backupdrill 2.4.0 → 2.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +113 -2
- package/dist/config.d.ts +10 -0
- package/dist/config.js +22 -1
- package/dist/deadline.d.ts +9 -0
- package/dist/deadline.js +29 -0
- package/dist/drill-record.d.ts +53 -0
- package/dist/drill-record.js +126 -0
- package/dist/exporter.d.ts +57 -0
- package/dist/exporter.js +305 -0
- package/dist/index.js +78 -6
- package/dist/ping.d.ts +31 -0
- package/dist/ping.js +217 -0
- package/dist/snapshots.d.ts +5 -0
- package/dist/snapshots.js +16 -8
- package/package.json +10 -2
- package/plugin/LICENSE +21 -0
- package/plugin/README.md +79 -0
- package/plugin/skills/supabase-backup/SKILL.md +137 -0
- package/plugin/skills/supabase-backup/assets/scheduled-backup.yml +119 -0
package/README.md
CHANGED
|
@@ -42,6 +42,16 @@ The [`plugin/`](plugin/) folder is a plugin for Claude Code and Cursor. It teach
|
|
|
42
42
|
|
|
43
43
|
Details are in [plugin/README.md](plugin/README.md).
|
|
44
44
|
|
|
45
|
+
## Use it from Pi
|
|
46
|
+
|
|
47
|
+
This npm package is also a Pi package. Installing it gives Pi the same skill the Claude Code plugin uses:
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
pi install npm:backupdrill
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
Start a new Pi session and ask it to back up your Supabase project or run a restore drill. The skill runs the CLI through `npx`, so Node.js 20 or newer has to be on your `PATH`. Everything else it needs, and where your keys go, is in [plugin/README.md](plugin/README.md).
|
|
54
|
+
|
|
45
55
|
## Usage
|
|
46
56
|
|
|
47
57
|
Configure via **env vars**, a **config file**, or **flags** (precedence: flags > env > file).
|
|
@@ -76,6 +86,7 @@ All recognized variables:
|
|
|
76
86
|
| `BACKUPDRILL_SUPABASE_STORAGE_*` | Storage-file source (endpoint/region/keys/buckets) — see [Backing up Storage files](#backing-up-storage-files-too) | — |
|
|
77
87
|
| `BACKUPDRILL_PG_DUMP` | Path to the `pg_dump` binary to use | `pg_dump` on `PATH` |
|
|
78
88
|
| `BACKUPDRILL_PG_RESTORE` | Path to `pg_restore` (used by `drill` and `restore`) | next to `BACKUPDRILL_PG_DUMP`, else `pg_restore` on `PATH` |
|
|
89
|
+
| `BACKUPDRILL_PING_URL` | Healthchecks.io ping URL for `backup` and `drill` — see [Alerts with Healthchecks.io](#alerts-with-healthchecksio) | — |
|
|
79
90
|
|
|
80
91
|
### Config file
|
|
81
92
|
|
|
@@ -147,7 +158,9 @@ backupdrill estimate --plan pro # measures your DB + Storage, prints the proje
|
|
|
147
158
|
<prefix>/<project>/<timestamp>/
|
|
148
159
|
├── dump.pgcustom # pg_dump --format=custom (public schema by default)
|
|
149
160
|
├── storage/<bucket>/<key> # your Storage files (if configured)
|
|
161
|
+
├── RECOVERY.md # how to restore this snapshot
|
|
150
162
|
└── manifest.json # schemas, tables, row counts, file list, sizes + sha256
|
|
163
|
+
<prefix>/<project>.drills/<finished-at>-<passed|failed>-<tag>.json # only with drill --record: one file per drill
|
|
151
164
|
```
|
|
152
165
|
|
|
153
166
|
## Restoring & drilling
|
|
@@ -224,10 +237,106 @@ Want this running on a schedule without paying for a server? Run it in GitHub Ac
|
|
|
224
237
|
2. Add these repo secrets (Settings → Secrets and variables → Actions):
|
|
225
238
|
- **Required:** `BACKUPDRILL_DATABASE_URL`, `BACKUPDRILL_S3_BUCKET`, `BACKUPDRILL_S3_ACCESS_KEY_ID`, `BACKUPDRILL_S3_SECRET_ACCESS_KEY`
|
|
226
239
|
- **Optional (destination):** `BACKUPDRILL_S3_ENDPOINT` (for R2/B2), `BACKUPDRILL_S3_REGION`, `BACKUPDRILL_PROJECT_NAME`
|
|
240
|
+
- **Optional (alerts):** `BACKUPDRILL_PING_URL` — see [Alerts with Healthchecks.io](#alerts-with-healthchecksio)
|
|
227
241
|
- **Optional (Storage files):** set all four of `BACKUPDRILL_SUPABASE_STORAGE_ENDPOINT`, `BACKUPDRILL_SUPABASE_STORAGE_REGION`, `BACKUPDRILL_SUPABASE_STORAGE_ACCESS_KEY_ID`, `BACKUPDRILL_SUPABASE_STORAGE_SECRET_ACCESS_KEY` to also back up Storage files; leave them unset to back up the database only.
|
|
228
242
|
|
|
229
243
|
Before switching to a **daily** schedule, read the egress-cost section above — every run is egress on your Supabase bill.
|
|
230
244
|
|
|
245
|
+
## Alerts with Healthchecks.io
|
|
246
|
+
|
|
247
|
+
A scheduled backup can fail quietly, or not run at all: a cron entry that never fires, a runner that dies, a workflow GitHub disabled. `backup` and `drill` can report every run to a [Healthchecks.io](https://healthchecks.io) check (a self-hosted Healthchecks works the same), and Healthchecks alerts you when a run fails or a success ping doesn't arrive on time.
|
|
248
|
+
|
|
249
|
+
Create a check whose schedule and grace time match your job, copy its ping URL, then:
|
|
250
|
+
|
|
251
|
+
```bash
|
|
252
|
+
export BACKUPDRILL_PING_URL="https://hc-ping.com/<uuid>"
|
|
253
|
+
backupdrill backup
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
`--ping-url <url>` and a `"pingUrl"` key in the config file work too (same precedence as everything else). Each run sends a start ping, then either a success or a failure ping:
|
|
257
|
+
|
|
258
|
+
- `<url>/start` before it begins, so Healthchecks shows how long the run took and alerts if it runs past the grace time;
|
|
259
|
+
- `<url>` when it succeeds, with a one-line summary in the body (tables, Storage files and the snapshot name; for `drill`, rows restored and restore time);
|
|
260
|
+
- `<url>/fail` when it fails, with the error in the body. For a drill that ran but didn't pass, that's the failed checks. Before it's sent, the CLI filters out the connection strings, passwords, hostnames, IP addresses, project refs and bucket keys it can recognise, plus the parts of Postgres errors that usually quote row values or SQL (`DETAIL`, `CONTEXT`, `Command was`). It's a filter, not a guarantee: a sensitive value it doesn't recognise can still reach Healthchecks. The full error stays in the job's own output.
|
|
261
|
+
|
|
262
|
+
Both pings of a run carry the same `rid`, so Healthchecks pairs them even if two runs overlap. If you schedule both `backup` and `drill`, give each its own check: on a shared check, the daily backup's success pings would keep it up while a weekly drill that stopped running went unnoticed.
|
|
263
|
+
|
|
264
|
+
A ping never changes the result of a run. Each request times out after 10 seconds and is retried twice on network errors and 5xx responses; if it still fails, the CLI prints a warning and carries on, with the same exit code it would have had without a ping URL.
|
|
265
|
+
|
|
266
|
+
## Prometheus metrics
|
|
267
|
+
|
|
268
|
+
`backupdrill exporter` serves [Prometheus](https://prometheus.io) metrics about the backups in your bucket, and about the restore drills you run with `--record`. It reads the bucket each time Prometheus scrapes it and never writes, so a key with read-only access to the bucket is enough.
|
|
269
|
+
|
|
270
|
+
```bash
|
|
271
|
+
# reads the same BACKUPDRILL_S3_* and BACKUPDRILL_PROJECT_NAME settings as backup and drill
|
|
272
|
+
backupdrill exporter # http://127.0.0.1:10059/metrics
|
|
273
|
+
backupdrill exporter --host 0.0.0.0 # when Prometheus runs on another machine
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
It listens on `127.0.0.1` unless you pass `--host`, because every scrape costs a few requests against your bucket and the port shouldn't end up open to the internet by accident. `--port` changes the port.
|
|
277
|
+
|
|
278
|
+
Drill results only show up if the drill leaves them in the bucket. Add `--record` to your drill job: when the drill finishes, it writes one small JSON file to `<prefix>/<project>.drills/`, named after the time it finished, whether it passed and a random tag (`2026-10-20T04-12-09-311Z-passed-3f9a1c07.json`). Each drill writes a file under its own name instead of updating a shared one, so the order in which two overlapping drills finish writing doesn't matter. Only the outcome, the time, the snapshot name and the restore time are stored, never the check details. A drill that stops early (Docker isn't running, the bucket is empty) is recorded as not passed. `--record` needs write access to the bucket; if the write fails, the drill prints a warning and its own result and exit code stay the same. Without `--record`, `drill` only reads the bucket, as before.
|
|
279
|
+
|
|
280
|
+
```bash
|
|
281
|
+
backupdrill drill --record
|
|
282
|
+
```
|
|
283
|
+
|
|
284
|
+
Point Prometheus at it. One exporter watches one project. The exporter doesn't add a project label, so set one in the scrape config:
|
|
285
|
+
|
|
286
|
+
```yaml
|
|
287
|
+
scrape_configs:
|
|
288
|
+
- job_name: backupdrill
|
|
289
|
+
scrape_interval: 1m
|
|
290
|
+
static_configs:
|
|
291
|
+
- targets: ["backup-host:10059"]
|
|
292
|
+
labels:
|
|
293
|
+
project: my-app
|
|
294
|
+
```
|
|
295
|
+
|
|
296
|
+
For a second project, run a second exporter with `--project-name` and a different `--port`.
|
|
297
|
+
|
|
298
|
+
| Metric | Meaning |
|
|
299
|
+
|---|---|
|
|
300
|
+
| `backupdrill_up` | 1 if the bucket was read, 0 if not (the reason goes to the exporter's log) |
|
|
301
|
+
| `backupdrill_scrape_duration_seconds` | How long reading the bucket took |
|
|
302
|
+
| `backupdrill_build_info{version}` | CLI version, always 1 |
|
|
303
|
+
| `backupdrill_snapshots` | Snapshot folders for the project, complete or not |
|
|
304
|
+
| `backupdrill_backup_last_success_timestamp_seconds` | When the newest complete snapshot finished (`createdAt` in its `manifest.json`); 0 if there is none |
|
|
305
|
+
| `backupdrill_latest_snapshot_dump_bytes` | Size of the database dump in that snapshot |
|
|
306
|
+
| `backupdrill_latest_snapshot_storage_files`, `_storage_bytes` | Storage files in that snapshot and their total size; 0 for a database-only backup |
|
|
307
|
+
| `backupdrill_latest_snapshot_tables`, `_estimated_rows` | Tables captured, and the row count Postgres estimated at backup time |
|
|
308
|
+
| `backupdrill_drill_last_run_timestamp_seconds` | When the last recorded drill finished, passed or not; 0 if none |
|
|
309
|
+
| `backupdrill_drill_last_run_passed` | 1 if that drill passed, 0 if not |
|
|
310
|
+
| `backupdrill_drill_last_success_timestamp_seconds` | When the last recorded drill that passed finished; 0 if none |
|
|
311
|
+
| `backupdrill_drill_last_success_restore_duration_seconds` | How long the database restore took in that drill |
|
|
312
|
+
|
|
313
|
+
A snapshot counts as complete once its `manifest.json` exists, because the backup writes it last. While there is no complete snapshot, the `latest_snapshot` metrics are left out. `backupdrill_drill_last_run_passed` appears once a drill has been recorded, and the restore duration once a recorded drill has passed. A manifest dated more than 10 minutes in the future counts as a failed read (`backupdrill_up 0`), and a drill file named with such a time is skipped with a warning in the log: either would hold back the alerts that work out an age from that timestamp. Check the clock on the machine that wrote it.
|
|
314
|
+
|
|
315
|
+
Alert rules to start from. Save them as `backupdrill-alerts.yml`, list that file under `rule_files` in `prometheus.yml` and reload Prometheus; `promtool check rules` passes on them:
|
|
316
|
+
|
|
317
|
+
```yaml
|
|
318
|
+
groups:
|
|
319
|
+
- name: backupdrill
|
|
320
|
+
rules:
|
|
321
|
+
- alert: BackupdrillBucketUnreadable
|
|
322
|
+
expr: backupdrill_up == 0
|
|
323
|
+
for: 15m
|
|
324
|
+
- alert: SupabaseBackupMissing
|
|
325
|
+
# daily backups; use 8 * 86400 for weekly
|
|
326
|
+
expr: time() - backupdrill_backup_last_success_timestamp_seconds > 26 * 3600
|
|
327
|
+
for: 15m
|
|
328
|
+
- alert: SupabaseBackupShrank
|
|
329
|
+
expr: backupdrill_latest_snapshot_dump_bytes < 0.7 * max_over_time(backupdrill_latest_snapshot_dump_bytes[7d])
|
|
330
|
+
- alert: SupabaseRestoreDrillFailed
|
|
331
|
+
expr: backupdrill_drill_last_run_passed == 0
|
|
332
|
+
- alert: SupabaseRestoreDrillMissing
|
|
333
|
+
# weekly drills
|
|
334
|
+
expr: time() - backupdrill_drill_last_success_timestamp_seconds > 8 * 86400
|
|
335
|
+
for: 1h
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
What a scrape costs: one list request per 1,000 snapshot folders and one per 1,000 drill files, plus reads of the newest `manifest.json` and the newest passing drill file. The manifest is fetched with `If-None-Match`, so when it hasn't changed the bucket answers `304 Not Modified` and nothing is downloaded. That matters once you back up Storage files, since the manifest lists every file. At a one-minute interval this comes to about 86,000 list requests and 86,000 small reads a month; check your storage provider's request pricing. Each read of the bucket stops half a second before Prometheus's scrape timeout (10 seconds if there's no Prometheus header), so a slow bucket shows up as `backupdrill_up 0` instead of a failed scrape.
|
|
339
|
+
|
|
231
340
|
## Roadmap
|
|
232
341
|
|
|
233
342
|
- [x] Database backup → your bucket, with checksummed manifest
|
|
@@ -237,6 +346,7 @@ Before switching to a **daily** schedule, read the egress-cost section above —
|
|
|
237
346
|
- [x] Storage-file integrity check inside the drill (checksums, sampled by default)
|
|
238
347
|
- [x] `restore` — recover a snapshot into a fresh Supabase project: database through the drill-grade engine with verification, Storage uploaded back with per-file reconciliation (local download remains an option)
|
|
239
348
|
- [x] GitHub Action wrapper
|
|
349
|
+
- [x] `exporter` — Prometheus metrics for backups and recorded drills
|
|
240
350
|
|
|
241
351
|
For scheduling, automated restore drills, and alerting, use the hosted service at [backupdrill.com](https://backupdrill.com).
|
|
242
352
|
|
|
@@ -245,9 +355,10 @@ For scheduling, automated restore drills, and alerting, use the hosted service a
|
|
|
245
355
|
`backupdrill` follows [semantic versioning](https://semver.org/). From **1.0.0** onwards the
|
|
246
356
|
following is the stable public surface — breaking it requires a major version bump:
|
|
247
357
|
|
|
248
|
-
- the commands (`backup`, `drill`, `restore`, `estimate`) and their flags
|
|
358
|
+
- the commands (`backup`, `drill`, `restore`, `estimate`, `exporter`) and their flags
|
|
249
359
|
- the `BACKUPDRILL_*` environment variables and the config-file keys
|
|
250
|
-
- the layout written to your bucket, and the shape of `manifest.json`
|
|
360
|
+
- the layout written to your bucket, and the shape of `manifest.json` and the drill record files
|
|
361
|
+
- the names and meaning of the metrics `exporter` serves
|
|
251
362
|
|
|
252
363
|
`manifest.json` carries a `schemaVersion` so the format can evolve safely. A snapshot written by
|
|
253
364
|
a newer release is **refused with a clear message** rather than parsed on a best-effort basis —
|
package/dist/config.d.ts
CHANGED
|
@@ -41,7 +41,17 @@ interface RawConfig {
|
|
|
41
41
|
schemas?: string[];
|
|
42
42
|
storage?: Partial<StorageConfig>;
|
|
43
43
|
supabaseStorage?: Partial<SupabaseStorageConfig>;
|
|
44
|
+
pingUrl?: string;
|
|
44
45
|
}
|
|
46
|
+
/**
|
|
47
|
+
* 监控 ping 网址(Healthchecks.io 那种):flag > 环境变量 > 配置文件,空串算没配
|
|
48
|
+
* (GitHub Actions 里没设的 secret 展开就是空串)。不放进 loadConfig:配置缺项时
|
|
49
|
+
* loadConfig 直接抛错,而这恰恰是要报 /fail 的情况,所以网址得先单独读出来。
|
|
50
|
+
*/
|
|
51
|
+
export declare function resolvePingUrl(opts: {
|
|
52
|
+
flag?: string;
|
|
53
|
+
configPath: string;
|
|
54
|
+
}): Promise<string | undefined>;
|
|
45
55
|
/** 合并优先级:显式 flag > 环境变量 > 配置文件。 */
|
|
46
56
|
export declare function loadConfig(opts: {
|
|
47
57
|
configPath: string;
|
package/dist/config.js
CHANGED
|
@@ -54,9 +54,30 @@ async function fromFile(path) {
|
|
|
54
54
|
catch (error) {
|
|
55
55
|
if (error.code === "ENOENT")
|
|
56
56
|
return {};
|
|
57
|
-
|
|
57
|
+
// JSON.parse 的报错会引用出错位置附近的原文,里面可能正好是密钥。本机日志里留着它方便排错;
|
|
58
|
+
// 发给监控(Healthchecks 会把它放进告警邮件)的只用 pingMessage 这句不带原文的
|
|
59
|
+
throw Object.assign(new Error(`Could not read config file ${path}: ${error.message}`), { pingMessage: `Could not read config file ${path}. The job's log has the details.` });
|
|
58
60
|
}
|
|
59
61
|
}
|
|
62
|
+
/**
|
|
63
|
+
* 监控 ping 网址(Healthchecks.io 那种):flag > 环境变量 > 配置文件,空串算没配
|
|
64
|
+
* (GitHub Actions 里没设的 secret 展开就是空串)。不放进 loadConfig:配置缺项时
|
|
65
|
+
* loadConfig 直接抛错,而这恰恰是要报 /fail 的情况,所以网址得先单独读出来。
|
|
66
|
+
*/
|
|
67
|
+
export async function resolvePingUrl(opts) {
|
|
68
|
+
let file = {};
|
|
69
|
+
try {
|
|
70
|
+
file = await fromFile(opts.configPath);
|
|
71
|
+
}
|
|
72
|
+
catch {
|
|
73
|
+
// 配置文件读不了时由 loadConfig 报错,这里只是拿不到文件里的网址
|
|
74
|
+
}
|
|
75
|
+
// 文件内容可能是 null、数组这类非对象的合法 JSON:那是 loadConfig 要报的错,这里只是拿不到网址
|
|
76
|
+
const filePingUrl = file && typeof file === "object" ? file.pingUrl : undefined;
|
|
77
|
+
return [opts.flag, process.env.BACKUPDRILL_PING_URL, filePingUrl]
|
|
78
|
+
.map((value) => (typeof value === "string" ? value.trim() : undefined))
|
|
79
|
+
.find(Boolean);
|
|
80
|
+
}
|
|
60
81
|
// spread 里显式的 undefined 会覆盖前一层的值(CLI 未传的 flag 正是 undefined),
|
|
61
82
|
// 合并前必须剔除,否则 env/配置文件里的 storage 配置会被整体清空。
|
|
62
83
|
function definedOnly(obj) {
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* 到点就放弃,不管 run 停没停。只把 AbortSignal 交给 SDK 不够:它在两次重试之间按
|
|
3
|
+
* Retry-After 睡觉时不看 signal,503 带 Retry-After: 2 就能把 1 秒的时限拖到 2 秒多。
|
|
4
|
+
* signal 仍然传下去;结束时(成功、失败、超时)一律 abort,让并行中还没回来的请求收手,
|
|
5
|
+
* 不然一个先失败、一个挂着的 Promise.all 会把挂着的那个连接一直留着。
|
|
6
|
+
*/
|
|
7
|
+
export declare function withDeadline<T>(run: (signal: AbortSignal) => Promise<T>, ms: number): Promise<T>;
|
|
8
|
+
/** 超时类错误(withDeadline 的、AbortSignal 的)统一认出来,好给一句人话 */
|
|
9
|
+
export declare function isTimeout(error: unknown): boolean;
|
package/dist/deadline.js
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* 到点就放弃,不管 run 停没停。只把 AbortSignal 交给 SDK 不够:它在两次重试之间按
|
|
3
|
+
* Retry-After 睡觉时不看 signal,503 带 Retry-After: 2 就能把 1 秒的时限拖到 2 秒多。
|
|
4
|
+
* signal 仍然传下去;结束时(成功、失败、超时)一律 abort,让并行中还没回来的请求收手,
|
|
5
|
+
* 不然一个先失败、一个挂着的 Promise.all 会把挂着的那个连接一直留着。
|
|
6
|
+
*/
|
|
7
|
+
export async function withDeadline(run, ms) {
|
|
8
|
+
const controller = new AbortController();
|
|
9
|
+
let timer;
|
|
10
|
+
const deadline = new Promise((_, reject) => {
|
|
11
|
+
timer = setTimeout(() => {
|
|
12
|
+
const error = new Error(`timed out after ${ms}ms`);
|
|
13
|
+
error.name = "TimeoutError";
|
|
14
|
+
reject(error);
|
|
15
|
+
}, ms);
|
|
16
|
+
});
|
|
17
|
+
try {
|
|
18
|
+
return await Promise.race([run(controller.signal), deadline]);
|
|
19
|
+
}
|
|
20
|
+
finally {
|
|
21
|
+
clearTimeout(timer);
|
|
22
|
+
controller.abort();
|
|
23
|
+
}
|
|
24
|
+
}
|
|
25
|
+
/** 超时类错误(withDeadline 的、AbortSignal 的)统一认出来,好给一句人话 */
|
|
26
|
+
export function isTimeout(error) {
|
|
27
|
+
const name = error?.name;
|
|
28
|
+
return name === "TimeoutError" || name === "AbortError";
|
|
29
|
+
}
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
import { type S3Client } from "@aws-sdk/client-s3";
|
|
2
|
+
import { type BackupConfig } from "./config.js";
|
|
3
|
+
import type { DrillReport } from "./drill.js";
|
|
4
|
+
/**
|
|
5
|
+
* `drill --record` 写进桶里的演练结论。`exporter` 只读桶,演练结果不落桶它就无从得知。
|
|
6
|
+
*
|
|
7
|
+
* 每次演练写一个新对象,从不覆盖:
|
|
8
|
+
* <prefix>/<project>.drills/<结束时间>-<passed|failed>-<8 位随机>.json
|
|
9
|
+
* 为什么不是一份可覆盖的"最近一次":两次演练重叠时,先结束的那次可能后写进桶,把更新的
|
|
10
|
+
* "没通过"盖成更早的"通过",失败告警就没了;先读后写也挡不住(不是原子的),而各家
|
|
11
|
+
* S3 兼容存储对条件写入的支持参差不齐。每次一个对象,"哪次最新"留给读的一方按名字判断,
|
|
12
|
+
* 写入顺序就无所谓了。
|
|
13
|
+
*
|
|
14
|
+
* 为什么放在 <project>.drills/,和快照目录并排而不是在它里面:resolveSnapshot(包括已经发出去的
|
|
15
|
+
* 旧版本)按字典序取 <project>/ 下最大的目录当最新快照,里面多一个 drills/ 会被当成快照。
|
|
16
|
+
*
|
|
17
|
+
* 文件名本身带着结束时间和结论,exporter 列一下目录就知道最近一次和最近一次通过是哪个;
|
|
18
|
+
* 只为了恢复用时才读一次通过的那份。只记结论,不记检查明细:失败明细里可能有 pg_restore
|
|
19
|
+
* 回显的表数据,而桶的读权限可能比演练日志给得更宽(exporter 只要只读 key)。
|
|
20
|
+
*/
|
|
21
|
+
/** 版本规则同 manifest:缺省不接受,更高的版本明确报错让用户升级,不按旧结构硬读。 */
|
|
22
|
+
export declare const DRILL_RECORD_SCHEMA_VERSION = 1;
|
|
23
|
+
export interface DrillRecord {
|
|
24
|
+
schemaVersion: number;
|
|
25
|
+
tool: "backupdrill-cli";
|
|
26
|
+
toolVersion: string;
|
|
27
|
+
/** 演练结束的时间(ISO8601),与文件名里的时间相同 */
|
|
28
|
+
finishedAt: string;
|
|
29
|
+
pass: boolean;
|
|
30
|
+
/** 演练的快照名;演练在选定快照之前就失败了(没装 Docker、桶里没有快照)时没有 */
|
|
31
|
+
snapshot?: string;
|
|
32
|
+
/** 恢复用时,只有通过的演练才记:没通过时这个数可能只是半程,拿来估恢复时间会误导 */
|
|
33
|
+
restoreSeconds?: number;
|
|
34
|
+
}
|
|
35
|
+
/** 演练记录所在的"目录",带尾斜杠 */
|
|
36
|
+
export declare function drillRecordsPrefix(config: BackupConfig): string;
|
|
37
|
+
export declare function drillRecordName(record: Pick<DrillRecord, "finishedAt" | "pass">, unique?: string): string;
|
|
38
|
+
/** 从文件名读出结束时间(毫秒)和结论;不是记录文件的名字返回 null */
|
|
39
|
+
export declare function parseDrillRecordName(name: string): {
|
|
40
|
+
finishedAt: number;
|
|
41
|
+
pass: boolean;
|
|
42
|
+
} | null;
|
|
43
|
+
/** report 为 null = 演练中途抛错、没有报告;这也算一次没通过的演练。 */
|
|
44
|
+
export declare function buildDrillRecord(report: DrillReport | null, finishedAt: Date): DrillRecord;
|
|
45
|
+
/** 写一个新对象,返回它的 key。不读桶:只需要写权限 */
|
|
46
|
+
export declare function writeDrillRecord(s3: S3Client, config: BackupConfig, record: DrillRecord, abortSignal?: AbortSignal): Promise<string>;
|
|
47
|
+
/**
|
|
48
|
+
* CLI 的 `drill --record`:写不进去只警告,演练本身的结论和退出码不变。
|
|
49
|
+
* 常见原因是给演练配的是只读 key;这种情况下 exporter 的演练时间会一直不动,告警规则会发现。
|
|
50
|
+
*/
|
|
51
|
+
export declare function recordDrillOutcome(config: BackupConfig, report: DrillReport | null, s3?: S3Client, timeoutMs?: number): Promise<void>;
|
|
52
|
+
/** 读桶里的演练记录。结构不对就报错:exporter 宁可报 up=0,也不报一个读错的结论。 */
|
|
53
|
+
export declare function parseDrillRecord(json: string, file: string): DrillRecord;
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
import { randomBytes } from "node:crypto";
|
|
2
|
+
import { PutObjectCommand } from "@aws-sdk/client-s3";
|
|
3
|
+
import { requireBucket } from "./config.js";
|
|
4
|
+
import { isTimeout, withDeadline } from "./deadline.js";
|
|
5
|
+
import { log } from "./log.js";
|
|
6
|
+
import { targetClient } from "./snapshots.js";
|
|
7
|
+
import { TOOL_VERSION } from "./version.js";
|
|
8
|
+
/**
|
|
9
|
+
* `drill --record` 写进桶里的演练结论。`exporter` 只读桶,演练结果不落桶它就无从得知。
|
|
10
|
+
*
|
|
11
|
+
* 每次演练写一个新对象,从不覆盖:
|
|
12
|
+
* <prefix>/<project>.drills/<结束时间>-<passed|failed>-<8 位随机>.json
|
|
13
|
+
* 为什么不是一份可覆盖的"最近一次":两次演练重叠时,先结束的那次可能后写进桶,把更新的
|
|
14
|
+
* "没通过"盖成更早的"通过",失败告警就没了;先读后写也挡不住(不是原子的),而各家
|
|
15
|
+
* S3 兼容存储对条件写入的支持参差不齐。每次一个对象,"哪次最新"留给读的一方按名字判断,
|
|
16
|
+
* 写入顺序就无所谓了。
|
|
17
|
+
*
|
|
18
|
+
* 为什么放在 <project>.drills/,和快照目录并排而不是在它里面:resolveSnapshot(包括已经发出去的
|
|
19
|
+
* 旧版本)按字典序取 <project>/ 下最大的目录当最新快照,里面多一个 drills/ 会被当成快照。
|
|
20
|
+
*
|
|
21
|
+
* 文件名本身带着结束时间和结论,exporter 列一下目录就知道最近一次和最近一次通过是哪个;
|
|
22
|
+
* 只为了恢复用时才读一次通过的那份。只记结论,不记检查明细:失败明细里可能有 pg_restore
|
|
23
|
+
* 回显的表数据,而桶的读权限可能比演练日志给得更宽(exporter 只要只读 key)。
|
|
24
|
+
*/
|
|
25
|
+
/** 版本规则同 manifest:缺省不接受,更高的版本明确报错让用户升级,不按旧结构硬读。 */
|
|
26
|
+
export const DRILL_RECORD_SCHEMA_VERSION = 1;
|
|
27
|
+
/** 演练记录所在的"目录",带尾斜杠 */
|
|
28
|
+
export function drillRecordsPrefix(config) {
|
|
29
|
+
const storage = requireBucket(config);
|
|
30
|
+
return `${storage.prefix}/${config.projectName}.drills/`;
|
|
31
|
+
}
|
|
32
|
+
/**
|
|
33
|
+
* 时间戳写法同快照目录名(把 ISO 里的 : 和 . 换成 -),定长,按名字排序就是按时间排序。
|
|
34
|
+
* 末尾的随机段:两次演练在同一毫秒结束、结论又相同时,只靠时间和结论会撞名、后写的覆盖先写的
|
|
35
|
+
*/
|
|
36
|
+
const RECORD_NAME = /^(\d{4}-\d{2}-\d{2})T(\d{2})-(\d{2})-(\d{2})-(\d{3})Z-(passed|failed)-[0-9a-f]{8}\.json$/;
|
|
37
|
+
export function drillRecordName(record, unique = randomBytes(4).toString("hex")) {
|
|
38
|
+
return `${record.finishedAt.replace(/[:.]/g, "-")}-${record.pass ? "passed" : "failed"}-${unique}.json`;
|
|
39
|
+
}
|
|
40
|
+
/** 从文件名读出结束时间(毫秒)和结论;不是记录文件的名字返回 null */
|
|
41
|
+
export function parseDrillRecordName(name) {
|
|
42
|
+
const m = RECORD_NAME.exec(name);
|
|
43
|
+
if (!m)
|
|
44
|
+
return null;
|
|
45
|
+
const finishedAt = Date.parse(`${m[1]}T${m[2]}:${m[3]}:${m[4]}.${m[5]}Z`);
|
|
46
|
+
return Number.isFinite(finishedAt) ? { finishedAt, pass: m[6] === "passed" } : null;
|
|
47
|
+
}
|
|
48
|
+
/** report 为 null = 演练中途抛错、没有报告;这也算一次没通过的演练。 */
|
|
49
|
+
export function buildDrillRecord(report, finishedAt) {
|
|
50
|
+
return {
|
|
51
|
+
schemaVersion: DRILL_RECORD_SCHEMA_VERSION,
|
|
52
|
+
tool: "backupdrill-cli",
|
|
53
|
+
toolVersion: TOOL_VERSION,
|
|
54
|
+
finishedAt: finishedAt.toISOString(),
|
|
55
|
+
pass: report?.pass === true,
|
|
56
|
+
...(report ? { snapshot: report.snapshot } : {}),
|
|
57
|
+
...(report?.pass ? { restoreSeconds: report.restoreSeconds } : {}),
|
|
58
|
+
};
|
|
59
|
+
}
|
|
60
|
+
/** 写一个新对象,返回它的 key。不读桶:只需要写权限 */
|
|
61
|
+
export async function writeDrillRecord(s3, config, record, abortSignal) {
|
|
62
|
+
const key = drillRecordsPrefix(config) + drillRecordName(record);
|
|
63
|
+
await s3.send(new PutObjectCommand({
|
|
64
|
+
Bucket: requireBucket(config).bucket,
|
|
65
|
+
Key: key,
|
|
66
|
+
Body: JSON.stringify(record, null, 2),
|
|
67
|
+
ContentType: "application/json",
|
|
68
|
+
}), { abortSignal });
|
|
69
|
+
return key;
|
|
70
|
+
}
|
|
71
|
+
/** 记一笔最多等这么久:桶接了请求却不回应时,演练不能卡在这里,后面还要退出、发监控 ping */
|
|
72
|
+
const RECORD_TIMEOUT_MS = 30_000;
|
|
73
|
+
/**
|
|
74
|
+
* CLI 的 `drill --record`:写不进去只警告,演练本身的结论和退出码不变。
|
|
75
|
+
* 常见原因是给演练配的是只读 key;这种情况下 exporter 的演练时间会一直不动,告警规则会发现。
|
|
76
|
+
*/
|
|
77
|
+
export async function recordDrillOutcome(config, report, s3 = targetClient(config), timeoutMs = RECORD_TIMEOUT_MS) {
|
|
78
|
+
const record = buildDrillRecord(report, new Date());
|
|
79
|
+
try {
|
|
80
|
+
const key = await withDeadline((signal) => writeDrillRecord(s3, config, record, signal), timeoutMs);
|
|
81
|
+
log.ok(`Drill result recorded → s3://${requireBucket(config).bucket}/${key}`);
|
|
82
|
+
}
|
|
83
|
+
catch (error) {
|
|
84
|
+
const reason = isTimeout(error)
|
|
85
|
+
? `no answer from the bucket within ${timeoutMs / 1000}s`
|
|
86
|
+
: error.message;
|
|
87
|
+
log.warn(`Could not record the drill result (${reason}). ` +
|
|
88
|
+
`--record needs write access to the bucket. The drill's own result is unchanged.`);
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
function malformed(file, what) {
|
|
92
|
+
throw new Error(`${file} is malformed: ${what}`);
|
|
93
|
+
}
|
|
94
|
+
/** 读桶里的演练记录。结构不对就报错:exporter 宁可报 up=0,也不报一个读错的结论。 */
|
|
95
|
+
export function parseDrillRecord(json, file) {
|
|
96
|
+
let parsed;
|
|
97
|
+
try {
|
|
98
|
+
parsed = JSON.parse(json);
|
|
99
|
+
}
|
|
100
|
+
catch (error) {
|
|
101
|
+
throw new Error(`${file} is not valid JSON: ${error.message}`);
|
|
102
|
+
}
|
|
103
|
+
if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) {
|
|
104
|
+
malformed(file, "root is not an object");
|
|
105
|
+
}
|
|
106
|
+
const r = parsed;
|
|
107
|
+
if (typeof r.schemaVersion !== "number" || !Number.isInteger(r.schemaVersion) || r.schemaVersion < 1) {
|
|
108
|
+
malformed(file, `schemaVersion ${JSON.stringify(r.schemaVersion)}`);
|
|
109
|
+
}
|
|
110
|
+
if (r.schemaVersion > DRILL_RECORD_SCHEMA_VERSION) {
|
|
111
|
+
throw new Error(`${file} uses schema version ${r.schemaVersion}, but this BackupDrill understands up to ` +
|
|
112
|
+
`${DRILL_RECORD_SCHEMA_VERSION}. Upgrade the CLI (npm i -g backupdrill@latest) to read it.`);
|
|
113
|
+
}
|
|
114
|
+
if (typeof r.finishedAt !== "string" || !Number.isFinite(Date.parse(r.finishedAt))) {
|
|
115
|
+
malformed(file, "finishedAt");
|
|
116
|
+
}
|
|
117
|
+
if (typeof r.pass !== "boolean")
|
|
118
|
+
malformed(file, "pass");
|
|
119
|
+
if (r.snapshot !== undefined && typeof r.snapshot !== "string")
|
|
120
|
+
malformed(file, "snapshot");
|
|
121
|
+
if (r.restoreSeconds !== undefined &&
|
|
122
|
+
(typeof r.restoreSeconds !== "number" || !Number.isFinite(r.restoreSeconds) || r.restoreSeconds < 0)) {
|
|
123
|
+
malformed(file, "restoreSeconds");
|
|
124
|
+
}
|
|
125
|
+
return r;
|
|
126
|
+
}
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
import { type Server } from "node:http";
|
|
2
|
+
import { type S3Client } from "@aws-sdk/client-s3";
|
|
3
|
+
import { type BackupConfig } from "./config.js";
|
|
4
|
+
/**
|
|
5
|
+
* `backupdrill exporter`:按 Prometheus 文本格式暴露桶里备份与演练的状态。
|
|
6
|
+
* 规矩照 https://prometheus.io/docs/instrumenting/writing_exporters/ :
|
|
7
|
+
* - 每次抓取现读桶,不按自己的定时器预取,不报旧值;
|
|
8
|
+
* - 读不了桶时照样回 200,backupdrill_up 为 0,其余业务指标不出(半套数据比没有更误导);
|
|
9
|
+
* - 指标不带标签(一个 exporter 进程只看一个项目,项目名属于 Prometheus 那边的 target 标签),
|
|
10
|
+
* 连接串、key、桶名一律不进指标。
|
|
11
|
+
*/
|
|
12
|
+
/** 10059:Prometheus wiki「Default port allocations」上 10058 之后的第一个空位(2026-10-20 查)。 */
|
|
13
|
+
export declare const DEFAULT_EXPORTER_PORT = 10059;
|
|
14
|
+
/** 最新一份完整快照里 exporter 要用的数,不留整份 manifest(Storage 文件多时 manifest 很大) */
|
|
15
|
+
export interface SnapshotSummary {
|
|
16
|
+
createdAt: number;
|
|
17
|
+
dumpBytes: number;
|
|
18
|
+
storageFiles: number;
|
|
19
|
+
storageBytes: number;
|
|
20
|
+
tables: number;
|
|
21
|
+
estimatedRows: number;
|
|
22
|
+
}
|
|
23
|
+
export interface BucketState {
|
|
24
|
+
/** <prefix>/<project>/ 下的快照目录数,含没写完 manifest 的 */
|
|
25
|
+
snapshots: number;
|
|
26
|
+
/** 最新一份有 manifest.json 的快照;一份都没有时为 null */
|
|
27
|
+
latest: SnapshotSummary | null;
|
|
28
|
+
/** 最近一次记下的演练(drill --record);没有时为 null */
|
|
29
|
+
lastDrill: {
|
|
30
|
+
finishedAt: number;
|
|
31
|
+
pass: boolean;
|
|
32
|
+
} | null;
|
|
33
|
+
/** 最近一次通过的演练;restoreSeconds 来自它的记录内容 */
|
|
34
|
+
lastPassedDrill: {
|
|
35
|
+
finishedAt: number;
|
|
36
|
+
restoreSeconds?: number;
|
|
37
|
+
} | null;
|
|
38
|
+
}
|
|
39
|
+
/**
|
|
40
|
+
* 读桶的函数。每次调用都现读;唯一记住的是最新一份 manifest 的 ETag 和摘要:
|
|
41
|
+
* 下次带 If-None-Match 去问,没变就是 304、不传正文。快照写完就不再改,而一份 manifest 列着
|
|
42
|
+
* 每个 Storage 文件,可能有几十 MB,每 30 秒整份下载一遍会变成用户账单上的流量。
|
|
43
|
+
*/
|
|
44
|
+
export declare function createBucketReader(s3: S3Client, config: BackupConfig): (abortSignal: AbortSignal) => Promise<BucketState>;
|
|
45
|
+
/** state 为 null = 这次没读成桶。全部是 gauge:都是"某一刻的状态",没有一个会只增不减。 */
|
|
46
|
+
export declare function renderMetrics(state: BucketState | null, scrapeSeconds: number): string;
|
|
47
|
+
/**
|
|
48
|
+
* 读桶的时限跟着 Prometheus 走:它在请求头里告诉我们这次抓取最多等多久,留半秒余量,
|
|
49
|
+
* 好让超时的这次还能回一个 up=0,而不是被 Prometheus 当成 exporter 挂了。
|
|
50
|
+
*/
|
|
51
|
+
export declare function scrapeTimeoutMs(header: string | string[] | undefined): number;
|
|
52
|
+
export declare function createExporterServer(read: (abortSignal: AbortSignal) => Promise<BucketState>): Server;
|
|
53
|
+
/** CLI 的 `backupdrill exporter`:起 HTTP 服务,直到进程被停掉。 */
|
|
54
|
+
export declare function startExporter(config: BackupConfig, opts: {
|
|
55
|
+
host: string;
|
|
56
|
+
port: number;
|
|
57
|
+
}): Promise<Server>;
|