backupproof 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Calas Automations Inc.
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,334 @@
1
+ Metadata-Version: 2.4
2
+ Name: backupproof
3
+ Version: 0.2.0
4
+ Summary: Backup restore testing: restore the latest restic/borg/Proxmox backup into a throwaway target, boot it, run smoke checks, report PASS or FAIL with proof
5
+ Author: Calas Automations Inc.
6
+ License: MIT
7
+ Keywords: backup,restore,restic,borg,proxmox,testing
8
+ Classifier: Development Status :: 3 - Alpha
9
+ Classifier: Environment :: Console
10
+ Classifier: Intended Audience :: System Administrators
11
+ Classifier: License :: OSI Approved :: MIT License
12
+ Classifier: Programming Language :: Python :: 3
13
+ Classifier: Topic :: System :: Archiving :: Backup
14
+ Requires-Python: >=3.9
15
+ Description-Content-Type: text/markdown
16
+ License-File: LICENSE
17
+ Dynamic: license-file
18
+
19
+ # backupproof — we prove your backup weekly
20
+
21
+ The #1 backup failure is not "no backup" — it's "backup never
22
+ tested, restore fails." backupproof takes the latest restic, borg,
23
+ or Proxmox backup, restores it into a throwaway target, verifies
24
+ it, and reports PASS or FAIL **with proof**: which checks passed,
25
+ which failed, and the evidence for each.
26
+
27
+ One verdict, three smoke packs only — web, DB, files. Nothing else.
28
+
29
+ **What "verify" means depends on the source — and the report says
30
+ which (v0.2.0 honesty fix):** restic/borg backups are file-level,
31
+ so the restored *files* are content-checked; there is no disk
32
+ image to boot, and the PASS line says so. Proxmox backups restore
33
+ into a throwaway VMID on the Proxmox host and are probed there
34
+ (web pack via a probe URL). A PASS line never claims a boot that
35
+ did not happen.
36
+
37
+ ## Install (local)
38
+
39
+ ```sh
40
+ pip install .
41
+ # or, for development:
42
+ pip install -e .
43
+ # or without installing, from this directory:
44
+ PYTHONPATH=. python3 -m backupproof --help
45
+ ```
46
+
47
+ Requires Python 3.9+. Stdlib only — no third-party dependencies,
48
+ no keys, no accounts, no telemetry, no phone-home.
49
+
50
+ ## Usage
51
+
52
+ ```sh
53
+ # The verdict: latest backup -> restore -> boot -> smoke -> PASS/FAIL
54
+ backupproof run --config backupproof.json
55
+ backupproof run --config backupproof.json --json --report proof.json
56
+
57
+ # List backups for a source (newest first)
58
+ backupproof list --config backupproof.json
59
+ backupproof list --source-type restic --repo /srv/backups --json
60
+
61
+ # Validate a config without restoring or booting anything
62
+ backupproof validate --config backupproof.json
63
+
64
+ # Write example config + fixture files (runnable: run --config <dir>/config.json)
65
+ backupproof init --output backupproof-examples
66
+ ```
67
+
68
+ Exit codes, CI-usable: **0 = PASS, 1 = FAIL (a check failed),
69
+ 2 = ERROR** (no backup found, restore failure, boot failure or
70
+ timeout, bad config, missing tool). `--json` works on all commands.
71
+
72
+ ### Config
73
+
74
+ A JSON file naming the source and the smoke checks:
75
+
76
+ ```json
77
+ {
78
+ "source": {"type": "restic", "repo": "/srv/backups"},
79
+ "checks": {
80
+ "web": [{"name": "homepage", "path": "/",
81
+ "expect_status": 200, "expect_contains": "Welcome"}],
82
+ "db": [{"name": "users table", "db_file": "app.db",
83
+ "query": "SELECT count(*) FROM users", "expect_rows_min": 1}],
84
+ "files": [{"name": "config", "path": "etc/app.conf",
85
+ "sha256": "<hex>", "size_min": 10}]
86
+ }
87
+ }
88
+ ```
89
+
90
+ Source types: `restic` (`repo`), `borg` (`repo`), `proxmox`
91
+ (`dump_dir` — a vzdump directory), `fixture` (`dir` — see below).
92
+ DB checks run the query against a SQLite file in the restored
93
+ target (stdlib `sqlite3`), or run a `command` (e.g. `psql`) and
94
+ check its output. Web checks fetch a real URL: `path` is resolved
95
+ against the served/restored target's base URL, or give a full `url`.
96
+ Files checks take `path` plus any of `sha256`, `size_min`,
97
+ `size_max`, `size_exact`, and `max_age_days`: the restored file's
98
+ modification time (as preserved by restic/borg) must be within N
99
+ days of the **backup's timestamp**, not of now — catches a nightly
100
+ dump file that stopped updating inside a fresh backup (`FILE_STALE`,
101
+ both dates reported). An unknown backup timestamp is a FAIL, never
102
+ a pass.
103
+ Optional `"mode": "0640"` and `"owner": "postgres"` compare the
104
+ restored file's permission bits and owning user (`pwd`)
105
+ (`FILE_MODE_MISMATCH` / `FILE_OWNER_MISMATCH`). On non-POSIX
106
+ hosts (Windows) both are a FAIL "not verifiable on this OS", never
107
+ a pass. Note: restic and borg restore file ownership only when run
108
+ as root; run as another user, restored files belong to that user.
109
+
110
+ ### Per-source fields
111
+
112
+ - `restic` / `borg`: `repo` (required). Optional `serve_dir` —
113
+ a directory inside the restored tree (e.g. `"webroot"`) that
114
+ backupproof serves over real localhost HTTP, so web checks can
115
+ hit the *restored* content (labelled "restored content, not a
116
+ VM" in the report). Without `serve_dir`, web checks need full
117
+ `url` values. **Passwords pass through the operator's env:**
118
+ `RESTIC_PASSWORD` for restic, `BORG_PASSPHRASE` for borg —
119
+ backupproof adds no credential handling of its own.
120
+ - `restic` / `borg` optional `"restore": "checked-paths"`
121
+ (default `"full"`): restore only the paths the checks read —
122
+ files-check `path`s, db `db_file`s and `serve_dir` (restic:
123
+ repeated `--include /<path>`; borg: `extract` with path
124
+ arguments). The restore line and the PASS line then say
125
+ `partial (N paths) — only checked paths restored; the rest of
126
+ the backup was not restore-tested`. A config with no
127
+ path-bearing check is a `CONFIG_ERROR`. Opt-in only: a partial
128
+ restore does not prove the whole backup restores.
129
+ - `proxmox`: `dump_dir` (required), `throwaway_vmid` (REQUIRED —
130
+ backupproof refuses to guess a VMID it will later destroy),
131
+ `storage` (default `"local"`). The restore runs `qmrestore` /
132
+ `pct restore` into the throwaway VMID, starts it, probes it
133
+ (with `--probe-url`), then stops and destroys **only that
134
+ VMID**. A pre-flight `qm/pct status <vmid>` refuses the run if
135
+ the VMID already exists. **Pack guard:** db/files packs cannot
136
+ verify a VM's filesystem from the host, so a Proxmox real-mode
137
+ config containing them is an ERROR (`PROXMOX_PACK_UNSUPPORTED`,
138
+ exit 2) naming the packs and the remedy (web checks with
139
+ `--probe-url`, or run backupproof inside the guest) — packs are
140
+ never silently dropped and PASSed.
141
+
142
+ ### Repository integrity step (opt-in)
143
+
144
+ `--integrity` runs `restic check` (add `--read-data-subset PCT`
145
+ to also read that share of the pack data) or `borg check
146
+ --repository-only` (`--verify-data` for `borg check
147
+ --verify-data`) **before** restoring. A failed check is an ERROR
148
+ (`INTEGRITY_FAILED`, exit 2) carrying the tool's last 20 output
149
+ lines; a passed one adds `integrity: {tool, args, ok}` to the
150
+ report. Cost warning: restic's documentation notes that reading
151
+ pack data downloads it from the repository, which is slow and can
152
+ cost money on cloud storage. Plain `restic check` (no subset)
153
+ checks structure only. Use a small subset, not full reads, weekly.
154
+
155
+ ### Dead-man's-switch ping (opt-in)
156
+
157
+ `--ping-url URL` sends one HTTP GET after the verdict: `URL` on
158
+ PASS, `URL/fail` on FAIL or ERROR — the healthchecks.io
159
+ convention, so a missed weekly run also alerts. 10-second
160
+ timeout. A ping failure prints a warning on stderr and never
161
+ changes the exit code. Nothing is sent unless the flag is given.
162
+
163
+ ### Staleness gate
164
+
165
+ Top-level `max_age_days` (or `--max-age-days`, CLI wins): when
166
+ the latest backup is older than that, the verdict carries a
167
+ warning. With `--fail-on-stale`, the run refuses outright
168
+ (`STALE_BACKUP`, exit 2) before restoring anything. A backup
169
+ whose timestamp is missing or unparseable warns ("cannot
170
+ determine age") and never fails on staleness alone.
171
+
172
+ ### Which packs verify what, per source
173
+
174
+ | Source | web pack | db pack | files pack |
175
+ |---|---|---|---|
176
+ | restic / borg (real) | served restored tree (`serve_dir`) or explicit URL | restored files | restored files |
177
+ | proxmox (real) | booted VM via probe URL | **refused** (pack guard) | **refused** (pack guard) |
178
+ | fixture (simulated) | served fixture webroot over real HTTP | fixture tree | fixture tree |
179
+
180
+ The report's `verified against:` line states this per run, and
181
+ `--report` JSON carries `boot_kind`, `boot_probed`,
182
+ `verified_against`, `warnings`, `restore_partial`, `integrity`
183
+ and `restore_seconds` fields. `restore_seconds` is the wall time
184
+ of the restore step only (not boot or probe) — RTO evidence; the
185
+ text shows `restore: … in 37.2s`. In simulated (fixture) mode it
186
+ times the fixture copy and is labelled `(simulated)`.
187
+
188
+ A run with no checks configured is an ERROR, never a PASS. A
189
+ check that cannot be verified is a FAIL, never skipped silently.
190
+
191
+ ### Example PASS output
192
+
193
+ ```
194
+ PASS (simulated): backup restores and boots in simulation, all smoke checks passed
195
+ backup: fixture snap-20261028 (2026-10-28T03:00:00Z)
196
+ mode: simulated (SIMULATED restore/boot — not a real VM boot)
197
+ restore: SIMULATED restore into /tmp/backupproof-yoi9_m9l/restored (3 files copied from fixture)
198
+ boot: SIMULATED boot completed in 42s (fixture boot.json; not a real VM boot); serving webroot webroot at http://127.0.0.1:34147
199
+ verified against: restored fixture tree (web pack served over real HTTP from the restored webroot)
200
+ PASS [web] homepage: HTTP 200, body contains 'Welcome'
201
+ PASS [db] users: query returned 1 row(s): [('alice',)]
202
+ PASS [files] config: app.conf: 10 bytes, sha256 matches
203
+ ```
204
+
205
+ A real restic/borg run reports instead:
206
+ `PASS: backup restores, all smoke checks passed (file-level backup — content verified, no VM boot)` —
207
+ and a Proxmox run with no probe URL reports
208
+ `PASS: backup restores, VM started (unprobed), all smoke checks passed`.
209
+
210
+ ### Example FAIL output (failing check named)
211
+
212
+ Change the restored homepage so it no longer contains "Welcome"
213
+ and the same run reports:
214
+
215
+ ```
216
+ FAIL: 1 of 3 smoke check(s) failed
217
+ backup: fixture snap-20261028 (2026-10-28T03:00:00Z)
218
+ mode: simulated (SIMULATED restore/boot — not a real VM boot)
219
+ restore: SIMULATED restore into /tmp/backupproof-y8b2y4d3/restored (3 files copied from fixture)
220
+ boot: SIMULATED boot completed in 42s (fixture boot.json; not a real VM boot); serving webroot webroot at http://127.0.0.1:45629
221
+ FAIL [web] homepage (WEB_CONTENT_MISSING): HTTP 200 but body lacks 'Welcome' (first 200 chars: 'Something else entirely\n')
222
+ PASS [db] users: query returned 1 row(s): [('alice',)]
223
+ PASS [files] config: app.conf: 10 bytes, sha256 matches
224
+ ```
225
+
226
+ Named failures: web — `WEB_UNREACHABLE`, `WEB_STATUS_MISMATCH`,
227
+ `WEB_CONTENT_MISSING`; DB — `DB_FILE_MISSING`, `DB_QUERY_FAILED`,
228
+ `DB_NO_ROWS`, `DB_CONTENT_MISSING`, `DB_COMMAND_FAILED`; files —
229
+ `FILE_MISSING`, `FILE_SIZE_MISMATCH`, `FILE_HASH_MISMATCH`,
230
+ `FILE_STALE`, `FILE_MODE_MISMATCH`, `FILE_OWNER_MISMATCH`.
231
+ Errors (exit 2): `NO_BACKUP_FOUND`, `RESTORE_FAILED`,
232
+ `BOOT_FAILED`, `BOOT_TIMEOUT`, `MISSING_TOOL`, `CONFIG_ERROR`,
233
+ `INTEGRITY_FAILED`.
234
+
235
+ ### Try it (fixture example, no backup tools needed)
236
+
237
+ ```sh
238
+ backupproof run --config examples/config.json
239
+ ```
240
+
241
+ The example fixture restores a small tree (webroot, `app.conf`,
242
+ SQLite `app.db`) and simulates boot while genuinely serving the
243
+ webroot over real localhost HTTP, so the web check is really
244
+ exercised.
245
+
246
+ ## Weekly cadence (doc example only — no built scheduler)
247
+
248
+ backupproof has no daemon and no built-in scheduler by design.
249
+ Run it weekly from cron or a systemd timer. Cron example (Sundays
250
+ 03:30, proof report kept per run):
251
+
252
+ ```cron
253
+ 30 3 * * 0 /usr/local/bin/backupproof run --config /etc/backupproof.json --report /var/log/backupproof/proof-$(date +\%F).json >> /var/log/backupproof/weekly.log 2>&1
254
+ ```
255
+
256
+ systemd timer example (same cadence, same single command):
257
+
258
+ ```ini
259
+ # /etc/systemd/system/backupproof.service
260
+ [Unit]
261
+ Description=Weekly backup restore test
262
+ [Service]
263
+ Type=oneshot
264
+ ExecStart=/usr/local/bin/backupproof run --config /etc/backupproof.json --report /var/log/backupproof/proof.json
265
+
266
+ # /etc/systemd/system/backupproof.timer
267
+ [Unit]
268
+ Description=Weekly backupproof run
269
+ [Timer]
270
+ OnCalendar=Sun 03:30
271
+ Persistent=true
272
+ [Install]
273
+ WantedBy=timers.target
274
+ ```
275
+
276
+ ## What was live-tested here vs. simulated — read this first
277
+
278
+ The sandbox has **no `/dev/kvm`, no QEMU, no Proxmox host**.
279
+ Restic and borg were NOT available at v0.1.0 build time — but
280
+ v0.2.0 installed real release binaries (restic 0.17.3, borg
281
+ 1.4.1, downloaded from their GitHub releases; `apt` was unusable
282
+ — its update hung on a slow mirror) and ran the real paths live.
283
+ So, honestly:
284
+
285
+ - **Live-tested here (real code, real execution, covered by the
286
+ test suite):** backup discovery parsing for restic JSON,
287
+ borg JSON, and Proxmox vzdump filenames; latest-backup
288
+ selection; restic/borg/qmrestore/pct command construction;
289
+ all three smoke packs (real HTTP fetches, real SQLite queries,
290
+ real file hashing); verdict logic including mixed results;
291
+ error honesty (no backup, restore failure, boot timeout);
292
+ exit codes; end-to-end fixture runs (PASS and FAIL); and —
293
+ **new in v0.2.0 — real restic and real borg end-to-end runs:**
294
+ a real restic repo and a real borg repo were created, backed
295
+ up, listed, restored by `RealRunner`, and smoke-checked by the
296
+ full CLI — both reported `PASS: backup restores, all smoke
297
+ checks passed (file-level backup — content verified, no VM
298
+ boot)`, exit 0, with all three packs genuinely passing
299
+ (web served from the restored tree, real SQLite query, real
300
+ sha256). The restic/borg real mode is no longer simulated.
301
+ - **Fixture/simulated here, labeled as such everywhere it
302
+ appears:** the restore+boot layer when run with a `fixture`
303
+ source. Fixture restore copies a fixture tree; fixture boot is
304
+ simulated from `boot.json` (it does start a real localhost HTTP
305
+ server for the webroot, so the web pack is genuinely checked).
306
+ Every simulated result prints `mode: simulated` and
307
+ `SIMULATED restore/boot` in text and `"mode": "simulated"` in
308
+ JSON. A simulated result is never a claim that a real backup
309
+ restored or booted.
310
+ - **Command-sequence tested, NOT executed against a real host:**
311
+ the Proxmox flow (pre-flight status, `qmrestore`/`pct restore`,
312
+ start, probe, stop, destroy) is tested end-to-end against fake
313
+ `qm`/`qmrestore` binaries that log every invocation — the
314
+ sequence, the existing-VMID refusal, the missing-VMID refusal,
315
+ and cleanup-on-failure are all proven, but no real Proxmox host
316
+ was touched. The old "boot the restored `.qcow2` with QEMU"
317
+ path was **removed** in v0.2.0: no v0.2.0 flow boots a local
318
+ disk image (Proxmox boots on the Proxmox host; restic/borg are
319
+ file-level), and the QEMU helper survives only as a tested
320
+ utility for a future local-image source.
321
+
322
+ **What a real host needs for the full path:** restic and/or borg
323
+ installed (passwords via `RESTIC_PASSWORD` / `BORG_PASSPHRASE`
324
+ env), or a Proxmox host with a dump dir + a throwaway VMID.
325
+ If a tool is missing, `run` exits 2 with `MISSING_TOOL` — it
326
+ never reports PASS on an unverified restore.
327
+
328
+ ## What v1 does NOT do (deliberately)
329
+
330
+ Fleet dashboards, a built scheduler service, team-tier features,
331
+ Windows support, VMware estates, any paid gating, and any smoke
332
+ pack beyond web, DB, and files. The paid team tier (C$19/mo,
333
+ later, not v1) and any fourth pack are out of scope until the owner
334
+ verdicts this pair.
@@ -0,0 +1,316 @@
1
+ # backupproof — we prove your backup weekly
2
+
3
+ The #1 backup failure is not "no backup" — it's "backup never
4
+ tested, restore fails." backupproof takes the latest restic, borg,
5
+ or Proxmox backup, restores it into a throwaway target, verifies
6
+ it, and reports PASS or FAIL **with proof**: which checks passed,
7
+ which failed, and the evidence for each.
8
+
9
+ One verdict, three smoke packs only — web, DB, files. Nothing else.
10
+
11
+ **What "verify" means depends on the source — and the report says
12
+ which (v0.2.0 honesty fix):** restic/borg backups are file-level,
13
+ so the restored *files* are content-checked; there is no disk
14
+ image to boot, and the PASS line says so. Proxmox backups restore
15
+ into a throwaway VMID on the Proxmox host and are probed there
16
+ (web pack via a probe URL). A PASS line never claims a boot that
17
+ did not happen.
18
+
19
+ ## Install (local)
20
+
21
+ ```sh
22
+ pip install .
23
+ # or, for development:
24
+ pip install -e .
25
+ # or without installing, from this directory:
26
+ PYTHONPATH=. python3 -m backupproof --help
27
+ ```
28
+
29
+ Requires Python 3.9+. Stdlib only — no third-party dependencies,
30
+ no keys, no accounts, no telemetry, no phone-home.
31
+
32
+ ## Usage
33
+
34
+ ```sh
35
+ # The verdict: latest backup -> restore -> boot -> smoke -> PASS/FAIL
36
+ backupproof run --config backupproof.json
37
+ backupproof run --config backupproof.json --json --report proof.json
38
+
39
+ # List backups for a source (newest first)
40
+ backupproof list --config backupproof.json
41
+ backupproof list --source-type restic --repo /srv/backups --json
42
+
43
+ # Validate a config without restoring or booting anything
44
+ backupproof validate --config backupproof.json
45
+
46
+ # Write example config + fixture files (runnable: run --config <dir>/config.json)
47
+ backupproof init --output backupproof-examples
48
+ ```
49
+
50
+ Exit codes, CI-usable: **0 = PASS, 1 = FAIL (a check failed),
51
+ 2 = ERROR** (no backup found, restore failure, boot failure or
52
+ timeout, bad config, missing tool). `--json` works on all commands.
53
+
54
+ ### Config
55
+
56
+ A JSON file naming the source and the smoke checks:
57
+
58
+ ```json
59
+ {
60
+ "source": {"type": "restic", "repo": "/srv/backups"},
61
+ "checks": {
62
+ "web": [{"name": "homepage", "path": "/",
63
+ "expect_status": 200, "expect_contains": "Welcome"}],
64
+ "db": [{"name": "users table", "db_file": "app.db",
65
+ "query": "SELECT count(*) FROM users", "expect_rows_min": 1}],
66
+ "files": [{"name": "config", "path": "etc/app.conf",
67
+ "sha256": "<hex>", "size_min": 10}]
68
+ }
69
+ }
70
+ ```
71
+
72
+ Source types: `restic` (`repo`), `borg` (`repo`), `proxmox`
73
+ (`dump_dir` — a vzdump directory), `fixture` (`dir` — see below).
74
+ DB checks run the query against a SQLite file in the restored
75
+ target (stdlib `sqlite3`), or run a `command` (e.g. `psql`) and
76
+ check its output. Web checks fetch a real URL: `path` is resolved
77
+ against the served/restored target's base URL, or give a full `url`.
78
+ Files checks take `path` plus any of `sha256`, `size_min`,
79
+ `size_max`, `size_exact`, and `max_age_days`: the restored file's
80
+ modification time (as preserved by restic/borg) must be within N
81
+ days of the **backup's timestamp**, not of now — catches a nightly
82
+ dump file that stopped updating inside a fresh backup (`FILE_STALE`,
83
+ both dates reported). An unknown backup timestamp is a FAIL, never
84
+ a pass.
85
+ Optional `"mode": "0640"` and `"owner": "postgres"` compare the
86
+ restored file's permission bits and owning user (`pwd`)
87
+ (`FILE_MODE_MISMATCH` / `FILE_OWNER_MISMATCH`). On non-POSIX
88
+ hosts (Windows) both are a FAIL "not verifiable on this OS", never
89
+ a pass. Note: restic and borg restore file ownership only when run
90
+ as root; run as another user, restored files belong to that user.
91
+
92
+ ### Per-source fields
93
+
94
+ - `restic` / `borg`: `repo` (required). Optional `serve_dir` —
95
+ a directory inside the restored tree (e.g. `"webroot"`) that
96
+ backupproof serves over real localhost HTTP, so web checks can
97
+ hit the *restored* content (labelled "restored content, not a
98
+ VM" in the report). Without `serve_dir`, web checks need full
99
+ `url` values. **Passwords pass through the operator's env:**
100
+ `RESTIC_PASSWORD` for restic, `BORG_PASSPHRASE` for borg —
101
+ backupproof adds no credential handling of its own.
102
+ - `restic` / `borg` optional `"restore": "checked-paths"`
103
+ (default `"full"`): restore only the paths the checks read —
104
+ files-check `path`s, db `db_file`s and `serve_dir` (restic:
105
+ repeated `--include /<path>`; borg: `extract` with path
106
+ arguments). The restore line and the PASS line then say
107
+ `partial (N paths) — only checked paths restored; the rest of
108
+ the backup was not restore-tested`. A config with no
109
+ path-bearing check is a `CONFIG_ERROR`. Opt-in only: a partial
110
+ restore does not prove the whole backup restores.
111
+ - `proxmox`: `dump_dir` (required), `throwaway_vmid` (REQUIRED —
112
+ backupproof refuses to guess a VMID it will later destroy),
113
+ `storage` (default `"local"`). The restore runs `qmrestore` /
114
+ `pct restore` into the throwaway VMID, starts it, probes it
115
+ (with `--probe-url`), then stops and destroys **only that
116
+ VMID**. A pre-flight `qm/pct status <vmid>` refuses the run if
117
+ the VMID already exists. **Pack guard:** db/files packs cannot
118
+ verify a VM's filesystem from the host, so a Proxmox real-mode
119
+ config containing them is an ERROR (`PROXMOX_PACK_UNSUPPORTED`,
120
+ exit 2) naming the packs and the remedy (web checks with
121
+ `--probe-url`, or run backupproof inside the guest) — packs are
122
+ never silently dropped and PASSed.
123
+
124
+ ### Repository integrity step (opt-in)
125
+
126
+ `--integrity` runs `restic check` (add `--read-data-subset PCT`
127
+ to also read that share of the pack data) or `borg check
128
+ --repository-only` (`--verify-data` for `borg check
129
+ --verify-data`) **before** restoring. A failed check is an ERROR
130
+ (`INTEGRITY_FAILED`, exit 2) carrying the tool's last 20 output
131
+ lines; a passed one adds `integrity: {tool, args, ok}` to the
132
+ report. Cost warning: restic's documentation notes that reading
133
+ pack data downloads it from the repository, which is slow and can
134
+ cost money on cloud storage. Plain `restic check` (no subset)
135
+ checks structure only. Use a small subset, not full reads, weekly.
136
+
137
+ ### Dead-man's-switch ping (opt-in)
138
+
139
+ `--ping-url URL` sends one HTTP GET after the verdict: `URL` on
140
+ PASS, `URL/fail` on FAIL or ERROR — the healthchecks.io
141
+ convention, so a missed weekly run also alerts. 10-second
142
+ timeout. A ping failure prints a warning on stderr and never
143
+ changes the exit code. Nothing is sent unless the flag is given.
144
+
145
+ ### Staleness gate
146
+
147
+ Top-level `max_age_days` (or `--max-age-days`, CLI wins): when
148
+ the latest backup is older than that, the verdict carries a
149
+ warning. With `--fail-on-stale`, the run refuses outright
150
+ (`STALE_BACKUP`, exit 2) before restoring anything. A backup
151
+ whose timestamp is missing or unparseable warns ("cannot
152
+ determine age") and never fails on staleness alone.
153
+
154
+ ### Which packs verify what, per source
155
+
156
+ | Source | web pack | db pack | files pack |
157
+ |---|---|---|---|
158
+ | restic / borg (real) | served restored tree (`serve_dir`) or explicit URL | restored files | restored files |
159
+ | proxmox (real) | booted VM via probe URL | **refused** (pack guard) | **refused** (pack guard) |
160
+ | fixture (simulated) | served fixture webroot over real HTTP | fixture tree | fixture tree |
161
+
162
+ The report's `verified against:` line states this per run, and
163
+ `--report` JSON carries `boot_kind`, `boot_probed`,
164
+ `verified_against`, `warnings`, `restore_partial`, `integrity`
165
+ and `restore_seconds` fields. `restore_seconds` is the wall time
166
+ of the restore step only (not boot or probe) — RTO evidence; the
167
+ text shows `restore: … in 37.2s`. In simulated (fixture) mode it
168
+ times the fixture copy and is labelled `(simulated)`.
169
+
170
+ A run with no checks configured is an ERROR, never a PASS. A
171
+ check that cannot be verified is a FAIL, never skipped silently.
172
+
173
+ ### Example PASS output
174
+
175
+ ```
176
+ PASS (simulated): backup restores and boots in simulation, all smoke checks passed
177
+ backup: fixture snap-20261028 (2026-10-28T03:00:00Z)
178
+ mode: simulated (SIMULATED restore/boot — not a real VM boot)
179
+ restore: SIMULATED restore into /tmp/backupproof-yoi9_m9l/restored (3 files copied from fixture)
180
+ boot: SIMULATED boot completed in 42s (fixture boot.json; not a real VM boot); serving webroot webroot at http://127.0.0.1:34147
181
+ verified against: restored fixture tree (web pack served over real HTTP from the restored webroot)
182
+ PASS [web] homepage: HTTP 200, body contains 'Welcome'
183
+ PASS [db] users: query returned 1 row(s): [('alice',)]
184
+ PASS [files] config: app.conf: 10 bytes, sha256 matches
185
+ ```
186
+
187
+ A real restic/borg run reports instead:
188
+ `PASS: backup restores, all smoke checks passed (file-level backup — content verified, no VM boot)` —
189
+ and a Proxmox run with no probe URL reports
190
+ `PASS: backup restores, VM started (unprobed), all smoke checks passed`.
191
+
192
+ ### Example FAIL output (failing check named)
193
+
194
+ Change the restored homepage so it no longer contains "Welcome"
195
+ and the same run reports:
196
+
197
+ ```
198
+ FAIL: 1 of 3 smoke check(s) failed
199
+ backup: fixture snap-20261028 (2026-10-28T03:00:00Z)
200
+ mode: simulated (SIMULATED restore/boot — not a real VM boot)
201
+ restore: SIMULATED restore into /tmp/backupproof-y8b2y4d3/restored (3 files copied from fixture)
202
+ boot: SIMULATED boot completed in 42s (fixture boot.json; not a real VM boot); serving webroot webroot at http://127.0.0.1:45629
203
+ FAIL [web] homepage (WEB_CONTENT_MISSING): HTTP 200 but body lacks 'Welcome' (first 200 chars: 'Something else entirely\n')
204
+ PASS [db] users: query returned 1 row(s): [('alice',)]
205
+ PASS [files] config: app.conf: 10 bytes, sha256 matches
206
+ ```
207
+
208
+ Named failures: web — `WEB_UNREACHABLE`, `WEB_STATUS_MISMATCH`,
209
+ `WEB_CONTENT_MISSING`; DB — `DB_FILE_MISSING`, `DB_QUERY_FAILED`,
210
+ `DB_NO_ROWS`, `DB_CONTENT_MISSING`, `DB_COMMAND_FAILED`; files —
211
+ `FILE_MISSING`, `FILE_SIZE_MISMATCH`, `FILE_HASH_MISMATCH`,
212
+ `FILE_STALE`, `FILE_MODE_MISMATCH`, `FILE_OWNER_MISMATCH`.
213
+ Errors (exit 2): `NO_BACKUP_FOUND`, `RESTORE_FAILED`,
214
+ `BOOT_FAILED`, `BOOT_TIMEOUT`, `MISSING_TOOL`, `CONFIG_ERROR`,
215
+ `INTEGRITY_FAILED`.
216
+
217
+ ### Try it (fixture example, no backup tools needed)
218
+
219
+ ```sh
220
+ backupproof run --config examples/config.json
221
+ ```
222
+
223
+ The example fixture restores a small tree (webroot, `app.conf`,
224
+ SQLite `app.db`) and simulates boot while genuinely serving the
225
+ webroot over real localhost HTTP, so the web check is really
226
+ exercised.
227
+
228
+ ## Weekly cadence (doc example only — no built scheduler)
229
+
230
+ backupproof has no daemon and no built-in scheduler by design.
231
+ Run it weekly from cron or a systemd timer. Cron example (Sundays
232
+ 03:30, proof report kept per run):
233
+
234
+ ```cron
235
+ 30 3 * * 0 /usr/local/bin/backupproof run --config /etc/backupproof.json --report /var/log/backupproof/proof-$(date +\%F).json >> /var/log/backupproof/weekly.log 2>&1
236
+ ```
237
+
238
+ systemd timer example (same cadence, same single command):
239
+
240
+ ```ini
241
+ # /etc/systemd/system/backupproof.service
242
+ [Unit]
243
+ Description=Weekly backup restore test
244
+ [Service]
245
+ Type=oneshot
246
+ ExecStart=/usr/local/bin/backupproof run --config /etc/backupproof.json --report /var/log/backupproof/proof.json
247
+
248
+ # /etc/systemd/system/backupproof.timer
249
+ [Unit]
250
+ Description=Weekly backupproof run
251
+ [Timer]
252
+ OnCalendar=Sun 03:30
253
+ Persistent=true
254
+ [Install]
255
+ WantedBy=timers.target
256
+ ```
257
+
258
+ ## What was live-tested here vs. simulated — read this first
259
+
260
+ The sandbox has **no `/dev/kvm`, no QEMU, no Proxmox host**.
261
+ Restic and borg were NOT available at v0.1.0 build time — but
262
+ v0.2.0 installed real release binaries (restic 0.17.3, borg
263
+ 1.4.1, downloaded from their GitHub releases; `apt` was unusable
264
+ — its update hung on a slow mirror) and ran the real paths live.
265
+ So, honestly:
266
+
267
+ - **Live-tested here (real code, real execution, covered by the
268
+ test suite):** backup discovery parsing for restic JSON,
269
+ borg JSON, and Proxmox vzdump filenames; latest-backup
270
+ selection; restic/borg/qmrestore/pct command construction;
271
+ all three smoke packs (real HTTP fetches, real SQLite queries,
272
+ real file hashing); verdict logic including mixed results;
273
+ error honesty (no backup, restore failure, boot timeout);
274
+ exit codes; end-to-end fixture runs (PASS and FAIL); and —
275
+ **new in v0.2.0 — real restic and real borg end-to-end runs:**
276
+ a real restic repo and a real borg repo were created, backed
277
+ up, listed, restored by `RealRunner`, and smoke-checked by the
278
+ full CLI — both reported `PASS: backup restores, all smoke
279
+ checks passed (file-level backup — content verified, no VM
280
+ boot)`, exit 0, with all three packs genuinely passing
281
+ (web served from the restored tree, real SQLite query, real
282
+ sha256). The restic/borg real mode is no longer simulated.
283
+ - **Fixture/simulated here, labeled as such everywhere it
284
+ appears:** the restore+boot layer when run with a `fixture`
285
+ source. Fixture restore copies a fixture tree; fixture boot is
286
+ simulated from `boot.json` (it does start a real localhost HTTP
287
+ server for the webroot, so the web pack is genuinely checked).
288
+ Every simulated result prints `mode: simulated` and
289
+ `SIMULATED restore/boot` in text and `"mode": "simulated"` in
290
+ JSON. A simulated result is never a claim that a real backup
291
+ restored or booted.
292
+ - **Command-sequence tested, NOT executed against a real host:**
293
+ the Proxmox flow (pre-flight status, `qmrestore`/`pct restore`,
294
+ start, probe, stop, destroy) is tested end-to-end against fake
295
+ `qm`/`qmrestore` binaries that log every invocation — the
296
+ sequence, the existing-VMID refusal, the missing-VMID refusal,
297
+ and cleanup-on-failure are all proven, but no real Proxmox host
298
+ was touched. The old "boot the restored `.qcow2` with QEMU"
299
+ path was **removed** in v0.2.0: no v0.2.0 flow boots a local
300
+ disk image (Proxmox boots on the Proxmox host; restic/borg are
301
+ file-level), and the QEMU helper survives only as a tested
302
+ utility for a future local-image source.
303
+
304
+ **What a real host needs for the full path:** restic and/or borg
305
+ installed (passwords via `RESTIC_PASSWORD` / `BORG_PASSPHRASE`
306
+ env), or a Proxmox host with a dump dir + a throwaway VMID.
307
+ If a tool is missing, `run` exits 2 with `MISSING_TOOL` — it
308
+ never reports PASS on an unverified restore.
309
+
310
+ ## What v1 does NOT do (deliberately)
311
+
312
+ Fleet dashboards, a built scheduler service, team-tier features,
313
+ Windows support, VMware estates, any paid gating, and any smoke
314
+ pack beyond web, DB, and files. The paid team tier (C$19/mo,
315
+ later, not v1) and any fourth pack are out of scope until the owner
316
+ verdicts this pair.