backupproof 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- backupproof-0.2.0/LICENSE +21 -0
- backupproof-0.2.0/PKG-INFO +334 -0
- backupproof-0.2.0/README.md +316 -0
- backupproof-0.2.0/backupproof/__init__.py +23 -0
- backupproof-0.2.0/backupproof/__main__.py +5 -0
- backupproof-0.2.0/backupproof/cli.py +420 -0
- backupproof-0.2.0/backupproof/config.py +121 -0
- backupproof-0.2.0/backupproof/discovery.py +185 -0
- backupproof-0.2.0/backupproof/errors.py +55 -0
- backupproof-0.2.0/backupproof/models.py +85 -0
- backupproof-0.2.0/backupproof/runners.py +436 -0
- backupproof-0.2.0/backupproof/smoke.py +202 -0
- backupproof-0.2.0/backupproof/verdict.py +111 -0
- backupproof-0.2.0/backupproof.egg-info/PKG-INFO +334 -0
- backupproof-0.2.0/backupproof.egg-info/SOURCES.txt +25 -0
- backupproof-0.2.0/backupproof.egg-info/dependency_links.txt +1 -0
- backupproof-0.2.0/backupproof.egg-info/entry_points.txt +2 -0
- backupproof-0.2.0/backupproof.egg-info/top_level.txt +1 -0
- backupproof-0.2.0/pyproject.toml +28 -0
- backupproof-0.2.0/setup.cfg +4 -0
- backupproof-0.2.0/tests/test_bp.py +273 -0
- backupproof-0.2.0/tests/test_cli_e2e.py +136 -0
- backupproof-0.2.0/tests/test_discovery.py +83 -0
- backupproof-0.2.0/tests/test_runners.py +97 -0
- backupproof-0.2.0/tests/test_smoke.py +155 -0
- backupproof-0.2.0/tests/test_v11.py +286 -0
- backupproof-0.2.0/tests/test_verdict_config.py +91 -0
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Calas Automations Inc.
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,334 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: backupproof
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Backup restore testing: restore the latest restic/borg/Proxmox backup into a throwaway target, boot it, run smoke checks, report PASS or FAIL with proof
|
|
5
|
+
Author: Calas Automations Inc.
|
|
6
|
+
License: MIT
|
|
7
|
+
Keywords: backup,restore,restic,borg,proxmox,testing
|
|
8
|
+
Classifier: Development Status :: 3 - Alpha
|
|
9
|
+
Classifier: Environment :: Console
|
|
10
|
+
Classifier: Intended Audience :: System Administrators
|
|
11
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
12
|
+
Classifier: Programming Language :: Python :: 3
|
|
13
|
+
Classifier: Topic :: System :: Archiving :: Backup
|
|
14
|
+
Requires-Python: >=3.9
|
|
15
|
+
Description-Content-Type: text/markdown
|
|
16
|
+
License-File: LICENSE
|
|
17
|
+
Dynamic: license-file
|
|
18
|
+
|
|
19
|
+
# backupproof — we prove your backup weekly
|
|
20
|
+
|
|
21
|
+
The #1 backup failure is not "no backup" — it's "backup never
|
|
22
|
+
tested, restore fails." backupproof takes the latest restic, borg,
|
|
23
|
+
or Proxmox backup, restores it into a throwaway target, verifies
|
|
24
|
+
it, and reports PASS or FAIL **with proof**: which checks passed,
|
|
25
|
+
which failed, and the evidence for each.
|
|
26
|
+
|
|
27
|
+
One verdict, three smoke packs only — web, DB, files. Nothing else.
|
|
28
|
+
|
|
29
|
+
**What "verify" means depends on the source — and the report says
|
|
30
|
+
which (v0.2.0 honesty fix):** restic/borg backups are file-level,
|
|
31
|
+
so the restored *files* are content-checked; there is no disk
|
|
32
|
+
image to boot, and the PASS line says so. Proxmox backups restore
|
|
33
|
+
into a throwaway VMID on the Proxmox host and are probed there
|
|
34
|
+
(web pack via a probe URL). A PASS line never claims a boot that
|
|
35
|
+
did not happen.
|
|
36
|
+
|
|
37
|
+
## Install (local)
|
|
38
|
+
|
|
39
|
+
```sh
|
|
40
|
+
pip install .
|
|
41
|
+
# or, for development:
|
|
42
|
+
pip install -e .
|
|
43
|
+
# or without installing, from this directory:
|
|
44
|
+
PYTHONPATH=. python3 -m backupproof --help
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Requires Python 3.9+. Stdlib only — no third-party dependencies,
|
|
48
|
+
no keys, no accounts, no telemetry, no phone-home.
|
|
49
|
+
|
|
50
|
+
## Usage
|
|
51
|
+
|
|
52
|
+
```sh
|
|
53
|
+
# The verdict: latest backup -> restore -> boot -> smoke -> PASS/FAIL
|
|
54
|
+
backupproof run --config backupproof.json
|
|
55
|
+
backupproof run --config backupproof.json --json --report proof.json
|
|
56
|
+
|
|
57
|
+
# List backups for a source (newest first)
|
|
58
|
+
backupproof list --config backupproof.json
|
|
59
|
+
backupproof list --source-type restic --repo /srv/backups --json
|
|
60
|
+
|
|
61
|
+
# Validate a config without restoring or booting anything
|
|
62
|
+
backupproof validate --config backupproof.json
|
|
63
|
+
|
|
64
|
+
# Write example config + fixture files (runnable: run --config <dir>/config.json)
|
|
65
|
+
backupproof init --output backupproof-examples
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Exit codes, CI-usable: **0 = PASS, 1 = FAIL (a check failed),
|
|
69
|
+
2 = ERROR** (no backup found, restore failure, boot failure or
|
|
70
|
+
timeout, bad config, missing tool). `--json` works on all commands.
|
|
71
|
+
|
|
72
|
+
### Config
|
|
73
|
+
|
|
74
|
+
A JSON file naming the source and the smoke checks:
|
|
75
|
+
|
|
76
|
+
```json
|
|
77
|
+
{
|
|
78
|
+
"source": {"type": "restic", "repo": "/srv/backups"},
|
|
79
|
+
"checks": {
|
|
80
|
+
"web": [{"name": "homepage", "path": "/",
|
|
81
|
+
"expect_status": 200, "expect_contains": "Welcome"}],
|
|
82
|
+
"db": [{"name": "users table", "db_file": "app.db",
|
|
83
|
+
"query": "SELECT count(*) FROM users", "expect_rows_min": 1}],
|
|
84
|
+
"files": [{"name": "config", "path": "etc/app.conf",
|
|
85
|
+
"sha256": "<hex>", "size_min": 10}]
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Source types: `restic` (`repo`), `borg` (`repo`), `proxmox`
|
|
91
|
+
(`dump_dir` — a vzdump directory), `fixture` (`dir` — see below).
|
|
92
|
+
DB checks run the query against a SQLite file in the restored
|
|
93
|
+
target (stdlib `sqlite3`), or run a `command` (e.g. `psql`) and
|
|
94
|
+
check its output. Web checks fetch a real URL: `path` is resolved
|
|
95
|
+
against the served/restored target's base URL, or give a full `url`.
|
|
96
|
+
Files checks take `path` plus any of `sha256`, `size_min`,
|
|
97
|
+
`size_max`, `size_exact`, and `max_age_days`: the restored file's
|
|
98
|
+
modification time (as preserved by restic/borg) must be within N
|
|
99
|
+
days of the **backup's timestamp**, not of now — catches a nightly
|
|
100
|
+
dump file that stopped updating inside a fresh backup (`FILE_STALE`,
|
|
101
|
+
both dates reported). An unknown backup timestamp is a FAIL, never
|
|
102
|
+
a pass.
|
|
103
|
+
Optional `"mode": "0640"` and `"owner": "postgres"` compare the
|
|
104
|
+
restored file's permission bits and owning user (`pwd`)
|
|
105
|
+
(`FILE_MODE_MISMATCH` / `FILE_OWNER_MISMATCH`). On non-POSIX
|
|
106
|
+
hosts (Windows) both are a FAIL "not verifiable on this OS", never
|
|
107
|
+
a pass. Note: restic and borg restore file ownership only when run
|
|
108
|
+
as root; run as another user, restored files belong to that user.
|
|
109
|
+
|
|
110
|
+
### Per-source fields
|
|
111
|
+
|
|
112
|
+
- `restic` / `borg`: `repo` (required). Optional `serve_dir` —
|
|
113
|
+
a directory inside the restored tree (e.g. `"webroot"`) that
|
|
114
|
+
backupproof serves over real localhost HTTP, so web checks can
|
|
115
|
+
hit the *restored* content (labelled "restored content, not a
|
|
116
|
+
VM" in the report). Without `serve_dir`, web checks need full
|
|
117
|
+
`url` values. **Passwords pass through the operator's env:**
|
|
118
|
+
`RESTIC_PASSWORD` for restic, `BORG_PASSPHRASE` for borg —
|
|
119
|
+
backupproof adds no credential handling of its own.
|
|
120
|
+
- `restic` / `borg` optional `"restore": "checked-paths"`
|
|
121
|
+
(default `"full"`): restore only the paths the checks read —
|
|
122
|
+
files-check `path`s, db `db_file`s and `serve_dir` (restic:
|
|
123
|
+
repeated `--include /<path>`; borg: `extract` with path
|
|
124
|
+
arguments). The restore line and the PASS line then say
|
|
125
|
+
`partial (N paths) — only checked paths restored; the rest of
|
|
126
|
+
the backup was not restore-tested`. A config with no
|
|
127
|
+
path-bearing check is a `CONFIG_ERROR`. Opt-in only: a partial
|
|
128
|
+
restore does not prove the whole backup restores.
|
|
129
|
+
- `proxmox`: `dump_dir` (required), `throwaway_vmid` (REQUIRED —
|
|
130
|
+
backupproof refuses to guess a VMID it will later destroy),
|
|
131
|
+
`storage` (default `"local"`). The restore runs `qmrestore` /
|
|
132
|
+
`pct restore` into the throwaway VMID, starts it, probes it
|
|
133
|
+
(with `--probe-url`), then stops and destroys **only that
|
|
134
|
+
VMID**. A pre-flight `qm/pct status <vmid>` refuses the run if
|
|
135
|
+
the VMID already exists. **Pack guard:** db/files packs cannot
|
|
136
|
+
verify a VM's filesystem from the host, so a Proxmox real-mode
|
|
137
|
+
config containing them is an ERROR (`PROXMOX_PACK_UNSUPPORTED`,
|
|
138
|
+
exit 2) naming the packs and the remedy (web checks with
|
|
139
|
+
`--probe-url`, or run backupproof inside the guest) — packs are
|
|
140
|
+
never silently dropped and PASSed.
|
|
141
|
+
|
|
142
|
+
### Repository integrity step (opt-in)
|
|
143
|
+
|
|
144
|
+
`--integrity` runs `restic check` (add `--read-data-subset PCT`
|
|
145
|
+
to also read that share of the pack data) or `borg check
|
|
146
|
+
--repository-only` (`--verify-data` for `borg check
|
|
147
|
+
--verify-data`) **before** restoring. A failed check is an ERROR
|
|
148
|
+
(`INTEGRITY_FAILED`, exit 2) carrying the tool's last 20 output
|
|
149
|
+
lines; a passed one adds `integrity: {tool, args, ok}` to the
|
|
150
|
+
report. Cost warning: restic's documentation notes that reading
|
|
151
|
+
pack data downloads it from the repository, which is slow and can
|
|
152
|
+
cost money on cloud storage. Plain `restic check` (no subset)
|
|
153
|
+
checks structure only. Use a small subset, not full reads, weekly.
|
|
154
|
+
|
|
155
|
+
### Dead-man's-switch ping (opt-in)
|
|
156
|
+
|
|
157
|
+
`--ping-url URL` sends one HTTP GET after the verdict: `URL` on
|
|
158
|
+
PASS, `URL/fail` on FAIL or ERROR — the healthchecks.io
|
|
159
|
+
convention, so a missed weekly run also alerts. 10-second
|
|
160
|
+
timeout. A ping failure prints a warning on stderr and never
|
|
161
|
+
changes the exit code. Nothing is sent unless the flag is given.
|
|
162
|
+
|
|
163
|
+
### Staleness gate
|
|
164
|
+
|
|
165
|
+
Top-level `max_age_days` (or `--max-age-days`, CLI wins): when
|
|
166
|
+
the latest backup is older than that, the verdict carries a
|
|
167
|
+
warning. With `--fail-on-stale`, the run refuses outright
|
|
168
|
+
(`STALE_BACKUP`, exit 2) before restoring anything. A backup
|
|
169
|
+
whose timestamp is missing or unparseable warns ("cannot
|
|
170
|
+
determine age") and never fails on staleness alone.
|
|
171
|
+
|
|
172
|
+
### Which packs verify what, per source
|
|
173
|
+
|
|
174
|
+
| Source | web pack | db pack | files pack |
|
|
175
|
+
|---|---|---|---|
|
|
176
|
+
| restic / borg (real) | served restored tree (`serve_dir`) or explicit URL | restored files | restored files |
|
|
177
|
+
| proxmox (real) | booted VM via probe URL | **refused** (pack guard) | **refused** (pack guard) |
|
|
178
|
+
| fixture (simulated) | served fixture webroot over real HTTP | fixture tree | fixture tree |
|
|
179
|
+
|
|
180
|
+
The report's `verified against:` line states this per run, and
|
|
181
|
+
`--report` JSON carries `boot_kind`, `boot_probed`,
|
|
182
|
+
`verified_against`, `warnings`, `restore_partial`, `integrity`
|
|
183
|
+
and `restore_seconds` fields. `restore_seconds` is the wall time
|
|
184
|
+
of the restore step only (not boot or probe) — RTO evidence; the
|
|
185
|
+
text shows `restore: … in 37.2s`. In simulated (fixture) mode it
|
|
186
|
+
times the fixture copy and is labelled `(simulated)`.
|
|
187
|
+
|
|
188
|
+
A run with no checks configured is an ERROR, never a PASS. A
|
|
189
|
+
check that cannot be verified is a FAIL, never skipped silently.
|
|
190
|
+
|
|
191
|
+
### Example PASS output
|
|
192
|
+
|
|
193
|
+
```
|
|
194
|
+
PASS (simulated): backup restores and boots in simulation, all smoke checks passed
|
|
195
|
+
backup: fixture snap-20261028 (2026-10-28T03:00:00Z)
|
|
196
|
+
mode: simulated (SIMULATED restore/boot — not a real VM boot)
|
|
197
|
+
restore: SIMULATED restore into /tmp/backupproof-yoi9_m9l/restored (3 files copied from fixture)
|
|
198
|
+
boot: SIMULATED boot completed in 42s (fixture boot.json; not a real VM boot); serving webroot webroot at http://127.0.0.1:34147
|
|
199
|
+
verified against: restored fixture tree (web pack served over real HTTP from the restored webroot)
|
|
200
|
+
PASS [web] homepage: HTTP 200, body contains 'Welcome'
|
|
201
|
+
PASS [db] users: query returned 1 row(s): [('alice',)]
|
|
202
|
+
PASS [files] config: app.conf: 10 bytes, sha256 matches
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
A real restic/borg run reports instead:
|
|
206
|
+
`PASS: backup restores, all smoke checks passed (file-level backup — content verified, no VM boot)` —
|
|
207
|
+
and a Proxmox run with no probe URL reports
|
|
208
|
+
`PASS: backup restores, VM started (unprobed), all smoke checks passed`.
|
|
209
|
+
|
|
210
|
+
### Example FAIL output (failing check named)
|
|
211
|
+
|
|
212
|
+
Change the restored homepage so it no longer contains "Welcome"
|
|
213
|
+
and the same run reports:
|
|
214
|
+
|
|
215
|
+
```
|
|
216
|
+
FAIL: 1 of 3 smoke check(s) failed
|
|
217
|
+
backup: fixture snap-20261028 (2026-10-28T03:00:00Z)
|
|
218
|
+
mode: simulated (SIMULATED restore/boot — not a real VM boot)
|
|
219
|
+
restore: SIMULATED restore into /tmp/backupproof-y8b2y4d3/restored (3 files copied from fixture)
|
|
220
|
+
boot: SIMULATED boot completed in 42s (fixture boot.json; not a real VM boot); serving webroot webroot at http://127.0.0.1:45629
|
|
221
|
+
FAIL [web] homepage (WEB_CONTENT_MISSING): HTTP 200 but body lacks 'Welcome' (first 200 chars: 'Something else entirely\n')
|
|
222
|
+
PASS [db] users: query returned 1 row(s): [('alice',)]
|
|
223
|
+
PASS [files] config: app.conf: 10 bytes, sha256 matches
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
Named failures: web — `WEB_UNREACHABLE`, `WEB_STATUS_MISMATCH`,
|
|
227
|
+
`WEB_CONTENT_MISSING`; DB — `DB_FILE_MISSING`, `DB_QUERY_FAILED`,
|
|
228
|
+
`DB_NO_ROWS`, `DB_CONTENT_MISSING`, `DB_COMMAND_FAILED`; files —
|
|
229
|
+
`FILE_MISSING`, `FILE_SIZE_MISMATCH`, `FILE_HASH_MISMATCH`,
|
|
230
|
+
`FILE_STALE`, `FILE_MODE_MISMATCH`, `FILE_OWNER_MISMATCH`.
|
|
231
|
+
Errors (exit 2): `NO_BACKUP_FOUND`, `RESTORE_FAILED`,
|
|
232
|
+
`BOOT_FAILED`, `BOOT_TIMEOUT`, `MISSING_TOOL`, `CONFIG_ERROR`,
|
|
233
|
+
`INTEGRITY_FAILED`.
|
|
234
|
+
|
|
235
|
+
### Try it (fixture example, no backup tools needed)
|
|
236
|
+
|
|
237
|
+
```sh
|
|
238
|
+
backupproof run --config examples/config.json
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
The example fixture restores a small tree (webroot, `app.conf`,
|
|
242
|
+
SQLite `app.db`) and simulates boot while genuinely serving the
|
|
243
|
+
webroot over real localhost HTTP, so the web check is really
|
|
244
|
+
exercised.
|
|
245
|
+
|
|
246
|
+
## Weekly cadence (doc example only — no built scheduler)
|
|
247
|
+
|
|
248
|
+
backupproof has no daemon and no built-in scheduler by design.
|
|
249
|
+
Run it weekly from cron or a systemd timer. Cron example (Sundays
|
|
250
|
+
03:30, proof report kept per run):
|
|
251
|
+
|
|
252
|
+
```cron
|
|
253
|
+
30 3 * * 0 /usr/local/bin/backupproof run --config /etc/backupproof.json --report /var/log/backupproof/proof-$(date +\%F).json >> /var/log/backupproof/weekly.log 2>&1
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
systemd timer example (same cadence, same single command):
|
|
257
|
+
|
|
258
|
+
```ini
|
|
259
|
+
# /etc/systemd/system/backupproof.service
|
|
260
|
+
[Unit]
|
|
261
|
+
Description=Weekly backup restore test
|
|
262
|
+
[Service]
|
|
263
|
+
Type=oneshot
|
|
264
|
+
ExecStart=/usr/local/bin/backupproof run --config /etc/backupproof.json --report /var/log/backupproof/proof.json
|
|
265
|
+
|
|
266
|
+
# /etc/systemd/system/backupproof.timer
|
|
267
|
+
[Unit]
|
|
268
|
+
Description=Weekly backupproof run
|
|
269
|
+
[Timer]
|
|
270
|
+
OnCalendar=Sun 03:30
|
|
271
|
+
Persistent=true
|
|
272
|
+
[Install]
|
|
273
|
+
WantedBy=timers.target
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
## What was live-tested here vs. simulated — read this first
|
|
277
|
+
|
|
278
|
+
The sandbox has **no `/dev/kvm`, no QEMU, no Proxmox host**.
|
|
279
|
+
Restic and borg were NOT available at v0.1.0 build time — but
|
|
280
|
+
v0.2.0 installed real release binaries (restic 0.17.3, borg
|
|
281
|
+
1.4.1, downloaded from their GitHub releases; `apt` was unusable
|
|
282
|
+
— its update hung on a slow mirror) and ran the real paths live.
|
|
283
|
+
So, honestly:
|
|
284
|
+
|
|
285
|
+
- **Live-tested here (real code, real execution, covered by the
|
|
286
|
+
test suite):** backup discovery parsing for restic JSON,
|
|
287
|
+
borg JSON, and Proxmox vzdump filenames; latest-backup
|
|
288
|
+
selection; restic/borg/qmrestore/pct command construction;
|
|
289
|
+
all three smoke packs (real HTTP fetches, real SQLite queries,
|
|
290
|
+
real file hashing); verdict logic including mixed results;
|
|
291
|
+
error honesty (no backup, restore failure, boot timeout);
|
|
292
|
+
exit codes; end-to-end fixture runs (PASS and FAIL); and —
|
|
293
|
+
**new in v0.2.0 — real restic and real borg end-to-end runs:**
|
|
294
|
+
a real restic repo and a real borg repo were created, backed
|
|
295
|
+
up, listed, restored by `RealRunner`, and smoke-checked by the
|
|
296
|
+
full CLI — both reported `PASS: backup restores, all smoke
|
|
297
|
+
checks passed (file-level backup — content verified, no VM
|
|
298
|
+
boot)`, exit 0, with all three packs genuinely passing
|
|
299
|
+
(web served from the restored tree, real SQLite query, real
|
|
300
|
+
sha256). The restic/borg real mode is no longer simulated.
|
|
301
|
+
- **Fixture/simulated here, labeled as such everywhere it
|
|
302
|
+
appears:** the restore+boot layer when run with a `fixture`
|
|
303
|
+
source. Fixture restore copies a fixture tree; fixture boot is
|
|
304
|
+
simulated from `boot.json` (it does start a real localhost HTTP
|
|
305
|
+
server for the webroot, so the web pack is genuinely checked).
|
|
306
|
+
Every simulated result prints `mode: simulated` and
|
|
307
|
+
`SIMULATED restore/boot` in text and `"mode": "simulated"` in
|
|
308
|
+
JSON. A simulated result is never a claim that a real backup
|
|
309
|
+
restored or booted.
|
|
310
|
+
- **Command-sequence tested, NOT executed against a real host:**
|
|
311
|
+
the Proxmox flow (pre-flight status, `qmrestore`/`pct restore`,
|
|
312
|
+
start, probe, stop, destroy) is tested end-to-end against fake
|
|
313
|
+
`qm`/`qmrestore` binaries that log every invocation — the
|
|
314
|
+
sequence, the existing-VMID refusal, the missing-VMID refusal,
|
|
315
|
+
and cleanup-on-failure are all proven, but no real Proxmox host
|
|
316
|
+
was touched. The old "boot the restored `.qcow2` with QEMU"
|
|
317
|
+
path was **removed** in v0.2.0: no v0.2.0 flow boots a local
|
|
318
|
+
disk image (Proxmox boots on the Proxmox host; restic/borg are
|
|
319
|
+
file-level), and the QEMU helper survives only as a tested
|
|
320
|
+
utility for a future local-image source.
|
|
321
|
+
|
|
322
|
+
**What a real host needs for the full path:** restic and/or borg
|
|
323
|
+
installed (passwords via `RESTIC_PASSWORD` / `BORG_PASSPHRASE`
|
|
324
|
+
env), or a Proxmox host with a dump dir + a throwaway VMID.
|
|
325
|
+
If a tool is missing, `run` exits 2 with `MISSING_TOOL` — it
|
|
326
|
+
never reports PASS on an unverified restore.
|
|
327
|
+
|
|
328
|
+
## What v1 does NOT do (deliberately)
|
|
329
|
+
|
|
330
|
+
Fleet dashboards, a built scheduler service, team-tier features,
|
|
331
|
+
Windows support, VMware estates, any paid gating, and any smoke
|
|
332
|
+
pack beyond web, DB, and files. The paid team tier (C$19/mo,
|
|
333
|
+
later, not v1) and any fourth pack are out of scope until the owner
|
|
334
|
+
verdicts this pair.
|
|
@@ -0,0 +1,316 @@
|
|
|
1
|
+
# backupproof — we prove your backup weekly
|
|
2
|
+
|
|
3
|
+
The #1 backup failure is not "no backup" — it's "backup never
|
|
4
|
+
tested, restore fails." backupproof takes the latest restic, borg,
|
|
5
|
+
or Proxmox backup, restores it into a throwaway target, verifies
|
|
6
|
+
it, and reports PASS or FAIL **with proof**: which checks passed,
|
|
7
|
+
which failed, and the evidence for each.
|
|
8
|
+
|
|
9
|
+
One verdict, three smoke packs only — web, DB, files. Nothing else.
|
|
10
|
+
|
|
11
|
+
**What "verify" means depends on the source — and the report says
|
|
12
|
+
which (v0.2.0 honesty fix):** restic/borg backups are file-level,
|
|
13
|
+
so the restored *files* are content-checked; there is no disk
|
|
14
|
+
image to boot, and the PASS line says so. Proxmox backups restore
|
|
15
|
+
into a throwaway VMID on the Proxmox host and are probed there
|
|
16
|
+
(web pack via a probe URL). A PASS line never claims a boot that
|
|
17
|
+
did not happen.
|
|
18
|
+
|
|
19
|
+
## Install (local)
|
|
20
|
+
|
|
21
|
+
```sh
|
|
22
|
+
pip install .
|
|
23
|
+
# or, for development:
|
|
24
|
+
pip install -e .
|
|
25
|
+
# or without installing, from this directory:
|
|
26
|
+
PYTHONPATH=. python3 -m backupproof --help
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Requires Python 3.9+. Stdlib only — no third-party dependencies,
|
|
30
|
+
no keys, no accounts, no telemetry, no phone-home.
|
|
31
|
+
|
|
32
|
+
## Usage
|
|
33
|
+
|
|
34
|
+
```sh
|
|
35
|
+
# The verdict: latest backup -> restore -> boot -> smoke -> PASS/FAIL
|
|
36
|
+
backupproof run --config backupproof.json
|
|
37
|
+
backupproof run --config backupproof.json --json --report proof.json
|
|
38
|
+
|
|
39
|
+
# List backups for a source (newest first)
|
|
40
|
+
backupproof list --config backupproof.json
|
|
41
|
+
backupproof list --source-type restic --repo /srv/backups --json
|
|
42
|
+
|
|
43
|
+
# Validate a config without restoring or booting anything
|
|
44
|
+
backupproof validate --config backupproof.json
|
|
45
|
+
|
|
46
|
+
# Write example config + fixture files (runnable: run --config <dir>/config.json)
|
|
47
|
+
backupproof init --output backupproof-examples
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Exit codes, CI-usable: **0 = PASS, 1 = FAIL (a check failed),
|
|
51
|
+
2 = ERROR** (no backup found, restore failure, boot failure or
|
|
52
|
+
timeout, bad config, missing tool). `--json` works on all commands.
|
|
53
|
+
|
|
54
|
+
### Config
|
|
55
|
+
|
|
56
|
+
A JSON file naming the source and the smoke checks:
|
|
57
|
+
|
|
58
|
+
```json
|
|
59
|
+
{
|
|
60
|
+
"source": {"type": "restic", "repo": "/srv/backups"},
|
|
61
|
+
"checks": {
|
|
62
|
+
"web": [{"name": "homepage", "path": "/",
|
|
63
|
+
"expect_status": 200, "expect_contains": "Welcome"}],
|
|
64
|
+
"db": [{"name": "users table", "db_file": "app.db",
|
|
65
|
+
"query": "SELECT count(*) FROM users", "expect_rows_min": 1}],
|
|
66
|
+
"files": [{"name": "config", "path": "etc/app.conf",
|
|
67
|
+
"sha256": "<hex>", "size_min": 10}]
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Source types: `restic` (`repo`), `borg` (`repo`), `proxmox`
|
|
73
|
+
(`dump_dir` — a vzdump directory), `fixture` (`dir` — see below).
|
|
74
|
+
DB checks run the query against a SQLite file in the restored
|
|
75
|
+
target (stdlib `sqlite3`), or run a `command` (e.g. `psql`) and
|
|
76
|
+
check its output. Web checks fetch a real URL: `path` is resolved
|
|
77
|
+
against the served/restored target's base URL, or give a full `url`.
|
|
78
|
+
Files checks take `path` plus any of `sha256`, `size_min`,
|
|
79
|
+
`size_max`, `size_exact`, and `max_age_days`: the restored file's
|
|
80
|
+
modification time (as preserved by restic/borg) must be within N
|
|
81
|
+
days of the **backup's timestamp**, not of now — catches a nightly
|
|
82
|
+
dump file that stopped updating inside a fresh backup (`FILE_STALE`,
|
|
83
|
+
both dates reported). An unknown backup timestamp is a FAIL, never
|
|
84
|
+
a pass.
|
|
85
|
+
Optional `"mode": "0640"` and `"owner": "postgres"` compare the
|
|
86
|
+
restored file's permission bits and owning user (`pwd`)
|
|
87
|
+
(`FILE_MODE_MISMATCH` / `FILE_OWNER_MISMATCH`). On non-POSIX
|
|
88
|
+
hosts (Windows) both are a FAIL "not verifiable on this OS", never
|
|
89
|
+
a pass. Note: restic and borg restore file ownership only when run
|
|
90
|
+
as root; run as another user, restored files belong to that user.
|
|
91
|
+
|
|
92
|
+
### Per-source fields
|
|
93
|
+
|
|
94
|
+
- `restic` / `borg`: `repo` (required). Optional `serve_dir` —
|
|
95
|
+
a directory inside the restored tree (e.g. `"webroot"`) that
|
|
96
|
+
backupproof serves over real localhost HTTP, so web checks can
|
|
97
|
+
hit the *restored* content (labelled "restored content, not a
|
|
98
|
+
VM" in the report). Without `serve_dir`, web checks need full
|
|
99
|
+
`url` values. **Passwords pass through the operator's env:**
|
|
100
|
+
`RESTIC_PASSWORD` for restic, `BORG_PASSPHRASE` for borg —
|
|
101
|
+
backupproof adds no credential handling of its own.
|
|
102
|
+
- `restic` / `borg` optional `"restore": "checked-paths"`
|
|
103
|
+
(default `"full"`): restore only the paths the checks read —
|
|
104
|
+
files-check `path`s, db `db_file`s and `serve_dir` (restic:
|
|
105
|
+
repeated `--include /<path>`; borg: `extract` with path
|
|
106
|
+
arguments). The restore line and the PASS line then say
|
|
107
|
+
`partial (N paths) — only checked paths restored; the rest of
|
|
108
|
+
the backup was not restore-tested`. A config with no
|
|
109
|
+
path-bearing check is a `CONFIG_ERROR`. Opt-in only: a partial
|
|
110
|
+
restore does not prove the whole backup restores.
|
|
111
|
+
- `proxmox`: `dump_dir` (required), `throwaway_vmid` (REQUIRED —
|
|
112
|
+
backupproof refuses to guess a VMID it will later destroy),
|
|
113
|
+
`storage` (default `"local"`). The restore runs `qmrestore` /
|
|
114
|
+
`pct restore` into the throwaway VMID, starts it, probes it
|
|
115
|
+
(with `--probe-url`), then stops and destroys **only that
|
|
116
|
+
VMID**. A pre-flight `qm/pct status <vmid>` refuses the run if
|
|
117
|
+
the VMID already exists. **Pack guard:** db/files packs cannot
|
|
118
|
+
verify a VM's filesystem from the host, so a Proxmox real-mode
|
|
119
|
+
config containing them is an ERROR (`PROXMOX_PACK_UNSUPPORTED`,
|
|
120
|
+
exit 2) naming the packs and the remedy (web checks with
|
|
121
|
+
`--probe-url`, or run backupproof inside the guest) — packs are
|
|
122
|
+
never silently dropped and PASSed.
|
|
123
|
+
|
|
124
|
+
### Repository integrity step (opt-in)
|
|
125
|
+
|
|
126
|
+
`--integrity` runs `restic check` (add `--read-data-subset PCT`
|
|
127
|
+
to also read that share of the pack data) or `borg check
|
|
128
|
+
--repository-only` (`--verify-data` for `borg check
|
|
129
|
+
--verify-data`) **before** restoring. A failed check is an ERROR
|
|
130
|
+
(`INTEGRITY_FAILED`, exit 2) carrying the tool's last 20 output
|
|
131
|
+
lines; a passed one adds `integrity: {tool, args, ok}` to the
|
|
132
|
+
report. Cost warning: restic's documentation notes that reading
|
|
133
|
+
pack data downloads it from the repository, which is slow and can
|
|
134
|
+
cost money on cloud storage. Plain `restic check` (no subset)
|
|
135
|
+
checks structure only. Use a small subset, not full reads, weekly.
|
|
136
|
+
|
|
137
|
+
### Dead-man's-switch ping (opt-in)
|
|
138
|
+
|
|
139
|
+
`--ping-url URL` sends one HTTP GET after the verdict: `URL` on
|
|
140
|
+
PASS, `URL/fail` on FAIL or ERROR — the healthchecks.io
|
|
141
|
+
convention, so a missed weekly run also alerts. 10-second
|
|
142
|
+
timeout. A ping failure prints a warning on stderr and never
|
|
143
|
+
changes the exit code. Nothing is sent unless the flag is given.
|
|
144
|
+
|
|
145
|
+
### Staleness gate
|
|
146
|
+
|
|
147
|
+
Top-level `max_age_days` (or `--max-age-days`, CLI wins): when
|
|
148
|
+
the latest backup is older than that, the verdict carries a
|
|
149
|
+
warning. With `--fail-on-stale`, the run refuses outright
|
|
150
|
+
(`STALE_BACKUP`, exit 2) before restoring anything. A backup
|
|
151
|
+
whose timestamp is missing or unparseable warns ("cannot
|
|
152
|
+
determine age") and never fails on staleness alone.
|
|
153
|
+
|
|
154
|
+
### Which packs verify what, per source
|
|
155
|
+
|
|
156
|
+
| Source | web pack | db pack | files pack |
|
|
157
|
+
|---|---|---|---|
|
|
158
|
+
| restic / borg (real) | served restored tree (`serve_dir`) or explicit URL | restored files | restored files |
|
|
159
|
+
| proxmox (real) | booted VM via probe URL | **refused** (pack guard) | **refused** (pack guard) |
|
|
160
|
+
| fixture (simulated) | served fixture webroot over real HTTP | fixture tree | fixture tree |
|
|
161
|
+
|
|
162
|
+
The report's `verified against:` line states this per run, and
|
|
163
|
+
`--report` JSON carries `boot_kind`, `boot_probed`,
|
|
164
|
+
`verified_against`, `warnings`, `restore_partial`, `integrity`
|
|
165
|
+
and `restore_seconds` fields. `restore_seconds` is the wall time
|
|
166
|
+
of the restore step only (not boot or probe) — RTO evidence; the
|
|
167
|
+
text shows `restore: … in 37.2s`. In simulated (fixture) mode it
|
|
168
|
+
times the fixture copy and is labelled `(simulated)`.
|
|
169
|
+
|
|
170
|
+
A run with no checks configured is an ERROR, never a PASS. A
|
|
171
|
+
check that cannot be verified is a FAIL, never skipped silently.
|
|
172
|
+
|
|
173
|
+
### Example PASS output
|
|
174
|
+
|
|
175
|
+
```
|
|
176
|
+
PASS (simulated): backup restores and boots in simulation, all smoke checks passed
|
|
177
|
+
backup: fixture snap-20261028 (2026-10-28T03:00:00Z)
|
|
178
|
+
mode: simulated (SIMULATED restore/boot — not a real VM boot)
|
|
179
|
+
restore: SIMULATED restore into /tmp/backupproof-yoi9_m9l/restored (3 files copied from fixture)
|
|
180
|
+
boot: SIMULATED boot completed in 42s (fixture boot.json; not a real VM boot); serving webroot webroot at http://127.0.0.1:34147
|
|
181
|
+
verified against: restored fixture tree (web pack served over real HTTP from the restored webroot)
|
|
182
|
+
PASS [web] homepage: HTTP 200, body contains 'Welcome'
|
|
183
|
+
PASS [db] users: query returned 1 row(s): [('alice',)]
|
|
184
|
+
PASS [files] config: app.conf: 10 bytes, sha256 matches
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
A real restic/borg run reports instead:
|
|
188
|
+
`PASS: backup restores, all smoke checks passed (file-level backup — content verified, no VM boot)` —
|
|
189
|
+
and a Proxmox run with no probe URL reports
|
|
190
|
+
`PASS: backup restores, VM started (unprobed), all smoke checks passed`.
|
|
191
|
+
|
|
192
|
+
### Example FAIL output (failing check named)
|
|
193
|
+
|
|
194
|
+
Change the restored homepage so it no longer contains "Welcome"
|
|
195
|
+
and the same run reports:
|
|
196
|
+
|
|
197
|
+
```
|
|
198
|
+
FAIL: 1 of 3 smoke check(s) failed
|
|
199
|
+
backup: fixture snap-20261028 (2026-10-28T03:00:00Z)
|
|
200
|
+
mode: simulated (SIMULATED restore/boot — not a real VM boot)
|
|
201
|
+
restore: SIMULATED restore into /tmp/backupproof-y8b2y4d3/restored (3 files copied from fixture)
|
|
202
|
+
boot: SIMULATED boot completed in 42s (fixture boot.json; not a real VM boot); serving webroot webroot at http://127.0.0.1:45629
|
|
203
|
+
FAIL [web] homepage (WEB_CONTENT_MISSING): HTTP 200 but body lacks 'Welcome' (first 200 chars: 'Something else entirely\n')
|
|
204
|
+
PASS [db] users: query returned 1 row(s): [('alice',)]
|
|
205
|
+
PASS [files] config: app.conf: 10 bytes, sha256 matches
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
Named failures: web — `WEB_UNREACHABLE`, `WEB_STATUS_MISMATCH`,
|
|
209
|
+
`WEB_CONTENT_MISSING`; DB — `DB_FILE_MISSING`, `DB_QUERY_FAILED`,
|
|
210
|
+
`DB_NO_ROWS`, `DB_CONTENT_MISSING`, `DB_COMMAND_FAILED`; files —
|
|
211
|
+
`FILE_MISSING`, `FILE_SIZE_MISMATCH`, `FILE_HASH_MISMATCH`,
|
|
212
|
+
`FILE_STALE`, `FILE_MODE_MISMATCH`, `FILE_OWNER_MISMATCH`.
|
|
213
|
+
Errors (exit 2): `NO_BACKUP_FOUND`, `RESTORE_FAILED`,
|
|
214
|
+
`BOOT_FAILED`, `BOOT_TIMEOUT`, `MISSING_TOOL`, `CONFIG_ERROR`,
|
|
215
|
+
`INTEGRITY_FAILED`.
|
|
216
|
+
|
|
217
|
+
### Try it (fixture example, no backup tools needed)
|
|
218
|
+
|
|
219
|
+
```sh
|
|
220
|
+
backupproof run --config examples/config.json
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
The example fixture restores a small tree (webroot, `app.conf`,
|
|
224
|
+
SQLite `app.db`) and simulates boot while genuinely serving the
|
|
225
|
+
webroot over real localhost HTTP, so the web check is really
|
|
226
|
+
exercised.
|
|
227
|
+
|
|
228
|
+
## Weekly cadence (doc example only — no built scheduler)
|
|
229
|
+
|
|
230
|
+
backupproof has no daemon and no built-in scheduler by design.
|
|
231
|
+
Run it weekly from cron or a systemd timer. Cron example (Sundays
|
|
232
|
+
03:30, proof report kept per run):
|
|
233
|
+
|
|
234
|
+
```cron
|
|
235
|
+
30 3 * * 0 /usr/local/bin/backupproof run --config /etc/backupproof.json --report /var/log/backupproof/proof-$(date +\%F).json >> /var/log/backupproof/weekly.log 2>&1
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
systemd timer example (same cadence, same single command):
|
|
239
|
+
|
|
240
|
+
```ini
|
|
241
|
+
# /etc/systemd/system/backupproof.service
|
|
242
|
+
[Unit]
|
|
243
|
+
Description=Weekly backup restore test
|
|
244
|
+
[Service]
|
|
245
|
+
Type=oneshot
|
|
246
|
+
ExecStart=/usr/local/bin/backupproof run --config /etc/backupproof.json --report /var/log/backupproof/proof.json
|
|
247
|
+
|
|
248
|
+
# /etc/systemd/system/backupproof.timer
|
|
249
|
+
[Unit]
|
|
250
|
+
Description=Weekly backupproof run
|
|
251
|
+
[Timer]
|
|
252
|
+
OnCalendar=Sun 03:30
|
|
253
|
+
Persistent=true
|
|
254
|
+
[Install]
|
|
255
|
+
WantedBy=timers.target
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
## What was live-tested here vs. simulated — read this first
|
|
259
|
+
|
|
260
|
+
The sandbox has **no `/dev/kvm`, no QEMU, no Proxmox host**.
|
|
261
|
+
Restic and borg were NOT available at v0.1.0 build time — but
|
|
262
|
+
v0.2.0 installed real release binaries (restic 0.17.3, borg
|
|
263
|
+
1.4.1, downloaded from their GitHub releases; `apt` was unusable
|
|
264
|
+
— its update hung on a slow mirror) and ran the real paths live.
|
|
265
|
+
So, honestly:
|
|
266
|
+
|
|
267
|
+
- **Live-tested here (real code, real execution, covered by the
|
|
268
|
+
test suite):** backup discovery parsing for restic JSON,
|
|
269
|
+
borg JSON, and Proxmox vzdump filenames; latest-backup
|
|
270
|
+
selection; restic/borg/qmrestore/pct command construction;
|
|
271
|
+
all three smoke packs (real HTTP fetches, real SQLite queries,
|
|
272
|
+
real file hashing); verdict logic including mixed results;
|
|
273
|
+
error honesty (no backup, restore failure, boot timeout);
|
|
274
|
+
exit codes; end-to-end fixture runs (PASS and FAIL); and —
|
|
275
|
+
**new in v0.2.0 — real restic and real borg end-to-end runs:**
|
|
276
|
+
a real restic repo and a real borg repo were created, backed
|
|
277
|
+
up, listed, restored by `RealRunner`, and smoke-checked by the
|
|
278
|
+
full CLI — both reported `PASS: backup restores, all smoke
|
|
279
|
+
checks passed (file-level backup — content verified, no VM
|
|
280
|
+
boot)`, exit 0, with all three packs genuinely passing
|
|
281
|
+
(web served from the restored tree, real SQLite query, real
|
|
282
|
+
sha256). The restic/borg real mode is no longer simulated.
|
|
283
|
+
- **Fixture/simulated here, labeled as such everywhere it
|
|
284
|
+
appears:** the restore+boot layer when run with a `fixture`
|
|
285
|
+
source. Fixture restore copies a fixture tree; fixture boot is
|
|
286
|
+
simulated from `boot.json` (it does start a real localhost HTTP
|
|
287
|
+
server for the webroot, so the web pack is genuinely checked).
|
|
288
|
+
Every simulated result prints `mode: simulated` and
|
|
289
|
+
`SIMULATED restore/boot` in text and `"mode": "simulated"` in
|
|
290
|
+
JSON. A simulated result is never a claim that a real backup
|
|
291
|
+
restored or booted.
|
|
292
|
+
- **Command-sequence tested, NOT executed against a real host:**
|
|
293
|
+
the Proxmox flow (pre-flight status, `qmrestore`/`pct restore`,
|
|
294
|
+
start, probe, stop, destroy) is tested end-to-end against fake
|
|
295
|
+
`qm`/`qmrestore` binaries that log every invocation — the
|
|
296
|
+
sequence, the existing-VMID refusal, the missing-VMID refusal,
|
|
297
|
+
and cleanup-on-failure are all proven, but no real Proxmox host
|
|
298
|
+
was touched. The old "boot the restored `.qcow2` with QEMU"
|
|
299
|
+
path was **removed** in v0.2.0: no v0.2.0 flow boots a local
|
|
300
|
+
disk image (Proxmox boots on the Proxmox host; restic/borg are
|
|
301
|
+
file-level), and the QEMU helper survives only as a tested
|
|
302
|
+
utility for a future local-image source.
|
|
303
|
+
|
|
304
|
+
**What a real host needs for the full path:** restic and/or borg
|
|
305
|
+
installed (passwords via `RESTIC_PASSWORD` / `BORG_PASSPHRASE`
|
|
306
|
+
env), or a Proxmox host with a dump dir + a throwaway VMID.
|
|
307
|
+
If a tool is missing, `run` exits 2 with `MISSING_TOOL` — it
|
|
308
|
+
never reports PASS on an unverified restore.
|
|
309
|
+
|
|
310
|
+
## What v1 does NOT do (deliberately)
|
|
311
|
+
|
|
312
|
+
Fleet dashboards, a built scheduler service, team-tier features,
|
|
313
|
+
Windows support, VMware estates, any paid gating, and any smoke
|
|
314
|
+
pack beyond web, DB, and files. The paid team tier (C$19/mo,
|
|
315
|
+
later, not v1) and any fourth pack are out of scope until the owner
|
|
316
|
+
verdicts this pair.
|