lts-backup-cli 2.0.0__tar.gz → 3.2.0__tar.gz

{lts_backup_cli-2.0.0 → lts_backup_cli-3.2.0}/CLAUDE.md

@@ -1,6 +1,6 @@
 # CLAUDE.md — lts-backup-cli engineering guide
 
-`lts-backup-cli` (command: `ltsbu`, module: `lts_backup_cli`) is a Python console application. Running on one **admin box**, it backs up **servers** over SSH: one restic repo per **Apache/httpd vhost** (DocumentRoot), one **database repo** per server (MariaDB 10.4 databases, one snapshot each) and one repo per extra **folder**, all stored on **SFTP storage**, run by **systemd timers** (every 2h + weekly) and rotated after each run. `ltsbu discover <host>` creates and edits all of it. It targets **Rocky Linux 8.10** and **Ubuntu 22.04**, requires **Python >= 3.9**, and is distributed on **PyPI**. Requirements: `docs/SPEC2.md` (v2; `docs/SPEC.md` is the superseded v1).
+`lts-backup-cli` (command: `ltsbu`, module: `lts_backup_cli`) is a Python console application. Running on one **backup server** (the admin box, e.g. nas.mylogitools.pt), it **pulls** **servers** over SSH into **local restic repos**: one repo per **Apache/httpd vhost** (DocumentRoot), one **database repo** per server (MariaDB 10.4 databases, one snapshot each), one repo per extra **folder** and one **docker** repo per compose project (site) plus a host `docker` repo (volumes, container definitions, compose dirs, binds, plus a dump per MariaDB container), run by **systemd timers** (every 2h + weekly) and rotated after each run. `ltsbu discover <host>` creates and edits all of it. It targets **Rocky Linux 8.10** and **Ubuntu 22.04**, requires **Python >= 3.9**, and is distributed on **PyPI**. Requirements: `docs/SPEC3.md` (v3; `docs/SPEC2.md` and `docs/SPEC.md` are the superseded v2 and v1).
 
 ## Role
 
@@ -44,7 +44,7 @@ Adapted from the Ponytail plugin (`DietrichGebert/ponytail`). These rules overri
 
 | Pattern | Where |
 |---|---|
-| Adapter | `ResticClient`, `MariaDBClient` wrap external binaries |
+| Adapter | `ResticClient`, `MariaDBClient`, `Rsync` wrap external binaries |
 | Strategy | `doctor.Check` implementations |
 | Facade | `services.py` (`BackupService`, `RestoreService`, `ForgetService`, `DiscoverService`) |
 | Command | click subcommands in `cli.py` |
@@ -52,8 +52,8 @@ Adapted from the Ponytail plugin (`DietrichGebert/ponytail`). These rules overri
 
 ### Code isolation (ports & adapters)
 
-- **Pure core**: `validation.py`, `sshconfig.py` and `apache.py` (text in, text out), `config.py` (dataclasses, lint) and the rendering half of `schedule.py`. These modules do no I/O beyond reading and writing files.
-- **The only place that spawns processes** is `runner.py`. Nothing else imports `subprocess`. Remote execution is `SshRunner`, a drop-in `Runner`; services get a runner already scoped to the server (`services.runner_for`).
+- **Pure core**: `validation.py`, `sshconfig.py`, `apache.py` and `docker.py` (text in, text out), `config.py` (dataclasses, lint) and the rendering half of `schedule.py`. These modules do no I/O beyond reading and writing files.
+- **The only place that spawns processes** is `runner.py`. Nothing else imports `subprocess`. Remote execution is `SshRunner`, a drop-in `Runner`. Services get the **local** runner (restic, rsync and pipelines run here) and reach the server with `services.runner_for` / `services.remote_argv` (one side of a local pipeline).
 - **`cli.py`** parses arguments, builds objects, prints output and maps exceptions to exit codes. It holds no business logic.
 
 ## Security rules (non-negotiable)
@@ -63,14 +63,15 @@ Adapted from the Ponytail plugin (`DietrichGebert/ponytail`). These rules overri
 - Never put secrets in argv or logs:
   - restic uses `--password-file`.
   - MariaDB uses `--defaults-extra-file=<0600 my.cnf>` on the server.
-  - Secrets reach a server on **stdin** (`services.put_secret`, `umask 077`), never in argv.
+  - The only secret written to a server (the MariaDB `my.cnf`) travels on **stdin** (`services.put_secret`, `umask 077`), never in argv. restic passwords never leave the backup server.
 - Secret files are created with mode `0600`, and secret dirs with mode `0700`.
 - Validate **and** sanitise all user input at the boundary (`validation.py`): labels, hostnames, ports, paths, DB identifiers and unix users.
 - `~/.ssh/config` edits:
   - Only touch the content between `# >>> ltsbu:<label> >>>` and `# <<< ltsbu:<label> <<<`.
   - Back up the file before writing.
-  - Write atomically (temp file + `os.replace` locally, temp file + `mv` on a server).
-- DB dumps are streamed (`mariadb-dump | restic backup --stdin`) and never touch a disk.
+  - Write atomically (temp file + `os.replace`).
+- DB dumps are streamed (`ssh … mariadb-dump | restic backup --stdin`, the pipe running on the backup server) and never touch a disk.
+- rsync's `--delete` only ever targets the local mirror; restores push with rsync **without** `--delete`.
 - Removing a repo (discover) stops timers and drops config; it **never deletes restic data**.
 
 ## Testing strategy (ISTQB-aligned)
@@ -79,13 +80,13 @@ Adapted from the Ponytail plugin (`DietrichGebert/ponytail`). These rules overri
 |---|---|---|---|
 | Unit / component | pytest | `tests/unit/` | pure functions, adapters' argv via fake `Runner`, services orchestration |
 | Integration / system | behave | `features/` (default tags) | full CLI through `click.testing.CliRunner`, fake runner, temp `$HOME` |
-| Smoke | sh | `tests/smoke/smoke.sh` | the **installed wheel** starts, every command is wired, empty/v1 configs fail correctly; CI after BDD and first in each E2E container |
-| Acceptance / E2E | behave `@e2e` | `features/` + `tests/docker/` | real SSH, restic, MariaDB 10.4, SFTP storage, Rocky 8.10 and Ubuntu 22.04 images |
+| Smoke | sh | `tests/smoke/smoke.sh` | the **installed wheel** starts, every command is wired, empty/v1/v2 configs fail correctly; CI after BDD and first in each E2E container |
+| Acceptance / E2E | behave `@e2e` | `features/` + `tests/docker/` | real SSH, rsync, restic, MariaDB 10.4 (docker CLI shimmed), Rocky 8.10 and Ubuntu 22.04 images |
 
 ### Test design techniques
 
 - Use equivalence partitioning and boundary values for every validator, e.g. ports `0, 1, 65535, 65536` and label lengths `0, 1, 63, 64`.
-- Use decision tables for restore modes (repo kind × `--target` / `--database` / `--into`).
+- Use decision tables for restore modes (repo kind × `--target` / `--database` / `--into` / `--container` / `--image`).
 - Use state transitions for SSH config blocks (absent → present → updated → removed, all idempotent) and for `discover` repos (new → edited → unchecked-kept → removed-on-confirm).
 
 ### Development workflow
@@ -96,8 +97,10 @@ Adapted from the Ponytail plugin (`DietrichGebert/ponytail`). These rules overri
 
 ## Domain notes (hard-won; keep them true)
 
-- **Commands:** `discover`, `doctor`, `list`, `backup`, `snapshot`, `restore`, `forget` and `credentials` (`export` / `import`). `discover` is the wizard (create + edit + remove); restic's own `init` is only an action inside it. v2 is a clean break: a config without `version = 2` is rejected with a hint, never migrated.
-- **Central pull:** ltsbu runs on the admin box; restic and the MariaDB client run **on the web server** over SSH, and the server reaches the storage through its own managed block `ltsbu:storage-<server>` in **its** `~/.ssh/config`. A server with `ssh_host = "localhost"` uses the plain local runner.
+- **Commands:** `discover`, `doctor`, `list`, `backup`, `snapshot`, `restore`, `forget` and `credentials` (`export` / `import`). `discover` is the wizard (create + edit + remove); restic's own `init` is only an action inside it. v3 is a clean break: a config without `version = 3` is rejected with a hint, never migrated.
+- **Central pull (v3):** ltsbu, restic, the repos (`repo_root`, default `/srv/restic`) and their passwords live **only on the backup server**. Files are rsynced into `mirror_root/<server>/<repo>` (absolute layout kept, `--relative`) and restic backs up that mirror; dumps run on the server and are piped into local restic. A server needs only sshd + rsync (+ MariaDB client / docker). There is no storage SSH block and nothing restic-related on servers. v1/v2 configs are rejected, never migrated. A server with `ssh_host = "localhost"` uses the plain local runner.
+- **Docker repos:** one per compose project (= one per site) plus the host repo `docker`. Discovery (`docker.PROBE`, `docker.propose`) gives each project its named volume dirs, compose working dir and bind sources that exist on the host (Portainer's `/data/compose/N` are container paths, dropped; shared binds are copied into each site), and the host repo `<DockerRootDir>/containers` (container definitions: the `docker run` Bitbucket runners exist only there) + `<DockerRootDir>/volumes` **excluding every project-owned volume**, so each volume is copied exactly once. Excludes `*-json.log` and the datadir volume of each dumped container. Containers with a `/var/lib/mysql` mount are dumped with `docker exec <c> mariadb-dump --all-databases` (constant script, name as argv). The files snapshot is tagged `files` so restore never picks a dump. Images listed in a docker repo's `images` are `docker save`d into restic on the **weekly** run only (GBs each; `<flat-name>.tar`, tag `image:<ref>`); `restore --image` streams one back into `docker load`. After each pull, mirror entries no repo path covers are removed (`rsync.stale`), so a dropped path leaves later snapshots. Reading the Docker root needs SSH as root.
+- **rsync exit 24** (files vanished during the transfer) is a warning, not a failure: servers are live.
 - **Binaries are resolved on the server** (`mariadb.probe_binaries`), never with a local `shutil.which`: MariaDB 10.4 may only ship the `mysql*` names.
 - **One snapshot per database:** `--stdin-filename <db>.sql` gives each DB a stable path, so restic finds the parent and `forget`'s default grouping (host, paths) rotates per DB. Every backup passes `--host <server label>`.
 - **Tag filters are AND-ed with one comma-joined `--tag a,b`.** Repeated `--tag` flags would OR them.
@@ -108,9 +111,9 @@ Adapted from the Ponytail plugin (`DietrichGebert/ponytail`). These rules overri
 - **Restore runs on the server** against its localhost MariaDB; there is no cross-server restore.
 - **ssh reads `~/.ssh/config` from the passwd home, not `$HOME`.** `sshconfig.default_path()` mirrors that. Tests must inject `ssh_config_path` and never touch the developer's real file.
 - **The managed SSH block uses `StrictHostKeyChecking accept-new`.** Unattended runs on a fresh machine would otherwise fail with BatchMode.
-- **Rocky 8: EPEL 8 has no restic.** Use the official release binary (README and `Dockerfile.rocky8`).
-- **Ubuntu 22.04 ships restic 0.12.1.** Only use restic flags that exist in 0.12 (`--stdin-filename`, `--host`, `--keep-hourly` do).
-- **E2E:** each OS image is the web server (sshd, vhost config, restic, MariaDB client on the shared `/run/mysqld` socket, user `webadmin`) **and** the admin box, which reaches it over real SSH. systemd isn't available in the containers, so E2E runs `discover --no-timers` and unit files are covered by unit tests. The smoke script (`tests/smoke/smoke.sh`) runs first, against the installed package.
+- **Rocky 8: EPEL 8 has no restic.** Use the official release binary (README and `Dockerfile.rocky8`). restic is only needed on the backup server.
+- **Ubuntu 22.04 ships restic 0.12.1.** Only use restic flags that exist in 0.12 (`--stdin-filename`, `--host`, `--keep-hourly` do) and keep `--stdin-filename` flat (no `/`).
+- **E2E:** each OS image is the web server (sshd, rsync, vhost config, MariaDB client on the shared `/run/mysqld` socket, user `webadmin`, a `docker` CLI shim in `tests/docker/docker-shim.sh` whose `exec` runs locally against the shared MariaDB) **and** the backup server (restic, local repos under the scenario's tmp dir), which pulls from it over real SSH. systemd isn't available in the containers, so E2E runs `discover --no-timers` and unit files are covered by unit tests. The smoke script (`tests/smoke/smoke.sh`) runs first, against the installed package.
 
 ## Knowledge graph (graphify)
 
@@ -128,7 +131,7 @@ pytest -q --cov=lts_backup_cli                                # unit
 behave --tags=~@e2e                                    # BDD (fast, no network)
 docker compose -f tests/docker/docker-compose.yml run --rm --build rocky8       # E2E (entrypoint runs behave @e2e)
 docker compose -f tests/docker/docker-compose.yml run --rm --build ubuntu2204
-docker compose -f tests/docker/docker-compose.yml down && rm -rf tests/docker/.e2e
+docker compose -f tests/docker/docker-compose.yml down && docker run --rm --entrypoint rm -v "$PWD/tests/docker:/d" mariadb:10.4 -rf /d/.e2e
 python -m build && twine check dist/*                  # packaging
 python -m venv /tmp/smoke && /tmp/smoke/bin/pip install dist/*.whl \
   && PATH=/tmp/smoke/bin:$PATH PYTHON=/tmp/smoke/bin/python sh tests/smoke/smoke.sh   # smoke