dar-backup 1.0.2__tar.gz → 1.1.0__tar.gz

{dar_backup-1.0.2 → dar_backup-1.1.0}/.gitignore

@@ -19,5 +19,6 @@ v2/.coverage
 v2/.coverage*
 v2/dar-backup.tar
 v2/coverage.xml
+v2/doc/test-report/coverage.xml
 v2/packages/*
 v2/.coverage.*.*.*.*

{dar_backup-1.0.2 → dar_backup-1.1.0}/Changelog.md

@@ -1,6 +1,33 @@
 <!-- markdownlint-disable MD024 -->
 # dar-backup Changelog
 
+## v2-1.1.0 - not released
+
+### Added
+
+- Point-in-Time Recovery (PITR): restore paths as of a specific time via `manager --restore-path --when --target`, with safety checks, logging, and fallback restore when catalogs can’t resolve a dated restore.
+- PITR integration torture tests for rename/mtime traps and catalog rebuild after DB loss.
+- PITR dry-run chain report (`manager --pitr-report`) to preview archive selection and detect missing archives.
+- PITR fallback now reports missing archive slices and can notify via Discord when configured.
+- CommandRunner debug diagnostics now include process PID and elapsed time for each executed command.
+- Manager command to relocate archive paths inside catalog databases (`--relocate-archive-path`, with dry-run).
+- PITR restore logging now includes the exact archive chain (FULL/DIFF/INCR + timestamps + basenames).
+- Tests covering PITR direct-restore flow and relocate-archive-path CLI safeguards.
+- PITR preflight flag `--pitr-report-first` to run a chain report before restore and fail fast on missing archives.
+
+### Changed
+
+- The release.sh script is more strict and runs the full pytest suite and commits test reports to doc/test-report
+- Github `pytest` workflow uploads test reports in .json and .txt formats
+- [Snyk] An XML parsing function now strips DTD to avoid a class of XXE vulnerabilities
+- PITR fallback now restores via the latest FULL → DIFF → INCR chain and fails fast when required archives are missing.
+- PITR restore now requires `--target` and blocks unsafe restore targets by default.
+- Filtered `.darrc` temp files are created in a writable location and cleaned up reliably after runs.
+- PITR fallback now validates chain completeness instead of skipping missing archives.
+- `COMMAND_TIMEOUT_SECS = -1` disables timeouts for long-running operations.
+- Catalog rebuild (`manager --add-dir`) now adds archives in FULL → DIFF → INCR order for the same date to avoid dar_manager prompts.
+- PITR restores now always use direct `dar` chain application (skipping `dar_manager -w` restores) to avoid interactive prompts on non-monotonic mtimes.
+
 ## v2-1.0.2 - 2026-01-25
 
 ### Added

{dar_backup-1.0.2 → dar_backup-1.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dar-backup
-Version: 1.0.2
+Version: 1.1.0
 Summary: A script to do full, differential and incremental backups using dar.     Some files are restored from the backups during verification, after which par2 redundancy files are created.     The script also has a cleanup feature to remove old backups and par2 files.
 Project-URL: GPG Public Key, https://keys.openpgp.org/search?q=dar-backup@pm.me
 Project-URL: Homepage, https://github.com/per2jensen/dar-backup/tree/main/v2
@@ -690,23 +690,26 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Topic :: System :: Archiving :: Backup
 Requires-Python: >=3.11
 Requires-Dist: argcomplete>=3.6.2
+Requires-Dist: dateparser>=1.2.0
 Requires-Dist: inputimeout>=1.0.4
 Requires-Dist: jinja2>=3.1.6
 Requires-Dist: rich>=13.0.0
 Provides-Extra: dev
 Requires-Dist: anyio>=4.4.0; extra == 'dev'
-Requires-Dist: black>=25.1.0; extra == 'dev'
 Requires-Dist: coverage>=7.8.2; extra == 'dev'
 Requires-Dist: h11>=0.16.0; extra == 'dev'
 Requires-Dist: httpcore>=0.17.3; extra == 'dev'
 Requires-Dist: matplotlib>=3.10.3; extra == 'dev'
+Requires-Dist: mypy; extra == 'dev'
 Requires-Dist: pandas>=2.3.0; extra == 'dev'
 Requires-Dist: psutil>=7.0.0; extra == 'dev'
-Requires-Dist: pytest; extra == 'dev'
 Requires-Dist: pytest-cov>=6.1.1; extra == 'dev'
+Requires-Dist: pytest-json-report; extra == 'dev'
+Requires-Dist: pytest-mock; extra == 'dev'
 Requires-Dist: pytest-timeout>=2.4.0; extra == 'dev'
 Requires-Dist: pytest>=8.4.0; extra == 'dev'
 Requires-Dist: requests>=2.32.2; extra == 'dev'
+Requires-Dist: ruff; extra == 'dev'
 Requires-Dist: wheel>=0.45.1; extra == 'dev'
 Requires-Dist: zipp>=3.19.1; extra == 'dev'
 Provides-Extra: packaging
@@ -742,6 +745,18 @@ You can see the [v2 Changelog](v2/Changelog.md) for details on features and prog
 
 Version **1.0.0** was reached on October 9, 2025.
 
+**High-level flow**
+
+[![dar-backup overview](v2/doc/dar-backup-overview-small.png)](#dar-backup-principles)
+
+**Quick links**
+
+- [Quick Guide](#quick-guide)
+- [Redundancy to fix bitrot](#par2)
+- [Point In Time Recovery](#point-in-time-recovery-pitr)
+- [550+ unit tests & integration tests](#test-coverage)
+  - par2 bitrot repair proof, full/diff/incr with restore, PITR verifications, ...  
+
 ## Table of Contents
 
 - [`dar-backup`](#dar-backup)
@@ -785,6 +800,7 @@ Version **1.0.0** was reached on October 9, 2025.
     - [select a directory](#select-a-directory)
     - [select files with "Z50" in the file name and exclude .xmp files](#select-files-with-z50-in-the-file-name-and-exclude-xmp-files)
   - [Restoring](#restoring)
+    - [Point-in-Time Recovery (PITR)](#point-in-time-recovery-pitr)
     - [default location for restores](#default-location-for-restores)
     - [--restore-dir option](#--restore-dir-option)
     - [a single file](#a-single-file)
@@ -826,7 +842,10 @@ Version **1.0.0** was reached on October 9, 2025.
   - [Projects these scripts benefit from](#projects-these-scripts-benefit-from)
   - [Reference](#reference)
     - [CLI Tools Overview](#cli-tools-overview)
-    - [test coverage](#test-coverage)
+    - [Test coverage](#test-coverage)
+      - [Test selection with markers](#test-selection-with-markers)
+      - [Common runs](#common-runs)
+      - [Run all tests](#run-all-tests)
     - [Dar-backup options](#dar-backup-options)
       - [Dar-backup exit codes](#dar-backup-exit-codes)
       - [Dar-backup env vars](#dar-backup-env-vars)
@@ -844,6 +863,7 @@ Version **1.0.0** was reached on October 9, 2025.
       - [1.0.2](#102)
         - [Trace Logging](#trace-logging)
         - [Command output Capture](#command-output-capture)
+      - [1.1.0](#110)
   
 ## My use case
 
@@ -1490,6 +1510,7 @@ NO_FILES_VERIFICATION = 5
 # timeout in seconds for backup, test, restore and par2 operations
 # The author has such `dar` tasks running for 10-15 hours on the yearly backups, so a value of 24 hours is used.
 # If a timeout is not specified when using the util.run_command(), a default timeout of 30 secs is used.
+# Use -1 to disable timeouts.
 COMMAND_TIMEOUT_SECS = 86400
 
 [DIRECTORIES]
@@ -1930,6 +1951,97 @@ gives something like
 
 ## Restoring
 
+### Point-in-Time Recovery (PITR)
+
+Use the `manager` CLI to restore files as they existed at a specific time:
+
+```bash
+. <the virtual env>/bin/activate
+manager --config-file <dar-backup.conf> \
+  --backup-def <definition> \
+  --restore-path tmp/path/to/file.txt \
+  --when "2026-01-29 15:00:39" \
+  --target /tmp/restore_pitr \
+  --log-stdout --verbose
+deactivate
+```
+
+Restore a directory (same idea, but the path is a directory):
+
+```bash
+. <the virtual env>/bin/activate
+manager --config-file <dar-backup.conf> \
+  --backup-def <definition> \
+  --restore-path tmp/path/to/directory/ \
+  --when "2026-01-29 15:00:39" \
+  --target /tmp/restore_pitr \
+  --log-stdout --verbose
+deactivate
+```
+
+Dry-run the archive chain selection before restoring:
+
+```bash
+. <the virtual env>/bin/activate
+manager --config-file <dar-backup.conf> \
+  --backup-def <definition> \
+  --restore-path tmp/path/to/directory \
+  --when "2026-01-29 15:00:39" \
+  --pitr-report \
+  --log-stdout --verbose
+deactivate
+```
+
+**Notes**:
+- `--restore-path` must be a relative path as stored in the catalog (no leading slash).
+- If a restore path is a **directory** and its name has no file extension, add a trailing `/` to make the intent explicit (e.g., `photos/2026/01/`). This avoids ambiguity with file paths that also lack extensions.
+  - Example (directory name has no extension):
+    - `manager --backup-def <definition> --restore-path "Automatic Upload/Per - iPhone/2026/01/" --when "now" --target /tmp/restore_pitr`
+- `--target` is required to avoid accidental restores into the current working directory.
+- Protected targets are blocked (e.g., `/etc`, `/usr`, `/bin`, `/var`, `/root`, `/boot`, `/lib`, `/proc`, `/sys`, `/dev`).
+- `--pitr-report` does a **dry-run** chain selection; if it reports missing archives, a restore will fail until the catalog is rebuilt or missing archives are restored.
+- `--pitr-report-first` runs the same chain report before a restore and aborts if any archive is missing (useful as a safety preflight).
+- `--when` accepts natural-language date expressions via `dateparser`. Examples:
+  - `"now"`
+  - `"2 weeks ago"`
+  - `"2025-10-05 14:30"`
+  - `yesterday 23:00`
+- PITR restores use the catalog to select the correct archive chain (FULL → DIFF → INCR) and then restore **directly with `dar`** in that order.
+  - This avoids interactive `dar_manager` prompts (e.g., non‑monotonic mtimes often seen on pCloud/FUSE).
+  - Directories can get a **new mtime** when files inside them are added/removed; the chain restore ensures the correct tree is rebuilt even if mtimes look “too new”.
+- Missing archives:
+  - PITR uses the latest FULL, the latest DIFF after that FULL, and the latest INCR after that DIFF.
+  - If any archive slice in that chain is missing on disk, PITR restore **fails** and logs which archive slices are missing.
+  - A short Discord notice is sent (if configured) so missing archives are visible immediately.
+- Relocating archive paths in the catalog:
+  - The catalog stores **absolute archive paths**. If archives move (or a mountpoint changes), the catalog will still point to the old path.
+  - This can happen when manager DBs are moved to another disk and the archives are re-added from a different mountpoint.
+  - Use the built-in relocate command to rewrite a path prefix in-place:
+    - Dry run:
+      - `manager --relocate-archive-path /old/path /new/path --relocate-archive-path-dry-run --backup-def <definition>`
+    - Apply:
+      - `manager --relocate-archive-path /old/path /new/path --backup-def <definition>`
+  - Example (move `/home/pj/mnt/dar` to `/mnt/dar`):
+    - `manager --relocate-archive-path /home/pj/mnt/dar /mnt/dar --backup-def pCloudDrive`
+  - Alternative quick fix: create a symlink from the old path to the new path.
+- Rebuilding a catalog after archive loss:
+  - If PITR fails due to missing archives, the catalog may no longer match what is actually on disk.
+  - You can rebuild the catalog from the remaining archives and then retry PITR (with the understanding that older restore points may no longer be possible).
+  - Example:
+    - `manager --create-db --config-file <dar-backup.conf>`
+    - `manager --add-dir <backup_dir> --backup-def <definition> --config-file <dar-backup.conf>`
+  - Or add individual archives:
+    - `manager --add-specific-archive <path/to/archive> --config-file <dar-backup.conf>`
+
+Example of the issue:
+1) FULL backup at 10:00 with `/data/photos/`  
+2) You add files at 11:00 (directory mtime updates)  
+3) DIFF backup at 11:05  
+4) You request PITR restore of `/data/photos/` at 10:30  
+
+`dar_manager -w` may say “directory did not exist before that time” because the directory mtime is now 11:00+.  
+The fallback still restores the correct tree as of 10:30 by applying the archive chain.
+
 ### default location for restores
 
 dar-backup will use the TEST_RESTORE_DIR location as the Root for restores, if the --restore-dir option has not been supplied.
@@ -2479,40 +2591,135 @@ One backup definition per file
 | [installer](#installer-options) | Set up directories and optionally create catalog databases according to a config file |
 | [demo](#demo-options) | Set up required directories and config files for a demo|
 
-### test coverage
+### Test coverage
+
+#### Test selection with markers
+
+The test suite is annotated with these markers:
+
+- `unit` (fast, pure logic)
+- `component` (subprocess boundary with mocks/lightweight commands)
+- `integration` (end-to-end workflows; external tools)
+- `slow` (long-running/heavier integration)
+- `live_discord` (sends real webhook messages; opt-in only)
+
+#### Common runs
 
-Running
+```bash
+# Fast local loop (unit + component)
+pytest -m "unit or component"
+
+# Integration (exclude slow + live webhook)
+pytest -m "integration and not slow and not live_discord"
+
+# Slow-only
+pytest -m slow
+
+# Full suite (default pytest.ini already excludes live_discord)
+pytest -m "not live_discord"
+
+# Live webhook (requires DAR_BACKUP_DISCORD_WEBHOOK_URL)
+pytest -m live_discord
+```
+
+#### Run all tests
 
 ```bash
-pytest --cov=dar_backup tests/
+pytest
 ```
 
-Results for version 1.0.0 in this report:
+Results for version 1.1.0 (pre-release, Feb 1, 2026) in this report:
 
 ```text
-====================================== tests coverage ======================================
-_____________________ coverage: platform linux, python 3.12.3-final-0 ______________________
+===================================== test session starts =====================================
+platform linux -- Python 3.12.3, pytest-8.4.0, pluggy-1.6.0
+rootdir: /home/pj/git/dar-backup/v2
+configfile: pytest.ini
+testpaths: tests
+plugins: anyio-4.9.0, timeout-2.4.0, cov-6.1.1, mock-3.15.1
+timeout: 1800.0s
+timeout method: signal
+timeout func_only: False
+collected 559 items / 1 deselected / 558 selected                                             
+
+tests/test_add_old_archive_confirmation.py ....                                         [  0%]
+tests/test_alternate_reference_archive.py ...                                           [  1%]
+tests/test_autocompletion_install.py ...                                                [  1%]
+tests/test_binary_info.py ......                                                        [  2%]
+tests/test_bitrot.py ..                                                                 [  3%]
+tests/test_clean_log.py ....................                                            [  6%]
+tests/test_cleanup.py ...............................                                   [ 12%]
+tests/test_command_runner.py .........................................                  [ 19%]
+tests/test_config_comments.py .                                                         [ 19%]
+tests/test_config_settings.py ..............                                            [ 22%]
+tests/test_create_backup_command.py ...                                                 [ 22%]
+tests/test_create_full_diff_incr_backup.py ..........                                   [ 24%]
+tests/test_dar_backup.py .......................................................        [ 34%]
+tests/test_dar_backup_additional_coverage.py .......................                    [ 38%]
+tests/test_dar_backup_startup.py ..                                                     [ 39%]
+tests/test_darrc.py ..                                                                  [ 39%]
+tests/test_demo.py .............                                                        [ 41%]
+tests/test_discord_webhook.py ..                                                        [ 42%]
+tests/test_filter_darrc_file.py .                                                       [ 42%]
+tests/test_generic_backup_command_execution.py .                                        [ 42%]
+tests/test_get_config_file.py .......                                                   [ 43%]
+tests/test_gpt_file_compression.py .                                                    [ 43%]
+tests/test_gpt_tests.py .......                                                         [ 45%]
+tests/test_installer.py ..................                                              [ 48%]
+tests/test_links.py .                                                                   [ 48%]
+tests/test_list_definitions.py .                                                        [ 48%]
+tests/test_listing.py ...                                                               [ 49%]
+tests/test_logging_trace.py ..                                                          [ 49%]
+tests/test_manager.py ................................................................. [ 61%]
+                                                                                        [ 61%]
+tests/test_manager_coverage.py ..................................                       [ 67%]
+tests/test_par2.py .                                                                    [ 67%]
+tests/test_par2_manifest.py .                                                           [ 67%]
+tests/test_par2_multi_definitions.py .                                                  [ 67%]
+tests/test_par2_overrides.py .                                                          [ 68%]
+tests/test_pitr.py ..............................................                       [ 76%]
+tests/test_pitr_integration.py .....                                                    [ 77%]
+tests/test_postreq.py .                                                                 [ 77%]
+tests/test_preflight.py ...                                                             [ 77%]
+tests/test_prereq.py .                                                                  [ 78%]
+tests/test_readme_changelog.py ..........                                               [ 79%]
+tests/test_restore.py ....                                                              [ 80%]
+tests/test_run_command.py ......s                                                       [ 81%]
+tests/test_sanity_checks.py .................                                           [ 84%]
+tests/test_space_definition.py .                                                        [ 85%]
+tests/test_startup_cleanup.py .......                                                   [ 86%]
+tests/test_status_indicators.py .......                                                 [ 87%]
+tests/test_stress.py .                                                                  [ 87%]
+tests/test_systemd_unit_generation.py ..........                                        [ 89%]
+tests/test_trace_logging.py ..                                                          [ 89%]
+tests/test_util.py ..........................................                           [ 97%]
+tests/test_util_completers.py ..........                                                [ 99%]
+tests/test_verbose.py ...                                                               [ 99%]
+tests/test_verify_cleanup.py .                                                          [100%]
+
+======================================= tests coverage ========================================
+_______________________ coverage: platform linux, python 3.12.3-final-0 _______________________
 
 Name                                   Stmts   Miss Branch BrPart  Cover
 ------------------------------------------------------------------------
-src/dar_backup/__about__.py                2      0      0      0   100%
+src/dar_backup/__about__.py                3      0      0      0   100%
 src/dar_backup/__init__.py                 0      0      0      0   100%
-src/dar_backup/clean_log.py               76     15     40      7    81%
-src/dar_backup/cleanup.py                203     16     66      7    91%
-src/dar_backup/command_runner.py         129     16     26      5    86%
-src/dar_backup/config_settings.py         73      3     18      1    96%
-src/dar_backup/dar_backup.py             551     28    180     21    93%
-src/dar_backup/dar_backup_systemd.py      56      7     10      2    86%
-src/dar_backup/demo.py                    92     20     26      6    75%
+src/dar_backup/clean_log.py              114     10     46      2    91%
+src/dar_backup/cleanup.py                294     42    100      9    87%
+src/dar_backup/command_runner.py         231     20     82     10    90%
+src/dar_backup/config_settings.py        162      9     62     10    92%
+src/dar_backup/dar_backup.py            1027     55    420     49    92%
+src/dar_backup/dar_backup_systemd.py      56      1     10      1    97%
+src/dar_backup/demo.py                   100      1     34      2    98%
 src/dar_backup/exceptions.py               2      0      0      0   100%
-src/dar_backup/installer.py              107     14     46      9    82%
-src/dar_backup/manager.py                427     42    152     15    90%
-src/dar_backup/rich_progress.py           70      7     30      4    89%
-src/dar_backup/util.py                   339     41     88     21    85%
+src/dar_backup/installer.py              120      4     54      4    95%
+src/dar_backup/manager.py               1105     91    454     38    91%
+src/dar_backup/rich_progress.py           70      2     30      3    95%
+src/dar_backup/util.py                   521     43    142     24    90%
 ------------------------------------------------------------------------
-TOTAL                                   2127    209    682     98    89%
-Coverage XML written to file coverage.xml
-======================== 257 passed, 1 skipped in 223.21s (0:03:43) ========================
+TOTAL                                   3805    278   1434    152    91%
+Coverage XML written to file /tmp/coverage.xml
+================== 557 passed, 1 skipped, 1 deselected in 409.66s (0:06:49) ===================
 ```
 
 ### Dar-backup options
@@ -2584,6 +2791,13 @@ Available options:
 -l, --list-catalogs                  List catalogs in databases for all backup definitions.
 --list-archive-contents <archive>    List the contents of an archive’s catalog by archive name.
 --find-file <file>                   Search catalogs for a specific file.
+--restore-path <path> [<path> ...]   Restore specific path(s) (Point-in-Time Recovery).
+--when <timestamp>                   Date/time for restoration (used with --restore-path).
+--target <path>                      Target directory for restoration (default: current dir).
+--pitr-report                        Report PITR archive chain for --restore-path/--when without restoring.
+--pitr-report-first                  Run PITR chain report before restore and abort if missing archives.
+--relocate-archive-path <old> <new>  Rewrite archive path prefix in the catalog DB (requires --backup-def).
+--relocate-archive-path-dry-run      Show archive path changes without applying them (use with --relocate-archive-path).
 --verbose                            Enable verbose output.
 --log-level <level>                  Set log level (`debug` or `trace`, default is `info`).
 
@@ -2821,3 +3035,7 @@ Example:
 [MISC]
 COMMAND_CAPTURE_MAX_BYTES = 102400
 ```
+
+#### 1.1.0
+
+COMMAND_TIMEOUT_SECS=-1 now disables timeout for commands executed.