dar-backup 1.0.1__py3-none-any.whl → 1.0.2__py3-none-any.whl

dar_backup/Changelog.md

@@ -1,6 +1,35 @@
 <!-- markdownlint-disable MD024 -->
 # dar-backup Changelog
 
+## v2-1.0.2 - 2026-01-25
+
+### Added
+
+- Streaming restore-test sampling using reservoir sampling to avoid holding full file lists in memory.
+- Configurable command output capture cap (`COMMAND_CAPTURE_MAX_BYTES`, default 100 KB) to limit in-memory stdout/stderr while still logging full output.
+- Streaming list output for `dar-backup --list-contents` and `manager --list-archive-contents` to avoid large in-memory buffers.
+- Test coverage additions for config parsing, util helpers, restore-test sampling edge cases, par2 slice helpers, and get_backed_up_files error paths.
+- Tests for missing source files during restore verification and for par2 generation order after verification.
+- CommandRunner test coverage for sanitize failure notes, text/binary output handling, timeouts, Popen failures, and TTY restore logic.
+- Tests for COMMAND_CAPTURE_MAX_BYTES defaults (0 and 1k) and binary stdout/stderr capture with truncation and log_output disabled.
+- Manager test coverage for create-db guardrails and catalog listing parsing/sorting across runner/subprocess paths.
+- Cleanup now reports PREREQ/POSTREQ failures cleanly and sends Discord failure notifications when configured.
+- New trace logger that always logs at DEBUG and captures stacktraces if they happens. Default max size is 10MB + 1 rollover file.
+
+### Changed
+
+- BUGFIX: Ensure existing files are removed before restore verification to prevent false positives.
+- Clears out restore-test directory on program start to ensure a clean slate.
+- Restore-test selection now streams DAR XML listings and samples candidates without loading all entries into RAM.
+- `get_backed_up_files` uses incremental XML parsing to reduce memory use for large archives.
+- Restore verification now logs a warning and continues when a source or restored file is missing during comparison.
+- CommandRunner supports per-command capture cap overrides (disable cap with `capture_output_limit_bytes=-1`).
+- Cleanup now rejects unsafe archive names when `--cleanup-specific-archives` is used to prevent accidental deletions.
+- Removed deprecated PAR2 layout/mode settings and simplified PAR2 cleanup to delete all matching .par2 artifacts.
+- Config templates/docs updated to drop PAR2_LAYOUT/PAR2_MODE references.
+- [Snyk] Python 3.11 required in pyproject.toml. Snyk has flagged a vulnerabilly in an xml parser that requires the bump to 3.11.
+- Config parsing errors now emit concise messages (no stack trace) and trigger Discord failure notifications in CLI tools.
+
 ## v2-1.0.1 - 2026-01-09
 
 ### Added

dar_backup/README.md

@@ -95,8 +95,8 @@ Version **1.0.0** was reached on October 9, 2025.
     - [Performance tip due to par2](#performance-tip-due-to-par2)
     - [.darrc sets -vd -vf (since v0.6.4)](#darrc-sets--vd--vf-since-v064)
     - [Separate log file for command output](#separate-log-file-for-command-output)
+    - [Trace Logging (Debug details)](#trace-logging-debug-details)
     - [Skipping cache directories](#skipping-cache-directories)
-    - [Progress bar and current directory](#progress-bar-and-current-directory)
     - [Shell autocompletion](#shell-autocompletion)
       - [Use it](#use-it)
       - [Archive name completion (smart, context-aware)](#archive-name-completion-smart-context-aware)
@@ -123,6 +123,9 @@ Version **1.0.0** was reached on October 9, 2025.
         - [DISCORD WEBHOOK](#discord-webhook)
         - [Restore test config](#restore-test-config)
         - [Par2](#par2-1)
+      - [1.0.2](#102)
+        - [Trace Logging](#trace-logging)
+        - [Command output Capture](#command-output-capture)
   
 ## My use case
 
@@ -791,7 +794,6 @@ ERROR_CORRECTION_PERCENT = 5
 ENABLED = True
 # Optional PAR2 configuration
 # PAR2_DIR = /path/to/par2-store
-# PAR2_MODE = per-slice
 # PAR2_RATIO_FULL = 10
 # PAR2_RATIO_DIFF = 5
 # PAR2_RATIO_INCR = 5
@@ -800,7 +802,6 @@ ENABLED = True
 # Optional per-backup overrides (section name = backup definition)
 [media-files]
 PAR2_DIR = /mnt/par2/media-files
-PAR2_MODE = per-archive
 PAR2_RATIO_FULL = 10
 
 # scripts to run before the backup to setup the environment
@@ -818,6 +819,7 @@ PAR2 notes:
 - If `PAR2_DIR` is unset, par2 files are created next to the archive slices (legacy behavior) and no manifest is written
 - When `PAR2_DIR` is set, dar-backup writes a manifest next to the par2 set:
   `archive_base.par2.manifest.ini`
+- When generating a par2 set, par2 reads all archive slices before writing any output files; for large backups, this initial read can take hours
 - Verify or repair using:
   `par2 verify -B <archive_dir> <par2_set.par2>`
   `par2 repair -B <archive_dir> <par2_set.par2>`
@@ -1143,6 +1145,25 @@ This happens when the shell splits the quoted string or interprets globs before
 
 > 💡 **Tip:** See [dar's documentation](http://dar.linux.free.fr/doc/man/dar.html#COMMANDS%20AND%20OPTIONS)
 
+>
+> 💡💡 **Tip:** To filter all the empty directories away that `dar` emits when listing  contents, append this grep:
+>
+> ```bash
+> |grep -vE '\s+d[rwx-]{9}\s'
+>```
+>
+>Example using the grep to discard directory noise from `dar's` output:
+>
+> ```bash
+> dar-backup --list-contents media-files_INCR_2025-05-10 --selection="-I '*Z50*' -X '*.xmp'" | grep -vE '\s+d[rwx-]{9}\s'
+>[Saved][ ]       [-L-][   0%][X]  -rw-rw-r--   user user 26  Mio Fri May  9 11:26:16 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0633.NEF
+>[Saved][ ]       [-L-][   0%][X]  -rw-rw-r--   user user 26  Mio Fri May  9 11:26:16 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0632.NEF
+>[Saved][ ]       [-L-][   0%][X]  -rw-rw-r--   user user 28  Mio Fri May  9 11:09:04 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0631.NEF
+>[Saved][ ]       [-L-][   0%][X]  -rw-rw-r--   user user 29  Mio Fri May  9 11:09:03 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0630.NEF
+>[Saved][ ]       [-L-][   0%][X]  -rw-rw-r--   user user 29  Mio Fri May  9 11:09:03 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0629.NEF
+>...
+>```
+
 ### select a directory
 
 Select files and sub directories in `home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling`
@@ -1499,23 +1520,29 @@ In order to not clutter that log file with the output of commands being run, a n
 
 The secondary log file can get quite cluttered, if you want to remove the clutter, run the `clean-log`script with the `--file` option, or simply delete it.
 
-### Skipping cache directories
+### Trace Logging (Debug details)
 
-The author uses the `--cache-directory-tagging` option in his [backup definitions](#backup-definition-example).
+To keep the main log file clean while preserving essential debugging information, `dar-backup` creates a separate trace log file (e.g., `dar-backup.trace.log`) alongside the main log.
 
-The effect is that directories with the [CACHEDIR.TAG](https://bford.info/cachedir/) file are not backed up. Those directories contain content fetched from the net, which is of an ephemeral nature and probably not what you want to back up.
+- **Main Log (`dar-backup.log`)**: Contains clean, human-readable INFO/ERROR messages. Stack traces are suppressed here.
+- **Trace Log (`dar-backup.trace.log`)**: Captures ALL messages at `DEBUG` level, including full exception stack traces. Use this file for debugging crashes or unexpected behavior.
 
-If the option is not in the backup definition, the cache directories are backed up as any other.
+You can configure the rotation of this file in `[MISC]`:
 
-### Progress bar and current directory
+```ini
+[MISC]
+# ... other settings ...
+TRACE_LOG_MAX_BYTES = 10485760  # 10 MB default
+TRACE_LOG_BACKUP_COUNT = 1      # Keep 1 old trace file (default)
+```
 
-If you run dar-backup interactively in a "normal" console on your computer,
-dar-backup displays 2 visual artifacts to show progress.
+### Skipping cache directories
 
-1. a progress bar that fills up and starts over
-2. a status line showing the directory being backed up. If the directory is big and takes time to backup, the line is not changing, but you will probably know there is a lot to backup.
+The author uses the `--cache-directory-tagging` option in his [backup definitions](#backup-definition-example).
 
-The indicators are not shown if dar-backup is run from systemd or if it is used in terminal multiplexers like `tmux` or `screen`. So no polluting of journald logs.
+The effect is that directories with the [CACHEDIR.TAG](https://bford.info/cachedir/) file are not backed up. Those directories contain content fetched from the net, which is of an ephemeral nature and probably not what you want to back up.
+
+If the option is not in the backup definition, the cache directories are backed up as any other.
 
 ### Shell autocompletion
 
@@ -1700,7 +1727,6 @@ pytest                   # run the test suite
 
 - Perhaps look into pre-processing backup definitions. As `dar` does not expand env vars
   `dar-backup` could do so and feed the result to `dar`.
-- When run interactively, a progress bar during test and par2 generation would be nice.
 - Look into a way to move the .par2 files away from the `dar` slices, to maximize chance of good redundancy.
 - Add option to dar-backup to use the `dar` option `--fsa-scope none`
 
@@ -2036,10 +2062,44 @@ PAR2_RUN_VERIFY = true
 #
 #[etc]
 # Keep global PAR2 settings but tweak ratios for this backup definition
-# RATIO is i percent number
+# RATIO is given in percent (%)
 #PAR2_RATIO_FULL = 15  
 #PAR2_RATIO_DIFF = 8
 #PAR2_RATIO_INCR = 8
 ```
 
 [Per-backup override test case: `tests/test_par2_overrides.py`](v2/tests/test_par2_overrides.py)
+
+#### 1.0.2
+
+##### Trace Logging
+
+To support debugging without cluttering the main log file, a secondary trace log is now created (e.g., `dar-backup.trace.log`).
+This file captures all `DEBUG` level messages and full exception stack traces.
+
+You can configure its rotation in the `[MISC]` section:
+
+- `TRACE_LOG_MAX_BYTES`: Max size of the trace log file in bytes. Default is `10485760` (10 MB).
+- `TRACE_LOG_BACKUP_COUNT`: Number of rotated trace log files to keep. Default is `1`.
+
+Example:
+
+```ini
+[MISC]
+TRACE_LOG_MAX_BYTES = 10485760
+TRACE_LOG_BACKUP_COUNT = 1
+```
+
+##### Command output Capture
+
+- New optional `[MISC]` setting: `COMMAND_CAPTURE_MAX_BYTES` (default 102400).
+  - Limits how much stdout/stderr is kept in memory per command while still logging full output.
+  - Set to `0` to disable buffering entirely. Command output is still streamed to dar-backup-commands.log
+  - If set to `0`, the calling function cannot rely on output from the executed command. The exit value is the only result provided.
+
+Example:
+
+```ini
+[MISC]
+COMMAND_CAPTURE_MAX_BYTES = 102400
+```

dar_backup/__about__.py

@@ -1,8 +1,7 @@
-__version__ = "1.0.1"
+__version__ = "1.0.2"
 
 __author__ = "Per Jensen"
 
 __license__ = '''Licensed under GNU GENERAL PUBLIC LICENSE v3, see the supplied file "LICENSE" for details.
 THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW, not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 See section 15 and section 16 in the supplied "LICENSE" file.'''
-

dar_backup/clean_log.py

@@ -23,13 +23,54 @@ import re
 import os
 import sys 
 
+from datetime import datetime
+
 from dar_backup import __about__ as about
 from dar_backup.config_settings import ConfigSettings
+from dar_backup.util import send_discord_message, get_logger
 
 LICENSE = '''Licensed under GNU GENERAL PUBLIC LICENSE v3, see the supplied file "LICENSE" for details.
 THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW, not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 See section 15 and section 16 in the supplied "LICENSE" file.'''
 
+TIMESTAMP_RE = re.compile(r"^\d{4}-\d{2}-\d{2}\b")
+CLEAN_MESSAGE_PREFIXES = (
+    "Inspecting directory",
+    "Finished Inspecting",
+    "<File",
+    "</File",
+    "<Attributes",
+    "</Attributes",
+    "<Directory",
+    "</Directory",
+    "<Catalog",
+    "</Catalog",
+    "<Symlink",
+    "</Symlink",
+)
+
+def _split_level_and_message(line):
+    line = line.rstrip("\n")
+    if " - " not in line:
+        return None, None
+
+    parts = line.split(" - ")
+    if len(parts) >= 3 and TIMESTAMP_RE.match(parts[0].strip()):
+        level = parts[1]
+        message = " - ".join(parts[2:])
+    else:
+        level = parts[0]
+        message = " - ".join(parts[1:])
+
+    return level.strip(), message
+
+def _should_remove_line(line):
+    level, message = _split_level_and_message(line)
+    if level != "INFO" or message is None:
+        return False
+    message = message.lstrip()
+    return any(message.startswith(prefix) for prefix in CLEAN_MESSAGE_PREFIXES)
+
 def clean_log_file(log_file_path, dry_run=False):
     """Removes specific log lines from the given file using a memory-efficient streaming approach."""
 
@@ -42,7 +83,7 @@ def clean_log_file(log_file_path, dry_run=False):
         print(f"No read permission for '{log_file_path}'")
         sys.exit(1)
 
-    if not os.access(log_file_path, os.W_OK):
+    if not dry_run and not os.access(log_file_path, os.W_OK):
         print(f"Error: No write permission for '{log_file_path}'")
         sys.exit(1)
 
@@ -51,47 +92,20 @@ def clean_log_file(log_file_path, dry_run=False):
         print(f"Performing a dry run on: {log_file_path}")
 
     temp_file_path = log_file_path + ".tmp"
-    
-    patterns = [
-        r"INFO\s*-\s*Inspecting\s*directory",
-        r"INFO\s*-\s*Finished\s*Inspecting",
-        r"INFO\s*-\s*<File",
-        r"INFO\s*-\s*</File",
-        r"INFO\s*-\s*<Attributes",
-        r"INFO\s*-\s*</Attributes",
-        r"INFO\s*-\s*</Directory",
-        r"INFO\s*-\s*<Directory",
-        r"INFO\s*-\s*<Catalog",
-        r"INFO\s*-\s*</Catalog",
-        r"INFO\s*-\s*<Symlink",
-        r"INFO\s*-\s*</Symlink",
-    ]
 
     try:
-        with open(log_file_path, "r", errors="ignore") as infile, open(temp_file_path, "w") as outfile:
+        if dry_run:
+            with open(log_file_path, "r", errors="ignore") as infile:
+                for line in infile:
+                    if _should_remove_line(line):
+                        print(f"Would remove: {line.strip()}")
+            return
 
+        with open(log_file_path, "r", errors="ignore") as infile, open(temp_file_path, "w") as outfile:
             for line in infile:
-                original_line = line  # Store the original line before modifying it
-                matched = False  # Track if a pattern is matched
-
-                for pattern in patterns:
-                    if re.search(pattern, line):  # Check if the pattern matches
-                        if dry_run:
-                            print(f"Would remove: {original_line.strip()}")  # Print full line for dry-run
-                        matched = True  # Mark that a pattern matched
-                        break  # No need to check other patterns if one matches
-
-                if not dry_run and not matched:  # In normal mode, only write non-empty lines
+                if not _should_remove_line(line):
                     outfile.write(line.rstrip() + "\n")
 
-                if dry_run and matched:
-                    continue  # In dry-run mode, skip writing (since we’re just showing)
-
-        
-        # Ensure the temp file exists before renaming
-        if not os.path.exists(temp_file_path):
-            open(temp_file_path, "w").close()  # Create an empty file if nothing was written
-
         os.replace(temp_file_path, log_file_path)
         print(f"Successfully cleaned log file: {log_file_path}")
 
@@ -131,43 +145,68 @@ def main():
 
     args = parser.parse_args()
 
-    config_settings = ConfigSettings(os.path.expanduser(os.path.expandvars(args.config_file)))
+    try:
+        config_settings = ConfigSettings(os.path.expanduser(os.path.expandvars(args.config_file)))
+    except Exception as exc:
+        msg = f"Config error: {exc}"
+        print(msg, file=sys.stderr)
+        ts = datetime.now().strftime("%Y-%m-%d_%H:%M")
+        send_discord_message(f"{ts} - clean-log: FAILURE - {msg}")
+        sys.exit(127)
 
-    if not args.file:
-        args.file = [config_settings.logfile_location]
+    try:
+        files_to_clean = args.file if args.file else [config_settings.logfile_location]
+        logfile_dir = os.path.dirname(os.path.realpath(config_settings.logfile_location))
+        validated_files = []
 
-    for file_path in args.file:
+        for file_path in files_to_clean:
+            if not isinstance(file_path, (str, bytes, os.PathLike)):
+                print(f"Error: Invalid file path type: {file_path}")
+                sys.exit(1)
 
-        if ".." in os.path.normpath(file_path).split(os.sep):
-            print(f"Error: Path traversal is not allowed: '{file_path}'")
-            sys.exit(1)
+            file_path = os.fspath(file_path)
+            if isinstance(file_path, bytes):
+                file_path = os.fsdecode(file_path)
 
-        logfile_dir = os.path.dirname(os.path.realpath(config_settings.logfile_location))
-        resolved_path = os.path.realpath(file_path)
+            if file_path.strip() == "":
+                print(f"Error: Invalid empty filename '{file_path}'.")
+                sys.exit(1)
 
-        if not resolved_path.startswith(logfile_dir + os.sep):
-            print(f"Error: File is outside allowed directory: '{file_path}'")
-            sys.exit(1)
+            if ".." in os.path.normpath(file_path).split(os.sep):
+                print(f"Error: Path traversal is not allowed: '{file_path}'")
+                sys.exit(1)
 
-        # Validate the file path type and existence        
-        if not isinstance(file_path, (str, bytes, os.PathLike)):
-            print(f"Error: Invalid file path type: {file_path}")
-            sys.exit(1)
+            resolved_path = os.path.realpath(file_path)
 
-        if not os.path.exists(file_path):
-            print(f"Error: Log file '{file_path}' does not exist.")
-            sys.exit(1)
+            if not resolved_path.startswith(logfile_dir + os.sep):
+                print(f"Error: File is outside allowed directory: '{file_path}'")
+                sys.exit(1)
 
-        if file_path.strip() == "":
-            print(f"Error: Invalid empty filename '{file_path}'.")
-            sys.exit(1)
+            if not os.path.exists(file_path):
+                print(f"Error: Log file '{file_path}' does not exist.")
+                sys.exit(1)
 
-
-    # Run the log file cleaning function
-    for log_file in args.file:
-        clean_log_file(log_file, dry_run=args.dry_run)
-    print(f"Log file '{args.file}' has been cleaned successfully.")
+            validated_files.append(file_path)
 
 
+        # Run the log file cleaning function
+        for log_file in validated_files:
+            clean_log_file(log_file, dry_run=args.dry_run)
+        file_list = ", ".join(validated_files)
+        if args.dry_run:
+            print(f"Dry run complete for: {file_list}")
+        else:
+            print(f"Log file '{file_list}' has been cleaned successfully.")
+    except Exception as e:
+        msg = f"Unexpected error during clean-log: {e}"
+        logger = get_logger()
+        if logger:
+            logger.error(msg, exc_info=True)
+        else:
+            print(msg, file=sys.stderr)
+        
+        ts = datetime.now().strftime("%Y-%m-%d_%H:%M")
+        send_discord_message(f"{ts} - clean-log: FAILURE - {msg}", config_settings=config_settings)
+        sys.exit(1)
 if __name__ == "__main__":
     main()