PyHardLinkBackup 1.3.0__tar.gz → 1.4.0__tar.gz

{pyhardlinkbackup-1.3.0 → pyhardlinkbackup-1.4.0}/.github/workflows/tests.yml

@@ -13,11 +13,12 @@ on:
 
 jobs:
   test:
-    runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
         python-version: ["3.14", "3.13", "3.12"]
+        os: [ubuntu-latest, macos-latest] # TODO: windows-latest
+    runs-on: ${{ matrix.os }}
     steps:
     - name: Checkout
       run: |

{pyhardlinkbackup-1.3.0 → pyhardlinkbackup-1.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: PyHardLinkBackup
-Version: 1.3.0
+Version: 1.4.0
 Summary: HardLink/Deduplication Backups with Python
 Project-URL: Documentation, https://github.com/jedie/PyHardLinkBackup
 Project-URL: Source, https://github.com/jedie/PyHardLinkBackup
@@ -42,22 +42,23 @@ The main command is `phlb backup <source> <destination>`:
 
 [comment]: <> (✂✂✂ auto generated backup help start ✂✂✂)
 ```
-usage: phlb backup [-h] source destination [--excludes [STR [STR ...]]] [-v]
+usage: phlb backup [-h] [BACKUP OPTIONS]
 
 Backup the source directory to the destination directory using hard links for deduplication.
 
-╭─ positional arguments ───────────────────────────────────────────────────────────────────────────────────────────────╮
-│ source       Source directory to back up. (required)                                                                 │
-│ destination  Destination directory for the backup. (required)                                                        │
-╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
-╭─ options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ -h, --help   show this help message and exit                                                                         │
-│ --excludes [STR [STR ...]]                                                                                           │
-│              List of directory or file names to exclude from backup. (default: __pycache__ .cache .temp .tmp .tox    │
-│              .nox)                                                                                                   │
-│ -v, --verbosity                                                                                                      │
-│              Verbosity level; e.g.: -v, -vv, -vvv, etc. (repeatable)                                                 │
-╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ positional arguments ──────────────────────────────────────────────────────────────────────────────────────╮
+│ source       Source directory to back up. (required)                                                        │
+│ destination  Destination directory for the backup. (required)                                               │
+╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ options ───────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ -h, --help   show this help message and exit                                                                │
+│ --excludes [STR [STR ...]]                                                                                  │
+│              List of directories to exclude from backup. (default: __pycache__ .cache .temp .tmp .tox .nox) │
+│ --verbosity {debug,info,warning,error}                                                                      │
+│              Log level for console logging. (default: warning)                                              │
+│ --log-file-level {debug,info,warning,error}                                                                 │
+│              Log level for the log file (default: info)                                                     │
+╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 [comment]: <> (✂✂✂ auto generated backup help end ✂✂✂)
 
@@ -174,7 +175,7 @@ The file contains only the relative path to the first hardlink of this file cont
 
 [comment]: <> (✂✂✂ auto generated dev help start ✂✂✂)
 ```
-usage: ./dev-cli.py [-h] {benchmark-hashes,coverage,install,lint,mypy,nox,pip-audit,publish,shell-completion,test,update,update-readme-history,update-test-snapshot-files,version}
+usage: ./dev-cli.py [-h] {benchmark-hashes,coverage,install,lint,mypy,nox,pip-audit,publish,scan-benchmark,shell-completion,test,update,update-readme-history,update-test-snapshot-files,version}
 
 
 
@@ -192,6 +193,8 @@ usage: ./dev-cli.py [-h] {benchmark-hashes,coverage,install,lint,mypy,nox,pip-au
 │   • nox        Run nox                                                                                               │
 │   • pip-audit  Run pip-audit check against current requirements files                                                │
 │   • publish    Build and upload this project to PyPi                                                                 │
+│   • scan-benchmark                                                                                                   │
+│                Benchmark our filesystem scan routine.                                                                │
 │   • shell-completion                                                                                                 │
 │                Setup shell completion for this CLI (Currently only for bash shell)                                   │
 │   • test       Run unittests                                                                                         │
@@ -228,6 +231,10 @@ Overview of main changes:
 
 [comment]: <> (✂✂✂ auto generated history start ✂✂✂)
 
+* [v1.4.0](https://github.com/jedie/PyHardLinkBackup/compare/v1.3.0...v1.4.0)
+  * 2026-01-16 - Create log file in backup and a summary.txt
+  * 2026-01-16 - Run CI tests on macos, too.
+  * 2026-01-16 - add dev cli command "scan-benchmark"
 * [v1.3.0](https://github.com/jedie/PyHardLinkBackup/compare/v1.2.0...v1.3.0)
   * 2026-01-15 - Verify SHA256SUMS files in "rebuild" command, too.
   * 2026-01-15 - Code cleanup: use more generic names for and in BackupProgress
@@ -243,11 +250,11 @@ Overview of main changes:
 * [v1.1.0](https://github.com/jedie/PyHardLinkBackup/compare/v1.0.1...v1.1.0)
   * 2026-01-14 - Change backup timestamp directory to old schema: '%Y-%m-%d-%H%M%S'
   * 2026-01-14 - Add "Overview of main changes" to README
-* [v1.0.1](https://github.com/jedie/PyHardLinkBackup/compare/v1.0.0...v1.0.1)
-  * 2026-01-13 - Store SHA256SUMS files in backup directories
 
 <details><summary>Expand older history entries ...</summary>
 
+* [v1.0.1](https://github.com/jedie/PyHardLinkBackup/compare/v1.0.0...v1.0.1)
+  * 2026-01-13 - Store SHA256SUMS files in backup directories
 * [v1.0.0](https://github.com/jedie/PyHardLinkBackup/compare/v0.13.0...v1.0.0)
   * 2026-01-13 - Change "./cli.py" to "phlb" (because it's the name installed via pipx)
   * 2026-01-13 - Update README

{pyhardlinkbackup-1.3.0 → pyhardlinkbackup-1.4.0}/PyHardLinkBackup/__init__.py

@@ -3,5 +3,5 @@
 """
 
 # See https://packaging.python.org/en/latest/specifications/version-specifiers/
-__version__ = '1.3.0'
+__version__ = '1.4.0'
 __author__ = 'Jens Diemer <PyHardLinkBackup@jensdiemer.de>'

{pyhardlinkbackup-1.3.0 → pyhardlinkbackup-1.4.0}/PyHardLinkBackup/backup.py

@@ -1,15 +1,16 @@
 import dataclasses
+import datetime
 import logging
 import os
 import shutil
 import sys
 import time
-from datetime import datetime
 from pathlib import Path
 
 from rich import print  # noqa
 
 from PyHardLinkBackup.constants import CHUNK_SIZE
+from PyHardLinkBackup.logging_setup import LoggingManager
 from PyHardLinkBackup.utilities.file_hash_database import FileHashDatabase
 from PyHardLinkBackup.utilities.file_size_database import FileSizeDatabase
 from PyHardLinkBackup.utilities.filesystem import (
@@ -20,9 +21,10 @@ from PyHardLinkBackup.utilities.filesystem import (
     read_and_hash_file,
     supports_hardlinks,
 )
-from PyHardLinkBackup.utilities.humanize import human_filesize
+from PyHardLinkBackup.utilities.humanize import PrintTimingContextManager, human_filesize
 from PyHardLinkBackup.utilities.rich_utils import DisplayFileTreeProgress
 from PyHardLinkBackup.utilities.sha256sums import store_hash
+from PyHardLinkBackup.utilities.tee import TeeStdoutContext
 
 
 logger = logging.getLogger(__name__)
@@ -31,6 +33,7 @@ logger = logging.getLogger(__name__)
 @dataclasses.dataclass
 class BackupResult:
     backup_dir: Path
+    log_file: Path
     #
     backup_count: int = 0
     backup_size: int = 0
@@ -152,7 +155,13 @@ def backup_one_file(
     store_hash(dst_path, file_hash)
 
 
-def backup_tree(*, src_root: Path, backup_root: Path, excludes: set[str]) -> BackupResult:
+def backup_tree(
+    *,
+    src_root: Path,
+    backup_root: Path,
+    excludes: tuple[str, ...],
+    log_manager: LoggingManager,
+) -> BackupResult:
     src_root = src_root.resolve()
     if not src_root.is_dir():
         print('Error: Source directory does not exist!')
@@ -176,15 +185,23 @@ def backup_tree(*, src_root: Path, backup_root: Path, excludes: set[str]) -> Bac
         sys.exit(1)
 
     # Step 1: Scan source directory:
-    src_file_count, src_total_size = humanized_fs_scan(src_root, excludes)
+    excludes: set = set(excludes)
+    with PrintTimingContextManager('Filesystem scan completed in'):
+        src_file_count, src_total_size = humanized_fs_scan(src_root, excludes=excludes)
 
     phlb_conf_dir = backup_root / '.phlb'
     phlb_conf_dir.mkdir(parents=False, exist_ok=True)
 
-    backup_dir = backup_root / src_root.name / datetime.now().strftime('%Y-%m-%d-%H%M%S')
-    logger.info('Backup %s to %s', src_root, backup_dir)
+    timestamp = datetime.datetime.now().strftime('%Y-%m-%d-%H%M%S')
+    backup_main_dir = backup_root / src_root.name
+    backup_dir = backup_main_dir / timestamp
     backup_dir.mkdir(parents=True, exist_ok=False)
 
+    log_file = backup_main_dir / f'{timestamp}-backup.log'
+    log_manager.start_file_logging(log_file)
+
+    logger.info('Backup %s to %s', src_root, backup_dir)
+
     print(f'\nBackup to {backup_dir}...\n')
 
     with DisplayFileTreeProgress(src_file_count, src_total_size) as progress:
@@ -192,7 +209,7 @@ def backup_tree(*, src_root: Path, backup_root: Path, excludes: set[str]) -> Bac
         size_db = FileSizeDatabase(phlb_conf_dir)
         hash_db = FileHashDatabase(backup_root, phlb_conf_dir)
 
-        backup_result = BackupResult(backup_dir=backup_dir)
+        backup_result = BackupResult(backup_dir=backup_dir, log_file=log_file)
 
         next_update = 0
         for entry in iter_scandir_files(src_root, excludes=excludes):
@@ -219,21 +236,25 @@ def backup_tree(*, src_root: Path, backup_root: Path, excludes: set[str]) -> Bac
         # Finalize progress indicator values:
         progress.update(completed_file_count=backup_result.backup_count, completed_size=backup_result.backup_size)
 
-    print(f'\nBackup complete: {backup_dir} (total size {human_filesize(backup_result.backup_size)})\n')
-    print(f'  Total files processed: {backup_result.backup_count}')
-    print(f'   * Symlinked files: {backup_result.symlink_files}')
-    print(
-        f'   * Hardlinked files: {backup_result.hardlinked_files}'
-        f' (saved {human_filesize(backup_result.hardlinked_size)})'
-    )
-    print(f'   * Copied files: {backup_result.copied_files} (total {human_filesize(backup_result.copied_size)})')
-    print(
-        f'     of which small (<{size_db.MIN_SIZE} Bytes)'
-        f' files: {backup_result.copied_small_files}'
-        f' (total {human_filesize(backup_result.copied_small_size)})'
-    )
-    if backup_result.error_count > 0:
-        print(f'  Errors during backup: {backup_result.error_count} (see log for details)')
-    print()
+    summary_file = backup_main_dir / f'{timestamp}-summary.txt'
+    with TeeStdoutContext(summary_file):
+        print(f'\nBackup complete: {backup_dir} (total size {human_filesize(backup_result.backup_size)})\n')
+        print(f'  Total files processed: {backup_result.backup_count}')
+        print(f'   * Symlinked files: {backup_result.symlink_files}')
+        print(
+            f'   * Hardlinked files: {backup_result.hardlinked_files}'
+            f' (saved {human_filesize(backup_result.hardlinked_size)})'
+        )
+        print(f'   * Copied files: {backup_result.copied_files} (total {human_filesize(backup_result.copied_size)})')
+        print(
+            f'     of which small (<{size_db.MIN_SIZE} Bytes)'
+            f' files: {backup_result.copied_small_files}'
+            f' (total {human_filesize(backup_result.copied_small_size)})'
+        )
+        if backup_result.error_count > 0:
+            print(f'  Errors during backup: {backup_result.error_count} (see log for details)')
+        print()
+
+    logger.info('Backup completed. Summary created: %s', summary_file)
 
     return backup_result

{pyhardlinkbackup-1.3.0 → pyhardlinkbackup-1.4.0}/PyHardLinkBackup/cli_app/phlb.py

@@ -3,13 +3,22 @@ from pathlib import Path
 from typing import Annotated
 
 import tyro
-from cli_base.cli_tools.verbosity import setup_logging
-from cli_base.tyro_commands import TyroVerbosityArgType
 from rich import print  # noqa
 
 from PyHardLinkBackup import rebuild_databases
 from PyHardLinkBackup.backup import backup_tree
 from PyHardLinkBackup.cli_app import app
+from PyHardLinkBackup.logging_setup import (
+    DEFAULT_CONSOLE_LOG_LEVEL,
+    DEFAULT_LOG_FILE_LEVEL,
+    LoggingManager,
+    TyroConsoleLogLevelArgType,
+    TyroLogFileLevelArgType,
+)
+from PyHardLinkBackup.utilities.tyro_cli_shared_args import (
+    DEFAULT_EXCLUDE_DIRECTORIES,
+    TyroExcludeDirectoriesArgType,
+)
 
 
 logger = logging.getLogger(__name__)
@@ -32,22 +41,22 @@ def backup(
         ),
     ],
     /,
-    excludes: Annotated[
-        tuple,
-        tyro.conf.arg(
-            help='List of directory or file names to exclude from backup.',
-        ),
-    ] = ('__pycache__', '.cache', '.temp', '.tmp', '.tox', '.nox'),
-    verbosity: TyroVerbosityArgType = 2,
+    excludes: TyroExcludeDirectoriesArgType = DEFAULT_EXCLUDE_DIRECTORIES,
+    verbosity: TyroConsoleLogLevelArgType = DEFAULT_CONSOLE_LOG_LEVEL,
+    log_file_level: TyroLogFileLevelArgType = DEFAULT_LOG_FILE_LEVEL,
 ) -> None:
     """
     Backup the source directory to the destination directory using hard links for deduplication.
     """
-    setup_logging(verbosity=verbosity)
+    log_manager = LoggingManager(
+        console_level=verbosity,
+        file_level=log_file_level,
+    )
     backup_tree(
         src_root=src,
         backup_root=dst,
-        excludes=set(excludes),
+        excludes=excludes,
+        log_manager=log_manager,
     )
 
 
@@ -61,11 +70,18 @@ def rebuild(
         ),
     ],
     /,
-    verbosity: TyroVerbosityArgType = 2,
+    verbosity: TyroConsoleLogLevelArgType = DEFAULT_CONSOLE_LOG_LEVEL,
+    log_file_level: TyroLogFileLevelArgType = DEFAULT_LOG_FILE_LEVEL,
 ) -> None:
     """
     Rebuild the file hash and size database by scanning all backup files. And also verify SHA256SUMS
     and/or store missing hashes in SHA256SUMS files.
     """
-    setup_logging(verbosity=verbosity)
-    rebuild_databases.rebuild(backup_root=backup_root)
+    log_manager = LoggingManager(
+        console_level=verbosity,
+        file_level=log_file_level,
+    )
+    rebuild_databases.rebuild(
+        backup_root=backup_root,
+        log_manager=log_manager,
+    )

{pyhardlinkbackup-1.3.0 → pyhardlinkbackup-1.4.0}/PyHardLinkBackup/cli_dev/benchmark.py

@@ -10,16 +10,34 @@ from cli_base.tyro_commands import TyroVerbosityArgType
 from rich import print  # noqa
 
 from PyHardLinkBackup.cli_dev import app
-from PyHardLinkBackup.utilities.filesystem import iter_scandir_files
+from PyHardLinkBackup.utilities.filesystem import humanized_fs_scan, iter_scandir_files
+from PyHardLinkBackup.utilities.humanize import PrintTimingContextManager
+from PyHardLinkBackup.utilities.tyro_cli_shared_args import DEFAULT_EXCLUDE_DIRECTORIES, TyroExcludeDirectoriesArgType
 
 
 logger = logging.getLogger(__name__)
 
 
+@app.command
+def scan_benchmark(
+    base_path: Path,
+    /,
+    excludes: TyroExcludeDirectoriesArgType = DEFAULT_EXCLUDE_DIRECTORIES,
+    verbosity: TyroVerbosityArgType = 1,
+) -> None:
+    """
+    Benchmark our filesystem scan routine.
+    """
+    setup_logging(verbosity=verbosity)
+    with PrintTimingContextManager('Filesystem scan completed in'):
+        humanized_fs_scan(path=base_path, excludes=set(excludes))
+
+
 @app.command
 def benchmark_hashes(
     base_path: Path,
     /,
+    excludes: TyroExcludeDirectoriesArgType = DEFAULT_EXCLUDE_DIRECTORIES,
     max_duration: int = 30,  # in seconds
     min_file_size: int = 15 * 1024,  # 15 KiB
     max_file_size: int = 100 * 1024 * 1024,  # 100 MiB
@@ -70,40 +88,41 @@ def benchmark_hashes(
     stop_time = start_time + max_duration
     next_update = start_time + 2
 
-    for dir_entry in iter_scandir_files(base_path):
-        entry_stat = dir_entry.stat()
-        file_size = entry_stat.st_size
-        if not (min_file_size <= file_size <= max_file_size):
-            continue
+    with PrintTimingContextManager('Filesystem scan completed in'):
+        for dir_entry in iter_scandir_files(path=base_path, excludes=set(excludes)):
+            entry_stat = dir_entry.stat()
+            file_size = entry_stat.st_size
+            if not (min_file_size <= file_size <= max_file_size):
+                continue
 
-        start_time = time.perf_counter()
-        file_content = Path(dir_entry.path).read_bytes()
-        duration = time.perf_counter() - start_time
-        total_read_time += duration
-
-        for algo in algorithms:
-            # Actual measurement:
             start_time = time.perf_counter()
-            hashlib.new(algo, file_content)
+            file_content = Path(dir_entry.path).read_bytes()
             duration = time.perf_counter() - start_time
-
-            results[algo].add(duration)
-
-        file_count += 1
-        total_size += entry_stat.st_size
-
-        now = time.time()
-        if now >= stop_time:
-            print('Reached max duration limit, stopping benchmark...')
-            break
-
-        if now >= next_update:
-            percent = (now - (stop_time - max_duration)) / max_duration * 100
-            print(
-                f'{int(percent)}% Processed {file_count} files so far,'
-                f' total size: {total_size / 1024 / 1024:.1f} MiB...'
-            )
-            next_update = now + 2
+            total_read_time += duration
+
+            for algo in algorithms:
+                # Actual measurement:
+                start_time = time.perf_counter()
+                hashlib.new(algo, file_content)
+                duration = time.perf_counter() - start_time
+
+                results[algo].add(duration)
+
+            file_count += 1
+            total_size += entry_stat.st_size
+
+            now = time.time()
+            if now >= stop_time:
+                print('Reached max duration limit, stopping benchmark...')
+                break
+
+            if now >= next_update:
+                percent = (now - (stop_time - max_duration)) / max_duration * 100
+                print(
+                    f'{int(percent)}% Processed {file_count} files so far,'
+                    f' total size: {total_size / 1024 / 1024:.1f} MiB...'
+                )
+                next_update = now + 2
 
     print(f'\nTotal files hashed: {file_count}, total size: {total_size / 1024 / 1024:.1f} MiB')
 

pyhardlinkbackup-1.4.0/PyHardLinkBackup/logging_setup.py

@@ -0,0 +1,124 @@
+import logging
+import sys
+from pathlib import Path
+from typing import Annotated, Literal
+
+import tyro
+from bx_py_utils.path import assert_is_dir
+from rich import (
+    get_console,
+    print,  # noqa
+)
+from rich.logging import RichHandler
+
+
+logger = logging.getLogger(__name__)
+
+LogLevelLiteral = Literal['debug', 'info', 'warning', 'error']
+
+
+TyroConsoleLogLevelArgType = Annotated[
+    LogLevelLiteral,
+    tyro.conf.arg(
+        help='Log level for console logging.',
+    ),
+]
+DEFAULT_CONSOLE_LOG_LEVEL: TyroConsoleLogLevelArgType = 'warning'
+
+
+TyroLogFileLevelArgType = Annotated[
+    LogLevelLiteral,
+    tyro.conf.arg(
+        help='Log level for the log file',
+    ),
+]
+DEFAULT_LOG_FILE_LEVEL: TyroLogFileLevelArgType = 'info'
+
+
+def log_level_name2int(level_name: str) -> int:
+    level_name = level_name.upper()
+    level_mapping = logging.getLevelNamesMapping()
+    try:
+        return level_mapping[level_name]
+    except KeyError as err:
+        raise ValueError(f'Invalid log level name: {level_name}') from err
+
+
+console = get_console()
+
+
+class LoggingManager:
+    def __init__(
+        self,
+        *,
+        console_level: TyroConsoleLogLevelArgType,
+        file_level: TyroLogFileLevelArgType,
+    ):
+        self.console_level_name = console_level
+        self.console_level: int = log_level_name2int(console_level)
+        self.file_level_name = file_level
+        self.file_level: int = log_level_name2int(file_level)
+
+        self.lowest_level = min(self.console_level, self.file_level)
+
+        if console_level == logging.DEBUG:
+            log_format = '(%(name)s) %(message)s'
+        else:
+            log_format = '%(message)s'
+
+        console.print(
+            f'(Set [bold]console[bold] log level: [cyan]{self.console_level_name}[/cyan])',
+            justify='right',
+        )
+        handler = RichHandler(console=console, omit_repeated_times=False)
+        handler.setLevel(self.console_level)
+        logging.basicConfig(
+            level=self.lowest_level,
+            format=log_format,
+            datefmt='[%x %X.%f]',
+            handlers=[handler],
+            force=True,
+        )
+        sys.excepthook = self.log_unhandled_exception
+
+    def start_file_logging(self, log_file: Path):
+        console.print(
+            f'(initialize log file [bold]{log_file}[/bold] with level: [cyan]{self.file_level_name}[/cyan])',
+            justify='right',
+        )
+
+        assert_is_dir(log_file.parent)
+
+        root_logger = logging.getLogger()
+
+        file_handler = logging.FileHandler(log_file, mode='a', encoding='utf-8')
+        file_handler.setLevel(self.file_level)
+
+        formatter = logging.Formatter(
+            '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
+            datefmt='%Y-%m-%d %H:%M:%S',
+        )
+        file_handler.setFormatter(formatter)
+
+        root_logger.addHandler(file_handler)
+
+    def log_unhandled_exception(self, exc_type, exc_value, exc_traceback):
+        if issubclass(exc_type, KeyboardInterrupt):
+            logger.info('Program interrupted by user (KeyboardInterrupt). Exiting...')
+            sys.__excepthook__(exc_type, exc_value, exc_traceback)
+        else:
+            logger.exception(
+                'Unhandled exception occurred:',
+                exc_info=(exc_type, exc_value, exc_traceback),
+            )
+
+
+class NoopLoggingManager(LoggingManager):
+    """
+    Only for tests: A logging manager that does nothing.
+    """
+    def __init__(self, *args, **kwargs):
+        pass
+
+    def start_file_logging(self, log_file: Path):
+        pass