PyHardLinkBackup 1.5.0__tar.gz → 1.7.0__tar.gz

{pyhardlinkbackup-1.5.0 → pyhardlinkbackup-1.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: PyHardLinkBackup
-Version: 1.5.0
+Version: 1.7.0
 Summary: HardLink/Deduplication Backups with Python
 Project-URL: Documentation, https://github.com/jedie/PyHardLinkBackup
 Project-URL: Source, https://github.com/jedie/PyHardLinkBackup
@@ -21,11 +21,20 @@ Description-Content-Type: text/markdown
 [![Python Versions](https://img.shields.io/pypi/pyversions/PyHardLinkBackup)](https://github.com/jedie/PyHardLinkBackup/blob/main/pyproject.toml)
 [![License GPL-3.0-or-later](https://img.shields.io/pypi/l/PyHardLinkBackup)](https://github.com/jedie/PyHardLinkBackup/blob/main/LICENSE)
 
-HardLink/Deduplication Backups with Python
+PyHardLinkBackup is a cross-platform backup tool designed for efficient, reliable, and accessible backups.
+Similar to `rsync --link-dest`, but with global deduplication across all backups and all paths, not just between two directories.
 
-**WIP:** v1.0.0 is a complete rewrite of PyHardLinkBackup.
+Some aspects:
 
-It's similar to `rsync --link-dest` but the deduplication is done globally for all backups and all paths.
+- Creates deduplicated, versioned backups using hardlinks, minimizing storage usage by linking identical files across all backup snapshots.
+- Employs a global deduplication database (by file size and SHA256 hash) per backup root, ensuring that duplicate files are detected and hardlinked even if they are moved or renamed between backups.
+- Backups are stored as regular files and directories—no proprietary formats—so you can access your data directly without special tools.
+- Deleting old snapshots does not affect the integrity of remaining backups.
+- Linux and macOS are fully supported (Windows support is experimental)
+
+Limitations:
+
+- Requires a filesystem that supports hardlinks (e.g., btrfs, zfs, ext4, APFS, NTFS with limitations).
 
 ## installation
 
@@ -38,7 +47,16 @@ pipx install PyHardLinkBackup
 ```
 
 After this you can call the CLI via `phlb` command.
-The main command is `phlb backup <source> <destination>`:
+The main command is `phlb backup <source> <destination>` to create a backup.
+
+e.g.:
+
+```bash
+phlb backup /path/to/source /path/to/destination
+```
+
+This will create a snapshot in `/path/to/destination` using hard links for deduplication. You can safely delete old snapshots without affecting others.
+
 
 [comment]: <> (✂✂✂ auto generated backup help start ✂✂✂)
 ```
@@ -63,20 +81,59 @@ Backup the source directory to the destination directory using hard links for de
 [comment]: <> (✂✂✂ auto generated backup help end ✂✂✂)
 
 
+## Screenshots
+### Screenshot - running a backup
+
+----
+
+![2026-01-19_phlb1.png](https://raw.githubusercontent.com/jedie/jedie.github.io/main/screenshots/PyHardLinkBackup/2026-01-19_phlb1.png "2026-01-19_phlb1.png")
+
+----
 
-Running a backup looks like:
+### Screenshot - backup finished
 
-![2026-01-15-phlb1.png](https://raw.githubusercontent.com/jedie/jedie.github.io/main/screenshots/PyHardLinkBackup/2026-01-15-phlb1.png "2026-01-15-phlb1.png")
+----
 
+![2026-01-19_phlb2.png](https://raw.githubusercontent.com/jedie/jedie.github.io/main/screenshots/PyHardLinkBackup/2026-01-19_phlb2.png "2026-01-19_phlb2.png")
 
+----
+
+(more screenshots here: [jedie.github.io/tree/main/screenshots/PyHardLinkBackup](https://github.com/jedie/jedie.github.io/tree/main/screenshots/PyHardLinkBackup))
+
+
+### update
+
+If you use pipx, just call:
+```bash
+pipx upgrade PyHardLinkBackup
+```
+see: https://pipx.pypa.io/stable/docs/#pipx-upgrade
 
-If it's finished it display a summary:
 
-![2026-01-15-phlb2.png](https://raw.githubusercontent.com/jedie/jedie.github.io/main/screenshots/PyHardLinkBackup/2026-01-15-phlb2.png "2026-01-15-phlb2.png")
+### Troubleshooting
 
+- **Permission Errors:** Ensure you have read access to source and write access to destination.
+- **Hardlink Limits:** Some filesystems (e.g., NTFS) have limits on the number of hardlinks per file.
+- **Symlink Handling:** Broken symlinks are handled gracefully; see logs for details.
+- **Backup Deletion:** Deleting a snapshot does not affect deduplication of other backups.
+- **Log Files:** Check the log file in each backup directory for error details.
 
 
-complete help for main CLI app:
+To lower the priority of the backup process (useful to reduce system impact during heavy backups), you can use `nice` and `ionice` on Linux systems:
+
+```bash
+nice -n 19 ionice -c3 phlb backup /path/to/source /path/to/destination
+```
+- `nice -n 19` sets the lowest CPU priority.
+- `ionice -c3` sets the lowest I/O priority (idle class).
+
+Adjust priority of an already running backup:
+```bash
+renice 19 -p $(pgrep phlb) && ionice -c3 -p $(pgrep phlb)
+```
+
+
+### complete help for main CLI app
 
 [comment]: <> (✂✂✂ auto generated main help start ✂✂✂)
 ```
@@ -99,13 +156,7 @@ usage: phlb [-h] {backup,compare,rebuild,version}
 [comment]: <> (✂✂✂ auto generated main help end ✂✂✂)
 
 
-### update
 
-If you use pipx, just call:
-```bash
-pipx upgrade PyHardLinkBackup
-```
-see: https://pipx.pypa.io/stable/docs/#pipx-upgrade
 
 
 ## concept
@@ -232,10 +283,26 @@ Overview of main changes:
 
 [comment]: <> (✂✂✂ auto generated history start ✂✂✂)
 
+* [v1.7.0](https://github.com/jedie/PyHardLinkBackup/compare/v1.6.0...v1.7.0)
+  * 2026-01-19 - Speedup and enhance unittest
+  * 2026-01-17 - Remove unfinished copied files on errors
+  * 2026-01-17 - Display/update progress on very lage files #75 and enhance all bars
+  * 2026-01-18 - Expand tests: Check file open calls
+  * 2026-01-17 - expand tests
+  * 2026-01-17 - simplify tests
+  * 2026-01-17 - Warn if broken symlink found
+  * 2026-01-17 - Update README
+* [v1.6.0](https://github.com/jedie/PyHardLinkBackup/compare/v1.5.0...v1.6.0)
+  * 2026-01-17 - Fix flaky test, because of terminal size
+  * 2026-01-17 - Bugfix: Don't hash new large files twice
+  * 2026-01-17 - Use compare also in backup tests
 * [v1.5.0](https://github.com/jedie/PyHardLinkBackup/compare/v1.4.1...v1.5.0)
   * 2026-01-17 - NEW: Compare command to verify source tree with last backup
 * [v1.4.1](https://github.com/jedie/PyHardLinkBackup/compare/v1.4.0...v1.4.1)
   * 2026-01-16 - Bugfix large file handling
+
+<details><summary>Expand older history entries ...</summary>
+
 * [v1.4.0](https://github.com/jedie/PyHardLinkBackup/compare/v1.3.0...v1.4.0)
   * 2026-01-16 - Create log file in backup and a summary.txt
   * 2026-01-16 - Run CI tests on macos, too.
@@ -246,9 +313,6 @@ Overview of main changes:
   * 2026-01-15 - Add tests for rebuild
   * 2026-01-15 - Add command to "rebuld" the size and hash filesystem database
   * 2026-01-15 - Add screenshots in the README
-
-<details><summary>Expand older history entries ...</summary>
-
 * [v1.2.0](https://github.com/jedie/PyHardLinkBackup/compare/v1.1.0...v1.2.0)
   * 2026-01-15 - Add error handling: Log exception but continue with the backup
   * 2026-01-15 - Check permission and hadlink support on destination path

{pyhardlinkbackup-1.5.0 → pyhardlinkbackup-1.7.0}/PyHardLinkBackup/__init__.py

@@ -3,5 +3,5 @@
 """
 
 # See https://packaging.python.org/en/latest/specifications/version-specifiers/
-__version__ = '1.5.0'
+__version__ = '1.7.0'
 __author__ = 'Jens Diemer <PyHardLinkBackup@jensdiemer.de>'

{pyhardlinkbackup-1.5.0 → pyhardlinkbackup-1.7.0}/PyHardLinkBackup/backup.py

@@ -14,6 +14,7 @@ from PyHardLinkBackup.logging_setup import LoggingManager
 from PyHardLinkBackup.utilities.file_hash_database import FileHashDatabase
 from PyHardLinkBackup.utilities.file_size_database import FileSizeDatabase
 from PyHardLinkBackup.utilities.filesystem import (
+    RemoveFileOnError,
     copy_and_hash,
     hash_file,
     humanized_fs_scan,
@@ -59,6 +60,7 @@ def backup_one_file(
     hash_db: FileHashDatabase,
     backup_dir: Path,
     backup_result: BackupResult,
+    progress: DisplayFileTreeProgress,
 ) -> None:
     backup_result.backup_count += 1
     src_path = Path(entry.path)
@@ -70,8 +72,8 @@ def backup_one_file(
 
     try:
         size = entry.stat().st_size
-    except FileNotFoundError:
-        # e.g.: Handle broken symlink
+    except FileNotFoundError as err:
+        logger.warning(f'Broken symlink {src_path}: {err.__class__.__name__}: {err}')
         target = os.readlink(src_path)
         dst_path.symlink_to(target)
         backup_result.symlink_files += 1
@@ -95,65 +97,66 @@ def backup_one_file(
     # Process regular files
     assert entry.is_file(follow_symlinks=False), f'Unexpected non-file: {src_path}'
 
-    # Deduplication logic
+    with RemoveFileOnError(dst_path):
+        # Deduplication logic
+
+        if size < size_db.MIN_SIZE:
+            # Small file -> always copy without deduplication
+            logger.info('Copy small file: %s to %s', src_path, dst_path)
+            file_hash = copy_and_hash(src_path, dst_path, progress=progress, total_size=size)
+            backup_result.copied_files += 1
+            backup_result.copied_size += size
+            backup_result.copied_small_files += 1
+            backup_result.copied_small_size += size
+            store_hash(dst_path, file_hash)
+            return
+
+        if size in size_db:
+            logger.debug('File with size %iBytes found before -> hash: %s', size, src_path)
+
+            if size <= CHUNK_SIZE:
+                # File can be read complete into memory
+                logger.debug('File size %iBytes <= CHUNK_SIZE (%iBytes) -> read complete into memory', size, CHUNK_SIZE)
+                file_content, file_hash = read_and_hash_file(src_path)
+                if existing_path := hash_db.get(file_hash):
+                    logger.info('Hardlink duplicate file: %s to %s', dst_path, existing_path)
+                    os.link(existing_path, dst_path)
+                    backup_result.hardlinked_files += 1
+                    backup_result.hardlinked_size += size
+                else:
+                    logger.info('Store unique file: %s to %s', src_path, dst_path)
+                    dst_path.write_bytes(file_content)
+                    hash_db[file_hash] = dst_path
+                    backup_result.copied_files += 1
+                    backup_result.copied_size += size
 
-    if size < size_db.MIN_SIZE:
-        # Small file -> always copy without deduplication
-        logger.info('Copy small file: %s to %s', src_path, dst_path)
-        file_hash = copy_and_hash(src_path, dst_path)
-        backup_result.copied_files += 1
-        backup_result.copied_size += size
-        backup_result.copied_small_files += 1
-        backup_result.copied_small_size += size
-        store_hash(dst_path, file_hash)
-        return
-
-    if size in size_db:
-        logger.debug('File with size %iBytes found before -> hash: %s', size, src_path)
-
-        if size <= CHUNK_SIZE:
-            # File can be read complete into memory
-            logger.debug('File size %iBytes <= CHUNK_SIZE (%iBytes) -> read complete into memory', size, CHUNK_SIZE)
-            file_content, file_hash = read_and_hash_file(src_path)
-            if existing_path := hash_db.get(file_hash):
-                logger.info('Hardlink duplicate file: %s to %s', dst_path, existing_path)
-                os.link(existing_path, dst_path)
-                backup_result.hardlinked_files += 1
-                backup_result.hardlinked_size += size
             else:
-                logger.info('Store unique file: %s to %s', src_path, dst_path)
-                dst_path.write_bytes(file_content)
-                hash_db[file_hash] = dst_path
-                backup_result.copied_files += 1
-                backup_result.copied_size += size
-
+                # Large file
+                file_hash = hash_file(src_path, progress=progress, total_size=size)  # Calculate hash without copying
+
+                if existing_path := hash_db.get(file_hash):
+                    logger.info('Hardlink duplicate file: %s to %s', dst_path, existing_path)
+                    os.link(existing_path, dst_path)
+                    backup_result.hardlinked_files += 1
+                    backup_result.hardlinked_size += size
+                else:
+                    logger.info('Copy unique file: %s to %s', src_path, dst_path)
+                    shutil.copyfile(src_path, dst_path)
+                    hash_db[file_hash] = dst_path
+                    backup_result.copied_files += 1
+                    backup_result.copied_size += size
+
+            # Keep original file metadata (permission bits, time stamps, and flags)
+            shutil.copystat(src_path, dst_path)
         else:
-            # Large file
-            file_hash = hash_file(src_path)  # Calculate hash without copying
-
-            if existing_path := hash_db.get(file_hash):
-                logger.info('Hardlink duplicate file: %s to %s', dst_path, existing_path)
-                os.link(existing_path, dst_path)
-                backup_result.hardlinked_files += 1
-                backup_result.hardlinked_size += size
-            else:
-                logger.info('Copy unique file: %s to %s', src_path, dst_path)
-                file_hash = copy_and_hash(src_path, dst_path)
-                hash_db[file_hash] = dst_path
-                backup_result.copied_files += 1
-                backup_result.copied_size += size
-
-        # Keep original file metadata (permission bits, time stamps, and flags)
-        shutil.copystat(src_path, dst_path)
-    else:
-        # A file with this size not backuped before -> Can't be duplicate -> copy and hash
-        file_hash = copy_and_hash(src_path, dst_path)
-        size_db.add(size)
-        hash_db[file_hash] = dst_path
-        backup_result.copied_files += 1
-        backup_result.copied_size += size
+            # A file with this size not backuped before -> Can't be duplicate -> copy and hash
+            file_hash = copy_and_hash(src_path, dst_path, progress=progress, total_size=size)
+            size_db.add(size)
+            hash_db[file_hash] = dst_path
+            backup_result.copied_files += 1
+            backup_result.copied_size += size
 
-    store_hash(dst_path, file_hash)
+        store_hash(dst_path, file_hash)
 
 
 def backup_tree(
@@ -205,7 +208,11 @@ def backup_tree(
 
     print(f'\nBackup to {backup_dir}...\n')
 
-    with DisplayFileTreeProgress(src_file_count, src_total_size) as progress:
+    with DisplayFileTreeProgress(
+        description=f'Backup {src_root}...',
+        total_file_count=src_file_count,
+        total_size=src_total_size,
+    ) as progress:
         # "Databases" for deduplication
         size_db = FileSizeDatabase(phlb_conf_dir)
         hash_db = FileHashDatabase(backup_root, phlb_conf_dir)
@@ -222,6 +229,7 @@ def backup_tree(
                     hash_db=hash_db,
                     backup_dir=backup_dir,
                     backup_result=backup_result,
+                    progress=progress,
                 )
             except Exception as err:
                 logger.exception(f'Backup {entry.path} {err.__class__.__name__}: {err}')

{pyhardlinkbackup-1.5.0 → pyhardlinkbackup-1.7.0}/PyHardLinkBackup/compare_backup.py

@@ -26,6 +26,7 @@ logger = logging.getLogger(__name__)
 
 @dataclasses.dataclass
 class CompareResult:
+    last_timestamp: str
     compare_dir: Path
     log_file: Path
     #
@@ -52,6 +53,7 @@ def compare_one_file(
     hash_db: FileHashDatabase,
     compare_dir: Path,
     compare_result: CompareResult,
+    progress: DisplayFileTreeProgress,
 ) -> None:
     src_size = entry.stat().st_size
 
@@ -79,8 +81,8 @@ def compare_one_file(
         compare_result.file_size_missmatch += 1
         return
 
-    src_hash = hash_file(src_path)
-    dst_hash = hash_file(dst_path)
+    src_hash = hash_file(src_path, progress=progress, total_size=src_size)
+    dst_hash = hash_file(dst_path, progress=progress, total_size=dst_size)
 
     if src_hash != dst_hash:
         logger.warning(
@@ -157,12 +159,16 @@ def compare_tree(
     with PrintTimingContextManager('Filesystem scan completed in'):
         src_file_count, src_total_size = humanized_fs_scan(src_root, excludes=excludes)
 
-    with DisplayFileTreeProgress(src_file_count, src_total_size) as progress:
+    with DisplayFileTreeProgress(
+        description=f'Compare {src_root}...',
+        total_file_count=src_file_count,
+        total_size=src_total_size,
+    ) as progress:
         # init "databases":
         size_db = FileSizeDatabase(phlb_conf_dir)
         hash_db = FileHashDatabase(backup_root, phlb_conf_dir)
 
-        compare_result = CompareResult(compare_dir=compare_dir, log_file=log_file)
+        compare_result = CompareResult(last_timestamp=last_timestamp, compare_dir=compare_dir, log_file=log_file)
 
         next_update = 0
         for entry in iter_scandir_files(src_root, excludes=excludes):
@@ -174,6 +180,7 @@ def compare_tree(
                     hash_db=hash_db,
                     compare_dir=compare_dir,
                     compare_result=compare_result,
+                    progress=progress,
                 )
             except Exception as err:
                 logger.exception(f'Compare {entry.path} {err.__class__.__name__}: {err}')
@@ -183,12 +190,12 @@ def compare_tree(
                 if now >= next_update:
                     progress.update(
                         completed_file_count=compare_result.total_file_count,
-                        completed_size=compare_result.total_size,
+                        advance_size=compare_result.total_size,
                     )
                     next_update = now + 0.5
 
         # Finalize progress indicator values:
-        progress.update(completed_file_count=compare_result.total_file_count, completed_size=compare_result.total_size)
+        progress.update(completed_file_count=compare_result.total_file_count, advance_size=compare_result.total_size)
 
     summary_file = compare_main_dir / f'{now_timestamp}-summary.txt'
     with TeeStdoutContext(summary_file):

{pyhardlinkbackup-1.5.0 → pyhardlinkbackup-1.7.0}/PyHardLinkBackup/constants.py

@@ -11,6 +11,8 @@ BASE_PATH = Path(PyHardLinkBackup.__file__).parent
 ##########################################################################
 # "Settings" for PyHardLinkBackup:
 
-CHUNK_SIZE = 64 * 1024 * 1024  # 64 MB
-SMALL_FILE_THRESHOLD = 1000  # bytes
 HASH_ALGO = 'sha256'
+SMALL_FILE_THRESHOLD = 1000  # bytes
+CHUNK_SIZE = 64 * 1024 * 1024  # 64 MB
+LAGE_FILE_PROGRESS_MIN_SIZE = CHUNK_SIZE * 3
+

{pyhardlinkbackup-1.5.0 → pyhardlinkbackup-1.7.0}/PyHardLinkBackup/rebuild_databases.py

@@ -41,6 +41,7 @@ def rebuild_one_file(
     size_db: FileSizeDatabase,
     hash_db: FileHashDatabase,
     rebuild_result: RebuildResult,
+    progress: DisplayFileTreeProgress,
 ):
     file_path = Path(entry.path)
 
@@ -62,7 +63,7 @@ def rebuild_one_file(
         # Small files will never deduplicate, skip them
         return
 
-    file_hash = hash_file(file_path)
+    file_hash = hash_file(file_path, progress=progress, total_size=size)
 
     if size not in size_db:
         size_db.add(size)
@@ -121,7 +122,11 @@ def rebuild(
                 file_count -= 1
                 total_size -= file.stat().st_size
 
-    with DisplayFileTreeProgress(file_count, total_size) as progress:
+    with DisplayFileTreeProgress(
+        description=f'Rebuild {backup_root}...',
+        total_file_count=file_count,
+        total_size=total_size,
+    ) as progress:
         # "Databases" for deduplication
         size_db = FileSizeDatabase(phlb_conf_dir)
         hash_db = FileHashDatabase(backup_root, phlb_conf_dir)
@@ -137,6 +142,7 @@ def rebuild(
                     size_db=size_db,
                     hash_db=hash_db,
                     rebuild_result=rebuild_result,
+                    progress=progress,
                 )
             except Exception as err:
                 logger.exception(f'Backup {entry.path} {err.__class__.__name__}: {err}')
@@ -145,12 +151,12 @@ def rebuild(
                 now = time.monotonic()
                 if now >= next_update:
                     progress.update(
-                        completed_file_count=rebuild_result.process_count, completed_size=rebuild_result.process_size
+                        completed_file_count=rebuild_result.process_count, advance_size=rebuild_result.process_size
                     )
                     next_update = now + 0.5
 
         # Finalize progress indicator values:
-        progress.update(completed_file_count=rebuild_result.process_count, completed_size=rebuild_result.process_size)
+        progress.update(completed_file_count=rebuild_result.process_count, advance_size=rebuild_result.process_size)
 
     summary_file = backup_root / f'{timestamp}-rebuild-summary.txt'
     with TeeStdoutContext(summary_file):