backup-helper 0.2__tar.gz

backup-helper-0.2/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.1
+Name: backup-helper
+Version: 0.2
+Summary: Helper tool for creating plain-file cold-storage archives including checksum files
+Author-email: omgitsmoe <60219950+omgitsmoe@users.noreply.github.com>
+Project-URL: Homepage, https://github.com/omgitsmoe/backup_helper
+Project-URL: Bug Tracker, https://github.com/omgitsmoe/backup_helper/issues
+Keywords: script,verify,backup,archival,bit-rot
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: checksum_helper<0.3,>=0.2.2
+Provides-Extra: test
+Requires-Dist: pytest<8,>=7.2; extra == "test"
+
+# BackupHelper
+
+A tool for simplifying the process of archiving multiple directories
+onto several different drives. For each directory a checksum file 
+will be created, which will be verified after the transfer.
+
+You can stage multiple sources and add targets to them.
+Once you're done you can start the transfer, which will run
+all copy operations at the same time, while making sure that
+all disks in a transfer aren't busy with another BackupHelper operation.
+
+## Quick start
+
+Add a directory as a source for copying/archiving:
+```
+python -m backup_helper stage ~/Documents --alias docs
+Staged: /home/m/Documents
+    with alias: docs
+```
+
+By default the BackupHelper state will be saved in the file
+`backup_status.json` in the current working directory.
+Alternatively a custom path can be used by passing
+`--status-file /path/to/status.json` to __each__ command.
+
+Add targets to that source. Either the normalized absolute path
+can be used as `source` or the alias (here: _"docs"_) if present:
+
+```
+$ python -m backup_helper add-target docs /media/storage1/docs_2024 --alias storage1
+Added target /media/storage1/docs_2024
+    with alias: storage1
+$ python -m backup_helper add-target docs /media/storage1/docs_2024 --alias storage2
+Added target /media/storage1/docs_2024
+    with alias: storage2
+```
+
+Now you can use the command `start` run the whole backup process
+in sequence.
+
+```
+python -m backup_helper start
+18:22:01 - INFO - Wrote /home/m/Documents/Documents_bh_2024-02-25T18-22-01.cshd
+...
+18:22:02 - INFO - 
+
+NO MISSING FILES!
+
+NO FAILED CHECKSUMS!
+
+SUMMARY:
+    TOTAL FILES: 3
+    MATCHES: 3
+    FAILED CHECKSUMS: 0
+    MISSING: 0
+
+...
+
+18:22:02 - INFO - /home/m/Documents/Documents_bh_2024-02-25T18-22-01.cshd: No missing files and all files matching their hashes
+
+...
+
+18:22:02 - INFO - Successfully completed the following 5 operation(s):
+Hashed '/home/m/Documents':
+  Hash file: /home/m/Documents/Documents_bh_2024-02-25T18-22-01.cshd
+Transfer successful:
+  From: /home/m/Documents
+  To: /media/storage1/docs_2024
+Transfer successful:
+  From: /home/m/Documents
+  To: /media/storage2/docs_2024
+Verified transfer '/media/storage1/docs_2024':
+  Checked: 3
+  CRC Errors: 0
+  Missing: 0
+Verified transfer '/media/storage2/docs_2024':
+  Checked: 3
+  CRC Errors: 0
+  Missing: 0
+```
+
+Each part of the backup process can be run on its own and on a
+specific source/target combination only. For more information
+see the [backup process section](#backup-process).
+
+## Backup process
+
+The backup process, which can be run automatically using the
+`start` command is split into the subprocesses:
+
+1) Hash all source directories. The checksum file will be added to
+   the directory. A log file of creating the checksum file will
+   be written next to status JSON file.
+2) Transfer all sources to their targets. Only one read __or__ write
+   operation per disk will be allowed at the same time.
+3) Verify the transfer by comparing the hashes of the generated
+   checksum file with the hashes of the files on the target.
+   A log of the verification process will be written to the target.
+
+The verification process (3) will be run last if there are more
+transfer operations on a disk, so:
+
+1) More expensive write operations are performed first.
+2) The transferred files are less likely to be in cache when hashing.
+
+Each part of the backup process can be run on its own and/or on a
+specific source/target combination only. Required previous steps
+will be run automatically.
+
+Using the `interactive` command it's possible to add sources/targets
+while the transfer is running, otherwise all running operations would
+need to be completed before executing further commands.
+
+## Commands
+
+See `python -m backup_helper --help`

backup-helper-0.2/README.md

@@ -0,0 +1,116 @@
+# BackupHelper
+
+A tool for simplifying the process of archiving multiple directories
+onto several different drives. For each directory a checksum file 
+will be created, which will be verified after the transfer.
+
+You can stage multiple sources and add targets to them.
+Once you're done you can start the transfer, which will run
+all copy operations at the same time, while making sure that
+all disks in a transfer aren't busy with another BackupHelper operation.
+
+## Quick start
+
+Add a directory as a source for copying/archiving:
+```
+python -m backup_helper stage ~/Documents --alias docs
+Staged: /home/m/Documents
+    with alias: docs
+```
+
+By default the BackupHelper state will be saved in the file
+`backup_status.json` in the current working directory.
+Alternatively a custom path can be used by passing
+`--status-file /path/to/status.json` to __each__ command.
+
+Add targets to that source. Either the normalized absolute path
+can be used as `source` or the alias (here: _"docs"_) if present:
+
+```
+$ python -m backup_helper add-target docs /media/storage1/docs_2024 --alias storage1
+Added target /media/storage1/docs_2024
+    with alias: storage1
+$ python -m backup_helper add-target docs /media/storage1/docs_2024 --alias storage2
+Added target /media/storage1/docs_2024
+    with alias: storage2
+```
+
+Now you can use the command `start` run the whole backup process
+in sequence.
+
+```
+python -m backup_helper start
+18:22:01 - INFO - Wrote /home/m/Documents/Documents_bh_2024-02-25T18-22-01.cshd
+...
+18:22:02 - INFO - 
+
+NO MISSING FILES!
+
+NO FAILED CHECKSUMS!
+
+SUMMARY:
+    TOTAL FILES: 3
+    MATCHES: 3
+    FAILED CHECKSUMS: 0
+    MISSING: 0
+
+...
+
+18:22:02 - INFO - /home/m/Documents/Documents_bh_2024-02-25T18-22-01.cshd: No missing files and all files matching their hashes
+
+...
+
+18:22:02 - INFO - Successfully completed the following 5 operation(s):
+Hashed '/home/m/Documents':
+  Hash file: /home/m/Documents/Documents_bh_2024-02-25T18-22-01.cshd
+Transfer successful:
+  From: /home/m/Documents
+  To: /media/storage1/docs_2024
+Transfer successful:
+  From: /home/m/Documents
+  To: /media/storage2/docs_2024
+Verified transfer '/media/storage1/docs_2024':
+  Checked: 3
+  CRC Errors: 0
+  Missing: 0
+Verified transfer '/media/storage2/docs_2024':
+  Checked: 3
+  CRC Errors: 0
+  Missing: 0
+```
+
+Each part of the backup process can be run on its own and on a
+specific source/target combination only. For more information
+see the [backup process section](#backup-process).
+
+## Backup process
+
+The backup process, which can be run automatically using the
+`start` command is split into the subprocesses:
+
+1) Hash all source directories. The checksum file will be added to
+   the directory. A log file of creating the checksum file will
+   be written next to status JSON file.
+2) Transfer all sources to their targets. Only one read __or__ write
+   operation per disk will be allowed at the same time.
+3) Verify the transfer by comparing the hashes of the generated
+   checksum file with the hashes of the files on the target.
+   A log of the verification process will be written to the target.
+
+The verification process (3) will be run last if there are more
+transfer operations on a disk, so:
+
+1) More expensive write operations are performed first.
+2) The transferred files are less likely to be in cache when hashing.
+
+Each part of the backup process can be run on its own and/or on a
+specific source/target combination only. Required previous steps
+will be run automatically.
+
+Using the `interactive` command it's possible to add sources/targets
+while the transfer is running, otherwise all running operations would
+need to be completed before executing further commands.
+
+## Commands
+
+See `python -m backup_helper --help`

backup-helper-0.2/backup_helper/__main__.py

@@ -0,0 +1,11 @@
+import sys
+
+from backup_helper import cli
+
+
+def main():
+    cli.main(sys.argv[1:])
+
+
+if __name__ == '__main__':
+    main()

backup-helper-0.2/backup_helper/backup_helper.py

@@ -0,0 +1,193 @@
+import sys
+import os
+import dataclasses
+import json
+import contextlib
+import logging
+import time
+
+from typing import (
+    List, Optional, Dict, Any, Union, cast, Callable, Set,
+    Iterator, Iterable, TYPE_CHECKING, Tuple, overload
+)
+
+from backup_helper import helpers
+from backup_helper.exceptions import (
+    SourceNotFound, TargetNotFound,
+    SourceAlreadyExists, AliasAlreadyExists,
+)
+from backup_helper.source import Source
+from backup_helper.target import Target
+from backup_helper import work
+
+logger = logging.getLogger(__name__)
+
+
+class BackupHelper:
+    def __init__(self, sources: List[Source]):
+        self._sources = {}
+        # don't serialize this, will be set when loading, so the file can be moved!
+        self._working_dir = '.'
+        for source in sources:
+            self._sources[source.path] = source
+            if source.alias:
+                self._sources[source.alias] = source
+        self._queue = work.setup_work_queue([])
+
+    @ classmethod
+    def load_state(cls, path: str) -> 'BackupHelper':
+        if os.path.exists(path):
+            with open(path, "r", encoding='utf-8') as f:
+                contents = f.read()
+            bh = cls.from_json(contents)
+            bh._working_dir = os.path.dirname(path)
+            return bh
+        else:
+            return cls([])
+
+    def to_json(self) -> Dict[Any, Any]:
+        result = {"version": 1, "type": type(self).__name__}
+
+        sources: List[Dict[str, Any]] = []
+        # sources contain both path as well as alias as keys, so we have to
+        # deduplicate them
+        for source in self.unique_sources():
+            sources.append(source.to_json())
+
+        result["sources"] = sources
+
+        return result
+
+    @ staticmethod
+    def from_json(json_str: str) -> 'BackupHelper':
+        d = json.loads(json_str, object_hook=BackupHelper.from_json_hook)
+        return d
+
+    @ staticmethod
+    def from_json_hook(json_object: Dict[Any, Any]) -> Union[
+            'BackupHelper', Source, Target, Dict[Any, Any]]:
+        # if this is used as object_hook in json_loads it
+        # will call the method iteratively as it builds the object bottom up
+
+        if "type" not in json_object:
+            return json_object
+
+        # version = json_object["version"]
+        obj_type = json_object["type"]
+
+        # dispatch to appropriate from_json method
+        if obj_type == "BackupHelper":
+            sources = json_object["sources"]
+            return BackupHelper(sources)
+        elif obj_type == "Source":
+            return Source.from_json(json_object)
+        elif obj_type == "Target":
+            return Target.from_json(json_object)
+        else:
+            return json_object
+
+    def unique_sources(self) -> Iterator[Source]:
+        # sources contain both path as well as alias as keys, so we have to
+        # deduplicate them
+        yield from helpers.unique_iterator(self._sources.values())
+
+    def save_state(self, path: str):
+        d = self.to_json()
+        with open(path, "w", encoding='utf-8') as f:
+            f.write(json.dumps(d))
+
+    def add_source(self, source: Source):
+        if source.path in self._sources:
+            raise SourceAlreadyExists(
+                f"Source '{source.path}' already exists!", source.path)
+
+        self._sources[source.path] = source
+        if source.alias:
+            if source.alias in self._sources:
+                raise AliasAlreadyExists(
+                    f"Alias '{source.alias}' already exists!", source.alias)
+            self._sources[source.alias] = source
+
+    def get_source(self, source_key: str) -> Source:
+        try:
+            return self._sources[source_key]
+        except KeyError:
+            raise SourceNotFound(
+                f"Source '{source_key}' not found!", source_key)
+
+    def hash_all(self) -> None:
+        for s in self.unique_sources():
+            s.hash_queue(self._queue, log_dir=self._working_dir)
+        success, errors = self._queue.start_and_join_all()
+        work.report_results(success, errors)
+
+    def transfer_all(self) -> None:
+        for src in self.unique_sources():
+            src.transfer_queue_all(self._queue)
+
+        success, errors = self._queue.start_and_join_all()
+        work.report_results(success, errors)
+
+    def verify_all(self) -> None:
+        for src in self.unique_sources():
+            src.verify_target_queue_all(self._queue)
+
+        success, errors = self._queue.start_and_join_all()
+        work.report_results(success, errors)
+
+    def start_all(self) -> None:
+        for src in self.unique_sources():
+            src.hash_queue(self._queue, log_dir=self._working_dir)
+            src.transfer_queue_all(self._queue)
+            src.verify_target_queue_all(self._queue)
+
+        success, errors = self._queue.start_and_join_all()
+        work.report_results(success, errors)
+
+    def workers_running(self) -> bool:
+        return self._queue.workers_running()
+
+    def join(self) -> None:
+        self._queue.join()
+
+    def status(self, source_key: str) -> str:
+        try:
+            src = self.get_source(source_key)
+        except SourceNotFound as e:
+            return f"Source '{e.source}' not found!"
+        else:
+            return src.status()
+
+    def status_all(self) -> str:
+        builder = []
+        for source in self.unique_sources():
+            builder.append(f"--- Source: {source.path} ---")
+            builder.append(source.status())
+
+        return "\n".join(builder)
+
+
+@ contextlib.contextmanager
+def load_backup_state(
+        path: str, instance: Optional[BackupHelper] = None) -> Iterator[BackupHelper]:
+    """Contextmangaer that saves state on Exception"""
+    if instance is None:
+        bh = BackupHelper.load_state(path)
+    else:
+        bh = instance
+
+    try:
+        yield bh
+    except Exception:
+        fn, ext = os.path.splitext(path)
+        bh.save_state(helpers.unique_filename(f"{fn}_crash{ext}"))
+        raise
+
+
+@ contextlib.contextmanager
+def load_backup_state_save_always(
+        path: str, instance: Optional[BackupHelper] = None) -> Iterator[BackupHelper]:
+    """Contextmangaer that saves state on exit"""
+    with load_backup_state(path, instance) as bh:
+        yield bh
+        bh.save_state(path)