declarative-data-backup 0.1.0__tar.gz → 0.2.0__tar.gz

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: declarative-data-backup
-Version: 0.1.0
+Version: 0.2.0
 Summary: Declarative Borg-based data backups
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
@@ -120,6 +120,18 @@ postgres_backup:
     - /server_root/path/to/postgres-backups
   required_mounts:
     - /server_root/path/to/postgres-backups
+
+apprise_backup:
+  files:
+    - /server_root/data/apprise/apprise.yml
+
+jellyfin_backup:
+  folders:
+    - /server_root/data/jellyfin
+  exclude:
+    - "**/cache/**"
+    - "**/log/**"
+    - "**/transcodes/**"
 ```
 
 Set the required environment variables before starting the process:
@@ -159,14 +171,22 @@ the run fails and pruning is skipped.
 The top level must be a nonempty YAML mapping of backup group names to group
 mappings. Group names may contain ASCII letters, digits, `_`, and `-`, and
 must be at most 64 characters. Duplicate YAML keys, unknown group fields,
-invalid values, and malformed or empty configuration are rejected before any
-create or prune operation.
+invalid types, unsupported pattern selectors, and empty configuration are
+rejected before any create or prune operation.
 
 Each group supports:
 
-- `folders` (required): a nonempty list of absolute directory paths. Each
-  source must exist, be readable, and resolve inside `SERVER_ROOT`. Duplicate
-  canonical paths and parent/child overlaps across groups are rejected.
+- `folders` (optional): list of absolute directory paths.
+- `files` (optional): list of absolute regular-file paths. Folders and files
+  share the same containment, readability, duplicate, and overlap checks;
+  configure at least one source per group.
+- `exclude` (optional): list of Borg 1.4 glob patterns, scoped to this group
+  and passed directly to Borg for folder sources. To avoid approximating
+  Borg's pattern matching, a group with explicit `files` cannot also configure
+  `exclude`; this is rejected before any Borg operation.
+  Patterns are otherwise passed unchanged to Borg, which evaluates their
+  matching semantics. If Borg rejects a pattern during create, the run stops
+  before pruning any source.
 - `compression` (optional): defaults to `auto,zstd,22`. Supported Borg 1.4
   codecs include `none`, `lz4`, `zstd`, `zlib`, and `lzma`; `auto` may precede
   a codec. Codec levels are checked against supported ranges: zstd `1`-`22`,
@@ -187,6 +207,11 @@ does not prove that nested host filesystems are mounted. In particular,
 `allow_empty: true` does not prove that a disk is present: an empty source may
 be legitimate and can be archived.
 
+Backing up a live SQLite database with `files` or `folders` does not produce a
+consistent database snapshot. Use an application-supported export, backup API,
+or coordinated filesystem snapshot for live database data. Exclusions do not
+make a database backup consistent.
+
 Paths are resolved before backup. A top-level source symlink is allowed only
 when its canonical target remains inside `SERVER_ROOT`; symlinks within a
 source are stored as links rather than followed.

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/README.md

@@ -107,6 +107,18 @@ postgres_backup:
     - /server_root/path/to/postgres-backups
   required_mounts:
     - /server_root/path/to/postgres-backups
+
+apprise_backup:
+  files:
+    - /server_root/data/apprise/apprise.yml
+
+jellyfin_backup:
+  folders:
+    - /server_root/data/jellyfin
+  exclude:
+    - "**/cache/**"
+    - "**/log/**"
+    - "**/transcodes/**"
 ```
 
 Set the required environment variables before starting the process:
@@ -146,14 +158,22 @@ the run fails and pruning is skipped.
 The top level must be a nonempty YAML mapping of backup group names to group
 mappings. Group names may contain ASCII letters, digits, `_`, and `-`, and
 must be at most 64 characters. Duplicate YAML keys, unknown group fields,
-invalid values, and malformed or empty configuration are rejected before any
-create or prune operation.
+invalid types, unsupported pattern selectors, and empty configuration are
+rejected before any create or prune operation.
 
 Each group supports:
 
-- `folders` (required): a nonempty list of absolute directory paths. Each
-  source must exist, be readable, and resolve inside `SERVER_ROOT`. Duplicate
-  canonical paths and parent/child overlaps across groups are rejected.
+- `folders` (optional): list of absolute directory paths.
+- `files` (optional): list of absolute regular-file paths. Folders and files
+  share the same containment, readability, duplicate, and overlap checks;
+  configure at least one source per group.
+- `exclude` (optional): list of Borg 1.4 glob patterns, scoped to this group
+  and passed directly to Borg for folder sources. To avoid approximating
+  Borg's pattern matching, a group with explicit `files` cannot also configure
+  `exclude`; this is rejected before any Borg operation.
+  Patterns are otherwise passed unchanged to Borg, which evaluates their
+  matching semantics. If Borg rejects a pattern during create, the run stops
+  before pruning any source.
 - `compression` (optional): defaults to `auto,zstd,22`. Supported Borg 1.4
   codecs include `none`, `lz4`, `zstd`, `zlib`, and `lzma`; `auto` may precede
   a codec. Codec levels are checked against supported ranges: zstd `1`-`22`,
@@ -174,6 +194,11 @@ does not prove that nested host filesystems are mounted. In particular,
 `allow_empty: true` does not prove that a disk is present: an empty source may
 be legitimate and can be archived.
 
+Backing up a live SQLite database with `files` or `folders` does not produce a
+consistent database snapshot. Use an application-supported export, backup API,
+or coordinated filesystem snapshot for live database data. Exclusions do not
+make a database backup consistent.
+
 Paths are resolved before backup. A top-level source symlink is allowed only
 when its canonical target remains inside `SERVER_ROOT`; symlinks within a
 source are stored as links rather than followed.

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "declarative-data-backup"
-version = "0.1.0"
+version = "0.2.0"
 description = "Declarative Borg-based data backups"
 readme = "README.md"
 requires-python = ">=3.11"

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/src/declarative_data_backup/backup.py

@@ -44,11 +44,11 @@ def run_backup(
     settings: Environment,
     definitions: list[BackupDefinition],
 ) -> None:
-    total_folders = sum(len(definition.folders) for definition in definitions)
+    total_sources = sum(len(definition.sources) for definition in definitions)
     logger.info(
-        "Validated %d backup groups containing %d folders",
+        "Validated %d backup groups containing %d sources",
         len(definitions),
-        total_folders,
+        total_sources,
     )
 
     run_id = uuid.uuid4().hex
@@ -56,11 +56,11 @@ def run_backup(
         created_at = datetime.now(UTC)
         for definition in definitions:
             logger.info(
-                "Starting backup group '%s' (%d folders).",
+                "Starting backup group '%s' (%d sources).",
                 definition.name,
-                len(definition.folders),
+                len(definition.sources),
             )
-            for folder in definition.folders:
+            for folder in definition.sources:
                 backup_folder(
                     folder,
                     definition.compression,
@@ -68,10 +68,11 @@ def run_backup(
                     settings,
                     created_at,
                     run_id,
+                    definition.excludes,
                 )
 
         for definition in definitions:
-            for folder in definition.folders:
+            for folder in definition.sources:
                 prune(
                     folder,
                     definition.retention_days,

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/src/declarative_data_backup/borg/borg.py

@@ -29,6 +29,7 @@ def backup_folder(
     settings: Environment,
     created_at: datetime,
     run_id: str,
+    excludes: tuple[str, ...] = (),
 ) -> None:
     name = archive_name_escaper(
         folder,
@@ -38,17 +39,19 @@ def backup_folder(
     context = f"group={group!r} source={folder}"
     started_at = time.monotonic()
     logger.info("Starting backup for %s as archive %s.", context, name)
+    args = [
+        "create",
+        "--verbose",
+        "--stats",
+        "--compression",
+        compression,
+    ]
+    for pattern in excludes:
+        args.extend(("--exclude", pattern))
+    args.extend((f"{settings.borg_repo}::{name}", str(folder)))
     try:
         run_borg(
-            [
-                "create",
-                "--verbose",
-                "--stats",
-                "--compression",
-                compression,
-                f"{settings.borg_repo}::{name}",
-                str(folder),
-            ],
+            args,
             settings,
             "create",
             context,

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/src/declarative_data_backup/config/backup_config/models.py

@@ -10,3 +10,9 @@ class BackupDefinition:
     retention_days: int
     required_mounts: tuple[Path, ...] = ()
     allow_empty: bool = True
+    files: tuple[Path, ...] = ()
+    excludes: tuple[str, ...] = ()
+
+    @property
+    def sources(self) -> tuple[Path, ...]:
+        return self.folders + self.files

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/src/declarative_data_backup/config/backup_config/utils.py

@@ -37,8 +37,8 @@ def parse_folders(
     group: str,
     allow_empty: bool,
 ) -> tuple[Path, ...]:
-    if not isinstance(raw_folders, list) or not raw_folders:
-        fail(f"'{group}.folders' must be a nonempty YAML list.")
+    if not isinstance(raw_folders, list):
+        fail(f"'{group}.folders' must be a YAML list.")
 
     folders = []
     for raw_folder in raw_folders:
@@ -60,6 +60,47 @@ def parse_folders(
     return tuple(folders)
 
 
+def parse_files(
+    raw_files: Any,
+    server_root: Path,
+    group: str,
+) -> tuple[Path, ...]:
+    if not isinstance(raw_files, list):
+        fail(f"'{group}.files' must be a YAML list.")
+
+    files = []
+    for raw_file in raw_files:
+        file = resolve_path(raw_file, server_root, f"'{group}.files' entry")
+        if not file.is_file():
+            fail(f"Backup source is not a regular file: {file}.")
+        if not os.access(file, os.R_OK):
+            fail(f"Backup source is not readable: {file}.")
+        files.append(file)
+    return tuple(files)
+
+
+def parse_excludes(raw_excludes: Any, group: str) -> tuple[str, ...]:
+    if not isinstance(raw_excludes, list):
+        fail(f"'{group}.exclude' must be a YAML list.")
+
+    excludes = []
+    for pattern in raw_excludes:
+        if not isinstance(pattern, str) or not pattern.strip() or "\0" in pattern:
+            fail(f"'{group}.exclude' entries must be nonempty Borg glob patterns.")
+
+        pattern_type, separator, pattern_body = pattern.partition(":")
+        if separator and len(pattern_type) == 2:
+            if pattern_type not in {"fm", "sh", "pf", "pp"}:
+                fail(
+                    f"'{group}.exclude' entries must use Borg glob patterns "
+                    "(fm:, sh:, pf:, or pp:)."
+                )
+            if not pattern_body:
+                fail(f"'{group}.exclude' entries must be nonempty Borg patterns.")
+        excludes.append(pattern)
+    return tuple(excludes)
+
+
 def parse_required_mounts(
     raw_mounts: Any,
     server_root: Path,
@@ -80,23 +121,23 @@ def parse_required_mounts(
     return mounts
 
 
-def check_folders_for_overlappings(definitions: list[BackupDefinition]) -> None:
+def check_sources_for_overlappings(definitions: list[BackupDefinition]) -> None:
     owners: dict[Path, str] = {}
     for definition in definitions:
-        for folder in definition.folders:
+        for source in definition.sources:
             for existing, owner in owners.items():
-                if folder == existing:
+                if source == existing:
                     fail(
-                        f"Duplicate canonical backup source {folder} "
+                        f"Duplicate canonical backup source {source} "
                         f"(already configured by '{owner}')."
                     )
-                if folder in existing.parents or existing in folder.parents:
+                if source in existing.parents or existing in source.parents:
                     fail(
                         f"Overlapping backup sources are not allowed: "
                         f"'{owner}' uses {existing}, while "
-                        f"'{definition.name}' uses {folder}."
+                        f"'{definition.name}' uses {source}."
                     )
-            owners[folder] = definition.name
+            owners[source] = definition.name
 
 
 def _mountinfo_path(raw_mount: str) -> Path:

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/src/declarative_data_backup/config/backup_config/yaml_parser.py

@@ -6,8 +6,10 @@ from ...environment import DEFAULT_COMPRESSION, DEFAULT_RETENTION_DAYS
 from ...errors import fail
 from .models import BackupDefinition
 from .utils import (
-    check_folders_for_overlappings,
     check_required_mounts_are_mounted,
+    check_sources_for_overlappings,
+    parse_excludes,
+    parse_files,
     parse_folders,
     parse_required_mounts,
 )
@@ -15,12 +17,17 @@ from .utils import (
 _GROUP_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9_-]{0,63}\Z")
 _GROUP_KEYS = {
     "folders",
+    "files",
+    "exclude",
     "compression",
     "retention_days",
     "required_mounts",
     "allow_empty",
 }
 _GROUP_DEFAULTS = {
+    "folders": [],
+    "files": [],
+    "exclude": [],
     "compression": DEFAULT_COMPRESSION,
     "retention_days": DEFAULT_RETENTION_DAYS,
     "required_mounts": [],
@@ -102,9 +109,20 @@ def _parse_definition(name: Any, raw: Any, server_root: Path) -> BackupDefinitio
     if not isinstance(allow_empty, bool):
         fail(f"'{name}.allow_empty' must be a boolean.")
 
+    folders = parse_folders(group["folders"], server_root, name, allow_empty)
+    files = parse_files(group["files"], server_root, name)
+    excludes = parse_excludes(group["exclude"], name)
+    if not folders and not files:
+        fail(f"'{name}' must configure at least one folder or file source.")
+    if files and excludes:
+        fail(
+            f"'{name}' cannot combine 'files' and 'exclude': Borg exclusion "
+            "of an explicitly selected file cannot be verified safely."
+        )
+
     return BackupDefinition(
         name=name,
-        folders=parse_folders(group.get("folders"), server_root, name, allow_empty),
+        folders=folders,
         compression=compression,
         retention_days=retention_days,
         required_mounts=parse_required_mounts(
@@ -113,6 +131,8 @@ def _parse_definition(name: Any, raw: Any, server_root: Path) -> BackupDefinitio
             name,
         ),
         allow_empty=allow_empty,
+        files=files,
+        excludes=excludes,
     )
 
 
@@ -131,6 +151,6 @@ def parse_backup_definitions(
     definitions = [
         _parse_definition(name, raw, server_root) for name, raw in data.items()
     ]
-    check_folders_for_overlappings(definitions)
+    check_sources_for_overlappings(definitions)
     check_required_mounts_are_mounted(definitions, mountinfo_path)
     return definitions

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/src/declarative_data_backup.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: declarative-data-backup
-Version: 0.1.0
+Version: 0.2.0
 Summary: Declarative Borg-based data backups
 Requires-Python: >=3.11
 Description-Content-Type: text/markdown
@@ -120,6 +120,18 @@ postgres_backup:
     - /server_root/path/to/postgres-backups
   required_mounts:
     - /server_root/path/to/postgres-backups
+
+apprise_backup:
+  files:
+    - /server_root/data/apprise/apprise.yml
+
+jellyfin_backup:
+  folders:
+    - /server_root/data/jellyfin
+  exclude:
+    - "**/cache/**"
+    - "**/log/**"
+    - "**/transcodes/**"
 ```
 
 Set the required environment variables before starting the process:
@@ -159,14 +171,22 @@ the run fails and pruning is skipped.
 The top level must be a nonempty YAML mapping of backup group names to group
 mappings. Group names may contain ASCII letters, digits, `_`, and `-`, and
 must be at most 64 characters. Duplicate YAML keys, unknown group fields,
-invalid values, and malformed or empty configuration are rejected before any
-create or prune operation.
+invalid types, unsupported pattern selectors, and empty configuration are
+rejected before any create or prune operation.
 
 Each group supports:
 
-- `folders` (required): a nonempty list of absolute directory paths. Each
-  source must exist, be readable, and resolve inside `SERVER_ROOT`. Duplicate
-  canonical paths and parent/child overlaps across groups are rejected.
+- `folders` (optional): list of absolute directory paths.
+- `files` (optional): list of absolute regular-file paths. Folders and files
+  share the same containment, readability, duplicate, and overlap checks;
+  configure at least one source per group.
+- `exclude` (optional): list of Borg 1.4 glob patterns, scoped to this group
+  and passed directly to Borg for folder sources. To avoid approximating
+  Borg's pattern matching, a group with explicit `files` cannot also configure
+  `exclude`; this is rejected before any Borg operation.
+  Patterns are otherwise passed unchanged to Borg, which evaluates their
+  matching semantics. If Borg rejects a pattern during create, the run stops
+  before pruning any source.
 - `compression` (optional): defaults to `auto,zstd,22`. Supported Borg 1.4
   codecs include `none`, `lz4`, `zstd`, `zlib`, and `lzma`; `auto` may precede
   a codec. Codec levels are checked against supported ranges: zstd `1`-`22`,
@@ -187,6 +207,11 @@ does not prove that nested host filesystems are mounted. In particular,
 `allow_empty: true` does not prove that a disk is present: an empty source may
 be legitimate and can be archived.
 
+Backing up a live SQLite database with `files` or `folders` does not produce a
+consistent database snapshot. Use an application-supported export, backup API,
+or coordinated filesystem snapshot for live database data. Exclusions do not
+make a database backup consistent.
+
 Paths are resolved before backup. A top-level source symlink is allowed only
 when its canonical target remains inside `SERVER_ROOT`; symlinks within a
 source are stored as links rather than followed.

{declarative_data_backup-0.1.0 → declarative_data_backup-0.2.0}/tests/test_backup.py

@@ -48,8 +48,8 @@ def test_main_runs_all_creates_before_any_prune(
     monkeypatch.setattr(
         backup,
         "backup_folder",
-        lambda folder, compression, group, settings, created_at, run_id: events.append(
-            ("create", group, folder, compression, run_id)
+        lambda folder, compression, group, settings, created_at, run_id, excludes: (
+            events.append(("create", group, folder, compression, run_id, excludes))
         ),
     )
     monkeypatch.setattr(
@@ -153,7 +153,7 @@ def test_retry_after_partial_create_uses_a_new_run_id_before_pruning(
     prunes = []
     fail_second = True
 
-    def create(folder, compression, group, settings, created_at, run_id):
+    def create(folder, compression, group, settings, created_at, run_id, excludes):
         nonlocal fail_second
         creates.append((folder, run_id))
         if fail_second and folder == second:
@@ -268,6 +268,41 @@ def test_main_prune_failure_is_nonzero_and_scoped(
     ]
 
 
+def test_exclusions_are_passed_only_to_their_backup_group(
+    settings_factory,
+    monkeypatch,
+):
+    settings = settings_factory()
+    first = settings.server_root / "first"
+    first.mkdir()
+    second = settings.server_root / "second.yml"
+    second.write_text("config\n", encoding="utf-8")
+    _write_config(
+        settings,
+        "first:\n"
+        f"  folders: [{first}]\n"
+        "  exclude: ['**/cache/**']\n"
+        "second:\n"
+        f"  files: [{second}]\n",
+    )
+    _configure_preflight(monkeypatch)
+    calls = []
+    monkeypatch.setattr(
+        backup,
+        "backup_folder",
+        lambda source, compression, group, settings, created_at, run_id, excludes: (
+            calls.append((group, source, excludes))
+        ),
+    )
+    monkeypatch.setattr(backup, "prune", lambda *args: None)
+
+    assert backup.main(_settings_environment(settings)) == 0
+    assert calls == [
+        ("first", first, ("**/cache/**",)),
+        ("second", second, ()),
+    ]
+
+
 def test_main_reports_unexpected_io_errors_cleanly(
     settings_factory,
     monkeypatch,