dar-backup 1.0.0.1__py3-none-any.whl → 1.0.2__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dar_backup/Changelog.md +64 -1
- dar_backup/README.md +355 -34
- dar_backup/__about__.py +3 -2
- dar_backup/clean_log.py +102 -63
- dar_backup/cleanup.py +225 -76
- dar_backup/command_runner.py +198 -103
- dar_backup/config_settings.py +158 -1
- dar_backup/dar-backup.conf +18 -0
- dar_backup/dar-backup.conf.j2 +44 -0
- dar_backup/dar_backup.py +806 -131
- dar_backup/demo.py +18 -9
- dar_backup/installer.py +18 -1
- dar_backup/manager.py +304 -91
- dar_backup/util.py +502 -141
- {dar_backup-1.0.0.1.dist-info → dar_backup-1.0.2.dist-info}/METADATA +358 -37
- dar_backup-1.0.2.dist-info/RECORD +25 -0
- {dar_backup-1.0.0.1.dist-info → dar_backup-1.0.2.dist-info}/WHEEL +1 -1
- dar_backup-1.0.0.1.dist-info/RECORD +0 -25
- {dar_backup-1.0.0.1.dist-info → dar_backup-1.0.2.dist-info}/entry_points.txt +0 -0
- {dar_backup-1.0.0.1.dist-info → dar_backup-1.0.2.dist-info}/licenses/LICENSE +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: dar-backup
|
|
3
|
-
Version: 1.0.
|
|
3
|
+
Version: 1.0.2
|
|
4
4
|
Summary: A script to do full, differential and incremental backups using dar. Some files are restored from the backups during verification, after which par2 redundancy files are created. The script also has a cleanup feature to remove old backups and par2 files.
|
|
5
5
|
Project-URL: GPG Public Key, https://keys.openpgp.org/search?q=dar-backup@pm.me
|
|
6
6
|
Project-URL: Homepage, https://github.com/per2jensen/dar-backup/tree/main/v2
|
|
@@ -686,9 +686,9 @@ Classifier: Development Status :: 5 - Production/Stable
|
|
|
686
686
|
Classifier: Intended Audience :: End Users/Desktop
|
|
687
687
|
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
|
|
688
688
|
Classifier: Operating System :: POSIX :: Linux
|
|
689
|
-
Classifier: Programming Language :: Python :: 3.
|
|
689
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
690
690
|
Classifier: Topic :: System :: Archiving :: Backup
|
|
691
|
-
Requires-Python: >=3.
|
|
691
|
+
Requires-Python: >=3.11
|
|
692
692
|
Requires-Dist: argcomplete>=3.6.2
|
|
693
693
|
Requires-Dist: inputimeout>=1.0.4
|
|
694
694
|
Requires-Dist: jinja2>=3.1.6
|
|
@@ -722,17 +722,19 @@ Description-Content-Type: text/markdown
|
|
|
722
722
|
**Reliable DAR backups, automated in clean Python**
|
|
723
723
|
|
|
724
724
|
[](https://codecov.io/gh/per2jensen/dar-backup)
|
|
725
|
-
[](https://snyk.io/
|
|
725
|
+
[](https://security.snyk.io/vuln/?search=dar-backup)
|
|
726
726
|

|
|
727
727
|
[](https://pypi.org/project/dar-backup/)
|
|
728
728
|
[](https://pypi.org/project/dar-backup/)
|
|
729
|
-
[](https://github.com/per2jensen/dar-backup/blob/main/clonepulse/weekly_clones.png)
|
|
730
|
+
[](https://github.com/per2jensen/dar-backup/blob/main/clonepulse/weekly_clones.png) <sub>🎯 Stats powered by [ClonePulse](https://github.com/per2jensen/clonepulse)</sub>
|
|
731
731
|
|
|
732
732
|
The wonderful 'dar' [Disk Archiver](https://github.com/Edrusb/DAR) is used for
|
|
733
733
|
the heavy lifting, together with the [parchive](https://github.com/Parchive/par2cmdline) suite in these scripts.
|
|
734
734
|
|
|
735
|
-
This is the `Python` based [**version 2**](
|
|
735
|
+
This is the `Python` based [**version 2**](v2) of `dar-backup`.
|
|
736
|
+
|
|
737
|
+
You can see the [v2 Changelog](v2/Changelog.md) for details on features and progress.
|
|
736
738
|
|
|
737
739
|
## TL;DR
|
|
738
740
|
|
|
@@ -746,6 +748,9 @@ Version **1.0.0** was reached on October 9, 2025.
|
|
|
746
748
|
- [TL;DR](#tldr)
|
|
747
749
|
- [Table of Contents](#table-of-contents)
|
|
748
750
|
- [My use case](#my-use-case)
|
|
751
|
+
- [My setup](#my-setup)
|
|
752
|
+
- [Why PAR2 is especially good for portable / offsite copies](#why-par2-is-especially-good-for-portable--offsite-copies)
|
|
753
|
+
- [Design choices](#design-choices)
|
|
749
754
|
- [Features](#features)
|
|
750
755
|
- [License](#license)
|
|
751
756
|
- [Quick Guide](#quick-guide)
|
|
@@ -789,6 +794,8 @@ Version **1.0.0** was reached on October 9, 2025.
|
|
|
789
794
|
- [restore test fails with exit code 5](#restore-test-fails-with-exit-code-5)
|
|
790
795
|
- [Par2](#par2)
|
|
791
796
|
- [Par2 to verify/repair](#par2-to-verifyrepair)
|
|
797
|
+
- [Par2 files kept with archives](#par2-files-kept-with-archives)
|
|
798
|
+
- [Par2 files in separate directory](#par2-files-in-separate-directory)
|
|
792
799
|
- [Par2 create redundancy files](#par2-create-redundancy-files)
|
|
793
800
|
- [Points of interest](#points-of-interest)
|
|
794
801
|
- [Limitations on File Names with Special Characters](#limitations-on-file-names-with-special-characters)
|
|
@@ -806,8 +813,8 @@ Version **1.0.0** was reached on October 9, 2025.
|
|
|
806
813
|
- [Performance tip due to par2](#performance-tip-due-to-par2)
|
|
807
814
|
- [.darrc sets -vd -vf (since v0.6.4)](#darrc-sets--vd--vf-since-v064)
|
|
808
815
|
- [Separate log file for command output](#separate-log-file-for-command-output)
|
|
816
|
+
- [Trace Logging (Debug details)](#trace-logging-debug-details)
|
|
809
817
|
- [Skipping cache directories](#skipping-cache-directories)
|
|
810
|
-
- [Progress bar and current directory](#progress-bar-and-current-directory)
|
|
811
818
|
- [Shell autocompletion](#shell-autocompletion)
|
|
812
819
|
- [Use it](#use-it)
|
|
813
820
|
- [Archive name completion (smart, context-aware)](#archive-name-completion-smart-context-aware)
|
|
@@ -821,12 +828,22 @@ Version **1.0.0** was reached on October 9, 2025.
|
|
|
821
828
|
- [CLI Tools Overview](#cli-tools-overview)
|
|
822
829
|
- [test coverage](#test-coverage)
|
|
823
830
|
- [Dar-backup options](#dar-backup-options)
|
|
831
|
+
- [Dar-backup exit codes](#dar-backup-exit-codes)
|
|
832
|
+
- [Dar-backup env vars](#dar-backup-env-vars)
|
|
824
833
|
- [Manager Options](#manager-options)
|
|
825
|
-
|
|
834
|
+
- [Cleanup env vars](#cleanup-env-vars)
|
|
826
835
|
- [Clean-log options](#clean-log-options)
|
|
827
836
|
- [Dar-backup-systemd options](#dar-backup-systemd-options)
|
|
828
837
|
- [Installer options](#installer-options)
|
|
829
838
|
- [Demo options](#demo-options)
|
|
839
|
+
- [Config changes](#config-changes)
|
|
840
|
+
- [1.0.1](#101)
|
|
841
|
+
- [DISCORD WEBHOOK](#discord-webhook)
|
|
842
|
+
- [Restore test config](#restore-test-config)
|
|
843
|
+
- [Par2](#par2-1)
|
|
844
|
+
- [1.0.2](#102)
|
|
845
|
+
- [Trace Logging](#trace-logging)
|
|
846
|
+
- [Command output Capture](#command-output-capture)
|
|
830
847
|
|
|
831
848
|
## My use case
|
|
832
849
|
|
|
@@ -836,7 +853,7 @@ I needed the following:
|
|
|
836
853
|
- Backup primarily photos, home made video and different types of documents
|
|
837
854
|
- I have cloud storage mounted on a directory within my home dir. The filesystem is [FUSE based](https://www.kernel.org/doc/html/latest/filesystems/fuse.html), which gives it a few special features
|
|
838
855
|
|
|
839
|
-
- Backup cloud storage (cloud is convenient, but I want control over my backups)
|
|
856
|
+
- Backup my cloud storage (cloud is convenient, but I want control over my backups)
|
|
840
857
|
- A non-privileged user can perform a mount
|
|
841
858
|
- A privileged user cannot look into the filesystem --> a backup script running as root is not suitable
|
|
842
859
|
|
|
@@ -847,6 +864,48 @@ I needed the following:
|
|
|
847
864
|
|
|
848
865
|
I do not need the encryption features of dar, as all storage is already encrypted.
|
|
849
866
|
|
|
867
|
+
## My setup
|
|
868
|
+
|
|
869
|
+
1. Primary backup to server with an ext4 file system on mdadm RAID1
|
|
870
|
+
|
|
871
|
+
2. Secondary copies to multiple USB disks / cloud
|
|
872
|
+
|
|
873
|
+
3. Archive integrity verification anywhere using [Par2](#par2) and `dar -t`.
|
|
874
|
+
|
|
875
|
+
4. Archive repair anywhere if needed. By default `dar-backup` creates par2 redundancy files with 5% coverage. Enough to fix localized bitrot.
|
|
876
|
+
|
|
877
|
+
5. No dependency on original system
|
|
878
|
+
|
|
879
|
+
### Why PAR2 is especially good for portable / offsite copies
|
|
880
|
+
|
|
881
|
+
PAR2 parity is:
|
|
882
|
+
|
|
883
|
+
> Self-contained (travels with the data)
|
|
884
|
+
>
|
|
885
|
+
>Format-agnostic (works on any filesystem)
|
|
886
|
+
>
|
|
887
|
+
>Location-agnostic (local disk, USB, cloud object storage)
|
|
888
|
+
>
|
|
889
|
+
>Tool-stable (PAR2 spec has not changed in years)
|
|
890
|
+
>
|
|
891
|
+
>That means:
|
|
892
|
+
>
|
|
893
|
+
>**Integrity protection moves with the archive**.
|
|
894
|
+
|
|
895
|
+
### Design choices
|
|
896
|
+
|
|
897
|
+
My design choices are boring, proven and pragmatic:
|
|
898
|
+
>
|
|
899
|
+
>mdadm handles disks
|
|
900
|
+
>
|
|
901
|
+
>PAR2 handles data integrity
|
|
902
|
+
>
|
|
903
|
+
>You control when and how verification happens
|
|
904
|
+
>
|
|
905
|
+
>Errors have a fair chance of being diagnosed and fixed, due to well known tooling.
|
|
906
|
+
>
|
|
907
|
+
>No hidden magic, no lock-in
|
|
908
|
+
|
|
850
909
|
## Features
|
|
851
910
|
|
|
852
911
|
- The battle tested [dar](https://github.com/Edrusb/DAR) Disk Archiver is used for the actual backups - it comes highly recommended.
|
|
@@ -869,7 +928,7 @@ I needed the following:
|
|
|
869
928
|
## License
|
|
870
929
|
|
|
871
930
|
These scripts are licensed under the GPLv3 license.
|
|
872
|
-
Read more here: [GNU GPL3.0](https://www.gnu.org/licenses/gpl-3.0.en.html), or have a look at the ["LICENSE"](
|
|
931
|
+
Read more here: [GNU GPL3.0](https://www.gnu.org/licenses/gpl-3.0.en.html), or have a look at the ["LICENSE"](LICENSE) file in this repository.
|
|
873
932
|
|
|
874
933
|
## Quick Guide
|
|
875
934
|
|
|
@@ -1214,7 +1273,7 @@ This python version is v2 of dar-backup, v1 is made in bash.
|
|
|
1214
1273
|
|
|
1215
1274
|
## Community
|
|
1216
1275
|
|
|
1217
|
-
Please review the [Code of Conduct](
|
|
1276
|
+
Please review the [Code of Conduct](CODE_OF_CONDUCT.md) to help keep this project welcoming and focused.
|
|
1218
1277
|
|
|
1219
1278
|
## Requirements
|
|
1220
1279
|
|
|
@@ -1234,7 +1293,7 @@ On Ubuntu, install the requirements this way:
|
|
|
1234
1293
|
|
|
1235
1294
|
### dar-backup overview
|
|
1236
1295
|
|
|
1237
|
-
](v2/doc/dar-backup-overview.png)
|
|
1238
1297
|
|
|
1239
1298
|
### dar-backup
|
|
1240
1299
|
|
|
@@ -1263,6 +1322,15 @@ The `cleanup` application deletes DIFF and INCR if the archives are older than t
|
|
|
1263
1322
|
|
|
1264
1323
|
`cleanup` will only remove FULL archives if the option `--cleanup-specific-archives` is used. It requires the user to confirm deletion of FULL archives.
|
|
1265
1324
|
|
|
1325
|
+
Use `--dry-run` to preview which archives, PAR2 files, and catalogs would be removed without deleting anything.
|
|
1326
|
+
|
|
1327
|
+
Examples:
|
|
1328
|
+
|
|
1329
|
+
```bash
|
|
1330
|
+
cleanup --dry-run -d media-files --log-stdout
|
|
1331
|
+
cleanup --dry-run --cleanup-specific-archives -d media-files media-files_INCR_2025-12-22
|
|
1332
|
+
```
|
|
1333
|
+
|
|
1266
1334
|
### manager
|
|
1267
1335
|
|
|
1268
1336
|
`dar`has the concept of catalogs which can be exported and optionally be added to a catalog database. That database makes it much easier to restore the correct version of a backed up file if for example a target date has been set.
|
|
@@ -1406,7 +1474,7 @@ deactivate
|
|
|
1406
1474
|
|
|
1407
1475
|
The configuration file's default location is: ~/.config/dar-backup/dar-backup.conf
|
|
1408
1476
|
|
|
1409
|
-
If you have your config file somewhere else, use the `--config` option to point to it.
|
|
1477
|
+
If you have your config file somewhere else, use the `--config-file` option to point to it.
|
|
1410
1478
|
|
|
1411
1479
|
Tilde `~` and environment variables can be used in the paths for various file locations.
|
|
1412
1480
|
|
|
@@ -1442,6 +1510,17 @@ INCR_AGE = 40
|
|
|
1442
1510
|
[PAR2]
|
|
1443
1511
|
ERROR_CORRECTION_PERCENT = 5
|
|
1444
1512
|
ENABLED = True
|
|
1513
|
+
# Optional PAR2 configuration
|
|
1514
|
+
# PAR2_DIR = /path/to/par2-store
|
|
1515
|
+
# PAR2_RATIO_FULL = 10
|
|
1516
|
+
# PAR2_RATIO_DIFF = 5
|
|
1517
|
+
# PAR2_RATIO_INCR = 5
|
|
1518
|
+
# PAR2_RUN_VERIFY = false
|
|
1519
|
+
|
|
1520
|
+
# Optional per-backup overrides (section name = backup definition)
|
|
1521
|
+
[media-files]
|
|
1522
|
+
PAR2_DIR = /mnt/par2/media-files
|
|
1523
|
+
PAR2_RATIO_FULL = 10
|
|
1445
1524
|
|
|
1446
1525
|
# scripts to run before the backup to setup the environment
|
|
1447
1526
|
[PREREQ]
|
|
@@ -1453,6 +1532,16 @@ SCRIPT_1 = df -h
|
|
|
1453
1532
|
#SCRIPT_2 = another_script.sh
|
|
1454
1533
|
```
|
|
1455
1534
|
|
|
1535
|
+
PAR2 notes:
|
|
1536
|
+
|
|
1537
|
+
- If `PAR2_DIR` is unset, par2 files are created next to the archive slices (legacy behavior) and no manifest is written
|
|
1538
|
+
- When `PAR2_DIR` is set, dar-backup writes a manifest next to the par2 set:
|
|
1539
|
+
`archive_base.par2.manifest.ini`
|
|
1540
|
+
- When generating a par2 set, par2 reads all archive slices before writing any output files; for large backups, this initial read can take hours
|
|
1541
|
+
- Verify or repair using:
|
|
1542
|
+
`par2 verify -B <archive_dir> <par2_set.par2>`
|
|
1543
|
+
`par2 repair -B <archive_dir> <par2_set.par2>`
|
|
1544
|
+
|
|
1456
1545
|
### .darrc
|
|
1457
1546
|
|
|
1458
1547
|
The package includes a default `darrc` file which configures `dar`.
|
|
@@ -1774,6 +1863,25 @@ This happens when the shell splits the quoted string or interprets globs before
|
|
|
1774
1863
|
|
|
1775
1864
|
> 💡 **Tip:** See [dar's documentation](http://dar.linux.free.fr/doc/man/dar.html#COMMANDS%20AND%20OPTIONS)
|
|
1776
1865
|
|
|
1866
|
+
>
|
|
1867
|
+
> 💡💡 **Tip:** To filter all the empty directories away that `dar` emits when listing contents, append this grep:
|
|
1868
|
+
>
|
|
1869
|
+
> ```bash
|
|
1870
|
+
> |grep -vE '\s+d[rwx-]{9}\s'
|
|
1871
|
+
>```
|
|
1872
|
+
>
|
|
1873
|
+
>Example using the grep to discard directory noise from `dar's` output:
|
|
1874
|
+
>
|
|
1875
|
+
> ```bash
|
|
1876
|
+
> dar-backup --list-contents media-files_INCR_2025-05-10 --selection="-I '*Z50*' -X '*.xmp'" | grep -vE '\s+d[rwx-]{9}\s'
|
|
1877
|
+
>[Saved][ ] [-L-][ 0%][X] -rw-rw-r-- user user 26 Mio Fri May 9 11:26:16 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0633.NEF
|
|
1878
|
+
>[Saved][ ] [-L-][ 0%][X] -rw-rw-r-- user user 26 Mio Fri May 9 11:26:16 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0632.NEF
|
|
1879
|
+
>[Saved][ ] [-L-][ 0%][X] -rw-rw-r-- user user 28 Mio Fri May 9 11:09:04 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0631.NEF
|
|
1880
|
+
>[Saved][ ] [-L-][ 0%][X] -rw-rw-r-- user user 29 Mio Fri May 9 11:09:03 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0630.NEF
|
|
1881
|
+
>[Saved][ ] [-L-][ 0%][X] -rw-rw-r-- user user 29 Mio Fri May 9 11:09:03 2025 home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling/Z50_0629.NEF
|
|
1882
|
+
>...
|
|
1883
|
+
>```
|
|
1884
|
+
|
|
1777
1885
|
### select a directory
|
|
1778
1886
|
|
|
1779
1887
|
Select files and sub directories in `home/user/data/2025/2025-05-09-Roskilde-Nordisk-udstilling`
|
|
@@ -1900,9 +2008,39 @@ If you need to use this option, un-comment it in the [.darrc](#darrc) file (loca
|
|
|
1900
2008
|
|
|
1901
2009
|
## Par2
|
|
1902
2010
|
|
|
2011
|
+
Why keep PAR2 on a different storage device:
|
|
2012
|
+
|
|
2013
|
+
- Reduces single-disk failure impact: bitrot on the archive disk does not affect the parity.
|
|
2014
|
+
- Easier offsite rotation: you can sync only the PAR2 sets to a different failure domain.
|
|
2015
|
+
|
|
2016
|
+
Redundancy guidance:
|
|
2017
|
+
|
|
2018
|
+
- FULL backups: 10% is a practical default for larger data sets and longer retention.
|
|
2019
|
+
- DIFF/INCR: 5% is often enough because the delta is smaller and easier to re-create.
|
|
2020
|
+
- Increase the ratio if the storage is flaky or the backup is hard to re-run.
|
|
2021
|
+
|
|
2022
|
+
Rule of thumb table:
|
|
2023
|
+
|
|
2024
|
+
| Backup type | Suggested PAR2 ratio | Notes |
|
|
2025
|
+
|-------------|----------------------|-------|
|
|
2026
|
+
| FULL | 10% | Longer retention, larger data set |
|
|
2027
|
+
| DIFF | 5% | Smaller delta |
|
|
2028
|
+
| INCR | 5% | Smaller delta |
|
|
2029
|
+
|
|
2030
|
+
>
|
|
2031
|
+
>For large, contiguous archives on reliable local storage, 7–8% has proven sufficient in practice; 10% remains a conservative default.
|
|
2032
|
+
>
|
|
2033
|
+
|
|
2034
|
+
Cloud sync / air-gap note:
|
|
2035
|
+
|
|
2036
|
+
- Syncing PAR2 sets to a different device or remote store protects against bitrot and small corruption, but it cannot recover a completely lost archive.
|
|
2037
|
+
- An air-gapped PAR2 store is useful when the archive disk is exposed to ransomware or accidental deletion.
|
|
2038
|
+
|
|
1903
2039
|
### Par2 to verify/repair
|
|
1904
2040
|
|
|
1905
|
-
|
|
2041
|
+
#### Par2 files kept with archives
|
|
2042
|
+
|
|
2043
|
+
If PAR2 files are stored next to the archives (legacy per-slice behavior), you can verify like this:
|
|
1906
2044
|
|
|
1907
2045
|
```bash
|
|
1908
2046
|
for file in <archive>*.dar.par2; do
|
|
@@ -1916,6 +2054,14 @@ if there are problems with a slice, try to repair it like this:
|
|
|
1916
2054
|
par2 repair <archive>.<slice number>.dar.par2
|
|
1917
2055
|
```
|
|
1918
2056
|
|
|
2057
|
+
#### Par2 files in separate directory
|
|
2058
|
+
|
|
2059
|
+
See [docs on disk layout matters](v2/doc/portable-par2-layout.md)
|
|
2060
|
+
|
|
2061
|
+
>Test case proving this flow:
|
|
2062
|
+
>
|
|
2063
|
+
>[tests/test_par2_manifest.py](v2/tests/test_par2_manifest.py)
|
|
2064
|
+
|
|
1919
2065
|
### Par2 create redundancy files
|
|
1920
2066
|
|
|
1921
2067
|
If you have merged archives, you will need to create the .par2 redundency files manually.
|
|
@@ -1929,6 +2075,15 @@ done
|
|
|
1929
2075
|
|
|
1930
2076
|
where "c" is create, -r5 is 5% redundency and -n1 is 1 redundency file
|
|
1931
2077
|
|
|
2078
|
+
If you want to create a single parity set for all slices in an archive:
|
|
2079
|
+
|
|
2080
|
+
```bash
|
|
2081
|
+
par2 create -B <archive_dir> -r5 <par2_dir>/<archive_base>.par2 <archive_dir>/<archive_base>.*.dar
|
|
2082
|
+
```
|
|
2083
|
+
|
|
2084
|
+
**OBSERVE** [docs on disk layout matters](v2/doc/portable-par2-layout.md)
|
|
2085
|
+
|
|
2086
|
+
|
|
1932
2087
|
## Points of interest
|
|
1933
2088
|
|
|
1934
2089
|
### Limitations on File Names with Special Characters
|
|
@@ -2083,23 +2238,29 @@ In order to not clutter that log file with the output of commands being run, a n
|
|
|
2083
2238
|
|
|
2084
2239
|
The secondary log file can get quite cluttered, if you want to remove the clutter, run the `clean-log`script with the `--file` option, or simply delete it.
|
|
2085
2240
|
|
|
2086
|
-
###
|
|
2241
|
+
### Trace Logging (Debug details)
|
|
2087
2242
|
|
|
2088
|
-
|
|
2243
|
+
To keep the main log file clean while preserving essential debugging information, `dar-backup` creates a separate trace log file (e.g., `dar-backup.trace.log`) alongside the main log.
|
|
2089
2244
|
|
|
2090
|
-
|
|
2245
|
+
- **Main Log (`dar-backup.log`)**: Contains clean, human-readable INFO/ERROR messages. Stack traces are suppressed here.
|
|
2246
|
+
- **Trace Log (`dar-backup.trace.log`)**: Captures ALL messages at `DEBUG` level, including full exception stack traces. Use this file for debugging crashes or unexpected behavior.
|
|
2091
2247
|
|
|
2092
|
-
|
|
2248
|
+
You can configure the rotation of this file in `[MISC]`:
|
|
2249
|
+
|
|
2250
|
+
```ini
|
|
2251
|
+
[MISC]
|
|
2252
|
+
# ... other settings ...
|
|
2253
|
+
TRACE_LOG_MAX_BYTES = 10485760 # 10 MB default
|
|
2254
|
+
TRACE_LOG_BACKUP_COUNT = 1 # Keep 1 old trace file (default)
|
|
2255
|
+
```
|
|
2093
2256
|
|
|
2094
|
-
###
|
|
2257
|
+
### Skipping cache directories
|
|
2095
2258
|
|
|
2096
|
-
|
|
2097
|
-
dar-backup displays 2 visual artifacts to show progress.
|
|
2259
|
+
The author uses the `--cache-directory-tagging` option in his [backup definitions](#backup-definition-example).
|
|
2098
2260
|
|
|
2099
|
-
|
|
2100
|
-
2. a status line showing the directory being backed up. If the directory is big and takes time to backup, the line is not changing, but you will probably know there is a lot to backup.
|
|
2261
|
+
The effect is that directories with the [CACHEDIR.TAG](https://bford.info/cachedir/) file are not backed up. Those directories contain content fetched from the net, which is of an ephemeral nature and probably not what you want to back up.
|
|
2101
2262
|
|
|
2102
|
-
|
|
2263
|
+
If the option is not in the backup definition, the cache directories are backed up as any other.
|
|
2103
2264
|
|
|
2104
2265
|
### Shell autocompletion
|
|
2105
2266
|
|
|
@@ -2284,7 +2445,6 @@ pytest # run the test suite
|
|
|
2284
2445
|
|
|
2285
2446
|
- Perhaps look into pre-processing backup definitions. As `dar` does not expand env vars
|
|
2286
2447
|
`dar-backup` could do so and feed the result to `dar`.
|
|
2287
|
-
- When run interactively, a progress bar during test and par2 generation would be nice.
|
|
2288
2448
|
- Look into a way to move the .par2 files away from the `dar` slices, to maximize chance of good redundancy.
|
|
2289
2449
|
- Add option to dar-backup to use the `dar` option `--fsa-scope none`
|
|
2290
2450
|
|
|
@@ -2309,15 +2469,15 @@ One backup definition per file
|
|
|
2309
2469
|
|
|
2310
2470
|
### CLI Tools Overview
|
|
2311
2471
|
|
|
2312
|
-
| Command
|
|
2313
|
-
|
|
2314
|
-
| [dar-backup](#dar-backup-options)| Perform full, differential, or incremental backups with verification and restore testing |
|
|
2315
|
-
| [manager](#manager-options)
|
|
2316
|
-
| [cleanup](#cleanup-options)
|
|
2317
|
-
| [clean-log](#clean-log-options)
|
|
2472
|
+
| Command | Description |
|
|
2473
|
+
| --- | --- |
|
|
2474
|
+
| [dar-backup](#dar-backup-options) | Perform full, differential, or incremental backups with verification and restore testing |
|
|
2475
|
+
| [manager](#manager-options) | Maintain and query catalog databases for archives |
|
|
2476
|
+
| [cleanup](#cleanup-options) | Remove outdated DIFF/INCR archives (and optionally FULLs) |
|
|
2477
|
+
| [clean-log](#clean-log-options) | Clean up excessive log output from dar command logs |
|
|
2318
2478
|
| [dar-backup-systemd](#dar-backup-systemd-options) | Generate (and optionally install) systemd timers and services for automated backups |
|
|
2319
|
-
| [installer](#installer-options)
|
|
2320
|
-
| [demo](#demo-options)
|
|
2479
|
+
| [installer](#installer-options) | Set up directories and optionally create catalog databases according to a config file |
|
|
2480
|
+
| [demo](#demo-options) | Set up required directories and config files for a demo|
|
|
2321
2481
|
|
|
2322
2482
|
### test coverage
|
|
2323
2483
|
|
|
@@ -2372,6 +2532,7 @@ Available options:
|
|
|
2372
2532
|
--examples Show examples of using dar-backup.py.
|
|
2373
2533
|
-l, --list List available backups.
|
|
2374
2534
|
--list-contents <archive> List the contents of a specified archive.
|
|
2535
|
+
--list-definitions List backup definitions from BACKUP.D_DIR.
|
|
2375
2536
|
--selection <params> Define file selection for listing/restoring.
|
|
2376
2537
|
--restore <archive> Restore a specified archive.
|
|
2377
2538
|
-r, --restore <archive> Restore archive.
|
|
@@ -2381,6 +2542,7 @@ Available options:
|
|
|
2381
2542
|
--log-level <level> `debug` or `trace`, default is `info`.
|
|
2382
2543
|
--log-stdout Also print log messages to stdout.
|
|
2383
2544
|
--do-not-compare Do not compare restores to file system.
|
|
2545
|
+
--preflight-check Run preflight checks and exit (runs automatically; this flag just exits after checks).
|
|
2384
2546
|
--examples Show examples of using dar-backup.
|
|
2385
2547
|
--readme Print README.md and exit
|
|
2386
2548
|
--readme-pretty Print README.md with Markdown styling and exit
|
|
@@ -2389,6 +2551,22 @@ Available options:
|
|
|
2389
2551
|
-v, --version Show version and license information.
|
|
2390
2552
|
```
|
|
2391
2553
|
|
|
2554
|
+
#### Dar-backup exit codes
|
|
2555
|
+
|
|
2556
|
+
- 0: Success.
|
|
2557
|
+
- 1: Error (backup/restore/preflight failure).
|
|
2558
|
+
- 2: Warning (restore test failed or backup already exists and is skipped).
|
|
2559
|
+
- 127: Typically an error during startup, file or config value missing
|
|
2560
|
+
- if the `dar -t` test fails, exit code 1 is emitted
|
|
2561
|
+
- restore tests could fail if the source file has changed after the backup
|
|
2562
|
+
|
|
2563
|
+
#### Dar-backup env vars
|
|
2564
|
+
|
|
2565
|
+
| Env var | Value | Description |
|
|
2566
|
+
| --- | --- | --- |
|
|
2567
|
+
| DAR_BACKUP_CONFIG_FILE | Full path to config file | Overrides built-in default, overridden by --config-file |
|
|
2568
|
+
| DAR_BACKUP_DISCORD_WEBHOOK_URL | https://discord.com/api/webhooks/\<userID\>/\<webhook UUID\> | The full url |
|
|
2569
|
+
|
|
2392
2570
|
### Manager Options
|
|
2393
2571
|
|
|
2394
2572
|
This script manages `dar` databases and catalogs.
|
|
@@ -2408,7 +2586,12 @@ Available options:
|
|
|
2408
2586
|
--find-file <file> Search catalogs for a specific file.
|
|
2409
2587
|
--verbose Enable verbose output.
|
|
2410
2588
|
--log-level <level> Set log level (`debug` or `trace`, default is `info`).
|
|
2411
|
-
|
|
2589
|
+
|
|
2590
|
+
#### Manager env vars
|
|
2591
|
+
|
|
2592
|
+
| Env var | Value | Description |
|
|
2593
|
+
| --- | --- | --- |
|
|
2594
|
+
| DAR_BACKUP_CONFIG_FILE | path to the config file | Default is $HOME/.config/dar-backup/dar-backup.conf |```
|
|
2412
2595
|
|
|
2413
2596
|
### Cleanup options
|
|
2414
2597
|
|
|
@@ -2425,12 +2608,19 @@ Supported options:
|
|
|
2425
2608
|
--alternate-archive-dir Clean up in this directory instead of the default one.
|
|
2426
2609
|
--cleanup-specific-archives "<archive>, <>, ..." Comma separated list of archives to cleanup.
|
|
2427
2610
|
-l, --list List available archives (filter using the -d option).
|
|
2611
|
+
--dry-run Show what would be deleted without removing files.
|
|
2428
2612
|
--verbose Print various status messages to screen.
|
|
2429
2613
|
--log-level <level> `debug` or `trace`, default is `info`", default="info".
|
|
2430
2614
|
--log-stdout Print log messages to stdout.
|
|
2431
2615
|
--test-mode This is used when running pytest test cases
|
|
2432
2616
|
```
|
|
2433
2617
|
|
|
2618
|
+
#### Cleanup env vars
|
|
2619
|
+
|
|
2620
|
+
| Env var | Value | Description |
|
|
2621
|
+
| --- | --- | --- |
|
|
2622
|
+
| DAR_BACKUP_CONFIG_FILE | path to the config file | Default is $HOME/.config/dar-backup/dar-backup.conf |
|
|
2623
|
+
|
|
2434
2624
|
### Clean-log options
|
|
2435
2625
|
|
|
2436
2626
|
This script removes excessive logging output from `dar` logs, improving readability and efficiency. Available options:
|
|
@@ -2480,8 +2670,8 @@ Create directories:
|
|
|
2480
2670
|
- ~/.config/dar-backup/
|
|
2481
2671
|
- ~/.config/dar-backup/backup.d/
|
|
2482
2672
|
- ~/dar-backup/
|
|
2483
|
-
- ~/dar-backup/backups
|
|
2484
|
-
- ~/dar-backup/restore
|
|
2673
|
+
- ~/dar-backup/backups/
|
|
2674
|
+
- ~/dar-backup/restore/
|
|
2485
2675
|
|
|
2486
2676
|
Sets up demo config files:
|
|
2487
2677
|
|
|
@@ -2500,3 +2690,134 @@ Sets up demo config files:
|
|
|
2500
2690
|
-v, --version Display version and licensing information.
|
|
2501
2691
|
-h, --help Displays usage info
|
|
2502
2692
|
```
|
|
2693
|
+
|
|
2694
|
+
### Config changes
|
|
2695
|
+
|
|
2696
|
+
#### 1.0.1
|
|
2697
|
+
|
|
2698
|
+
##### DISCORD WEBHOOK
|
|
2699
|
+
|
|
2700
|
+
For Discord notifications use the `DAR_BACKUP_DISCORD_WEBHOOK_URL` environment variable. It should not be placed in the config file.
|
|
2701
|
+
|
|
2702
|
+
DAR_BACKUP_DISCORD_WEBHOOK_URL is the entire endpoint like this:
|
|
2703
|
+
|
|
2704
|
+
```bash
|
|
2705
|
+
export DAR_BACKUP_DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/\<userId\>/\<uuid\>
|
|
2706
|
+
```
|
|
2707
|
+
|
|
2708
|
+
##### Restore test config
|
|
2709
|
+
|
|
2710
|
+
Restore tests choose random files from the archive and compare them with the live filesystem.
|
|
2711
|
+
To avoid noisy paths (caches, temp files, logs), you can exclude candidates before the random
|
|
2712
|
+
selection happens. All matching is case-insensitive.
|
|
2713
|
+
|
|
2714
|
+
Config keys (in [MISC]):
|
|
2715
|
+
|
|
2716
|
+
- RESTORETEST_EXCLUDE_PREFIXES: comma-separated path prefixes to skip. Matches from the start of
|
|
2717
|
+
the path (after trimming a leading "/"). Use trailing "/" for directories.
|
|
2718
|
+
- RESTORETEST_EXCLUDE_SUFFIXES: comma-separated filename suffixes to skip.
|
|
2719
|
+
- RESTORETEST_EXCLUDE_REGEX: optional regex to skip anything matching the path.
|
|
2720
|
+
|
|
2721
|
+
Example:
|
|
2722
|
+
|
|
2723
|
+
```ini
|
|
2724
|
+
[MISC]
|
|
2725
|
+
RESTORETEST_EXCLUDE_PREFIXES = .cache/, .local/share/Trash/, .mozilla/, snap/firefox/common/.mozilla/
|
|
2726
|
+
RESTORETEST_EXCLUDE_SUFFIXES = .sqlite-wal, .sqlite-shm, .log, .tmp, .lock, .journal
|
|
2727
|
+
RESTORETEST_EXCLUDE_REGEX = (^|/)(Cache|cache|Logs|log)/
|
|
2728
|
+
```
|
|
2729
|
+
|
|
2730
|
+
Regex tips (case-insensitive):
|
|
2731
|
+
|
|
2732
|
+
- Match common cache/log directories anywhere:
|
|
2733
|
+
`(^|/)(cache|logs)/`
|
|
2734
|
+
- Skip thumbnails and temp dirs:
|
|
2735
|
+
`(^|/)(thumbnails|tmp|temp)/`
|
|
2736
|
+
- Exclude browser profile noise while keeping other files:
|
|
2737
|
+
`(^|/)\.mozilla/|/snap/firefox/common/\.mozilla/`
|
|
2738
|
+
|
|
2739
|
+
##### Par2
|
|
2740
|
+
|
|
2741
|
+
New optional PAR2 settings were added to the config file. If none of these keys are added, dar-backup behaves exactly as before (PAR2 files next to archives, per-slice parity).
|
|
2742
|
+
|
|
2743
|
+
| Name | Description | When it is in effect | Suggested value |
|
|
2744
|
+
|------|-------------|----------------------|-----------------|
|
|
2745
|
+
| PAR2_DIR | Directory to store .par2 and .vol*.par2 files | When set | A different device or mount from BACKUP_DIR |
|
|
2746
|
+
| PAR2_RATIO_FULL | Redundancy percent for FULL | When set | 10 (%) |
|
|
2747
|
+
| PAR2_RATIO_DIFF | Redundancy percent for DIFF | When set | 5 (%)|
|
|
2748
|
+
| PAR2_RATIO_INCR | Redundancy percent for INCR | When set | 5 (%)|
|
|
2749
|
+
| PAR2_RUN_VERIFY | Verify after create | When set | false |
|
|
2750
|
+
|
|
2751
|
+
Notes:
|
|
2752
|
+
|
|
2753
|
+
- PAR2_RATIO_*, and PAR2_RUN_VERIFY apply even if PAR2_DIR is not set (i.e. par2 output stays next to the archives).
|
|
2754
|
+
|
|
2755
|
+
Per-backup overrides use a section named after the backup definition with the same PAR2_* keys:
|
|
2756
|
+
|
|
2757
|
+
```text
|
|
2758
|
+
|
|
2759
|
+
######################################################################
|
|
2760
|
+
# Per-backup configuration example overrides
|
|
2761
|
+
######################################################################
|
|
2762
|
+
|
|
2763
|
+
# --------------------------------------------------------------------
|
|
2764
|
+
# Per-backup overrides (section name must match backup.d filename stem)
|
|
2765
|
+
# Example: backup.d/home.conf -> [home]
|
|
2766
|
+
# --------------------------------------------------------------------
|
|
2767
|
+
|
|
2768
|
+
#[home]
|
|
2769
|
+
# Disable PAR2 entirely for this backup definition
|
|
2770
|
+
PAR2_ENABLED = false
|
|
2771
|
+
#
|
|
2772
|
+
#[media]
|
|
2773
|
+
# Store PAR2 files in a separate location for this backup definition
|
|
2774
|
+
#PAR2_DIR = /samba/par2/media
|
|
2775
|
+
# Raise redundancy only for FULL
|
|
2776
|
+
#
|
|
2777
|
+
[documents]
|
|
2778
|
+
# Run verify par2 sets after creation
|
|
2779
|
+
PAR2_RUN_VERIFY = true
|
|
2780
|
+
#
|
|
2781
|
+
#[etc]
|
|
2782
|
+
# Keep global PAR2 settings but tweak ratios for this backup definition
|
|
2783
|
+
# RATIO is given in percent (%)
|
|
2784
|
+
#PAR2_RATIO_FULL = 15
|
|
2785
|
+
#PAR2_RATIO_DIFF = 8
|
|
2786
|
+
#PAR2_RATIO_INCR = 8
|
|
2787
|
+
```
|
|
2788
|
+
|
|
2789
|
+
[Per-backup override test case: `tests/test_par2_overrides.py`](v2/tests/test_par2_overrides.py)
|
|
2790
|
+
|
|
2791
|
+
#### 1.0.2
|
|
2792
|
+
|
|
2793
|
+
##### Trace Logging
|
|
2794
|
+
|
|
2795
|
+
To support debugging without cluttering the main log file, a secondary trace log is now created (e.g., `dar-backup.trace.log`).
|
|
2796
|
+
This file captures all `DEBUG` level messages and full exception stack traces.
|
|
2797
|
+
|
|
2798
|
+
You can configure its rotation in the `[MISC]` section:
|
|
2799
|
+
|
|
2800
|
+
- `TRACE_LOG_MAX_BYTES`: Max size of the trace log file in bytes. Default is `10485760` (10 MB).
|
|
2801
|
+
- `TRACE_LOG_BACKUP_COUNT`: Number of rotated trace log files to keep. Default is `1`.
|
|
2802
|
+
|
|
2803
|
+
Example:
|
|
2804
|
+
|
|
2805
|
+
```ini
|
|
2806
|
+
[MISC]
|
|
2807
|
+
TRACE_LOG_MAX_BYTES = 10485760
|
|
2808
|
+
TRACE_LOG_BACKUP_COUNT = 1
|
|
2809
|
+
```
|
|
2810
|
+
|
|
2811
|
+
##### Command output Capture
|
|
2812
|
+
|
|
2813
|
+
- New optional `[MISC]` setting: `COMMAND_CAPTURE_MAX_BYTES` (default 102400).
|
|
2814
|
+
- Limits how much stdout/stderr is kept in memory per command while still logging full output.
|
|
2815
|
+
- Set to `0` to disable buffering entirely. Command output is still streamed to dar-backup-commands.log
|
|
2816
|
+
- If set to `0`, the calling function cannot rely on output from the executed command. The exit value is the only result provided.
|
|
2817
|
+
|
|
2818
|
+
Example:
|
|
2819
|
+
|
|
2820
|
+
```ini
|
|
2821
|
+
[MISC]
|
|
2822
|
+
COMMAND_CAPTURE_MAX_BYTES = 102400
|
|
2823
|
+
```
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
dar_backup/.darrc,sha256=-aerqivZmOsW_XBCh9IfbYTUvw0GkzDSr3Vx4GcNB1g,2113
|
|
2
|
+
dar_backup/Changelog.md,sha256=gIAlrXElafjmReneKZlhaPd5FcXU7SJ58_P-huzLTO8,19436
|
|
3
|
+
dar_backup/README.md,sha256=b9w97562AlVTE7ob_5kmf1pc9EiUi7swv8g11M4NOrE,78420
|
|
4
|
+
dar_backup/__about__.py,sha256=0lLpLB4_DxcwxlIZF1F8V8O6WVfjzKepl2PPulVwn2Q,370
|
|
5
|
+
dar_backup/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
|
6
|
+
dar_backup/clean_log.py,sha256=Uf6MjpS_v1IVZdfx7ogRo6j5EIscdxjLImn8GKZo_rc,7038
|
|
7
|
+
dar_backup/cleanup.py,sha256=aC7ogT68u570ufXrzFgpCcxWgzN5dX0o5LIgZhgbQKo,19147
|
|
8
|
+
dar_backup/command_runner.py,sha256=ktK_LmVUZATZlfXPniGLC4W_8x187OzKreK7Dhb5LUA,13123
|
|
9
|
+
dar_backup/config_settings.py,sha256=PKe6iTVeSsN1DW4kTgFbUHsl8S5E-f0I3Lhn4P9a4hs,12723
|
|
10
|
+
dar_backup/dar-backup.conf,sha256=ejucmhEbSdqFU4JEVfrWbFj9H6U87TWSnhex3LraEUY,2124
|
|
11
|
+
dar_backup/dar-backup.conf.j2,sha256=kD13msmP2IXLjnJ9lZmqtLTnsUZq1EkPZAI1LlOkYXo,3852
|
|
12
|
+
dar_backup/dar_backup.py,sha256=997dp6zjXluoenldo19KYtDVt91uMJxoHOI6cIVzohs,69805
|
|
13
|
+
dar_backup/dar_backup_systemd.py,sha256=PwAc2H2J3hQLWpnC6Ib95NZYtB2G2NDgkSblfLj1n10,3875
|
|
14
|
+
dar_backup/demo.py,sha256=KaW-KOqP1sBX4TYJoO-Wr84e81ILI2kN2iV57QU6rbg,8243
|
|
15
|
+
dar_backup/demo_backup_def.j2,sha256=hQW2Glp0QGV3Kt8cwjS0mpOCdyzjVlpgbgL6LpXTKJA,1793
|
|
16
|
+
dar_backup/exceptions.py,sha256=dBQwgKUilgfb1Tnlm_m1Dl1IqoY75L0n1uRkm0DBVTk,158
|
|
17
|
+
dar_backup/installer.py,sha256=yJ0xXi24C1Vo6gnj2BlhV1X55ZIG5t21t8tNKTYSclM,7616
|
|
18
|
+
dar_backup/manager.py,sha256=rctD1WNdSwyEiDw81wnhPyzIVKjrax2QU71PRSx4MqQ,34995
|
|
19
|
+
dar_backup/rich_progress.py,sha256=SfwFxebBl6jnDQMUQr4McknkW1yQWaJVo1Ju1OD3okA,3221
|
|
20
|
+
dar_backup/util.py,sha256=wVyHzdZIQSnXPSup2m0Wx1w-PKx_ornsg4MMEW5AoKs,38608
|
|
21
|
+
dar_backup-1.0.2.dist-info/METADATA,sha256=7_shfQw0tpdJbnlfR46Abd03LxkTzQZj_Xv1mtjNAIM,121126
|
|
22
|
+
dar_backup-1.0.2.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
|
|
23
|
+
dar_backup-1.0.2.dist-info/entry_points.txt,sha256=pOK9M8cHeAcGIatrYzkm_1O89kPk0enyYONALYjFBx4,286
|
|
24
|
+
dar_backup-1.0.2.dist-info/licenses/LICENSE,sha256=OXLcl0T2SZ8Pmy2_dmlvKuetivmyPd5m1q-Gyd-zaYY,35149
|
|
25
|
+
dar_backup-1.0.2.dist-info/RECORD,,
|