borgbackup 2.0.0b9__tar.gz → 2.0.0b10__tar.gz

{borgbackup-2.0.0b9 → borgbackup-2.0.0b10}/.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
 -   repo: https://github.com/psf/black
-    rev: 23.1.0
+    rev: 24.8.0
     hooks:
     -   id: black
 - repo: https://github.com/astral-sh/ruff-pre-commit

{borgbackup-2.0.0b9 → borgbackup-2.0.0b10}/CHANGES.rst

@@ -12,8 +12,8 @@ This section provides information about security and corruption issues.
 Upgrade Notes
 =============
 
-borg 1.2.x to borg 2.0
-----------------------
+borg 1.2.x/1.4.x to borg 2.0
+----------------------------
 
 Compatibility notes:
 
@@ -21,11 +21,11 @@ Compatibility notes:
 
   We tried to put all the necessary "breaking" changes into this release, so we
   hopefully do not need another breaking release in the near future. The changes
-  were necessary for improved security, improved speed, unblocking future
-  improvements, getting rid of legacy crap / design limitations, having less and
-  simpler code to maintain.
+  were necessary for improved security, improved speed and parallelism,
+  unblocking future improvements, getting rid of legacy crap and design
+  limitations, having less and simpler code to maintain.
 
-  You can use "borg transfer" to transfer archives from borg 1.1/1.2 repos to
+  You can use "borg transfer" to transfer archives from borg 1.2/1.4 repos to
   a new borg 2.0 repo, but it will need some time and space.
 
   Before using "borg transfer", you must have upgraded to borg >= 1.2.6 (or
@@ -58,10 +58,12 @@ Compatibility notes:
 
     - borg 2 repo commands:
 
-      - borg rcreate  # "repo create", was: borg init
-      - borg rlist  # "repo list"
-      - borg rinfo  # "repo info"
-      - borg rdelete  # "repo delete"
+      - borg repo-create  # was: borg init
+      - borg repo-list
+      - borg repo-info
+      - borg repo-delete
+      - borg repo-compress
+      - borg repo-space
     - borg 2 archive commands:
 
       - borg create ARCHIVE ...
@@ -84,6 +86,8 @@ Compatibility notes:
   - removed --nobsdflags (use --noflags)
   - removed --noatime (default now, see also --atime)
   - removed --save-space option (does not change behaviour)
+- removed --bypass-lock option
+- removed borg config command (only worked locally anyway)
 - using --list together with --progress is now disallowed (except with --log-json), #7219
 - the --glob-archives option was renamed to --match-archives (the short option
   name -a is unchanged) and extended to support different pattern styles:
@@ -114,14 +118,13 @@ Compatibility notes:
   fail now that somehow "worked" before (but maybe didn't work as intended due to
   the contradicting options).
 
-
 .. _changelog:
 
 Change Log 2.x
 ==============
 
-Version 2.0.0b9 (2024-07-20)
-----------------------------
+Version 2.0.0b10 (2024-09-09)
+-----------------------------
 
 Please note:
 
@@ -130,6 +133,60 @@ This is a beta release, only for testing - do not use for production repos.
 For upgrade and compatibility hints, please also read the section "Upgrade Notes"
 above.
 
+TL;DR: this is a huge change and the first very fundamental change in how borg
+works since ever:
+
+- you will need to create new repos.
+- likely more exciting than previous betas, definitely not for production.
+- remove "borg check --repair --archives-only" from your script in case you
+  added that for b9. "borg compact -v" now finds and deletes orphan chunks
+  and also outputs some new stats.
+
+New features:
+
+- borgstore based repository, file:, ssh: and sftp: for now, more possible.
+- repository stores objects separately now, not using segment files.
+  this has more fs overhead, but needs much less I/O because no segment
+  files compaction is required anymore. also, no repository index is
+  needed anymore because we can directly find the objects by their ID.
+- locking: new borgstore based repository locking with automatic stale
+  lock removal (if lock does not get refreshed, if lock owner process is dead).
+- simultaneous repository access for many borg commands except check/compact.
+  the cache lock for adhocwithfiles is still exclusive though, so use
+  BORG_CACHE_IMPL=adhoc if you want to try that out using only 1 machine
+  and 1 user (that implementation doesn't use a cache lock). When using
+  multiple client machines or users, it also works with the default cache.
+- delete/prune: much quicker now and can be undone.
+- check --repair --undelete-archives: bring archives back from the dead.
+- repo-space: manage reserved space in repository (avoid dead-end situation if
+  repository filesystem runs full).
+
+Bugs/issues fixed:
+
+- a lot! all linked from PR #8332.
+
+Other changes:
+
+- repository: remove transactions, solved differently and much simpler now
+  (convergence and write order primarily).
+- repository: replaced precise reference counting with "object exists in repo?"
+  and "garbage collection of unused objects".
+- cache: remove transactions, remove chunks cache.
+  removed LocalCache, BORG_CACHE_IMPL=local, solving all related issues.
+  as in beta 9, adhowwithfiles is the default implementation.
+- compact: needs the borg key now (run it clientside), -v gives nice stats.
+- transfer: archive transfers from borg 1.x need the --from-borg1 option
+- check: reimplemented / bigger changes.
+- code: got rid of a metric ton of not needed complexity.
+  when borg does not need to read borg 1.x repos/archives anymore, after
+  users have transferred their archives, even much more can be removed.
+- docs: updated / removed outdated stuff
+- renamed r* commands to repo-*
+
+
+Version 2.0.0b9 (2024-07-20)
+----------------------------
+
 New features:
 
 - add BORG_CACHE_IMPL, default is "adhocwithfiles" to test the new cache

{borgbackup-2.0.0b9 → borgbackup-2.0.0b10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: borgbackup
-Version: 2.0.0b9
+Version: 2.0.0b10
 Summary: Deduplicated, encrypted, authenticated and compressed backups
 Author: The Borg Collective (see AUTHORS file)
 Maintainer-email: Thomas Waldmann <tw@waldmann-edv.de>
@@ -26,11 +26,13 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Security :: Cryptography
 Classifier: Topic :: System :: Archiving :: Backup
 Requires-Python: >=3.9
 License-File: LICENSE
 License-File: AUTHORS
+Requires-Dist: borgstore~=0.0.1
 Requires-Dist: msgpack<=1.1.0,>=1.0.3
 Requires-Dist: packaging
 Requires-Dist: platformdirs<5.0.0,>=3.0.0; sys_platform == "darwin"
@@ -93,7 +95,7 @@ Main features
 **Speed**
   * performance-critical code (chunking, compression, encryption) is
     implemented in C/Cython
-  * local caching of files/chunks index data
+  * local caching
   * quick detection of unmodified files
 
 **Data encryption**
@@ -146,9 +148,9 @@ For ease of use, set the BORG_REPO environment variable::
 
     $ export BORG_REPO=/path/to/repo
 
-Create a new backup repository (see ``borg rcreate --help`` for encryption options)::
+Create a new backup repository (see ``borg repo-create --help`` for encryption options)::
 
-    $ borg rcreate -e repokey-aes-ocb
+    $ borg repo-create -e repokey-aes-ocb
 
 Create a new backup archive::
 

{borgbackup-2.0.0b9 → borgbackup-2.0.0b10}/README.rst

@@ -69,7 +69,7 @@ Main features
 **Speed**
   * performance-critical code (chunking, compression, encryption) is
     implemented in C/Cython
-  * local caching of files/chunks index data
+  * local caching
   * quick detection of unmodified files
 
 **Data encryption**
@@ -122,9 +122,9 @@ For ease of use, set the BORG_REPO environment variable::
 
     $ export BORG_REPO=/path/to/repo
 
-Create a new backup repository (see ``borg rcreate --help`` for encryption options)::
+Create a new backup repository (see ``borg repo-create --help`` for encryption options)::
 
-    $ borg rcreate -e repokey-aes-ocb
+    $ borg repo-create -e repokey-aes-ocb
 
 Create a new backup archive::
 

{borgbackup-2.0.0b9 → borgbackup-2.0.0b10}/docs/changes.rst

@@ -12,8 +12,8 @@ This section provides information about security and corruption issues.
 Upgrade Notes
 =============
 
-borg 1.2.x to borg 2.0
-----------------------
+borg 1.2.x/1.4.x to borg 2.0
+----------------------------
 
 Compatibility notes:
 
@@ -21,11 +21,11 @@ Compatibility notes:
 
   We tried to put all the necessary "breaking" changes into this release, so we
   hopefully do not need another breaking release in the near future. The changes
-  were necessary for improved security, improved speed, unblocking future
-  improvements, getting rid of legacy crap / design limitations, having less and
-  simpler code to maintain.
+  were necessary for improved security, improved speed and parallelism,
+  unblocking future improvements, getting rid of legacy crap and design
+  limitations, having less and simpler code to maintain.
 
-  You can use "borg transfer" to transfer archives from borg 1.1/1.2 repos to
+  You can use "borg transfer" to transfer archives from borg 1.2/1.4 repos to
   a new borg 2.0 repo, but it will need some time and space.
 
   Before using "borg transfer", you must have upgraded to borg >= 1.2.6 (or
@@ -58,10 +58,12 @@ Compatibility notes:
 
     - borg 2 repo commands:
 
-      - borg rcreate  # "repo create", was: borg init
-      - borg rlist  # "repo list"
-      - borg rinfo  # "repo info"
-      - borg rdelete  # "repo delete"
+      - borg repo-create  # was: borg init
+      - borg repo-list
+      - borg repo-info
+      - borg repo-delete
+      - borg repo-compress
+      - borg repo-space
     - borg 2 archive commands:
 
       - borg create ARCHIVE ...
@@ -84,6 +86,8 @@ Compatibility notes:
   - removed --nobsdflags (use --noflags)
   - removed --noatime (default now, see also --atime)
   - removed --save-space option (does not change behaviour)
+- removed --bypass-lock option
+- removed borg config command (only worked locally anyway)
 - using --list together with --progress is now disallowed (except with --log-json), #7219
 - the --glob-archives option was renamed to --match-archives (the short option
   name -a is unchanged) and extended to support different pattern styles:
@@ -114,14 +118,13 @@ Compatibility notes:
   fail now that somehow "worked" before (but maybe didn't work as intended due to
   the contradicting options).
 
-
 .. _changelog:
 
 Change Log 2.x
 ==============
 
-Version 2.0.0b9 (2024-07-20)
-----------------------------
+Version 2.0.0b10 (2024-09-09)
+-----------------------------
 
 Please note:
 
@@ -130,6 +133,60 @@ This is a beta release, only for testing - do not use for production repos.
 For upgrade and compatibility hints, please also read the section "Upgrade Notes"
 above.
 
+TL;DR: this is a huge change and the first very fundamental change in how borg
+works since ever:
+
+- you will need to create new repos.
+- likely more exciting than previous betas, definitely not for production.
+- remove "borg check --repair --archives-only" from your script in case you
+  added that for b9. "borg compact -v" now finds and deletes orphan chunks
+  and also outputs some new stats.
+
+New features:
+
+- borgstore based repository, file:, ssh: and sftp: for now, more possible.
+- repository stores objects separately now, not using segment files.
+  this has more fs overhead, but needs much less I/O because no segment
+  files compaction is required anymore. also, no repository index is
+  needed anymore because we can directly find the objects by their ID.
+- locking: new borgstore based repository locking with automatic stale
+  lock removal (if lock does not get refreshed, if lock owner process is dead).
+- simultaneous repository access for many borg commands except check/compact.
+  the cache lock for adhocwithfiles is still exclusive though, so use
+  BORG_CACHE_IMPL=adhoc if you want to try that out using only 1 machine
+  and 1 user (that implementation doesn't use a cache lock). When using
+  multiple client machines or users, it also works with the default cache.
+- delete/prune: much quicker now and can be undone.
+- check --repair --undelete-archives: bring archives back from the dead.
+- repo-space: manage reserved space in repository (avoid dead-end situation if
+  repository filesystem runs full).
+
+Bugs/issues fixed:
+
+- a lot! all linked from PR #8332.
+
+Other changes:
+
+- repository: remove transactions, solved differently and much simpler now
+  (convergence and write order primarily).
+- repository: replaced precise reference counting with "object exists in repo?"
+  and "garbage collection of unused objects".
+- cache: remove transactions, remove chunks cache.
+  removed LocalCache, BORG_CACHE_IMPL=local, solving all related issues.
+  as in beta 9, adhowwithfiles is the default implementation.
+- compact: needs the borg key now (run it clientside), -v gives nice stats.
+- transfer: archive transfers from borg 1.x need the --from-borg1 option
+- check: reimplemented / bigger changes.
+- code: got rid of a metric ton of not needed complexity.
+  when borg does not need to read borg 1.x repos/archives anymore, after
+  users have transferred their archives, even much more can be removed.
+- docs: updated / removed outdated stuff
+- renamed r* commands to repo-*
+
+
+Version 2.0.0b9 (2024-07-20)
+----------------------------
+
 New features:
 
 - add BORG_CACHE_IMPL, default is "adhocwithfiles" to test the new cache

{borgbackup-2.0.0b9 → borgbackup-2.0.0b10}/docs/changes_1.x.rst

@@ -3469,7 +3469,7 @@ Other changes:
   - archiver tests: add check_cache tool - lints refcounts
 
 - fixed cache sync performance regression from 1.1.0b1 onwards, #1940
-- syncing the cache without chunks.archive.d (see :ref:`disable_archive_chunks`)
+- syncing the cache without chunks.archive.d
   now avoids any merges and is thus faster, #1940
 - borg check --verify-data: faster due to linear on-disk-order scan
 - borg debug-xxx commands removed, we use "debug xxx" subcommands now, #1627

{borgbackup-2.0.0b9 → borgbackup-2.0.0b10}/docs/deployment/automated-local.rst

@@ -105,7 +105,7 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.).
     #
 
     # Options for borg create
-    BORG_OPTS="--stats --one-file-system --compression lz4 --checkpoint-interval 86400"
+    BORG_OPTS="--stats --one-file-system --compression lz4"
 
     # Set BORG_PASSPHRASE or BORG_PASSCOMMAND somewhere around here, using export,
     # if encryption is used.
@@ -150,7 +150,7 @@ modify it to suit your needs (e.g. more backup sets, dumping databases etc.).
     fi
 
 Create the ``/etc/backups/autoeject`` file to have the script automatically eject the drive
-after creating the backup. Rename the file to something else (e.g. ``/etc/backup/autoeject-no``)
+after creating the backup. Rename the file to something else (e.g. ``/etc/backups/autoeject-no``)
 when you want to do something with the drive after creating backups (e.g running check).
 
 Create the ``/etc/backups/backup-suspend`` file if the machine should suspend after completing
@@ -177,7 +177,7 @@ Find the UUID of the file system that backups should be stored on::
 
     lsblk -o+uuid,label
 
-Note the UUID into the ``/etc/backup/backup.disks`` file.
+Note the UUID into the ``/etc/backups/backup.disks`` file.
 
 Mount the drive to /mnt/backup.
 
@@ -198,7 +198,7 @@ Security considerations
 -----------------------
 
 The script as shown above will mount any file system with an UUID listed in
-``/etc/backup/backup.disks``. The UUID check is a safety / annoyance-reduction
+``/etc/backups/backup.disks``. The UUID check is a safety / annoyance-reduction
 mechanism to keep the script from blowing up whenever a random USB thumb drive is connected.
 It is not meant as a security mechanism. Mounting file systems and reading repository
 data exposes additional attack surfaces (kernel file system drivers,

{borgbackup-2.0.0b9 → borgbackup-2.0.0b10}/docs/deployment/hosting-repositories.rst

@@ -68,8 +68,6 @@ can be filled to the specified quota.
 If storage quotas are used, ensure that all deployed Borg releases
 support storage quotas.
 
-Refer to :ref:`internals_storage_quota` for more details on storage quotas.
-
 **Specificities: Append-only repositories**
 
 Running ``borg init`` via a ``borg serve --append-only`` server will **not**