borgstore 0.0.1__tar.gz

borgstore-0.0.1/CHANGES.rst

@@ -0,0 +1,7 @@
+ChangeLog
+=========
+
+Version 0.0.1 2024-08-23
+------------------------
+
+First PyPi release.

borgstore-0.0.1/LICENSE.rst

@@ -0,0 +1,28 @@
+Copyright (C) 2024 Thomas Waldmann <tw@waldmann-edv.de>
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+ 1. Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+ 2. Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+ 3. The name of the author may not be used to endorse or promote
+    products derived from this software without specific prior
+    written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

borgstore-0.0.1/PKG-INFO

@@ -0,0 +1,184 @@
+Metadata-Version: 2.1
+Name: borgstore
+Version: 0.0.1
+Summary: key/value store
+Author-email: Thomas Waldmann <tw@waldmann-edv.de>
+License: BSD
+Project-URL: Homepage, https://github.com/borgbackup/borgstore
+Keywords: kv,key/value,store
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Operating System :: POSIX
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/x-rst
+License-File: LICENSE.rst
+Requires-Dist: cryptography<43.0.0
+Requires-Dist: paramiko
+
+BorgStore
+=========
+
+A key/value store implementation in Python, supporting multiple backends,
+data redundancy and distribution.
+
+Keys
+----
+
+A key (str) can look like:
+
+- 0123456789abcdef...  (usually a long, hex-encoded hash value)
+- Any other pure ASCII string without "/" or ".." or " ".
+
+
+Namespaces
+----------
+
+To keep stuff apart, keys should get prefixed with a namespace, like:
+
+- config/settings
+- meta/0123456789abcdef...
+- data/0123456789abcdef...
+
+Please note:
+
+1. you should always use namespaces.
+2. nested namespaces like namespace1/namespace2/key are not supported.
+3. the code could work without a namespace (namespace ""), but then you
+   can't add another namespace later, because then you would have created
+   nested namespaces.
+
+Values
+------
+
+Values can be any arbitrary binary data (bytes).
+
+Store Operations
+----------------
+
+The high-level Store API implementation transparently deals with nesting and
+soft deletion, so the caller doesn't have to care much for that and the Backend
+API can be much simpler:
+
+- create/destroy: initialize or remove the whole store.
+- list: flat list of the items in the given namespace, with or without soft
+  deleted items.
+- store: write a new item into the store (giving its key/value pair)
+- load: read a value from the store (giving its key), partial loads giving
+  offset and/or size are supported.
+- info: get information about an item via its key (exists? size? ...)
+- delete: immediately remove an item from the store (giving its key)
+- move: implements rename, soft delete / undelete, move to current
+  nesting level
+
+Automatic Nesting
+-----------------
+
+For the Store user, items have names like e.g.:
+
+namespace/0123456789abcdef...
+namespace/abcdef0123456789...
+
+If there are very many items in the namespace, this could lead to scalability
+issues in the backend, thus the Store implementation offers transparent
+nesting, so that internally the Backend API will be called with
+names like e.g.:
+
+namespace/01/23/56/0123456789abcdef...
+namespace/ab/cd/ef/abcdef0123456789...
+
+The nesting depth can be configured from 0 (= no nesting) to N levels and
+there can be different nesting configurations depending on the namespace.
+
+The Store supports operating at different nesting levels in the same
+namespace at the same time.
+
+Soft deletion
+-------------
+
+To soft delete an item (so its value could be still read or it could be
+undeleted), the store just renames the item, appending ".del" to its name.
+
+Undelete reverses this by removing the ".del" suffix from the name.
+
+Some store operations have a boolean flag "deleted" to choose whether they
+shall consider soft deleted items.
+
+Backends
+--------
+
+The backend API is rather simple, one only needs to provide some very
+basic operations.
+
+Currently, these storage backends are implemented:
+
+- POSIX filesystems (namespaces: directories, values: in key-named files)
+- SFTP (access a server via sftp, namespaces: directories, values: in key-named files)
+- (more might come in future)
+
+MStore
+------
+
+API of MStore is very similar to Store, but instead of directly using one backend
+only (like Store does), it uses multiple Stores internally to implement:
+
+- redundancy (keep same data at multiple places)
+- distribution (keep different data at multiple places)
+
+Scalability
+-----------
+
+- Count of key/value pairs stored in a namespace: automatic nesting is
+  provided for keys to address common scalability issues.
+- Key size: there are no special provisions for extremely long keys (like:
+  more than backend limitations). Usually this is not a problem though.
+- Value size: there are no special provisions for dealing with large value
+  sizes (like: more than free memory, more than backend storage limitations,
+  etc.). If one deals with very large values, one usually cuts them into
+  chunks before storing them into the store.
+- Partial loads improve performance by avoiding a full load if only a part
+  of the value is needed (e.g. a header with metadata).
+
+Want a demo?
+------------
+
+Run this to get instructions how to run the demo:
+
+python3 -m borgstore
+
+State of this project
+---------------------
+
+**API is still unstable and expected to change as development goes on.**
+
+**There will be no data migration tools involving development/testing releases,
+like e.g. upgrading a store from alpha1 to alpha2 or beta13 to release.**
+
+There are tests and they succeed for the basic functionality, so some of the
+stuff is already working well.
+
+There might be missing features or optimization potential, feedback welcome!
+
+There are a lot of possible, but still missing backends (like e.g. for cloud
+storage). If you want to create and support one: pull requests are welcome.
+
+Borg?
+-----
+
+Please note that this code is developed by the Borg Collective and hosted
+by the BorgBackup github organisation, but is currently **not** used by
+the BorgBackup (aka "borg") backup tool stable release.
+
+License
+-------
+
+BSD license.
+

borgstore-0.0.1/README.rst

@@ -0,0 +1,158 @@
+BorgStore
+=========
+
+A key/value store implementation in Python, supporting multiple backends,
+data redundancy and distribution.
+
+Keys
+----
+
+A key (str) can look like:
+
+- 0123456789abcdef...  (usually a long, hex-encoded hash value)
+- Any other pure ASCII string without "/" or ".." or " ".
+
+
+Namespaces
+----------
+
+To keep stuff apart, keys should get prefixed with a namespace, like:
+
+- config/settings
+- meta/0123456789abcdef...
+- data/0123456789abcdef...
+
+Please note:
+
+1. you should always use namespaces.
+2. nested namespaces like namespace1/namespace2/key are not supported.
+3. the code could work without a namespace (namespace ""), but then you
+   can't add another namespace later, because then you would have created
+   nested namespaces.
+
+Values
+------
+
+Values can be any arbitrary binary data (bytes).
+
+Store Operations
+----------------
+
+The high-level Store API implementation transparently deals with nesting and
+soft deletion, so the caller doesn't have to care much for that and the Backend
+API can be much simpler:
+
+- create/destroy: initialize or remove the whole store.
+- list: flat list of the items in the given namespace, with or without soft
+  deleted items.
+- store: write a new item into the store (giving its key/value pair)
+- load: read a value from the store (giving its key), partial loads giving
+  offset and/or size are supported.
+- info: get information about an item via its key (exists? size? ...)
+- delete: immediately remove an item from the store (giving its key)
+- move: implements rename, soft delete / undelete, move to current
+  nesting level
+
+Automatic Nesting
+-----------------
+
+For the Store user, items have names like e.g.:
+
+namespace/0123456789abcdef...
+namespace/abcdef0123456789...
+
+If there are very many items in the namespace, this could lead to scalability
+issues in the backend, thus the Store implementation offers transparent
+nesting, so that internally the Backend API will be called with
+names like e.g.:
+
+namespace/01/23/56/0123456789abcdef...
+namespace/ab/cd/ef/abcdef0123456789...
+
+The nesting depth can be configured from 0 (= no nesting) to N levels and
+there can be different nesting configurations depending on the namespace.
+
+The Store supports operating at different nesting levels in the same
+namespace at the same time.
+
+Soft deletion
+-------------
+
+To soft delete an item (so its value could be still read or it could be
+undeleted), the store just renames the item, appending ".del" to its name.
+
+Undelete reverses this by removing the ".del" suffix from the name.
+
+Some store operations have a boolean flag "deleted" to choose whether they
+shall consider soft deleted items.
+
+Backends
+--------
+
+The backend API is rather simple, one only needs to provide some very
+basic operations.
+
+Currently, these storage backends are implemented:
+
+- POSIX filesystems (namespaces: directories, values: in key-named files)
+- SFTP (access a server via sftp, namespaces: directories, values: in key-named files)
+- (more might come in future)
+
+MStore
+------
+
+API of MStore is very similar to Store, but instead of directly using one backend
+only (like Store does), it uses multiple Stores internally to implement:
+
+- redundancy (keep same data at multiple places)
+- distribution (keep different data at multiple places)
+
+Scalability
+-----------
+
+- Count of key/value pairs stored in a namespace: automatic nesting is
+  provided for keys to address common scalability issues.
+- Key size: there are no special provisions for extremely long keys (like:
+  more than backend limitations). Usually this is not a problem though.
+- Value size: there are no special provisions for dealing with large value
+  sizes (like: more than free memory, more than backend storage limitations,
+  etc.). If one deals with very large values, one usually cuts them into
+  chunks before storing them into the store.
+- Partial loads improve performance by avoiding a full load if only a part
+  of the value is needed (e.g. a header with metadata).
+
+Want a demo?
+------------
+
+Run this to get instructions how to run the demo:
+
+python3 -m borgstore
+
+State of this project
+---------------------
+
+**API is still unstable and expected to change as development goes on.**
+
+**There will be no data migration tools involving development/testing releases,
+like e.g. upgrading a store from alpha1 to alpha2 or beta13 to release.**
+
+There are tests and they succeed for the basic functionality, so some of the
+stuff is already working well.
+
+There might be missing features or optimization potential, feedback welcome!
+
+There are a lot of possible, but still missing backends (like e.g. for cloud
+storage). If you want to create and support one: pull requests are welcome.
+
+Borg?
+-----
+
+Please note that this code is developed by the Borg Collective and hosted
+by the BorgBackup github organisation, but is currently **not** used by
+the BorgBackup (aka "borg") backup tool stable release.
+
+License
+-------
+
+BSD license.
+

borgstore-0.0.1/pyproject.toml

@@ -0,0 +1,62 @@
+[project]
+name = "borgstore"
+dynamic = ["version"]
+authors = [{name="Thomas Waldmann", email="tw@waldmann-edv.de"}, ]
+description = "key/value store"
+readme = "README.rst"
+keywords = ["kv", "key/value", "store"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: BSD License",
+    "Operating System :: POSIX",
+    "Programming Language :: Python",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Libraries",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+license = {text="BSD"}
+requires-python = ">=3.9"
+dependencies = [
+    "cryptography < 43.0.0",  # using a more recent version triggers annoying warnings with paramiko
+    "paramiko",
+]
+
+[project.urls]
+Homepage = "https://github.com/borgbackup/borgstore"
+
+[build-system]
+requires = ["setuptools", "setuptools_scm[toml]>=6.2"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools_scm]
+# make sure we have the same versioning scheme with all setuptools_scm versions, to avoid different autogenerated files
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1015052
+# https://github.com/borgbackup/borg/issues/6875
+write_to = "src/borgstore/_version.py"
+write_to_template = "__version__ = version = {version!r}\n"
+
+[tool.black]
+line-length = 120
+skip-magic-trailing-comma = true
+
+[tool.pytest.ini_options]
+minversion = "6.0"
+testpaths = ["tests"]
+
+[tool.flake8]
+# Ignoring E203 due to https://github.com/PyCQA/pycodestyle/issues/373
+ignore = ['E226', 'W503', 'E203']
+max_line_length = 120
+exclude = ['build', 'dist', '.git', '.idea', '.mypy_cache', '.tox']
+
+[tool.mypy]
+python_version = '3.10'
+strict_optional = false
+local_partial_types = true
+show_error_codes = true
+files = 'src/borgstore/**/*.py'

borgstore-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

borgstore-0.0.1/src/borgstore/__init__.py

@@ -0,0 +1,5 @@
+"""
+BorgStore - a key/value store.
+"""
+
+from ._version import __version__, version  # noqa

borgstore-0.0.1/src/borgstore/__main__.py

@@ -0,0 +1,71 @@
+"""
+Demo for BorgStore
+==================
+
+Usage:  python -m borgstore <borgstore_storage_url>
+
+E.g.:   python -m borgstore file:///tmp/borgstore_storage
+
+Please be careful: the given storage will be created, used and **completely deleted**!
+"""
+
+
+def run_demo(storage_url):
+    from .store import Store
+
+    def id_key(data: bytes):
+        from hashlib import new
+
+        h = new("sha256", data)
+        return f"data/{h.hexdigest()}"
+
+    levels_config = {
+        "config/": [0],  # no nesting needed/wanted for the configs
+        "data/": [2],  # 2 nesting levels wanted for the data
+    }
+    store = Store(url=storage_url, levels=levels_config)
+    try:
+        store.create()
+    except FileExistsError:
+        # currently, we only have file:// storages, so this should be fine.
+        print("Error: you must not give an existing directory.")
+        return
+
+    print("Writing 2 items to config namespace...")
+    settings1_key = "config/settings1"
+    store.store(settings1_key, b"value1 = 42")
+    settings2_key = "config/settings2"
+    store.store(settings2_key, b"value2 = 23")
+
+    print(f"Listing config namespace contents: {list(store.list('config'))}")
+
+    settings1_value = store.load(settings1_key)
+    print(f"Loaded from store: {settings1_key}: {settings1_value.decode()}")
+    settings2_value = store.load(settings2_key)
+    print(f"Loaded from store: {settings2_key}: {settings2_value.decode()}")
+
+    print("Writing 2 items to data namespace...")
+    data1 = b"some arbitrary binary data."
+    key1 = id_key(data1)
+    store.store(key1, data1)
+    data2 = b"more arbitrary binary data. " * 2
+    key2 = id_key(data2)
+    store.store(key2, data2)
+    print(f"Soft deleting item {key2} ...")
+    store.move(key2, delete=True)
+
+    print(f"Listing data namespace contents: {list(store.list('data', deleted=False))}")
+    print(f"Listing data namespace contents, incl. deleted: {list(store.list('data', deleted=True))}")
+
+    answer = input("After you've inspected the storage, enter DESTROY to destroy the storage, anything else to abort: ")
+    if answer == "DESTROY":
+        store.destroy()
+
+
+if __name__ == "__main__":
+    import sys
+
+    if len(sys.argv) == 2:
+        run_demo(sys.argv[1])
+    else:
+        print(__doc__)

borgstore-0.0.1/src/borgstore/_version.py

@@ -0,0 +1 @@
+__version__ = version = '0.0.1'

borgstore-0.0.1/src/borgstore/backends/__init__.py

@@ -0,0 +1,3 @@
+"""
+Package with misc. backend implementations. See backends._base for details.
+"""

borgstore-0.0.1/src/borgstore/backends/_base.py

@@ -0,0 +1,104 @@
+"""
+Base class and type definitions for all backend implementations in this package.
+
+Docs that are not backend-specific are also found here.
+"""
+
+from abc import ABC, abstractmethod
+from collections import namedtuple
+from typing import Iterator
+
+from ..constants import MAX_NAME_LENGTH
+
+ItemInfo = namedtuple("ItemInfo", "name exists size directory")
+
+
+def validate_name(name):
+    """validate a backend key / name"""
+    if not isinstance(name, str):
+        raise TypeError(f"name must be str, but got: {type(name)}")
+    # name must not be too long
+    if len(name) > MAX_NAME_LENGTH:
+        raise ValueError(f"name is too long (max: {MAX_NAME_LENGTH}): {name}")
+    # avoid encoding issues
+    try:
+        name.encode("ascii")
+    except UnicodeEncodeError:
+        raise ValueError(f"name must encode to plain ascii, but failed with: {name}")
+    # security: name must be relative - can be foo or foo/bar/baz, but must never be /foo or ../foo
+    if name.startswith("/") or name.endswith("/") or ".." in name:
+        raise ValueError(f"name must be relative and not contain '..': {name}")
+    # names used here always have '/' as separator, never '\' -
+    # this is to avoid confusion in case this is ported to e.g. Windows.
+    # also: no blanks - simplifies usage via CLI / shell.
+    if "\\" in name or " " in name:
+        raise ValueError(f"name must not contain backslashes or blanks: {name}")
+    # name must be lowercase - this is to avoid troubles in case this is ported to a non-case-sensitive backend.
+    # also, guess we want to avoid that a key "config" would address a different item than a key "CONFIG" or
+    # a key "1234CAFE5678BABE" would address a different item than a key "1234cafe5678babe".
+    if name != name.lower():
+        raise ValueError(f"name must be lowercase, but got: {name}")
+
+
+class BackendBase(ABC):
+    @abstractmethod
+    def create(self):
+        """create (initialize) a backend storage"""
+
+    @abstractmethod
+    def destroy(self):
+        """completely remove the backend storage (and its contents)"""
+
+    def __enter__(self):
+        self.open()
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        self.close()
+        return False
+
+    @abstractmethod
+    def open(self):
+        """open (start using) a backend storage"""
+
+    @abstractmethod
+    def close(self):
+        """close (stop using) a backend storage"""
+
+    @abstractmethod
+    def mkdir(self, name: str) -> None:
+        """create directory/namespace <name>"""
+
+    @abstractmethod
+    def rmdir(self, name: str) -> None:
+        """remove directory/namespace <name>"""
+
+    @abstractmethod
+    def info(self, name) -> ItemInfo:
+        """return information about <name>"""
+
+    @abstractmethod
+    def load(self, name: str, *, size=None, offset=0) -> bytes:
+        """load value from <name>"""
+
+    @abstractmethod
+    def store(self, name: str, value: bytes) -> None:
+        """store <value> into <name>"""
+
+    @abstractmethod
+    def delete(self, name: str) -> None:
+        """delete <name>"""
+
+    @abstractmethod
+    def move(self, curr_name: str, new_name: str) -> None:
+        """rename curr_name to new_name (overwrite target)"""
+
+    @abstractmethod
+    def list(self, name: str) -> Iterator[ItemInfo]:
+        """list the contents of <name>, non-recursively.
+
+        Does not yield TMP_SUFFIX items - usually they are either not finished
+        uploading or they are leftover crap from aborted uploads.
+
+        The yielded ItemInfos are sorted alphabetically by name.
+        """

borgstore-0.0.1/src/borgstore/backends/errors.py

@@ -0,0 +1,27 @@
+"""
+Generic exception classes used by all backends.
+"""
+
+
+class BackendError(Exception):
+    """Base class for exceptions in this module."""
+
+
+class BackendAlreadyExists(BackendError):
+    """Raised when a backend already exists."""
+
+
+class BackendDoesNotExist(BackendError):
+    """Raised when a backend does not exist."""
+
+
+class BackendMustNotBeOpen(BackendError):
+    """Backend must not be open."""
+
+
+class BackendMustBeOpen(BackendError):
+    """Backend must be open."""
+
+
+class ObjectNotFound(BackendError):
+    """Object not found."""