scalio-backup 0.1.0__tar.gz

scalio_backup-0.1.0/PKG-INFO

@@ -0,0 +1,80 @@
+Metadata-Version: 2.4
+Name: scalio-backup
+Version: 0.1.0
+Summary: Encrypted export and import (backup, restore, move) shared by the Scalio apps
+License: Proprietary
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+Requires-Dist: cryptography>=42
+Requires-Dist: pymongo>=4.6
+Requires-Dist: packaging>=23
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: mongomock-motor>=0.0.29; extra == "dev"
+Requires-Dist: typing-extensions>=4.8; extra == "dev"
+Requires-Dist: pyrage>=1.1; extra == "dev"
+Requires-Dist: motor>=3.3; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+
+# scalio-backup
+
+Encrypted export and import of a whole Scalio app instance, shared by the Scalio apps (Nexus first, then
+Foundry, Rosetta, Citadel). Used for backup/restore and for moving an instance between self-hosted and SaaS.
+Concept and decisions E1-E12: https://claude.ai/artifact/Ej8DFrgi4hcFDQ69dvKWZ7
+
+Nothing in this package is secret. The password is the customer's; without it nobody (Scalio included) can
+open a bundle.
+
+## Bundle format (`.scaliobak`)
+
+- Outside: standard [age v1](https://age-encryption.org/v1) with a passphrase (scrypt, work factor 2^18),
+  ChaCha20-Poly1305 in 64 KiB chunks. `age -d backup.scaliobak > backup.tar` opens it without Scalio.
+- Inside: a tar stream. `manifest.json` first (format `scalio-backup/1`, product, app and schema version,
+  source, options, SHA-256 per part), then `collections/<name>.jsonl` (MongoDB Extended JSON, canonical, so
+  dates, Int64 and ObjectIds stay exact), `meta.jsonl`, and for file folders `files/<dir>.index.jsonl` plus the
+  files. The import checks every hash; a changed or incomplete bundle is refused before anything is written.
+
+## Classes
+
+Every collection (and every document of a mixed `meta` collection) has one class: `config` (always exported),
+`data` (exported unless "configuration only"), `cache`, `ephemeral`, `instance` (never exported). Each app keeps
+its classes in its own `backup_spec.py` and a test that fails on an unclassified collection.
+
+## Use in an app
+
+```python
+from scalio_backup import AppSpec, Preserve, apply_import, export_instance, open_bundle, recover, rollback, purge
+
+SPEC = AppSpec(product="scalio-nexus", app_version=VERSION, collections=..., meta_ids=...,
+               secret_fields={"settings": ("entra.clientSecret", ...)}, file_dirs={"hr": HR_UPLOAD_DIR},
+               schema_version=current_schema, export_hook=decrypt_instance_secrets, import_hook=encrypt_again)
+
+# export (background job): writes the encrypted bundle to a file object
+manifest = await export_instance(db, SPEC, out, password, include_data=True, include_secrets=True,
+                                 created_by=user_email, source={"mode": "selfhosted", "host": host, "tenant": tid})
+
+# import, step 1: decrypt and verify into a work folder, nothing changes yet (WrongPassword, BundleError,
+# IncompatibleBundle when the product differs or the bundle is newer than this instance)
+bundle = open_bundle(upload, password, SPEC, target_schema=await current_schema(db))
+show(bundle.preview())          # source, counts, warnings; the app adds its own (other tenant, license limits)
+
+# import, step 2: inside the app's maintenance window (no writes, background jobs paused)
+await apply_import(db, SPEC, bundle, preserve=(Preserve("app_users", {"email": me}),), imported_by=me)
+bundle.cleanup()
+# then: migrations, end all sessions, re-sync caches, show the checklist
+
+await rollback(db, SPEC)        # back to the state before the import (until purge)
+await recover(db, SPEC)         # on startup: cleans up an import that stopped half way
+await purge(db, SPEC)           # daily: drops the rollback copy after 7 days
+```
+
+How the swap works: documents go into `_import_<name>`, then each collection is renamed to
+`_preimport_<name>` and the staging copy into its place. A journal (`_backup_journal`) records each step, so an
+interrupted import is rolled back on the next start. File folders keep their old content in `.preimport/`.
+
+## Development
+
+`pip install -e ".[dev]"`, then `pytest -q` and `ruff check .`. Engine tests use mongomock; with `MONGO_URL` set
+they run against a real MongoDB (CI does both). `tests/vector.scaliobak` is the shared test bundle
+(`tests/make_vector.py`); the age layer is checked against rage (`pyrage`) in both directions.

scalio_backup-0.1.0/README.md

@@ -0,0 +1,61 @@
+# scalio-backup
+
+Encrypted export and import of a whole Scalio app instance, shared by the Scalio apps (Nexus first, then
+Foundry, Rosetta, Citadel). Used for backup/restore and for moving an instance between self-hosted and SaaS.
+Concept and decisions E1-E12: https://claude.ai/artifact/Ej8DFrgi4hcFDQ69dvKWZ7
+
+Nothing in this package is secret. The password is the customer's; without it nobody (Scalio included) can
+open a bundle.
+
+## Bundle format (`.scaliobak`)
+
+- Outside: standard [age v1](https://age-encryption.org/v1) with a passphrase (scrypt, work factor 2^18),
+  ChaCha20-Poly1305 in 64 KiB chunks. `age -d backup.scaliobak > backup.tar` opens it without Scalio.
+- Inside: a tar stream. `manifest.json` first (format `scalio-backup/1`, product, app and schema version,
+  source, options, SHA-256 per part), then `collections/<name>.jsonl` (MongoDB Extended JSON, canonical, so
+  dates, Int64 and ObjectIds stay exact), `meta.jsonl`, and for file folders `files/<dir>.index.jsonl` plus the
+  files. The import checks every hash; a changed or incomplete bundle is refused before anything is written.
+
+## Classes
+
+Every collection (and every document of a mixed `meta` collection) has one class: `config` (always exported),
+`data` (exported unless "configuration only"), `cache`, `ephemeral`, `instance` (never exported). Each app keeps
+its classes in its own `backup_spec.py` and a test that fails on an unclassified collection.
+
+## Use in an app
+
+```python
+from scalio_backup import AppSpec, Preserve, apply_import, export_instance, open_bundle, recover, rollback, purge
+
+SPEC = AppSpec(product="scalio-nexus", app_version=VERSION, collections=..., meta_ids=...,
+               secret_fields={"settings": ("entra.clientSecret", ...)}, file_dirs={"hr": HR_UPLOAD_DIR},
+               schema_version=current_schema, export_hook=decrypt_instance_secrets, import_hook=encrypt_again)
+
+# export (background job): writes the encrypted bundle to a file object
+manifest = await export_instance(db, SPEC, out, password, include_data=True, include_secrets=True,
+                                 created_by=user_email, source={"mode": "selfhosted", "host": host, "tenant": tid})
+
+# import, step 1: decrypt and verify into a work folder, nothing changes yet (WrongPassword, BundleError,
+# IncompatibleBundle when the product differs or the bundle is newer than this instance)
+bundle = open_bundle(upload, password, SPEC, target_schema=await current_schema(db))
+show(bundle.preview())          # source, counts, warnings; the app adds its own (other tenant, license limits)
+
+# import, step 2: inside the app's maintenance window (no writes, background jobs paused)
+await apply_import(db, SPEC, bundle, preserve=(Preserve("app_users", {"email": me}),), imported_by=me)
+bundle.cleanup()
+# then: migrations, end all sessions, re-sync caches, show the checklist
+
+await rollback(db, SPEC)        # back to the state before the import (until purge)
+await recover(db, SPEC)         # on startup: cleans up an import that stopped half way
+await purge(db, SPEC)           # daily: drops the rollback copy after 7 days
+```
+
+How the swap works: documents go into `_import_<name>`, then each collection is renamed to
+`_preimport_<name>` and the staging copy into its place. A journal (`_backup_journal`) records each step, so an
+interrupted import is rolled back on the next start. File folders keep their old content in `.preimport/`.
+
+## Development
+
+`pip install -e ".[dev]"`, then `pytest -q` and `ruff check .`. Engine tests use mongomock; with `MONGO_URL` set
+they run against a real MongoDB (CI does both). `tests/vector.scaliobak` is the shared test bundle
+(`tests/make_vector.py`); the age layer is checked against rage (`pyrage`) in both directions.

scalio_backup-0.1.0/pyproject.toml

@@ -0,0 +1,29 @@
+[build-system]
+requires = ["setuptools>=69"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "scalio-backup"
+version = "0.1.0"
+description = "Encrypted export and import (backup, restore, move) shared by the Scalio apps"
+readme = "README.md"
+requires-python = ">=3.11"
+license = { text = "Proprietary" }
+dependencies = ["cryptography>=42", "pymongo>=4.6", "packaging>=23"]
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "pytest-asyncio>=0.23", "mongomock-motor>=0.0.29", "typing-extensions>=4.8", "pyrage>=1.1", "motor>=3.3", "ruff>=0.6"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B"]

scalio_backup-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

scalio_backup-0.1.0/src/scalio_backup/__init__.py

@@ -0,0 +1,24 @@
+"""Encrypted export and import of a Scalio app instance (backup, restore, self-hosted <-> SaaS move)."""
+from .age import AgeError, WrongPassword
+from .engine import (
+    Bundle,
+    BundleError,
+    IncompatibleBundle,
+    apply_import,
+    blank_paths,
+    check_compatible,
+    check_password,
+    export_instance,
+    open_bundle,
+    purge,
+    recover,
+    rollback,
+    status,
+)
+from .spec import CACHE, CLASSES, CONFIG, DATA, EPHEMERAL, INSTANCE, AppSpec, Preserve
+
+__version__ = "0.1.0"
+__all__ = ["AgeError", "AppSpec", "Bundle", "BundleError", "CACHE", "CLASSES", "CONFIG", "DATA", "EPHEMERAL",
+           "INSTANCE", "IncompatibleBundle", "Preserve", "WrongPassword", "apply_import", "blank_paths",
+           "check_compatible", "check_password", "export_instance", "open_bundle", "purge", "recover",
+           "rollback", "status"]

scalio_backup-0.1.0/src/scalio_backup/age.py

@@ -0,0 +1,237 @@
+"""age v1 file format with a passphrase (scrypt recipient), streaming in both directions.
+
+Spec: https://age-encryption.org/v1 . Files written here open with the `age` tool (`age -d file`) and files
+from `age -p` open here, so a customer can always reach the data without Scalio. Only the scrypt
+recipient is supported, and it must be the only stanza (as the spec requires).
+"""
+import base64
+import hashlib
+import hmac
+import io
+import os
+
+from cryptography.exceptions import InvalidTag
+from cryptography.hazmat.primitives import hashes
+from cryptography.hazmat.primitives.ciphers.aead import ChaCha20Poly1305
+from cryptography.hazmat.primitives.kdf.hkdf import HKDF
+from cryptography.hazmat.primitives.kdf.scrypt import Scrypt
+
+INTRO = b"age-encryption.org/v1\n"
+SCRYPT_LABEL = b"age-encryption.org/v1/scrypt"
+CHUNK = 64 * 1024
+TAG = 16
+DEFAULT_LOG_N = 18          # same work factor as the age tool on a current machine (256 MiB, about 1 s)
+MAX_LOG_N = 20              # refuse files that would make us spend more memory than that
+
+
+class AgeError(Exception):
+    """The file is not a valid age file, or it was changed."""
+
+
+class WrongPassword(AgeError):
+    """The passphrase does not open this file."""
+
+
+def _b64(data: bytes) -> bytes:
+    return base64.b64encode(data).rstrip(b"=")
+
+
+def _unb64(data: bytes) -> bytes:
+    if b"=" in data or b"\r" in data:
+        raise AgeError("bad base64 in header")
+    try:
+        out = base64.b64decode(data + b"=" * (-len(data) % 4), validate=True)
+    except ValueError as e:
+        raise AgeError("bad base64 in header") from e
+    if _b64(out) != data:
+        raise AgeError("non-canonical base64 in header")
+    return out
+
+
+def _hkdf(ikm: bytes, salt: bytes, info: bytes) -> bytes:
+    return HKDF(algorithm=hashes.SHA256(), length=32, salt=salt, info=info).derive(ikm)
+
+
+def _scrypt(password: bytes, salt: bytes, log_n: int) -> bytes:
+    return Scrypt(salt=SCRYPT_LABEL + salt, length=32, n=1 << log_n, r=8, p=1).derive(password)
+
+
+def _nonce(counter: int, last: bool) -> bytes:
+    if counter >= 1 << 88:
+        raise AgeError("file too large")
+    return counter.to_bytes(11, "big") + (b"\x01" if last else b"\x00")
+
+
+def _pw(password) -> bytes:
+    if isinstance(password, str):
+        password = password.encode("utf-8")
+    if not password:
+        raise ValueError("empty passphrase")
+    return password
+
+
+class EncryptWriter(io.RawIOBase):
+    """File-like object: write() plaintext, close() finishes the file. Writes the age file to `out`."""
+
+    def __init__(self, out, password, log_n: int = DEFAULT_LOG_N):
+        super().__init__()
+        self._out = out
+        file_key = os.urandom(16)
+        salt = os.urandom(16)
+        wrap = ChaCha20Poly1305(_scrypt(_pw(password), salt, log_n)).encrypt(b"\x00" * 12, file_key, None)
+        header = INTRO + b"-> scrypt " + _b64(salt) + b" " + str(log_n).encode() + b"\n" + _b64(wrap) + b"\n---"
+        mac = hmac.new(_hkdf(file_key, b"", b"header"), header, hashlib.sha256).digest()
+        out.write(header + b" " + _b64(mac) + b"\n")
+        nonce = os.urandom(16)
+        out.write(nonce)
+        self._aead = ChaCha20Poly1305(_hkdf(file_key, nonce, b"payload"))
+        self._buf = bytearray()
+        self._counter = 0
+        self._done = False
+
+    def writable(self):
+        return True
+
+    def write(self, data) -> int:
+        if self._done:
+            raise ValueError("write to closed age writer")
+        self._buf += data
+        # keep at least one byte back: the last chunk is only known on close
+        while len(self._buf) > CHUNK:
+            self._emit(bytes(self._buf[:CHUNK]), last=False)
+            del self._buf[:CHUNK]
+        return len(data)
+
+    def _emit(self, chunk: bytes, last: bool):
+        self._out.write(self._aead.encrypt(_nonce(self._counter, last), chunk, None))
+        self._counter += 1
+
+    def close(self):
+        if not self._done:
+            self._emit(bytes(self._buf), last=True)
+            self._buf.clear()
+            self._done = True
+        super().close()
+
+
+def _read_line(src, limit: int = 4096) -> bytes:
+    line = bytearray()
+    while True:
+        c = src.read(1)
+        if not c:
+            raise AgeError("unexpected end of header")
+        line += c
+        if c == b"\n":
+            return bytes(line)
+        if len(line) > limit:
+            raise AgeError("header line too long")
+
+
+class DecryptReader(io.RawIOBase):
+    """File-like object over an age file: read() returns verified plaintext. A changed chunk raises AgeError
+    at the read that reaches it; nothing unverified is ever returned."""
+
+    def __init__(self, src, password, max_log_n: int = MAX_LOG_N):
+        super().__init__()
+        self._src = src
+        if _read_line(src) != INTRO:
+            raise AgeError("not an age v1 file")
+        header = bytearray(INTRO)
+        stanzas = []
+        while True:
+            line = _read_line(src)
+            if line.startswith(b"---"):
+                break
+            if not line.startswith(b"-> "):
+                raise AgeError("bad header")
+            header += line
+            args = line[3:-1].split(b" ")
+            body = bytearray()
+            while True:
+                bl = _read_line(src)
+                header += bl
+                chunk = bl[:-1]
+                if len(chunk) > 64:
+                    raise AgeError("bad stanza body")
+                body += chunk
+                if len(chunk) < 64:
+                    break
+            stanzas.append((args, _unb64(bytes(body))))
+        if not line.startswith(b"--- ") or not line.endswith(b"\n"):
+            raise AgeError("bad header end")
+        header += b"---"
+        mac = _unb64(line[4:-1])
+        if len(stanzas) != 1 or stanzas[0][0][:1] != [b"scrypt"]:
+            raise AgeError("this file is not protected by a passphrase only")
+        args, body = stanzas[0]
+        if len(args) != 3:
+            raise AgeError("bad scrypt stanza")
+        salt = _unb64(args[1])
+        if len(salt) != 16 or not args[2].isdigit() or args[2].startswith(b"0") or len(body) != 32:
+            raise AgeError("bad scrypt stanza")
+        log_n = int(args[2])
+        if log_n > max_log_n:
+            raise AgeError(f"work factor 2^{log_n} is above the allowed 2^{max_log_n}")
+        try:
+            file_key = ChaCha20Poly1305(_scrypt(_pw(password), salt, log_n)).decrypt(b"\x00" * 12, body, None)
+        except InvalidTag as e:
+            raise WrongPassword("wrong password") from e
+        if not hmac.compare_digest(hmac.new(_hkdf(file_key, b"", b"header"), bytes(header), hashlib.sha256).digest(),
+                                   mac):
+            raise AgeError("header was changed")
+        nonce = src.read(16)
+        if len(nonce) != 16:
+            raise AgeError("file is truncated")
+        self._aead = ChaCha20Poly1305(_hkdf(file_key, nonce, b"payload"))
+        self._counter = 0
+        self._plain = b""
+        self._pos = 0
+        self._eof = False
+        self._next = src.read(CHUNK + TAG)  # one chunk read ahead to know which is the last
+
+    def readable(self):
+        return True
+
+    def _decrypt_next(self):
+        cur = self._next
+        if not cur:
+            raise AgeError("file is truncated")
+        nxt = self._src.read(CHUNK + TAG) if len(cur) == CHUNK + TAG else b""
+        last = not nxt
+        try:
+            plain = self._aead.decrypt(_nonce(self._counter, last), cur, None)
+        except InvalidTag as e:
+            raise AgeError("file was changed or is truncated") from e
+        if last and not plain and self._counter > 0:
+            raise AgeError("empty last chunk")
+        self._counter += 1
+        self._next = nxt
+        self._eof = last
+        self._plain, self._pos = plain, 0
+
+    def readinto(self, b) -> int:
+        while self._pos >= len(self._plain):
+            if self._eof:
+                return 0
+            self._decrypt_next()
+        n = min(len(b), len(self._plain) - self._pos)
+        b[:n] = self._plain[self._pos:self._pos + n]
+        self._pos += n
+        return n
+
+    def read_all_and_verify(self):
+        """Read to the end; raises AgeError if anything was changed."""
+        while self.read(CHUNK):
+            pass
+
+
+def encrypt_bytes(data: bytes, password, log_n: int = DEFAULT_LOG_N) -> bytes:
+    out = io.BytesIO()
+    w = EncryptWriter(out, password, log_n)
+    w.write(data)
+    w.close()
+    return out.getvalue()
+
+
+def decrypt_bytes(data: bytes, password, max_log_n: int = MAX_LOG_N) -> bytes:
+    return DecryptReader(io.BytesIO(data), password, max_log_n).read()