hbkit 0.4.5__tar.gz → 0.4.6__tar.gz

{hbkit-0.4.5 → hbkit-0.4.6}/FORMAT.md

@@ -16,8 +16,9 @@ in file-signature databases, or in DFIR tooling.
 
 Coverage caveat: this was derived from **one** archive — DSM 7, Hyper Backup 4.1.2-4039,
 unencrypted, LZ4, single version, single pool, plus a second small **encrypted** archive
-from the same DSM version. Variant handling marked [D] is implemented from disassembly but
-has never met a real archive of that kind.
+from the same DSM version. A user's 7-version encrypted archive (Hyper Backup 4.2.2,
+issue #4) later corrected §7.2. Variant handling marked [D] is implemented from
+disassembly but has never met a real archive of that kind.
 
 ---
 
@@ -254,7 +255,7 @@ Extra files in an encrypted archive:
 |---|---|
 | `Config/public.pem[.n]` | raw 32-byte X25519 **public** key (not PEM despite the name) |
 | `Config/encKeys[.n]` | header only: magic `ekhtar`, u16 version, u8 KeyWrapType |
-| `Pool/vkey.db[.n]` | `vkey(version_id, rsa_vkey, rsa_vkey_iv, checksum, ref_count)` |
+| `Pool/vkey.db[.n]` | `vkey(version_id, rsa_vkey, rsa_vkey_iv, checksum, ref_count)`, one row per backup version |
 
 Despite the `rsa_` column names, no RSA is used in this version: the blobs are libsodium
 **sealed boxes** (X25519 + XSalsa20-Poly1305). Sizes give it away — 80 = 48 overhead + 32
@@ -282,10 +283,22 @@ iv    = crypto_box_seal_open(vkey.rsa_vkey_iv, pub, priv)   # 16 bytes
 plain = LZ4( PKCS7-strip( AES-256-CBC-decrypt(ciphertext, key, iv) ) )
 ```
 
+**Every backup version has its own key and IV**: one `vkey.db` row per version, all
+unsealed with the same keypair, and each version encrypts only the chunks it writes.
+Dedup means a file from the newest version can mix chunks from all of them. On a
+7-version archive (Hyper Backup 4.2.2, issue #4), every one of 399,676 chunks decoded
+under exactly one key, the keys covered contiguous bucket ranges, and each row's
+`ref_count` equalled the number of buckets in its range. [V, from that report]
+
+No field found so far names the version that wrote a bucket, so hbkit tries the key that
+last worked for the bucket, then the others, and accepts one only when the decompressed
+output matches the chunk's MD5. A wrong key can occasionally still produce a valid LZ4
+block of the right length, so the MD5 test is required even when verification is off.
+
 The IV is fixed per backup version, so identical plaintext chunks produce identical
-ciphertext — visible directly in a bucket file, and a useful sanity check when
-implementing. The `uncomp_len` and MD5 in the bucket index still describe the *final*
-decompressed plaintext, so the existing verification works unchanged.
+ciphertext within a version — visible directly in a bucket file, and a useful sanity
+check when implementing. The `uncomp_len` and MD5 in the bucket index still describe the
+*final* decompressed plaintext, so the existing verification works unchanged.
 
 ### 7.3 Filenames [V]
 

{hbkit-0.4.5 → hbkit-0.4.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hbkit
-Version: 0.4.5
+Version: 0.4.6
 Summary: Recover files from Synology Hyper Backup (.hbk) archives without Synology software
 Project-URL: Homepage, https://github.com/YordiLorenzo/hbkit
 Project-URL: Source, https://github.com/YordiLorenzo/hbkit
@@ -280,8 +280,9 @@ Read this before trusting it with the only copy of anything.
   contain plaintext names (not file contents). Delete it if that matters to you.
 - **Proven against a limited set of archives.** A 3 TB DSM 7 / Hyper Backup 4.1.2
   archive (unencrypted, LZ4, single version, single pool) and a small encrypted one from
-  the same DSM version, both verified byte-exact. Older record layouts
-  (16-byte `chunk_index`, 28-byte bucket records), zlib chunks and multi-version archives
+  the same DSM version, both verified byte-exact, plus a 7-version encrypted archive from
+  Hyper Backup 4.2.2 reported in [#4](https://github.com/YordiLorenzo/hbkit/issues/4).
+  Older record layouts (16-byte `chunk_index`, 28-byte bucket records) and zlib chunks
   are implemented from disassembly but have not met a real archive of that kind. `doctor`
   exists precisely so you can find out in seconds rather than mid-restore.
 - **Unknown layouts are refused, not guessed.** A wrong guess would mean silently wrong

{hbkit-0.4.5 → hbkit-0.4.6}/README.md

@@ -251,8 +251,9 @@ Read this before trusting it with the only copy of anything.
   contain plaintext names (not file contents). Delete it if that matters to you.
 - **Proven against a limited set of archives.** A 3 TB DSM 7 / Hyper Backup 4.1.2
   archive (unencrypted, LZ4, single version, single pool) and a small encrypted one from
-  the same DSM version, both verified byte-exact. Older record layouts
-  (16-byte `chunk_index`, 28-byte bucket records), zlib chunks and multi-version archives
+  the same DSM version, both verified byte-exact, plus a 7-version encrypted archive from
+  Hyper Backup 4.2.2 reported in [#4](https://github.com/YordiLorenzo/hbkit/issues/4).
+  Older record layouts (16-byte `chunk_index`, 28-byte bucket records) and zlib chunks
   are implemented from disassembly but have not met a real archive of that kind. `doctor`
   exists precisely so you can find out in seconds rather than mid-restore.
 - **Unknown layouts are refused, not guessed.** A wrong guess would mean silently wrong

{hbkit-0.4.5 → hbkit-0.4.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hbkit"
-version = "0.4.5"
+version = "0.4.6"
 description = "Recover files from Synology Hyper Backup (.hbk) archives without Synology software"
 readme = "README.md"
 requires-python = ">=3.10"

{hbkit-0.4.5 → hbkit-0.4.6}/src/hbkit/__init__.py

@@ -1,6 +1,6 @@
 """hbkit - recover files from Synology Hyper Backup (.hbk) archives without Synology software."""
 
-__version__ = "0.4.5"
+__version__ = "0.4.6"
 
 from .archive import (Archive, NeedPassword, UnsupportedArchive,  # noqa: F401
                       find_archive_root, is_archive)

{hbkit-0.4.5 → hbkit-0.4.6}/src/hbkit/archive.py

@@ -81,6 +81,17 @@ def lz4():
     return _lz4
 
 
+def _decompress(raw: bytes, ulen: int) -> bytes | None:
+    """A chunk payload: a raw LZ4 block, else zlib (compress type 4). None if neither fits."""
+    dst = ctypes.create_string_buffer(ulen)
+    if lz4().LZ4_decompress_safe(raw, dst, len(raw), ulen) == ulen:
+        return dst.raw
+    try:
+        return zlib.decompress(raw)
+    except zlib.error:
+        return None
+
+
 # ------------------------------------------------------------ path resolution
 
 _SHARD = re.compile(r"^(\d+)\.idx(?:\.(\d+))?$")
@@ -280,6 +291,7 @@ class Archive:
         self._bidx: OrderedDict[int, bytes] = OrderedDict()
         self._bdat: OrderedDict[int, object] = OrderedDict()
         self._brec: dict[int, int] = {}
+        self._bkey: dict[int, int] = {}      # bucket id -> index into crypto.data_keys
         self._password = None
         if self.is_encrypted():
             if password is None:
@@ -311,7 +323,7 @@ class Archive:
         return out
 
     def _open_crypto(self, password: str):
-        """Unwrap the version key using the password. Raises WrongPassword on mismatch."""
+        """Unwrap every version's data key using the password. Raises WrongPassword on mismatch."""
         import sqlite3
 
         from .crypto import ArchiveCrypto
@@ -323,11 +335,11 @@ class Archive:
         with open(resolve(f"{self.root}/Config", "public.pem"), "rb") as fh:
             pub = fh.read()
         db = sqlite3.connect(f"file:{resolve(f'{self.root}/Pool', 'vkey.db')}?immutable=1", uri=True)
-        row = db.execute("SELECT rsa_vkey, rsa_vkey_iv FROM vkey ORDER BY version_id").fetchone()
+        rows = db.execute("SELECT rsa_vkey, rsa_vkey_iv FROM vkey ORDER BY version_id").fetchall()
         db.close()
-        if not row:
+        if not rows:
             raise UnsupportedArchive("encrypted archive has no rows in vkey db")
-        return ArchiveCrypto(password, unikey, pub, bytes(row[0]), bytes(row[1]))
+        return ArchiveCrypto(password, unikey, pub, [(bytes(k), bytes(iv)) for k, iv in rows])
 
     def decrypt_name(self, stored: str) -> str:
         """Decrypt a stored filename, or return it unchanged for plaintext archives."""
@@ -415,21 +427,33 @@ class Archive:
             raise ValueError(f"bucket index CRC32 mismatch b{bid}@{boff}")
         fh.seek(off)
         raw = fh.read(clen)
+        md5 = rec[12:28]
         if self.crypto is not None:
-            raw = self.crypto.decrypt_chunk(raw)
-            clen = len(raw)
-        dst = ctypes.create_string_buffer(ulen)
-        n = lz4().LZ4_decompress_safe(raw, dst, clen, ulen)
-        data = dst.raw[:n] if n > 0 else b""
-        if n != ulen:
-            try:
-                data = zlib.decompress(raw)                 # compress type 4
-            except Exception:
-                raise ValueError(f"decompress failed b{bid}@{boff} lz4={n} want={ulen}") from None
-        if verify and hashlib.md5(data).digest() != rec[12:28]:
+            return self._decrypt(bid, boff, raw, ulen, md5)
+        data = _decompress(raw, ulen)
+        if data is None:
+            raise ValueError(f"decompress failed b{bid}@{boff} want={ulen}")
+        if verify and hashlib.md5(data).digest() != md5:
             raise ValueError(f"MD5 mismatch b{bid}@{boff}")
         return data
 
+    def _decrypt(self, bid: int, boff: int, raw: bytes, ulen: int, md5: bytes) -> bytes:
+        """Decrypt and decompress one chunk of an encrypted archive.
+
+        Each backup version encrypts its chunks with its own data key, and nothing records
+        which version wrote a bucket. Try the key that last worked for this bucket, then
+        the others, and accept a key only when the output matches the chunk's MD5, whatever
+        `verify` says: a wrong key can still decompress to the right length.
+        """
+        n = len(self.crypto.data_keys)
+        first = self._bkey.get(bid, 0)
+        for k in (first, *(i for i in range(n) if i != first)):
+            data = _decompress(self.crypto.decrypt_chunk(raw, k), ulen)
+            if data is not None and hashlib.md5(data).digest() == md5:
+                self._bkey[bid] = k
+                return data
+        raise ValueError(f"no data key decodes b{bid}@{boff} ({n} tried, want={ulen})")
+
     def extract(self, ovf: int, size: int, verify: bool = True, out=None, on_bytes=None):
         """Rebuild one file. Writes to `out` if given (returns byte count), else returns bytes.
 

{hbkit-0.4.5 → hbkit-0.4.6}/src/hbkit/cli.py

@@ -268,6 +268,15 @@ def main() -> int:
             print(f"  share {s}   versions={arc.share_versions(s)}")
         return 0
 
+    if cmd in ("get", "verify"):
+        # globs match paths, which start with "/", so a leading "-" is always a stray option
+        opts = [x for x in a[2:] if x.startswith("-")]
+        if opts:
+            print(f"unknown option {opts[0]}" + (
+                " (hbkit always reads the newest backup version)" if opts[0] == "--version" else ""),
+                file=sys.stderr)
+            return 2
+
     arc, db = open_index(archive, password)
 
     if cmd == "list":

{hbkit-0.4.5 → hbkit-0.4.6}/src/hbkit/crypto.py

@@ -13,6 +13,10 @@ instant correctness check - a wrong password is detected before reading any data
     viv   = crypto_box_seal_open(vkey.rsa_vkey_iv)   # 64B sealed box -> 16B AES IV
     chunk = LZ4( unpad( AES-256-CBC(ciphertext, vkey, viv) ) )
 
+vkey.db holds one (vkey, viv) row per backup version, and each version encrypts the
+chunks it writes with its own pair. Nothing records which version wrote a chunk, so a
+reader tries each pair and keeps the one whose output matches the chunk's MD5.
+
 Filenames are encrypted separately, with their own key, and stored base64 with '/'
 replaced by '_' (illegal in a filename):
 
@@ -69,7 +73,8 @@ class ArchiveCrypto:
     """Holds the unwrapped keys for one encrypted archive."""
 
     def __init__(self, password: str, unikey: str, public_key: bytes,
-                 sealed_vkey: bytes, sealed_viv: bytes):
+                 sealed_keys: list[tuple[bytes, bytes]]):
+        """`sealed_keys` is the (rsa_vkey, rsa_vkey_iv) rows of vkey.db, one per version."""
         nb, Cipher, algorithms, modes = _deps()
         self._Cipher, self._algorithms, self._modes = Cipher, algorithms, modes
 
@@ -80,8 +85,10 @@ class ArchiveCrypto:
         if pub != public_key:
             raise WrongPassword("password does not match this archive")
 
-        self.key = nb.crypto_box_seal_open(sealed_vkey, pub, priv)
-        self.iv = nb.crypto_box_seal_open(sealed_viv, pub, priv)
+        # (key, iv) per backup version, in version order, without repeats
+        self.data_keys = list(dict.fromkeys(
+            (nb.crypto_box_seal_open(k, pub, priv), nb.crypto_box_seal_open(iv, pub, priv))
+            for k, iv in sealed_keys))
         self.fn_key = hashlib.sha256(priv + uni).digest()
         self.fn_iv = hashlib.md5(uni + FILENAME_IV_SALT).digest()
 
@@ -89,8 +96,9 @@ class ArchiveCrypto:
         d = self._Cipher(self._algorithms.AES(key), self._modes.CBC(iv)).decryptor()
         return unpad(d.update(ct) + d.finalize())
 
-    def decrypt_chunk(self, ct: bytes) -> bytes:
-        return self._dec(ct, self.key, self.iv)
+    def decrypt_chunk(self, ct: bytes, which: int) -> bytes:
+        key, iv = self.data_keys[which]
+        return self._dec(ct, key, iv)
 
     def decrypt_name(self, stored: str) -> str:
         """Decrypt one path component. '_' stands in for '/' in the base64 alphabet."""

{hbkit-0.4.5 → hbkit-0.4.6}/src/hbkit/doctor.py

@@ -74,6 +74,8 @@ def diagnose(path: str, sample: int = 10, seed: int = 0, password: str | None =
     codec = CODECS.get(cfg.get("data_compress_type", ""), f"unknown({cfg.get('data_compress_type')})")
     r.fact("chunk codec", codec)
     r.fact("encrypted", "yes (unlocked)" if arc.crypto else "no")
+    if arc.crypto:
+        r.fact("data keys", len(arc.crypto.data_keys))
     r.fact("dedup across files", cfg.get("support_cross_file_dedup", "?"))
 
     # --- blockers -------------------------------------------------------
@@ -125,8 +127,8 @@ def diagnose(path: str, sample: int = 10, seed: int = 0, password: str | None =
         if len(vs) > 1:
             multi.append(f"{s}={vs}")
     if multi:
-        r.warnings.append("multiple backup versions present; the newest is used by default "
-                          "(--version selects another): " + "; ".join(multi))
+        r.warnings.append("multiple backup versions present; hbkit reads the newest: "
+                          + "; ".join(multi))
 
     # --- prove it by actually rebuilding files ---------------------------
     if not r.blockers:
@@ -141,10 +143,16 @@ def diagnose(path: str, sample: int = 10, seed: int = 0, password: str | None =
             except Exception as e:                           # noqa: BLE001
                 r.warnings.append(f"share {s}: cannot read file list ({e})")
                 continue
-            rows = [x for x in rows if stat.S_ISREG(x[3] or 0)]
+            # One random file from each of `sample` equal slices of the virtual_file stream,
+            # which backups append to, so the newest files are always covered. On an
+            # encrypted archive their new chunks sit under the newest version's own key.
+            rows = sorted((x for x in rows if stat.S_ISREG(x[3] or 0)), key=lambda x: x[2])
             if rows:
                 rnd = random.Random(seed or 1234)
-                picked += [(s, *rnd.choice(rows)[:3]) for _ in range(min(sample, len(rows)))]
+                k = min(sample, len(rows))
+                for i in range(k):
+                    row = rows[rnd.randrange(i * len(rows) // k, (i + 1) * len(rows) // k)]
+                    picked.append((s, *row[:3]))
 
         if not picked:
             r.warnings.append("no regular files found to sample")
@@ -168,6 +176,9 @@ def diagnose(path: str, sample: int = 10, seed: int = 0, password: str | None =
             el = max(time.time() - t0, 1e-6)
             r.check(f"rebuilt {len(picked)} sampled files, all chunks verified",
                     bad == 0, f"{ok} ok, {bad} failed" + (f" - {errs[0]}" if errs else ""))
+            if arc.crypto:
+                r.fact("data keys used", f"{len(set(arc._bkey.values()))} of "
+                                         f"{len(arc.crypto.data_keys)} by the sample")
             r.fact("sample throughput", f"{human(nbytes)} in {el:.1f}s "
                                         f"({nbytes/el/1e6:,.0f} MB/s, single-threaded)")
     arc.close()

{hbkit-0.4.5 → hbkit-0.4.6}/tests/test_hbkit.py

@@ -231,6 +231,78 @@ def test_encrypted_names_and_content():
     arc.close()
 
 
+# ------------------------------------- one data key per backup version (no archive needed)
+
+def _multi_key_archive():
+    """An Archive holding just enough state for read_chunk: bucket N contains one chunk
+    encrypted with backup version N+1's data key, the way vkey.db lays them out."""
+    import ctypes
+    import hashlib
+    import io
+    import struct
+    import zlib
+
+    nb = pytest.importorskip("nacl.bindings")
+    try:
+        lib = hbk.lz4()
+    except RuntimeError:
+        pytest.skip("liblz4 not available")
+    from cryptography.hazmat.primitives import padding
+    from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
+
+    from hbkit.crypto import ARGON2_ALG, ARGON2_MEMLIMIT, ARGON2_OPSLIMIT, ArchiveCrypto
+
+    pw, unikey = "pw", "unikey"
+    seed = nb.crypto_pwhash_alg(32, pw.encode(), hashlib.md5(unikey.encode()).digest(),
+                                ARGON2_OPSLIMIT, ARGON2_MEMLIMIT, ARGON2_ALG)
+    pub, _ = nb.crypto_box_seed_keypair(seed)
+    keys = [(os.urandom(32), os.urandom(16)) for _ in range(3)]
+
+    arc = object.__new__(hbk.Archive)
+    arc.crypto = ArchiveCrypto(pw, unikey, pub, [(nb.crypto_box_seal(k, pub),
+                                                  nb.crypto_box_seal(iv, pub)) for k, iv in keys])
+    arc._brec, arc._bkey, buckets, plain = {}, {}, {}, []
+    lib.LZ4_compress_default.argtypes = [ctypes.c_char_p, ctypes.c_char_p,
+                                         ctypes.c_int, ctypes.c_int]
+    for bid, (key, iv) in enumerate([*keys, (os.urandom(32), os.urandom(16))]):
+        data = f"chunk written by version {bid + 1}. ".encode() * 40
+        dst = ctypes.create_string_buffer(len(data) + 64)
+        n = lib.LZ4_compress_default(data, dst, len(data), len(dst))
+        lz = dst.raw[:n]
+        pad = padding.PKCS7(128).padder()
+        enc = Cipher(algorithms.AES(key), modes.CBC(iv)).encryptor()
+        ct = enc.update(pad.update(lz) + pad.finalize()) + enc.finalize()
+        rec = struct.pack(">III", len(ct), 0, len(data)) + hashlib.md5(data).digest()
+        buckets[bid] = (rec + struct.pack(">I", zlib.crc32(rec)), io.BytesIO(ct))
+        arc._brec[bid] = 32
+        plain.append(data)
+    arc._bucket = buckets.__getitem__
+    return arc, plain
+
+
+def test_chunks_decode_under_the_key_of_the_version_that_wrote_them():
+    """Issue #4: vkey.db has a row per backup version, each a different AES key. Only the
+    first was ever loaded, so every chunk a later version wrote failed to decompress."""
+    arc, plain = _multi_key_archive()
+    for bid in range(3):
+        assert arc.read_chunk(bid, 0, verify=False) == plain[bid]
+    assert arc._bkey == {0: 0, 1: 1, 2: 2}
+
+
+def test_a_chunk_no_key_decodes_is_an_error_even_without_verify():
+    arc, _ = _multi_key_archive()
+    with pytest.raises(ValueError, match="no data key decodes b3@0"):
+        arc.read_chunk(3, 0, verify=False)
+
+
+def test_verify_rejects_options_it_does_not_take():
+    """`verify '*' --version 3` used to verify the newest version and say nothing."""
+    r = subprocess.run([sys.executable, "-m", "hbkit.cli", "/no/such/archive",
+                        "verify", "*", "--version", "3"], capture_output=True, text=True)
+    assert r.returncode == 2
+    assert "unknown option --version" in r.stderr
+
+
 # ------------------------------------------------- shard maths (no archive needed)
 
 def _make_shards(tmp_path, sizes, shard_size, monkeypatch):