hbkit 0.2.1__tar.gz → 0.3.0__tar.gz

{hbkit-0.2.1 → hbkit-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hbkit
-Version: 0.2.1
+Version: 0.3.0
 Summary: Recover files from Synology Hyper Backup (.hbk) archives without Synology software
 Project-URL: Homepage, https://github.com/YordiLorenzo/hbkit
 Project-URL: Source, https://github.com/YordiLorenzo/hbkit
@@ -143,27 +143,69 @@ readable by rebuilding a random sample of real files with full checksum verifica
 
 ## Network mounts (rclone / S3 / R2)
 
-Opening an archive no longer measures every index shard. Shards are a fixed 8 MiB except
-the last, so offsets are computed instead — one directory listing and a single stat per
-index family, and `file_chunk<N>.index` families are opened only if a file references them.
-On a 3 TB archive that removed roughly **2,600 network round-trips** from startup.
+You can point hbkit at a bucket mounted with rclone instead of copying the archive down
+first. Opening an archive no longer measures every index shard — shards are a fixed 8 MiB
+except the last, so offsets are computed: one directory listing and a single stat per index
+family, and `file_chunk<N>.index` families open only if a file references them. On a 3 TB
+archive that removed roughly **2,600 network round-trips** from startup.
 
-If you do mount a bucket, the flags matter more than anything hbkit does:
+### Setting up the mount
+
+hbkit can drive `rclone` for you. It stores no credentials and talks to no cloud API —
+`rclone config` still owns all of that — it just picks flags that are easy to get wrong.
 
 ```sh
-rclone mount r2:bucket ~/mnt/r2 --read-only \
-  --vfs-cache-mode off \      # range requests; 'full' downloads whole files
-  --dir-cache-time 72h         # first listing of ~2000 shards is slow, then cached
+rclone config                                   # add an s3 remote; for R2 pick "Cloudflare"
+hbk remotes                                     # what's configured
+hbk mount r2:mybucket ~/mnt/backup --for browse # mount + pre-warm
+hbk ~/mnt/backup/target.hbk doctor
+hbk unmount ~/mnt/backup
 ```
 
-**Do not use `--vfs-cache-mode full`.** hbkit reads a 32-byte index record and a ~5 KB
-chunk at a time; in `full` mode each of those pulls an entire file, so a 10 KB extraction
-downloaded ~82 MB (an 8 MiB index shard plus a ~50 MB bucket) and took ten minutes.
-
-Even configured well, a network mount is dramatically slower than local storage — the
-access pattern is thousands of small scattered reads. **If you can, copy the archive to a
-local disk first.** Treat mounted-bucket recovery as workable for pulling out a handful of
-files, not for restoring terabytes.
+`--for browse` (default) uses range requests; `--for restore` uses whole-file fetching as
+read-ahead with the cache capped at 50G (`--cache-size 10G` to change it). `--no-warm`
+skips pre-warming, `--dry-run` just prints the rclone command.
+
+Measured on a 3 TB archive in Cloudflare R2:
+
+| | before tuning | with `hbk mount` |
+|---|---|---|
+| cold mount + directory pre-warm | ~12 min | **16 s** |
+| opening the archive | 9 min 21 s | **1.8 s** |
+| extracting a 10 KB file | 584 s | **11 s** |
+
+Three things get you there, and you can apply them by hand if you'd rather not use
+`hbk mount`:
+
+- **`--no-modtime`** — the big one. Without it rclone issues a HEAD per object just to fill
+  in modification times for a listing: ~0.27 s per entry, so the 2,021-shard `chunk_index`
+  directory took **546.8 s**. With it, **4.07 s** — 134× faster. hbkit never uses the
+  modtimes of an archive's internal files (restored mtimes come from the archive's own
+  metadata), so this costs nothing.
+- **`--dir-cache-time 72h`** — the first listing of a large shard directory is the expensive
+  one; cache it and every later open is instant.
+- **pre-warming** — `hbk mount` walks the index directories once up front, so the wait
+  happens visibly at mount time instead of looking like a hang on your first command. Run
+  it separately with `hbk warm ~/mnt/backup`.
+
+### Choosing `--vfs-cache-mode` — it depends on what you're doing
+
+`hbk mount --for` picks this for you, but if you mount by hand: rclone's `full` mode
+downloads **whole files** while `off` issues **range requests**.
+
+| What you're doing | Mode | Why |
+|---|---|---|
+| First index build | `full` | Reads one share database end to end (356 MB on a 1.1M-file archive). Sequential — what whole-file fetching is good at. One time; cached in `~/.cache/hbkit` after. |
+| Browsing, `list`, the TUI tree | either | Served from the local index, no network at all. |
+| Pulling out a few files | `off` | hbkit reads a 32-byte index record and a ~5 KB chunk at a time. In `full` mode each pulls a whole file: a measured 10 KB extraction fetched ~82 MB and took ten minutes. |
+| Bulk restoring a folder | `full` | You touch most of each ~50 MB bucket anyway, so whole-file fetching becomes read-ahead. **Cap it** with `--vfs-cache-max-size`, or it will fill your disk. |
+
+### Expectations
+
+Even configured well, a mounted bucket is dramatically slower than local storage — the
+access pattern is thousands of small scattered reads, and each one crosses the network.
+**If you can, copy the archive to a local disk first.** Start with one small folder and
+watch the throughput before committing to a large restore.
 
 ## Performance
 

{hbkit-0.2.1 → hbkit-0.3.0}/README.md

@@ -114,27 +114,69 @@ readable by rebuilding a random sample of real files with full checksum verifica
 
 ## Network mounts (rclone / S3 / R2)
 
-Opening an archive no longer measures every index shard. Shards are a fixed 8 MiB except
-the last, so offsets are computed instead — one directory listing and a single stat per
-index family, and `file_chunk<N>.index` families are opened only if a file references them.
-On a 3 TB archive that removed roughly **2,600 network round-trips** from startup.
+You can point hbkit at a bucket mounted with rclone instead of copying the archive down
+first. Opening an archive no longer measures every index shard — shards are a fixed 8 MiB
+except the last, so offsets are computed: one directory listing and a single stat per index
+family, and `file_chunk<N>.index` families open only if a file references them. On a 3 TB
+archive that removed roughly **2,600 network round-trips** from startup.
 
-If you do mount a bucket, the flags matter more than anything hbkit does:
+### Setting up the mount
+
+hbkit can drive `rclone` for you. It stores no credentials and talks to no cloud API —
+`rclone config` still owns all of that — it just picks flags that are easy to get wrong.
 
 ```sh
-rclone mount r2:bucket ~/mnt/r2 --read-only \
-  --vfs-cache-mode off \      # range requests; 'full' downloads whole files
-  --dir-cache-time 72h         # first listing of ~2000 shards is slow, then cached
+rclone config                                   # add an s3 remote; for R2 pick "Cloudflare"
+hbk remotes                                     # what's configured
+hbk mount r2:mybucket ~/mnt/backup --for browse # mount + pre-warm
+hbk ~/mnt/backup/target.hbk doctor
+hbk unmount ~/mnt/backup
 ```
 
-**Do not use `--vfs-cache-mode full`.** hbkit reads a 32-byte index record and a ~5 KB
-chunk at a time; in `full` mode each of those pulls an entire file, so a 10 KB extraction
-downloaded ~82 MB (an 8 MiB index shard plus a ~50 MB bucket) and took ten minutes.
-
-Even configured well, a network mount is dramatically slower than local storage — the
-access pattern is thousands of small scattered reads. **If you can, copy the archive to a
-local disk first.** Treat mounted-bucket recovery as workable for pulling out a handful of
-files, not for restoring terabytes.
+`--for browse` (default) uses range requests; `--for restore` uses whole-file fetching as
+read-ahead with the cache capped at 50G (`--cache-size 10G` to change it). `--no-warm`
+skips pre-warming, `--dry-run` just prints the rclone command.
+
+Measured on a 3 TB archive in Cloudflare R2:
+
+| | before tuning | with `hbk mount` |
+|---|---|---|
+| cold mount + directory pre-warm | ~12 min | **16 s** |
+| opening the archive | 9 min 21 s | **1.8 s** |
+| extracting a 10 KB file | 584 s | **11 s** |
+
+Three things get you there, and you can apply them by hand if you'd rather not use
+`hbk mount`:
+
+- **`--no-modtime`** — the big one. Without it rclone issues a HEAD per object just to fill
+  in modification times for a listing: ~0.27 s per entry, so the 2,021-shard `chunk_index`
+  directory took **546.8 s**. With it, **4.07 s** — 134× faster. hbkit never uses the
+  modtimes of an archive's internal files (restored mtimes come from the archive's own
+  metadata), so this costs nothing.
+- **`--dir-cache-time 72h`** — the first listing of a large shard directory is the expensive
+  one; cache it and every later open is instant.
+- **pre-warming** — `hbk mount` walks the index directories once up front, so the wait
+  happens visibly at mount time instead of looking like a hang on your first command. Run
+  it separately with `hbk warm ~/mnt/backup`.
+
+### Choosing `--vfs-cache-mode` — it depends on what you're doing
+
+`hbk mount --for` picks this for you, but if you mount by hand: rclone's `full` mode
+downloads **whole files** while `off` issues **range requests**.
+
+| What you're doing | Mode | Why |
+|---|---|---|
+| First index build | `full` | Reads one share database end to end (356 MB on a 1.1M-file archive). Sequential — what whole-file fetching is good at. One time; cached in `~/.cache/hbkit` after. |
+| Browsing, `list`, the TUI tree | either | Served from the local index, no network at all. |
+| Pulling out a few files | `off` | hbkit reads a 32-byte index record and a ~5 KB chunk at a time. In `full` mode each pulls a whole file: a measured 10 KB extraction fetched ~82 MB and took ten minutes. |
+| Bulk restoring a folder | `full` | You touch most of each ~50 MB bucket anyway, so whole-file fetching becomes read-ahead. **Cap it** with `--vfs-cache-max-size`, or it will fill your disk. |
+
+### Expectations
+
+Even configured well, a mounted bucket is dramatically slower than local storage — the
+access pattern is thousands of small scattered reads, and each one crosses the network.
+**If you can, copy the archive to a local disk first.** Start with one small folder and
+watch the throughput before committing to a large restore.
 
 ## Performance
 

{hbkit-0.2.1 → hbkit-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hbkit"
-version = "0.2.1"
+version = "0.3.0"
 description = "Recover files from Synology Hyper Backup (.hbk) archives without Synology software"
 readme = "README.md"
 requires-python = ">=3.10"

{hbkit-0.2.1 → hbkit-0.3.0}/src/hbkit/__init__.py

@@ -1,6 +1,6 @@
 """hbkit - recover files from Synology Hyper Backup (.hbk) archives without Synology software."""
 
-__version__ = "0.2.1"
+__version__ = "0.3.0"
 
 from .archive import (Archive, NeedPassword, UnsupportedArchive,  # noqa: F401
                       find_archive_root, is_archive)

{hbkit-0.2.1 → hbkit-0.3.0}/src/hbkit/cli.py

@@ -8,12 +8,19 @@
   hbk <archive> verify <glob>      [-j N]            integrity-check, write nothing
   hbk <archive> tui                                  browse in the full-screen UI
 
+  hbk mount   <remote:bucket> <dir> [--for browse|restore] [--no-warm]
+  hbk warm    <dir>                                  pre-cache dir listings
+  hbk unmount <dir>                                          unmount it
+  hbk remotes                                                list configured rclone remotes
+
 Encrypted archives: add -p/--password, or set HBK_PASSWORD, or you'll be prompted.
 
 <archive> is a .hbk directory, or any drive/folder containing one.
 Globs match the full archive path, which starts with the share name.
 Extraction is resumable: correctly-sized files are skipped.
 
+  hbk mount r2:mybucket ~/mnt/backup --for browse
+  hbk ~/mnt/backup/target.hbk doctor
   hbk /Volumes/Backup doctor
   hbk /Volumes/Backup list holiday
   hbk /Volumes/Backup get "/Photos/2019/*" ~/restore
@@ -149,9 +156,47 @@ def main() -> int:
             i = a.index(flag)
             password = a[i + 1]
             del a[i:i + 2]
-    if len(a) < 2:
+    if len(a) < 2 and (not a or a[0] != "remotes"):
         print(__doc__)
         return 1
+    if a[0] in ("mount", "unmount", "remotes", "warm"):
+        from . import mount as m
+        try:
+            if a[0] == "remotes":
+                rs = m.remotes()
+                print("\n".join(f"  {r}" for r in rs) if rs
+                      else "  no rclone remotes configured - run: rclone config")
+                return 0
+            if a[0] == "warm":
+                if len(a) < 2:
+                    print("usage: hbk warm <dir>")
+                    return 1
+                return m.warm(a[1])
+            if a[0] == "unmount":
+                if len(a) < 2:
+                    print("usage: hbk unmount <dir>")
+                    return 1
+                return m.unmount(a[1])
+            if len(a) < 3:
+                print("usage: hbk mount <remote:bucket> <dir> [--for browse|restore]")
+                return 1
+            prof, rest = "browse", a[3:]
+            if "--for" in rest:
+                i = rest.index("--for")
+                prof = rest[i + 1]
+            cache = None
+            if "--cache-size" in rest:
+                cache = rest[rest.index("--cache-size") + 1]
+            rc = m.mount(a[1], a[2], prof, cache, dry_run="--dry-run" in rest)
+            if rc == 0 and "--dry-run" not in rest and "--no-warm" not in rest:
+                print("\npre-warming directory cache (first open over a network mount "
+                      "is slow; doing it here so later commands are instant)")
+                m.warm(a[2])
+            return rc
+        except (m.RcloneMissing, ValueError) as e:
+            print(f"{e}", file=sys.stderr)
+            return 2
+
     archive, cmd = a[0], a[1]
 
     if cmd == "doctor":

hbkit-0.3.0/src/hbkit/mount.py

@@ -0,0 +1,174 @@
+"""Convenience wrapper around `rclone mount` for archives kept in object storage.
+
+This stores no credentials and talks to no cloud API - `rclone config` still owns all of
+that. All it does is pick the flags, because the flags are easy to get badly wrong:
+rclone's `--vfs-cache-mode full` fetches *whole files*, and hbkit reads a 32-byte index
+record and a ~5 KB chunk at a time, so a 10 KB extraction can pull ~82 MB. Meanwhile
+`off` issues range requests, which is wrong for the one-time index build (a single
+sequential read of a share database that can be hundreds of MB).
+
+So the mode depends on the job, and that is what `--for` selects:
+
+    browse   -> off,  range requests; cheap metadata, good for cherry-picking files
+    restore  -> full, whole-file fetch acts as read-ahead when pulling a whole folder,
+                capped so it cannot fill the disk
+"""
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+import sys
+
+PROFILES = {
+    # name: (vfs-cache-mode, extra flags, one-line rationale)
+    "browse": ("off", [],
+               "range requests - best for opening an archive and pulling a few files"),
+    "restore": ("full", ["--vfs-cache-max-size", "50G"],
+                "whole-file fetch as read-ahead for bulk restores, cache capped at 50G"),
+}
+
+# --no-modtime is the single biggest win. Without it rclone issues a HEAD per object just
+# to fill in modification times for a directory listing, which is ~0.27s per entry: the
+# 2,021-shard chunk_index directory took 546.8s cold. With it, 4.07s - a 134x speedup.
+# hbkit never uses the modtimes of an archive's internal files (restored file mtimes come
+# from the archive's own SQLite metadata), so dropping them costs nothing.
+COMMON = ["--read-only", "--no-modtime", "--dir-cache-time", "72h", "--daemon"]
+
+
+class RcloneMissing(Exception):
+    pass
+
+
+def require_rclone() -> str:
+    p = shutil.which("rclone")
+    if not p:
+        raise RcloneMissing(
+            "rclone is not installed. Install it (brew install rclone / "
+            "apt install rclone), then run `rclone config` to add your S3/R2 remote.")
+    return p
+
+
+def remotes() -> list[str]:
+    """Configured rclone remotes, e.g. ['r2:', 's3:']."""
+    try:
+        out = subprocess.run([require_rclone(), "listremotes"],
+                             capture_output=True, text=True, timeout=15)
+        return [x for x in out.stdout.split() if x]
+    except (RcloneMissing, subprocess.SubprocessError):
+        return []
+
+
+def is_mounted(path: str) -> bool:
+    path = os.path.abspath(os.path.expanduser(path))
+    try:
+        out = subprocess.run(["mount"], capture_output=True, text=True, timeout=10).stdout
+    except subprocess.SubprocessError:
+        return False
+    return any(f" {path} " in ln or ln.endswith(f" {path}") for ln in out.splitlines())
+
+
+def build_command(remote: str, mountpoint: str, profile: str = "browse",
+                  cache_size: str | None = None) -> list[str]:
+    if profile not in PROFILES:
+        raise ValueError(f"unknown profile {profile!r}; choose from {', '.join(PROFILES)}")
+    mode, extra, _ = PROFILES[profile]
+    extra = list(extra)
+    if cache_size and mode == "full":
+        extra = ["--vfs-cache-max-size", cache_size]
+    return ([require_rclone(), "mount", remote, os.path.abspath(os.path.expanduser(mountpoint)),
+             "--vfs-cache-mode", mode] + extra + COMMON)
+
+
+def mount(remote: str, mountpoint: str, profile: str = "browse",
+          cache_size: str | None = None, dry_run: bool = False) -> int:
+    mp = os.path.abspath(os.path.expanduser(mountpoint))
+    if is_mounted(mp):
+        print(f"already mounted: {mp}")
+        return 0
+    cmd = build_command(remote, mp, profile, cache_size)
+    mode, _, why = PROFILES[profile]
+    print(f"profile {profile}: --vfs-cache-mode {mode}  ({why})")
+    print("  " + " ".join(cmd))
+    if dry_run:
+        return 0
+    os.makedirs(mp, exist_ok=True)
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        print((r.stderr or r.stdout).strip(), file=sys.stderr)
+        return r.returncode
+    # --daemon returns immediately; wait for the mountpoint to become live
+    import time
+    for _ in range(60):
+        if is_mounted(mp) and os.listdir(mp):
+            break
+        time.sleep(0.5)
+    if not is_mounted(mp):
+        print(f"rclone returned success but {mp} is not mounted", file=sys.stderr)
+        return 1
+    print(f"mounted {remote} at {mp}")
+    for e in sorted(os.listdir(mp))[:10]:
+        print(f"    {e}")
+    return 0
+
+
+# Directories hbkit lists when opening an archive. rclone caches a listing for
+# --dir-cache-time, so walking these once up front turns a ~9 minute "did it hang?" on
+# the user's first command into a visible, attributable wait here. Measured on a 3 TB
+# archive over R2: 9m21s cold, then 1.6s, then 0.07s.
+def warm(mountpoint: str, quiet: bool = False) -> int:
+    """Populate rclone's directory cache for every .hbk archive under `mountpoint`."""
+    import time
+    mp = os.path.abspath(os.path.expanduser(mountpoint))
+    archives = []
+    for e in sorted(os.listdir(mp)):
+        d = os.path.join(mp, e)
+        if os.path.isdir(d) and os.path.isdir(os.path.join(d, "Config")):
+            archives.append(d)
+    if not archives:
+        return 0
+    n = 0
+    t0 = time.time()
+    for arc in archives:
+        targets = [arc, os.path.join(arc, "Config"), os.path.join(arc, "Pool")]
+        for sub in ("Config/virtual_file.index", "Pool/chunk_index", "Config/@Share"):
+            targets.append(os.path.join(arc, sub))
+        cfg = os.path.join(arc, "Config")
+        try:
+            targets += [os.path.join(cfg, x) for x in os.listdir(cfg)
+                        if x.startswith("file_chunk") and os.path.isdir(os.path.join(cfg, x))]
+        except OSError:
+            pass
+        share = os.path.join(arc, "Config", "@Share")
+        if os.path.isdir(share):
+            targets += [os.path.join(share, x) for x in os.listdir(share)]
+        for t in targets:
+            if not os.path.isdir(t):
+                continue
+            if not quiet:
+                print(f"  warming {os.path.relpath(t, mp)} ...", end="", flush=True)
+            s0 = time.time()
+            try:
+                c = len(os.listdir(t))
+            except OSError as e:
+                c = -1
+            if not quiet:
+                print(f" {c} entries, {time.time()-s0:.1f}s")
+            n += 1
+    if not quiet:
+        print(f"warmed {n} directories in {time.time()-t0:.0f}s "
+              f"(cached for --dir-cache-time; later opens are instant)")
+    return 0
+
+
+def unmount(mountpoint: str) -> int:
+    mp = os.path.abspath(os.path.expanduser(mountpoint))
+    if not is_mounted(mp):
+        print(f"not mounted: {mp}")
+        return 0
+    for cmd in (["umount", mp], ["fusermount", "-u", mp], ["diskutil", "unmount", "force", mp]):
+        if shutil.which(cmd[0]) and subprocess.run(cmd, capture_output=True).returncode == 0:
+            print(f"unmounted {mp}")
+            return 0
+    print(f"could not unmount {mp}", file=sys.stderr)
+    return 1

{hbkit-0.2.1 → hbkit-0.3.0}/tests/test_hbkit.py

@@ -285,3 +285,56 @@ def test_cat_single_shard_still_works(tmp_path, monkeypatch):
     c = A.Cat(str(tmp_path))
     assert c.total == 100 and c.read(70, 20) == blob[70:90]
     c.close()
+
+
+# --------------------------------------------------- rclone mount helper (no network)
+
+def test_mount_profiles_pick_the_right_cache_mode(monkeypatch):
+    from hbkit import mount as m
+    monkeypatch.setattr(m, "require_rclone", lambda: "/usr/bin/rclone")
+    browse = m.build_command("r2:b", "/tmp/mp", "browse")
+    restore = m.build_command("r2:b", "/tmp/mp", "restore")
+    assert "off" == browse[browse.index("--vfs-cache-mode") + 1]
+    assert "full" == restore[restore.index("--vfs-cache-mode") + 1]
+    # bulk restores cache whole ~50MB buckets, so the cache MUST be capped
+    assert "--vfs-cache-max-size" in restore
+    assert "--vfs-cache-max-size" not in browse
+
+
+def test_mount_command_is_always_read_only_and_skips_modtimes(monkeypatch):
+    """--no-modtime avoids a HEAD per object: 2021-entry listing 546.8s -> 4.07s."""
+    from hbkit import mount as m
+    monkeypatch.setattr(m, "require_rclone", lambda: "/usr/bin/rclone")
+    for profile in ("browse", "restore"):
+        cmd = m.build_command("r2:b", "/tmp/mp", profile)
+        assert "--read-only" in cmd, profile
+        assert "--no-modtime" in cmd, profile
+        assert "--dir-cache-time" in cmd, profile
+
+
+def test_mount_rejects_unknown_profile(monkeypatch):
+    from hbkit import mount as m
+    monkeypatch.setattr(m, "require_rclone", lambda: "/usr/bin/rclone")
+    with pytest.raises(ValueError):
+        m.build_command("r2:b", "/tmp/mp", "nonsense")
+
+
+def test_mount_cache_size_override(monkeypatch):
+    from hbkit import mount as m
+    monkeypatch.setattr(m, "require_rclone", lambda: "/usr/bin/rclone")
+    cmd = m.build_command("r2:b", "/tmp/mp", "restore", cache_size="10G")
+    assert cmd[cmd.index("--vfs-cache-max-size") + 1] == "10G"
+
+
+def test_missing_rclone_is_a_clear_error(monkeypatch):
+    from hbkit import mount as m
+    monkeypatch.setattr(m.shutil, "which", lambda _: None)
+    with pytest.raises(m.RcloneMissing) as e:
+        m.require_rclone()
+    assert "rclone config" in str(e.value)
+
+
+def test_warm_ignores_a_directory_with_no_archives(tmp_path):
+    from hbkit import mount as m
+    (tmp_path / "not-an-archive").mkdir()
+    assert m.warm(str(tmp_path), quiet=True) == 0