hbkit 0.2.0__tar.gz → 0.3.0__tar.gz

{hbkit-0.2.0 → hbkit-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hbkit
-Version: 0.2.0
+Version: 0.3.0
 Summary: Recover files from Synology Hyper Backup (.hbk) archives without Synology software
 Project-URL: Homepage, https://github.com/YordiLorenzo/hbkit
 Project-URL: Source, https://github.com/YordiLorenzo/hbkit
@@ -35,9 +35,9 @@ Description-Content-Type: text/markdown
 
 Recover files from **Synology Hyper Backup (`.hbk`)** archives without any Synology software.
 
-Point it at a backup — on a local disk, an external drive, or an S3/R2 bucket mounted with
-rclone — browse it as a tree, and pull out what you want. Works headless on Linux and macOS,
-including Apple Silicon, where Synology's own Hyper Backup Explorer is awkward or unavailable.
+Point it at a backup on a local disk, an external drive, or a network mount, browse it as
+a tree, and pull out what you want. Works headless on Linux and macOS, including Apple Silicon, where Synology's
+own Hyper Backup Explorer is awkward or unavailable.
 
 ```sh
 brew install lz4          # or: sudo apt install liblz4-1
@@ -141,6 +141,72 @@ readable by rebuilding a random sample of real files with full checksum verifica
 - **Index cached** per archive in `~/.cache/hbkit`, rebuilt automatically when the archive
   changes. Browsing 1.1M files is instant after the first open.
 
+## Network mounts (rclone / S3 / R2)
+
+You can point hbkit at a bucket mounted with rclone instead of copying the archive down
+first. Opening an archive no longer measures every index shard — shards are a fixed 8 MiB
+except the last, so offsets are computed: one directory listing and a single stat per index
+family, and `file_chunk<N>.index` families open only if a file references them. On a 3 TB
+archive that removed roughly **2,600 network round-trips** from startup.
+
+### Setting up the mount
+
+hbkit can drive `rclone` for you. It stores no credentials and talks to no cloud API —
+`rclone config` still owns all of that — it just picks flags that are easy to get wrong.
+
+```sh
+rclone config                                   # add an s3 remote; for R2 pick "Cloudflare"
+hbk remotes                                     # what's configured
+hbk mount r2:mybucket ~/mnt/backup --for browse # mount + pre-warm
+hbk ~/mnt/backup/target.hbk doctor
+hbk unmount ~/mnt/backup
+```
+
+`--for browse` (default) uses range requests; `--for restore` uses whole-file fetching as
+read-ahead with the cache capped at 50G (`--cache-size 10G` to change it). `--no-warm`
+skips pre-warming, `--dry-run` just prints the rclone command.
+
+Measured on a 3 TB archive in Cloudflare R2:
+
+| | before tuning | with `hbk mount` |
+|---|---|---|
+| cold mount + directory pre-warm | ~12 min | **16 s** |
+| opening the archive | 9 min 21 s | **1.8 s** |
+| extracting a 10 KB file | 584 s | **11 s** |
+
+Three things get you there, and you can apply them by hand if you'd rather not use
+`hbk mount`:
+
+- **`--no-modtime`** — the big one. Without it rclone issues a HEAD per object just to fill
+  in modification times for a listing: ~0.27 s per entry, so the 2,021-shard `chunk_index`
+  directory took **546.8 s**. With it, **4.07 s** — 134× faster. hbkit never uses the
+  modtimes of an archive's internal files (restored mtimes come from the archive's own
+  metadata), so this costs nothing.
+- **`--dir-cache-time 72h`** — the first listing of a large shard directory is the expensive
+  one; cache it and every later open is instant.
+- **pre-warming** — `hbk mount` walks the index directories once up front, so the wait
+  happens visibly at mount time instead of looking like a hang on your first command. Run
+  it separately with `hbk warm ~/mnt/backup`.
+
+### Choosing `--vfs-cache-mode` — it depends on what you're doing
+
+`hbk mount --for` picks this for you, but if you mount by hand: rclone's `full` mode
+downloads **whole files** while `off` issues **range requests**.
+
+| What you're doing | Mode | Why |
+|---|---|---|
+| First index build | `full` | Reads one share database end to end (356 MB on a 1.1M-file archive). Sequential — what whole-file fetching is good at. One time; cached in `~/.cache/hbkit` after. |
+| Browsing, `list`, the TUI tree | either | Served from the local index, no network at all. |
+| Pulling out a few files | `off` | hbkit reads a 32-byte index record and a ~5 KB chunk at a time. In `full` mode each pulls a whole file: a measured 10 KB extraction fetched ~82 MB and took ten minutes. |
+| Bulk restoring a folder | `full` | You touch most of each ~50 MB bucket anyway, so whole-file fetching becomes read-ahead. **Cap it** with `--vfs-cache-max-size`, or it will fill your disk. |
+
+### Expectations
+
+Even configured well, a mounted bucket is dramatically slower than local storage — the
+access pattern is thousands of small scattered reads, and each one crosses the network.
+**If you can, copy the archive to a local disk first.** Start with one small folder and
+watch the throughput before committing to a large restore.
+
 ## Performance
 
 Use `-j` to set worker processes (default 8). Threads do not help — extraction is

{hbkit-0.2.0 → hbkit-0.3.0}/README.md

@@ -6,9 +6,9 @@
 
 Recover files from **Synology Hyper Backup (`.hbk`)** archives without any Synology software.
 
-Point it at a backup — on a local disk, an external drive, or an S3/R2 bucket mounted with
-rclone — browse it as a tree, and pull out what you want. Works headless on Linux and macOS,
-including Apple Silicon, where Synology's own Hyper Backup Explorer is awkward or unavailable.
+Point it at a backup on a local disk, an external drive, or a network mount, browse it as
+a tree, and pull out what you want. Works headless on Linux and macOS, including Apple Silicon, where Synology's
+own Hyper Backup Explorer is awkward or unavailable.
 
 ```sh
 brew install lz4          # or: sudo apt install liblz4-1
@@ -112,6 +112,72 @@ readable by rebuilding a random sample of real files with full checksum verifica
 - **Index cached** per archive in `~/.cache/hbkit`, rebuilt automatically when the archive
   changes. Browsing 1.1M files is instant after the first open.
 
+## Network mounts (rclone / S3 / R2)
+
+You can point hbkit at a bucket mounted with rclone instead of copying the archive down
+first. Opening an archive no longer measures every index shard — shards are a fixed 8 MiB
+except the last, so offsets are computed: one directory listing and a single stat per index
+family, and `file_chunk<N>.index` families open only if a file references them. On a 3 TB
+archive that removed roughly **2,600 network round-trips** from startup.
+
+### Setting up the mount
+
+hbkit can drive `rclone` for you. It stores no credentials and talks to no cloud API —
+`rclone config` still owns all of that — it just picks flags that are easy to get wrong.
+
+```sh
+rclone config                                   # add an s3 remote; for R2 pick "Cloudflare"
+hbk remotes                                     # what's configured
+hbk mount r2:mybucket ~/mnt/backup --for browse # mount + pre-warm
+hbk ~/mnt/backup/target.hbk doctor
+hbk unmount ~/mnt/backup
+```
+
+`--for browse` (default) uses range requests; `--for restore` uses whole-file fetching as
+read-ahead with the cache capped at 50G (`--cache-size 10G` to change it). `--no-warm`
+skips pre-warming, `--dry-run` just prints the rclone command.
+
+Measured on a 3 TB archive in Cloudflare R2:
+
+| | before tuning | with `hbk mount` |
+|---|---|---|
+| cold mount + directory pre-warm | ~12 min | **16 s** |
+| opening the archive | 9 min 21 s | **1.8 s** |
+| extracting a 10 KB file | 584 s | **11 s** |
+
+Three things get you there, and you can apply them by hand if you'd rather not use
+`hbk mount`:
+
+- **`--no-modtime`** — the big one. Without it rclone issues a HEAD per object just to fill
+  in modification times for a listing: ~0.27 s per entry, so the 2,021-shard `chunk_index`
+  directory took **546.8 s**. With it, **4.07 s** — 134× faster. hbkit never uses the
+  modtimes of an archive's internal files (restored mtimes come from the archive's own
+  metadata), so this costs nothing.
+- **`--dir-cache-time 72h`** — the first listing of a large shard directory is the expensive
+  one; cache it and every later open is instant.
+- **pre-warming** — `hbk mount` walks the index directories once up front, so the wait
+  happens visibly at mount time instead of looking like a hang on your first command. Run
+  it separately with `hbk warm ~/mnt/backup`.
+
+### Choosing `--vfs-cache-mode` — it depends on what you're doing
+
+`hbk mount --for` picks this for you, but if you mount by hand: rclone's `full` mode
+downloads **whole files** while `off` issues **range requests**.
+
+| What you're doing | Mode | Why |
+|---|---|---|
+| First index build | `full` | Reads one share database end to end (356 MB on a 1.1M-file archive). Sequential — what whole-file fetching is good at. One time; cached in `~/.cache/hbkit` after. |
+| Browsing, `list`, the TUI tree | either | Served from the local index, no network at all. |
+| Pulling out a few files | `off` | hbkit reads a 32-byte index record and a ~5 KB chunk at a time. In `full` mode each pulls a whole file: a measured 10 KB extraction fetched ~82 MB and took ten minutes. |
+| Bulk restoring a folder | `full` | You touch most of each ~50 MB bucket anyway, so whole-file fetching becomes read-ahead. **Cap it** with `--vfs-cache-max-size`, or it will fill your disk. |
+
+### Expectations
+
+Even configured well, a mounted bucket is dramatically slower than local storage — the
+access pattern is thousands of small scattered reads, and each one crosses the network.
+**If you can, copy the archive to a local disk first.** Start with one small folder and
+watch the throughput before committing to a large restore.
+
 ## Performance
 
 Use `-j` to set worker processes (default 8). Threads do not help — extraction is

{hbkit-0.2.0 → hbkit-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hbkit"
-version = "0.2.0"
+version = "0.3.0"
 description = "Recover files from Synology Hyper Backup (.hbk) archives without Synology software"
 readme = "README.md"
 requires-python = ">=3.10"

{hbkit-0.2.0 → hbkit-0.3.0}/src/hbkit/__init__.py

@@ -1,6 +1,6 @@
 """hbkit - recover files from Synology Hyper Backup (.hbk) archives without Synology software."""
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 from .archive import (Archive, NeedPassword, UnsupportedArchive,  # noqa: F401
                       find_archive_root, is_archive)

{hbkit-0.2.0 → hbkit-0.3.0}/src/hbkit/archive.py

@@ -45,6 +45,7 @@ BUCKET_BUF = int(os.environ.get("HBK_BUCKET_BUF", 1 << 22))   # read-ahead; plat
 INDEX_BUF = int(os.environ.get("HBK_INDEX_BUF", 1 << 16))     # chunk keys are mostly sequential;
                                                              # unbuffered = one syscall per 29 B
 MAGIC = 0x7053A86E
+SHARD_SIZE = 8 << 20        # every index shard is exactly 8 MiB except the last
 
 _LZ4_CANDIDATES = [
     os.environ.get("HBK_LZ4"),
@@ -112,7 +113,14 @@ def resolve(d: str, base: str) -> str:
 
 
 class Cat:
-    """Numbered <N>.idx[.gen] shards presented as a single logical byte stream."""
+    """Numbered <N>.idx[.gen] shards presented as a single logical byte stream.
+
+    Shards are a fixed SHARD_SIZE except the last, so offsets are computed rather than
+    measured. That matters enormously on network mounts: stat-ing every shard cost ~2,600
+    round-trips on a 3 TB archive (minutes before reading a byte). We now do one directory
+    listing plus a single stat of the final shard. If the assumption were ever wrong, a
+    short read surfaces as a chunk MD5 failure - loudly - never as silent corruption.
+    """
 
     def __init__(self, d: str):
         self.d = d
@@ -129,12 +137,11 @@ class Cat:
         if not found:
             raise FileNotFoundError(f"no .idx shards in {d}")
         self.files = [found[k][1] for k in sorted(found)]
-        self.sizes = [os.path.getsize(os.path.join(d, f)) for f in self.files]
-        self.starts, t = [], 0
-        for s in self.sizes:
-            self.starts.append(t)
-            t += s
-        self.total = t
+        n = len(self.files)
+        last = os.path.getsize(os.path.join(d, self.files[-1]))
+        self.sizes = [SHARD_SIZE] * (n - 1) + [last]
+        self.starts = [i * SHARD_SIZE for i in range(n)]
+        self.total = (n - 1) * SHARD_SIZE + last
         self._fh: dict[int, object] = {}
         # Shard 0 carries a 64-byte header that declares the record size, so readers do
         # not have to hardcode per-DSM-version layouts.
@@ -150,16 +157,26 @@ class Cat:
         self.declared_total = (w[5] << 32) | w[6]
 
     def read(self, off: int, n: int) -> bytes:
+        """Read n bytes at logical offset off, spanning shards as needed."""
         out = b""
-        for i, (f, st, sz) in enumerate(zip(self.files, self.starts, self.sizes)):
-            if off + n <= st or off >= st + sz:
-                continue
-            a, b = max(off, st), min(off + n, st + sz)
+        i = off // SHARD_SIZE
+        while n > 0 and i < len(self.files):
+            st = self.starts[i]
+            take = min(n, st + self.sizes[i] - off)
+            if take <= 0:
+                break
             fh = self._fh.get(i)
             if fh is None:
-                fh = self._fh[i] = open(os.path.join(self.d, f), "rb", buffering=INDEX_BUF)
-            fh.seek(a - st)
-            out += fh.read(b - a)
+                fh = self._fh[i] = open(os.path.join(self.d, self.files[i]), "rb",
+                                        buffering=INDEX_BUF)
+            fh.seek(off - st)
+            got = fh.read(take)
+            out += got
+            if len(got) < take:            # short shard: stop rather than mis-align
+                break
+            off += take
+            n -= take
+            i += 1
         return out
 
     def close(self):
@@ -168,6 +185,32 @@ class Cat:
         self._fh.clear()
 
 
+class _LazyCats(dict):
+    """file_chunk<N>.index families, opened only when a file actually references one."""
+
+    def __init__(self, dirs: dict[int, str]):
+        super().__init__()
+        self._dirs = dirs
+
+    def get(self, k, default=None):
+        if k in self:
+            return self[k]
+        d = self._dirs.get(k)
+        if d is None:
+            return default
+        self[k] = c = Cat(d)
+        return c
+
+    def __missing__(self, k):
+        c = self.get(k)
+        if c is None:
+            raise KeyError(k)
+        return c
+
+    def values(self):
+        return [v for v in dict.values(self)]
+
+
 def is_archive(d: str) -> bool:
     return os.path.isdir(os.path.join(d, "Pool")) and os.path.isdir(os.path.join(d, "Config"))
 
@@ -198,13 +241,14 @@ class Archive:
         c, p = f"{self.root}/Config", f"{self.root}/Pool"
         self.vf = Cat(f"{c}/virtual_file.index")
         self.ci = Cat(f"{p}/chunk_index")
-        self.fc: dict[int, Cat] = {}
+        self._fc_dirs: dict[int, str] = {}
         for e in os.listdir(c):
             m = re.match(r"^file_chunk(\d+)\.index$", e)
             if m and os.path.isdir(os.path.join(c, e)):
-                self.fc[int(m.group(1))] = Cat(os.path.join(c, e))
-        if not self.fc:
+                self._fc_dirs[int(m.group(1))] = os.path.join(c, e)
+        if not self._fc_dirs:
             raise FileNotFoundError(f"no file_chunk<N>.index directories in {c}")
+        self.fc: dict[int, Cat] = _LazyCats(self._fc_dirs)
         pools = [e for e in os.listdir(p) if e.isdigit() and os.path.isdir(os.path.join(p, e))]
         if not pools:
             raise FileNotFoundError(f"no numbered pool directory under {p}")

{hbkit-0.2.0 → hbkit-0.3.0}/src/hbkit/cli.py

@@ -8,12 +8,19 @@
   hbk <archive> verify <glob>      [-j N]            integrity-check, write nothing
   hbk <archive> tui                                  browse in the full-screen UI
 
+  hbk mount   <remote:bucket> <dir> [--for browse|restore] [--no-warm]
+  hbk warm    <dir>                                  pre-cache dir listings
+  hbk unmount <dir>                                          unmount it
+  hbk remotes                                                list configured rclone remotes
+
 Encrypted archives: add -p/--password, or set HBK_PASSWORD, or you'll be prompted.
 
 <archive> is a .hbk directory, or any drive/folder containing one.
 Globs match the full archive path, which starts with the share name.
 Extraction is resumable: correctly-sized files are skipped.
 
+  hbk mount r2:mybucket ~/mnt/backup --for browse
+  hbk ~/mnt/backup/target.hbk doctor
   hbk /Volumes/Backup doctor
   hbk /Volumes/Backup list holiday
   hbk /Volumes/Backup get "/Photos/2019/*" ~/restore
@@ -149,9 +156,47 @@ def main() -> int:
             i = a.index(flag)
             password = a[i + 1]
             del a[i:i + 2]
-    if len(a) < 2:
+    if len(a) < 2 and (not a or a[0] != "remotes"):
         print(__doc__)
         return 1
+    if a[0] in ("mount", "unmount", "remotes", "warm"):
+        from . import mount as m
+        try:
+            if a[0] == "remotes":
+                rs = m.remotes()
+                print("\n".join(f"  {r}" for r in rs) if rs
+                      else "  no rclone remotes configured - run: rclone config")
+                return 0
+            if a[0] == "warm":
+                if len(a) < 2:
+                    print("usage: hbk warm <dir>")
+                    return 1
+                return m.warm(a[1])
+            if a[0] == "unmount":
+                if len(a) < 2:
+                    print("usage: hbk unmount <dir>")
+                    return 1
+                return m.unmount(a[1])
+            if len(a) < 3:
+                print("usage: hbk mount <remote:bucket> <dir> [--for browse|restore]")
+                return 1
+            prof, rest = "browse", a[3:]
+            if "--for" in rest:
+                i = rest.index("--for")
+                prof = rest[i + 1]
+            cache = None
+            if "--cache-size" in rest:
+                cache = rest[rest.index("--cache-size") + 1]
+            rc = m.mount(a[1], a[2], prof, cache, dry_run="--dry-run" in rest)
+            if rc == 0 and "--dry-run" not in rest and "--no-warm" not in rest:
+                print("\npre-warming directory cache (first open over a network mount "
+                      "is slow; doing it here so later commands are instant)")
+                m.warm(a[2])
+            return rc
+        except (m.RcloneMissing, ValueError) as e:
+            print(f"{e}", file=sys.stderr)
+            return 2
+
     archive, cmd = a[0], a[1]
 
     if cmd == "doctor":

hbkit-0.3.0/src/hbkit/mount.py

@@ -0,0 +1,174 @@
+"""Convenience wrapper around `rclone mount` for archives kept in object storage.
+
+This stores no credentials and talks to no cloud API - `rclone config` still owns all of
+that. All it does is pick the flags, because the flags are easy to get badly wrong:
+rclone's `--vfs-cache-mode full` fetches *whole files*, and hbkit reads a 32-byte index
+record and a ~5 KB chunk at a time, so a 10 KB extraction can pull ~82 MB. Meanwhile
+`off` issues range requests, which is wrong for the one-time index build (a single
+sequential read of a share database that can be hundreds of MB).
+
+So the mode depends on the job, and that is what `--for` selects:
+
+    browse   -> off,  range requests; cheap metadata, good for cherry-picking files
+    restore  -> full, whole-file fetch acts as read-ahead when pulling a whole folder,
+                capped so it cannot fill the disk
+"""
+from __future__ import annotations
+
+import os
+import shutil
+import subprocess
+import sys
+
+PROFILES = {
+    # name: (vfs-cache-mode, extra flags, one-line rationale)
+    "browse": ("off", [],
+               "range requests - best for opening an archive and pulling a few files"),
+    "restore": ("full", ["--vfs-cache-max-size", "50G"],
+                "whole-file fetch as read-ahead for bulk restores, cache capped at 50G"),
+}
+
+# --no-modtime is the single biggest win. Without it rclone issues a HEAD per object just
+# to fill in modification times for a directory listing, which is ~0.27s per entry: the
+# 2,021-shard chunk_index directory took 546.8s cold. With it, 4.07s - a 134x speedup.
+# hbkit never uses the modtimes of an archive's internal files (restored file mtimes come
+# from the archive's own SQLite metadata), so dropping them costs nothing.
+COMMON = ["--read-only", "--no-modtime", "--dir-cache-time", "72h", "--daemon"]
+
+
+class RcloneMissing(Exception):
+    pass
+
+
+def require_rclone() -> str:
+    p = shutil.which("rclone")
+    if not p:
+        raise RcloneMissing(
+            "rclone is not installed. Install it (brew install rclone / "
+            "apt install rclone), then run `rclone config` to add your S3/R2 remote.")
+    return p
+
+
+def remotes() -> list[str]:
+    """Configured rclone remotes, e.g. ['r2:', 's3:']."""
+    try:
+        out = subprocess.run([require_rclone(), "listremotes"],
+                             capture_output=True, text=True, timeout=15)
+        return [x for x in out.stdout.split() if x]
+    except (RcloneMissing, subprocess.SubprocessError):
+        return []
+
+
+def is_mounted(path: str) -> bool:
+    path = os.path.abspath(os.path.expanduser(path))
+    try:
+        out = subprocess.run(["mount"], capture_output=True, text=True, timeout=10).stdout
+    except subprocess.SubprocessError:
+        return False
+    return any(f" {path} " in ln or ln.endswith(f" {path}") for ln in out.splitlines())
+
+
+def build_command(remote: str, mountpoint: str, profile: str = "browse",
+                  cache_size: str | None = None) -> list[str]:
+    if profile not in PROFILES:
+        raise ValueError(f"unknown profile {profile!r}; choose from {', '.join(PROFILES)}")
+    mode, extra, _ = PROFILES[profile]
+    extra = list(extra)
+    if cache_size and mode == "full":
+        extra = ["--vfs-cache-max-size", cache_size]
+    return ([require_rclone(), "mount", remote, os.path.abspath(os.path.expanduser(mountpoint)),
+             "--vfs-cache-mode", mode] + extra + COMMON)
+
+
+def mount(remote: str, mountpoint: str, profile: str = "browse",
+          cache_size: str | None = None, dry_run: bool = False) -> int:
+    mp = os.path.abspath(os.path.expanduser(mountpoint))
+    if is_mounted(mp):
+        print(f"already mounted: {mp}")
+        return 0
+    cmd = build_command(remote, mp, profile, cache_size)
+    mode, _, why = PROFILES[profile]
+    print(f"profile {profile}: --vfs-cache-mode {mode}  ({why})")
+    print("  " + " ".join(cmd))
+    if dry_run:
+        return 0
+    os.makedirs(mp, exist_ok=True)
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        print((r.stderr or r.stdout).strip(), file=sys.stderr)
+        return r.returncode
+    # --daemon returns immediately; wait for the mountpoint to become live
+    import time
+    for _ in range(60):
+        if is_mounted(mp) and os.listdir(mp):
+            break
+        time.sleep(0.5)
+    if not is_mounted(mp):
+        print(f"rclone returned success but {mp} is not mounted", file=sys.stderr)
+        return 1
+    print(f"mounted {remote} at {mp}")
+    for e in sorted(os.listdir(mp))[:10]:
+        print(f"    {e}")
+    return 0
+
+
+# Directories hbkit lists when opening an archive. rclone caches a listing for
+# --dir-cache-time, so walking these once up front turns a ~9 minute "did it hang?" on
+# the user's first command into a visible, attributable wait here. Measured on a 3 TB
+# archive over R2: 9m21s cold, then 1.6s, then 0.07s.
+def warm(mountpoint: str, quiet: bool = False) -> int:
+    """Populate rclone's directory cache for every .hbk archive under `mountpoint`."""
+    import time
+    mp = os.path.abspath(os.path.expanduser(mountpoint))
+    archives = []
+    for e in sorted(os.listdir(mp)):
+        d = os.path.join(mp, e)
+        if os.path.isdir(d) and os.path.isdir(os.path.join(d, "Config")):
+            archives.append(d)
+    if not archives:
+        return 0
+    n = 0
+    t0 = time.time()
+    for arc in archives:
+        targets = [arc, os.path.join(arc, "Config"), os.path.join(arc, "Pool")]
+        for sub in ("Config/virtual_file.index", "Pool/chunk_index", "Config/@Share"):
+            targets.append(os.path.join(arc, sub))
+        cfg = os.path.join(arc, "Config")
+        try:
+            targets += [os.path.join(cfg, x) for x in os.listdir(cfg)
+                        if x.startswith("file_chunk") and os.path.isdir(os.path.join(cfg, x))]
+        except OSError:
+            pass
+        share = os.path.join(arc, "Config", "@Share")
+        if os.path.isdir(share):
+            targets += [os.path.join(share, x) for x in os.listdir(share)]
+        for t in targets:
+            if not os.path.isdir(t):
+                continue
+            if not quiet:
+                print(f"  warming {os.path.relpath(t, mp)} ...", end="", flush=True)
+            s0 = time.time()
+            try:
+                c = len(os.listdir(t))
+            except OSError as e:
+                c = -1
+            if not quiet:
+                print(f" {c} entries, {time.time()-s0:.1f}s")
+            n += 1
+    if not quiet:
+        print(f"warmed {n} directories in {time.time()-t0:.0f}s "
+              f"(cached for --dir-cache-time; later opens are instant)")
+    return 0
+
+
+def unmount(mountpoint: str) -> int:
+    mp = os.path.abspath(os.path.expanduser(mountpoint))
+    if not is_mounted(mp):
+        print(f"not mounted: {mp}")
+        return 0
+    for cmd in (["umount", mp], ["fusermount", "-u", mp], ["diskutil", "unmount", "force", mp]):
+        if shutil.which(cmd[0]) and subprocess.run(cmd, capture_output=True).returncode == 0:
+            print(f"unmounted {mp}")
+            return 0
+    print(f"could not unmount {mp}", file=sys.stderr)
+    return 1

{hbkit-0.2.0 → hbkit-0.3.0}/tests/test_hbkit.py

@@ -229,3 +229,112 @@ def test_encrypted_names_and_content():
         data = arc.extract(ovf, size, verify=True)   # verify=True checks every chunk MD5
         assert len(data) == size, path
     arc.close()
+
+
+# ------------------------------------------------- shard maths (no archive needed)
+
+def _make_shards(tmp_path, sizes, shard_size, monkeypatch):
+    """Write synthetic <N>.idx.2 shards. Shard 0 carries a valid 64-byte header, as real
+    archives do; the header is part of the logical stream so offsets stay comparable."""
+    import struct
+
+    from hbkit import archive as A
+    monkeypatch.setattr(A, "SHARD_SIZE", shard_size)
+    total = sum(sizes)
+    header = struct.pack(">16I", A.MAGIC, 0, 0, 0, 32, total >> 32, total & 0xFFFFFFFF,
+                         *([0] * 9))
+    blob = b""
+    for i, n in enumerate(sizes):
+        if i == 0:
+            part = header + bytes((64 + j) % 251 for j in range(n - 64))
+        else:
+            part = bytes((len(blob) + j) % 251 for j in range(n))
+        (tmp_path / f"{i}.idx.2").write_bytes(part)
+        blob += part
+    return blob
+
+
+def test_cat_computes_offsets_without_stating_every_shard(tmp_path, monkeypatch):
+    """Sizes/offsets must be derived from the fixed shard size + one final stat."""
+    from hbkit import archive as A
+    blob = _make_shards(tmp_path, [128, 128, 10], 128, monkeypatch)
+    c = A.Cat(str(tmp_path))
+    assert len(c.files) == 3
+    assert c.total == len(blob) == 266
+    assert c.starts == [0, 128, 256]
+    assert c.sizes == [128, 128, 10]
+    assert c.record_size == 32          # read from the header, not guessed
+
+
+def test_cat_reads_across_shard_boundaries(tmp_path, monkeypatch):
+    """The multi-shard read path: every offset/length combination must match one blob."""
+    from hbkit import archive as A
+    blob = _make_shards(tmp_path, [128, 128, 10], 128, monkeypatch)
+    c = A.Cat(str(tmp_path))
+    for off in range(0, len(blob), 7):
+        for n in (1, 5, 127, 128, 129, 260):
+            assert c.read(off, n) == blob[off:off + n], f"off={off} n={n}"
+    assert c.read(0, len(blob)) == blob          # whole stream in one read
+    assert c.read(len(blob), 10) == b""          # past the end
+    c.close()
+
+
+def test_cat_single_shard_still_works(tmp_path, monkeypatch):
+    from hbkit import archive as A
+    blob = _make_shards(tmp_path, [100], 128, monkeypatch)
+    c = A.Cat(str(tmp_path))
+    assert c.total == 100 and c.read(70, 20) == blob[70:90]
+    c.close()
+
+
+# --------------------------------------------------- rclone mount helper (no network)
+
+def test_mount_profiles_pick_the_right_cache_mode(monkeypatch):
+    from hbkit import mount as m
+    monkeypatch.setattr(m, "require_rclone", lambda: "/usr/bin/rclone")
+    browse = m.build_command("r2:b", "/tmp/mp", "browse")
+    restore = m.build_command("r2:b", "/tmp/mp", "restore")
+    assert "off" == browse[browse.index("--vfs-cache-mode") + 1]
+    assert "full" == restore[restore.index("--vfs-cache-mode") + 1]
+    # bulk restores cache whole ~50MB buckets, so the cache MUST be capped
+    assert "--vfs-cache-max-size" in restore
+    assert "--vfs-cache-max-size" not in browse
+
+
+def test_mount_command_is_always_read_only_and_skips_modtimes(monkeypatch):
+    """--no-modtime avoids a HEAD per object: 2021-entry listing 546.8s -> 4.07s."""
+    from hbkit import mount as m
+    monkeypatch.setattr(m, "require_rclone", lambda: "/usr/bin/rclone")
+    for profile in ("browse", "restore"):
+        cmd = m.build_command("r2:b", "/tmp/mp", profile)
+        assert "--read-only" in cmd, profile
+        assert "--no-modtime" in cmd, profile
+        assert "--dir-cache-time" in cmd, profile
+
+
+def test_mount_rejects_unknown_profile(monkeypatch):
+    from hbkit import mount as m
+    monkeypatch.setattr(m, "require_rclone", lambda: "/usr/bin/rclone")
+    with pytest.raises(ValueError):
+        m.build_command("r2:b", "/tmp/mp", "nonsense")
+
+
+def test_mount_cache_size_override(monkeypatch):
+    from hbkit import mount as m
+    monkeypatch.setattr(m, "require_rclone", lambda: "/usr/bin/rclone")
+    cmd = m.build_command("r2:b", "/tmp/mp", "restore", cache_size="10G")
+    assert cmd[cmd.index("--vfs-cache-max-size") + 1] == "10G"
+
+
+def test_missing_rclone_is_a_clear_error(monkeypatch):
+    from hbkit import mount as m
+    monkeypatch.setattr(m.shutil, "which", lambda _: None)
+    with pytest.raises(m.RcloneMissing) as e:
+        m.require_rclone()
+    assert "rclone config" in str(e.value)
+
+
+def test_warm_ignores_a_directory_with_no_archives(tmp_path):
+    from hbkit import mount as m
+    (tmp_path / "not-an-archive").mkdir()
+    assert m.warm(str(tmp_path), quiet=True) == 0