hbkit 0.4.7__tar.gz → 0.5.0__tar.gz

hbkit-0.5.0/CHANGELOG.md

@@ -0,0 +1,46 @@
+# Changelog
+
+## 0.5.0
+
+### New
+
+- **Backup versions.** `list`, `get`, `verify`, `info`, `doctor` and `tui` read any
+  version: `--version N`, `--version @YYYY-MM-DD[ HH:MM[:SS]]` / `--before '…'` (newest
+  complete version that started no later than that local time), `--allow-incomplete`.
+  The default is still the newest version, but now the newest *complete* one, read for
+  every share alike.
+- **`hbk <archive> versions [--porcelain]`** lists versions with start time and status.
+  It needs no password.
+- `list` accepts the same glob as `get` when the pattern contains `*`, `?` or `[`.
+- `--` ends the options.
+- Each backup version gets its own cached index; superseded ones are deleted.
+
+### Changed — check your scripts
+
+Exit codes are now distinct and documented (README, "Exit codes"):
+
+| situation | 0.4.8 | 0.5.0 |
+|---|---|---|
+| password missing (non-interactive) | 1 | 3 |
+| wrong password | 1 or 2 | 3 |
+| glob/pattern matched nothing (`get`, `verify`) | 1 | 5 |
+| pattern matched nothing (`list`) | 0 | 5 |
+| not enough disk space | 1 | 6 |
+| usage error / usage text shown | 1 | 2 |
+| interrupted `get`/`verify` | 0 or 1 | 130 |
+| unexpected internal error | 1 | 2 |
+| requested version missing / not complete | — | 4 |
+| output pipe closed (`… \| head`), `list`/`versions`/`info` | 1 (traceback) | 0 |
+| output pipe closed, `get`/`verify` | 1 (traceback) | 141 |
+
+Arguments and options a command used to ignore are refused with code 2: a stray word
+after the destination (an unquoted path with a space), `--strict` outside `get`, `-j` on
+`list`/`info`/`doctor`, a repeated option, `--porcelain` outside `versions`.
+
+`hbk --version` alone still prints hbkit's version; `hbk --version 3 <archive> …` now
+selects backup version 3.
+
+### Fixed
+
+- Resuming into a directory that holds another version's restore no longer keeps a file
+  just because its size matches; the restore record now names each file's source.

{hbkit-0.4.7 → hbkit-0.5.0}/FORMAT.md

@@ -231,7 +231,75 @@ Reconstruct paths by joining `pname_id_v2 -> name_id_v2`. Roots are rows whose p
 absent from the table or which are self-parented. `mode` is a POSIX mode word — use
 `S_ISDIR` to separate directories from files.
 
-`Config/version_info.db` lists backup versions with timestamps and completion status.
+### 5.1 Backup versions [V for the columns below, I for their meaning]
+
+Each backup run is a **version** with an integer id (1, 2, 3, …). A version writes one
+`Config/@Share/<share>/<id>.db` per share it covered. Everything else — `virtual_file`,
+`file_chunk`, `chunk_index`, the buckets — is shared by all versions, so the same file in
+two versions differs only in its `version_list` row (`size`, `mtime_sec`,
+`off_virtual_file`). Reading an older version means reading an older `<id>.db`; nothing
+below the share databases changes.
+
+`Config/version_info.db`, observed on Hyper Backup 4.2.2-4262 (DSM 7, two versions):
+
+```sql
+CREATE TABLE version_info (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,  -- the version id, as in <id>.db
+  timestamp INTEGER,                     -- unix seconds, when the version started
+  name TEXT, source TEXT,                -- source is encrypted on encrypted archives
+  share TEXT,                            -- "share1,share2," - shares in this version
+  status TEXT,                           -- 'Complete' for a finished version
+  statistics TEXT,                       -- JSON: sizes and per-kind counts
+  diff_size INTEGER, locked INTEGER NOT NULL default 0, share_info BLOB,
+  missing_src TEXT, tag_db_magic BLOB, tag_db_file_size_thr INTEGER,
+  has_suspend_dup INTEGER DEFAULT 0, bad_share TEXT DEFAULT '',
+  internal_share TEXT DEFAULT '',
+  suspend_history TEXT DEFAULT '',       -- JSON events: create, complete, ...
+  enc_cksum BLOB DEFAULT NULL, depose_time INTEGER DEFAULT -1);
+```
+
+Only `id`, `timestamp` and `status` are relied on. `'Complete'` and `'Failure'` are the
+status values seen so far; hbkit treats **any value other than `'Complete'` as not
+complete** rather than guessing what it means. `timestamp` matched the `create` event in
+`suspend_history`. [V]
+
+A second archive (Hyper Backup 4.2.2-4262, encrypted) was read at 8 and again at 11
+versions. Version 8 is a `Failure`: a run that stopped on an error. Its `suspend_history`
+ends in event 5 `fail` where finished versions end in event 6 `complete`, and in the
+8-version copy its `<id>.db` file lists existed, holding what was backed up before the
+error. Reading it would silently give an incomplete restore, which is why it is never the
+default. The next successful run became version 9 and deleted version 8's `<id>.db` files
+(`Config/version_delete.prog` was written then, with `is_rotate=false`), but left its
+`version_info` row, now naming shares that have no file list for it. `id` is
+`AUTOINCREMENT`, so ids are never reused. [V for the values]
+
+Version 11 is a run Hyper Backup reported as **partially completed**, because a source
+folder of the task had been renamed away. It is recorded as `'Complete'`, with event 6
+`complete` in `suspend_history`, like any finished version. Only `missing_src` differs: it
+holds the missing source in the same `/*eNC00*...*/` form as `source` (`NULL` on every
+other version). Its file lists omit that folder and are otherwise whole, so hbkit reads it
+by default. Whether other causes of "partially completed" (unreadable files, say) are also
+recorded as `'Complete'` is not known. hbkit cannot decrypt `missing_src` yet: it is not
+encoded like file names. [V for the values, ? for other causes and the encoding]
+
+`Config/@Share/<share>/complete_list.db` holds `complete_list(version_id INTEGER PRIMARY
+KEY)`. It is **not** a reliable completion record: on the same archive the `@AppConfig`
+share's list was empty although both versions were `Complete` in `version_info`, while
+the data share listed both. On the second archive (11 versions) the data share listed
+every `Complete` version but not the `Failure` one, and `@AppConfig` was again empty. hbkit
+does not use it. [V for the observation, ? for meaning]
+
+On that archive version 1 had `depose_time` set to the start time of version 2, while
+version 2 kept the default `-1`. On the second archive each `Complete` version had the
+start time of the next **`Complete`** version (version 7 holds version 9's, skipping the
+failed 8), the newest `Complete` version had `0` and the `Failure` version `-1`: it looks
+like "superseded as the newest good version at". The meaning is not known and hbkit
+ignores it, as it does `bad_share` and `locked`.
+[V for the values, ? for meaning]
+
+The same archive also showed `Pool/vkey.db` with one row per version (`version_id` 1, 2),
+and index shards with a `_dup` sibling (`0.idx_dup`) instead of a generation suffix; the
+`_dup` copies are ignored.
 
 Entries named `@eaDir`, or ending `@SynoEAStream` / `@SynoResource`, are Synology
 metadata — thumbnails, extended attributes, resource forks — not user data. In the

{hbkit-0.4.7 → hbkit-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hbkit
-Version: 0.4.7
+Version: 0.5.0
 Summary: Recover files from Synology Hyper Backup (.hbk) archives without Synology software
 Project-URL: Homepage, https://github.com/YordiLorenzo/hbkit
 Project-URL: Source, https://github.com/YordiLorenzo/hbkit
@@ -97,8 +97,9 @@ live progress bar, throughput, ETA and a failure log.
 
 ```sh
 hbk <archive> doctor                            # probe an archive, prove it's readable
-hbk <archive> info                              # task name, codec, shares, encryption
-hbk <archive> list [pattern]                    # search the file index
+hbk <archive> info                              # task name, codec, shares, versions
+hbk <archive> versions [--porcelain]            # backup versions, start times, status
+hbk <archive> list [pattern|glob]               # search the file index
 hbk <archive> get <glob> <dest> [-j N]          # extract, preserving tree and mtimes
 hbk <archive> get <glob> <dest> --strict        # ...re-verifying checksums on resume
 hbk <archive> verify <glob> [-j N]              # integrity-check, write nothing
@@ -114,7 +115,81 @@ For encrypted archives pass `-p/--password`, set `HBK_PASSWORD`, or let it promp
 shows a password field when it detects encryption.
 
 `<archive>` is a `.hbk` directory, or any drive or folder containing one — it will find it.
-Globs match the full archive path, which begins with the share name.
+Globs match the full archive path, which begins with a slash and the share name
+(`"/Photos/*"`, not `"Photos/*"`). `list` takes either plain text (case-insensitive
+substring) or, if it contains `*`, `?` or `[`, the same glob `get` would use.
+
+### Backup versions
+
+Hyper Backup keeps many versions of the same data. `list`, `get`, `verify`, `info`,
+`doctor` and `tui` read the **newest complete** version unless told otherwise:
+
+| option | reads |
+|---|---|
+| *(none)* | the newest complete version |
+| `--version N` | exactly version `N` (the numbers `versions` prints) |
+| `--version @2026-10-15` | the newest complete version that started **by the end of** that day |
+| `--version '@2026-10-15 03:30'` | … that started no later than 03:30:59 that day |
+| `--before '2026-10-15 23:59'` | the same as `--version '@2026-10-15 23:59'` |
+| `--allow-incomplete` | also accept a version whose backup did not finish |
+
+Dates are in the local time of the machine running `hbk` (set `TZ` to change it); a bare
+date runs to local midnight (also on daylight-saving days), a minute to :59, a time with
+seconds is exact. A clock time that does not exist on a spring-forward night is read as
+the hour after; one that occurs twice in autumn means its first occurrence. A version
+counts as complete only if Hyper Backup recorded its status
+as `Complete`, which it also does for a "partially completed" backup, at least one that
+skipped a missing source folder; an unfinished or failed one is refused unless you pass `--allow-incomplete`, which
+on its own reads the newest version whether or not it finished. If no version is
+complete, commands exit with code 4 rather than showing an empty archive.
+
+Every share is read from the **same** version, so the default and `--version N` for the
+newest complete `N` give identical results; a share that was not part of that version is
+absent rather than silently taken from an older one. Archives without
+`Config/version_info.db` fall back to the newest database of each share. If the file is
+there but cannot be read (damaged, an unknown layout, caught mid-write, or with no
+versions in it), hbkit cannot tell finished versions from unfinished ones, so it refuses
+(code 2) unless you pass
+`--allow-incomplete`, which then reads the newest file list of each share and says that
+completion was not checked. Whenever there is a choice, the version being read is
+printed on stderr.
+
+```sh
+hbk "$HBK_DIR" versions                         # human-readable, marks the default
+hbk "$HBK_DIR" versions --porcelain             # id<TAB>ISO time<TAB>unix time<TAB>status
+hbk "$HBK_DIR" list "/db_staging/firebird/*" --version @2026-10-15
+hbk "$HBK_DIR" get  "/db_staging/firebird/*.gbk" "$TMP" --version @2026-10-15 -j 8
+```
+
+`--porcelain` prints one line per version, oldest first, for example
+`12	2026-10-15T03:00:04+02:00	1792026004	Complete`. `versions` needs no password: the
+version list is stored unencrypted. Each version gets its own index in the cache, so
+switching between them only builds each index once. A rebuilt index replaces the stale
+one for the same archive and version.
+
+### Exit codes
+
+Scripts can rely on these:
+
+| code | meaning |
+|---|---|
+| 0 | success |
+| 1 | one or more files failed to extract or verify — a checksum mismatch or an unreadable chunk; the failures are listed on stdout |
+| 2 | usage error (including a malformed `--version`), the archive is missing, unreadable or of an unsupported layout, or an unexpected internal error |
+| 3 | the archive is encrypted and no password was given, or it is wrong |
+| 4 | the requested version does not exist, is not complete, or no version matches the date |
+| 5 | the glob (`get`, `verify`) or pattern (`list`) matched no file |
+| 6 | the destination does not have enough free space (`get`) |
+| 130 | interrupted (Ctrl-C) |
+| 141 | `get`, `verify` or another command that acts rather than only prints stopped because its output pipe closed (`… \| head`) before it finished; the outcome is unknown. `list`, `versions` and `info` exit 0 then; a pipe that closes after a command finished keeps that command's code (Linux and macOS; on Windows a closed pipe is a non-zero I/O error for every command) |
+
+`doctor` has its own simple contract: 0 when the archive is recoverable, 1 otherwise, with
+the reason in its report. Any argument or option a command would ignore — a second
+`--version`, `--strict` outside `get`, a stray word after the destination (an unquoted
+path with a space) — is refused with code 2 rather than silently dropped. `--version`
+on its own prints hbkit's version; followed by anything it selects a backup version. To
+search for a name that starts with `-`, end the options with `--`:
+`hbk <archive> list -- -draft`.
 
 **Start with `doctor`.** It reports the layout it found and then *proves* the archive is
 readable by rebuilding a random sample of real files with full checksum verification:
@@ -140,7 +215,12 @@ readable by rebuilding a random sample of real files with full checksum verifica
 
 ## Behaviour worth knowing
 
-- **Resumable.** Correctly-sized files are skipped, so re-running a big job is cheap.
+- **Resumable.** Correctly-sized files are skipped, so re-running a big job is cheap. A
+  file recorded as restored from a *different* backup version is extracted again even if
+  its size matches, so restoring another version into the same directory is safe. A file
+  whose record does not name its source — restored by hbkit before 0.5 — is extracted
+  again once too, since nothing says which version it came from. (A file with no record
+  at all, e.g. copied in by hand, is still judged by size alone; `--strict` refuses it.)
 - **Crash-safe.** Files are written to `.part` and atomically renamed, so an interrupted
   run never leaves a truncated file that a later resume would trust.
 - **Layout preserved.** Output goes to `<dest>/<share>/<original path>` with original mtimes.
@@ -148,7 +228,7 @@ readable by rebuilding a random sample of real files with full checksum verifica
 - **Sidecars skipped.** `@eaDir`, `@SynoEAStream` and `@SynoResource` are Synology
   metadata — thumbnails and xattr streams, not your data. In one real archive they were
   half of all entries but under 1% of the bytes.
-- **Index cached** per archive in `~/.cache/hbkit` (override with `HBK_CACHE`), rebuilt
+- **Index cached** per archive and version in `~/.cache/hbkit` (override with `HBK_CACHE`), rebuilt
   automatically when the archive changes. Browsing 1.1M files is instant after the first
   open; building it takes ~10s locally, ~48s over a network mount.
 
@@ -271,6 +351,10 @@ Read this before trusting it with the only copy of anything.
   archive (unencrypted, LZ4, single version, single pool) and a small encrypted one from
   the same DSM version, both verified byte-exact, plus a 7-version encrypted archive from
   Hyper Backup 4.2.2 reported in [#4](https://github.com/YordiLorenzo/hbkit/issues/4).
+  Version selection was checked byte-exact on a small two-version encrypted Hyper Backup
+  4.2.2 archive (each version's copy of a changed file, by number and by date), and on
+  synthetic archives; the version list comes from `Config/version_info.db`, whose layout
+  is checked before use.
   Older record layouts (16-byte `chunk_index`, 28-byte bucket records) and zlib chunks
   are implemented from disassembly but have not met a real archive of that kind. `doctor`
   exists precisely so you can find out in seconds rather than mid-restore.

{hbkit-0.4.7 → hbkit-0.5.0}/README.md

@@ -68,8 +68,9 @@ live progress bar, throughput, ETA and a failure log.
 
 ```sh
 hbk <archive> doctor                            # probe an archive, prove it's readable
-hbk <archive> info                              # task name, codec, shares, encryption
-hbk <archive> list [pattern]                    # search the file index
+hbk <archive> info                              # task name, codec, shares, versions
+hbk <archive> versions [--porcelain]            # backup versions, start times, status
+hbk <archive> list [pattern|glob]               # search the file index
 hbk <archive> get <glob> <dest> [-j N]          # extract, preserving tree and mtimes
 hbk <archive> get <glob> <dest> --strict        # ...re-verifying checksums on resume
 hbk <archive> verify <glob> [-j N]              # integrity-check, write nothing
@@ -85,7 +86,81 @@ For encrypted archives pass `-p/--password`, set `HBK_PASSWORD`, or let it promp
 shows a password field when it detects encryption.
 
 `<archive>` is a `.hbk` directory, or any drive or folder containing one — it will find it.
-Globs match the full archive path, which begins with the share name.
+Globs match the full archive path, which begins with a slash and the share name
+(`"/Photos/*"`, not `"Photos/*"`). `list` takes either plain text (case-insensitive
+substring) or, if it contains `*`, `?` or `[`, the same glob `get` would use.
+
+### Backup versions
+
+Hyper Backup keeps many versions of the same data. `list`, `get`, `verify`, `info`,
+`doctor` and `tui` read the **newest complete** version unless told otherwise:
+
+| option | reads |
+|---|---|
+| *(none)* | the newest complete version |
+| `--version N` | exactly version `N` (the numbers `versions` prints) |
+| `--version @2026-10-15` | the newest complete version that started **by the end of** that day |
+| `--version '@2026-10-15 03:30'` | … that started no later than 03:30:59 that day |
+| `--before '2026-10-15 23:59'` | the same as `--version '@2026-10-15 23:59'` |
+| `--allow-incomplete` | also accept a version whose backup did not finish |
+
+Dates are in the local time of the machine running `hbk` (set `TZ` to change it); a bare
+date runs to local midnight (also on daylight-saving days), a minute to :59, a time with
+seconds is exact. A clock time that does not exist on a spring-forward night is read as
+the hour after; one that occurs twice in autumn means its first occurrence. A version
+counts as complete only if Hyper Backup recorded its status
+as `Complete`, which it also does for a "partially completed" backup, at least one that
+skipped a missing source folder; an unfinished or failed one is refused unless you pass `--allow-incomplete`, which
+on its own reads the newest version whether or not it finished. If no version is
+complete, commands exit with code 4 rather than showing an empty archive.
+
+Every share is read from the **same** version, so the default and `--version N` for the
+newest complete `N` give identical results; a share that was not part of that version is
+absent rather than silently taken from an older one. Archives without
+`Config/version_info.db` fall back to the newest database of each share. If the file is
+there but cannot be read (damaged, an unknown layout, caught mid-write, or with no
+versions in it), hbkit cannot tell finished versions from unfinished ones, so it refuses
+(code 2) unless you pass
+`--allow-incomplete`, which then reads the newest file list of each share and says that
+completion was not checked. Whenever there is a choice, the version being read is
+printed on stderr.
+
+```sh
+hbk "$HBK_DIR" versions                         # human-readable, marks the default
+hbk "$HBK_DIR" versions --porcelain             # id<TAB>ISO time<TAB>unix time<TAB>status
+hbk "$HBK_DIR" list "/db_staging/firebird/*" --version @2026-10-15
+hbk "$HBK_DIR" get  "/db_staging/firebird/*.gbk" "$TMP" --version @2026-10-15 -j 8
+```
+
+`--porcelain` prints one line per version, oldest first, for example
+`12	2026-10-15T03:00:04+02:00	1792026004	Complete`. `versions` needs no password: the
+version list is stored unencrypted. Each version gets its own index in the cache, so
+switching between them only builds each index once. A rebuilt index replaces the stale
+one for the same archive and version.
+
+### Exit codes
+
+Scripts can rely on these:
+
+| code | meaning |
+|---|---|
+| 0 | success |
+| 1 | one or more files failed to extract or verify — a checksum mismatch or an unreadable chunk; the failures are listed on stdout |
+| 2 | usage error (including a malformed `--version`), the archive is missing, unreadable or of an unsupported layout, or an unexpected internal error |
+| 3 | the archive is encrypted and no password was given, or it is wrong |
+| 4 | the requested version does not exist, is not complete, or no version matches the date |
+| 5 | the glob (`get`, `verify`) or pattern (`list`) matched no file |
+| 6 | the destination does not have enough free space (`get`) |
+| 130 | interrupted (Ctrl-C) |
+| 141 | `get`, `verify` or another command that acts rather than only prints stopped because its output pipe closed (`… \| head`) before it finished; the outcome is unknown. `list`, `versions` and `info` exit 0 then; a pipe that closes after a command finished keeps that command's code (Linux and macOS; on Windows a closed pipe is a non-zero I/O error for every command) |
+
+`doctor` has its own simple contract: 0 when the archive is recoverable, 1 otherwise, with
+the reason in its report. Any argument or option a command would ignore — a second
+`--version`, `--strict` outside `get`, a stray word after the destination (an unquoted
+path with a space) — is refused with code 2 rather than silently dropped. `--version`
+on its own prints hbkit's version; followed by anything it selects a backup version. To
+search for a name that starts with `-`, end the options with `--`:
+`hbk <archive> list -- -draft`.
 
 **Start with `doctor`.** It reports the layout it found and then *proves* the archive is
 readable by rebuilding a random sample of real files with full checksum verification:
@@ -111,7 +186,12 @@ readable by rebuilding a random sample of real files with full checksum verifica
 
 ## Behaviour worth knowing
 
-- **Resumable.** Correctly-sized files are skipped, so re-running a big job is cheap.
+- **Resumable.** Correctly-sized files are skipped, so re-running a big job is cheap. A
+  file recorded as restored from a *different* backup version is extracted again even if
+  its size matches, so restoring another version into the same directory is safe. A file
+  whose record does not name its source — restored by hbkit before 0.5 — is extracted
+  again once too, since nothing says which version it came from. (A file with no record
+  at all, e.g. copied in by hand, is still judged by size alone; `--strict` refuses it.)
 - **Crash-safe.** Files are written to `.part` and atomically renamed, so an interrupted
   run never leaves a truncated file that a later resume would trust.
 - **Layout preserved.** Output goes to `<dest>/<share>/<original path>` with original mtimes.
@@ -119,7 +199,7 @@ readable by rebuilding a random sample of real files with full checksum verifica
 - **Sidecars skipped.** `@eaDir`, `@SynoEAStream` and `@SynoResource` are Synology
   metadata — thumbnails and xattr streams, not your data. In one real archive they were
   half of all entries but under 1% of the bytes.
-- **Index cached** per archive in `~/.cache/hbkit` (override with `HBK_CACHE`), rebuilt
+- **Index cached** per archive and version in `~/.cache/hbkit` (override with `HBK_CACHE`), rebuilt
   automatically when the archive changes. Browsing 1.1M files is instant after the first
   open; building it takes ~10s locally, ~48s over a network mount.
 
@@ -242,6 +322,10 @@ Read this before trusting it with the only copy of anything.
   archive (unencrypted, LZ4, single version, single pool) and a small encrypted one from
   the same DSM version, both verified byte-exact, plus a 7-version encrypted archive from
   Hyper Backup 4.2.2 reported in [#4](https://github.com/YordiLorenzo/hbkit/issues/4).
+  Version selection was checked byte-exact on a small two-version encrypted Hyper Backup
+  4.2.2 archive (each version's copy of a changed file, by number and by date), and on
+  synthetic archives; the version list comes from `Config/version_info.db`, whose layout
+  is checked before use.
   Older record layouts (16-byte `chunk_index`, 28-byte bucket records) and zlib chunks
   are implemented from disassembly but have not met a real archive of that kind. `doctor`
   exists precisely so you can find out in seconds rather than mid-restore.

{hbkit-0.4.7 → hbkit-0.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hbkit"
-version = "0.4.7"
+version = "0.5.0"
 description = "Recover files from Synology Hyper Backup (.hbk) archives without Synology software"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -43,4 +43,4 @@ hbk-tui = "hbkit.tui:main"
 packages = ["src/hbkit"]
 
 [tool.hatch.build.targets.sdist]
-include = ["src/hbkit", "tests", "README.md", "FORMAT.md", "LICENSE"]
+include = ["src/hbkit", "tests", "README.md", "FORMAT.md", "CHANGELOG.md", "LICENSE"]

{hbkit-0.4.7 → hbkit-0.5.0}/src/hbkit/__init__.py

@@ -1,6 +1,6 @@
 """hbkit - recover files from Synology Hyper Backup (.hbk) archives without Synology software."""
 
-__version__ = "0.4.7"
+__version__ = "0.5.0"
 
 from .archive import (Archive, NeedPassword, UnsupportedArchive,  # noqa: F401
                       find_archive_root, is_archive)