hbkit 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
hbkit-0.1.0/.gitignore ADDED
@@ -0,0 +1,10 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ .venv/
4
+ venv/
5
+ dist/
6
+ build/
7
+ *.egg-info/
8
+ .pytest_cache/
9
+ .DS_Store
10
+ *.part
hbkit-0.1.0/FORMAT.md ADDED
@@ -0,0 +1,282 @@
1
+ # The Synology Hyper Backup (`.hbk`) on-disk format
2
+
3
+ A reverse-engineered specification, written so that anyone can implement a reader in any
4
+ language. To our knowledge no public description of this format existed before this
5
+ document: searching the distinctive magic `70 53 A8 6E` returns nothing on the open web,
6
+ in file-signature databases, or in DFIR tooling.
7
+
8
+ **Status legend** — every claim below is tagged:
9
+
10
+ - **[V]** Verified. Proven by reconstructing real files byte-exactly and checking them
11
+ against the archive's own MD5 and CRC32 values.
12
+ - **[D]** Disassembly. Read from exported C++ symbols in Synology's own
13
+ `HyperBackupExplorer` binary, but not independently exercised by us.
14
+ - **[I]** Inferred. Consistent with observation, not proven. Treat with suspicion.
15
+ - **[?]** Unknown. Documented so the next person knows where the edges are.
16
+
17
+ Coverage caveat: this was derived from **one** archive — DSM 7, Hyper Backup 4.1.2-4039,
18
+ unencrypted, LZ4, single version, single pool. Variant handling marked [D] is implemented
19
+ from disassembly but has never met a real archive of that kind.
20
+
21
+ ---
22
+
23
+ ## 1. Conventions
24
+
25
+ - **All integers are big-endian.** No exceptions were found. [V]
26
+ - **Generation suffixes.** Most files carry a trailing `.<n>`: `0.idx.2`, `1.db.2`,
27
+ `index_ver.json.1`. A reader should resolve `name`, `name.1`, `name.2`… and prefer the
28
+ highest generation. [V]
29
+ - **macOS junk.** Archives stored on exFAT/HFS accumulate AppleDouble sidecars named
30
+ `._<something>`. These are not part of the format and must be filtered, or they will be
31
+ mistaken for real shards. [V]
32
+
33
+ ## 2. Directory layout
34
+
35
+ ```
36
+ <target>.hbk/
37
+ _Syno_TaskConfig INI-ish task settings (plain text)
38
+ SynologyHyperBackup.bkpi marker, may be zero bytes
39
+ Config/
40
+ index_ver.json[.n] {"major":0,"minor":9,"sub_minor":1}
41
+ target_ver.json[.n]
42
+ version_info.db[.n] SQLite: one row per backup version
43
+ virtual_file.index/ shard dir (56-byte records)
44
+ file_chunk<N>.index/ shard dirs (flat i64 arrays), N = 1..4 observed
45
+ @Share/<share>/
46
+ <version>.db[.n] SQLite: the file tree for that share+version
47
+ complete_list.db[.n] SQLite: which versions completed
48
+ Pool/
49
+ chunk_index/ shard dir (29-byte records, or 16 on older archives)
50
+ bucketID.counter[.n] u64 big-endian: number of buckets
51
+ file_pool/ whole-file dedup store [?]
52
+ <pool>/<dir>/<n>.bucket[.n] ~50 MB of concatenated compressed chunks
53
+ <pool>/<dir>/<n>.index[.n] the bucket's chunk directory
54
+ Control/, Guard/ bookkeeping, not needed to read data
55
+ ```
56
+
57
+ ## 3. The shard container
58
+
59
+ Index families are directories of shards named `<N>.idx[.gen]`, N = 0,1,2,…
60
+ **Concatenate them in numeric order into one logical byte stream.** Shard size is 8 MiB;
61
+ records may straddle a shard boundary, so a reader must treat the family as one stream, not
62
+ as independent files. [V]
63
+
64
+ Only shard 0 carries a **64-byte header**, at stream offset 0. Records therefore begin at
65
+ **stream offset 64**. [V]
66
+
67
+ Header, as 16 big-endian `uint32` words:
68
+
69
+ | word | meaning |
70
+ |---|---|
71
+ | 0 | magic `0x7053A86E` [V] |
72
+ | 1 | kind — 0 virtual_file, 1 chunk_index/file_chunk, 2 bucket index [I] |
73
+ | 2 | variant [I] |
74
+ | 4 | **record size in bytes** [V] |
75
+ | 5:6 | `uint64` total stream length, as `(w5 << 32) | w6` [V] |
76
+
77
+ Word 4 is the important one: it lets a reader support layout variants without hardcoding
78
+ DSM versions. Observed: 56 (virtual_file), 29 (chunk_index), 32 (bucket index), 0 for
79
+ `file_chunk` — which is a flat `int64` array with no per-record framing. [V]
80
+
81
+ The declared length in words 5:6 may slightly exceed the bytes actually present; treat it
82
+ as advisory, not as a truncation check. [I]
83
+
84
+ ## 4. The lookup chain
85
+
86
+ ```
87
+ version_list.off_virtual_file (SQLite, per share)
88
+ -> virtual_file record (56 B)
89
+ -> (shard, offset) into file_chunk<shard>.index
90
+ -> flat array of int64 keys, 8 B each
91
+ -> each key = byte offset of a chunk_index record
92
+ -> chunk_index record -> (bucket_id, bucket_index_offset)
93
+ -> bucket .index record -> (offset, compressed length, uncompressed length, MD5)
94
+ -> bucket data file -> one raw LZ4 block
95
+ ```
96
+
97
+ ### 4.1 virtual_file record — 56 bytes [V for the pointer, D for the rest]
98
+
99
+ `off_virtual_file` from SQLite is a **byte offset into the virtual_file stream**.
100
+
101
+ | offset | size | field |
102
+ |---|---|---|
103
+ | 0 | 8 | **chunk-list pointer** — `(shard << 48) | (byte_offset & 0xFFFF_FFFF_FFFF)` [V] |
104
+ | 8 | 4 | ref_count [D] |
105
+ | 12 | 4 | uid [D] |
106
+ | 16 | 4 | gid [D] |
107
+ | 20 | 8 | atime seconds [D] |
108
+ | 28 | 4 | atime nanoseconds [D] |
109
+ | 32 | 8 | crtime seconds [D] |
110
+ | 40 | 4 | crtime nanoseconds [D] |
111
+ | 44 | 4 | mod_ver, or CRC depending on variant [D] |
112
+ | 48 | 8 | acl_offset, same packed encoding [D] |
113
+
114
+ The packing is exactly what the vendor binary does:
115
+ `FileChunkIndexIdParse(x) = x >> 48`, `FileChunkOffsetParse(x) = x & 0xFFFFFFFFFFFF`. [D]
116
+
117
+ The shard number is **read, not computed** — it varies per file (values 1–4 observed).
118
+
119
+ Note the record carries **no size, no name, no mode, no mtime**. All of that lives in
120
+ SQLite. The extractor needs the file size from `version_list` to know when to stop
121
+ consuming chunks. [V]
122
+
123
+ `off_virtual_file = -1` is a sentinel meaning the file is whole-file deduplicated into
124
+ `Pool/file_pool` rather than chunked. One occurrence in 501,278 files. The file_pool
125
+ format is **not decoded**. [?]
126
+
127
+ ### 4.2 file_chunk arrays [V]
128
+
129
+ At the offset above, the `file_chunk<shard>.index` stream holds a **flat array of
130
+ big-endian `int64` values, 8 bytes each**, with no header and no per-record framing.
131
+ Each value is the **byte offset of a chunk_index record**.
132
+
133
+ Read entries in order, resolve each, and stop when the summed *uncompressed* chunk lengths
134
+ equal the file size from SQLite. There is no terminator and no stored chunk count.
135
+
136
+ Immediately before the array sits a **12-byte per-file header** (the `acl_offset` pointer
137
+ is exactly 12 bytes lower on every file sampled, 1547/1547). Its contents relate to
138
+ ACL/xattr storage and are **not decoded**. [?]
139
+
140
+ ### 4.3 chunk_index record
141
+
142
+ Two layouts, distinguished by the record size in the shard header.
143
+
144
+ **29 bytes (v3)** [V]
145
+
146
+ | offset | size | field |
147
+ |---|---|---|
148
+ | 0 | 1 | mode; **bit 0 set = indirect** |
149
+ | 1 | 8 | if indirect: byte offset of another chunk_index record — follow it |
150
+ | 1 | 4 | if direct: `int32` bucket_id |
151
+ | 5 | 4 | if direct: `int32` byte offset into that bucket's `.index` |
152
+ | 25 | 4 | CRC [?] |
153
+
154
+ The indirect form is content deduplication: many files' chunks redirect to one canonical
155
+ chunk. Follow the chain until bit 0 is clear. Guard the recursion — a malformed archive
156
+ could otherwise loop. [V]
157
+
158
+ **16 bytes (v1/v2)** [D] — older archives:
159
+
160
+ | offset | size | field |
161
+ |---|---|---|
162
+ | 0 | 4 | `int32` bucket_id |
163
+ | 4 | 4 | `int32` bucket index offset |
164
+ | 8 | 4 | ref_count |
165
+ | 12 | 4 | mod_ver (v1) or CRC (v2) |
166
+
167
+ ### 4.4 Bucket addressing [V]
168
+
169
+ ```
170
+ dir = bucket_id >> 11 (2048 buckets per directory)
171
+ file = bucket_id & 0x7FF
172
+ path = Pool/<pool>/<dir>/<file>.bucket[.gen] and .index[.gen]
173
+ ```
174
+
175
+ The directory is part of the address: the same filename `<n>.bucket` exists in *every*
176
+ directory. Getting this wrong is the single easiest way to read the wrong data — and
177
+ because chunk MD5s are checked, it surfaces as a verification failure rather than silent
178
+ corruption.
179
+
180
+ ### 4.5 Bucket index record
181
+
182
+ **32 bytes (LAYOUT_D)** [V]
183
+
184
+ | offset | size | field |
185
+ |---|---|---|
186
+ | 0 | 4 | `uint32` compressed length |
187
+ | 4 | 4 | `uint32` byte offset within the `.bucket` file |
188
+ | 8 | 4 | `uint32` uncompressed length |
189
+ | 12 | 16 | **MD5 of the decompressed chunk** |
190
+ | 28 | 4 | **CRC32 of bytes [0:28] of this record**, poly `0xEDB88320` |
191
+
192
+ **28 bytes (legacy)** — identical without the trailing CRC32. [D]
193
+
194
+ Both checks verified across all 82,313 chunks of a 674 MB file. Earlier public
195
+ descriptions call the 20-byte tail a SHA-1; it is not. It is MD5 + CRC32, confirmed both
196
+ empirically and from the vendor's `getChecksum`, which copies 16 bytes. [V]
197
+
198
+ ### 4.6 Chunk payload [V]
199
+
200
+ At `offset` in the `.bucket` file, read `compressed length` bytes and decompress to exactly
201
+ `uncompressed length`.
202
+
203
+ The payload is a **raw LZ4 block** — *not* an LZ4 frame. There is no frame header and no
204
+ magic; call the block API directly (`LZ4_decompress_safe`) with the known output size.
205
+
206
+ The codec comes from `data_compress_type` in `_Syno_TaskConfig`. The vendor dispatcher
207
+ `SYNO::Backup::decompress(type, …)` branches on **1 = lz4, 2 = lz4-hc, 4 = zlib**; lz4 and
208
+ lz4-hc share a decompressor. A robust reader attempts LZ4 and falls back to zlib. [D for
209
+ the mapping, V for LZ4]
210
+
211
+ Existing open-source tools implement only LZ4 and report corruption on some chunks; the
212
+ zlib path is the likely explanation.
213
+
214
+ ## 5. SQLite metadata [V]
215
+
216
+ `Config/@Share/<share>/<version>.db` holds the file tree:
217
+
218
+ ```sql
219
+ CREATE TABLE version_list (
220
+ name_id_v2 BLOB PRIMARY KEY, -- 20-byte node id
221
+ pname_id_v2 BLOB, -- parent's name_id_v2; join on this to build paths
222
+ off_virtual_file INTEGER, -- byte offset into the virtual_file stream, or -1
223
+ file_name TEXT, size INTEGER, mode INTEGER,
224
+ mtime_sec INTEGER, ctime_sec INTEGER, inode INTEGER, ...
225
+ );
226
+ ```
227
+
228
+ Reconstruct paths by joining `pname_id_v2 -> name_id_v2`. Roots are rows whose parent is
229
+ absent from the table or which are self-parented. `mode` is a POSIX mode word — use
230
+ `S_ISDIR` to separate directories from files.
231
+
232
+ `Config/version_info.db` lists backup versions with timestamps and completion status.
233
+
234
+ Entries named `@eaDir`, or ending `@SynoEAStream` / `@SynoResource`, are Synology
235
+ metadata — thumbnails, extended attributes, resource forks — not user data. In the
236
+ reference archive they were 553,899 of 1,135,405 rows but only ~40 GB of 4.88 TB.
237
+
238
+ ## 6. Verification
239
+
240
+ Every chunk is self-checking: MD5 over the decompressed bytes, plus CRC32 over the index
241
+ record. **A reader that verifies both cannot silently return wrong data** — the worst case
242
+ is a loud failure. This matters more than throughput in a recovery tool, and it is what
243
+ makes it safe to attempt undecoded variants: a wrong guess fails rather than corrupts.
244
+
245
+ ## 7. Encryption
246
+
247
+ `enable_data_encrypt` in `_Syno_TaskConfig` indicates client-side encryption. This
248
+ document covers **unencrypted archives only**. The encrypted variant is not implemented
249
+ here; a 2016 Python 2 script by "mrsandman" and an accompanying synology-forum.de thread
250
+ document the key derivation, and remain the only public reference.
251
+
252
+ ## 8. Not decoded
253
+
254
+ - `Pool/file_pool` whole-file dedup store, and the `off_virtual_file = -1` reference.
255
+ - The 12-byte per-file header preceding each chunk array (ACL/xattr related).
256
+ - The trailing 4 bytes of a v3 chunk_index record.
257
+ - How Synology *chooses* which `file_chunk` shard a file lands in (readers only need to
258
+ read the value, never predict it).
259
+ - Encrypted archives.
260
+
261
+ ## 9. How this was derived
262
+
263
+ Two independent routes, cross-checked against each other:
264
+
265
+ 1. **Empirical.** Anchor on a file whose content could be recognised — a 10,244-byte
266
+ `.DS_Store` that turned out to be exactly two chunks (9232 + 1012) — then generalise and
267
+ confirm by rebuilding progressively larger files until an 674 MB video reproduced
268
+ byte-exactly across 82,313 chunks.
269
+ 2. **Disassembly.** Synology's macOS `HyperBackupExplorer` ships **full C++ symbols**
270
+ (~24k mangled names, plus original source paths). `nm -a` and `objdump -d` on named
271
+ functions such as `ChunkIndexAdapter::getChunkIndexInfo`,
272
+ `VirtualFileAdapter::getVirtualFileInfo` and `VirtualFile::FileChunkOffsetParse` give
273
+ exact field offsets and endianness. This turns guesswork into reading the vendor's own
274
+ field arithmetic.
275
+
276
+ Where the two disagreed, the empirical result won and the disassembly note was corrected —
277
+ that is how the "SHA-1" error was caught.
278
+
279
+ No vendor code is reproduced here. Field offsets, record sizes and wire layouts are facts
280
+ about a data format, not authorship. The independent Rust implementation
281
+ [TeamDman/teamy-hyper-backup-explorer](https://github.com/TeamDman/teamy-hyper-backup-explorer)
282
+ (MPL-2.0) was consulted; its constants agree with what we derived separately.
hbkit-0.1.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 YordiLorenzo
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
hbkit-0.1.0/PKG-INFO ADDED
@@ -0,0 +1,206 @@
1
+ Metadata-Version: 2.4
2
+ Name: hbkit
3
+ Version: 0.1.0
4
+ Summary: Recover files from Synology Hyper Backup (.hbk) archives without Synology software
5
+ Project-URL: Homepage, https://github.com/YordiLorenzo/hbkit
6
+ Project-URL: Source, https://github.com/YordiLorenzo/hbkit
7
+ Project-URL: Issues, https://github.com/YordiLorenzo/hbkit/issues
8
+ Project-URL: Format spec, https://github.com/YordiLorenzo/hbkit/blob/main/FORMAT.md
9
+ Author-email: YordiLorenzo <yordilorenzo@gmail.com>
10
+ License-Expression: MIT
11
+ License-File: LICENSE
12
+ Keywords: backup,data-recovery,dsm,extract,hbk,hyper-backup,nas,recovery,synology
13
+ Classifier: Development Status :: 4 - Beta
14
+ Classifier: Environment :: Console
15
+ Classifier: Intended Audience :: End Users/Desktop
16
+ Classifier: Intended Audience :: System Administrators
17
+ Classifier: Operating System :: MacOS
18
+ Classifier: Operating System :: POSIX :: Linux
19
+ Classifier: Programming Language :: Python :: 3
20
+ Classifier: Topic :: System :: Archiving :: Backup
21
+ Classifier: Topic :: System :: Recovery Tools
22
+ Requires-Python: >=3.10
23
+ Requires-Dist: textual>=0.80
24
+ Provides-Extra: dev
25
+ Requires-Dist: pytest>=7; extra == 'dev'
26
+ Description-Content-Type: text/markdown
27
+
28
+ # hbkit
29
+
30
+ Recover files from **Synology Hyper Backup (`.hbk`)** archives without any Synology software.
31
+
32
+ Point it at a backup — on a local disk, an external drive, or an S3/R2 bucket mounted with
33
+ rclone — browse it as a tree, and pull out what you want. Works headless on Linux and macOS,
34
+ including Apple Silicon, where Synology's own Hyper Backup Explorer is awkward or unavailable.
35
+
36
+ ```sh
37
+ pip install hbkit
38
+
39
+ hbk /Volumes/Backup doctor # can this archive be recovered?
40
+ hbk-tui /Volumes/Backup # browse and select interactively
41
+ hbk /Volumes/Backup get "/Photos/*" ~/restore
42
+ ```
43
+
44
+ ---
45
+
46
+ ## Why
47
+
48
+ Hyper Backup Explorer is a GUI, has no command line, ships x86-only on Linux, and gets
49
+ unhappy with large archives. If your NAS died and the backup is all you have, you want
50
+ something you can point at a drive, script, and trust.
51
+
52
+ `hbkit` reads the format directly. Every chunk it returns has been checked against the
53
+ archive's own MD5 and CRC32, so **it cannot silently hand you corrupt data** — the worst
54
+ case is a loud failure naming the file.
55
+
56
+ ## The TUI
57
+
58
+ ```
59
+ ⭘ Hyper Backup Recovery 17:11:17
60
+ ┌──────────────────────────────────────────────────────────────┐ │
61
+ │ search filename… (/) │ │ Selection
62
+ └──────────────────────────────────────────────────────────────┘ │ 197,607 files
63
+ ▼ ◪ 📁 NAS Volume 1 4.4T 499,745 │ 625.0G in 1 item(s)
64
+ ├─ ▶ ☐ 📁 Archive 2022 24.2G 358 │
65
+ ├─ ▶ ☐ 📁 Backups 23.5G 63,693 │ Destination
66
+ ├─ ▶ ☐ 📁 Video Projects 1.9T 64,473 │ ┌──────────────────────────┐
67
+ ├─ ▶ ☐ 📁 Media Library 1.3T 66,629 │ │ ~/restore │
68
+ ├─ ▼ ☑ 📁 Photo Libraries 625.0G 197,607 │ └──────────────────────────┘
69
+ │ ├─ ▶ ☑ 📁 Photos Library - Laptop… 28.8G 76,844 │ ⚠ needs 625.0G, only 70.3G free
70
+ │ ├─ ▶ ☑ 📁 Photos Library - Old Backup… 19.8G 15,505 │
71
+ │ Start recovery
72
+ a All n Clear d Destination r Recover / Search q Quit │
73
+ ```
74
+
75
+ `space` tick · `a` all · `n` clear · `/` search · `d` destination · `r` recover.
76
+
77
+ Folders show subtree size and file count. Ticking a folder takes its whole subtree; the
78
+ destination panel warns before you start if the selection will not fit. Recovery shows a
79
+ live progress bar, throughput, ETA and a failure log.
80
+
81
+ ## Commands
82
+
83
+ ```sh
84
+ hbk <archive> doctor # probe an unknown archive, prove it's readable
85
+ hbk <archive> info # task name, codec, shares, encryption
86
+ hbk <archive> list [pattern] # search the file index
87
+ hbk <archive> get <glob> <dest> [-j N] # extract, preserving tree and mtimes
88
+ hbk <archive> verify <glob> [-j N] # integrity-check, write nothing
89
+ hbk <archive> tui # same as hbk-tui
90
+ ```
91
+
92
+ `<archive>` is a `.hbk` directory, or any drive or folder containing one — it will find it.
93
+ Globs match the full archive path, which begins with the share name.
94
+
95
+ **Start with `doctor`.** It reports the layout it found and then *proves* the archive is
96
+ readable by rebuilding a random sample of real files with full checksum verification:
97
+
98
+ ```
99
+ archive : /Volumes/Backup/nas_1.hbk
100
+ task : Daily Backup
101
+ source host : nas
102
+ source model : DS...
103
+ chunk codec : lz4
104
+ virtual_file record : 56 B
105
+ chunk_index record : 29 B (v3)
106
+ bucket index record : 32 B (md5+crc32)
107
+ shares : Photos, Documents
108
+
109
+ PASS virtual_file layout known (56 B)
110
+ PASS chunk_index layout known (29 B)
111
+ PASS bucket layout known (32 B)
112
+ PASS rebuilt 9 sampled files, all chunks verified (9 ok, 0 failed)
113
+
114
+ VERDICT: recoverable. Sampled files rebuilt byte-exact and checksum-verified.
115
+ ```
116
+
117
+ ## Behaviour worth knowing
118
+
119
+ - **Resumable.** Correctly-sized files are skipped, so re-running a big job is cheap.
120
+ - **Crash-safe.** Files are written to `.part` and atomically renamed, so an interrupted
121
+ run never leaves a truncated file that a later resume would trust.
122
+ - **Layout preserved.** Output goes to `<dest>/<share>/<original path>` with original mtimes.
123
+ - **Read-only.** Nothing is ever written to the archive.
124
+ - **Sidecars skipped.** `@eaDir`, `@SynoEAStream` and `@SynoResource` are Synology
125
+ metadata — thumbnails and xattr streams, not your data. In one real archive they were
126
+ half of all entries but under 1% of the bytes.
127
+ - **Index cached** per archive in `~/.cache/hbkit`, rebuilt automatically when the archive
128
+ changes. Browsing 1.1M files is instant after the first open.
129
+
130
+ ## Performance
131
+
132
+ Use `-j` to set worker processes (default 8). Threads do not help — extraction is
133
+ GIL-bound in Python, measured flat at ~32 MB/s from 1 to 12 threads — so `hbkit` fans out
134
+ to real processes.
135
+
136
+ Throughput is bounded by the source device, not by `hbkit`. On a USB spinning disk with a
137
+ 92 MB/s sequential ceiling, a cold parallel run reached 58 MB/s while the disk itself sat
138
+ at 49 MB/s; scattered reads across tens of thousands of bucket files never reach sequential
139
+ speed. Work is ordered by locality so each worker sweeps the pool in one direction rather
140
+ than several heads chasing several regions.
141
+
142
+ Media does not compress — measured ratio 1.004 on video. The space saving in a Hyper Backup
143
+ archive comes from cross-file dedup, not per-file compression, so expect bytes-off-disk to
144
+ roughly equal bytes-delivered.
145
+
146
+ ## Scope and limits
147
+
148
+ Read this before trusting it with the only copy of anything.
149
+
150
+ - **Encrypted archives are not supported.** They are detected and refused, never
151
+ half-decoded. If `enable_data_encrypt` is set, this tool will not help you.
152
+ - **Proven against a limited set of archives.** The reference archive is DSM 7,
153
+ Hyper Backup 4.1.2, unencrypted, LZ4, single version, single pool. Older record layouts
154
+ (16-byte `chunk_index`, 28-byte bucket records), zlib chunks and multi-version archives
155
+ are implemented from disassembly but have not met a real archive of that kind. `doctor`
156
+ exists precisely so you can find out in seconds rather than mid-restore.
157
+ - **Unknown layouts are refused, not guessed.** A wrong guess would mean silently wrong
158
+ bytes, which is the one thing a recovery tool must never do.
159
+ - **Whole-file dedup** (`off_virtual_file = -1`, files living in `Pool/file_pool`) is not
160
+ decoded. One file in 501,278 in the reference archive.
161
+ - Requires `liblz4` (`brew install lz4`, or `apt install liblz4-1`). Set `HBK_LZ4` if it
162
+ is somewhere unusual.
163
+
164
+ ## The format
165
+
166
+ [`FORMAT.md`](FORMAT.md) is a full specification of the on-disk format, written so you can
167
+ implement a reader in any language. Every claim is tagged **verified / from disassembly /
168
+ inferred / unknown**, and there is an explicit list of what is still undecoded.
169
+
170
+ As far as we can tell no public description of this format existed before it — searching
171
+ the container magic `70 53 A8 6E` returns nothing on the open web or in file-signature
172
+ databases. If the tool is useless to you, the spec may not be.
173
+
174
+ It was derived two ways and cross-checked: empirically, by anchoring on a file whose bytes
175
+ could be recognised and then rebuilding progressively larger files until a 674 MB video
176
+ reproduced exactly across 82,313 chunks; and by reading exported C++ symbols in Synology's
177
+ own `HyperBackupExplorer` binary, which ships with full symbols and gives exact field
178
+ offsets. Where the two disagreed, the empirical result won.
179
+
180
+ ## Prior art
181
+
182
+ - [TeamDman/teamy-hyper-backup-explorer](https://github.com/TeamDman/teamy-hyper-backup-explorer) — independent Rust implementation (MPL-2.0). Its constants agree with what we derived separately.
183
+ - [mistersandman/hyperbackup_decrypt](https://github.com/mistersandman/hyperbackup_decrypt) — 2016 Python 2 script, and the only public reference for the **encrypted** variant.
184
+
185
+ ## Development
186
+
187
+ ```sh
188
+ git clone https://github.com/YordiLorenzo/hbkit && cd hbkit
189
+ python3 -m venv .venv && ./.venv/bin/pip install -e ".[dev]"
190
+ HBK_TEST_ARCHIVE=/path/to/backup ./.venv/bin/python -m pytest tests -v
191
+ ```
192
+
193
+ The test suite needs a real archive — correctness is checked against the archive's own
194
+ checksums and against file-format markers, so a pass means the bytes are genuinely right,
195
+ not merely the right length. Tests skip cleanly when no archive is available.
196
+
197
+ Contributions especially welcome for: encrypted archives, the legacy record layouts, and
198
+ `Pool/file_pool`. If you have an archive `doctor` cannot read, an issue with its output
199
+ is genuinely useful.
200
+
201
+ ## License
202
+
203
+ MIT — see [LICENSE](LICENSE).
204
+
205
+ Not affiliated with or endorsed by Synology. "Synology" and "Hyper Backup" are trademarks
206
+ of Synology Inc., used here only to describe what this software reads.