hbkit 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- hbkit-0.1.0/.gitignore +10 -0
- hbkit-0.1.0/FORMAT.md +282 -0
- hbkit-0.1.0/LICENSE +21 -0
- hbkit-0.1.0/PKG-INFO +206 -0
- hbkit-0.1.0/README.md +179 -0
- hbkit-0.1.0/pyproject.toml +46 -0
- hbkit-0.1.0/src/hbkit/__init__.py +7 -0
- hbkit-0.1.0/src/hbkit/archive.py +387 -0
- hbkit-0.1.0/src/hbkit/cli.py +206 -0
- hbkit-0.1.0/src/hbkit/doctor.py +208 -0
- hbkit-0.1.0/src/hbkit/index.py +203 -0
- hbkit-0.1.0/src/hbkit/runner.py +192 -0
- hbkit-0.1.0/src/hbkit/tui.py +738 -0
- hbkit-0.1.0/tests/test_hbkit.py +186 -0
hbkit-0.1.0/.gitignore
ADDED
hbkit-0.1.0/FORMAT.md
ADDED
|
@@ -0,0 +1,282 @@
|
|
|
1
|
+
# The Synology Hyper Backup (`.hbk`) on-disk format
|
|
2
|
+
|
|
3
|
+
A reverse-engineered specification, written so that anyone can implement a reader in any
|
|
4
|
+
language. To our knowledge no public description of this format existed before this
|
|
5
|
+
document: searching the distinctive magic `70 53 A8 6E` returns nothing on the open web,
|
|
6
|
+
in file-signature databases, or in DFIR tooling.
|
|
7
|
+
|
|
8
|
+
**Status legend** — every claim below is tagged:
|
|
9
|
+
|
|
10
|
+
- **[V]** Verified. Proven by reconstructing real files byte-exactly and checking them
|
|
11
|
+
against the archive's own MD5 and CRC32 values.
|
|
12
|
+
- **[D]** Disassembly. Read from exported C++ symbols in Synology's own
|
|
13
|
+
`HyperBackupExplorer` binary, but not independently exercised by us.
|
|
14
|
+
- **[I]** Inferred. Consistent with observation, not proven. Treat with suspicion.
|
|
15
|
+
- **[?]** Unknown. Documented so the next person knows where the edges are.
|
|
16
|
+
|
|
17
|
+
Coverage caveat: this was derived from **one** archive — DSM 7, Hyper Backup 4.1.2-4039,
|
|
18
|
+
unencrypted, LZ4, single version, single pool. Variant handling marked [D] is implemented
|
|
19
|
+
from disassembly but has never met a real archive of that kind.
|
|
20
|
+
|
|
21
|
+
---
|
|
22
|
+
|
|
23
|
+
## 1. Conventions
|
|
24
|
+
|
|
25
|
+
- **All integers are big-endian.** No exceptions were found. [V]
|
|
26
|
+
- **Generation suffixes.** Most files carry a trailing `.<n>`: `0.idx.2`, `1.db.2`,
|
|
27
|
+
`index_ver.json.1`. A reader should resolve `name`, `name.1`, `name.2`… and prefer the
|
|
28
|
+
highest generation. [V]
|
|
29
|
+
- **macOS junk.** Archives stored on exFAT/HFS accumulate AppleDouble sidecars named
|
|
30
|
+
`._<something>`. These are not part of the format and must be filtered, or they will be
|
|
31
|
+
mistaken for real shards. [V]
|
|
32
|
+
|
|
33
|
+
## 2. Directory layout
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
<target>.hbk/
|
|
37
|
+
_Syno_TaskConfig INI-ish task settings (plain text)
|
|
38
|
+
SynologyHyperBackup.bkpi marker, may be zero bytes
|
|
39
|
+
Config/
|
|
40
|
+
index_ver.json[.n] {"major":0,"minor":9,"sub_minor":1}
|
|
41
|
+
target_ver.json[.n]
|
|
42
|
+
version_info.db[.n] SQLite: one row per backup version
|
|
43
|
+
virtual_file.index/ shard dir (56-byte records)
|
|
44
|
+
file_chunk<N>.index/ shard dirs (flat i64 arrays), N = 1..4 observed
|
|
45
|
+
@Share/<share>/
|
|
46
|
+
<version>.db[.n] SQLite: the file tree for that share+version
|
|
47
|
+
complete_list.db[.n] SQLite: which versions completed
|
|
48
|
+
Pool/
|
|
49
|
+
chunk_index/ shard dir (29-byte records, or 16 on older archives)
|
|
50
|
+
bucketID.counter[.n] u64 big-endian: number of buckets
|
|
51
|
+
file_pool/ whole-file dedup store [?]
|
|
52
|
+
<pool>/<dir>/<n>.bucket[.n] ~50 MB of concatenated compressed chunks
|
|
53
|
+
<pool>/<dir>/<n>.index[.n] the bucket's chunk directory
|
|
54
|
+
Control/, Guard/ bookkeeping, not needed to read data
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## 3. The shard container
|
|
58
|
+
|
|
59
|
+
Index families are directories of shards named `<N>.idx[.gen]`, N = 0,1,2,…
|
|
60
|
+
**Concatenate them in numeric order into one logical byte stream.** Shard size is 8 MiB;
|
|
61
|
+
records may straddle a shard boundary, so a reader must treat the family as one stream, not
|
|
62
|
+
as independent files. [V]
|
|
63
|
+
|
|
64
|
+
Only shard 0 carries a **64-byte header**, at stream offset 0. Records therefore begin at
|
|
65
|
+
**stream offset 64**. [V]
|
|
66
|
+
|
|
67
|
+
Header, as 16 big-endian `uint32` words:
|
|
68
|
+
|
|
69
|
+
| word | meaning |
|
|
70
|
+
|---|---|
|
|
71
|
+
| 0 | magic `0x7053A86E` [V] |
|
|
72
|
+
| 1 | kind — 0 virtual_file, 1 chunk_index/file_chunk, 2 bucket index [I] |
|
|
73
|
+
| 2 | variant [I] |
|
|
74
|
+
| 4 | **record size in bytes** [V] |
|
|
75
|
+
| 5:6 | `uint64` total stream length, as `(w5 << 32) | w6` [V] |
|
|
76
|
+
|
|
77
|
+
Word 4 is the important one: it lets a reader support layout variants without hardcoding
|
|
78
|
+
DSM versions. Observed: 56 (virtual_file), 29 (chunk_index), 32 (bucket index), 0 for
|
|
79
|
+
`file_chunk` — which is a flat `int64` array with no per-record framing. [V]
|
|
80
|
+
|
|
81
|
+
The declared length in words 5:6 may slightly exceed the bytes actually present; treat it
|
|
82
|
+
as advisory, not as a truncation check. [I]
|
|
83
|
+
|
|
84
|
+
## 4. The lookup chain
|
|
85
|
+
|
|
86
|
+
```
|
|
87
|
+
version_list.off_virtual_file (SQLite, per share)
|
|
88
|
+
-> virtual_file record (56 B)
|
|
89
|
+
-> (shard, offset) into file_chunk<shard>.index
|
|
90
|
+
-> flat array of int64 keys, 8 B each
|
|
91
|
+
-> each key = byte offset of a chunk_index record
|
|
92
|
+
-> chunk_index record -> (bucket_id, bucket_index_offset)
|
|
93
|
+
-> bucket .index record -> (offset, compressed length, uncompressed length, MD5)
|
|
94
|
+
-> bucket data file -> one raw LZ4 block
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
### 4.1 virtual_file record — 56 bytes [V for the pointer, D for the rest]
|
|
98
|
+
|
|
99
|
+
`off_virtual_file` from SQLite is a **byte offset into the virtual_file stream**.
|
|
100
|
+
|
|
101
|
+
| offset | size | field |
|
|
102
|
+
|---|---|---|
|
|
103
|
+
| 0 | 8 | **chunk-list pointer** — `(shard << 48) | (byte_offset & 0xFFFF_FFFF_FFFF)` [V] |
|
|
104
|
+
| 8 | 4 | ref_count [D] |
|
|
105
|
+
| 12 | 4 | uid [D] |
|
|
106
|
+
| 16 | 4 | gid [D] |
|
|
107
|
+
| 20 | 8 | atime seconds [D] |
|
|
108
|
+
| 28 | 4 | atime nanoseconds [D] |
|
|
109
|
+
| 32 | 8 | crtime seconds [D] |
|
|
110
|
+
| 40 | 4 | crtime nanoseconds [D] |
|
|
111
|
+
| 44 | 4 | mod_ver, or CRC depending on variant [D] |
|
|
112
|
+
| 48 | 8 | acl_offset, same packed encoding [D] |
|
|
113
|
+
|
|
114
|
+
The packing is exactly what the vendor binary does:
|
|
115
|
+
`FileChunkIndexIdParse(x) = x >> 48`, `FileChunkOffsetParse(x) = x & 0xFFFFFFFFFFFF`. [D]
|
|
116
|
+
|
|
117
|
+
The shard number is **read, not computed** — it varies per file (values 1–4 observed).
|
|
118
|
+
|
|
119
|
+
Note the record carries **no size, no name, no mode, no mtime**. All of that lives in
|
|
120
|
+
SQLite. The extractor needs the file size from `version_list` to know when to stop
|
|
121
|
+
consuming chunks. [V]
|
|
122
|
+
|
|
123
|
+
`off_virtual_file = -1` is a sentinel meaning the file is whole-file deduplicated into
|
|
124
|
+
`Pool/file_pool` rather than chunked. One occurrence in 501,278 files. The file_pool
|
|
125
|
+
format is **not decoded**. [?]
|
|
126
|
+
|
|
127
|
+
### 4.2 file_chunk arrays [V]
|
|
128
|
+
|
|
129
|
+
At the offset above, the `file_chunk<shard>.index` stream holds a **flat array of
|
|
130
|
+
big-endian `int64` values, 8 bytes each**, with no header and no per-record framing.
|
|
131
|
+
Each value is the **byte offset of a chunk_index record**.
|
|
132
|
+
|
|
133
|
+
Read entries in order, resolve each, and stop when the summed *uncompressed* chunk lengths
|
|
134
|
+
equal the file size from SQLite. There is no terminator and no stored chunk count.
|
|
135
|
+
|
|
136
|
+
Immediately before the array sits a **12-byte per-file header** (the `acl_offset` pointer
|
|
137
|
+
is exactly 12 bytes lower on every file sampled, 1547/1547). Its contents relate to
|
|
138
|
+
ACL/xattr storage and are **not decoded**. [?]
|
|
139
|
+
|
|
140
|
+
### 4.3 chunk_index record
|
|
141
|
+
|
|
142
|
+
Two layouts, distinguished by the record size in the shard header.
|
|
143
|
+
|
|
144
|
+
**29 bytes (v3)** [V]
|
|
145
|
+
|
|
146
|
+
| offset | size | field |
|
|
147
|
+
|---|---|---|
|
|
148
|
+
| 0 | 1 | mode; **bit 0 set = indirect** |
|
|
149
|
+
| 1 | 8 | if indirect: byte offset of another chunk_index record — follow it |
|
|
150
|
+
| 1 | 4 | if direct: `int32` bucket_id |
|
|
151
|
+
| 5 | 4 | if direct: `int32` byte offset into that bucket's `.index` |
|
|
152
|
+
| 25 | 4 | CRC [?] |
|
|
153
|
+
|
|
154
|
+
The indirect form is content deduplication: many files' chunks redirect to one canonical
|
|
155
|
+
chunk. Follow the chain until bit 0 is clear. Guard the recursion — a malformed archive
|
|
156
|
+
could otherwise loop. [V]
|
|
157
|
+
|
|
158
|
+
**16 bytes (v1/v2)** [D] — older archives:
|
|
159
|
+
|
|
160
|
+
| offset | size | field |
|
|
161
|
+
|---|---|---|
|
|
162
|
+
| 0 | 4 | `int32` bucket_id |
|
|
163
|
+
| 4 | 4 | `int32` bucket index offset |
|
|
164
|
+
| 8 | 4 | ref_count |
|
|
165
|
+
| 12 | 4 | mod_ver (v1) or CRC (v2) |
|
|
166
|
+
|
|
167
|
+
### 4.4 Bucket addressing [V]
|
|
168
|
+
|
|
169
|
+
```
|
|
170
|
+
dir = bucket_id >> 11 (2048 buckets per directory)
|
|
171
|
+
file = bucket_id & 0x7FF
|
|
172
|
+
path = Pool/<pool>/<dir>/<file>.bucket[.gen] and .index[.gen]
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
The directory is part of the address: the same filename `<n>.bucket` exists in *every*
|
|
176
|
+
directory. Getting this wrong is the single easiest way to read the wrong data — and
|
|
177
|
+
because chunk MD5s are checked, it surfaces as a verification failure rather than silent
|
|
178
|
+
corruption.
|
|
179
|
+
|
|
180
|
+
### 4.5 Bucket index record
|
|
181
|
+
|
|
182
|
+
**32 bytes (LAYOUT_D)** [V]
|
|
183
|
+
|
|
184
|
+
| offset | size | field |
|
|
185
|
+
|---|---|---|
|
|
186
|
+
| 0 | 4 | `uint32` compressed length |
|
|
187
|
+
| 4 | 4 | `uint32` byte offset within the `.bucket` file |
|
|
188
|
+
| 8 | 4 | `uint32` uncompressed length |
|
|
189
|
+
| 12 | 16 | **MD5 of the decompressed chunk** |
|
|
190
|
+
| 28 | 4 | **CRC32 of bytes [0:28] of this record**, poly `0xEDB88320` |
|
|
191
|
+
|
|
192
|
+
**28 bytes (legacy)** — identical without the trailing CRC32. [D]
|
|
193
|
+
|
|
194
|
+
Both checks verified across all 82,313 chunks of a 674 MB file. Earlier public
|
|
195
|
+
descriptions call the 20-byte tail a SHA-1; it is not. It is MD5 + CRC32, confirmed both
|
|
196
|
+
empirically and from the vendor's `getChecksum`, which copies 16 bytes. [V]
|
|
197
|
+
|
|
198
|
+
### 4.6 Chunk payload [V]
|
|
199
|
+
|
|
200
|
+
At `offset` in the `.bucket` file, read `compressed length` bytes and decompress to exactly
|
|
201
|
+
`uncompressed length`.
|
|
202
|
+
|
|
203
|
+
The payload is a **raw LZ4 block** — *not* an LZ4 frame. There is no frame header and no
|
|
204
|
+
magic; call the block API directly (`LZ4_decompress_safe`) with the known output size.
|
|
205
|
+
|
|
206
|
+
The codec comes from `data_compress_type` in `_Syno_TaskConfig`. The vendor dispatcher
|
|
207
|
+
`SYNO::Backup::decompress(type, …)` branches on **1 = lz4, 2 = lz4-hc, 4 = zlib**; lz4 and
|
|
208
|
+
lz4-hc share a decompressor. A robust reader attempts LZ4 and falls back to zlib. [D for
|
|
209
|
+
the mapping, V for LZ4]
|
|
210
|
+
|
|
211
|
+
Existing open-source tools implement only LZ4 and report corruption on some chunks; the
|
|
212
|
+
zlib path is the likely explanation.
|
|
213
|
+
|
|
214
|
+
## 5. SQLite metadata [V]
|
|
215
|
+
|
|
216
|
+
`Config/@Share/<share>/<version>.db` holds the file tree:
|
|
217
|
+
|
|
218
|
+
```sql
|
|
219
|
+
CREATE TABLE version_list (
|
|
220
|
+
name_id_v2 BLOB PRIMARY KEY, -- 20-byte node id
|
|
221
|
+
pname_id_v2 BLOB, -- parent's name_id_v2; join on this to build paths
|
|
222
|
+
off_virtual_file INTEGER, -- byte offset into the virtual_file stream, or -1
|
|
223
|
+
file_name TEXT, size INTEGER, mode INTEGER,
|
|
224
|
+
mtime_sec INTEGER, ctime_sec INTEGER, inode INTEGER, ...
|
|
225
|
+
);
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Reconstruct paths by joining `pname_id_v2 -> name_id_v2`. Roots are rows whose parent is
|
|
229
|
+
absent from the table or which are self-parented. `mode` is a POSIX mode word — use
|
|
230
|
+
`S_ISDIR` to separate directories from files.
|
|
231
|
+
|
|
232
|
+
`Config/version_info.db` lists backup versions with timestamps and completion status.
|
|
233
|
+
|
|
234
|
+
Entries named `@eaDir`, or ending `@SynoEAStream` / `@SynoResource`, are Synology
|
|
235
|
+
metadata — thumbnails, extended attributes, resource forks — not user data. In the
|
|
236
|
+
reference archive they were 553,899 of 1,135,405 rows but only ~40 GB of 4.88 TB.
|
|
237
|
+
|
|
238
|
+
## 6. Verification
|
|
239
|
+
|
|
240
|
+
Every chunk is self-checking: MD5 over the decompressed bytes, plus CRC32 over the index
|
|
241
|
+
record. **A reader that verifies both cannot silently return wrong data** — the worst case
|
|
242
|
+
is a loud failure. This matters more than throughput in a recovery tool, and it is what
|
|
243
|
+
makes it safe to attempt undecoded variants: a wrong guess fails rather than corrupts.
|
|
244
|
+
|
|
245
|
+
## 7. Encryption
|
|
246
|
+
|
|
247
|
+
`enable_data_encrypt` in `_Syno_TaskConfig` indicates client-side encryption. This
|
|
248
|
+
document covers **unencrypted archives only**. The encrypted variant is not implemented
|
|
249
|
+
here; a 2016 Python 2 script by "mrsandman" and an accompanying synology-forum.de thread
|
|
250
|
+
document the key derivation, and remain the only public reference.
|
|
251
|
+
|
|
252
|
+
## 8. Not decoded
|
|
253
|
+
|
|
254
|
+
- `Pool/file_pool` whole-file dedup store, and the `off_virtual_file = -1` reference.
|
|
255
|
+
- The 12-byte per-file header preceding each chunk array (ACL/xattr related).
|
|
256
|
+
- The trailing 4 bytes of a v3 chunk_index record.
|
|
257
|
+
- How Synology *chooses* which `file_chunk` shard a file lands in (readers only need to
|
|
258
|
+
read the value, never predict it).
|
|
259
|
+
- Encrypted archives.
|
|
260
|
+
|
|
261
|
+
## 9. How this was derived
|
|
262
|
+
|
|
263
|
+
Two independent routes, cross-checked against each other:
|
|
264
|
+
|
|
265
|
+
1. **Empirical.** Anchor on a file whose content could be recognised — a 10,244-byte
|
|
266
|
+
`.DS_Store` that turned out to be exactly two chunks (9232 + 1012) — then generalise and
|
|
267
|
+
confirm by rebuilding progressively larger files until an 674 MB video reproduced
|
|
268
|
+
byte-exactly across 82,313 chunks.
|
|
269
|
+
2. **Disassembly.** Synology's macOS `HyperBackupExplorer` ships **full C++ symbols**
|
|
270
|
+
(~24k mangled names, plus original source paths). `nm -a` and `objdump -d` on named
|
|
271
|
+
functions such as `ChunkIndexAdapter::getChunkIndexInfo`,
|
|
272
|
+
`VirtualFileAdapter::getVirtualFileInfo` and `VirtualFile::FileChunkOffsetParse` give
|
|
273
|
+
exact field offsets and endianness. This turns guesswork into reading the vendor's own
|
|
274
|
+
field arithmetic.
|
|
275
|
+
|
|
276
|
+
Where the two disagreed, the empirical result won and the disassembly note was corrected —
|
|
277
|
+
that is how the "SHA-1" error was caught.
|
|
278
|
+
|
|
279
|
+
No vendor code is reproduced here. Field offsets, record sizes and wire layouts are facts
|
|
280
|
+
about a data format, not authorship. The independent Rust implementation
|
|
281
|
+
[TeamDman/teamy-hyper-backup-explorer](https://github.com/TeamDman/teamy-hyper-backup-explorer)
|
|
282
|
+
(MPL-2.0) was consulted; its constants agree with what we derived separately.
|
hbkit-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 YordiLorenzo
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
hbkit-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: hbkit
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Recover files from Synology Hyper Backup (.hbk) archives without Synology software
|
|
5
|
+
Project-URL: Homepage, https://github.com/YordiLorenzo/hbkit
|
|
6
|
+
Project-URL: Source, https://github.com/YordiLorenzo/hbkit
|
|
7
|
+
Project-URL: Issues, https://github.com/YordiLorenzo/hbkit/issues
|
|
8
|
+
Project-URL: Format spec, https://github.com/YordiLorenzo/hbkit/blob/main/FORMAT.md
|
|
9
|
+
Author-email: YordiLorenzo <yordilorenzo@gmail.com>
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: backup,data-recovery,dsm,extract,hbk,hyper-backup,nas,recovery,synology
|
|
13
|
+
Classifier: Development Status :: 4 - Beta
|
|
14
|
+
Classifier: Environment :: Console
|
|
15
|
+
Classifier: Intended Audience :: End Users/Desktop
|
|
16
|
+
Classifier: Intended Audience :: System Administrators
|
|
17
|
+
Classifier: Operating System :: MacOS
|
|
18
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
19
|
+
Classifier: Programming Language :: Python :: 3
|
|
20
|
+
Classifier: Topic :: System :: Archiving :: Backup
|
|
21
|
+
Classifier: Topic :: System :: Recovery Tools
|
|
22
|
+
Requires-Python: >=3.10
|
|
23
|
+
Requires-Dist: textual>=0.80
|
|
24
|
+
Provides-Extra: dev
|
|
25
|
+
Requires-Dist: pytest>=7; extra == 'dev'
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
|
|
28
|
+
# hbkit
|
|
29
|
+
|
|
30
|
+
Recover files from **Synology Hyper Backup (`.hbk`)** archives without any Synology software.
|
|
31
|
+
|
|
32
|
+
Point it at a backup — on a local disk, an external drive, or an S3/R2 bucket mounted with
|
|
33
|
+
rclone — browse it as a tree, and pull out what you want. Works headless on Linux and macOS,
|
|
34
|
+
including Apple Silicon, where Synology's own Hyper Backup Explorer is awkward or unavailable.
|
|
35
|
+
|
|
36
|
+
```sh
|
|
37
|
+
pip install hbkit
|
|
38
|
+
|
|
39
|
+
hbk /Volumes/Backup doctor # can this archive be recovered?
|
|
40
|
+
hbk-tui /Volumes/Backup # browse and select interactively
|
|
41
|
+
hbk /Volumes/Backup get "/Photos/*" ~/restore
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
---
|
|
45
|
+
|
|
46
|
+
## Why
|
|
47
|
+
|
|
48
|
+
Hyper Backup Explorer is a GUI, has no command line, ships x86-only on Linux, and gets
|
|
49
|
+
unhappy with large archives. If your NAS died and the backup is all you have, you want
|
|
50
|
+
something you can point at a drive, script, and trust.
|
|
51
|
+
|
|
52
|
+
`hbkit` reads the format directly. Every chunk it returns has been checked against the
|
|
53
|
+
archive's own MD5 and CRC32, so **it cannot silently hand you corrupt data** — the worst
|
|
54
|
+
case is a loud failure naming the file.
|
|
55
|
+
|
|
56
|
+
## The TUI
|
|
57
|
+
|
|
58
|
+
```
|
|
59
|
+
⭘ Hyper Backup Recovery 17:11:17
|
|
60
|
+
┌──────────────────────────────────────────────────────────────┐ │
|
|
61
|
+
│ search filename… (/) │ │ Selection
|
|
62
|
+
└──────────────────────────────────────────────────────────────┘ │ 197,607 files
|
|
63
|
+
▼ ◪ 📁 NAS Volume 1 4.4T 499,745 │ 625.0G in 1 item(s)
|
|
64
|
+
├─ ▶ ☐ 📁 Archive 2022 24.2G 358 │
|
|
65
|
+
├─ ▶ ☐ 📁 Backups 23.5G 63,693 │ Destination
|
|
66
|
+
├─ ▶ ☐ 📁 Video Projects 1.9T 64,473 │ ┌──────────────────────────┐
|
|
67
|
+
├─ ▶ ☐ 📁 Media Library 1.3T 66,629 │ │ ~/restore │
|
|
68
|
+
├─ ▼ ☑ 📁 Photo Libraries 625.0G 197,607 │ └──────────────────────────┘
|
|
69
|
+
│ ├─ ▶ ☑ 📁 Photos Library - Laptop… 28.8G 76,844 │ ⚠ needs 625.0G, only 70.3G free
|
|
70
|
+
│ ├─ ▶ ☑ 📁 Photos Library - Old Backup… 19.8G 15,505 │
|
|
71
|
+
│ Start recovery
|
|
72
|
+
a All n Clear d Destination r Recover / Search q Quit │
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
`space` tick · `a` all · `n` clear · `/` search · `d` destination · `r` recover.
|
|
76
|
+
|
|
77
|
+
Folders show subtree size and file count. Ticking a folder takes its whole subtree; the
|
|
78
|
+
destination panel warns before you start if the selection will not fit. Recovery shows a
|
|
79
|
+
live progress bar, throughput, ETA and a failure log.
|
|
80
|
+
|
|
81
|
+
## Commands
|
|
82
|
+
|
|
83
|
+
```sh
|
|
84
|
+
hbk <archive> doctor # probe an unknown archive, prove it's readable
|
|
85
|
+
hbk <archive> info # task name, codec, shares, encryption
|
|
86
|
+
hbk <archive> list [pattern] # search the file index
|
|
87
|
+
hbk <archive> get <glob> <dest> [-j N] # extract, preserving tree and mtimes
|
|
88
|
+
hbk <archive> verify <glob> [-j N] # integrity-check, write nothing
|
|
89
|
+
hbk <archive> tui # same as hbk-tui
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
`<archive>` is a `.hbk` directory, or any drive or folder containing one — it will find it.
|
|
93
|
+
Globs match the full archive path, which begins with the share name.
|
|
94
|
+
|
|
95
|
+
**Start with `doctor`.** It reports the layout it found and then *proves* the archive is
|
|
96
|
+
readable by rebuilding a random sample of real files with full checksum verification:
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
archive : /Volumes/Backup/nas_1.hbk
|
|
100
|
+
task : Daily Backup
|
|
101
|
+
source host : nas
|
|
102
|
+
source model : DS...
|
|
103
|
+
chunk codec : lz4
|
|
104
|
+
virtual_file record : 56 B
|
|
105
|
+
chunk_index record : 29 B (v3)
|
|
106
|
+
bucket index record : 32 B (md5+crc32)
|
|
107
|
+
shares : Photos, Documents
|
|
108
|
+
|
|
109
|
+
PASS virtual_file layout known (56 B)
|
|
110
|
+
PASS chunk_index layout known (29 B)
|
|
111
|
+
PASS bucket layout known (32 B)
|
|
112
|
+
PASS rebuilt 9 sampled files, all chunks verified (9 ok, 0 failed)
|
|
113
|
+
|
|
114
|
+
VERDICT: recoverable. Sampled files rebuilt byte-exact and checksum-verified.
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
## Behaviour worth knowing
|
|
118
|
+
|
|
119
|
+
- **Resumable.** Correctly-sized files are skipped, so re-running a big job is cheap.
|
|
120
|
+
- **Crash-safe.** Files are written to `.part` and atomically renamed, so an interrupted
|
|
121
|
+
run never leaves a truncated file that a later resume would trust.
|
|
122
|
+
- **Layout preserved.** Output goes to `<dest>/<share>/<original path>` with original mtimes.
|
|
123
|
+
- **Read-only.** Nothing is ever written to the archive.
|
|
124
|
+
- **Sidecars skipped.** `@eaDir`, `@SynoEAStream` and `@SynoResource` are Synology
|
|
125
|
+
metadata — thumbnails and xattr streams, not your data. In one real archive they were
|
|
126
|
+
half of all entries but under 1% of the bytes.
|
|
127
|
+
- **Index cached** per archive in `~/.cache/hbkit`, rebuilt automatically when the archive
|
|
128
|
+
changes. Browsing 1.1M files is instant after the first open.
|
|
129
|
+
|
|
130
|
+
## Performance
|
|
131
|
+
|
|
132
|
+
Use `-j` to set worker processes (default 8). Threads do not help — extraction is
|
|
133
|
+
GIL-bound in Python, measured flat at ~32 MB/s from 1 to 12 threads — so `hbkit` fans out
|
|
134
|
+
to real processes.
|
|
135
|
+
|
|
136
|
+
Throughput is bounded by the source device, not by `hbkit`. On a USB spinning disk with a
|
|
137
|
+
92 MB/s sequential ceiling, a cold parallel run reached 58 MB/s while the disk itself sat
|
|
138
|
+
at 49 MB/s; scattered reads across tens of thousands of bucket files never reach sequential
|
|
139
|
+
speed. Work is ordered by locality so each worker sweeps the pool in one direction rather
|
|
140
|
+
than several heads chasing several regions.
|
|
141
|
+
|
|
142
|
+
Media does not compress — measured ratio 1.004 on video. The space saving in a Hyper Backup
|
|
143
|
+
archive comes from cross-file dedup, not per-file compression, so expect bytes-off-disk to
|
|
144
|
+
roughly equal bytes-delivered.
|
|
145
|
+
|
|
146
|
+
## Scope and limits
|
|
147
|
+
|
|
148
|
+
Read this before trusting it with the only copy of anything.
|
|
149
|
+
|
|
150
|
+
- **Encrypted archives are not supported.** They are detected and refused, never
|
|
151
|
+
half-decoded. If `enable_data_encrypt` is set, this tool will not help you.
|
|
152
|
+
- **Proven against a limited set of archives.** The reference archive is DSM 7,
|
|
153
|
+
Hyper Backup 4.1.2, unencrypted, LZ4, single version, single pool. Older record layouts
|
|
154
|
+
(16-byte `chunk_index`, 28-byte bucket records), zlib chunks and multi-version archives
|
|
155
|
+
are implemented from disassembly but have not met a real archive of that kind. `doctor`
|
|
156
|
+
exists precisely so you can find out in seconds rather than mid-restore.
|
|
157
|
+
- **Unknown layouts are refused, not guessed.** A wrong guess would mean silently wrong
|
|
158
|
+
bytes, which is the one thing a recovery tool must never do.
|
|
159
|
+
- **Whole-file dedup** (`off_virtual_file = -1`, files living in `Pool/file_pool`) is not
|
|
160
|
+
decoded. One file in 501,278 in the reference archive.
|
|
161
|
+
- Requires `liblz4` (`brew install lz4`, or `apt install liblz4-1`). Set `HBK_LZ4` if it
|
|
162
|
+
is somewhere unusual.
|
|
163
|
+
|
|
164
|
+
## The format
|
|
165
|
+
|
|
166
|
+
[`FORMAT.md`](FORMAT.md) is a full specification of the on-disk format, written so you can
|
|
167
|
+
implement a reader in any language. Every claim is tagged **verified / from disassembly /
|
|
168
|
+
inferred / unknown**, and there is an explicit list of what is still undecoded.
|
|
169
|
+
|
|
170
|
+
As far as we can tell no public description of this format existed before it — searching
|
|
171
|
+
the container magic `70 53 A8 6E` returns nothing on the open web or in file-signature
|
|
172
|
+
databases. If the tool is useless to you, the spec may not be.
|
|
173
|
+
|
|
174
|
+
It was derived two ways and cross-checked: empirically, by anchoring on a file whose bytes
|
|
175
|
+
could be recognised and then rebuilding progressively larger files until a 674 MB video
|
|
176
|
+
reproduced exactly across 82,313 chunks; and by reading exported C++ symbols in Synology's
|
|
177
|
+
own `HyperBackupExplorer` binary, which ships with full symbols and gives exact field
|
|
178
|
+
offsets. Where the two disagreed, the empirical result won.
|
|
179
|
+
|
|
180
|
+
## Prior art
|
|
181
|
+
|
|
182
|
+
- [TeamDman/teamy-hyper-backup-explorer](https://github.com/TeamDman/teamy-hyper-backup-explorer) — independent Rust implementation (MPL-2.0). Its constants agree with what we derived separately.
|
|
183
|
+
- [mistersandman/hyperbackup_decrypt](https://github.com/mistersandman/hyperbackup_decrypt) — 2016 Python 2 script, and the only public reference for the **encrypted** variant.
|
|
184
|
+
|
|
185
|
+
## Development
|
|
186
|
+
|
|
187
|
+
```sh
|
|
188
|
+
git clone https://github.com/YordiLorenzo/hbkit && cd hbkit
|
|
189
|
+
python3 -m venv .venv && ./.venv/bin/pip install -e ".[dev]"
|
|
190
|
+
HBK_TEST_ARCHIVE=/path/to/backup ./.venv/bin/python -m pytest tests -v
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
The test suite needs a real archive — correctness is checked against the archive's own
|
|
194
|
+
checksums and against file-format markers, so a pass means the bytes are genuinely right,
|
|
195
|
+
not merely the right length. Tests skip cleanly when no archive is available.
|
|
196
|
+
|
|
197
|
+
Contributions especially welcome for: encrypted archives, the legacy record layouts, and
|
|
198
|
+
`Pool/file_pool`. If you have an archive `doctor` cannot read, an issue with its output
|
|
199
|
+
is genuinely useful.
|
|
200
|
+
|
|
201
|
+
## License
|
|
202
|
+
|
|
203
|
+
MIT — see [LICENSE](LICENSE).
|
|
204
|
+
|
|
205
|
+
Not affiliated with or endorsed by Synology. "Synology" and "Hyper Backup" are trademarks
|
|
206
|
+
of Synology Inc., used here only to describe what this software reads.
|