model-mirror-cli 0.2.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- model_mirror/__init__.py +7 -0
- model_mirror/audit.py +214 -0
- model_mirror/checksums.py +371 -0
- model_mirror/cli.py +2436 -0
- model_mirror/config.py +300 -0
- model_mirror/hub.py +901 -0
- model_mirror/lock.py +119 -0
- model_mirror/mirror.py +201 -0
- model_mirror/progress.py +260 -0
- model_mirror/removal.py +164 -0
- model_mirror/repair.py +399 -0
- model_mirror/state.py +150 -0
- model_mirror/torrent.py +453 -0
- model_mirror/torrent_coverage.py +400 -0
- model_mirror/torrent_hashes.py +168 -0
- model_mirror/torrent_import.py +620 -0
- model_mirror/torrent_publication.py +582 -0
- model_mirror/torrent_seed.py +228 -0
- model_mirror/verify.py +153 -0
- model_mirror_cli-0.2.0.dist-info/METADATA +550 -0
- model_mirror_cli-0.2.0.dist-info/RECORD +24 -0
- model_mirror_cli-0.2.0.dist-info/WHEEL +4 -0
- model_mirror_cli-0.2.0.dist-info/entry_points.txt +2 -0
- model_mirror_cli-0.2.0.dist-info/licenses/LICENSE +21 -0
|
@@ -0,0 +1,550 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: model-mirror-cli
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Mirror and validate Hugging Face archives without leaving payloads in the default cache.
|
|
5
|
+
Project-URL: Repository, https://github.com/xlr8harder/model-mirror
|
|
6
|
+
Project-URL: Issues, https://github.com/xlr8harder/model-mirror/issues
|
|
7
|
+
Author: xlr8harder
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Keywords: huggingface,mirror,model-archive,torrent
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Environment :: Console
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
19
|
+
Classifier: Topic :: System :: Archiving
|
|
20
|
+
Requires-Python: >=3.11
|
|
21
|
+
Requires-Dist: huggingface-hub>=1.0
|
|
22
|
+
Requires-Dist: pyyaml>=6.0
|
|
23
|
+
Provides-Extra: torrent
|
|
24
|
+
Requires-Dist: libtorrent<2.1,>=2.0.10; extra == 'torrent'
|
|
25
|
+
Description-Content-Type: text/markdown
|
|
26
|
+
|
|
27
|
+
# model-mirror
|
|
28
|
+
|
|
29
|
+
Mirror Hugging Face repositories into local bulk storage and verify that the
|
|
30
|
+
files remain complete.
|
|
31
|
+
|
|
32
|
+
`model-mirror` downloads directly into one archive directory, avoiding payload
|
|
33
|
+
files being left behind in the default Hugging Face cache. It records the exact
|
|
34
|
+
Hub commit mirrored, writes a local hash manifest, and keeps verification state
|
|
35
|
+
beside each model. If you already use `hf auth login`, model-mirror will try to
|
|
36
|
+
find that token automatically.
|
|
37
|
+
|
|
38
|
+
Online operations resolve the requested Hugging Face revision, usually `main`,
|
|
39
|
+
to a specific Hub commit before downloading, verifying, or repairing. The local
|
|
40
|
+
mirror stays tied to that commit so files are not mixed across upstream updates.
|
|
41
|
+
If the Hub repo later moves to a newer commit, `verify` reports the upstream
|
|
42
|
+
change without modifying local files.
|
|
43
|
+
|
|
44
|
+
## Quick Start
|
|
45
|
+
|
|
46
|
+
Install the `model-mirror-cli` distribution from PyPI with
|
|
47
|
+
[uv](https://docs.astral.sh/uv/) (Python 3.11 or newer):
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
uv tool install model-mirror-cli
|
|
51
|
+
|
|
52
|
+
model-mirror config directory /mnt/big-drive/huggingface
|
|
53
|
+
model-mirror config set hf-xet-reconstruct-write-sequentially true # useful for HDDs
|
|
54
|
+
# Optional if token autodetection does not find your Hugging Face token:
|
|
55
|
+
model-mirror config set token-path ~/.cache/huggingface/token
|
|
56
|
+
|
|
57
|
+
model-mirror mirror org/model
|
|
58
|
+
model-mirror status
|
|
59
|
+
model-mirror verify org/model
|
|
60
|
+
model-mirror repair org/model # if verify reports repair paths
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
The PyPI distribution is named `model-mirror-cli` to distinguish it from an
|
|
64
|
+
unrelated package; the installed command remains `model-mirror`. To include
|
|
65
|
+
experimental torrent support from the start, install
|
|
66
|
+
`uv tool install 'model-mirror-cli[torrent]'`.
|
|
67
|
+
|
|
68
|
+
Update and inspect an installed release with:
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
uv tool upgrade model-mirror-cli
|
|
72
|
+
model-mirror --version
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
For development, clone the
|
|
76
|
+
[source repository](https://github.com/xlr8harder/model-mirror), run
|
|
77
|
+
`uv sync --all-extras --dev`, and prefix commands with `uv run`. The rest of
|
|
78
|
+
this README assumes the tool installation above.
|
|
79
|
+
|
|
80
|
+
For periodic maintenance of the whole archive:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
model-mirror verify --all --max-age 30d || true
|
|
84
|
+
model-mirror repair --all
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
Mirrors are stored by repo type:
|
|
88
|
+
|
|
89
|
+
```text
|
|
90
|
+
/mnt/big-drive/huggingface/models/org/model/
|
|
91
|
+
/mnt/big-drive/huggingface/datasets/org/data/
|
|
92
|
+
/mnt/big-drive/huggingface/spaces/org/space/
|
|
93
|
+
/mnt/big-drive/huggingface/.model-mirror/cache/
|
|
94
|
+
/mnt/big-drive/huggingface/.model-mirror/tmp/
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
The archive has one top-level control directory, `.model-mirror/`. Its `cache/`
|
|
98
|
+
and `tmp/` children contain disposable Hugging Face/Xet state and resumable
|
|
99
|
+
download staging. Each mirrored repository also has its own `.model-mirror/`
|
|
100
|
+
directory for durable commit-scoped metadata. Payload directories never contain
|
|
101
|
+
a model-mirror virtual environment or package-installation cache.
|
|
102
|
+
|
|
103
|
+
Run `model-mirror --help` or `model-mirror COMMAND --help` for the full CLI
|
|
104
|
+
reference. Run `model-mirror config options` for every supported config key.
|
|
105
|
+
Commands exit non-zero for dirty, incomplete, busy, or invalid states where that
|
|
106
|
+
matters; see each subcommand's help for exact exit-status behavior.
|
|
107
|
+
|
|
108
|
+
Without a repository argument, `model-mirror status` prints a compact,
|
|
109
|
+
metadata-only archive table with repository type and ID, recorded file count
|
|
110
|
+
and payload size, abbreviated resolved commit, verification age, and
|
|
111
|
+
exceptional state. A healthy row uses `-` in `EXCEPTIONS` instead of repeating
|
|
112
|
+
`clean`. Torrent and live activity columns appear only when relevant.
|
|
113
|
+
`model-mirror list` is currently an alias for the same output.
|
|
114
|
+
|
|
115
|
+
Status reads model-mirror's existing verification, manifest, snapshot, lock,
|
|
116
|
+
progress, and torrent metadata. It does not recursively scan payload or cache
|
|
117
|
+
directories, read payload bytes, contact the Hub, or update metadata. Missing
|
|
118
|
+
recorded counts or sizes are displayed as unknown; use `verify` to reconcile
|
|
119
|
+
the local filesystem.
|
|
120
|
+
|
|
121
|
+
Pass a repository ID for a detailed, strictly local last-known report without
|
|
122
|
+
contacting the Hub:
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
model-mirror status org/model
|
|
126
|
+
model-mirror list org/model # same detailed view
|
|
127
|
+
model-mirror status --repo-type dataset org/data
|
|
128
|
+
model-mirror status --check-upstream org/model
|
|
129
|
+
model-mirror status --verbose org/model
|
|
130
|
+
model-mirror status --json
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
The default detail view emphasizes verification state and age, resolved commit,
|
|
134
|
+
recorded payload size, upstream state, torrent publication, exceptional state,
|
|
135
|
+
and useful next actions. Issues and repair paths appear only when present.
|
|
136
|
+
`--verbose` prints all recorded metadata fields.
|
|
137
|
+
|
|
138
|
+
`--check-upstream` performs an advisory live comparison between the locally
|
|
139
|
+
resolved commit and the commit currently resolved by the requested upstream
|
|
140
|
+
revision. It does not write verification state, snapshot metadata, timestamps,
|
|
141
|
+
or locks. The ordinary recorded upstream state is still shown separately from
|
|
142
|
+
the live observation in JSON.
|
|
143
|
+
|
|
144
|
+
`--json` emits the versioned `model-mirror-status` schema with one
|
|
145
|
+
`repositories` array for both single-repository and archive-wide output.
|
|
146
|
+
Numbers remain numeric, timestamps are ISO 8601, and issues and repair paths are
|
|
147
|
+
arrays. `--verbose` is accepted as a no-op when combined with `--json`. A
|
|
148
|
+
mismatch between verification and snapshot commits is reported as
|
|
149
|
+
`snapshot-stale`.
|
|
150
|
+
|
|
151
|
+
## Verification
|
|
152
|
+
|
|
153
|
+
`mirror` verifies by default. A clean mirror has:
|
|
154
|
+
|
|
155
|
+
- all expected Hub files present
|
|
156
|
+
- expected file sizes
|
|
157
|
+
- local SHA-256 and Git blob SHA-1 hashes in versioned `.manifest`
|
|
158
|
+
- LFS file hashes compared with Hub LFS SHA-256 metadata
|
|
159
|
+
- regular Git files compared with Hub Git blob ids
|
|
160
|
+
- `.verification` with `status: clean`
|
|
161
|
+
|
|
162
|
+
By default, verification ignores unexpected payload files, so they do not make
|
|
163
|
+
an otherwise correct mirror dirty. Use `verify --strict` to report unexpected
|
|
164
|
+
files and make verification fail. Model-mirror's own state and cache paths are
|
|
165
|
+
not treated as payload extras.
|
|
166
|
+
|
|
167
|
+
Useful verification commands:
|
|
168
|
+
|
|
169
|
+
```bash
|
|
170
|
+
model-mirror verify org/model
|
|
171
|
+
model-mirror verify --cached org/model
|
|
172
|
+
model-mirror verify --offline org/model
|
|
173
|
+
model-mirror verify --strict org/model
|
|
174
|
+
model-mirror verify --all
|
|
175
|
+
model-mirror verify --all --max-age 7d
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
`--cached` checks presence, sizes, and Hub-provided hashes from current
|
|
179
|
+
`.manifest` rows without rehashing payload files. If cached hash data is missing
|
|
180
|
+
or stale, cached verification exits non-zero and tells you to run full
|
|
181
|
+
verification. `--offline` does not contact the Hub, so it does not detect
|
|
182
|
+
whether the upstream repo has moved to a newer commit. Full offline verification
|
|
183
|
+
requires an existing `.manifest`; `--offline --cached` only reports the current
|
|
184
|
+
`.verification` state. `--max-age` is useful for periodic jobs that should skip
|
|
185
|
+
recently verified clean mirrors.
|
|
186
|
+
|
|
187
|
+
If the upstream repository is unavailable, online verification exits non-zero
|
|
188
|
+
and prints the command to mark the local mirror offline-only:
|
|
189
|
+
|
|
190
|
+
```bash
|
|
191
|
+
model-mirror offline org/model
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
Offline-only mirrors use local verification only and do not check whether the
|
|
195
|
+
Hub repo has moved or disappeared. Use `model-mirror online org/model` to
|
|
196
|
+
re-enable Hub checks.
|
|
197
|
+
|
|
198
|
+
If one repo is already locked, a single-repo `verify` exits non-zero. With
|
|
199
|
+
`verify --all`, locked repos are reported as skipped, remaining repos are still
|
|
200
|
+
checked, and the final exit status is non-zero.
|
|
201
|
+
|
|
202
|
+
Verification records missing or corrupt files as repair paths in
|
|
203
|
+
`.verification`. Repairs redownload only those paths:
|
|
204
|
+
|
|
205
|
+
```bash
|
|
206
|
+
model-mirror verify org/model
|
|
207
|
+
model-mirror repair org/model
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
`repair org/model` consumes the existing `.verification` state. If no
|
|
211
|
+
verification state exists, it tells you to run `verify` first. It prints how old
|
|
212
|
+
the verification result is, warns after 24 hours, updates manifest rows for
|
|
213
|
+
repaired files, and runs a final cached verification. In a `verify` then
|
|
214
|
+
`repair` workflow, the initial full verify hashes existing files once; repaired
|
|
215
|
+
files are hashed again after download, but unchanged large files are not
|
|
216
|
+
rehashed a second time.
|
|
217
|
+
|
|
218
|
+
Repair also reconciles a pinned snapshot plan whose commit is older than the
|
|
219
|
+
clean verification state. It fetches the recorded commit's Hub metadata and
|
|
220
|
+
reuses current manifest hashes, so payload files are not reread when that cached
|
|
221
|
+
evidence is complete. A successful commit update promotes the new snapshot plan
|
|
222
|
+
before recording the new commit as clean.
|
|
223
|
+
|
|
224
|
+
If repair sees incomplete cached verification data for untouched files, it stops
|
|
225
|
+
before downloading and tells you to run full `verify`. `repair --force-partial`
|
|
226
|
+
overrides that safety check, but may leave the repository inconsistent and still
|
|
227
|
+
exits non-zero if final cached verification cannot prove the result.
|
|
228
|
+
|
|
229
|
+
An offline-only mirror cannot be repaired because there is no upstream source to
|
|
230
|
+
repair against. A direct `repair org/model` exits non-zero with that explanation;
|
|
231
|
+
`repair --all` logs a warning and skips offline-only mirrors.
|
|
232
|
+
|
|
233
|
+
## Experimental Torrent Publishing And Recovery
|
|
234
|
+
|
|
235
|
+
Torrent support is optional and experimental. The `upgrade` and `torrent`
|
|
236
|
+
commands, their behavior, and the torrent-specific metadata formats may change
|
|
237
|
+
before stabilization. Ordinary mirror, verify, and repair workflows remain
|
|
238
|
+
stable and do not require the torrent extra:
|
|
239
|
+
|
|
240
|
+
```bash
|
|
241
|
+
# Install or replace the PyPI tool with torrent support:
|
|
242
|
+
uv tool install --force 'model-mirror-cli[torrent]'
|
|
243
|
+
|
|
244
|
+
# Or add the extra to the checkout's development environment:
|
|
245
|
+
uv sync --extra torrent
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
Every torrent is an immutable publication of one resolved
|
|
249
|
+
`repo@commit`. Publishing first requires a clean, pinned mirror. It completes
|
|
250
|
+
any missing torrent hash coverage, writes an ordinary hybrid v1/v2 `.torrent`,
|
|
251
|
+
prints a standard magnet URI and external-client data location, and creates a
|
|
252
|
+
persistent fence that prevents the canonical archive from moving to another
|
|
253
|
+
commit.
|
|
254
|
+
|
|
255
|
+
```bash
|
|
256
|
+
model-mirror upgrade org/model # optional explicit precomputation
|
|
257
|
+
model-mirror torrent create org/model # artifacts and fence; no seed intent
|
|
258
|
+
model-mirror torrent publish org/model # artifacts, fence, and durable seed intent
|
|
259
|
+
model-mirror torrent show org/model
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
Downloads and same-commit repairs accumulate reusable torrent hashes during
|
|
263
|
+
their existing payload pass. `upgrade` is for older archives or incomplete
|
|
264
|
+
coverage; it reads only files still missing hashes for the current publication
|
|
265
|
+
profile:
|
|
266
|
+
|
|
267
|
+
```bash
|
|
268
|
+
model-mirror upgrade org/model
|
|
269
|
+
model-mirror upgrade --all --dry-run
|
|
270
|
+
model-mirror upgrade --all
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
The publication profile is the versioned, deterministic set of rules that fixes
|
|
274
|
+
file order, piece sizing, padding, descriptor encoding, and therefore torrent
|
|
275
|
+
identity. The current and only profile is `hybrid-v1-v2-1`; model-mirror selects
|
|
276
|
+
it automatically. There is no user-selectable profile option yet. A future
|
|
277
|
+
identity-affecting algorithm change will use a new profile name rather than
|
|
278
|
+
silently changing an existing swarm.
|
|
279
|
+
|
|
280
|
+
The managed backend is a replaceable long-running process:
|
|
281
|
+
|
|
282
|
+
```bash
|
|
283
|
+
model-mirror torrent serve
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
Run it under your normal service manager with restart-on-failure and
|
|
287
|
+
restart-on-boot. Publication intent, verified file fingerprints, and the update
|
|
288
|
+
fence are durable; a restarted backend reconciles desired seeds without an
|
|
289
|
+
explicit reseed command or a payload-wide recheck. `model-mirror status` reports
|
|
290
|
+
desired and observed torrent state. A minimal systemd service uses the same
|
|
291
|
+
configured archive:
|
|
292
|
+
|
|
293
|
+
```ini
|
|
294
|
+
[Service]
|
|
295
|
+
ExecStart=/path/to/model-mirror --config /path/to/config.yaml torrent serve
|
|
296
|
+
Restart=always
|
|
297
|
+
```
|
|
298
|
+
|
|
299
|
+
The emitted `.torrent` and magnet are not tied to the managed backend. To use a
|
|
300
|
+
preferred client, either add the printed torrent with its printed data location
|
|
301
|
+
for seeding, or record external ownership explicitly:
|
|
302
|
+
|
|
303
|
+
```bash
|
|
304
|
+
model-mirror torrent publish --external org/model
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
Model-mirror does not control or infer the runtime health of an external
|
|
308
|
+
client. Stop that client before modifying payload. `torrent stop` stops managed
|
|
309
|
+
intent (or records that an external seed was stopped) but deliberately retains
|
|
310
|
+
the update fence. Only explicit retirement releases it:
|
|
311
|
+
|
|
312
|
+
```bash
|
|
313
|
+
model-mirror torrent stop org/model
|
|
314
|
+
model-mirror torrent retire org/model
|
|
315
|
+
model-mirror repair --update org/model
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
Same-commit `repair` enters maintenance, waits for the managed backend to
|
|
319
|
+
detach, repairs only recorded paths, refreshes their coverage, and resumes the
|
|
320
|
+
same publication when verification succeeds. `repair --update` stays blocked
|
|
321
|
+
until retirement.
|
|
322
|
+
|
|
323
|
+
On another host, model-mirror can download natively or provide an exact
|
|
324
|
+
standard-client handoff:
|
|
325
|
+
|
|
326
|
+
```bash
|
|
327
|
+
model-mirror torrent join /path/model@commit.torrent
|
|
328
|
+
model-mirror torrent join 'magnet:?xt=...' --seed
|
|
329
|
+
|
|
330
|
+
model-mirror torrent handoff /path/model@commit.torrent
|
|
331
|
+
# Download with any standard client to the printed data location, then run:
|
|
332
|
+
model-mirror torrent import /path/model@commit.torrent /printed/path/model
|
|
333
|
+
```
|
|
334
|
+
|
|
335
|
+
Import validates hostile paths, the commit-scoped descriptor, sizes, and
|
|
336
|
+
content before atomically moving the staged payload into the normal archive
|
|
337
|
+
layout. Native join reuses libtorrent's verified piece state and reconstructs
|
|
338
|
+
coverage without rereading the payload. External-client import independently
|
|
339
|
+
reads and hashes the downloaded files because model-mirror cannot assume that
|
|
340
|
+
client's runtime state.
|
|
341
|
+
|
|
342
|
+
A trusted torrent proves consistency with the supplied infohash; by itself it
|
|
343
|
+
does not prove that the publisher's bytes were authentic Hugging Face content.
|
|
344
|
+
Imported state therefore records `torrent-verified`,
|
|
345
|
+
`trusted-infohash`, and `not-upstream-verified` separately. It remains usable
|
|
346
|
+
when upstream has disappeared and can later be checked against upstream if it
|
|
347
|
+
becomes available.
|
|
348
|
+
|
|
349
|
+
## Periodic Jobs
|
|
350
|
+
|
|
351
|
+
For alert-only checks, run verification periodically and let its non-zero exit
|
|
352
|
+
status trigger normal alerting:
|
|
353
|
+
|
|
354
|
+
```bash
|
|
355
|
+
model-mirror verify --all --max-age 30d
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
For a repair pass after verification:
|
|
359
|
+
|
|
360
|
+
```bash
|
|
361
|
+
model-mirror verify --all --max-age 30d || true
|
|
362
|
+
model-mirror repair --all
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
Do not chain verification to repair with `&&`: `verify` exits non-zero when it
|
|
366
|
+
finds repairable damage. `verify --all` skips recently verified clean mirrors
|
|
367
|
+
when `--max-age` is set. Busy mirrors are reported and skipped, the rest of the
|
|
368
|
+
archive is still checked, and the final exit status is non-zero if any mirror is
|
|
369
|
+
dirty, failed, or busy.
|
|
370
|
+
|
|
371
|
+
## Upstream Updates
|
|
372
|
+
|
|
373
|
+
Every online operation resolves the requested revision, usually `main`, to a
|
|
374
|
+
specific Hub commit. The local mirror is tied to that commit.
|
|
375
|
+
|
|
376
|
+
If `verify` sees that upstream `main` now points at a different commit, it marks
|
|
377
|
+
the mirror with `upstream_status: changed` but does not overwrite local files.
|
|
378
|
+
|
|
379
|
+
Repair and update are deliberately separate operations:
|
|
380
|
+
|
|
381
|
+
- `repair` restores missing or damaged files from the mirror's currently
|
|
382
|
+
recorded commit. It never moves the mirror to a different commit, even when
|
|
383
|
+
`verify` has detected an upstream change.
|
|
384
|
+
- `repair --update` explicitly moves the mirror to the changed upstream commit
|
|
385
|
+
recorded by `verify`.
|
|
386
|
+
|
|
387
|
+
Updating one or all changed mirrors is therefore explicit:
|
|
388
|
+
|
|
389
|
+
```bash
|
|
390
|
+
model-mirror repair --update org/model
|
|
391
|
+
model-mirror repair --all --update
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
Use `model-mirror mirror --commit abc123 org/model` when you want a reproducible
|
|
395
|
+
archive pinned to an exact Hub revision.
|
|
396
|
+
|
|
397
|
+
## Common Commands
|
|
398
|
+
|
|
399
|
+
```bash
|
|
400
|
+
model-mirror mirror org/model # download and verify
|
|
401
|
+
model-mirror mirror --no-verify org/model # download without final verification
|
|
402
|
+
model-mirror verify org/model # full verification
|
|
403
|
+
model-mirror verify --cached org/model # use current .manifest hashes
|
|
404
|
+
model-mirror repair org/model # redownload paths from .verification
|
|
405
|
+
model-mirror repair --all # repair all mirrors with recorded repair paths
|
|
406
|
+
model-mirror repair --update org/model # apply a changed upstream commit recorded by verify
|
|
407
|
+
model-mirror offline org/model # local verification only; no Hub checks
|
|
408
|
+
model-mirror online org/model # re-enable Hub checks
|
|
409
|
+
model-mirror list # fast recorded state for every mirror
|
|
410
|
+
model-mirror status # metadata-only archive status
|
|
411
|
+
model-mirror status org/model # concise last-known repository state
|
|
412
|
+
model-mirror status --verbose org/model # full recorded metadata
|
|
413
|
+
model-mirror status --check-upstream org/model # advisory live upstream comparison
|
|
414
|
+
model-mirror status --json # stable machine-readable status
|
|
415
|
+
model-mirror remove org/model # inspect, confirm, and permanently remove one mirror
|
|
416
|
+
model-mirror upgrade org/model # fill missing torrent hash coverage
|
|
417
|
+
model-mirror torrent publish org/model # publish and request durable seeding
|
|
418
|
+
model-mirror torrent join FILE_OR_MAGNET # recover a normal local archive
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
Datasets and Spaces are supported with `--repo-type dataset` or
|
|
422
|
+
`--repo-type space`.
|
|
423
|
+
|
|
424
|
+
## Key Configuration
|
|
425
|
+
|
|
426
|
+
```bash
|
|
427
|
+
model-mirror config show
|
|
428
|
+
model-mirror config options
|
|
429
|
+
model-mirror config directory /mnt/big-drive/huggingface
|
|
430
|
+
model-mirror config set checksum-workers 1
|
|
431
|
+
model-mirror config set hf-xet-reconstruct-write-sequentially true
|
|
432
|
+
model-mirror config set hf-xet-num-concurrent-range-gets 1
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
`config set` accepts either kebab-case or snake_case keys; the saved YAML and
|
|
436
|
+
`config options` use canonical snake_case. `config directory PATH` is a
|
|
437
|
+
convenience form for the most common setting and creates `PATH` immediately.
|
|
438
|
+
`config set directory PATH` saves the same value without creating the directory
|
|
439
|
+
until it is needed.
|
|
440
|
+
|
|
441
|
+
Important configuration options:
|
|
442
|
+
|
|
443
|
+
- `directory`: archive root
|
|
444
|
+
- `repo_type`: default repo type, usually `model`
|
|
445
|
+
- `revision`: default branch, tag, or commit, usually `main`
|
|
446
|
+
- `checksum`: whether mirror/repair writes local hash manifest records
|
|
447
|
+
- `checksum_workers`: checksum hashing concurrency; `1` is HDD-friendly
|
|
448
|
+
- `verify_after_mirror`: run verification after `mirror`
|
|
449
|
+
- `token_path`: Hugging Face token file path; optional when autodetection finds
|
|
450
|
+
`HF_TOKEN_PATH`, `HF_HOME/token`, `~/.cache/huggingface/token`, or
|
|
451
|
+
`~/.huggingface/token`. If no token is found during Hub access,
|
|
452
|
+
model-mirror warns and prints the config command to set this path. Token
|
|
453
|
+
contents are never printed.
|
|
454
|
+
- `cache_dir`: optional override for the Hugging Face/Xet cache; defaults to
|
|
455
|
+
`DIRECTORY/.model-mirror/cache`
|
|
456
|
+
- `tmp_dir`: optional override for staging and temporary files; defaults to
|
|
457
|
+
`DIRECTORY/.model-mirror/tmp`
|
|
458
|
+
- `hf_xet_reconstruct_write_sequentially`: HDD-friendly Xet reconstruction
|
|
459
|
+
writes; uses the current `HF_XET_RECONSTRUCTION_USE_VECTORED_WRITE=false`
|
|
460
|
+
knob when supported
|
|
461
|
+
- `hf_xet_num_concurrent_range_gets`: Xet internal download concurrency.
|
|
462
|
+
Default `1` is HDD-friendly; increase for SSD/NVMe.
|
|
463
|
+
- `hf_xet_high_performance`: enable Xet high-performance mode. This is off by
|
|
464
|
+
default; use only on high-bandwidth machines with fast disks and ample memory,
|
|
465
|
+
typically 64 GB RAM or more.
|
|
466
|
+
|
|
467
|
+
## Disk Space During Transfers
|
|
468
|
+
|
|
469
|
+
Model-mirror does not stage a second complete snapshot. Each file is streamed
|
|
470
|
+
through the HTTP or Xet transport into a sibling `NAME.incomplete` file while
|
|
471
|
+
its integrity hashes are accumulated, then atomically renamed to `NAME`. The
|
|
472
|
+
partial file is the eventual payload allocation rather than an additional full
|
|
473
|
+
copy. With the default `download_workers: 1`, only one payload file is
|
|
474
|
+
incomplete at a time. Same-commit repair removes a damaged file before
|
|
475
|
+
redownloading it, so it likewise does not retain both the damaged and repaired
|
|
476
|
+
copy.
|
|
477
|
+
|
|
478
|
+
Allow space for the final snapshot, active partial files, and auxiliary
|
|
479
|
+
transport cache under `DIRECTORY/.model-mirror/tmp/downloads`. The auxiliary
|
|
480
|
+
cache is not specified as a fixed percentage and can vary with the HTTP/Xet
|
|
481
|
+
implementation, but model-mirror does not intentionally reconstruct a second
|
|
482
|
+
full payload there. Interrupted partial files and staging cache are retained
|
|
483
|
+
for resume.
|
|
484
|
+
`model-mirror clean-cache` scans and previews reclaimable cache space before
|
|
485
|
+
`clean-cache --force` removes it without deleting mirrored payloads. It also
|
|
486
|
+
detects legacy archive-root `.cache/` and `.tmp/` directories left by older
|
|
487
|
+
versions. The `.model-mirror/` control directory itself remains in place. The
|
|
488
|
+
metadata-only `status` command reports active work when its progress heartbeat
|
|
489
|
+
is available; do not run forced cache cleanup while it shows an active download
|
|
490
|
+
or repair.
|
|
491
|
+
|
|
492
|
+
## Locking And Interrupted Commands
|
|
493
|
+
|
|
494
|
+
Repository operations use an advisory kernel `flock` on
|
|
495
|
+
`.verification.lock`. `mirror`, `remove`, `verify`, `repair`, `offline`, `online`,
|
|
496
|
+
`upgrade`, and torrent control-plane transitions take this per-repository lock;
|
|
497
|
+
the managed seeder holds it only during short reconciliation transitions.
|
|
498
|
+
`list` and `status` remain non-blocking and report the command, PID, host, and
|
|
499
|
+
start time for a lock that is actually held.
|
|
500
|
+
|
|
501
|
+
The kernel releases the lock automatically when a process exits or the host
|
|
502
|
+
restarts. The `.verification.lock` file may remain after a crash, but its
|
|
503
|
+
presence alone does not mean the repository is locked; model-mirror probes the
|
|
504
|
+
lock itself and ignores stale contents. Do not delete the file to override a
|
|
505
|
+
genuinely running operation. After a killed download, rerun the same `mirror`
|
|
506
|
+
command to resume its pinned snapshot.
|
|
507
|
+
|
|
508
|
+
## Removing A Mirror
|
|
509
|
+
|
|
510
|
+
`remove` prints the repository identity, type, path, status, exceptional state,
|
|
511
|
+
resolved commit, last verification time and age, payload file count, and exact
|
|
512
|
+
size. By default it requires typing the full repository ID; `--yes` is available
|
|
513
|
+
for deliberate automation:
|
|
514
|
+
|
|
515
|
+
```bash
|
|
516
|
+
model-mirror remove org/model
|
|
517
|
+
model-mirror remove --repo-type dataset org/data
|
|
518
|
+
model-mirror remove --yes org/model
|
|
519
|
+
```
|
|
520
|
+
|
|
521
|
+
Removal is blocked while a torrent publication fence is active. Stop any
|
|
522
|
+
external client first when applicable, then release the local publication:
|
|
523
|
+
|
|
524
|
+
```bash
|
|
525
|
+
model-mirror torrent stop org/model
|
|
526
|
+
model-mirror torrent retire org/model
|
|
527
|
+
model-mirror remove org/model
|
|
528
|
+
```
|
|
529
|
+
|
|
530
|
+
After confirmation, model-mirror atomically moves the mirror from its canonical
|
|
531
|
+
path into `.model-mirror-removals/` under the same archive. It removes payload
|
|
532
|
+
files first, ordinary metadata next, and the preserved removal record and lock
|
|
533
|
+
last. If the command or host stops partway through, rerun the same `remove`
|
|
534
|
+
command to resume. A new mirror created at the original path while an older
|
|
535
|
+
removal is pending is never deleted by that resumed cleanup.
|
|
536
|
+
|
|
537
|
+
Torrent retirement releases local managed seed intent and the update fence, but
|
|
538
|
+
cannot revoke torrent metadata already shared with other peers. Removing a
|
|
539
|
+
repository does not remove archive-wide cache; inspect that separately with
|
|
540
|
+
`model-mirror clean-cache`.
|
|
541
|
+
|
|
542
|
+
## Notes
|
|
543
|
+
|
|
544
|
+
Model-mirror uses Hugging Face metadata and the HTTP/Xet transports, selecting
|
|
545
|
+
Xet automatically when available. It keeps Hugging Face cache and temporary
|
|
546
|
+
directories under the configured archive root so large downloads do not spill
|
|
547
|
+
into the default home cache.
|
|
548
|
+
|
|
549
|
+
See [CONTRIBUTORS.md](CONTRIBUTORS.md) for implementation details, testing, and
|
|
550
|
+
future design notes.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
model_mirror/__init__.py,sha256=dWnOBXDC-F5PQg3w9HtgEUK6tiI2tzlhLZ_71IW5fOY,165
|
|
2
|
+
model_mirror/audit.py,sha256=jmQZJSj3UYniFjSDe_rV9s8XIl8GoTb1893S-JZ0Oco,7888
|
|
3
|
+
model_mirror/checksums.py,sha256=RE0hPjcsXmPg8sxCW-hmpiZ1AJFNdAU7LeCxqSHinJw,11981
|
|
4
|
+
model_mirror/cli.py,sha256=_ztg97l_RYF47o1ToCfQmW7U8n9O2c_1ULGGIuTHdtc,95786
|
|
5
|
+
model_mirror/config.py,sha256=805bDaVtzsRltrL5bjUCzAE-U00HE6Fc_ZxiiwQar1k,11827
|
|
6
|
+
model_mirror/hub.py,sha256=lWcxP3x0nF5u21hniAtUbuzJwOFbYTjK1-ffbAkj520,30697
|
|
7
|
+
model_mirror/lock.py,sha256=lOS4y_jM9GAtBd7CZJo4mpO6b8d1S640RARYHuilo_o,3485
|
|
8
|
+
model_mirror/mirror.py,sha256=v1ocF6Z28n3Oy9zuGMS3flxwNQcpT5rzOZOf1AMt2T8,6933
|
|
9
|
+
model_mirror/progress.py,sha256=Dozscx_YKZqFN3ELqdo_WTzBOr_upU8FnrCwhNTvV5Y,8420
|
|
10
|
+
model_mirror/removal.py,sha256=rR-S3YVGvhyS_v8Q6VnfQ4cWEVFywSzOdMTw4d-2enE,5230
|
|
11
|
+
model_mirror/repair.py,sha256=96TgXqMfSUGouDFkzNAQ5iHH1mUpE3TBjaXiTkClFQs,12408
|
|
12
|
+
model_mirror/state.py,sha256=v771JrO8VjWgVRh7ztolW47vWtKgbeSVNe-8PNvCHG8,5535
|
|
13
|
+
model_mirror/torrent.py,sha256=gNldH3Ui5nrBq_O0MtxU5QaPrjj57akISssayOiK5cs,16579
|
|
14
|
+
model_mirror/torrent_coverage.py,sha256=CobMsF-ku0FKNV4ppuph-55MjBC6KXCB6DFTfPHnZRA,14928
|
|
15
|
+
model_mirror/torrent_hashes.py,sha256=qb62iU8r7wn6zk3AsoAOxib6Y7vzuBfeUUOeuY5vMbI,6227
|
|
16
|
+
model_mirror/torrent_import.py,sha256=4HWOkg_I6uB4OvK-VtbayPgMZ7xLRSwcqr8tex4e67c,25170
|
|
17
|
+
model_mirror/torrent_publication.py,sha256=yeKaaAWXZsKWr4LQuYm_53smOobeufapWp5W79ijPF4,22853
|
|
18
|
+
model_mirror/torrent_seed.py,sha256=uWNfby7isay5VmWq-R3a3a72m15yzKmPU7AMZLMCCoo,8398
|
|
19
|
+
model_mirror/verify.py,sha256=VJgcXcRRSQe_YVnnPYx7KpMKAHjJXHyTvBCfebpKmRY,4477
|
|
20
|
+
model_mirror_cli-0.2.0.dist-info/METADATA,sha256=PrAOzOAgqAEaNQCeDwfNDSoW0KiFSU0CIp12N013YT0,23967
|
|
21
|
+
model_mirror_cli-0.2.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
|
|
22
|
+
model_mirror_cli-0.2.0.dist-info/entry_points.txt,sha256=3TZd_iSp38e0qWULj0iKM39LqeP_N2vb2zRqsOTKFNE,55
|
|
23
|
+
model_mirror_cli-0.2.0.dist-info/licenses/LICENSE,sha256=xXLebneXQDu3LlHRaz2kR5WeEFqvomN4wEZcjDOuHIY,1067
|
|
24
|
+
model_mirror_cli-0.2.0.dist-info/RECORD,,
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 xlr8harder
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|