dsh-h3-image 0.0.0-stage → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +27 -0
- package/README.md +88 -2
- package/README.zh.md +80 -0
- package/SECURITY.md +34 -0
- package/THIRD_PARTY_NOTICES.md +32 -0
- package/assets/te_setup.py +129 -0
- package/cordis.patch.yml +8 -0
- package/icon.svg +6 -0
- package/lib/checks.js +426 -0
- package/lib/gate.js +63 -0
- package/lib/index.js +617 -0
- package/lib/job-supervisor.mjs +79 -0
- package/lib/jobs.js +302 -0
- package/lib/pinned.js +127 -0
- package/lib/runner.js +195 -0
- package/lib/setup.js +526 -0
- package/lib/state.js +74 -0
- package/locale/en.json +6 -0
- package/locale/zh.json +6 -0
- package/package.json +59 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 dsh-h3-image contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
NOTE: This license covers the plugin source code only. The MiniMax H3-Image
|
|
24
|
+
model weights are distributed separately by MiniMax under the MiniMax H3
|
|
25
|
+
Image Community License Agreement (shown inside the setup wizard before any
|
|
26
|
+
download), and the upstream MLX engine is Apache-2.0. See
|
|
27
|
+
THIRD_PARTY_NOTICES.md.
|
package/README.md
CHANGED
|
@@ -1,3 +1,89 @@
|
|
|
1
|
-
#
|
|
1
|
+
# dsh-h3-image
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**English** | [简体中文](README.zh.md)
|
|
4
|
+
|
|
5
|
+
> **Name reserved.** This package version (0.2.0) ships the full local
|
|
6
|
+
> pipeline: the read-only environment check, the one-time setup wizard
|
|
7
|
+
> (venv → pinned engine clone → license gate → sha256-verified weight
|
|
8
|
+
> download with resume → smoke test, each long step a background job that
|
|
9
|
+
> survives restarts), and text-to-image. Nothing here calls any cloud
|
|
10
|
+
> service.
|
|
11
|
+
|
|
12
|
+
A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH)
|
|
13
|
+
plugin bundle that runs the open-source **MiniMax H3-Image** text-to-image
|
|
14
|
+
model **entirely on your own Apple Silicon Mac**, through the
|
|
15
|
+
[MLX](https://github.com/ml-explore/mlx) engine. No API keys, no accounts, no
|
|
16
|
+
per-image fees, no egress during generation.
|
|
17
|
+
|
|
18
|
+
## What you get
|
|
19
|
+
|
|
20
|
+
| Tool | Purpose |
|
|
21
|
+
| --- | --- |
|
|
22
|
+
| `h3_status` | Read-only health check: platform, unified memory, venv, engine commit, weights, text encoder, free disk — each red item carries a fix hint. |
|
|
23
|
+
| `h3_setup` | The one-time setup wizard as a resumable state machine: `check → venv → engine → license → weights → smoke`. Long steps (engine install, ~59 GB of downloads + conversion) run as detached background jobs — submit returns a `job_id` in seconds, `action=poll` reads progress from the engine's own on-disk state, and an interrupted download resumes from `.part` on re-run. A dead job is reported `interrupted`, never "done". `action=license` shows the community license; acceptance is recorded only by you. `action=reset` clears the local state record. |
|
|
24
|
+
| `h3_image` | Text-to-image: one engine subprocess per call, internal deadline, one generation at a time (a second call gets an immediate `busy`), PNG verified before it is reported, output lands in your session workspace. |
|
|
25
|
+
|
|
26
|
+
## Requirements
|
|
27
|
+
|
|
28
|
+
- Apple Silicon Mac (the engine is arm64-only; Intel/Windows/Linux are
|
|
29
|
+
declined by `h3_status`, not by a crash).
|
|
30
|
+
- **48 GB+ unified memory recommended**; 32 GB machines are warned-then-
|
|
31
|
+
confirmed, below 32 GB is refused.
|
|
32
|
+
- ~60 GB free disk during setup (measured peak on a from-zero install; the
|
|
33
|
+
steady state after the engine deletes source shards is ~53 GB — see the
|
|
34
|
+
resource ledger below).
|
|
35
|
+
- DSH ≥ 0.2.2-rc.1, Node ≥ 22.19.
|
|
36
|
+
|
|
37
|
+
## Resource ledger (measured, bare-metal acceptance 2026-11-12/13)
|
|
38
|
+
|
|
39
|
+
Every number below comes from one from-zero acceptance run on a 275 GB
|
|
40
|
+
Apple Silicon machine (no reuse candidates, fresh venv/engine/weights) —
|
|
41
|
+
not from estimates.
|
|
42
|
+
|
|
43
|
+
| Item | Measured |
|
|
44
|
+
| --- | --- |
|
|
45
|
+
| Disk peak (incremental, whole setup) | 58.6 GB (54.6 GiB): DiT pack 21.5 + text encoder 28.2 + one in-flight source shard 5.4 + venv 1.1 |
|
|
46
|
+
| Disk steady state (post-setup) | 53.2 GB (source shards deleted by the engine after conversion) |
|
|
47
|
+
| venv creation | ~18 s (reuses a suitable standalone CPython when found; reports a miss otherwise) |
|
|
48
|
+
| Engine clone + pip install | ~72 s |
|
|
49
|
+
| Weights: per transformer shard (download 5.5 GB + Q8 convert + delete) | ~4–5 min |
|
|
50
|
+
| Weights: resume segment (7 shards + 28.2 GB text encoder) | 41 min |
|
|
51
|
+
| Smoke generation | 30 s wall / 10.8 s engine time (1 MP, 6 steps) |
|
|
52
|
+
| Generation peak memory (engine metrics) | 27.5 GB per image at 1 MP |
|
|
53
|
+
| Host memory in use during the run | 274 GB — includes a resident inference server; the plugin's own footprint is the 27.5 GB above |
|
|
54
|
+
|
|
55
|
+
Interrupted steps resume: a killed download continues from its `.part`
|
|
56
|
+
(Range requests), and finished shards are never re-fetched — verified by
|
|
57
|
+
the acceptance run's resume audit (zero re-downloads of completed shards).
|
|
58
|
+
|
|
59
|
+
## Licenses — read before downloading weights
|
|
60
|
+
|
|
61
|
+
This plugin's **code** is MIT. It does not ship, re-host or re-distribute the
|
|
62
|
+
model. The setup wizard downloads the weights from the official Hugging Face
|
|
63
|
+
repository at a pinned revision, and **shows you the MiniMax H3 Image
|
|
64
|
+
Community License Agreement — including its territorial exclusions — before
|
|
65
|
+
any byte is fetched. You accept it yourself; the plugin never accepts on your
|
|
66
|
+
behalf.** The upstream engine is Apache-2.0. See
|
|
67
|
+
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
|
|
68
|
+
|
|
69
|
+
## Install (DSH)
|
|
70
|
+
|
|
71
|
+
```
|
|
72
|
+
plugin_manager → install_bundle → <this repo or the npm package>
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Then enable the bundle in your profile and run `h3_status` — it tells you
|
|
76
|
+
exactly what the machine still needs.
|
|
77
|
+
|
|
78
|
+
## Development
|
|
79
|
+
|
|
80
|
+
```sh
|
|
81
|
+
npm test # unit tests (pure check logic)
|
|
82
|
+
npm run release # gates + pack + tarball scan
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
Two release gates run in CI and on `prepack`:
|
|
86
|
+
`scripts/check-release-safety.mjs` (no machine fingerprints, no AppleDouble,
|
|
87
|
+
no secrets — in source **and** in the built tarball) and
|
|
88
|
+
`scripts/check-version-consistency.mjs` (package.json ↔ CHANGELOG ↔ locales ↔
|
|
89
|
+
pinned revisions).
|
package/README.zh.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# dsh-h3-image
|
|
2
|
+
|
|
3
|
+
[English](README.md) | **简体中文**
|
|
4
|
+
|
|
5
|
+
> **名称已占位。** 当前版本(0.2.0)交付完整本地链路:只读环境体检、一键安装
|
|
6
|
+
> 向导(venv → 钉定引擎 clone → 许可确认 → sha256 校验的权重下载(支持断点
|
|
7
|
+
> 续传)→ 冒烟出图,每个长步骤都是重启不丢的后台任务),以及本地文生图。
|
|
8
|
+
> 全程不调用任何云服务。
|
|
9
|
+
|
|
10
|
+
一个 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(DSH)
|
|
11
|
+
插件包,通过 [MLX](https://github.com/ml-explore/mlx) 引擎在你的 **Apple
|
|
12
|
+
Silicon Mac 上完全本地**运行开源的 **MiniMax H3-Image** 文生图模型。
|
|
13
|
+
零 API 密钥、零账号、零按图计费,生图过程零出网。
|
|
14
|
+
|
|
15
|
+
## 提供什么
|
|
16
|
+
|
|
17
|
+
| 工具 | 用途 |
|
|
18
|
+
| --- | --- |
|
|
19
|
+
| `h3_status` | 只读体检:平台、统一内存、venv、引擎 commit、权重、文本编码器、可用磁盘——每一项红灯都附修复提示。 |
|
|
20
|
+
| `h3_setup` | 一次性安装向导,可续跑的状态机:`check → venv → engine → license → weights → smoke`。长步骤(引擎安装、约 59 GB 下载 + 转换)以脱离进程组的后台任务运行——提交秒级返回 `job_id`,`action=poll` 从引擎自己的磁盘状态读进度,中断的下载重跑即从 `.part` 续传。进程已死的任务如实报 `interrupted`,绝不谎报"完成"。`action=license` 展示社区许可,只有你本人确认才会记录接受。`action=reset` 清除本机状态记录。 |
|
|
21
|
+
| `h3_image` | 文生图:每次调用一个引擎子进程,内置截止时间,同一时刻只跑一张(第二路立即收到 `busy`),PNG 结构校验通过才报成功,产物落在会话工作区。 |
|
|
22
|
+
|
|
23
|
+
## 环境要求
|
|
24
|
+
|
|
25
|
+
- Apple Silicon Mac(引擎仅支持 arm64;Intel/Windows/Linux 在 `h3_status`
|
|
26
|
+
检查阶段即明确拒绝,而不是崩溃)。
|
|
27
|
+
- **推荐 48 GB 及以上统一内存**;32 GB 机器先警告再确认,低于 32 GB 拒绝。
|
|
28
|
+
- 安装期约 60 GB 可用磁盘(from-zero 安装的实测峰值;引擎删除源分片后的
|
|
29
|
+
稳态约 53 GB——见下方资源账表)。
|
|
30
|
+
- DSH ≥ 0.2.2-rc.1,Node ≥ 22.19。
|
|
31
|
+
|
|
32
|
+
## 资源账(实测,裸机验收 2026-11-12/13)
|
|
33
|
+
|
|
34
|
+
下表每个数字都来自 275 GB Apple Silicon 机器上的一次 from-zero 验收运行
|
|
35
|
+
(无复用候选、全新 venv/引擎/权重)——不是估算。
|
|
36
|
+
|
|
37
|
+
| 项目 | 实测 |
|
|
38
|
+
| --- | --- |
|
|
39
|
+
| 磁盘峰值(增量,整个安装过程) | 58.6 GB(54.6 GiB):DiT 打包 21.5 + 文本编码器 28.2 + 在途源分片 5.4 + venv 1.1 |
|
|
40
|
+
| 磁盘稳态(安装完成后) | 53.2 GB(源分片由引擎转换后自行删除) |
|
|
41
|
+
| venv 创建 | 约 18 秒(能发现合适的独立 CPython 就复用,探测不到如实报 miss) |
|
|
42
|
+
| 引擎 clone + pip 安装 | 约 72 秒 |
|
|
43
|
+
| 权重:每个 transformer 分片(下载 5.5 GB + Q8 转换 + 删除) | 约 4–5 分钟 |
|
|
44
|
+
| 权重:续传段(7 分片 + 28.2 GB 文本编码器) | 41 分钟 |
|
|
45
|
+
| 冒烟出图 | 墙钟 30 秒 / 引擎耗时 10.8 秒(1 MP,6 步) |
|
|
46
|
+
| 单图内存峰值(引擎 metrics) | 1 MP 每图 27.5 GB |
|
|
47
|
+
| 运行期间主机已用内存 | 274 GB——含常驻推理服务;插件自身占用即上行 27.5 GB |
|
|
48
|
+
|
|
49
|
+
中断可续:被杀的下载从 `.part` 继续(Range 请求),已完成分片绝不重下——
|
|
50
|
+
验收运行的续传对账为证(零重下已完成分片)。
|
|
51
|
+
|
|
52
|
+
## 许可——下载权重前必读
|
|
53
|
+
|
|
54
|
+
本插件的**代码**是 MIT。**不打包、不转载、不分发模型。** 安装向导从官方
|
|
55
|
+
Hugging Face 仓库按钉定 revision 下载权重,并且**在任何一个字节下载之前,
|
|
56
|
+
向你展示 MiniMax H3 Image 社区许可协议(含其地域排除条款)。由你本人接受;
|
|
57
|
+
插件永不代你接受。** 上游引擎为 Apache-2.0。见
|
|
58
|
+
[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)。
|
|
59
|
+
|
|
60
|
+
## 安装(DSH)
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
plugin_manager → install_bundle → <本仓库或 npm 包>
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
然后在 profile 中启用该 bundle,跑一次 `h3_status`——机器还缺什么它逐项
|
|
67
|
+
告诉你。
|
|
68
|
+
|
|
69
|
+
## 开发
|
|
70
|
+
|
|
71
|
+
```sh
|
|
72
|
+
npm test # 单元测试(纯检查逻辑)
|
|
73
|
+
npm run release # 门禁 + 打包 + tarball 扫描
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
两道发布门禁在 CI 和 `prepack` 上运行:
|
|
77
|
+
`scripts/check-release-safety.mjs`(零机器指纹、零 AppleDouble、零密钥——
|
|
78
|
+
源码**和**构建出的 tarball 都扫)与
|
|
79
|
+
`scripts/check-version-consistency.mjs`(package.json ↔ CHANGELOG ↔ locale ↔
|
|
80
|
+
钉定 revision 一致性)。
|
package/SECURITY.md
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
## Design summary
|
|
4
|
+
|
|
5
|
+
- **No secrets.** The plugin has no API keys, tokens, or credentials of any
|
|
6
|
+
kind. Local inference needs none.
|
|
7
|
+
- **Zero runtime egress.** Generation is pure local subprocess work. The only
|
|
8
|
+
network activity in the plugin's lifetime is the setup wizard fetching the
|
|
9
|
+
pinned engine (GitHub) and weights (huggingface.co, or an optional
|
|
10
|
+
hf-mirror you explicitly choose). Every fetched file is sha256-verified
|
|
11
|
+
against a value recorded in `lib/pinned.js`.
|
|
12
|
+
- **Sandboxed outputs.** Generated images are written inside the session
|
|
13
|
+
workspace by default; an `output_dir` outside the workspace is rejected.
|
|
14
|
+
- **Subprocess discipline.** The engine runs as a child process from the
|
|
15
|
+
pinned commit inside an isolated venv; prompts are passed as arguments,
|
|
16
|
+
never interpolated into a shell string. Concurrency gate = 1.
|
|
17
|
+
- **License self-acceptance.** Weight download requires the user's own
|
|
18
|
+
acceptance of the MiniMax community license (including its territorial
|
|
19
|
+
terms); the plugin records the acceptance timestamp locally and never
|
|
20
|
+
accepts on the user's behalf.
|
|
21
|
+
|
|
22
|
+
## Reporting a vulnerability
|
|
23
|
+
|
|
24
|
+
Open a **private security advisory** on the GitHub repository
|
|
25
|
+
(`aiohow/dsh-h3-image` → Security → Report a vulnerability). Please do not
|
|
26
|
+
open a public issue for unfixed vulnerabilities. We aim to respond within 7
|
|
27
|
+
days.
|
|
28
|
+
|
|
29
|
+
## Out of scope
|
|
30
|
+
|
|
31
|
+
- Vulnerabilities in the upstream engine (`minimax-h3-mlx`) or model weights
|
|
32
|
+
— report those to their respective upstreams; we will pin a fixed revision
|
|
33
|
+
promptly if a release affects users.
|
|
34
|
+
- Attacks that already require code execution on your machine.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Third-party notices
|
|
2
|
+
|
|
3
|
+
dsh-h3-image (plugin code) is MIT-licensed. It does **not** bundle, vendor,
|
|
4
|
+
re-host or re-distribute any of the following; the setup wizard downloads
|
|
5
|
+
each item from its official upstream at a pinned revision, on the user's
|
|
6
|
+
machine, after the user accepts the respective license where required.
|
|
7
|
+
|
|
8
|
+
## MLX engine — `minimax-h3-mlx`
|
|
9
|
+
|
|
10
|
+
- Upstream: <https://github.com/mrbizarro/minimax-h3-mlx>
|
|
11
|
+
- License: Apache-2.0 (see the upstream `LICENSE`)
|
|
12
|
+
- Pinned commit: recorded in `lib/pinned.js` (`ENGINE.commit`). The wizard
|
|
13
|
+
clones and checks out exactly this commit; `main` is never followed.
|
|
14
|
+
|
|
15
|
+
## Model weights — `MiniMaxAI/MiniMax-H3-Image` (+ text encoder from `MiniMaxAI/MiniMax-H3`)
|
|
16
|
+
|
|
17
|
+
- Upstream: <https://huggingface.co/MiniMaxAI/MiniMax-H3-Image>
|
|
18
|
+
- License: MiniMax H3 Image Community License Agreement (upstream `LICENSE`
|
|
19
|
+
file). **The license includes territorial exclusions** — read the upstream
|
|
20
|
+
LICENSE text yourself before downloading; the wizard shows it and requires
|
|
21
|
+
your explicit acceptance. The plugin never accepts on your behalf.
|
|
22
|
+
- Pinned revisions: recorded in `lib/pinned.js` (`WEIGHTS.revision`,
|
|
23
|
+
`TEXT_ENCODER.revision`); every downloaded file is verified against a
|
|
24
|
+
recorded sha256.
|
|
25
|
+
|
|
26
|
+
## Runtime — Python ≥ 3.13 + MLX
|
|
27
|
+
|
|
28
|
+
- The wizard creates an isolated virtual environment on your machine and
|
|
29
|
+
installs the engine's declared dependencies (MLX is Apple-licensed, other
|
|
30
|
+
packages under their own licenses). The plugin ships none of them.
|
|
31
|
+
|
|
32
|
+
No trademark or brand assets of MiniMax are included in this package.
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Text-encoder fetch driver for dsh-h3-image (bundled asset, not engine code).
|
|
3
|
+
|
|
4
|
+
Why this exists instead of calling the engine's download_selected.py: that
|
|
5
|
+
script downloads three repos including video/audio VAEs and a pruned bf16
|
|
6
|
+
checkpoint this plugin's image path never loads, and it takes no pinning
|
|
7
|
+
parameters. This driver downloads EXACTLY the pinned file set, verifies the
|
|
8
|
+
main safetensors against a pinned sha256 + byte size, and honours a reuse
|
|
9
|
+
candidate list (sha256 verdict only — a size match alone never authorizes
|
|
10
|
+
reuse; it is reused by symlink, never by copy).
|
|
11
|
+
|
|
12
|
+
Progress contract (parsed by lib/jobs.js, all flushed):
|
|
13
|
+
[te n/N] fetched <name>
|
|
14
|
+
[te n/N] reused <name> (<path>)
|
|
15
|
+
[te n/N] skipped <name> (already present)
|
|
16
|
+
TE OK
|
|
17
|
+
|
|
18
|
+
Resumability: huggingface_hub keeps .incomplete part files and resumes them;
|
|
19
|
+
small files are atomic via .part + rename.
|
|
20
|
+
|
|
21
|
+
Mirror: set HF_ENDPOINT before launching (e.g. https://hf-mirror.com) —
|
|
22
|
+
hf_hub_download honours it. Nothing here hardcodes an endpoint.
|
|
23
|
+
"""
|
|
24
|
+
|
|
25
|
+
from __future__ import annotations
|
|
26
|
+
|
|
27
|
+
import argparse
|
|
28
|
+
import hashlib
|
|
29
|
+
import os
|
|
30
|
+
import sys
|
|
31
|
+
from pathlib import Path
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
def sha256_of(path: Path) -> str:
|
|
35
|
+
h = hashlib.sha256()
|
|
36
|
+
with path.open("rb") as fh:
|
|
37
|
+
for chunk in iter(lambda: fh.read(1 << 22), b""):
|
|
38
|
+
h.update(chunk)
|
|
39
|
+
return h.hexdigest()
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
def main() -> int:
|
|
43
|
+
ap = argparse.ArgumentParser(description=__doc__)
|
|
44
|
+
ap.add_argument("--dest", type=Path, required=True, help="text-encoder folder (the engine's --text-dir)")
|
|
45
|
+
ap.add_argument("--meta-dest", type=Path, required=True, help="folder for upstream meta (FL2VA/...)")
|
|
46
|
+
ap.add_argument("--repo", required=True)
|
|
47
|
+
ap.add_argument("--revision", required=True)
|
|
48
|
+
ap.add_argument("--files", required=True, help="comma-separated file list from the pinned package")
|
|
49
|
+
ap.add_argument("--main-file", default="text_encoder.safetensors")
|
|
50
|
+
ap.add_argument("--sha256", required=True, help="pinned sha256 of the main file")
|
|
51
|
+
ap.add_argument("--bytes", type=int, required=True, help="pinned byte size of the main file")
|
|
52
|
+
ap.add_argument("--meta-repo", required=True)
|
|
53
|
+
ap.add_argument("--meta-revision", required=True)
|
|
54
|
+
ap.add_argument("--meta-files", required=True, help="comma-separated upstream paths (kept under meta-dest)")
|
|
55
|
+
ap.add_argument("--candidates-file", type=Path, default=None, help="newline list of size-matching candidate paths to hash")
|
|
56
|
+
args = ap.parse_args()
|
|
57
|
+
|
|
58
|
+
# Xet high-performance mode is documented by Hugging Face for large single
|
|
59
|
+
# files; harmless when the endpoint does not serve Xet.
|
|
60
|
+
os.environ.setdefault("HF_XET_HIGH_PERFORMANCE", "1")
|
|
61
|
+
from huggingface_hub import hf_hub_download
|
|
62
|
+
|
|
63
|
+
files = [f for f in args.files.split(",") if f]
|
|
64
|
+
total = len(files)
|
|
65
|
+
args.dest.mkdir(parents=True, exist_ok=True)
|
|
66
|
+
|
|
67
|
+
def download(repo: str, revision: str, name: str, dest: Path) -> Path:
|
|
68
|
+
dest.mkdir(parents=True, exist_ok=True)
|
|
69
|
+
return Path(hf_hub_download(repo, filename=name, revision=revision, local_dir=str(dest)))
|
|
70
|
+
|
|
71
|
+
reused = False
|
|
72
|
+
main_path = args.dest / args.main_file
|
|
73
|
+
if main_path.exists() and main_path.stat().st_size == args.bytes and sha256_of(main_path) == args.sha256:
|
|
74
|
+
print(f"[te 1/{total}] skipped {args.main_file} (already present, sha256 match)", flush=True)
|
|
75
|
+
reused = True
|
|
76
|
+
else:
|
|
77
|
+
candidate_ok = False
|
|
78
|
+
if args.candidates_file is not None and args.candidates_file.exists():
|
|
79
|
+
for line in args.candidates_file.read_text().splitlines():
|
|
80
|
+
cand = Path(line.strip())
|
|
81
|
+
if not cand.is_file() or cand.stat().st_size != args.bytes:
|
|
82
|
+
continue # size mismatch: not even worth hashing
|
|
83
|
+
print(f"[te 0/{total}] hashing reuse candidate {cand.name} ({args.bytes / 1e9:.1f} GB)", flush=True)
|
|
84
|
+
if sha256_of(cand) == args.sha256:
|
|
85
|
+
# Exact digest match (design D3): reuse by symlink, never copy.
|
|
86
|
+
if main_path.exists() or main_path.is_symlink():
|
|
87
|
+
main_path.unlink()
|
|
88
|
+
main_path.symlink_to(cand)
|
|
89
|
+
print(f"[te 1/{total}] reused {args.main_file} (sha256 verified)", flush=True)
|
|
90
|
+
candidate_ok = True
|
|
91
|
+
reused = True
|
|
92
|
+
break
|
|
93
|
+
print(f"[te 0/{total}] candidate {cand.name} hash MISMATCH — will download", flush=True)
|
|
94
|
+
if not candidate_ok:
|
|
95
|
+
got = download(args.repo, args.revision, args.main_file, args.dest)
|
|
96
|
+
size = got.stat().st_size
|
|
97
|
+
if size != args.bytes:
|
|
98
|
+
print(f"SIZE MISMATCH {args.main_file}: got {size}, expected {args.bytes}", flush=True)
|
|
99
|
+
return 3
|
|
100
|
+
digest = sha256_of(got)
|
|
101
|
+
if digest != args.sha256:
|
|
102
|
+
print(f"SHA256 MISMATCH {args.main_file}: got {digest}, expected {args.sha256}", flush=True)
|
|
103
|
+
got.unlink(missing_ok=True)
|
|
104
|
+
return 3
|
|
105
|
+
print(f"[te 1/{total}] fetched {args.main_file} (sha256 verified)", flush=True)
|
|
106
|
+
|
|
107
|
+
rest = [f for f in files if f != args.main_file]
|
|
108
|
+
for i, name in enumerate(rest, start=2):
|
|
109
|
+
target = args.dest / name
|
|
110
|
+
if target.exists() and target.stat().st_size > 0:
|
|
111
|
+
print(f"[te {i}/{total}] skipped {name} (already present)", flush=True)
|
|
112
|
+
continue
|
|
113
|
+
download(args.repo, args.revision, name, args.dest)
|
|
114
|
+
print(f"[te {i}/{total}] fetched {name}", flush=True)
|
|
115
|
+
|
|
116
|
+
for name in [f for f in args.meta_files.split(",") if f]:
|
|
117
|
+
target = args.meta_dest / name
|
|
118
|
+
if target.exists() and target.stat().st_size > 0:
|
|
119
|
+
print(f"[te meta] skipped {name} (already present)", flush=True)
|
|
120
|
+
continue
|
|
121
|
+
download(args.meta_repo, args.meta_revision, name, args.meta_dest)
|
|
122
|
+
print(f"[te meta] fetched {name}", flush=True)
|
|
123
|
+
|
|
124
|
+
print("TE OK", flush=True)
|
|
125
|
+
return 0
|
|
126
|
+
|
|
127
|
+
|
|
128
|
+
if __name__ == "__main__":
|
|
129
|
+
raise SystemExit(main())
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# dsh-h3-image bundle patch: insert this package's Host half.
|
|
2
|
+
# Loaded only when the profile selects the bundle
|
|
3
|
+
# (dsh.profile.bundles: [..., "dsh-h3-image"]) or the entry is added
|
|
4
|
+
# to the profile's cordis.patch.yml as an `insert`.
|
|
5
|
+
- insert:
|
|
6
|
+
- id: h3-image
|
|
7
|
+
name: dsh-h3-image
|
|
8
|
+
config: {}
|
package/icon.svg
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="36" height="36" viewBox="0 0 36 36">
|
|
2
|
+
<rect x="6" y="5.5" width="24" height="17" rx="3" fill="none" stroke="#4c6ef5" stroke-width="2.4"/>
|
|
3
|
+
<circle cx="13.5" cy="11.5" r="2.2" fill="#4c6ef5"/>
|
|
4
|
+
<path d="M8.6 19.6l5.2-5.8 3.8 4.2 3.4-3.9 5.4 5.5z" fill="#4c6ef5"/>
|
|
5
|
+
<path d="M22.4 23.4l-3.4 5.2h3.1l-1.1 3.9 4.9-5.9h-3.1l2.1-3.2z" fill="#2f9e8f"/>
|
|
6
|
+
</svg>
|