dsh-h3-image 0.0.0-stage → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +30 -0
- package/README.md +138 -2
- package/SECURITY.md +34 -0
- package/THIRD_PARTY_NOTICES.md +46 -0
- package/assets/te_setup.py +129 -0
- package/cordis.patch.yml +8 -0
- package/icon.svg +6 -0
- package/lib/checks.js +426 -0
- package/lib/gate.js +63 -0
- package/lib/index.js +617 -0
- package/lib/job-supervisor.mjs +79 -0
- package/lib/jobs.js +302 -0
- package/lib/pinned.js +135 -0
- package/lib/runner.js +195 -0
- package/lib/setup.js +526 -0
- package/lib/state.js +94 -0
- package/locale/en.json +6 -0
- package/locale/zh.json +6 -0
- package/package.json +58 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 dsh-h3-image contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
NOTE: This license covers the plugin source code only. The MiniMax H3-Image
|
|
24
|
+
model weights are distributed separately by MiniMax under the MiniMax H3
|
|
25
|
+
Image Community License Agreement, and the quantized text encoder under the
|
|
26
|
+
MiniMax H3 Community License Agreement; the setup wizard shows the key
|
|
27
|
+
points of both (including territorial exclusions) with direct links to the
|
|
28
|
+
full texts before any download, and the original agreement files land on
|
|
29
|
+
your disk with the downloads. The upstream MLX engine is Apache-2.0. See
|
|
30
|
+
THIRD_PARTY_NOTICES.md.
|
package/README.md
CHANGED
|
@@ -1,3 +1,139 @@
|
|
|
1
|
-
#
|
|
1
|
+
# dsh-h3-image
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[简体中文](#简体中文) | [English](#english)
|
|
4
|
+
|
|
5
|
+
## 简体中文
|
|
6
|
+
|
|
7
|
+
> **非官方声明**:本插件是社区独立开发的第三方工具,与 MiniMax 无任何关联、未获其背书;"MiniMax-H3-Image" 仅用于指示所支持的模型。权重依 MiniMax 社区许可协议提供,**授权地域不含欧盟、英国、韩国、美国**——请自行确认你所在司法辖区可合法使用。完整免责与商用提示见 [docs/DISCLAIMER.md](docs/DISCLAIMER.md)。
|
|
8
|
+
|
|
9
|
+
一个 [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)(DSH)插件包:在**你自己的 Apple Silicon Mac** 上,通过 [MLX](https://github.com/ml-explore/mlx) 引擎运行开源的 **MiniMax H3-Image** 文生图模型,全程完全本地。零 API 密钥、零账号、零按张费用,生图过程零出网。
|
|
10
|
+
|
|
11
|
+
当前版本交付完整本地链路:只读环境体检、一键安装向导(venv → 钉定引擎 clone → 许可确认 → sha256 校验的权重下载(支持断点续传)→ 冒烟出图,每个长步骤都是宿主重启不丢的后台任务),以及本地文生图。
|
|
12
|
+
|
|
13
|
+
### 你能得到什么
|
|
14
|
+
|
|
15
|
+
| 工具 | 用途 |
|
|
16
|
+
| --- | --- |
|
|
17
|
+
| `h3_status` | 只读环境体检:平台、统一内存、venv、引擎 commit、权重、文本编码器、可用磁盘——每个红项附修复提示。 |
|
|
18
|
+
| `h3_setup` | 一键安装向导,可续传状态机:`check → venv → engine → license → weights → smoke`。长步骤(引擎安装、约 59 GB 下载+转换)以分离后台任务运行——提交秒级返回 `job_id`,`action=poll` 从引擎自身的磁盘状态读进度,中断的下载重跑时从 `.part` 续传。死掉的任务如实报 `interrupted`,绝不谎报"完成"。`action=license` 展示社区许可,接受记录只能由你本人产生;`action=reset` 清除本地状态记录。 |
|
|
19
|
+
| `h3_image` | 文生图:每次调用一个引擎子进程、内部 deadline、同一时刻只跑一路(第二路立即收到 `busy`)、PNG 结构校验通过才上报,产物落在你的会话工作区。 |
|
|
20
|
+
|
|
21
|
+
### 运行要求
|
|
22
|
+
|
|
23
|
+
- Apple Silicon Mac(引擎仅支持 arm64;Intel/Windows/Linux 由 `h3_status` 明确拒绝,而不是崩溃)。
|
|
24
|
+
- **建议 48GB 及以上统一内存**;32GB 机型先警告再确认,低于 32GB 拒绝。
|
|
25
|
+
- 安装期约 60GB 可用磁盘(从零安装实测峰值;引擎删除源分片后稳态约 53GB——见下方资源账)。
|
|
26
|
+
- DSH ≥ 0.2.2-rc.1,Node ≥ 22.19。
|
|
27
|
+
|
|
28
|
+
### 资源账(实测,裸机验收 2026-11-12/13)
|
|
29
|
+
|
|
30
|
+
以下每个数字都来自 275GB Apple Silicon 机器上的一次从零验收运行(无复用候选、全新 venv/引擎/权重)——不是估算。
|
|
31
|
+
|
|
32
|
+
| 项目 | 实测 |
|
|
33
|
+
| --- | --- |
|
|
34
|
+
| 磁盘峰值(增量,全程) | 58.6 GB (54.6 GiB):DiT pack 21.5 + 文本编码器 28.2 + 在途源分片 5.4 + venv 1.1 |
|
|
35
|
+
| 磁盘稳态(安装完成后) | 53.2 GB(引擎转换后自删源分片) |
|
|
36
|
+
| venv 创建 | ~18 s(找到合适的独立 CPython 即复用,否则如实报 miss) |
|
|
37
|
+
| 引擎 clone + pip install | ~72 s |
|
|
38
|
+
| 权重:单个 transformer 分片(下载 5.5 GB + Q8 转换 + 删除) | ~4–5 分钟 |
|
|
39
|
+
| 权重:续传段(7 分片 + 28.2 GB 文本编码器) | 41 分钟 |
|
|
40
|
+
| 冒烟出图 | 30 s 墙钟 / 10.8 s 引擎时间(1 MP,6 步) |
|
|
41
|
+
| 生图内存峰值(引擎 metrics) | 27.5 GB / 张 @ 1 MP |
|
|
42
|
+
| 运行期间宿主已用内存 | 274 GB——含常驻推理服务;插件自身占用即上行的 27.5 GB |
|
|
43
|
+
|
|
44
|
+
中断的步骤可续传:被杀的下载从 `.part` 继续(Range 请求),已完成分片绝不重下——验收运行的续传审计实证(已完成分片零重下)。
|
|
45
|
+
|
|
46
|
+
### 许可——下载权重前必读
|
|
47
|
+
|
|
48
|
+
本插件的**代码**是 MIT,不打包、不托管、不再分发模型——模型与其许可条款与本插件无关,插件只是带许可门的下载器与运行器。安装向导从 Hugging Face 官方仓库按钉定 revision 下载权重,并在拉取任何字节之前**展示两份协议的要点与全文直链**(权重:MiniMax H3 Image 社区许可;量化文本编码器:MiniMax H3 社区许可,均来自各自上游仓库),**点名地域排除(EU/UK/KR/US)**;协议原件随下载落盘在你的模型目录里。由你本人确认接受;插件永不代你接受。上游引擎为 Apache-2.0。详见 [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)。
|
|
49
|
+
|
|
50
|
+
### 安装(DSH)
|
|
51
|
+
|
|
52
|
+
在 DSH 里用 `plugin_manager` 安装,两种来源任选:
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
install_bundle → dsh-h3-image # 从 npm 装最新发布版
|
|
56
|
+
install_bundle → /path/to/dsh-h3-image # 或本地仓库/解包目录
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
然后在 profile 中启用该 bundle,跑一次 `h3_status`——机器还缺什么它会逐项告诉你。
|
|
60
|
+
|
|
61
|
+
### 开发
|
|
62
|
+
|
|
63
|
+
```sh
|
|
64
|
+
npm test # 单测(纯检查逻辑)
|
|
65
|
+
npm run release # 门禁 + pack + tarball 扫描
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
两道发布门禁在 CI 与 `prepack` 运行:`scripts/check-release-safety.mjs`(源码**与**产物 tarball 双扫:零机器指纹、零 AppleDouble、零密钥)与 `scripts/check-version-consistency.mjs`(package.json ↔ CHANGELOG ↔ locale ↔ 钉定 revision 四处一致)。
|
|
69
|
+
|
|
70
|
+
---
|
|
71
|
+
|
|
72
|
+
## English
|
|
73
|
+
|
|
74
|
+
> **Unofficial**: a third-party community tool, not affiliated with or endorsed by MiniMax; "MiniMax-H3-Image" is referenced descriptively only. Weights are provided under the MiniMax Community License, whose authorized territory **excludes the EU, UK, South Korea and the US** — verify you may lawfully use the model in your jurisdiction. Full disclaimer: [docs/DISCLAIMER.md](docs/DISCLAIMER.md).
|
|
75
|
+
|
|
76
|
+
A [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness) (DSH) plugin bundle that runs the open-source **MiniMax H3-Image** text-to-image model **entirely on your own Apple Silicon Mac**, through the [MLX](https://github.com/ml-explore/mlx) engine. No API keys, no accounts, no per-image fees, no egress during generation.
|
|
77
|
+
|
|
78
|
+
The current version ships the full local pipeline: the read-only environment check, the one-time setup wizard (venv → pinned engine clone → license gate → sha256-verified weight download with resume → smoke test, each long step a background job that survives restarts), and text-to-image.
|
|
79
|
+
|
|
80
|
+
### What you get
|
|
81
|
+
|
|
82
|
+
| Tool | Purpose |
|
|
83
|
+
| --- | --- |
|
|
84
|
+
| `h3_status` | Read-only health check: platform, unified memory, venv, engine commit, weights, text encoder, free disk — each red item carries a fix hint. |
|
|
85
|
+
| `h3_setup` | The one-time setup wizard as a resumable state machine: `check → venv → engine → license → weights → smoke`. Long steps (engine install, ~59 GB of downloads + conversion) run as detached background jobs — submit returns a `job_id` in seconds, `action=poll` reads progress from the engine's own on-disk state, and an interrupted download resumes from `.part` on re-run. A dead job is reported `interrupted`, never "done". `action=license` shows the community license; acceptance is recorded only by you. `action=reset` clears the local state record. |
|
|
86
|
+
| `h3_image` | Text-to-image: one engine subprocess per call, internal deadline, one generation at a time (a second call gets an immediate `busy`), PNG verified before it is reported, output lands in your session workspace. |
|
|
87
|
+
|
|
88
|
+
### Requirements
|
|
89
|
+
|
|
90
|
+
- Apple Silicon Mac (the engine is arm64-only; Intel/Windows/Linux are declined by `h3_status`, not by a crash).
|
|
91
|
+
- **48 GB+ unified memory recommended**; 32 GB machines are warned-then-confirmed, below 32 GB is refused.
|
|
92
|
+
- ~60 GB free disk during setup (measured peak on a from-zero install; the steady state after the engine deletes source shards is ~53 GB — see the resource ledger below).
|
|
93
|
+
- DSH ≥ 0.2.2-rc.1, Node ≥ 22.19.
|
|
94
|
+
|
|
95
|
+
### Resource ledger (measured, bare-metal acceptance 2026-11-12/13)
|
|
96
|
+
|
|
97
|
+
Every number below comes from one from-zero acceptance run on a 275 GB Apple Silicon machine (no reuse candidates, fresh venv/engine/weights) — not from estimates.
|
|
98
|
+
|
|
99
|
+
| Item | Measured |
|
|
100
|
+
| --- | --- |
|
|
101
|
+
| Disk peak (incremental, whole setup) | 58.6 GB (54.6 GiB): DiT pack 21.5 + text encoder 28.2 + one in-flight source shard 5.4 + venv 1.1 |
|
|
102
|
+
| Disk steady state (post-setup) | 53.2 GB (source shards deleted by the engine after conversion) |
|
|
103
|
+
| venv creation | ~18 s (reuses a suitable standalone CPython when found; reports a miss otherwise) |
|
|
104
|
+
| Engine clone + pip install | ~72 s |
|
|
105
|
+
| Weights: per transformer shard (download 5.5 GB + Q8 convert + delete) | ~4–5 min |
|
|
106
|
+
| Weights: resume segment (7 shards + 28.2 GB text encoder) | 41 min |
|
|
107
|
+
| Smoke generation | 30 s wall / 10.8 s engine time (1 MP, 6 steps) |
|
|
108
|
+
| Generation peak memory (engine metrics) | 27.5 GB per image at 1 MP |
|
|
109
|
+
| Host memory in use during the run | 274 GB — includes a resident inference server; the plugin's own footprint is the 27.5 GB above |
|
|
110
|
+
|
|
111
|
+
Interrupted steps resume: a killed download continues from its `.part` (Range requests), and finished shards are never re-fetched — verified by the acceptance run's resume audit (zero re-downloads of completed shards).
|
|
112
|
+
|
|
113
|
+
### Licenses — read before downloading weights
|
|
114
|
+
|
|
115
|
+
This plugin's **code** is MIT. It does not ship, re-host or re-distribute the model — the model and its license terms have nothing to do with this plugin, which is only a downloader and runner with a license gate. Before any byte is fetched, the setup wizard **shows the key points of both agreements with direct links to their full texts** (weights: MiniMax H3 Image Community License; quantized text encoder: MiniMax H3 Community License, each from its own upstream repo) and **names the territorial exclusions (EU/UK/KR/US)**; the original agreement files land on your disk inside the model directory. You accept them yourself; the plugin never accepts on your behalf. The upstream engine is Apache-2.0. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
|
|
116
|
+
|
|
117
|
+
### Install (DSH)
|
|
118
|
+
|
|
119
|
+
Install with `plugin_manager`, from either source:
|
|
120
|
+
|
|
121
|
+
```
|
|
122
|
+
install_bundle → dsh-h3-image # latest release from npm
|
|
123
|
+
install_bundle → /path/to/dsh-h3-image # or a local checkout / unpacked dir
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Then enable the bundle in your profile and run `h3_status` — it tells you exactly what the machine still needs.
|
|
127
|
+
|
|
128
|
+
### Development
|
|
129
|
+
|
|
130
|
+
```sh
|
|
131
|
+
npm test # unit tests (pure check logic)
|
|
132
|
+
npm run release # gates + pack + tarball scan
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Two release gates run in CI and on `prepack`:
|
|
136
|
+
`scripts/check-release-safety.mjs` (no machine fingerprints, no AppleDouble,
|
|
137
|
+
no secrets — in source **and** in the built tarball) and
|
|
138
|
+
`scripts/check-version-consistency.mjs` (package.json ↔ CHANGELOG ↔ locales ↔
|
|
139
|
+
pinned revisions).
|
package/SECURITY.md
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
## Design summary
|
|
4
|
+
|
|
5
|
+
- **No secrets.** The plugin has no API keys, tokens, or credentials of any
|
|
6
|
+
kind. Local inference needs none.
|
|
7
|
+
- **Zero runtime egress.** Generation is pure local subprocess work. The only
|
|
8
|
+
network activity in the plugin's lifetime is the setup wizard fetching the
|
|
9
|
+
pinned engine (GitHub) and weights (huggingface.co, or an optional
|
|
10
|
+
hf-mirror you explicitly choose). Every fetched file is sha256-verified
|
|
11
|
+
against a value recorded in `lib/pinned.js`.
|
|
12
|
+
- **Sandboxed outputs.** Generated images are written inside the session
|
|
13
|
+
workspace by default; an `output_dir` outside the workspace is rejected.
|
|
14
|
+
- **Subprocess discipline.** The engine runs as a child process from the
|
|
15
|
+
pinned commit inside an isolated venv; prompts are passed as arguments,
|
|
16
|
+
never interpolated into a shell string. Concurrency gate = 1.
|
|
17
|
+
- **License self-acceptance.** Weight download requires the user's own
|
|
18
|
+
acceptance of the MiniMax community license (including its territorial
|
|
19
|
+
terms); the plugin records the acceptance timestamp locally and never
|
|
20
|
+
accepts on the user's behalf.
|
|
21
|
+
|
|
22
|
+
## Reporting a vulnerability
|
|
23
|
+
|
|
24
|
+
Open a **private security advisory** on the GitHub repository
|
|
25
|
+
(`aiohow/dsh-h3-image` → Security → Report a vulnerability). Please do not
|
|
26
|
+
open a public issue for unfixed vulnerabilities. We aim to respond within 7
|
|
27
|
+
days.
|
|
28
|
+
|
|
29
|
+
## Out of scope
|
|
30
|
+
|
|
31
|
+
- Vulnerabilities in the upstream engine (`minimax-h3-mlx`) or model weights
|
|
32
|
+
— report those to their respective upstreams; we will pin a fixed revision
|
|
33
|
+
promptly if a release affects users.
|
|
34
|
+
- Attacks that already require code execution on your machine.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# Third-party notices
|
|
2
|
+
|
|
3
|
+
dsh-h3-image (plugin code) is MIT-licensed. It does **not** bundle, vendor,
|
|
4
|
+
re-host or re-distribute any of the following; the setup wizard downloads
|
|
5
|
+
each item from its official upstream at a pinned revision, on the user's
|
|
6
|
+
machine, after the user accepts the respective license where required.
|
|
7
|
+
|
|
8
|
+
## MLX engine — `minimax-h3-mlx`
|
|
9
|
+
|
|
10
|
+
- Upstream: <https://github.com/mrbizarro/minimax-h3-mlx>
|
|
11
|
+
- License: Apache-2.0 (see the upstream `LICENSE`)
|
|
12
|
+
- Pinned commit: recorded in `lib/pinned.js` (`ENGINE.commit`). The wizard
|
|
13
|
+
clones and checks out exactly this commit; `main` is never followed.
|
|
14
|
+
|
|
15
|
+
## Model weights — `MiniMaxAI/MiniMax-H3-Image` (+ upstream meta from `MiniMaxAI/MiniMax-H3`)
|
|
16
|
+
|
|
17
|
+
- Upstream: <https://huggingface.co/MiniMaxAI/MiniMax-H3-Image>
|
|
18
|
+
- License: MiniMax H3 Image Community License Agreement (upstream `LICENSE`
|
|
19
|
+
file). **The license includes territorial exclusions (EU, UK, South Korea,
|
|
20
|
+
US)** — read the upstream LICENSE text yourself before downloading; the
|
|
21
|
+
wizard shows its key points with a direct link and requires your explicit
|
|
22
|
+
acceptance. The plugin never accepts on your behalf.
|
|
23
|
+
- Pinned revisions: recorded in `lib/pinned.js` (`WEIGHTS.revision`,
|
|
24
|
+
`TEXT_ENCODER.revision`); every downloaded file is verified against a
|
|
25
|
+
recorded sha256.
|
|
26
|
+
|
|
27
|
+
## Quantized text encoder — `ddalcu/MiniMax-H3-FL2VA-MLX-Serve-8bit`
|
|
28
|
+
|
|
29
|
+
- Upstream: <https://huggingface.co/ddalcu/MiniMax-H3-FL2VA-MLX-Serve-8bit>
|
|
30
|
+
- A community-quantized Model Derivative, hosted by an independent third
|
|
31
|
+
party; this plugin neither operates that repository nor is affiliated with
|
|
32
|
+
it. It is the package the pinned engine actually loads for the text path.
|
|
33
|
+
- License: **MiniMax H3 Community License Agreement** (a second agreement,
|
|
34
|
+
distinct from the H3-Image one, with the same territorial exclusions). Its
|
|
35
|
+
`LICENSE`, `NOTICE` and `MODIFICATIONS.md` are downloaded beside the
|
|
36
|
+
weights; the wizard lists both agreements before acceptance.
|
|
37
|
+
- Pinned revision + sha256: recorded in `lib/pinned.js`
|
|
38
|
+
(`TEXT_ENCODER_PACKAGE`).
|
|
39
|
+
|
|
40
|
+
## Runtime — Python ≥ 3.13 + MLX
|
|
41
|
+
|
|
42
|
+
- The wizard creates an isolated virtual environment on your machine and
|
|
43
|
+
installs the engine's declared dependencies (MLX is MIT-licensed by Apple,
|
|
44
|
+
other packages under their own licenses). The plugin ships none of them.
|
|
45
|
+
|
|
46
|
+
No trademark or brand assets of MiniMax are included in this package.
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Text-encoder fetch driver for dsh-h3-image (bundled asset, not engine code).
|
|
3
|
+
|
|
4
|
+
Why this exists instead of calling the engine's download_selected.py: that
|
|
5
|
+
script downloads three repos including video/audio VAEs and a pruned bf16
|
|
6
|
+
checkpoint this plugin's image path never loads, and it takes no pinning
|
|
7
|
+
parameters. This driver downloads EXACTLY the pinned file set, verifies the
|
|
8
|
+
main safetensors against a pinned sha256 + byte size, and honours a reuse
|
|
9
|
+
candidate list (sha256 verdict only — a size match alone never authorizes
|
|
10
|
+
reuse; it is reused by symlink, never by copy).
|
|
11
|
+
|
|
12
|
+
Progress contract (parsed by lib/jobs.js, all flushed):
|
|
13
|
+
[te n/N] fetched <name>
|
|
14
|
+
[te n/N] reused <name> (<path>)
|
|
15
|
+
[te n/N] skipped <name> (already present)
|
|
16
|
+
TE OK
|
|
17
|
+
|
|
18
|
+
Resumability: huggingface_hub keeps .incomplete part files and resumes them;
|
|
19
|
+
small files are atomic via .part + rename.
|
|
20
|
+
|
|
21
|
+
Mirror: set HF_ENDPOINT before launching (e.g. https://hf-mirror.com) —
|
|
22
|
+
hf_hub_download honours it. Nothing here hardcodes an endpoint.
|
|
23
|
+
"""
|
|
24
|
+
|
|
25
|
+
from __future__ import annotations
|
|
26
|
+
|
|
27
|
+
import argparse
|
|
28
|
+
import hashlib
|
|
29
|
+
import os
|
|
30
|
+
import sys
|
|
31
|
+
from pathlib import Path
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
def sha256_of(path: Path) -> str:
|
|
35
|
+
h = hashlib.sha256()
|
|
36
|
+
with path.open("rb") as fh:
|
|
37
|
+
for chunk in iter(lambda: fh.read(1 << 22), b""):
|
|
38
|
+
h.update(chunk)
|
|
39
|
+
return h.hexdigest()
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
def main() -> int:
|
|
43
|
+
ap = argparse.ArgumentParser(description=__doc__)
|
|
44
|
+
ap.add_argument("--dest", type=Path, required=True, help="text-encoder folder (the engine's --text-dir)")
|
|
45
|
+
ap.add_argument("--meta-dest", type=Path, required=True, help="folder for upstream meta (FL2VA/...)")
|
|
46
|
+
ap.add_argument("--repo", required=True)
|
|
47
|
+
ap.add_argument("--revision", required=True)
|
|
48
|
+
ap.add_argument("--files", required=True, help="comma-separated file list from the pinned package")
|
|
49
|
+
ap.add_argument("--main-file", default="text_encoder.safetensors")
|
|
50
|
+
ap.add_argument("--sha256", required=True, help="pinned sha256 of the main file")
|
|
51
|
+
ap.add_argument("--bytes", type=int, required=True, help="pinned byte size of the main file")
|
|
52
|
+
ap.add_argument("--meta-repo", required=True)
|
|
53
|
+
ap.add_argument("--meta-revision", required=True)
|
|
54
|
+
ap.add_argument("--meta-files", required=True, help="comma-separated upstream paths (kept under meta-dest)")
|
|
55
|
+
ap.add_argument("--candidates-file", type=Path, default=None, help="newline list of size-matching candidate paths to hash")
|
|
56
|
+
args = ap.parse_args()
|
|
57
|
+
|
|
58
|
+
# Xet high-performance mode is documented by Hugging Face for large single
|
|
59
|
+
# files; harmless when the endpoint does not serve Xet.
|
|
60
|
+
os.environ.setdefault("HF_XET_HIGH_PERFORMANCE", "1")
|
|
61
|
+
from huggingface_hub import hf_hub_download
|
|
62
|
+
|
|
63
|
+
files = [f for f in args.files.split(",") if f]
|
|
64
|
+
total = len(files)
|
|
65
|
+
args.dest.mkdir(parents=True, exist_ok=True)
|
|
66
|
+
|
|
67
|
+
def download(repo: str, revision: str, name: str, dest: Path) -> Path:
|
|
68
|
+
dest.mkdir(parents=True, exist_ok=True)
|
|
69
|
+
return Path(hf_hub_download(repo, filename=name, revision=revision, local_dir=str(dest)))
|
|
70
|
+
|
|
71
|
+
reused = False
|
|
72
|
+
main_path = args.dest / args.main_file
|
|
73
|
+
if main_path.exists() and main_path.stat().st_size == args.bytes and sha256_of(main_path) == args.sha256:
|
|
74
|
+
print(f"[te 1/{total}] skipped {args.main_file} (already present, sha256 match)", flush=True)
|
|
75
|
+
reused = True
|
|
76
|
+
else:
|
|
77
|
+
candidate_ok = False
|
|
78
|
+
if args.candidates_file is not None and args.candidates_file.exists():
|
|
79
|
+
for line in args.candidates_file.read_text().splitlines():
|
|
80
|
+
cand = Path(line.strip())
|
|
81
|
+
if not cand.is_file() or cand.stat().st_size != args.bytes:
|
|
82
|
+
continue # size mismatch: not even worth hashing
|
|
83
|
+
print(f"[te 0/{total}] hashing reuse candidate {cand.name} ({args.bytes / 1e9:.1f} GB)", flush=True)
|
|
84
|
+
if sha256_of(cand) == args.sha256:
|
|
85
|
+
# Exact digest match (design D3): reuse by symlink, never copy.
|
|
86
|
+
if main_path.exists() or main_path.is_symlink():
|
|
87
|
+
main_path.unlink()
|
|
88
|
+
main_path.symlink_to(cand)
|
|
89
|
+
print(f"[te 1/{total}] reused {args.main_file} (sha256 verified)", flush=True)
|
|
90
|
+
candidate_ok = True
|
|
91
|
+
reused = True
|
|
92
|
+
break
|
|
93
|
+
print(f"[te 0/{total}] candidate {cand.name} hash MISMATCH — will download", flush=True)
|
|
94
|
+
if not candidate_ok:
|
|
95
|
+
got = download(args.repo, args.revision, args.main_file, args.dest)
|
|
96
|
+
size = got.stat().st_size
|
|
97
|
+
if size != args.bytes:
|
|
98
|
+
print(f"SIZE MISMATCH {args.main_file}: got {size}, expected {args.bytes}", flush=True)
|
|
99
|
+
return 3
|
|
100
|
+
digest = sha256_of(got)
|
|
101
|
+
if digest != args.sha256:
|
|
102
|
+
print(f"SHA256 MISMATCH {args.main_file}: got {digest}, expected {args.sha256}", flush=True)
|
|
103
|
+
got.unlink(missing_ok=True)
|
|
104
|
+
return 3
|
|
105
|
+
print(f"[te 1/{total}] fetched {args.main_file} (sha256 verified)", flush=True)
|
|
106
|
+
|
|
107
|
+
rest = [f for f in files if f != args.main_file]
|
|
108
|
+
for i, name in enumerate(rest, start=2):
|
|
109
|
+
target = args.dest / name
|
|
110
|
+
if target.exists() and target.stat().st_size > 0:
|
|
111
|
+
print(f"[te {i}/{total}] skipped {name} (already present)", flush=True)
|
|
112
|
+
continue
|
|
113
|
+
download(args.repo, args.revision, name, args.dest)
|
|
114
|
+
print(f"[te {i}/{total}] fetched {name}", flush=True)
|
|
115
|
+
|
|
116
|
+
for name in [f for f in args.meta_files.split(",") if f]:
|
|
117
|
+
target = args.meta_dest / name
|
|
118
|
+
if target.exists() and target.stat().st_size > 0:
|
|
119
|
+
print(f"[te meta] skipped {name} (already present)", flush=True)
|
|
120
|
+
continue
|
|
121
|
+
download(args.meta_repo, args.meta_revision, name, args.meta_dest)
|
|
122
|
+
print(f"[te meta] fetched {name}", flush=True)
|
|
123
|
+
|
|
124
|
+
print("TE OK", flush=True)
|
|
125
|
+
return 0
|
|
126
|
+
|
|
127
|
+
|
|
128
|
+
if __name__ == "__main__":
|
|
129
|
+
raise SystemExit(main())
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# dsh-h3-image bundle patch: insert this package's Host half.
|
|
2
|
+
# Loaded only when the profile selects the bundle
|
|
3
|
+
# (dsh.profile.bundles: [..., "dsh-h3-image"]) or the entry is added
|
|
4
|
+
# to the profile's cordis.patch.yml as an `insert`.
|
|
5
|
+
- insert:
|
|
6
|
+
- id: h3-image
|
|
7
|
+
name: dsh-h3-image
|
|
8
|
+
config: {}
|
package/icon.svg
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="36" height="36" viewBox="0 0 36 36">
|
|
2
|
+
<rect x="6" y="5.5" width="24" height="17" rx="3" fill="none" stroke="#4c6ef5" stroke-width="2.4"/>
|
|
3
|
+
<circle cx="13.5" cy="11.5" r="2.2" fill="#4c6ef5"/>
|
|
4
|
+
<path d="M8.6 19.6l5.2-5.8 3.8 4.2 3.4-3.9 5.4 5.5z" fill="#4c6ef5"/>
|
|
5
|
+
<path d="M22.4 23.4l-3.4 5.2h3.1l-1.1 3.9 4.9-5.9h-3.1l2.1-3.2z" fill="#2f9e8f"/>
|
|
6
|
+
</svg>
|