dsh-h3-image 0.2.7 → 0.2.9

package/README.md

@@ -45,13 +45,17 @@
 
 ### 安装（DSH）
 
-在 DSH 里用 `plugin_manager` 安装，两种来源任选：
+图形界面：设置 → 插件 → 安装，输入 `dsh-h3-image`，装完启用（启用后重启一次 DSH 让工具生效）。
+
+也可以让 agent 用 `plugin_manager` 安装，两种来源任选：
 
 ```
 install_bundle → dsh-h3-image            # 从 npm 装最新发布版
 install_bundle → /path/to/dsh-h3-image   # 或本地仓库/解包目录
 ```
 
+装到旧版多半是 npm 镜像同步滞后（如 npmmirror 比官方源慢几分钟到几小时），切回官方源 `registry.npmjs.org` 重装即可。
+
 然后在 profile 中启用该 bundle，跑一次 `h3_status`——机器还缺什么它会逐项告诉你。
 
 ### 开发
@@ -108,13 +112,17 @@ This plugin's **code** is MIT. It does not ship, re-host or re-distribute the mo
 
 ### Install (DSH)
 
-Install with `plugin_manager`, from either source:
+From the GUI: Settings → Plugins → Install, type `dsh-h3-image`, then enable it (restart DSH once so the tools load).
+
+Or let the agent install it with `plugin_manager`, from either source:
 
 ```
 install_bundle → dsh-h3-image            # latest release from npm
 install_bundle → /path/to/dsh-h3-image   # or a local checkout / unpacked dir
 ```
 
+If you end up on an older version, the npm mirror you picked is probably behind the official registry (npmmirror lags by minutes to hours); reinstall from `registry.npmjs.org`.
+
 Then enable the bundle in your profile and run `h3_status` — it tells you exactly what the machine still needs.
 
 ### Development

package/assets/stage_weights.py

@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+"""Weight-staging driver for dsh-h3-image (bundled asset, not engine code).
+
+Why this exists: the pinned engine's fetch_h3_image.py hardcodes
+https://huggingface.co and has no mirror switch, and patching the pinned
+engine is out of scope (design D1). So instead of touching the engine, this
+driver places the EXACT pinned file set (CORE_FILES + the transformer shards,
+read live from the engine script — never re-listed or re-hashed here) into
+the engine's own layout ahead of time. The engine's fetch contract then does
+the rest: a file that already exists with the right size and sha256 is
+returned without any network access, so the following engine run is instant
+for the main weights while still converting shard by shard as usual.
+
+Endpoint honesty: every endpoint is tried DIRECT first (proxies explicitly
+disabled — a proxy download would silently spend the user's proxy data). Only
+after every endpoint failed direct does one final round run through the system
+proxy, announced by its own line before it starts. Redirects (301/302/303/
+307/308) are followed by hand because urllib does not follow 308, which is
+exactly what a mirror answers for every file; the Range header is kept across
+hops so a resumed download stays resumed through a 308.
+
+File-list truth: imported from the engine script (--engine-script), so a pin
+bump needs no change here.
+
+Progress contract (parsed by lib/jobs.js, all flushed):
+    [stage n/N] fetching <name> via <host>
+    [stage n/N] placed <name>
+    [stage n/N] skipped <name> (already present)
+    [stage n/N] switched endpoint: <host>
+    stage: retrying via system proxy (may count against proxy data)
+    STAGE OK            (terminal, stdout)
+    STAGE FAIL <rel>    (terminal, stderr, exit 1)
+
+Resumability: a `.part` survives a failed attempt and resumes with a Range
+request; a server that ignores Range restarts the file cleanly (200 →
+truncate). A part that fails sha256 is deleted, never renamed into place.
+
+Usage:
+    python stage_weights.py --dir MODEL_DIR \
+        --engine-script .../scripts/fetch_h3_image.py \
+        --endpoints hf,https://mirror.example
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+from pathlib import Path
+
+TIMEOUT = 60  # seconds without data on any single read — same as the engine fetch
+REDIRECT_CODES = (301, 302, 303, 307, 308)
+HF = "https://huggingface.co"
+
+
+class _NoRedirect(urllib.request.HTTPRedirectHandler):
+    """Neutralise urllib's built-in redirect handling (it ignores 308); the
+    loop in open_following() follows every redirect code itself."""
+
+    def redirect_request(self, req, fp, code, msg, headers, newurl):  # noqa: D102
+        return None
+
+
+def sha256_of(path: Path) -> str:
+    h = hashlib.sha256()
+    with path.open("rb") as fh:
+        for chunk in iter(lambda: fh.read(1 << 22), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def build_opener(direct: bool):
+    """direct=True → ProxyHandler({}) (never the user's proxy); direct=False
+    → the default proxy handler, which honours the system proxy settings."""
+    if direct:
+        return urllib.request.build_opener(_NoRedirect(), urllib.request.ProxyHandler({}))
+    return urllib.request.build_opener(_NoRedirect())
+
+
+def open_following(opener, url: str, headers: dict, max_redirects: int = 10):
+    """GET url, following 301/302/303/307/308 by hand. Returns the live
+    response; the caller streams from it and it closes with the caller's
+    `with` block."""
+    for _ in range(max_redirects + 1):
+        req = urllib.request.Request(url, headers=headers)
+        try:
+            return opener.open(req, timeout=TIMEOUT)
+        except urllib.error.HTTPError as err:
+            location = err.headers.get("Location") if err.headers is not None else None
+            err.close()
+            if err.code in REDIRECT_CODES and location:
+                url = urllib.parse.urljoin(url, location)
+                continue
+            raise
+    raise RuntimeError(f"more than {max_redirects} redirects for {url}")
+
+
+def attempt(opener, url: str, dest: Path, part: Path, size: int, sha, name: str,
+            index: int, total: int, first: bool) -> None:
+    """One endpoint, one file: stream into .part (resuming when the server
+    honours Range), then verify and rename. Any raise = this endpoint failed
+    for this file; the partial .part stays on disk for the next attempt."""
+    host = urllib.parse.urlsplit(url).hostname or url
+    if not first:
+        print(f"[stage {index}/{total}] switched endpoint: {host}", flush=True)
+    print(f"[stage {index}/{total}] fetching {name} via {host}", flush=True)
+    have = part.stat().st_size if part.exists() else 0
+    if have > size:
+        part.unlink()
+        have = 0
+    headers = {"Range": f"bytes={have}-"} if have else {}
+    with open_following(opener, url, headers) as resp:
+        resume = bool(have) and getattr(resp, "status", 200) == 206
+        with part.open("ab" if resume else "wb") as out:
+            while True:
+                block = resp.read(1 << 22)
+                if not block:
+                    break
+                out.write(block)
+    got = part.stat().st_size if part.exists() else 0
+    if got != size:
+        if got > size:
+            part.unlink(missing_ok=True)  # a corrupt oversize part can never resume
+        raise RuntimeError(f"expected {size} bytes, got {got}")
+    if sha is not None and sha256_of(part) != sha:
+        part.unlink(missing_ok=True)  # wrong content: never stage it, never resume it
+        raise RuntimeError("sha256 mismatch")
+    part.rename(dest)
+    print(f"[stage {index}/{total}] placed {name}", flush=True)
+
+
+def stage_file(*, name: str, rel: str, dest: Path, size: int, sha, endpoints,
+               middle: str, index: int, total: int) -> bool:
+    """Place one pinned file. Returns True on placed/skipped, False after every
+    endpoint failed in both the direct and the system-proxy round."""
+    dest.parent.mkdir(parents=True, exist_ok=True)
+    if dest.exists() and dest.stat().st_size == size and (sha is None or sha256_of(dest) == sha):
+        print(f"[stage {index}/{total}] skipped {name} (already present)", flush=True)
+        return True
+    part = dest.with_name(dest.name + ".part")
+    if part.exists() and part.stat().st_size == size:
+        # A part that is already complete goes straight to verification
+        # (same shortcut the engine fetch takes), skipping the network.
+        if sha is None or sha256_of(part) == sha:
+            part.rename(dest)
+            print(f"[stage {index}/{total}] placed {name}", flush=True)
+            return True
+        part.unlink()
+
+    url_of = lambda endpoint: f"{endpoint.rstrip('/')}/{middle}/{rel}"  # noqa: E731
+    last_error = None
+    attempts = 0
+    direct = build_opener(direct=True)
+    for endpoint in endpoints:
+        try:
+            attempt(direct, url_of(endpoint), dest, part, size, sha, name, index, total, first=attempts == 0)
+            return True
+        except Exception as error:  # noqa: BLE001 — any failure means: next endpoint
+            last_error = error
+            attempts += 1
+    print("stage: retrying via system proxy (may count against proxy data)", flush=True)
+    proxied = build_opener(direct=False)
+    for endpoint in endpoints:
+        try:
+            attempt(proxied, url_of(endpoint), dest, part, size, sha, name, index, total, first=attempts == 0)
+            return True
+        except Exception as error:  # noqa: BLE001
+            last_error = error
+            attempts += 1
+    print(f"stage: {name}: {last_error}", file=sys.stderr, flush=True)
+    return False
+
+
+def main(argv=None) -> int:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--dir", type=Path, required=True, help="model root (the engine fetch's --dir)")
+    ap.add_argument("--engine-script", type=Path, required=True, help="absolute path to the engine's fetch_h3_image.py")
+    ap.add_argument("--endpoints", required=True, help="comma-separated priority list; 'hf' or https://host[/prefix] per entry")
+    args = ap.parse_args(argv)
+
+    # The engine script's own directory goes on sys.path so `import
+    # fetch_h3_image` (and its sibling import of fetch_ph3_image) resolves.
+    # Importing it only reads constants — no network, no side effects.
+    sys.path.insert(0, str(args.engine_script.resolve().parent))
+    import fetch_h3_image as engine  # noqa: E402
+
+    middle = f"{engine.REPO}/resolve/{engine.REVISION}"
+
+    endpoints = []
+    for raw in args.endpoints.split(","):
+        raw = raw.strip().rstrip("/")
+        if raw:
+            endpoints.append(HF if raw == "hf" else raw)
+    if not endpoints:
+        print("stage: --endpoints resolved to nothing", file=sys.stderr)
+        return 2
+
+    # Layout truth = the engine's own fetch_h3_image.main: CORE_FILES go to
+    # root/<rel> (root/source/<rel> for transformer/), shards to
+    # root/source/transformer/<name> via the engine's SHARD_FILE template.
+    shard_file = getattr(engine, "SHARD_FILE", "transformer/{name}")
+    items = []
+    for rel, (size, sha) in engine.CORE_FILES.items():
+        dest = args.dir / rel if not rel.startswith("transformer/") else args.dir / "source" / rel
+        items.append((rel, rel, dest, size, sha))
+    for shard_name in sorted(engine.SHARDS):
+        size, sha = engine.SHARDS[shard_name]
+        rel = shard_file.format(name=shard_name)
+        items.append((shard_name, rel, args.dir / "source" / rel, size, sha))
+
+    total = len(items)
+    for index, (name, rel, dest, size, sha) in enumerate(items, 1):
+        ok = stage_file(name=name, rel=rel, dest=dest, size=size, sha=sha,
+                        endpoints=endpoints, middle=middle, index=index, total=total)
+        if not ok:
+            print(f"STAGE FAIL {rel}", file=sys.stderr, flush=True)
+            return 1
+    print("STAGE OK", flush=True)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/lib/checks.js

@@ -313,6 +313,18 @@ export function checkDisk({ freeGB, modelDirName }) {
 // Full status run (used by the h3_status tool).
 // ---------------------------------------------------------------------------
 
+/**
+ * Mirror normalization (single truth for the config surface):
+ *   "hf" | undefined | anything invalid → "hf"
+ *   "hf-mirror" (legacy literal)        → "https://hf-mirror.com"
+ *   any other https:// URL              → kept as-is (custom endpoint)
+ */
+export function normalizeMirror(value) {
+  if (value === "hf-mirror") return "https://hf-mirror.com";
+  if (typeof value === "string" && value.startsWith("https://") && !/\s/.test(value)) return value;
+  return "hf";
+}
+
 export function resolveConfig(config = {}) {
   const home = homedir();
   // With an explicit model_dir, sibling defaults hang off its parent (the
@@ -337,7 +349,11 @@ export function resolveConfig(config = {}) {
     venv_dir: config.venv_dir ?? join(root, "venv"),
     text_dir,
     text_config,
-    mirror: config.mirror === "hf-mirror" ? "hf-mirror" : "hf",
+    // Mirror: "hf" (default) or a custom endpoint. The legacy literal
+    // "hf-mirror" normalizes to its URL; any https:// URL is a valid custom
+    // endpoint (staged main weights + TE HF_ENDPOINT both use it). Anything
+    // else falls back to "hf" — a typo must never disable the default path.
+    mirror: normalizeMirror(config.mirror),
     // Reuse-probe scan roots (absolute or $HOME-relative). Default is the
     // one known asset location; an explicit [] forces a true bare-metal
     // simulation (no candidates → full download + verification path).

package/lib/index.js

@@ -106,7 +106,8 @@ function formatSetup(value) {
     lines.push(`h3_setup poll [${value.kind}/${value.state}] job ${value.job_id}`);
     const p = value.progress ?? {};
     if (value.kind === "weights") {
-      lines.push(`shards ${p.shard_done ?? 0}/${p.shard_total ?? 12} converted${p.current ? ` (fetching #${p.current})` : ""}; text encoder ${p.te_done ?? 0}/${p.te_total ?? 9}`);
+      const staged = p.stage_total ? `staged ${p.stage_done}/${p.stage_total}; ` : "";
+      lines.push(`${staged}shards ${p.shard_done ?? 0}/${p.shard_total ?? 12} converted${p.current ? ` (fetching #${p.current})` : ""}; text encoder ${p.te_done ?? 0}/${p.te_total ?? 9}`);
     } else if (p.stage) {
       lines.push(`stage: ${p.stage}`);
     }

package/lib/jobs.js

@@ -144,6 +144,7 @@ export async function tailLog(jDir, id, maxBytes = 2000, readFileP = readFile) {
  *   `PACK OK -> ...`                                               (engine fetch, terminal)
  *   `[te n/9] fetched <name>` / `[te n/9] reused <name>`           (our TE driver)
  *   `TE OK`                                                        (TE driver, terminal)
+ *   `[stage n/N] placed <name>`                                    (our staging driver)
  *   `== <label> ==`                                                (supervisor stage marker)
  */
 export function parseProgress(logTail) {
@@ -161,6 +162,13 @@ export function parseProgress(logTail) {
     out.te_total = Math.max(out.te_total, +m[2]);
     out.te_done = Math.max(out.te_done, +m[1]);
   }
+  // Staging only reports a total when a mirror run actually staged files;
+  // absent lines leave stage_total undefined so the renderer can skip it.
+  const stageLines = [...logTail.matchAll(/^\[stage (\d+)\/(\d+)\] (placed|skipped)/gm)];
+  for (const m of stageLines) {
+    out.stage_total = Math.max(out.stage_total ?? 0, +m[2]);
+    out.stage_done = Math.max(out.stage_done ?? 0, +m[1]);
+  }
   out.pack_ok = /^PACK OK/m.test(logTail);
   out.te_ok = /^TE OK/m.test(logTail);
   return out;

package/lib/setup.js

@@ -23,11 +23,14 @@
  * symlink, never by copy — only on an exact digest match. No match → download
  * from the pinned revision. A size match alone never authorizes reuse.
  *
- * Mirror honesty (documented in README): `mirror: "hf-mirror"` redirects the
- * text-encoder download (hf_hub_download honors HF_ENDPOINT). The main
- * weights come from the engine's own fetch script whose URL is hardcoded —
- * patching the pinned engine is out of scope (design D1), so the mirror
- * option never claims to cover the main weights.
+ * Mirror honesty (documented in README): a non-default `mirror` covers BOTH
+ * download faces without touching the pinned engine (design D1). The main
+ * weights are staged ahead of the engine fetch by assets/stage_weights.py,
+ * which places the exact pinned file set into the engine's own layout from
+ * the chosen endpoint; the engine fetch then skips every placed file (its
+ * contract: present + right size + right sha256 → no network). The
+ * text-encoder download honours HF_ENDPOINT (hf_hub_download). The default
+ * `hf` carries no stage step and no env override at all.
  */
 import { execFile } from "node:child_process";
 import { mkdir, readFile, readdir, writeFile } from "node:fs/promises";
@@ -192,9 +195,39 @@ function teDriverPath() {
   return join(import.meta.dirname, "..", "assets", "te_setup.py");
 }
 
-/** Engine fetch chain for the weights job (resumable by the engine itself). */
+function stageDriverPath() {
+  return join(import.meta.dirname, "..", "assets", "stage_weights.py");
+}
+
+/** Legacy literal kept working for old profile configs; anything else that
+ *  is not "hf" is treated as a custom endpoint URL. */
+const HF_MIRROR_URL = "https://hf-mirror.com";
+
+/**
+ * Engine fetch chain for the weights job (resumable by the engine itself).
+ * `mirror` "hf" (default) = plain engine fetch + TE, zero env. Any other
+ * value (resolveConfig normalizes "hf-mirror" to its URL) = a custom
+ * endpoint: a `stage` command runs FIRST and places every pinned main-weight
+ * file from `<mirror>,hf` (mirror first — a user who configured one means
+ * it; hf as fallback), so the engine fetch that follows skips them all;
+ * the TE command points HF_ENDPOINT at the same endpoint.
+ */
 export function buildWeightsCommands({ resolved, venvBin, mirror }) {
-  return [
+  const endpoint = mirror === "hf-mirror" ? HF_MIRROR_URL : mirror;
+  const commands = [];
+  if (mirror !== "hf") {
+    commands.push({
+      label: "stage",
+      argv: [
+        venvBin,
+        stageDriverPath(),
+        "--dir", resolved.model_dir,
+        "--engine-script", join(resolved.engine_dir, "scripts", "fetch_h3_image.py"),
+        "--endpoints", `${endpoint},hf`,
+      ],
+    });
+  }
+  commands.push(
     {
       label: "weights-fetch",
       argv: [venvBin, join(resolved.engine_dir, "scripts", "fetch_h3_image.py"), "--dir", resolved.model_dir, "--accepted"],
@@ -217,11 +250,13 @@ export function buildWeightsCommands({ resolved, venvBin, mirror }) {
         "--meta-files", UPSTREAM_META.files.join(","),
         "--candidates-file", join(resolved.model_dir, "h3-jobs", "reuse-candidates.txt"),
       ],
-      // hf-mirror only ever touches THIS command (hf_hub_download honours
-      // HF_ENDPOINT); the default carries no env override at all.
-      ...(mirror === "hf-mirror" ? { env: { HF_ENDPOINT: "https://hf-mirror.com" } } : {}),
+      // A custom endpoint only ever touches THIS command's env
+      // (hf_hub_download honours HF_ENDPOINT); the default carries no env
+      // override at all.
+      ...(mirror === "hf" ? {} : { env: { HF_ENDPOINT: endpoint } }),
     },
-  ];
+  );
+  return commands;
 }
 
 /** Smoke chain: one 1 MP image into the job area, metrics beside it. */

package/locale/en.json

@@ -1,6 +1,6 @@
 {
   "meta": {
-    "title": "H3 Image (local MLX)",
-    "description": "Unofficial third-party plugin, not affiliated with MiniMax. On Apple Silicon, fully local text-to-image with the open-source MiniMax H3-Image (MLX engine). A one-time setup wizard downloads a fixed version of the engine and weights to your machine (the MiniMax community license and its region terms are shown before any download; you accept it yourself), then generation runs offline — no keys, no accounts, no network access. Long setup steps are background jobs that survive restarts and resume where they stopped. Requires Apple Silicon with 48 GB+ unified memory (32 GB is not guaranteed) and ~60 GB free disk during setup; Intel Macs and other platforms are declined up front. Current version (0.2.7): read-only environment check, setup wizard, and local generation (the community license must be explicitly accepted by you before the first image)."
+    "title": "H3 Image (dsh-h3-image)",
+    "description": "Unofficial third-party plugin, not affiliated with MiniMax. On Apple Silicon, fully local text-to-image with the open-source MiniMax H3-Image (MLX engine). A one-time setup wizard downloads a fixed version of the engine and weights to your machine (the MiniMax community license and its region terms are shown before any download; you accept it yourself), then generation runs offline — no keys, no accounts, no network access. Long setup steps are background jobs that survive restarts and resume where they stopped. Requires Apple Silicon with 48 GB+ unified memory (32 GB is not guaranteed) and ~60 GB free disk during setup; Intel Macs and other platforms are declined up front. Current version (0.2.9): read-only environment check, setup wizard, and local generation (the community license must be explicitly accepted by you before the first image)."
   }
 }

package/locale/zh.json

@@ -1,6 +1,6 @@
 {
   "meta": {
-    "title": "H3 本地生图（MLX）",
-    "description": "非官方第三方插件，与 MiniMax 无关联。在 Apple Silicon 上用开源 MiniMax H3-Image（MLX 引擎）做完全本地文生图。安装向导一次性把固定版本的引擎与权重下载到本机（下载前展示 MiniMax 社区许可及地域条款，由你本人确认接受）；之后生图全程离线——不用密钥、不用账号、不联网。安装中的长步骤是后台任务，DSH 重启不丢、断点续传。需 Apple Silicon、48GB 及以上统一内存（32GB 不保证）、安装期约 60GB 可用磁盘；Intel Mac 与其他平台在检查阶段即明确拒绝。当前版本（0.2.7）：只读环境体检 + 安装向导 + 本地产图（社区许可需你本人确认接受后才会出图）。"
+    "title": "H3 本地生图 (dsh-h3-image)",
+    "description": "非官方第三方插件，与 MiniMax 无关联。在 Apple Silicon 上用开源 MiniMax H3-Image（MLX 引擎）做完全本地文生图。安装向导一次性把固定版本的引擎与权重下载到本机（下载前展示 MiniMax 社区许可及地域条款，由你本人确认接受）；之后生图全程离线——不用密钥、不用账号、不联网。安装中的长步骤是后台任务，DSH 重启不丢、断点续传。需 Apple Silicon、48GB 及以上统一内存（32GB 不保证）、安装期约 60GB 可用磁盘；Intel Mac 与其他平台在检查阶段即明确拒绝。当前版本（0.2.9）：只读环境体检 + 安装向导 + 本地产图（社区许可需你本人确认接受后才会出图）。"
   }
 }

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "dsh-h3-image",
-  "version": "0.2.7",
-  "description": "在 Apple Silicon 上用开源 MiniMax H3-Image（MLX 引擎）做完全本地文生图的 DeepSeek Harness 插件：一键安装向导 + 零出网本地生图。Fully local MiniMax H3-Image text-to-image on Apple Silicon MLX — a DeepSeek Harness plugin with a one-time setup wizard and zero-egress generation.",
+  "version": "0.2.9",
+  "description": "在 Apple Silicon 上用开源 MiniMax H3-Image（MLX 引擎）做完全本地文生图的 DeepSeek Harness 插件：一键安装向导 + 零出网本地生图。Fully local MiniMax H3-Image text-to-image on Apple Silicon MLX — a DeepSeek Harness plugin with a one-time setup wizard and no network access at generation.",
   "type": "module",
   "main": "./lib/index.js",
   "scripts": {