dsh-dl-server 0.1.1 → 0.3.0

package/README.md

@@ -17,14 +17,18 @@ A persistent download queue for DeepSeek Harness.
 Settings → Plugins → Install a third-party plugin → Add plugin
 ```
 
-Type **`dsh-dl-server`**, confirm, then **restart DSH once**.
+Type **`dsh-dl-server@0.1.1`**, confirm, then **restart DSH once**.
 
 ### From the command line
 
 ```sh
-dsh plugin add dsh-dl-server
+dsh plugin add dsh-dl-server@0.1.1
 ```
 
+> **Pin the version.** `0.1.0` declared `@deepseek-ai/dsh-storage-domain` as a regular dependency, so pnpm installed a second copy inside the plugin; the injected service then no longer matched the host and the plugin never mounted (every service stayed `pending`). `0.1.1` moves it to `peerDependencies`.
+>
+> pnpm also caches registry metadata — without an explicit version you may still resolve `0.1.0`.
+
 > `dsh` must be on your PATH. The desktop build ships one at
 > `…\DeepSeek Harness\resources\runtime\cli\bin\dsh.cmd`; if you have never added it, use the UI path.
 
@@ -63,6 +67,26 @@ A download started in one session is visible to every other session — progress
 | `download_cancel` | Cancel, optionally dropping the partial file |
 | `download_cleanup` | Drop records (the default scope leaves downloaded files alone) |
 
+## New in 0.2.0
+
+| | |
+| --- | --- |
+| **Mirror expansion** | GitHub Releases / HuggingFace / raw.githubusercontent / PyPI URLs gain a fallback mirror automatically. The original always stays first. |
+| **Mirror cooldown** | Two consecutive failures on one host parks it for 10 minutes instead of retrying it on every job. |
+| **Size-aware concurrency** | Under 1 MB skips aria2 entirely; 1–8 MB runs 4 connections; only 8 MB and up gets 16. |
+| **Session fairness** | When two sessions queue at once they share the slots instead of one of them taking the whole queue. |
+| **Cumulative disk budget** | Enqueue counts what already-queued jobs still need, so a batch cannot quietly fill the disk. |
+| **Failures are no longer silent** | Checkpoint write errors are counted and surfaced in `download_status` under `storage`; an unwritable destination is refused before the transfer starts. |
+
+## New in 0.3.0
+
+| | |
+| --- | --- |
+| **Queue de-duplication** | One destination is never downloaded twice. A finished job is reused while its file is still on disk; a different source claiming the same path fails with `dest-in-use`. |
+| **Mirror self-verification** | Before a mirror is used, it is HEADed once: the length must equal the origin's, and when the origin advertises an ETag the mirror's must match too. |
+
+> A mirror is a third party and serves whatever it likes. Length plus ETag catches a cached older build or an HTML error page; it does **not** catch a poisoned copy. Only `sha256` proves content, which is why mirrors are a fallback and not a guarantee.
+
 ## Example
 
 ```
@@ -108,12 +132,19 @@ The last row is the point: **on a per-connection-throttled host, segmentation be
 
 ## Roadmap
 
-```
-v1    queue + persistence + seven tools + aria2 transport   ← released
-v1.1  disk headroom / silent failures / session fairness / mirrors
-v2    shell interception / torrents / own segmented engine / content addressing
-v3    user panel + callbacks into the agent
-```
+The roadmap names product stages (`v1`, `v1.1`, `v1.2`, `v2`); npm versions follow semver. They do not map one-to-one — use this table:
+
+| npm | Roadmap | Content | Status |
+| --- | --- | --- | --- |
+| `0.1.x` | v1 | queue + persistence + seven tools + aria2 transport | released |
+| `0.2.x` | v1.1 | mirrors / mirror cooldown / size-aware concurrency / session fairness / disk budget / loud failures | released |
+| `0.3.x` | v1.2 | queue de-duplication / mirror self-verification | planned |
+| `1.0.0` | v2 | shell interception / torrents / own segmented engine / content addressing | planned |
+| `1.1.x` | v3 | user panel + callbacks into the agent | planned |
+
+**Why 0.x lasts until v2:** `1.0.0` means "the tool names and config keys will not change incompatibly". Only v2 touches the transport and the tool set, so that is where 1.0 belongs.
+
+**You never have to do the conversion yourself** — install the latest and you get the newest milestone.
 
 ## License
 

package/README.zh.md

@@ -17,18 +17,22 @@
 设置 → 插件 → 安装第三方插件 → 添加插件
 ```
 
-输入框里填 **`dsh-dl-server`**，确认，然后**重启一次 DSH**。
+输入框里填 **`dsh-dl-server@0.1.1`**，确认，然后**重启一次 DSH**。
 
 ### 方式二：命令行
 
 ```sh
-dsh plugin add dsh-dl-server
+dsh plugin add dsh-dl-server@0.1.1
 ```
 
+> **带上 `@0.1.1`。** `0.1.0` 把 `@deepseek-ai/dsh-storage-domain` 声明成了普通依赖，pnpm 会给插件装第二份，服务和宿主对不上、插件挂不住（全部 `pending`）。0.1.1 改成 `peerDependencies` 已修复。
+>
+> 另外 pnpm 会缓存 registry 元数据 —— 不带版本号时可能仍然解析到 `0.1.0`。
+
 > `dsh` 需要先在 PATH 里。桌面版自带的那份在
 > `…\DeepSeek Harness\resources\runtime\cli\bin\dsh.cmd`，没加过 PATH 的话用方式一。
 
-**要求 DSH 0.2.1-rc.5 或以上。**
+**要求 DSH 0.2.1 或以上。**
 **aria2 可选** —— 装了走 16 线程分段，没装走单流兜底，一样能下，只是慢一些。
 
 ---
@@ -63,6 +67,26 @@ DLS 做的是**队列层**，不是传输层。
 | `download_cancel` | 取消，可选删掉半截文件 |
 | `download_cleanup` | 清记录（默认不动已下载的文件） |
 
+## 0.2.0 新增
+
+| 能力 | 说明 |
+| --- | --- |
+| **镜像表** | GitHub Releases / HuggingFace / raw.githubusercontent / PyPI 的地址自动多带一个镜像，原地址永远排第一 |
+| **镜像冷却** | 同一 host 连续失败两次就冷却 10 分钟，不再每次撞 |
+| **并发分档** | < 1 MB 不进 aria2；1–8 MB 开 4 连接；≥ 8 MB 才 16 |
+| **会话级公平** | 两个会话同时排队时各占一半槽位，一个会话吃不下整条队列 |
+| **磁盘余量累加** | 入队时把「已在队列里还没下完的」一起算进去，批量入队写不满盘 |
+| **失败不再静默** | 断点写失败会计数并通过 `download_status` 的 `storage` 字段暴露；目标目录不可写在传输开始前就拒绝 |
+
+## 0.3.0 新增
+
+| 能力 | 说明 |
+| --- | --- |
+| **队列去重** | 同一个落地路径不会下两份。同源且文件已在盘上时直接复用已完成的任务；不同源抢同一路径会明确报 `dest-in-use` |
+| **镜像自证同源** | 用镜像前先 HEAD 一次，长度必须与原始一致；原始给了 ETag 时镜像的 ETag 也必须相等，否则不用 |
+
+> 镜像是第三方，它给什么我们不知道。长度 + ETag 能挡住「缓存了旧版本」「返回了错误页」这类问题，**挡不住投毒** —— 那只有 `sha256` 做得到。所以镜像只是兜底，真要保证内容就对 `sha256`。
+
 ## 用它
 
 ```
@@ -108,12 +132,19 @@ agent 会调 `download_enqueue` 两次拿到 `J#10` / `J#11`，然后 `download_
 
 ## 路线
 
-```
-v1    队列 + 持久化 + 七个工具 + aria2 传输      ← 已发布
-v1.1  磁盘余量 / 静默失败 / 会话公平 / 镜像表
-v2    拦截钩子 / 种子 / 自研分段 / 内容寻址
-v3    用户面板 + 操作回传
-```
+路线图用 `v1 / v1.1 / v1.2 / v2` 讲产品阶段，npm 版本按语义化版本走。两者对不上一一对应，查这张表：
+
+| npm 版本 | 路线图 | 内容 | 状态 |
+| --- | --- | --- | --- |
+| `0.1.x` | v1 | 队列 + 持久化 + 七个工具 + aria2 传输 | 已发布 |
+| `0.2.x` | v1.1 | 镜像表 / 镜像冷却 / 并发分档 / 会话公平 / 磁盘余量 / 失败显式化 | 已发布 |
+| `0.3.x` | v1.2 | 队列去重 / 镜像自证同源 | 计划 |
+| `1.0.0` | v2 | 拦截 shell 下载 / 种子与磁力 / 自研分段引擎 / 内容寻址 | 计划 |
+| `1.1.x` | v3 | 用户面板 + 操作回传 agent | 计划 |
+
+**为什么 0.x 一直拖到 v2：** 1.0.0 在语义化版本里意味着「工具名和配置项从此不破坏性变更」。v2 才会动到传输层和工具集合，在那之前不配叫 1.0。
+
+**升级时不用换算**：装最新版就行，`dsh plugin add dsh-dl-server` 永远指向当下最新。
 
 ## 许可
 

package/lib/domain.js

@@ -10,6 +10,10 @@ const JobRecordSchema = z.object({
   origin: z.string().optional(),
 
   urls: z.array(z.string()).min(1),
+  /** Mirrors derived from the original URL at enqueue time, for diagnostics. */
+  mirrors: z
+    .array(z.object({ from: z.string(), to: z.string(), rule: z.string() }))
+    .optional(),
   destPath: z.string(),
 
   state: z.enum(['queued', 'running', 'paused', 'completed', 'failed', 'canceled']),

package/lib/mirrors.js

@@ -0,0 +1,142 @@
+/**
+ * Mirror expansion for well-known hosts.
+ *
+ * Pure functions only: no network, no state. The point is to turn one URL into
+ * a small ordered list so the transport can fall back when the origin is slow or
+ * blocked — the original URL always stays first.
+ *
+ * What this deliberately does NOT do: prove that a mirror serves the same bytes.
+ * A third-party mirror can return an older build, an HTML error page, or worse.
+ * Only `expectSha256` can prove that, and it is checked after the download. The
+ * mitigation here is the cooldown in `queue.js`: a host that fails twice stops
+ * being tried for a while.
+ */
+
+/** @typedef {{ from: string, to: string, rule: string }} MirrorApplication */
+
+const RULES = [
+  {
+    name: 'huggingface',
+    test: (u) => u.hostname === 'huggingface.co',
+    rewrite: (u) => {
+      const out = new URL(u.href)
+      out.hostname = 'hf-mirror.com'
+      return out.href
+    },
+  },
+  {
+    name: 'github-release',
+    test: (u) => u.hostname === 'github.com' && /\/releases\/download\//.test(u.pathname),
+    rewrite: (u) => `https://ghproxy.net/${u.href}`,
+  },
+  {
+    name: 'raw-githubusercontent',
+    test: (u) => u.hostname === 'raw.githubusercontent.com',
+    rewrite: (u) => {
+      const out = new URL(u.href)
+      out.hostname = 'raw.gitmirror.com'
+      return out.href
+    },
+  },
+  {
+    name: 'pythonhosted',
+    test: (u) => u.hostname === 'files.pythonhosted.org',
+    rewrite: (u) => {
+      const out = new URL(u.href)
+      out.hostname = 'pypi.tuna.tsinghua.edu.cn'
+      return out.href
+    },
+  },
+]
+
+/** Hosts we already treat as mirrors; expanding them again would recurse. */
+const ALREADY_MIRROR = new Set(['hf-mirror.com', 'ghproxy.net', 'raw.gitmirror.com', 'pypi.tuna.tsinghua.edu.cn'])
+
+/**
+ * @param {string[]} urls
+ * @returns {{ urls: string[], applied: MirrorApplication[] }}
+ */
+export function expandMirrors(urls) {
+  const out = []
+  const seen = new Set()
+  /** @type {MirrorApplication[]} */
+  const applied = []
+
+  const push = (url) => {
+    if (seen.has(url)) return
+    seen.add(url)
+    out.push(url)
+  }
+
+  for (const url of urls) {
+    push(url)
+
+    let parsed
+    try {
+      parsed = new URL(url)
+    } catch {
+      continue
+    }
+    if (ALREADY_MIRROR.has(parsed.hostname)) continue
+
+    for (const rule of RULES) {
+      if (!rule.test(parsed)) continue
+      const mirror = rule.rewrite(parsed)
+      if (seen.has(mirror)) continue
+      push(mirror)
+      applied.push({ from: url, to: mirror, rule: rule.name })
+      break
+    }
+  }
+
+  return { urls: out, applied }
+}
+
+/** Hostname of a URL, or `undefined` when it does not parse. */
+export function hostOf(url) {
+  try {
+    return new URL(url).hostname
+  } catch {
+    return undefined
+  }
+}
+
+/** `W/"abc"` → `abc`; weak validators and quoting differ between servers. */
+export function normalizeEtag(etag) {
+  if (typeof etag !== 'string') return ''
+  return etag.trim().replace(/^W\//i, '').replace(/^"|"$/g, '').toLowerCase()
+}
+
+/**
+ * Can `candidate` be accepted as the same bytes the origin would serve?
+ *
+ * This is a heuristic, not a proof: an ETag is whatever the mirror declares.
+ * It catches the common failures — a mirror caching an older build, or answering
+ * with an HTML error page — and nothing more. Only `expectSha256` proves content.
+ *
+ * @param {{ length?: number, etag?: string }} origin
+ * @param {{ length?: number, etag?: string }} candidate
+ * @returns {{ ok: boolean, reason?: string }}
+ */
+export function sameResource(origin, candidate) {
+  const originLength = Number(origin?.length ?? -1)
+  const candidateLength = Number(candidate?.length ?? -1)
+
+  // No baseline (the origin probe failed): we cannot prove anything, and the
+  // mirror exists precisely for the case where the origin is unreachable.
+  if (!(originLength > 0)) return { ok: true, reason: 'origin-unknown' }
+
+  if (!(candidateLength > 0)) return { ok: false, reason: 'mirror did not report a length' }
+  if (candidateLength !== originLength) {
+    return { ok: false, reason: `length ${candidateLength} != ${originLength}` }
+  }
+
+  const originEtag = normalizeEtag(origin?.etag)
+  if (originEtag) {
+    const candidateEtag = normalizeEtag(candidate?.etag)
+    if (!candidateEtag) return { ok: false, reason: 'mirror reports no ETag' }
+    if (candidateEtag !== originEtag) return { ok: false, reason: 'ETag differs' }
+  }
+
+  return { ok: true }
+}