dsh-dl-server 0.1.0 → 0.2.0

package/README.md

@@ -3,15 +3,43 @@
 English | [中文](README.zh.md)
 
 **DLS** — DSH Download Server
-A global download service for DeepSeek Harness: visible across sessions, owned per session, persistent queue, pausable and resumable.
+A persistent download queue for DeepSeek Harness.
 
-> DLS is all you need
+> Downloads should not live inside a session.
+
+---
+
+## Install
+
+### From the UI (recommended)
+
+```
+Settings → Plugins → Install a third-party plugin → Add plugin
+```
+
+Type **`dsh-dl-server@0.1.1`**, confirm, then **restart DSH once**.
+
+### From the command line
+
+```sh
+dsh plugin add dsh-dl-server@0.1.1
+```
+
+> **Pin the version.** `0.1.0` declared `@deepseek-ai/dsh-storage-domain` as a regular dependency, so pnpm installed a second copy inside the plugin; the injected service then no longer matched the host and the plugin never mounted (every service stayed `pending`). `0.1.1` moves it to `peerDependencies`.
+>
+> pnpm also caches registry metadata — without an explicit version you may still resolve `0.1.0`.
+
+> `dsh` must be on your PATH. The desktop build ships one at
+> `…\DeepSeek Harness\resources\runtime\cli\bin\dsh.cmd`; if you have never added it, use the UI path.
+
+**Requires DSH 0.2.1-rc.5 or newer.**
+**aria2 is optional** — install it for 16-way segmented downloading, or skip it and the plugin falls back to a single-stream transfer. Same features, less speed.
 
 ---
 
 ## Why
 
-Every existing DSH downloader is one tool function that blocks until the file lands. They make downloads **fast**; they don't solve three things:
+Most DSH downloaders are one tool function that blocks until the file lands. They make downloads **fast**; they don't solve three things:
 
 - the download dies when the session or the turn ends
 - there is no queue, no pause, no resume
@@ -19,27 +47,87 @@ Every existing DSH downloader is one tool function that blocks until the file la
 
 DLS is the **queue layer**, not the transport layer.
 
-## Status
+```
+Jobs live in DSH, not in a session.
+```
+
+A download started in one session is visible to every other session — progress, destination, everything. But only the session that started it can pause, resume, or cancel it.
+
+---
+
+## The seven tools
+
+| Tool | What it does |
+| --- | --- |
+| `download_enqueue` | Queue a job, **return immediately** with a job id |
+| `download_status` | Inspect the queue or a single job (`J#1042`) |
+| `download_wait` | Block until a job reaches a terminal state, with a timeout |
+| `download_pause` | Pause, **keeping the bytes already on disk** |
+| `download_resume` | Continue from the last checkpoint |
+| `download_cancel` | Cancel, optionally dropping the partial file |
+| `download_cleanup` | Drop records (the default scope leaves downloaded files alone) |
 
-`v0.1` — scaffold. Design: [SPEC.md](SPEC.md).
+## New in 0.2.0
 
-| Module | State |
+| | |
 | --- | --- |
-| Data model | done |
-| Persistent domain | done |
-| Transport (aria2 JSON-RPC + single-stream fallback) | done |
-| Queue engine (scheduling, ownership, pause reasons, checkpoints, recovery) | done |
-| Seven tools | done |
-| Real DSH install / verification | done (DSH 0.2.1-rc.5) |
-| Hermetic tests (`npm test`) | done |
-| User panel (v3) | later |
+| **Mirror expansion** | GitHub Releases / HuggingFace / raw.githubusercontent / PyPI URLs gain a fallback mirror automatically. The original always stays first. |
+| **Mirror cooldown** | Two consecutive failures on one host parks it for 10 minutes instead of retrying it on every job. |
+| **Size-aware concurrency** | Under 1 MB skips aria2 entirely; 1–8 MB runs 4 connections; only 8 MB and up gets 16. |
+| **Session fairness** | When two sessions queue at once they share the slots instead of one of them taking the whole queue. |
+| **Cumulative disk budget** | Enqueue counts what already-queued jobs still need, so a batch cannot quietly fill the disk. |
+| **Failures are no longer silent** | Checkpoint write errors are counted and surfaced in `download_status` under `storage`; an unwritable destination is refused before the transfer starts. |
+
+## Example
+
+```
+You:   download these two PDFs into D:\Papers and tell me when they're done
+```
+
+The agent calls `download_enqueue` twice and gets `J#10` / `J#11` back, then `download_wait`.
+
+Because enqueue returns immediately, it can fire off ten downloads without waiting for any of them.
+
+---
+
+## Measured
+
+Same machine, same file:
+
+| Setup | Throughput |
+| --- | --- |
+| TUNA mirror · single connection | 43.98 MB/s |
+| TUNA mirror · 16 threads (another plugin) | 54.20 MB/s |
+| TUNA mirror · **DLS** | **61.08 MB/s** (saturates a 500 Mbps link) |
+| HuggingFace · single-stream fallback | 1.00 MB/s |
+| HuggingFace · **DLS** | **6.96 MB/s** (7×) |
+
+The last row is the point: **on a per-connection-throttled host, segmentation beats bandwidth.**
+
+---
+
+## Behaviour worth knowing
+
+- **Ownership is per session.** Any session on the machine can *see* every job; only the owner can mutate one.
+- **`cleanup(scope="all")` deletes files.** The default `allWithoutFiles` only drops records.
+- **Checkpoints live in DSH storage**, not as lock files dropped into your home directory — the plugin does not depend on `TMPDIR` or the working directory.
+- **When aria2 is unavailable** the transport degrades to a single Node stream. No features lost, some speed lost.
+
+---
+
+## Docs
+
+- [SPEC.md](SPEC.md) — the 565-line design doc: comparison with four sibling plugins, what was borrowed from PCL2, and the v1 trade-offs
+- [CONTRIBUTING.md](CONTRIBUTING.md) — layout, how to run the tests, what not to touch
+- [SECURITY.md](SECURITY.md) — how to report a vulnerability, and what counts as one
 
-## Tools (planned)
+## Roadmap
 
 ```
-download_enqueue    download_status     download_wait
-download_pause      download_resume     download_cancel
-download_cleanup
+v1    queue + persistence + seven tools + aria2 transport   ← released
+v1.1  disk headroom / silent failures / session fairness / mirrors
+v2    shell interception / torrents / own segmented engine / content addressing
+v3    user panel + callbacks into the agent
 ```
 
 ## License

package/README.zh.md

@@ -5,13 +5,41 @@
 **DLS** — DSH Download Server
 大肥鱼自己的下载服务器插件
 
-> DLS is all you need
+> 下载不该活在会话里。
 
 ---
 
-## 为什么
+## 装
 
-现成的 DSH 下载插件都是「一个工具函数 + 阻塞等到下完」。它们让下载**变快**，但没解决三件事：
+### 方式一：界面（推荐）
+
+```
+设置 → 插件 → 安装第三方插件 → 添加插件
+```
+
+输入框里填 **`dsh-dl-server@0.1.1`**，确认，然后**重启一次 DSH**。
+
+### 方式二：命令行
+
+```sh
+dsh plugin add dsh-dl-server@0.1.1
+```
+
+> **带上 `@0.1.1`。** `0.1.0` 把 `@deepseek-ai/dsh-storage-domain` 声明成了普通依赖，pnpm 会给插件装第二份，服务和宿主对不上、插件挂不住（全部 `pending`）。0.1.1 改成 `peerDependencies` 已修复。
+>
+> 另外 pnpm 会缓存 registry 元数据 —— 不带版本号时可能仍然解析到 `0.1.0`。
+
+> `dsh` 需要先在 PATH 里。桌面版自带的那份在
+> `…\DeepSeek Harness\resources\runtime\cli\bin\dsh.cmd`，没加过 PATH 的话用方式一。
+
+**要求 DSH 0.2.1 或以上。**
+**aria2 可选** —— 装了走 16 线程分段，没装走单流兜底，一样能下，只是慢一些。
+
+---
+
+## 它解决什么
+
+现有的 DSH 下载插件几乎都是「一个工具函数 + 阻塞等到文件落地」。它们让下载**变快**，但没解决三件事：
 
 - 会话或回合结束，下载就断了
 - 不能排队、不能暂停恢复
@@ -19,25 +47,87 @@
 
 DLS 做的是**队列层**，不是传输层。
 
-## 状态
+```
+任务活在 DSH 里，不活在会话里
+```
+
+一个会话发起的下载，其他会话都能看见进度和落盘路径；但只有发起它的那个会话能暂停、续传、取消。
+
+---
+
+## 七个工具
+
+| 工具 | 作用 |
+| --- | --- |
+| `download_enqueue` | 入队，**立刻返回** jobId，不等下载 |
+| `download_status` | 看队列和单个任务的进度（`J#1042` 这种 id） |
+| `download_wait` | 等某个任务到终态，带超时 |
+| `download_pause` | 暂停，**保留已下字节** |
+| `download_resume` | 从断点接着下 |
+| `download_cancel` | 取消，可选删掉半截文件 |
+| `download_cleanup` | 清记录（默认不动已下载的文件） |
 
-`v0.1` 脚手架阶段。完整设计见 [SPEC.md](SPEC.md)。
+## 0.2.0 新增
 
-| 模块 | 状态 |
+| 能力 | 说明 |
 | --- | --- |
-| 数据模型 | ✅ |
-| 持久化 domain | ✅ |
-| `download_status` | 最小实现 |
-| 队列引擎 | 待做 |
-| 传输层（aria2 / Node 兜底） | 待做 |
-| 其余六个工具 | 待做 |
+| **镜像表** | GitHub Releases / HuggingFace / raw.githubusercontent / PyPI 的地址自动多带一个镜像，原地址永远排第一 |
+| **镜像冷却** | 同一 host 连续失败两次就冷却 10 分钟，不再每次撞 |
+| **并发分档** | < 1 MB 不进 aria2；1–8 MB 开 4 连接；≥ 8 MB 才 16 |
+| **会话级公平** | 两个会话同时排队时各占一半槽位，一个会话吃不下整条队列 |
+| **磁盘余量累加** | 入队时把「已在队列里还没下完的」一起算进去，批量入队写不满盘 |
+| **失败不再静默** | 断点写失败会计数并通过 `download_status` 的 `storage` 字段暴露；目标目录不可写在传输开始前就拒绝 |
+
+## 用它
+
+```
+你： 把这两个 PDF 下到 D:\Papers，下完告诉我
+```
+
+agent 会调 `download_enqueue` 两次拿到 `J#10` / `J#11`，然后 `download_wait`。
+
+**入队就返回**，所以它可以一次发十个，不用等。
+
+---
+
+## 实测
+
+同一台机器，同一个文件：
+
+| 场景 | 结果 |
+| --- | --- |
+| 清华 TUNA 源 · 单连接 | 43.98 MB/s |
+| 清华 TUNA 源 · 16 线程（其他插件） | 54.20 MB/s |
+| 清华 TUNA 源 · **DLS** | **61.08 MB/s**（跑满 500M 链路） |
+| HuggingFace · 单流兜底 | 1.00 MB/s |
+| HuggingFace · **DLS** | **6.96 MB/s**（7 倍） |
+
+最后一个说明问题：**被限速的源上，分段比带宽更值钱。**
+
+---
+
+## 已知行为（不是 bug）
+
+- **归属按会话**：同一台机器上任何会话都能**看见**全部任务，但只有 owner 能改。
+- **`cleanup(scope="all")` 会删文件**；默认的 `allWithoutFiles` 只删记录。
+- **断点落在 DSH 的存储里**，不是自己往用户目录丢锁文件 —— 所以不看 TMPDIR 或 cwd 的脸色。
+- **aria2 不可用时自动降级**到 Node 单流，功能不减，速度降。
+
+---
+
+## 文档
+
+- [SPEC.md](SPEC.md) —— 565 行设计说明，含四个同类插件的对比、从 PCL2 借了什么、v1 的取舍
+- [CONTRIBUTING.md](CONTRIBUTING.md) —— 代码布局、怎么跑测试、哪些地方别乱动
+- [SECURITY.md](SECURITY.md) —— 漏洞怎么报，什么算漏洞什么不算
 
-## 工具（规划）
+## 路线
 
 ```
-download_enqueue    download_status     download_wait
-download_pause      download_resume     download_cancel
-download_cleanup
+v1    队列 + 持久化 + 七个工具 + aria2 传输      ← 已发布
+v1.1  磁盘余量 / 静默失败 / 会话公平 / 镜像表
+v2    拦截钩子 / 种子 / 自研分段 / 内容寻址
+v3    用户面板 + 操作回传
 ```
 
 ## 许可

package/lib/domain.js

@@ -10,6 +10,10 @@ const JobRecordSchema = z.object({
   origin: z.string().optional(),
 
   urls: z.array(z.string()).min(1),
+  /** Mirrors derived from the original URL at enqueue time, for diagnostics. */
+  mirrors: z
+    .array(z.object({ from: z.string(), to: z.string(), rule: z.string() }))
+    .optional(),
   destPath: z.string(),
 
   state: z.enum(['queued', 'running', 'paused', 'completed', 'failed', 'canceled']),

package/lib/mirrors.js

@@ -0,0 +1,102 @@
+/**
+ * Mirror expansion for well-known hosts.
+ *
+ * Pure functions only: no network, no state. The point is to turn one URL into
+ * a small ordered list so the transport can fall back when the origin is slow or
+ * blocked — the original URL always stays first.
+ *
+ * What this deliberately does NOT do: prove that a mirror serves the same bytes.
+ * A third-party mirror can return an older build, an HTML error page, or worse.
+ * Only `expectSha256` can prove that, and it is checked after the download. The
+ * mitigation here is the cooldown in `queue.js`: a host that fails twice stops
+ * being tried for a while.
+ */
+
+/** @typedef {{ from: string, to: string, rule: string }} MirrorApplication */
+
+const RULES = [
+  {
+    name: 'huggingface',
+    test: (u) => u.hostname === 'huggingface.co',
+    rewrite: (u) => {
+      const out = new URL(u.href)
+      out.hostname = 'hf-mirror.com'
+      return out.href
+    },
+  },
+  {
+    name: 'github-release',
+    test: (u) => u.hostname === 'github.com' && /\/releases\/download\//.test(u.pathname),
+    rewrite: (u) => `https://ghproxy.net/${u.href}`,
+  },
+  {
+    name: 'raw-githubusercontent',
+    test: (u) => u.hostname === 'raw.githubusercontent.com',
+    rewrite: (u) => {
+      const out = new URL(u.href)
+      out.hostname = 'raw.gitmirror.com'
+      return out.href
+    },
+  },
+  {
+    name: 'pythonhosted',
+    test: (u) => u.hostname === 'files.pythonhosted.org',
+    rewrite: (u) => {
+      const out = new URL(u.href)
+      out.hostname = 'pypi.tuna.tsinghua.edu.cn'
+      return out.href
+    },
+  },
+]
+
+/** Hosts we already treat as mirrors; expanding them again would recurse. */
+const ALREADY_MIRROR = new Set(['hf-mirror.com', 'ghproxy.net', 'raw.gitmirror.com', 'pypi.tuna.tsinghua.edu.cn'])
+
+/**
+ * @param {string[]} urls
+ * @returns {{ urls: string[], applied: MirrorApplication[] }}
+ */
+export function expandMirrors(urls) {
+  const out = []
+  const seen = new Set()
+  /** @type {MirrorApplication[]} */
+  const applied = []
+
+  const push = (url) => {
+    if (seen.has(url)) return
+    seen.add(url)
+    out.push(url)
+  }
+
+  for (const url of urls) {
+    push(url)
+
+    let parsed
+    try {
+      parsed = new URL(url)
+    } catch {
+      continue
+    }
+    if (ALREADY_MIRROR.has(parsed.hostname)) continue
+
+    for (const rule of RULES) {
+      if (!rule.test(parsed)) continue
+      const mirror = rule.rewrite(parsed)
+      if (seen.has(mirror)) continue
+      push(mirror)
+      applied.push({ from: url, to: mirror, rule: rule.name })
+      break
+    }
+  }
+
+  return { urls: out, applied }
+}
+
+/** Hostname of a URL, or `undefined` when it does not parse. */
+export function hostOf(url) {
+  try {
+    return new URL(url).hostname
+  } catch {
+    return undefined
+  }
+}