dsh-dl-server 0.3.1 → 0.4.1

package/README.md

@@ -17,23 +17,32 @@ A persistent download queue for DeepSeek Harness.
 Settings → Plugins → Install a third-party plugin → Add plugin
 ```
 
-Type **`dsh-dl-server@0.1.1`**, confirm, then **restart DSH once**.
+Type **`dsh-dl-server@0.4.0`**, confirm, then **restart DSH once**.
 
 ### From the command line
 
 ```sh
-dsh plugin add dsh-dl-server@0.1.1
+dsh plugin add dsh-dl-server@0.4.0
 ```
 
 > **Pin the version.** `0.1.0` declared `@deepseek-ai/dsh-storage-domain` as a regular dependency, so pnpm installed a second copy inside the plugin; the injected service then no longer matched the host and the plugin never mounted (every service stayed `pending`). `0.1.1` moves it to `peerDependencies`.
 >
-> pnpm also caches registry metadata — without an explicit version you may still resolve `0.1.0`.
+> pnpm also caches registry metadata — without an explicit version you may still resolve an older release.
 
 > `dsh` must be on your PATH. The desktop build ships one at
 > `…\DeepSeek Harness\resources\runtime\cli\bin\dsh.cmd`; if you have never added it, use the UI path.
 
 **Requires DSH 0.2.1-rc.5 or newer.**
-**aria2 is optional** — install it for 16-way segmented downloading, or skip it and the plugin falls back to a single-stream transfer. Same features, less speed.
+**aria2 is optional but strongly recommended** — install it for 16-way segmented downloading. Without it the plugin falls back to a single-stream transfer: same features, roughly a tenth of the speed. The plugin warns at startup and reports the reason through `download_status`, so a slow first install is not a mystery.
+
+| Platform | Install aria2 |
+| --- | --- |
+| Windows | `winget install aria2.aria2` |
+| Ubuntu / Debian | `sudo apt install aria2` |
+| Fedora | `sudo dnf install aria2` |
+| Arch | `sudo pacman -S aria2` |
+| macOS | `brew install aria2` |
+
 
 ---
 
@@ -87,6 +96,22 @@ A download started in one session is visible to every other session — progress
 
 > A mirror is a third party and serves whatever it likes. Length plus ETag catches a cached older build or an HTML error page; it does **not** catch a poisoned copy. Only `sha256` proves content, which is why mirrors are a fallback and not a guarantee.
 
+## New in 0.4.0
+
+| | |
+| --- | --- |
+| **Custom request headers** | `download_enqueue` takes an optional `headers` object — `Referer`, `Cookie`, `User-Agent`, anything the host wants. Sent on the origin probe, on every mirror probe, on the aria2 download and on the single-stream fallback. |
+| **Validated at enqueue** | Line breaks, control characters, illegal names, more than 32 headers, and transport-owned names (`Range`, `Host`, `Content-Length`, …) are rejected with `header-invalid` **before** any bytes move. |
+| **Redacted in output** | Values may be session cookies. Job output lists names only (`"requestHeaders": ["Referer"]`); the values stay in the transfer layer and the local store. |
+
+Why it matters: some URLs are correct and still refuse to download. Bilibili's media CDN answers **403 to any request without `Referer: https://www.bilibili.com`** — no number of mirrors helps. This is the same reason `curl -H` exists.
+
+```
+download_enqueue  urls=[...]  dest=...  headers={"Referer": "https://www.bilibili.com"}
+```
+
+Resume still works: headers are persisted with the job, so a download interrupted by a DSH restart sends them again.
+
 ## Example
 
 ```
@@ -120,7 +145,7 @@ The last row is the point: **on a per-connection-throttled host, segmentation be
 - **Ownership is per session.** Any session on the machine can *see* every job; only the owner can mutate one.
 - **`cleanup(scope="all")` deletes files.** The default `allWithoutFiles` only drops records.
 - **Checkpoints live in DSH storage**, not as lock files dropped into your home directory — the plugin does not depend on `TMPDIR` or the working directory.
-- **When aria2 is unavailable** the transport degrades to a single Node stream. No features lost, some speed lost.
+- **When aria2 is unavailable** the transport degrades to a single Node stream. No features lost, some speed lost — and the plugin says so instead of staying quiet, because a silent 10× slowdown looks like a bug.
 
 ---
 
@@ -132,13 +157,15 @@ The last row is the point: **on a per-connection-throttled host, segmentation be
 
 ## Roadmap
 
-The roadmap names product stages (`v1`, `v1.1`, `v1.2`, `v2`); npm versions follow semver. They do not map one-to-one — use this table:
+The roadmap names product stages (`v1`, `v1.1`, `v1.2`, `v1.3`, `v2`); npm versions follow semver. They do not map one-to-one — use this table:
 
 | npm | Roadmap | Content | Status |
 | --- | --- | --- | --- |
 | `0.1.x` | v1 | queue + persistence + seven tools + aria2 transport | released |
 | `0.2.x` | v1.1 | mirrors / mirror cooldown / size-aware concurrency / session fairness / disk budget / loud failures | released |
-| `0.3.x` | v1.2 | queue de-duplication / mirror self-verification | planned |
+| `0.3.x` | v1.2 | queue de-duplication / mirror self-verification | released |
+| `0.4.x` | v1.3 | custom request headers (`Referer` / `Cookie` / `User-Agent`) on every attempt, validated at enqueue, redacted in output | released |
+| `0.4.1` | patch | missing aria2 is now a loud, actionable warning instead of a silent speed cliff; `download_status` reports the channel | released |
 | `1.0.0` | v2 | shell interception / torrents / own segmented engine / content addressing | planned |
 | `1.1.x` | v3 | user panel + callbacks into the agent | planned |
 

package/README.zh.md

@@ -17,23 +17,32 @@
 设置 → 插件 → 安装第三方插件 → 添加插件
 ```
 
-输入框里填 **`dsh-dl-server@0.1.1`**，确认，然后**重启一次 DSH**。
+输入框里填 **`dsh-dl-server@0.4.0`**，确认，然后**重启一次 DSH**。
 
 ### 方式二：命令行
 
 ```sh
-dsh plugin add dsh-dl-server@0.1.1
+dsh plugin add dsh-dl-server@0.4.0
 ```
 
-> **带上 `@0.1.1`。** `0.1.0` 把 `@deepseek-ai/dsh-storage-domain` 声明成了普通依赖，pnpm 会给插件装第二份，服务和宿主对不上、插件挂不住（全部 `pending`）。0.1.1 改成 `peerDependencies` 已修复。
+> **带上版本号。** `0.1.0` 把 `@deepseek-ai/dsh-storage-domain` 声明成了普通依赖，pnpm 会给插件装第二份，服务和宿主对不上、插件挂不住（全部 `pending`）。0.1.1 改成 `peerDependencies` 已修复。
 >
-> 另外 pnpm 会缓存 registry 元数据 —— 不带版本号时可能仍然解析到 `0.1.0`。
+> 另外 pnpm 会缓存 registry 元数据 —— 不带版本号时可能仍然解析到旧版本。
 
 > `dsh` 需要先在 PATH 里。桌面版自带的那份在
 > `…\DeepSeek Harness\resources\runtime\cli\bin\dsh.cmd`，没加过 PATH 的话用方式一。
 
 **要求 DSH 0.2.1 或以上。**
-**aria2 可选** —— 装了走 16 线程分段，没装走单流兜底，一样能下，只是慢一些。
+**aria2 可选，但强烈建议装** —— 装了走 16 线程分段；没装走单流兜底，功能一个不少，速度大约只有十分之一。没装时插件会在启动日志里明确警告，`download_status` 也会报出原因，不会让你以为"DLS 就这速度"。
+
+| 平台 | 装 aria2 |
+| --- | --- |
+| Windows | `winget install aria2.aria2` |
+| Ubuntu / Debian | `sudo apt install aria2` |
+| Fedora | `sudo dnf install aria2` |
+| Arch | `sudo pacman -S aria2` |
+| macOS | `brew install aria2` |
+
 
 ---
 
@@ -87,6 +96,22 @@ DLS 做的是**队列层**，不是传输层。
 
 > 镜像是第三方，它给什么我们不知道。长度 + ETag 能挡住「缓存了旧版本」「返回了错误页」这类问题，**挡不住投毒** —— 那只有 `sha256` 做得到。所以镜像只是兜底，真要保证内容就对 `sha256`。
 
+## 0.4.0 新增
+
+| | |
+| --- | --- |
+| **自定义请求头** | `download_enqueue` 多了个可选 `headers` —— `Referer` / `Cookie` / `User-Agent`，站点要什么给什么。原始探测、每个镜像探测、aria2 下载、单流兜底，四处都会带 |
+| **入队时就校验** | 换行、控制符、非法名字、超过 32 条、以及传输层自己管的（`Range` / `Host` / `Content-Length` …）一律报 `header-invalid`，**一个字节都不会动** |
+| **输出脱敏** | 值可能是登录 cookie。任务输出只列名字（`"requestHeaders": ["Referer"]`），值只留在传输层和本地存储里 |
+
+**为什么需要它**：有些 URL 完全正确就是下不了。B站媒体 CDN 对**不带 `Referer: https://www.bilibili.com` 的请求一律 403**，镜像再多也没用 —— 这就是 `curl -H` 存在的理由。
+
+```
+download_enqueue  urls=[...]  dest=...  headers={"Referer": "https://www.bilibili.com"}
+```
+
+**断点续传照样能用**：header 跟 job 一起持久化，DSH 重启后接着下仍然会带上。
+
 ## 用它
 
 ```
@@ -132,13 +157,15 @@ agent 会调 `download_enqueue` 两次拿到 `J#10` / `J#11`，然后 `download_
 
 ## 路线
 
-路线图用 `v1 / v1.1 / v1.2 / v2` 讲产品阶段，npm 版本按语义化版本走。两者对不上一一对应，查这张表：
+路线图用 `v1 / v1.1 / v1.2 / v1.3 / v2` 讲产品阶段，npm 版本按语义化版本走。两者对不上一一对应，查这张表：
 
 | npm 版本 | 路线图 | 内容 | 状态 |
 | --- | --- | --- | --- |
 | `0.1.x` | v1 | 队列 + 持久化 + 七个工具 + aria2 传输 | 已发布 |
 | `0.2.x` | v1.1 | 镜像表 / 镜像冷却 / 并发分档 / 会话公平 / 磁盘余量 / 失败显式化 | 已发布 |
-| `0.3.x` | v1.2 | 队列去重 / 镜像自证同源 | 计划 |
+| `0.3.x` | v1.2 | 队列去重 / 镜像自证同源 | 已发布 |
+| `0.4.x` | v1.3 | 自定义请求头（`Referer` / `Cookie` / `User-Agent`）：每次尝试都带，入队时校验，输出只列名字 | 已发布 |
+| `0.4.1` | 补丁 | 没装 aria2 从"静默变慢"改成"明确警告 + 安装命令"；`download_status` 能看出走的哪条通道 | 已发布 |
 | `1.0.0` | v2 | 拦截 shell 下载 / 种子与磁力 / 自研分段引擎 / 内容寻址 | 计划 |
 | `1.1.x` | v3 | 用户面板 + 操作回传 agent | 计划 |
 

package/lib/headers.js

@@ -0,0 +1,138 @@
+/**
+ * Custom HTTP request headers for a download job.
+ *
+ * Why this exists (v1.3): some CDNs refuse a bare GET. Bilibili's media CDN
+ * answers 403 unless the request carries `Referer: https://www.bilibili.com`;
+ * private endpoints want `Cookie`; some hosts sniff `User-Agent`. Without a way
+ * to send headers, a whole class of direct links is undownloadable no matter how
+ * many mirrors are offered.
+ *
+ * What this module owns:
+ *   · validation — a bad header must fail at enqueue time, not mid-transfer
+ *   · the aria2 shape — `aria2.addUri` wants `["Name: Value", ...]`, not an object
+ *   · redaction — values may be cookies or tokens, so output lists names only
+ *
+ * SPEC §11.3.
+ */
+
+/** RFC 7230 token: what a header name may legally contain. */
+const NAME_RE = /^[!#$%&'*+\-.^_`|~0-9A-Za-z]+$/
+
+/**
+ * Headers the transfer layer already owns, or that would let a caller break it.
+ * Rejecting these is not paranoia: a user-supplied `Range` silently corrupts
+ * resume, and `Host`/`Content-Length` desync aria2's accounting.
+ */
+const RESERVED = new Set([
+  'host',
+  'content-length',
+  'transfer-encoding',
+  'connection',
+  'content-encoding',
+  'range',
+  'accept-encoding',
+])
+
+/** Upper bounds, so one job cannot smuggle an unbounded blob into the store. */
+const MAX_HEADERS = 32
+const MAX_NAME_LEN = 64
+const MAX_VALUE_LEN = 4096
+
+export class HeaderError extends Error {
+  /** @param {string} message */
+  constructor(message) {
+    super(message)
+    this.name = 'HeaderError'
+    this.code = 'header-invalid'
+  }
+}
+
+/**
+ * Validate and copy a caller-supplied header map.
+ *
+ * @param {unknown} input `{ "Referer": "https://www.bilibili.com" }`
+ * @returns {Record<string, string>} a fresh plain object, empty when `input` is empty
+ * @throws {HeaderError} on anything the transport could not send faithfully
+ */
+export function normalizeHeaders(input) {
+  if (input === undefined || input === null) return {}
+  if (typeof input !== 'object' || Array.isArray(input)) {
+    throw new HeaderError('headers must be an object of name/value strings')
+  }
+
+  const entries = Object.entries(input)
+  if (entries.length === 0) return {}
+  if (entries.length > MAX_HEADERS) {
+    throw new HeaderError(`too many headers (${entries.length}); at most ${MAX_HEADERS} are supported`)
+  }
+
+  /** @type {Record<string, string>} */
+  const out = {}
+  const seen = new Set()
+
+  for (const [rawName, rawValue] of entries) {
+    const name = rawName.trim()
+    if (!name) throw new HeaderError('header name must not be empty')
+    if (name.length > MAX_NAME_LEN) {
+      throw new HeaderError(`header name too long: ${name.slice(0, 32)}…`)
+    }
+    if (!NAME_RE.test(name)) {
+      throw new HeaderError(`invalid header name: ${JSON.stringify(name)}`)
+    }
+
+    const lower = name.toLowerCase()
+    if (RESERVED.has(lower)) {
+      throw new HeaderError(
+        `header ${name} is managed by the downloader and cannot be set per job`,
+      )
+    }
+    if (seen.has(lower)) {
+      throw new HeaderError(`duplicate header: ${name} (names are case-insensitive)`)
+    }
+    seen.add(lower)
+
+    if (typeof rawValue !== 'string') {
+      throw new HeaderError(`header ${name} must have a string value`)
+    }
+    // \r\n would let a value forge extra header lines; other control characters
+    // are either illegal on the wire or mangled by a proxy.
+    if (/[\r\n]/.test(rawValue)) {
+      throw new HeaderError(`header ${name} contains a line break`)
+    }
+    // eslint-disable-next-line no-control-regex
+    if (/[\u0000-\u0008\u000B\u000C\u000E-\u001F\u007F]/.test(rawValue)) {
+      throw new HeaderError(`header ${name} contains a control character`)
+    }
+    if (rawValue.length > MAX_VALUE_LEN) {
+      throw new HeaderError(`header ${name} value too long (${rawValue.length} > ${MAX_VALUE_LEN})`)
+    }
+
+    out[name] = rawValue
+  }
+
+  return out
+}
+
+/**
+ * aria2's JSON-RPC takes headers as a flat list of `"Name: Value"` strings and
+ * sends each one with `--header`. An empty list must stay absent, not `[]`.
+ * @param {Record<string, string> | undefined} headers
+ * @returns {string[] | undefined}
+ */
+export function toAria2Headers(headers) {
+  if (!headers) return undefined
+  const list = Object.entries(headers).map(([name, value]) => `${name}: ${value}`)
+  return list.length > 0 ? list : undefined
+}
+
+/**
+ * The names alone, for anything a human or a model will read.
+ * Values never leave the transfer layer — they may be cookies or tokens.
+ * @param {Record<string, string> | undefined} headers
+ * @returns {string[] | undefined}
+ */
+export function headerNames(headers) {
+  if (!headers) return undefined
+  const names = Object.keys(headers)
+  return names.length > 0 ? names : undefined
+}