dsh-dl-server 0.2.0 → 0.3.0

package/README.md

@@ -78,6 +78,15 @@ A download started in one session is visible to every other session — progress
 | **Cumulative disk budget** | Enqueue counts what already-queued jobs still need, so a batch cannot quietly fill the disk. |
 | **Failures are no longer silent** | Checkpoint write errors are counted and surfaced in `download_status` under `storage`; an unwritable destination is refused before the transfer starts. |
 
+## New in 0.3.0
+
+| | |
+| --- | --- |
+| **Queue de-duplication** | One destination is never downloaded twice. A finished job is reused while its file is still on disk; a different source claiming the same path fails with `dest-in-use`. |
+| **Mirror self-verification** | Before a mirror is used, it is HEADed once: the length must equal the origin's, and when the origin advertises an ETag the mirror's must match too. |
+
+> A mirror is a third party and serves whatever it likes. Length plus ETag catches a cached older build or an HTML error page; it does **not** catch a poisoned copy. Only `sha256` proves content, which is why mirrors are a fallback and not a guarantee.
+
 ## Example
 
 ```
@@ -123,12 +132,19 @@ The last row is the point: **on a per-connection-throttled host, segmentation be
 
 ## Roadmap
 
-```
-v1    queue + persistence + seven tools + aria2 transport   ← released
-v1.1  disk headroom / silent failures / session fairness / mirrors
-v2    shell interception / torrents / own segmented engine / content addressing
-v3    user panel + callbacks into the agent
-```
+The roadmap names product stages (`v1`, `v1.1`, `v1.2`, `v2`); npm versions follow semver. They do not map one-to-one — use this table:
+
+| npm | Roadmap | Content | Status |
+| --- | --- | --- | --- |
+| `0.1.x` | v1 | queue + persistence + seven tools + aria2 transport | released |
+| `0.2.x` | v1.1 | mirrors / mirror cooldown / size-aware concurrency / session fairness / disk budget / loud failures | released |
+| `0.3.x` | v1.2 | queue de-duplication / mirror self-verification | planned |
+| `1.0.0` | v2 | shell interception / torrents / own segmented engine / content addressing | planned |
+| `1.1.x` | v3 | user panel + callbacks into the agent | planned |
+
+**Why 0.x lasts until v2:** `1.0.0` means "the tool names and config keys will not change incompatibly". Only v2 touches the transport and the tool set, so that is where 1.0 belongs.
+
+**You never have to do the conversion yourself** — install the latest and you get the newest milestone.
 
 ## License
 

package/README.zh.md

@@ -78,6 +78,15 @@ DLS 做的是**队列层**，不是传输层。
 | **磁盘余量累加** | 入队时把「已在队列里还没下完的」一起算进去，批量入队写不满盘 |
 | **失败不再静默** | 断点写失败会计数并通过 `download_status` 的 `storage` 字段暴露；目标目录不可写在传输开始前就拒绝 |
 
+## 0.3.0 新增
+
+| 能力 | 说明 |
+| --- | --- |
+| **队列去重** | 同一个落地路径不会下两份。同源且文件已在盘上时直接复用已完成的任务；不同源抢同一路径会明确报 `dest-in-use` |
+| **镜像自证同源** | 用镜像前先 HEAD 一次，长度必须与原始一致；原始给了 ETag 时镜像的 ETag 也必须相等，否则不用 |
+
+> 镜像是第三方，它给什么我们不知道。长度 + ETag 能挡住「缓存了旧版本」「返回了错误页」这类问题，**挡不住投毒** —— 那只有 `sha256` 做得到。所以镜像只是兜底，真要保证内容就对 `sha256`。
+
 ## 用它
 
 ```
@@ -123,12 +132,19 @@ agent 会调 `download_enqueue` 两次拿到 `J#10` / `J#11`，然后 `download_
 
 ## 路线
 
-```
-v1    队列 + 持久化 + 七个工具 + aria2 传输      ← 已发布
-v1.1  磁盘余量 / 静默失败 / 会话公平 / 镜像表
-v2    拦截钩子 / 种子 / 自研分段 / 内容寻址
-v3    用户面板 + 操作回传
-```
+路线图用 `v1 / v1.1 / v1.2 / v2` 讲产品阶段，npm 版本按语义化版本走。两者对不上一一对应，查这张表：
+
+| npm 版本 | 路线图 | 内容 | 状态 |
+| --- | --- | --- | --- |
+| `0.1.x` | v1 | 队列 + 持久化 + 七个工具 + aria2 传输 | 已发布 |
+| `0.2.x` | v1.1 | 镜像表 / 镜像冷却 / 并发分档 / 会话公平 / 磁盘余量 / 失败显式化 | 已发布 |
+| `0.3.x` | v1.2 | 队列去重 / 镜像自证同源 | 计划 |
+| `1.0.0` | v2 | 拦截 shell 下载 / 种子与磁力 / 自研分段引擎 / 内容寻址 | 计划 |
+| `1.1.x` | v3 | 用户面板 + 操作回传 agent | 计划 |
+
+**为什么 0.x 一直拖到 v2：** 1.0.0 在语义化版本里意味着「工具名和配置项从此不破坏性变更」。v2 才会动到传输层和工具集合，在那之前不配叫 1.0。
+
+**升级时不用换算**：装最新版就行，`dsh plugin add dsh-dl-server` 永远指向当下最新。
 
 ## 许可
 

package/lib/mirrors.js

@@ -100,3 +100,43 @@ export function hostOf(url) {
     return undefined
   }
 }
+
+/** `W/"abc"` → `abc`; weak validators and quoting differ between servers. */
+export function normalizeEtag(etag) {
+  if (typeof etag !== 'string') return ''
+  return etag.trim().replace(/^W\//i, '').replace(/^"|"$/g, '').toLowerCase()
+}
+
+/**
+ * Can `candidate` be accepted as the same bytes the origin would serve?
+ *
+ * This is a heuristic, not a proof: an ETag is whatever the mirror declares.
+ * It catches the common failures — a mirror caching an older build, or answering
+ * with an HTML error page — and nothing more. Only `expectSha256` proves content.
+ *
+ * @param {{ length?: number, etag?: string }} origin
+ * @param {{ length?: number, etag?: string }} candidate
+ * @returns {{ ok: boolean, reason?: string }}
+ */
+export function sameResource(origin, candidate) {
+  const originLength = Number(origin?.length ?? -1)
+  const candidateLength = Number(candidate?.length ?? -1)
+
+  // No baseline (the origin probe failed): we cannot prove anything, and the
+  // mirror exists precisely for the case where the origin is unreachable.
+  if (!(originLength > 0)) return { ok: true, reason: 'origin-unknown' }
+
+  if (!(candidateLength > 0)) return { ok: false, reason: 'mirror did not report a length' }
+  if (candidateLength !== originLength) {
+    return { ok: false, reason: `length ${candidateLength} != ${originLength}` }
+  }
+
+  const originEtag = normalizeEtag(origin?.etag)
+  if (originEtag) {
+    const candidateEtag = normalizeEtag(candidate?.etag)
+    if (!candidateEtag) return { ok: false, reason: 'mirror reports no ETag' }
+    if (candidateEtag !== originEtag) return { ok: false, reason: 'ETag differs' }
+  }
+
+  return { ok: true }
+}

package/lib/queue.js

@@ -38,6 +38,26 @@ function storageKeyOf(id) {
   return seq === undefined ? id : `j${seq}`
 }
 
+/**
+ * Comparison key for a destination path. Windows folds case and accepts both
+ * separators, so `D:\A\b.bin` and `d:/a/B.bin` are the same file there.
+ * @param {string} destPath
+ */
+function destKeyOf(destPath) {
+  const resolved = path.resolve(destPath).replace(/[\\/]+$/, '')
+  return process.platform === 'win32' ? resolved.toLowerCase() : resolved
+}
+
+/** @param {string} file */
+async function fileExists(file) {
+  try {
+    const info = await fs.stat(file)
+    return info.isFile() && info.size > 0
+  } catch {
+    return false
+  }
+}
+
 export class QueueError extends Error {
   /** @param {string} code @param {string} message */
   constructor(code, message) {
@@ -99,6 +119,8 @@ export class Queue {
     this.closed = false
     this.timer = undefined
     this.checkpointTimer = undefined
+    /** Session served last, so the next pass starts with a different one. */
+    this.lastServed = undefined
 
     /** @type {{ errors: number, lastError?: string, lastAt?: number }} */
     this.storage = { errors: 0 }
@@ -206,9 +228,37 @@ export class Queue {
     }
 
     const expanded = expandMirrors(spec.urls)
+    const destPath = path.resolve(spec.destPath)
+    const key = destKeyOf(destPath)
+
+    // Two jobs writing the same path is a conflict no matter which URL each one
+    // uses: aria2 would fight over the same .aria2 control file. Dedupe on the
+    // destination, not on the URL — one file can legitimately have many URLs.
+    const sameDest = [...this.jobs.entries()].map(([, record]) => record).filter((record) => destKeyOf(record.destPath) === key)
+
+    const active = sameDest.find((record) => !isTerminal(record.state))
+    if (active) {
+      if (active.urls[0] === expanded.urls[0]) {
+        this.log('info', `${active.id} reused by ${caller.session.slice(0, 8)} (same destination and source)`)
+        return { ...this.#view(active), deduplicated: true }
+      }
+      throw new QueueError(
+        'dest-in-use',
+        `${destPath} is already being downloaded as ${active.id} from ${active.urls[0]}; cancel it first or choose another destination`,
+      )
+    }
+
+    // A finished job for the same file is what the caller wants far more often
+    // than a second download. Only reuse it while the file is still on disk —
+    // a deleted file must be fetched again.
+    const finished = sameDest.find((record) => record.state === 'completed' && record.urls[0] === expanded.urls[0])
+    if (finished && (await fileExists(finished.destPath))) {
+      this.log('info', `${finished.id} reused by ${caller.session.slice(0, 8)} (already downloaded)`)
+      return { ...this.#view(finished), deduplicated: true }
+    }
 
-    await this.#assertDestWritable(spec.destPath)
-    await this.#assertDiskSpace(spec.destPath, spec.expectedBytes)
+    await this.#assertDestWritable(destPath)
+    await this.#assertDiskSpace(destPath, spec.expectedBytes)
 
     const seq = await this.#nextSeq()
     const id = formatJobId(seq)
@@ -223,7 +273,7 @@ export class Queue {
       origin: caller.origin,
       urls: expanded.urls,
       mirrors: expanded.applied.length > 0 ? expanded.applied : undefined,
-      destPath: spec.destPath,
+      destPath,
       state: 'queued',
       createdAt: stamp,
       updatedAt: stamp,
@@ -397,7 +447,7 @@ export class Queue {
       owner: { workspace: record.ownerWorkspace, session: record.ownerSession },
       origin: record.origin,
       urls: record.urls,
-      mirrors: record.mirrors,
+      mirrors: this.#mirrorView(record),
       dest: record.destPath,
       bytesTotal: total,
       bytesDone: done,
@@ -419,6 +469,21 @@ export class Queue {
     })
   }
 
+  /**
+   * Candidates were recorded at enqueue time; the transport only learns which of
+   * them actually proved to serve the same bytes once the transfer starts.
+   * @param {import('./types.js').JobRecord} record
+   */
+  #mirrorView(record) {
+    if (!record.mirrors) return undefined
+    const report = this.transport?.mirrorReports?.get(record.urls[0])
+    return {
+      candidates: record.mirrors,
+      verified: report?.verified ?? [],
+      rejected: report?.rejected ?? [],
+    }
+  }
+
   #schedule() {
     if (this.closed || this.timer) return
     this.timer = setTimeout(() => {
@@ -462,12 +527,16 @@ export class Queue {
       else bySession.set(key, [record])
     }
 
-    // Round-robin across sessions, so a session that queued ten jobs cannot run
-    // ahead of one that queued a single job.
+    // Start each pass after the session that was served last, otherwise the
+    // first session in iteration order wins every tie and round-robin is a lie.
+    const order = [...bySession.keys()]
+    const startAt = this.lastServed && order.includes(this.lastServed) ? (order.indexOf(this.lastServed) + 1) % order.length : 0
+    const rotated = [...order.slice(startAt), ...order.slice(0, startAt)]
+
     let progressed = true
     while (progressed && this.active.size < this.config.maxConcurrentJobs) {
       progressed = false
-      for (const key of bySession.keys()) {
+      for (const key of rotated) {
         if (this.active.size >= this.config.maxConcurrentJobs) break
         const list = bySession.get(key)
         if (!list || list.length === 0) continue
@@ -475,6 +544,7 @@ export class Queue {
         const record = list.shift()
         if (this.active.has(record.id)) continue
         running.set(key, (running.get(key) ?? 0) + 1)
+        this.lastServed = key
         void this.#run(record)
         progressed = true
       }

package/lib/tools.js

@@ -90,7 +90,13 @@ export function registerTools({ ctx, queue, config }) {
               { urls: args.urls, destPath: args.dest, expectSha256: args.sha256 },
               withCaller(exec),
             )
-            return { ok: true, jobId: job.id, state: job.state, dest: job.dest }
+            return {
+              ok: true,
+              jobId: job.id,
+              state: job.state,
+              dest: job.dest,
+              ...(job.deduplicated ? { deduplicated: true } : {}),
+            }
           } catch (error) {
             return fail(error)
           }

package/lib/transport.js

@@ -6,6 +6,7 @@ import path from 'node:path'
 import { Readable } from 'node:stream'
 import { pipeline } from 'node:stream/promises'
 import { randomBytes } from 'node:crypto'
+import { sameResource } from './mirrors.js'
 
 /**
  * Transport layer. Two channels: an aria2c JSON-RPC daemon when one is
@@ -194,6 +195,8 @@ export class Transport {
     this.mode = 'direct'
     this.aria2 = undefined
     this.rpc = undefined
+    /** @type {Map<string, { verified: string[], rejected: { url: string, reason: string }[] }>} */
+    this.mirrorReports = new Map()
   }
 
   /** Detect aria2 and start the daemon if possible. Never throws. */
@@ -247,15 +250,16 @@ export class Transport {
 
   /** @param {TransferOptions} options @returns {Promise<TransferResult>} */
   async transfer(options) {
-    const urls = options.urls
-    const declared = await this.#probeSize(urls, options.signal)
-    const plan = planTransfer(declared, options.connectionsCap ?? options.connections ?? this.config.connectionsPerJob ?? 16)
+    const origin = await this.#probeHead(options.urls[0], options.signal)
+    const plan = planTransfer(origin.length, options.connectionsCap ?? options.connections ?? this.config.connectionsPerJob ?? 16)
     const connections = options.connections ?? plan.connections
+
+    const urls = await this.#verifiedUrls(options.urls, origin, options.signal)
     const useAria2 = this.mode === 'aria2' && this.rpc && !options.preferDirect && !plan.direct
 
     if (useAria2) {
       try {
-        const result = await this.#transferAria2({ ...options, connections })
+        const result = await this.#transferAria2({ ...options, urls, connections })
         options.onUrlResult?.(urls[0], true)
         return result
       } catch (error) {
@@ -263,19 +267,53 @@ export class Transport {
         this.log('warn', `aria2 transfer failed, retrying single-stream: ${error.message}`)
       }
     }
-    return this.#transferDirect(options)
+    return this.#transferDirect({ ...options, urls })
   }
 
   /**
-   * Best-effort Content-Length lookup. A failure is not an error: unknown size
-   * only means the planner falls back to its middle-of-the-road setting.
+   * Keep only the mirrors that can be shown to serve the same bytes as the
+   * origin. A mirror is a third party: it may hold an older build or answer with
+   * an HTML error page, and `sha256` is the only proof of content. Length plus
+   * ETag catches the common mistakes; the reasoning is documented in SPEC §11.2.
    * @param {string[]} urls
+   * @param {{ length: number, etag?: string }} origin
+   * @param {AbortSignal} [signal]
+   */
+  async #verifiedUrls(urls, origin, signal) {
+    if (urls.length <= 1) return urls
+
+    const accepted = [urls[0]]
+    /** @type {{ url: string, reason: string }[]} */
+    const rejected = []
+
+    for (const mirror of urls.slice(1)) {
+      const head = await this.#probeHead(mirror, signal)
+      const verdict = sameResource(origin, head)
+      if (verdict.ok) {
+        accepted.push(mirror)
+        if (verdict.reason === 'origin-unknown') {
+          this.log('warn', `mirror ${mirror} kept unverified: the origin did not report a length`)
+        }
+      } else {
+        rejected.push({ url: mirror, reason: verdict.reason ?? 'unverified' })
+        this.log('warn', `mirror ${mirror} rejected: ${verdict.reason}`)
+      }
+    }
+
+    this.mirrorReports.set(urls[0], { verified: accepted.slice(1), rejected })
+    return accepted
+  }
+
+  /**
+   * Best-effort HEAD: the length and validator used for sizing and for proving a
+   * mirror matches. A failure is not an error — unknown size only means the
+   * planner stays conservative.
+   * @param {string} url
    * @param {AbortSignal} [signal]
-   * @returns {Promise<number>} the size, or -1 when it could not be determined
+   * @returns {Promise<{ length: number, etag?: string }>}
    */
-  async #probeSize(urls, signal) {
-    const url = urls[0]
-    if (!url) return -1
+  async #probeHead(url, signal) {
+    if (!url) return { length: -1 }
     const controller = new AbortController()
     const timer = setTimeout(() => controller.abort(), 5000)
     timer.unref?.()
@@ -284,11 +322,12 @@ export class Transport {
 
     try {
       const response = await fetch(url, { method: 'HEAD', signal: controller.signal })
-      if (!response.ok) return -1
+      if (!response.ok) return { length: -1 }
       const length = Number(response.headers.get('content-length') ?? 0)
-      return Number.isFinite(length) && length > 0 ? length : -1
+      const etag = response.headers.get('etag') ?? undefined
+      return { length: Number.isFinite(length) && length > 0 ? length : -1, etag }
     } catch {
-      return -1
+      return { length: -1 }
     } finally {
       clearTimeout(timer)
       signal?.removeEventListener('abort', onAbort)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-dl-server",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "description": "DLS — DSH Download Server: a global, session-aware, persistent download queue for DeepSeek Harness.",
   "type": "module",
   "main": "lib/index.js",