dsh-layered-memory 0.8.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +53 -10
- package/README.md +50 -8
- package/assets/readme/bench-dialog.svg +87 -0
- package/assets/readme/bench-workflow.svg +79 -0
- package/assets/readme/flow.svg +74 -74
- package/assets/readme/storage.svg +56 -56
- package/dist/client.js +369 -0
- package/dist/config.d.ts +15 -0
- package/dist/config.js +1 -0
- package/dist/index.d.ts +4 -0
- package/dist/index.js +10 -3
- package/dist/llm.d.ts +15 -0
- package/dist/llm.js +16 -0
- package/dist/pipeline/l1.js +3 -3
- package/dist/pipeline/l2.js +2 -2
- package/dist/pipeline/l3.js +2 -2
- package/dist/pipeline/rebuild.js +2 -2
- package/dist/pipeline/runner.d.ts +9 -3
- package/dist/pipeline/runner.js +37 -4
- package/dist/settings.d.ts +14 -0
- package/dist/settings.js +41 -2
- package/dist/stats.js +113 -1
- package/dist/store/download-queue.d.ts +35 -2
- package/dist/store/download-queue.js +102 -5
- package/dist/store/model-catalog.js +1 -1
- package/package.json +5 -3
package/README.en.md
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
[简体中文](README.md) · [Latest release](https://github.com/JunNanLYS/dsh-layered-memory/releases/latest) · [Report issues](https://github.com/JunNanLYS/dsh-layered-memory/issues)
|
|
11
11
|
|
|
12
|
-
[](https://www.npmjs.com/package/dsh-layered-memory)
|
|
13
13
|
[](https://github.com/deepseek-ai/deepseek-harness)
|
|
14
14
|
[](LICENSE)
|
|
15
15
|
|
|
@@ -21,7 +21,7 @@ Requires Node ≥ 22.16. Two invocation styles — the `npx` prefix can replace
|
|
|
21
21
|
any command below:
|
|
22
22
|
|
|
23
23
|
```bash
|
|
24
|
-
# Option 1: run the official CLI directly via npx (no pre-installed dsh; version can be pinned, e.g. dsh-layered-memory@0.
|
|
24
|
+
# Option 1: run the official CLI directly via npx (no pre-installed dsh; version can be pinned, e.g. dsh-layered-memory@0.8.0)
|
|
25
25
|
npx -y @deepseek-ai/dsh plugin --profile web add dsh-layered-memory
|
|
26
26
|
|
|
27
27
|
# Option 2: with the dsh CLI installed (dsh is a pnpm forwarder; npm i -g pnpm first if missing)
|
|
@@ -142,6 +142,44 @@ trajectory view):
|
|
|
142
142
|
alt="The same settings memory browser overview in light theme: identical layout and information on light card backgrounds with the same accent family, theme switch without reload">
|
|
143
143
|
</p>
|
|
144
144
|
|
|
145
|
+
## Measured Comparison (DSH-MemBench: Automated Benchmark)
|
|
146
|
+
|
|
147
|
+
Screenshots show what the plugin looks like — this section answers "**what does enabling it actually buy you?**" with measured numbers from an **automated benchmark** ([`bench/`](./bench/), one command to reproduce). Method: the same scenario bank with verbatim-identical inputs runs in **Group A (memory on)** and **Group B (memory off)**, 3 repetitions each, merged; environment DeepSeek official `deepseek-v4-flash`, plugin 0.8.0, Windows; taxonomy adapted from [LongMemEval](https://github.com/xiaowu0162/longmemeval) / [LoCoMo](https://snap-research.github.io/locomo/) / [AMB](https://github.com/vectorize-io/agent-memory-benchmark).
|
|
148
|
+
|
|
149
|
+
### Dialog track (15 scenarios × 6 probe types × 3 reps = 270 questions/group): does it remember correctly
|
|
150
|
+
|
|
151
|
+
<p align="center">
|
|
152
|
+
<img src="./assets/readme/bench-dialog.svg" width="100%"
|
|
153
|
+
alt="DSH-MemBench dialog track, Group A vs Group B bar chart: overall accuracy A (memory on) 92.6% (250/270) vs B (memory off) 17.8% (48/270); per probe type, 45 each — extraction A 45/45 vs B 3/45, multi-hop A 45/45 vs B 0/45, temporal A 43/45 vs B 0/45, knowledge updates A 31/45 vs B 0/45, scene recall A 41/45 vs B 0/45, abstention both 45/45 with 0 fabricated">
|
|
154
|
+
</p>
|
|
155
|
+
|
|
156
|
+
**Dual-channel recall** (Group A): passive injection hit rate **75.1%** (the answer's key points appear in the recall injection, 169/225); most of the rest the model recovered by **actively calling the memory tools** — 84 questions with active queries, **60 rescued by tools**. The end-to-end 92.6% is the composite of both channels plus model utilization. With the memory store accumulating across scenarios for the whole run, 144 probe injections carried other scenarios' memories (honestly counted) — yet overall accuracy held at 92.6%: interference resistance under a growing store, measured.
|
|
157
|
+
|
|
158
|
+
### Workflow track (4 scenarios × 3 reps, real tool sandbox): does it do it right, and cheaper
|
|
159
|
+
|
|
160
|
+
<p align="center">
|
|
161
|
+
<img src="./assets/readme/bench-workflow.svg" width="100%"
|
|
162
|
+
alt="DSH-MemBench workflow track, Group A vs Group B: task completion A 24/33 (72.7%) vs B 11/33 (33.3%); cost comparison (Group B as the full-bar baseline) — steps 125 vs 186 (B +49%), tool calls 184 vs 296 (B +61%), input tokens 1.30M vs 1.86M (B +43%); asks-user-for-help A 0 vs B 3; login scenario input tokens A 241k vs B 453k (+88%)">
|
|
163
|
+
</p>
|
|
164
|
+
|
|
165
|
+
**Login scenario close-up** (credentials exist only in memory; the site is a local service with unforgeable tokens): Group A completed all three runs **in a single turn each** (6/6, 241k input); Group B had to **ask the user for credentials every time** (3 asks, double the turns) and still finished only 5/6, at 453k input — **+88%**. This is one of memory's core values: **what it saves is not task difficulty, but pointless round-trips and re-teaching**.
|
|
166
|
+
|
|
167
|
+
### Methodology & reproduction
|
|
168
|
+
|
|
169
|
+
```bash
|
|
170
|
+
node bench/harness/run.mjs --arm A --repeats 3 --provider deepseek-official --model deepseek-v4-flash
|
|
171
|
+
node bench/harness/run.mjs --arm B --repeats 3 --provider deepseek-official --model deepseek-v4-flash # dialog track
|
|
172
|
+
node bench/harness/run.mjs --track workflow --arm A/B --repeats 3 ... # workflow track
|
|
173
|
+
node bench/harness/report.mjs --latest [dialog|workflow] # aggregate report
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
- Scoring: programmatic `contains-all` plus an LLM judge against key points (every answer and verdict is preserved in `result.json` for human audit); workflow completion is verified programmatically from produced files and their contents;
|
|
177
|
+
- Metrics come from provider-reported usage (input with cache-hit split) and session-event folding; the **steady-state cache rate** excludes each session's first request (A 88.7% vs B 85.4% — memory injection does not hurt caching);
|
|
178
|
+
- Regression use: run before/after a plugin change and diff with `compare.mjs` (environment header check + Group-B control-drift warning);
|
|
179
|
+
- Limitations (stated honestly): single machine, 3 merged runs; the judge model is the same as the tested model; the scenario bank is author-built (biased toward memory-advantage scenarios — reproduce it yourself); the tool audit flags out-of-sandbox access (agents occasionally probed the user home dir in tests; this benchmark's answers never exist in the real memory store, so the numbers are unaffected).
|
|
180
|
+
|
|
181
|
+
Full reports and per-question data: [`bench/baseline/`](./bench/baseline/).
|
|
182
|
+
|
|
145
183
|
## Configuration
|
|
146
184
|
|
|
147
185
|
Override configs go into the profile's own `cordis.patch.yml` as a **top-level bare
|
|
@@ -153,9 +191,9 @@ the bundle layer appends and causes `duplicate loader entry id` startup failure)
|
|
|
153
191
|
name: dsh-layered-memory
|
|
154
192
|
config: # keys replace whole lines (no deep merge); write out all keys you want to keep
|
|
155
193
|
family: auto # default mode for new sessions: auto | chat | work
|
|
156
|
-
llm: # distillation
|
|
157
|
-
provider: ''
|
|
158
|
-
model: ''
|
|
194
|
+
llm: # static distillation route (both fields set = deployment pin,
|
|
195
|
+
provider: '' # which outranks the settings-page selection; when empty the route
|
|
196
|
+
model: '' # follows the settings-page "distillation model" picker or the default model)
|
|
159
197
|
```
|
|
160
198
|
|
|
161
199
|
| Field | Default | Description |
|
|
@@ -194,11 +232,12 @@ the bundle layer appends and causes `duplicate loader entry id` startup failure)
|
|
|
194
232
|
| `embedding.timeoutMs` | `10000` | Per-call embedding timeout (ms) |
|
|
195
233
|
| `embedding.allowLocalModels` | `true` | Allow the local embedding tier (deployment ceiling; when off, no model downloads and no local tier in settings) |
|
|
196
234
|
| `embedding.mirror` | `https://hf-mirror.com` | Download mirror root for local models (can be changed back to `https://huggingface.co`) |
|
|
197
|
-
| `
|
|
198
|
-
| `llm.
|
|
235
|
+
| `embedding.proxy` | `''` | Three-state download proxy: `''` (default) = auto-detect proxy env vars (`HTTPS_PROXY`/`ALL_PROXY` etc., honoring `NO_PROXY`); `none` = disable, always direct; any other value = proxy URL (e.g. `http://127.0.0.1:7890`). Direct connections to the mirror are intermittently unreachable on some networks (connect timeouts and poisoned bytes have both been observed) — keep the default auto-detection on machines with a proxy |
|
|
236
|
+
| `llm.provider/model` | empty | Static distillation route (deployment pin): when **both** fields are set the route is locked, outranking the settings-page selection and the default model (deployments can force distillation onto a specific route); when empty the route follows "settings-page selection → default model". At runtime, switch among **configured providers** (including custom ones added in dsh Settings → Models) via the "distillation model" picker in Settings → Memory → Overview — effective immediately, no restart needed |
|
|
237
|
+
| `llm.maxTokens` | `65536` | Fallback output cap for non-layered calls. Each distillation stage has its own budget (extraction 16k / dedup 8k / L2 32k / L3 16k; auto ×4 when reasoning effort is high/max, so thinking can't starve the text budget); the per-layer budgets are runtime-adjustable in Settings → Memory → Overview → distillation parameters (empty/0 = built-in defaults) |
|
|
199
238
|
| `llm.reasoningEffort` | `off` | Distillation reasoning-effort tier (deployment default): `off` / `high` / `max`; empty string = don't send (follow model default). Distillation is structured extraction, so thinking is off by default — a reasoning model (e.g. v4-flash) at its default `high` tier can consume the entire output budget on thinking, leaving 0 chars of text; set to empty string for models that don't recognize the effort parameter. Switchable at runtime in Settings → Memory → Overview ("follow config" falls back to this value) |
|
|
200
239
|
| `llm.temperature` | `0.3` | Distillation temperature |
|
|
201
|
-
| `llm.maxInputChars` | `700000` | Input character budget per distillation call (over-budget L1 inputs are chunked automatically) |
|
|
240
|
+
| `llm.maxInputChars` | `700000` | Input character budget per distillation call (over-budget L1 inputs are chunked automatically); runtime-adjustable in Settings → distillation parameters → input budget (empty/0 = follow this value) |
|
|
202
241
|
| `llm.timeoutMs` | `120000` | Per-call distillation timeout (ms) |
|
|
203
242
|
| `tools` | `true` | Whether to register model-callable memory tools |
|
|
204
243
|
|
|
@@ -231,8 +270,12 @@ machine). The local catalog is a built-in allowlist (each model pinned to a revi
|
|
|
231
270
|
with per-file sha256; arbitrary repos cannot be downloaded).
|
|
232
271
|
|
|
233
272
|
- **Download**: one click on the model card (default mirror `hf-mirror.com`, resumable
|
|
234
|
-
downloads + sha256 integrity checks
|
|
235
|
-
|
|
273
|
+
downloads + sha256 integrity checks; a proxy is used when direct access is
|
|
274
|
+
unreachable — proxy env vars like `HTTPS_PROXY`/`ALL_PROXY` are auto-detected by
|
|
275
|
+
default, see `embedding.proxy`). Per-file failures auto-retry with a rotated cache
|
|
276
|
+
key (`?dshmem-retry=N`, sidestepping occasionally bad CDN cache objects); hash
|
|
277
|
+
mismatches restart from zero, network errors resume from the checkpoint; stored
|
|
278
|
+
under `models/<id>/` in the data directory, deletable from the settings page at any time;
|
|
236
279
|
- **On-demand runtime**: the inference runtime (transformers.js, ~100–200MB) is
|
|
237
280
|
installed only on first switch to the local tier, into `runtime/` in the data
|
|
238
281
|
directory — never in the plugin's dependency tree or install directory;
|
package/README.md
CHANGED
|
@@ -9,7 +9,7 @@
|
|
|
9
9
|
|
|
10
10
|
[English](README.en.md) · [最新发行版](https://github.com/JunNanLYS/dsh-layered-memory/releases/latest) · [反馈问题](https://github.com/JunNanLYS/dsh-layered-memory/issues)
|
|
11
11
|
|
|
12
|
-
[](https://www.npmjs.com/package/dsh-layered-memory)
|
|
13
13
|
[](https://github.com/deepseek-ai/deepseek-harness)
|
|
14
14
|
[](LICENSE)
|
|
15
15
|
|
|
@@ -20,7 +20,7 @@
|
|
|
20
20
|
需要 Node ≥ 22.16。两种调用方式任选(`npx` 前缀可替换下面任何 `dsh` 命令):
|
|
21
21
|
|
|
22
22
|
```bash
|
|
23
|
-
# 方式一:npx 直接跑官方 CLI(无需预装 dsh;可 pin 版本,如 dsh-layered-memory@0.
|
|
23
|
+
# 方式一:npx 直接跑官方 CLI(无需预装 dsh;可 pin 版本,如 dsh-layered-memory@0.8.0)
|
|
24
24
|
npx -y @deepseek-ai/dsh plugin --profile web add dsh-layered-memory
|
|
25
25
|
|
|
26
26
|
# 方式二:已装 dsh CLI(dsh 是 pnpm 转发器,未装 pnpm 时先 npm i -g pnpm)
|
|
@@ -122,6 +122,44 @@ npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution no
|
|
|
122
122
|
alt="浅色主题下的同一设置页记忆浏览器概览:同款布局与信息,浅色卡片底与同套强调色,主题切换无需重载">
|
|
123
123
|
</p>
|
|
124
124
|
|
|
125
|
+
## 实测对比(DSH-MemBench:自动化基准)
|
|
126
|
+
|
|
127
|
+
图文回答"长什么样",这一节用**自动化基准**的实测数字回答"**开了到底有什么用**"([`bench/`](./bench/),一条命令可复现)。方法:同场景库、逐字相同输入,**A 组(记忆开)** 与 **B 组(记忆关)** 各跑 3 次取合并值;环境 DeepSeek 官方 `deepseek-v4-flash`、插件 0.8.0、Windows;题型设计借鉴 [LongMemEval](https://github.com/xiaowu0162/longmemeval) / [LoCoMo](https://snap-research.github.io/locomo/) / [AMB](https://github.com/vectorize-io/agent-memory-benchmark)。
|
|
128
|
+
|
|
129
|
+
### 对话赛道(15 场景 × 6 题型 × 3 次 = 270 题/组):答得准吗
|
|
130
|
+
|
|
131
|
+
<p align="center">
|
|
132
|
+
<img src="./assets/readme/bench-dialog.svg" width="100%"
|
|
133
|
+
alt="DSH-MemBench 对话赛道 A/B 对照条形图:A 组(记忆开)总准确率 92.6%(250/270),B 组(记忆关)17.8%(48/270);分题型各 45 题——抽取 A 45/45 对 B 3/45、多跳 A 45/45 对 B 0/45、时序 A 43/45 对 B 0/45、知识更新 A 31/45 对 B 0/45、场景回忆 A 41/45 对 B 0/45、拒答两组均 45/45 且 0 编造">
|
|
134
|
+
</p>
|
|
135
|
+
|
|
136
|
+
**召回双通道**(A 组):被动注入召回率 **75.1%**(该题要点出现在召回注入中,169/225),其余多数由模型**主动调用记忆工具**查回——84 题主动查询、**60 题靠工具兜底答对**;端到端 92.6% 是两通道 + 模型利用的合成结果。记忆库跨场景全程累积下,探针召回注入混入其他场景记忆 144 次(已如实计数),总准确率仍稳在 92.6%——抗干扰能力经受住了膨胀记忆库的考验。
|
|
137
|
+
|
|
138
|
+
### 工作流赛道(4 场景 × 3 次,真实工具沙箱):做得对、做得省吗
|
|
139
|
+
|
|
140
|
+
<p align="center">
|
|
141
|
+
<img src="./assets/readme/bench-workflow.svg" width="100%"
|
|
142
|
+
alt="DSH-MemBench 工作流赛道 A/B 对照图:任务完成度 A 组 24/33(72.7%)对 B 组 11/33(33.3%);成本对比(B 组为满格基准)——步骤 125 对 186(B +49%)、工具调用 184 对 296(B +61%)、输入 token 1.30M 对 1.86M(B +43%);向用户求助 A 0 次对 B 3 次;登录场景输入 token A 241k 对 B 453k(+88%)">
|
|
143
|
+
</p>
|
|
144
|
+
|
|
145
|
+
**登录场景特写**(凭据只存在于记忆,站点为本地服务、令牌不可伪造):A 组三次全部**一轮直取**完成(6/6,241k 输入);B 组每次都要**反问用户要凭据**(3 次求助、双倍轮次)才完成 5/6,输入 453k——**+88%**。这正是记忆的核心价值之一:**省掉的不是任务难度,是无谓的往返与重复教学**。
|
|
146
|
+
|
|
147
|
+
### 方法论与复现
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
node bench/harness/run.mjs --arm A --repeats 3 --provider deepseek-official --model deepseek-v4-flash
|
|
151
|
+
node bench/harness/run.mjs --arm B --repeats 3 --provider deepseek-official --model deepseek-v4-flash # 对话赛道
|
|
152
|
+
node bench/harness/run.mjs --track workflow --arm A/B --repeats 3 ... # 工作流赛道
|
|
153
|
+
node bench/harness/report.mjs --latest [dialog|workflow] # 汇总报告
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
- 判分:`contains-all` 程序判 + 判卷模型按要点判(答案原文与判分理由全部留痕 `result.json` 可人工复核);工作流完成度为产物文件 + 关键内容程序化校验;
|
|
157
|
+
- 指标全部来自供应商上报 usage(输入含缓存命中拆分)与会话事件折叠;**稳态缓存率**剔除每会话首请求(A 88.7% vs B 85.4%——记忆注入不伤缓存);
|
|
158
|
+
- 回归用途:改插件前后各跑一遍,`compare.mjs` 出对比表(环境头校验 + B 组对照组漂移告警);
|
|
159
|
+
- 局限(诚实声明):单机 ×3 次合并、判卷模型与被测模型同源、作者自建场景库(倾向记忆优势场景,欢迎自行复现);工具审计会标记越界访问(实测 Agent 偶发探测用户主目录,本基准场景答案不存于真实记忆库、数字不受影响)。
|
|
160
|
+
|
|
161
|
+
完整报告与逐题数据:[`bench/baseline/`](./bench/baseline/)。
|
|
162
|
+
|
|
125
163
|
## 存储布局
|
|
126
164
|
|
|
127
165
|
<p align="center">
|
|
@@ -147,7 +185,10 @@ ONNX 量化 **CPU 推理**——无需 API Key,数据不出本机)。本地
|
|
|
147
185
|
白名单(每款锁定 revision + 每文件 sha256,不可下载任意仓库)。
|
|
148
186
|
|
|
149
187
|
- **下载**:模型卡一键下载(默认镜像 `hf-mirror.com`,断点续传 + sha256 完整性
|
|
150
|
-
|
|
188
|
+
校验;直连不可达时可走代理——默认自动探测 `HTTPS_PROXY`/`ALL_PROXY` 等环境
|
|
189
|
+
变量,见 `embedding.proxy`)。单文件失败自动重试且换缓存键(`?dshmem-retry=N`,
|
|
190
|
+
绕开镜像 CDN 偶发的坏缓存对象),校验失配从零重下、网络错误保留断点续传;
|
|
191
|
+
落盘数据目录 `models/<id>/`,不用了随时在设置页删除;
|
|
151
192
|
- **按需运行时**:首次切换本地档才安装推理运行时(transformers.js,约 100~200MB,
|
|
152
193
|
装进数据目录 `runtime/`——不进插件依赖树,不碰插件安装目录);
|
|
153
194
|
- **活切换**:一键换源——自动后台全量重嵌(进度可见、可取消,期间检索自动降级
|
|
@@ -167,8 +208,8 @@ ONNX 量化 **CPU 推理**——无需 API Key,数据不出本机)。本地
|
|
|
167
208
|
name: dsh-layered-memory
|
|
168
209
|
config: # 键按行整体替换(不深合并),按需写全要保留的键
|
|
169
210
|
family: auto # 新会话默认档:auto | chat | work
|
|
170
|
-
llm: #
|
|
171
|
-
provider: ''
|
|
211
|
+
llm: # 蒸馏模型静态路由(双字段齐 = 部署 pin,优先于设置页选择;
|
|
212
|
+
provider: '' # 留空则跟随设置页"蒸馏模型"选择器或当前默认模型)
|
|
172
213
|
model: ''
|
|
173
214
|
```
|
|
174
215
|
|
|
@@ -208,11 +249,12 @@ ONNX 量化 **CPU 推理**——无需 API Key,数据不出本机)。本地
|
|
|
208
249
|
| `embedding.timeoutMs` | `10000` | 单次 embedding 调用超时(ms) |
|
|
209
250
|
| `embedding.allowLocalModels` | `true` | 允许本地嵌入档(部署上限:关闭后设置页不能下载模型、不能切本地档) |
|
|
210
251
|
| `embedding.mirror` | `https://hf-mirror.com` | 本地模型下载镜像根地址(可改回官方 `https://huggingface.co`) |
|
|
211
|
-
| `
|
|
212
|
-
| `llm.
|
|
252
|
+
| `embedding.proxy` | `''` | 模型下载代理三态:`''`(默认)= 自动探测代理环境变量(`HTTPS_PROXY`/`ALL_PROXY` 等,尊重 `NO_PROXY`);`none` = 禁用强制直连;其他值 = 代理 URL(如 `http://127.0.0.1:7890`)。镜像直连在国内网络间歇不可达(直连超时与污染字节交替出现过),开代理的机器建议保持默认自动探测 |
|
|
253
|
+
| `llm.provider/model` | 空 | 蒸馏模型静态路由(部署 pin):provider 与 model **双字段齐**时锁定蒸馏路由,优先于设置页的运行时选择与默认模型(部署可强制蒸馏走指定路由);留空则跟随"设置页选择 → 默认模型"。运行时可在设置页 → 记忆 → 概览的"蒸馏模型"选择器从**已配置的供应商**(含 dsh 设置 → 模型里添加的自定义供应商)中切换,即时生效无需重启 |
|
|
254
|
+
| `llm.maxTokens` | `65536` | 未分层调用的兜底输出总闸。各蒸馏层有独立预算(抽取 16k / 去重 8k / L2 32k / L3 16k;思考档 high/max 时自动 ×4,防 reasoning 吃光预算),分层预算可在设置页 → 记忆 → 概览 → 蒸馏参数运行时调整(留空/0 = 跟随内置默认) |
|
|
213
255
|
| `llm.reasoningEffort` | `off` | 蒸馏思考档位(部署默认):`off` / `high` / `max`,空串不传(跟随模型默认)。蒸馏是结构化抽取任务,默认关思考——推理模型(如 v4-flash)默认 high 档的思考可把任意输出预算全部吃光导致正文 0 字符;非推理模型不认识 effort 时需设为空串。运行时可在设置页 → 记忆 → 概览临时切换(选"跟随配置"即回退本值) |
|
|
214
256
|
| `llm.temperature` | `0.3` | 蒸馏温度 |
|
|
215
|
-
| `llm.maxInputChars` | `700000` | 单次蒸馏输入字符预算(超限的 L1
|
|
257
|
+
| `llm.maxInputChars` | `700000` | 单次蒸馏输入字符预算(超限的 L1 输入自动分块抽取);运行时可在设置页 → 蒸馏参数 → 输入预算调整(留空/0 = 跟随本值) |
|
|
216
258
|
| `llm.timeoutMs` | `120000` | 单次蒸馏调用超时(ms) |
|
|
217
259
|
| `tools` | `true` | 是否注册模型可调用的记忆工具 |
|
|
218
260
|
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg"
|
|
2
|
+
width="1200" height="690" viewBox="0 0 1200 690"
|
|
3
|
+
role="img" aria-labelledby="benchDialogTitle benchDialogDesc">
|
|
4
|
+
<title id="benchDialogTitle">DSH-MemBench 对话赛道:A/B 两组准确率对照</title>
|
|
5
|
+
<desc id="benchDialogDesc">同题同输入对照:A 组(记忆开)总准确率 92.6%(250/270),B 组(记忆关)17.8%(48/270)。分题型(每题型 45 题):抽取 A 45/45 对 B 3/45;多跳 A 45/45 对 B 0/45;时序 A 43/45 对 B 0/45;知识更新 A 31/45 对 B 0/45;场景回忆 A 41/45 对 B 0/45;拒答两组均 45/45、0 编造。</desc>
|
|
6
|
+
|
|
7
|
+
<rect width="1200" height="690" rx="26" fill="#0D1526"/>
|
|
8
|
+
|
|
9
|
+
<g font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','PingFang SC','Microsoft YaHei',sans-serif">
|
|
10
|
+
|
|
11
|
+
<!-- ── 标题与图例 ── -->
|
|
12
|
+
<text x="64" y="66" font-size="26" font-weight="700" fill="#EFF3FA">对话赛道:同题同输入 · A/B 对照准确率</text>
|
|
13
|
+
<text x="64" y="94" font-size="18" fill="#8296B3">15 场景 × 6 题型 × 3 次 = 270 题/组(每题型 45 题)· 条长 = 答对率,数值 = 答对题数</text>
|
|
14
|
+
|
|
15
|
+
<rect x="842" y="50" width="18" height="18" rx="5" fill="#FFD15D"/>
|
|
16
|
+
<text x="868" y="65" font-size="18" fill="#C2CDDE">A 组 · 记忆开</text>
|
|
17
|
+
<rect x="842" y="76" width="18" height="18" rx="5" fill="#4E6DA8"/>
|
|
18
|
+
<text x="868" y="91" font-size="18" fill="#C2CDDE">B 组 · 记忆关</text>
|
|
19
|
+
|
|
20
|
+
<!-- ── 网格线(0/25/50/75/100%)── -->
|
|
21
|
+
<g stroke="#1C2A47" stroke-width="1">
|
|
22
|
+
<line x1="260" y1="122" x2="260" y2="614"/>
|
|
23
|
+
<line x1="435" y1="122" x2="435" y2="614"/>
|
|
24
|
+
<line x1="610" y1="122" x2="610" y2="614"/>
|
|
25
|
+
<line x1="785" y1="122" x2="785" y2="614"/>
|
|
26
|
+
<line x1="960" y1="122" x2="960" y2="614"/>
|
|
27
|
+
</g>
|
|
28
|
+
<g font-size="17" fill="#8296B3" text-anchor="middle">
|
|
29
|
+
<text x="260" y="640">0%</text>
|
|
30
|
+
<text x="435" y="640">25%</text>
|
|
31
|
+
<text x="610" y="640">50%</text>
|
|
32
|
+
<text x="785" y="640">75%</text>
|
|
33
|
+
<text x="960" y="640">100%</text>
|
|
34
|
+
</g>
|
|
35
|
+
|
|
36
|
+
<!-- ── 总准确率(强调行)── -->
|
|
37
|
+
<text x="240" y="176" font-size="21" font-weight="700" fill="#EFF3FA" text-anchor="end">总准确率</text>
|
|
38
|
+
<rect x="260" y="148" width="648" height="26" rx="6" fill="#FFD15D"/>
|
|
39
|
+
<text x="922" y="169" font-size="22" font-weight="700" fill="#FFD15D">92.6%(250/270)</text>
|
|
40
|
+
<rect x="260" y="182" width="124" height="26" rx="6" fill="#4E6DA8"/>
|
|
41
|
+
<text x="396" y="203" font-size="22" font-weight="700" fill="#8FA6C6">17.8%(48/270)</text>
|
|
42
|
+
|
|
43
|
+
<line x1="64" y1="226" x2="1136" y2="226" stroke="#263A5C" stroke-width="1.5"/>
|
|
44
|
+
|
|
45
|
+
<!-- ── 分题型行 ── -->
|
|
46
|
+
<!-- 抽取 -->
|
|
47
|
+
<text x="240" y="286" font-size="19" fill="#C2CDDE" text-anchor="end">抽取</text>
|
|
48
|
+
<rect x="260" y="254" width="700" height="18" rx="5" fill="#FFD15D"/>
|
|
49
|
+
<text x="974" y="269" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">45/45</text>
|
|
50
|
+
<rect x="260" y="278" width="47" height="18" rx="5" fill="#4E6DA8"/>
|
|
51
|
+
<text x="319" y="293" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">3/45</text>
|
|
52
|
+
|
|
53
|
+
<!-- 多跳 -->
|
|
54
|
+
<text x="240" y="348" font-size="19" fill="#C2CDDE" text-anchor="end">多跳</text>
|
|
55
|
+
<rect x="260" y="316" width="700" height="18" rx="5" fill="#FFD15D"/>
|
|
56
|
+
<text x="974" y="331" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">45/45</text>
|
|
57
|
+
<text x="272" y="355" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">0/45</text>
|
|
58
|
+
|
|
59
|
+
<!-- 时序 -->
|
|
60
|
+
<text x="240" y="410" font-size="19" fill="#C2CDDE" text-anchor="end">时序</text>
|
|
61
|
+
<rect x="260" y="378" width="669" height="18" rx="5" fill="#FFD15D"/>
|
|
62
|
+
<text x="943" y="393" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">43/45</text>
|
|
63
|
+
<text x="272" y="417" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">0/45</text>
|
|
64
|
+
|
|
65
|
+
<!-- 知识更新 -->
|
|
66
|
+
<text x="240" y="472" font-size="19" fill="#C2CDDE" text-anchor="end">知识更新 *</text>
|
|
67
|
+
<rect x="260" y="440" width="482" height="18" rx="5" fill="#FFD15D"/>
|
|
68
|
+
<text x="756" y="455" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">31/45</text>
|
|
69
|
+
<text x="272" y="479" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">0/45</text>
|
|
70
|
+
|
|
71
|
+
<!-- 场景回忆 -->
|
|
72
|
+
<text x="240" y="534" font-size="19" fill="#C2CDDE" text-anchor="end">场景回忆</text>
|
|
73
|
+
<rect x="260" y="502" width="638" height="18" rx="5" fill="#FFD15D"/>
|
|
74
|
+
<text x="912" y="517" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">41/45</text>
|
|
75
|
+
<text x="272" y="541" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">0/45</text>
|
|
76
|
+
|
|
77
|
+
<!-- 拒答 -->
|
|
78
|
+
<text x="240" y="596" font-size="19" fill="#C2CDDE" text-anchor="end">拒答 *</text>
|
|
79
|
+
<rect x="260" y="564" width="700" height="18" rx="5" fill="#FFD15D"/>
|
|
80
|
+
<text x="974" y="579" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">45/45</text>
|
|
81
|
+
<rect x="260" y="588" width="700" height="18" rx="5" fill="#4E6DA8"/>
|
|
82
|
+
<text x="974" y="603" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">45/45</text>
|
|
83
|
+
|
|
84
|
+
<!-- ── 脚注 ── -->
|
|
85
|
+
<text x="64" y="672" font-size="16" fill="#8296B3">* 知识更新:改口后仍答旧值记 0 分(直接考核 L1 去重更新);拒答:没发生过的事编造即 0 分(两组均 0 编造)。</text>
|
|
86
|
+
</g>
|
|
87
|
+
</svg>
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg"
|
|
2
|
+
width="1200" height="660" viewBox="0 0 1200 660"
|
|
3
|
+
role="img" aria-labelledby="benchWorkflowTitle benchWorkflowDesc">
|
|
4
|
+
<title id="benchWorkflowTitle">DSH-MemBench 工作流赛道:A/B 两组完成度与成本对照</title>
|
|
5
|
+
<desc id="benchWorkflowDesc">真实工具沙箱(4 场景 × 3 次)探针会话合计:任务完成度 A 组 24/33(72.7%)对 B 组 11/33(33.3%);成本对比以 B 组为满格基准,A 组相对更省——步骤 125 对 186(B +49%)、工具调用 184 对 296(B +61%)、输入 token 1.30M 对 1.86M(B +43%);向用户求助 A 0 次对 B 3 次;登录场景输入 token A 241k 对 B 453k(+88%)。</desc>
|
|
6
|
+
|
|
7
|
+
<rect width="1200" height="660" rx="26" fill="#0D1526"/>
|
|
8
|
+
|
|
9
|
+
<g font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','PingFang SC','Microsoft YaHei',sans-serif">
|
|
10
|
+
|
|
11
|
+
<!-- ── 标题与图例 ── -->
|
|
12
|
+
<text x="64" y="66" font-size="26" font-weight="700" fill="#EFF3FA">工作流赛道:真实工具沙箱 · 做得对,也更省</text>
|
|
13
|
+
<text x="64" y="94" font-size="18" fill="#8296B3">4 场景 × 3 次 · 探针会话合计 · 成本行以 B 组为满格基准,蓝徽章 = B 组多付</text>
|
|
14
|
+
|
|
15
|
+
<rect x="842" y="50" width="18" height="18" rx="5" fill="#FFD15D"/>
|
|
16
|
+
<text x="868" y="65" font-size="18" fill="#C2CDDE">A 组 · 记忆开</text>
|
|
17
|
+
<rect x="842" y="76" width="18" height="18" rx="5" fill="#4E6DA8"/>
|
|
18
|
+
<text x="868" y="91" font-size="18" fill="#C2CDDE">B 组 · 记忆关</text>
|
|
19
|
+
|
|
20
|
+
<!-- ── 任务完成度 ── -->
|
|
21
|
+
<text x="64" y="152" font-size="20" font-weight="700" fill="#EFF3FA">任务完成度</text>
|
|
22
|
+
<text x="186" y="152" font-size="17" fill="#8296B3">(产物程序化校验)</text>
|
|
23
|
+
|
|
24
|
+
<rect x="260" y="170" width="509" height="26" rx="6" fill="#FFD15D"/>
|
|
25
|
+
<text x="783" y="191" font-size="22" font-weight="700" fill="#FFD15D">24/33 · 72.7%</text>
|
|
26
|
+
<rect x="260" y="206" width="233" height="26" rx="6" fill="#4E6DA8"/>
|
|
27
|
+
<text x="505" y="227" font-size="22" font-weight="700" fill="#8FA6C6">11/33 · 33.3%</text>
|
|
28
|
+
|
|
29
|
+
<line x1="64" y1="258" x2="1136" y2="258" stroke="#263A5C" stroke-width="1.5"/>
|
|
30
|
+
|
|
31
|
+
<!-- ── 成本对比(B = 满格基准)── -->
|
|
32
|
+
<text x="64" y="290" font-size="20" font-weight="700" fill="#EFF3FA">成本对比</text>
|
|
33
|
+
<text x="166" y="290" font-size="17" fill="#8296B3">(B 组 = 满格基准,条越短越省)</text>
|
|
34
|
+
|
|
35
|
+
<!-- 步骤 -->
|
|
36
|
+
<text x="240" y="348" font-size="19" fill="#C2CDDE" text-anchor="end">步骤</text>
|
|
37
|
+
<rect x="260" y="318" width="376" height="18" rx="5" fill="#FFD15D"/>
|
|
38
|
+
<text x="648" y="333" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">125</text>
|
|
39
|
+
<rect x="260" y="342" width="560" height="18" rx="5" fill="#4E6DA8"/>
|
|
40
|
+
<text x="834" y="357" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">186</text>
|
|
41
|
+
<rect x="980" y="321" width="136" height="36" rx="10" fill="#315DA6"/>
|
|
42
|
+
<text x="1048" y="345" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +49%</text>
|
|
43
|
+
|
|
44
|
+
<!-- 工具调用 -->
|
|
45
|
+
<text x="240" y="430" font-size="19" fill="#C2CDDE" text-anchor="end">工具调用</text>
|
|
46
|
+
<rect x="260" y="400" width="348" height="18" rx="5" fill="#FFD15D"/>
|
|
47
|
+
<text x="620" y="415" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">184</text>
|
|
48
|
+
<rect x="260" y="424" width="560" height="18" rx="5" fill="#4E6DA8"/>
|
|
49
|
+
<text x="834" y="439" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">296</text>
|
|
50
|
+
<rect x="980" y="403" width="136" height="36" rx="10" fill="#315DA6"/>
|
|
51
|
+
<text x="1048" y="427" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +61%</text>
|
|
52
|
+
|
|
53
|
+
<!-- 输入 token -->
|
|
54
|
+
<text x="240" y="512" font-size="19" fill="#C2CDDE" text-anchor="end">输入 token</text>
|
|
55
|
+
<rect x="260" y="482" width="391" height="18" rx="5" fill="#FFD15D"/>
|
|
56
|
+
<text x="663" y="497" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">1.30M</text>
|
|
57
|
+
<rect x="260" y="506" width="560" height="18" rx="5" fill="#4E6DA8"/>
|
|
58
|
+
<text x="834" y="521" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">1.86M</text>
|
|
59
|
+
<rect x="980" y="485" width="136" height="36" rx="10" fill="#315DA6"/>
|
|
60
|
+
<text x="1048" y="509" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +43%</text>
|
|
61
|
+
|
|
62
|
+
<!-- ── 底部特写卡片 ── -->
|
|
63
|
+
<rect x="64" y="544" width="512" height="72" rx="14" fill="#15223A" stroke="#263A5C" stroke-width="1.5"/>
|
|
64
|
+
<text x="88" y="574" font-size="17" fill="#8296B3">向用户求助(探针会话)</text>
|
|
65
|
+
<text x="88" y="600" font-size="20" font-weight="700">
|
|
66
|
+
<tspan fill="#FFD15D">A 0 次</tspan>
|
|
67
|
+
<tspan fill="#8FA6C6"> · B 3 次(每轮要凭据)</tspan>
|
|
68
|
+
</text>
|
|
69
|
+
|
|
70
|
+
<rect x="624" y="544" width="512" height="72" rx="14" fill="#15223A" stroke="#263A5C" stroke-width="1.5"/>
|
|
71
|
+
<text x="648" y="574" font-size="17" fill="#8296B3">登录场景 · 输入 token(凭据只存在记忆里)</text>
|
|
72
|
+
<text x="648" y="600" font-size="20" font-weight="700">
|
|
73
|
+
<tspan fill="#FFD15D">A 241k</tspan>
|
|
74
|
+
<tspan fill="#8FA6C6"> · B 453k</tspan>
|
|
75
|
+
</text>
|
|
76
|
+
<rect x="1040" y="578" width="72" height="26" rx="8" fill="#315DA6"/>
|
|
77
|
+
<text x="1076" y="597" font-size="18" font-weight="700" fill="#FFD15D" text-anchor="middle">+88%</text>
|
|
78
|
+
</g>
|
|
79
|
+
</svg>
|