dsh-layered-memory 0.8.3 → 0.8.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +38 -17
- package/README.md +34 -17
- package/assets/img/ui-dark.jpg +0 -0
- package/assets/img/ui-light.jpg +0 -0
- package/assets/readme/bench-dialog.svg +53 -70
- package/assets/readme/bench-workflow.svg +29 -29
- package/dist/bench-control.d.ts +35 -0
- package/dist/bench-control.js +16 -0
- package/dist/client.js +205 -29
- package/dist/config.d.ts +12 -0
- package/dist/config.js +9 -1
- package/dist/hooks/recall.d.ts +23 -0
- package/dist/hooks/recall.js +36 -9
- package/dist/index.d.ts +2 -0
- package/dist/index.js +16 -1
- package/dist/llm-usage.d.ts +27 -0
- package/dist/llm-usage.js +39 -0
- package/dist/llm.d.ts +11 -2
- package/dist/llm.js +20 -6
- package/dist/pipeline/l1.js +4 -2
- package/dist/pipeline/l2.js +1 -0
- package/dist/pipeline/l3.js +1 -0
- package/dist/pipeline/runner.d.ts +16 -0
- package/dist/pipeline/runner.js +60 -0
- package/dist/prompts/l1-extraction.d.ts +7 -1
- package/dist/prompts/l1-extraction.js +12 -3
- package/dist/settings.d.ts +3 -3
- package/dist/settings.js +3 -3
- package/dist/stats.d.ts +27 -1
- package/dist/stats.js +42 -5
- package/dist/store/l0.d.ts +2 -0
- package/dist/store/l0.js +4 -0
- package/dist/store/sqlite.d.ts +2 -0
- package/dist/store/sqlite.js +14 -0
- package/dist/types.d.ts +8 -0
- package/dist/types.js +8 -0
- package/package.json +17 -17
package/README.en.md
CHANGED
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
[简体中文](README.md) · [Latest release](https://github.com/JunNanLYS/dsh-layered-memory/releases/latest) · [Report issues](https://github.com/JunNanLYS/dsh-layered-memory/issues)
|
|
11
11
|
|
|
12
12
|
[](https://www.npmjs.com/package/dsh-layered-memory)
|
|
13
|
-
[](https://github.com/deepseek-ai/deepseek-harness)
|
|
14
14
|
[](LICENSE)
|
|
15
15
|
|
|
16
16
|
</div>
|
|
@@ -21,7 +21,7 @@ Requires Node ≥ 22.16. Two invocation styles — the `npx` prefix can replace
|
|
|
21
21
|
any command below:
|
|
22
22
|
|
|
23
23
|
```bash
|
|
24
|
-
# Option 1: run the official CLI directly via npx (no pre-installed dsh; version can be pinned, e.g. dsh-layered-memory@0.8.
|
|
24
|
+
# Option 1: run the official CLI directly via npx (no pre-installed dsh; version can be pinned, e.g. dsh-layered-memory@0.8.4)
|
|
25
25
|
npx -y @deepseek-ai/dsh plugin --profile web add dsh-layered-memory
|
|
26
26
|
|
|
27
27
|
# Option 2: with the dsh CLI installed (dsh is a pnpm forwarder; npm i -g pnpm first if missing)
|
|
@@ -129,6 +129,15 @@ trajectory view):
|
|
|
129
129
|
- **Control**: the pill next to the mode selector in the input bar (`Memory · Auto`);
|
|
130
130
|
clicking opens a macOS-style sliding picker above — release to snap to the nearest
|
|
131
131
|
mode; adapts to light/dark themes;
|
|
132
|
+
- The lower half of the popover is a **per-session info area**: recall hits
|
|
133
|
+
(hit/searched turns plus cumulative items), batching progress (this session's
|
|
134
|
+
slice x/effective threshold; the off mode shows parked slices instead), memories
|
|
135
|
+
produced for this session, and session message count — plus status lines for
|
|
136
|
+
anomalies (storage degraded / vector search unavailable) and a global summary
|
|
137
|
+
(pending distill count, last distill time). Data comes from the
|
|
138
|
+
`dsh-memory/session-stats` endpoint (in-memory registries + an indexed COUNT,
|
|
139
|
+
zero file I/O), adaptively polled while open (2s busy / 5s idle) and stopped on
|
|
140
|
+
close;
|
|
132
141
|
- Each session's choice is persisted by sessionId to `session-modes.json`, surviving
|
|
133
142
|
restarts/session restore; stacks with the global switches (global is the master gate);
|
|
134
143
|
L2/L3 are fully family-isolated — content never leaks across families.
|
|
@@ -144,39 +153,50 @@ trajectory view):
|
|
|
144
153
|
|
|
145
154
|
## Measured Comparison (DSH-MemBench: Automated Benchmark)
|
|
146
155
|
|
|
147
|
-
Screenshots show what the plugin looks like — this section answers "**what does enabling it actually buy you?**" with measured numbers from an **automated benchmark** ([`bench/`](./bench/), one command to reproduce). Method: the same scenario bank with verbatim-identical inputs runs in **Group A (memory on)** and **Group B (memory off)
|
|
156
|
+
Screenshots show what the plugin looks like — this section answers "**what does enabling it actually buy you?**" with measured numbers from an **automated benchmark** ([`bench/`](./bench/), one command to reproduce). Method: the same scenario bank with verbatim-identical inputs runs in **Group A (memory on)** with 3 merged repetitions and **Group B (memory off)** with 1 repetition (a memory-off long task burns multiples of the tokens per scenario — a cost guardrail); the dialog track now runs Group A only (memory-off probes in independent sessions cannot succeed, so the control carries no information — retired). Dialog-track environment: DeepSeek official `deepseek-v4-flash`, plugin 0.8.5 (judge same-source as tested; every answer archived for manual audit), Windows; taxonomy adapted from [LongMemEval](https://github.com/xiaowu0162/longmemeval) / [LoCoMo](https://snap-research.github.io/locomo/) / [AMB](https://github.com/vectorize-io/agent-memory-benchmark), with the extended probe types and lifecycle track informed by [MemoryAgentBench](https://arxiv.org/abs/2507.05257) / [GoodAI LTM](https://github.com/GoodAI/goodai-ltm-benchmark) / BEAM.
|
|
148
157
|
|
|
149
|
-
|
|
158
|
+
> The dialog track below is the **fresh 0.8.5 baseline** (fixed plugin + corrected judging criteria); the workflow-track numbers remain the archived 0.8.3 run (the bank has since grown to 8 scenarios with a prospective-memory addition — re-run pending).
|
|
159
|
+
|
|
160
|
+
### Dialog track (20 scenarios × 10 probe types × 3 reps = 420 questions): does it remember correctly
|
|
161
|
+
|
|
162
|
+
> 0.8.5 baseline (Group A data; the dialog-track B arm is retired — Group A only).
|
|
150
163
|
|
|
151
164
|
<p align="center">
|
|
152
165
|
<img src="./assets/readme/bench-dialog.svg" width="100%"
|
|
153
|
-
alt="DSH-MemBench dialog track
|
|
166
|
+
alt="DSH-MemBench dialog track accuracy chart (Group A, memory on): overall accuracy 95.2% (400/420); six core probe types, 60 questions each — extraction 58/60, multi-hop 60/60, temporal 56/60, updates 55/60, scene recall 52/60, abstention 60/60 with 0 fabricated; four extended probe types, 15 each — accretive completion 15/15, update chains 15/15, event ordering 14/15, paraphrase 15/15">
|
|
154
167
|
</p>
|
|
155
168
|
|
|
156
|
-
**Dual-channel recall** (Group A): passive injection hit rate **
|
|
169
|
+
**Dual-channel recall** (Group A): passive injection hit rate **78.1%** (the answer's key points appear in the recall injection, 281/360); most of the rest the model recovered by **actively calling the memory tools** — 106 questions with active queries, **75 rescued by tools**. The end-to-end 95.2% is the composite of both channels plus model utilization. With the memory store accumulating across scenarios for the whole run, 295 probe injections carried other scenarios' memories (honestly counted) — yet accuracy actually *rose* from 92.8% (early, small store) to 97.7% (late, largest store), and offline flooding with 600 extra synthetic records moved retrieval recall@5 by only −2.8pp: interference resistance under a growing store, measured.
|
|
157
170
|
|
|
158
|
-
|
|
171
|
+
**Layered weaknesses**: offline retrieval metrics (recall@5, controlled replay) total 73.3%, with event ordering at 0% and scene recall at 50% — end-to-end still 93%+ thanks to model robustness over adjacent injected memories. **Efficiency triangle** (the cost of memory): injections add no latency (injected turns respond 210ms *faster* on average), recall text is ~10.3% of per-turn input, and the whole distillation pipeline costs ≈2727 input / 240 output tokens per captured message (1172 calls, 0 failures).
|
|
172
|
+
|
|
173
|
+
### Workflow track (archived 0.8.3 · 7-scenario edition · Group A ×3 / Group B ×1, real tool sandbox): does it do it right, and cheaper
|
|
159
174
|
|
|
160
175
|
<p align="center">
|
|
161
176
|
<img src="./assets/readme/bench-workflow.svg" width="100%"
|
|
162
|
-
alt="DSH-MemBench workflow track, Group A vs Group B:
|
|
177
|
+
alt="DSH-MemBench workflow track, Group A vs Group B: probe-phase completion A 59/69 (85.5%) vs B 10/23 (43.5%); cost comparison (Group B as the full-bar baseline, per-scenario means) — steps 24.3 vs 41.4 (B +70%), tool calls 37.7 vs 62.1 (B +65%), input tokens 266k vs 1.81M (B 6.8×); style-convention scenario probes A 12/12 vs B 0/4; long-task input tokens per scenario A 266k vs B 1.81M">
|
|
163
178
|
</p>
|
|
164
179
|
|
|
165
|
-
**
|
|
180
|
+
**Probe-phase completion 85.5% vs 43.5% (+42pp)**: both groups have live context during teach/change phases — the probe phase (continuation task in a fresh session) is the pure memory window. Group A scored a perfect 12/12 on all three new probe archetypes (workflow knowledge update / twin-runbook disambiguation / style-convention continuity), consistent across all three reps; Group B scored **0/4** on style-convention probes (naming/structure/thousands-separator/footer conventions exist only in memory — they cannot be explored out of the sandbox), while on the workflow-update scenario it can reverse-engineer the procedure by reading the script (discrimination limited by sandbox affordances, honestly noted).
|
|
181
|
+
|
|
182
|
+
**Long-task cost: Group B burns 6.8× Group A's input tokens per scenario** (1.81M vs 266k) — without memory the agent advances by re-exploring, and under a high reasoning effort it even builds its own projects to probe what a one-line script convention would have done; output tokens 3× (46.2k vs 15.4k), steps +70%. This is memory's core value: **what it saves is not task difficulty, but pointless round-trips and re-exploration**.
|
|
166
183
|
|
|
167
184
|
### Methodology & reproduction
|
|
168
185
|
|
|
169
186
|
```bash
|
|
170
|
-
node bench/harness/run.mjs --arm A --repeats 3 --provider deepseek-official --model deepseek-v4-flash
|
|
171
|
-
node bench/harness/run.mjs --arm
|
|
172
|
-
node bench/harness/run.mjs --track
|
|
187
|
+
node bench/harness/run.mjs --arm A --repeats 3 --provider deepseek-official --model deepseek-v4-flash # dialog track (Group A only)
|
|
188
|
+
node bench/harness/run.mjs --track workflow --arm AB --repeats 3 ... # workflow track (A/B arms in parallel)
|
|
189
|
+
node bench/harness/run.mjs --track lifecycle --arm A ... # lifecycle track (gating/off/rebuild/forget)
|
|
173
190
|
node bench/harness/report.mjs --latest [dialog|workflow] # aggregate report
|
|
191
|
+
node bench/harness/retrieval-metrics.mjs <runDir> --flood 200,600 # retrieval metrics + flooding curve
|
|
174
192
|
```
|
|
175
193
|
|
|
176
|
-
- Scoring: programmatic `contains-all` plus an LLM judge against key points (every answer and verdict is preserved in `result.json` for human audit); workflow completion is verified programmatically from produced files and their contents;
|
|
177
|
-
-
|
|
178
|
-
-
|
|
179
|
-
-
|
|
194
|
+
- Scoring: programmatic `contains-all` plus an LLM judge against key points (every answer and verdict is preserved in `result.json` for human audit); for stale-bearing probes (updates/update-chains/forget) an old value only fails when stated *as the current answer*, and abstention probes allow citing real adjacent facts while denying the asked point; workflow completion is verified programmatically from produced files and their contents (four check kinds: positive / forbidden-word / must-not-exist / exists);
|
|
195
|
+
- Metric surface: beyond the per-type accuracy table (6 core + 4 extended types), reports automatically include **offline retrieval metrics** (recall@5 / injection precision / stale leakage), the **efficiency triangle** (injection latency differential / injection share / distillation accounting per captured message), **scale-position analysis** (accuracy & contamination vs store growth), and the lifecycle-track section (family-gating matrix / off-mode dual assertions / rebuild fidelity / forget requests);
|
|
196
|
+
- Live progress: running the benchmark auto-starts a local progress panel and opens the browser (`--no-panel` to disable) — per-arm scenario/phase/message-level progress, heartbeat & activity freshness (distinguishes "stuck" from "process died"), and cumulative cost as it accrues;
|
|
197
|
+
- Metrics come from provider-reported usage (input with cache-hit split) and session-event folding; the steady-state cache rate excludes each session's first request (0.8.5 baseline: 89.1% — memory injection does not hurt caching);
|
|
198
|
+
- Regression use: run before/after a plugin change and diff with `compare.mjs` (environment header check including git SHA + Group-B control-drift warning + retrieval-metric comparison);
|
|
199
|
+
- Limitations (stated honestly): single machine; Group A ×3 merged, Group B ×1 (cost guardrail — noisier); judge vs tested model: same model in the 0.8.5 dialog baseline, heterogeneous in the archived workflow run (glm-5.3 judging v4-flash); the scenario bank is author-built (biased toward memory-advantage scenarios — reproduce it yourself); sandbox-file affordances partially leak procedures (Group B can reverse-engineer by reading scripts — discrimination limits honestly noted); dual-tier tool audit (strict violation voids the scenario / loose heuristic flags only), with 0 violations measured on both sides.
|
|
180
200
|
|
|
181
201
|
Full reports and per-question data: [`bench/baseline/`](./bench/baseline/).
|
|
182
202
|
|
|
@@ -235,11 +255,12 @@ the bundle layer appends and causes `duplicate loader entry id` startup failure)
|
|
|
235
255
|
| `embedding.proxy` | `''` | Three-state download proxy: `''` (default) = auto-detect proxy env vars (`HTTPS_PROXY`/`ALL_PROXY` etc., honoring `NO_PROXY`); `none` = disable, always direct; any other value = proxy URL (e.g. `http://127.0.0.1:7890`). Direct connections to the mirror are intermittently unreachable on some networks (connect timeouts and poisoned bytes have both been observed) — keep the default auto-detection on machines with a proxy |
|
|
236
256
|
| `llm.provider/model` | empty | Static distillation route (deployment pin): when **both** fields are set the route is locked, outranking the settings-page selection and the default model (deployments can force distillation onto a specific route); when empty the route follows "settings-page selection → default model". At runtime, switch among **configured providers** (including custom ones added in dsh Settings → Models) via the "distillation model" picker in Settings → Memory → Overview — effective immediately, no restart needed |
|
|
237
257
|
| `llm.maxTokens` | `65536` | Fallback output cap for non-layered calls. Each distillation stage has its own budget (extraction 16k / dedup 8k / L2 32k / L3 16k; auto ×4 when the reasoning effort is high/xhigh/max, so thinking can't starve the text budget); the per-layer budgets are runtime-adjustable in Settings → Memory → Overview → distillation parameters (empty/0 = built-in defaults) |
|
|
238
|
-
| `llm.reasoningEffort` | empty | Distillation reasoning effort: empty = **auto** (resolved from model capability: the model's default tier, else `high`); an explicit value (`off`/`none`/`low`/`medium`/`high`/`xhigh`/`max`) is only sent when the model declares support — effort vocabularies differ across providers (deepseek accepts `off`, OpenAI-style APIs use `none`, models that declare no tiers get nothing), and unsupported tiers degrade to not-sending with a one-time warning; output budgets auto-×4 at high/xhigh/max. Switchable at runtime in Settings → Memory → Overview; the selectable tiers follow the current model live |
|
|
258
|
+
| `llm.reasoningEffort` | empty | Distillation reasoning effort: empty = **auto** (resolved from model capability: the model's default tier, else `high`); an explicit value (`off`/`none`/`minimal`/`low`/`medium`/`high`/`xhigh`/`max`) is only sent when the model declares support — effort vocabularies differ across providers (deepseek accepts `off`, OpenAI-style APIs use `none`, models that declare no tiers get nothing), and unsupported tiers degrade to not-sending with a one-time warning; output budgets auto-×4 at high/xhigh/max. Switchable at runtime in Settings → Memory → Overview; the selectable tiers follow the current model live |
|
|
239
259
|
| `llm.temperature` | `0.3` | Distillation temperature |
|
|
240
260
|
| `llm.maxInputChars` | `700000` | Input character budget per distillation call (over-budget L1 inputs are chunked automatically); runtime-adjustable in Settings → distillation parameters → input budget (empty/0 = follow this value) |
|
|
241
261
|
| `llm.timeoutMs` | `120000` | Per-call distillation timeout (ms) |
|
|
242
262
|
| `tools` | `true` | Whether to register model-callable memory tools |
|
|
263
|
+
| `benchControl` | `false` | Register the in-process bench control service (rebuild trigger / session-mode setting / distillation usage snapshot — used by the benchmark's lifecycle track). Off by default — zero surface in production deployments; do not enable casually |
|
|
243
264
|
|
|
244
265
|
## Storage Layout
|
|
245
266
|
|
package/README.md
CHANGED
|
@@ -10,7 +10,7 @@
|
|
|
10
10
|
[English](README.en.md) · [最新发行版](https://github.com/JunNanLYS/dsh-layered-memory/releases/latest) · [反馈问题](https://github.com/JunNanLYS/dsh-layered-memory/issues)
|
|
11
11
|
|
|
12
12
|
[](https://www.npmjs.com/package/dsh-layered-memory)
|
|
13
|
-
[](https://github.com/deepseek-ai/deepseek-harness)
|
|
14
14
|
[](LICENSE)
|
|
15
15
|
|
|
16
16
|
</div>
|
|
@@ -20,7 +20,7 @@
|
|
|
20
20
|
需要 Node ≥ 22.16。两种调用方式任选(`npx` 前缀可替换下面任何 `dsh` 命令):
|
|
21
21
|
|
|
22
22
|
```bash
|
|
23
|
-
# 方式一:npx 直接跑官方 CLI(无需预装 dsh;可 pin 版本,如 dsh-layered-memory@0.8.
|
|
23
|
+
# 方式一:npx 直接跑官方 CLI(无需预装 dsh;可 pin 版本,如 dsh-layered-memory@0.8.4)
|
|
24
24
|
npx -y @deepseek-ai/dsh plugin --profile web add dsh-layered-memory
|
|
25
25
|
|
|
26
26
|
# 方式二:已装 dsh CLI(dsh 是 pnpm 转发器,未装 pnpm 时先 npm i -g pnpm)
|
|
@@ -110,6 +110,11 @@ npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution no
|
|
|
110
110
|
</p>
|
|
111
111
|
|
|
112
112
|
- **控件**:输入栏内、模式选择器右侧的 pill(`记忆·自动`),点击在上方浮出档位滑块深浅主题自适应;
|
|
113
|
+
- 悬浮板下半部是**会话信息区**:召回命中(命中/检索轮次与累计条数)、攒批进度
|
|
114
|
+
(本会话切片 x/生效阈值;关闭档显示挂起切片数)、本会话产出记忆条数、会话消息数,
|
|
115
|
+
外加异常状态行(存储降级 / 向量检索不可用)与全局摘要(待蒸馏条数、上次蒸馏时间);
|
|
116
|
+
数据走 `dsh-memory/session-stats` 端点(纯内存注册表 + 索引 COUNT,零文件 I/O),
|
|
117
|
+
打开期间自适应轮询(忙 2s / 静 5s),关闭即停;
|
|
113
118
|
- 每会话的选择按 sessionId 持久化到 `session-modes.json`,重启/恢复会话不丢;
|
|
114
119
|
与全局开关叠加(全局是总闸);L2/L3 完全分类,分类内容不渗透。
|
|
115
120
|
|
|
@@ -124,39 +129,50 @@ npx tsc src/smoke.ts --outDir dist-smoke --module nodenext --moduleResolution no
|
|
|
124
129
|
|
|
125
130
|
## 实测对比(DSH-MemBench:自动化基准)
|
|
126
131
|
|
|
127
|
-
图文回答"长什么样",这一节用**自动化基准**的实测数字回答"**开了到底有什么用**"([`bench/`](./bench/),一条命令可复现)。方法:同场景库、逐字相同输入,**A
|
|
132
|
+
图文回答"长什么样",这一节用**自动化基准**的实测数字回答"**开了到底有什么用**"([`bench/`](./bench/),一条命令可复现)。方法:同场景库、逐字相同输入,**A 组(记忆开)跑 3 次取合并值,B 组(记忆关)跑 1 次**(无记忆的长任务每场景要吞数倍 token,成本护栏);对话赛道只跑 A 组(B 组会话独立无记忆必然失败,对照无信息量,已下线)。对话赛道环境:DeepSeek 官方 `deepseek-v4-flash`、插件 0.8.5(判卷与被测同源,答案原文全部留痕可人工复核)、Windows;题型设计借鉴 [LongMemEval](https://github.com/xiaowu0162/longmemeval) / [LoCoMo](https://snap-research.github.io/locomo/) / [AMB](https://github.com/vectorize-io/agent-memory-benchmark),扩展题型与生命周期赛道参照 [MemoryAgentBench](https://arxiv.org/abs/2507.05257) / [GoodAI LTM](https://github.com/GoodAI/goodai-ltm-benchmark) / BEAM。
|
|
128
133
|
|
|
129
|
-
|
|
134
|
+
> 对话赛道为 **0.8.5 新基线**(修复版插件 + 修正后的判卷口径);工作流赛道数字仍为 0.8.3 存档(0.8.5 起场景库扩至 8 个,新增前瞻记忆场景,重跑待做)。
|
|
135
|
+
|
|
136
|
+
### 对话赛道(20 场景 × 10 题型 × 3 次 = 420 题):答得准吗
|
|
137
|
+
|
|
138
|
+
> 0.8.5 基线(A 组数据;对话赛道 B 组已下线,只跑 A 组)。
|
|
130
139
|
|
|
131
140
|
<p align="center">
|
|
132
141
|
<img src="./assets/readme/bench-dialog.svg" width="100%"
|
|
133
|
-
alt="DSH-MemBench
|
|
142
|
+
alt="DSH-MemBench 对话赛道准确率图(A 组·记忆开):总准确率 95.2%(400/420);核心六题型各 60 题——抽取 58/60、多跳 60/60、时序 56/60、更新 55/60、场景回忆 52/60、拒答 60/60 且 0 编造;扩展四题型各 15 题——增量积累 15/15、连锁更新 15/15、事件排序 14/15、同义改写 15/15">
|
|
134
143
|
</p>
|
|
135
144
|
|
|
136
|
-
**召回双通道**(A 组):被动注入召回率 **
|
|
145
|
+
**召回双通道**(A 组):被动注入召回率 **78.1%**(该题要点出现在召回注入中,281/360),其余多数由模型**主动调用记忆工具**查回——106 题主动查询、**75 题靠工具兜底答对**;端到端 95.2% 是两通道 + 模型利用的合成结果。记忆库跨场景全程累积下,探针召回注入混入其他场景记忆 295 次(已如实计数),总准确率反而前段 92.8% → 后段 97.7%——抗干扰能力经受住了膨胀记忆库的考验(离线灌水再灌 600 条合成噪声,检索层 recall@5 也只降 2.8pp)。
|
|
137
146
|
|
|
138
|
-
|
|
147
|
+
**分层看短板**:检索层离线指标(recall@5 受控复现)总 73.3%,其中事件排序 0%、场景回忆 50%——端到端仍 93%+ 靠的是注入邻近记忆后模型的鲁棒性;**效率三角**(记忆的开销):注入非但不加延迟(注入轮响应比无注入轮平均快 210ms)、注入占每轮输入约 10.3%,蒸馏全链路摊到每条捕获消息 ≈2727 输入 / 240 输出 token(1172 次调用 0 失败)。
|
|
148
|
+
|
|
149
|
+
### 工作流赛道(0.8.3 存档 · 7 场景版 · A 组 3 次 / B 组 1 次,真实工具沙箱):做得对、做得省吗
|
|
139
150
|
|
|
140
151
|
<p align="center">
|
|
141
152
|
<img src="./assets/readme/bench-workflow.svg" width="100%"
|
|
142
|
-
alt="DSH-MemBench 工作流赛道 A/B
|
|
153
|
+
alt="DSH-MemBench 工作流赛道 A/B 对照图:探针段完成度 A 组 59/69(85.5%)对 B 组 10/23(43.5%);成本对比(B 组为满格基准,每场景均值)——步骤 24.3 对 41.4(B +70%)、工具调用 37.7 对 62.1(B +65%)、输入 token 266k 对 1.81M(B 6.8 倍);风格规范场景探针 A 12/12 对 B 0/4;长任务每场景输入 token A 266k 对 B 1.81M">
|
|
143
154
|
</p>
|
|
144
155
|
|
|
145
|
-
|
|
156
|
+
**探针段完成度 85.5% vs 43.5%(+42pp)**:教学/变更段两组都有现场上下文,探针段(新会话延续任务)才是纯记忆窗口——A 组三个新考法场景(流程知识更新 / 双胞胎消歧 / 风格规范延续)全部 12/12 满分且三轮一致;B 组在风格规范场景探针 **0/4**(命名/结构/千分位/页脚约定只存记忆,沙箱探不出来),在流程更新场景则能靠读脚本逆向(判别力受沙箱可供性限制,已如实标注)。
|
|
157
|
+
|
|
158
|
+
**长任务成本:B 组每场景输入 token 是 A 组的 6.8 倍**(1.81M vs 266k)——无记忆时 agent 靠重新探索前进,high 思考档下甚至会自建工程去探测本可用一条脚本约定完成的流程;输出 token 3 倍(46.2k vs 15.4k)、步骤 +70%。这正是记忆的核心价值:**省掉的不是任务难度,是无谓的往返与重复探索**。
|
|
146
159
|
|
|
147
160
|
### 方法论与复现
|
|
148
161
|
|
|
149
162
|
```bash
|
|
150
|
-
node bench/harness/run.mjs --arm A --repeats 3 --provider deepseek-official --model deepseek-v4-flash
|
|
151
|
-
node bench/harness/run.mjs --arm
|
|
152
|
-
node bench/harness/run.mjs --track
|
|
163
|
+
node bench/harness/run.mjs --arm A --repeats 3 --provider deepseek-official --model deepseek-v4-flash # 对话赛道(只跑 A 组)
|
|
164
|
+
node bench/harness/run.mjs --track workflow --arm AB --repeats 3 ... # 工作流赛道(A/B 双组并行)
|
|
165
|
+
node bench/harness/run.mjs --track lifecycle --arm A ... # 生命周期赛道(门控/off/rebuild/遗忘)
|
|
153
166
|
node bench/harness/report.mjs --latest [dialog|workflow] # 汇总报告
|
|
167
|
+
node bench/harness/retrieval-metrics.mjs <runDir> --flood 200,600 # 检索层指标 + 灌水曲线
|
|
154
168
|
```
|
|
155
169
|
|
|
156
|
-
- 判分:`contains-all` 程序判 + 判卷模型按要点判(答案原文与判分理由全部留痕 `result.json`
|
|
157
|
-
-
|
|
158
|
-
-
|
|
159
|
-
-
|
|
170
|
+
- 判分:`contains-all` 程序判 + 判卷模型按要点判(答案原文与判分理由全部留痕 `result.json` 可人工复核);带 stale 的题(更新/连锁/遗忘)"旧值当作现状陈述"才判负、拒答题允许引用真实背景解释"不知道被问点";工作流完成度为产物文件 + 关键内容程序化校验(四型判据:正检查/禁词/产物缺席/存在性);
|
|
171
|
+
- 指标面:准确率总表(6 核心 + 4 扩展题型)之外,自动产出**检索层离线指标**(recall@5 / 注入精度 / 作废泄漏)、**效率三角**(注入开销差分 / 注入占比 / 蒸馏记账摊到每消息)、**规模位置分析**(库容膨胀下的准确率/污染)与生命周期赛道专属节(分族门控矩阵 / off 双断言 / rebuild 保真 / 遗忘);
|
|
172
|
+
- 实时进度:跑基准时自动拉起本地进度面板并打开浏览器(`--no-panel` 关闭)——A/B 双臂场景/阶段/消息粒度进度、心跳与活动新鲜度(直判"卡住 vs 进程挂了")、累计成本随跑随涨;
|
|
173
|
+
- 指标全部来自供应商上报 usage(输入含缓存命中拆分)与会话事件折叠;稳态缓存率剔除每会话首请求(0.8.5 基线:89.1%——记忆注入不伤缓存);
|
|
174
|
+
- 回归用途:改插件前后各跑一遍,`compare.mjs` 出对比表(环境头校验含 gitSha + B 组对照组漂移告警 + 检索层指标对比);
|
|
175
|
+
- 局限(诚实声明):单机;A 组 ×3 合并、B 组 ×1(成本护栏,噪声更大);判卷与被测模型:对话 0.8.5 基线同源、工作流存档跑为异构(glm-5.3 判 v4-flash);作者自建场景库(倾向记忆优势场景,欢迎自行复现);沙箱文件的可供性会部分泄露流程(B 组可读脚本逆向,判别力受限处已如实标注);工具审计双档(严格违规判负/宽松提示),实测双方 0 违规。
|
|
160
176
|
|
|
161
177
|
完整报告与逐题数据:[`bench/baseline/`](./bench/baseline/)。
|
|
162
178
|
|
|
@@ -252,11 +268,12 @@ ONNX 量化 **CPU 推理**——无需 API Key,数据不出本机)。本地
|
|
|
252
268
|
| `embedding.proxy` | `''` | 模型下载代理三态:`''`(默认)= 自动探测代理环境变量(`HTTPS_PROXY`/`ALL_PROXY` 等,尊重 `NO_PROXY`);`none` = 禁用强制直连;其他值 = 代理 URL(如 `http://127.0.0.1:7890`)。镜像直连在国内网络间歇不可达(直连超时与污染字节交替出现过),开代理的机器建议保持默认自动探测 |
|
|
253
269
|
| `llm.provider/model` | 空 | 蒸馏模型静态路由(部署 pin):provider 与 model **双字段齐**时锁定蒸馏路由,优先于设置页的运行时选择与默认模型(部署可强制蒸馏走指定路由);留空则跟随"设置页选择 → 默认模型"。运行时可在设置页 → 记忆 → 概览的"蒸馏模型"选择器从**已配置的供应商**(含 dsh 设置 → 模型里添加的自定义供应商)中切换,即时生效无需重启 |
|
|
254
270
|
| `llm.maxTokens` | `65536` | 未分层调用的兜底输出总闸。各蒸馏层有独立预算(抽取 16k / 去重 8k / L2 32k / L3 16k;思考档 high/xhigh/max 时自动 ×4,防 reasoning 吃光预算),分层预算可在设置页 → 记忆 → 概览 → 蒸馏参数运行时调整(留空/0 = 跟随内置默认) |
|
|
255
|
-
| `llm.reasoningEffort` | 空 | 蒸馏思考档位:空串 = **自动**(按模型能力解析:模型默认档 → `high`);显式值(`off`/`none`/`low`/`medium`/`high`/`xhigh`/`max`)仅在该模型声明支持时发送——跨供应商 effort 词汇表不同(deepseek 认 `off`,OpenAI 系是 `none`,未声明档位的模型不传),不支持的档位自动降级为不传并告警一次;思考档 high/xhigh/max 时输出预算自动 ×4。运行时在设置页 → 记忆 → 概览切换,可选档位表跟随当前模型实时显示 |
|
|
271
|
+
| `llm.reasoningEffort` | 空 | 蒸馏思考档位:空串 = **自动**(按模型能力解析:模型默认档 → `high`);显式值(`off`/`none`/`minimal`/`low`/`medium`/`high`/`xhigh`/`max`)仅在该模型声明支持时发送——跨供应商 effort 词汇表不同(deepseek 认 `off`,OpenAI 系是 `none`,未声明档位的模型不传),不支持的档位自动降级为不传并告警一次;思考档 high/xhigh/max 时输出预算自动 ×4。运行时在设置页 → 记忆 → 概览切换,可选档位表跟随当前模型实时显示 |
|
|
256
272
|
| `llm.temperature` | `0.3` | 蒸馏温度 |
|
|
257
273
|
| `llm.maxInputChars` | `700000` | 单次蒸馏输入字符预算(超限的 L1 输入自动分块抽取);运行时可在设置页 → 蒸馏参数 → 输入预算调整(留空/0 = 跟随本值) |
|
|
258
274
|
| `llm.timeoutMs` | `120000` | 单次蒸馏调用超时(ms) |
|
|
259
275
|
| `tools` | `true` | 是否注册模型可调用的记忆工具 |
|
|
276
|
+
| `benchControl` | `false` | 注册 bench 控制服务(进程内 rebuild 触发/会话档位设置/蒸馏用量快照,供基准 lifecycle 赛道)。默认关——生产部署零表面积,勿随意开启 |
|
|
260
277
|
|
|
261
278
|
## 日志与故障排查
|
|
262
279
|
|
package/assets/img/ui-dark.jpg
CHANGED
|
Binary file
|
package/assets/img/ui-light.jpg
CHANGED
|
Binary file
|
|
@@ -1,87 +1,70 @@
|
|
|
1
1
|
<svg xmlns="http://www.w3.org/2000/svg"
|
|
2
|
-
width="1200" height="
|
|
2
|
+
width="1200" height="630" viewBox="0 0 1200 630"
|
|
3
3
|
role="img" aria-labelledby="benchDialogTitle benchDialogDesc">
|
|
4
|
-
<title id="benchDialogTitle">DSH-MemBench 对话赛道:A
|
|
5
|
-
<desc id="benchDialogDesc"
|
|
4
|
+
<title id="benchDialogTitle">DSH-MemBench 对话赛道:A 组(记忆开)准确率</title>
|
|
5
|
+
<desc id="benchDialogDesc">A 组(记忆开)总准确率 95.2%(400/420,20 场景 × 10 题型 × 3 次)。核心六题型各 60 题:抽取 58/60、多跳 60/60、时序 56/60、更新 55/60、场景回忆 52/60、拒答 60/60 且 0 编造;扩展四题型各 15 题:增量积累 15/15、连锁更新 15/15、事件排序 14/15、同义改写 15/15。对话赛道 B 组已下线(会话独立、无记忆必然失败,对照无信息量)。</desc>
|
|
6
6
|
|
|
7
|
-
<rect width="1200" height="
|
|
7
|
+
<rect width="1200" height="630" rx="26" fill="#0D1526"/>
|
|
8
8
|
|
|
9
9
|
<g font-family="-apple-system,BlinkMacSystemFont,'Segoe UI','PingFang SC','Microsoft YaHei',sans-serif">
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
<text x="64" y="
|
|
13
|
-
<text x="64" y="94" font-size="18" fill="#8296B3">15 场景 × 6 题型 × 3 次 = 270 题/组(每题型 45 题)· 条长 = 答对率,数值 = 答对题数</text>
|
|
11
|
+
<text x="64" y="66" font-size="26" font-weight="700" fill="#EFF3FA">对话赛道:记忆开 · 准确率</text>
|
|
12
|
+
<text x="64" y="94" font-size="18" fill="#8296B3">20 场景 × 10 题型 × 3 次 = 420 题(核心六题型各 60 题、扩展四题型各 15 题)· 条长 = 答对率,数值 = 答对题数</text>
|
|
14
13
|
|
|
15
14
|
<rect x="842" y="50" width="18" height="18" rx="5" fill="#FFD15D"/>
|
|
16
|
-
<text x="868" y="65" font-size="
|
|
17
|
-
<rect x="
|
|
18
|
-
<text x="
|
|
15
|
+
<text x="868" y="65" font-size="17" fill="#C2CDDE">核心六题型</text>
|
|
16
|
+
<rect x="966" y="50" width="18" height="18" rx="5" fill="#6FB1FF"/>
|
|
17
|
+
<text x="992" y="65" font-size="17" fill="#C2CDDE">扩展四题型</text>
|
|
19
18
|
|
|
20
|
-
<!-- ── 网格线(0/25/50/75/100%)── -->
|
|
21
19
|
<g stroke="#1C2A47" stroke-width="1">
|
|
22
|
-
<line x1="260" y1="
|
|
23
|
-
<line x1="435" y1="
|
|
24
|
-
<line x1="610" y1="
|
|
25
|
-
<line x1="785" y1="
|
|
26
|
-
<line x1="960" y1="
|
|
20
|
+
<line x1="260" y1="120" x2="260" y2="525"/>
|
|
21
|
+
<line x1="435" y1="120" x2="435" y2="525"/>
|
|
22
|
+
<line x1="610" y1="120" x2="610" y2="525"/>
|
|
23
|
+
<line x1="785" y1="120" x2="785" y2="525"/>
|
|
24
|
+
<line x1="960" y1="120" x2="960" y2="525"/>
|
|
27
25
|
</g>
|
|
28
26
|
<g font-size="17" fill="#8296B3" text-anchor="middle">
|
|
29
|
-
<text x="260" y="
|
|
30
|
-
<text x="435" y="
|
|
31
|
-
<text x="610" y="
|
|
32
|
-
<text x="785" y="
|
|
33
|
-
<text x="960" y="
|
|
27
|
+
<text x="260" y="552">0%</text>
|
|
28
|
+
<text x="435" y="552">25%</text>
|
|
29
|
+
<text x="610" y="552">50%</text>
|
|
30
|
+
<text x="785" y="552">75%</text>
|
|
31
|
+
<text x="960" y="552">100%</text>
|
|
34
32
|
</g>
|
|
35
33
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
34
|
+
<text x="240" y="149" font-size="18" font-weight="700" fill="#EFF3FA" text-anchor="end">总准确率</text>
|
|
35
|
+
<rect x="260" y="130" width="667" height="25" rx="6" fill="#FFD15D"/>
|
|
36
|
+
<text x="939" y="149" font-size="17" font-weight="600" fill="#FFD15D">400/420(95.2%)</text>
|
|
37
|
+
<text x="240" y="184" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">抽取</text>
|
|
38
|
+
<rect x="260" y="165" width="677" height="25" rx="6" fill="#FFD15D"/>
|
|
39
|
+
<text x="949" y="184" font-size="17" font-weight="600" fill="#FFD15D">58/60(96.7%)</text>
|
|
40
|
+
<text x="240" y="219" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">多跳</text>
|
|
41
|
+
<rect x="260" y="200" width="700" height="25" rx="6" fill="#FFD15D"/>
|
|
42
|
+
<text x="972" y="219" font-size="17" font-weight="600" fill="#FFD15D">60/60(100.0%)</text>
|
|
43
|
+
<text x="240" y="254" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">时序</text>
|
|
44
|
+
<rect x="260" y="235" width="653" height="25" rx="6" fill="#FFD15D"/>
|
|
45
|
+
<text x="925" y="254" font-size="17" font-weight="600" fill="#FFD15D">56/60(93.3%)</text>
|
|
46
|
+
<text x="240" y="289" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">更新</text>
|
|
47
|
+
<rect x="260" y="270" width="642" height="25" rx="6" fill="#FFD15D"/>
|
|
48
|
+
<text x="914" y="289" font-size="17" font-weight="600" fill="#FFD15D">55/60(91.7%)</text>
|
|
49
|
+
<text x="240" y="324" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">场景回忆</text>
|
|
50
|
+
<rect x="260" y="305" width="607" height="25" rx="6" fill="#FFD15D"/>
|
|
51
|
+
<text x="879" y="324" font-size="17" font-weight="600" fill="#FFD15D">52/60(86.7%)</text>
|
|
52
|
+
<text x="240" y="359" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">拒答(0 编造)</text>
|
|
53
|
+
<rect x="260" y="340" width="700" height="25" rx="6" fill="#FFD15D"/>
|
|
54
|
+
<text x="972" y="359" font-size="17" font-weight="600" fill="#FFD15D">60/60(100.0%)</text>
|
|
55
|
+
<text x="240" y="394" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">增量积累</text>
|
|
56
|
+
<rect x="260" y="375" width="700" height="25" rx="6" fill="#6FB1FF"/>
|
|
57
|
+
<text x="972" y="394" font-size="17" font-weight="600" fill="#6FB1FF">15/15(100.0%)</text>
|
|
58
|
+
<text x="240" y="429" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">连锁更新</text>
|
|
59
|
+
<rect x="260" y="410" width="700" height="25" rx="6" fill="#6FB1FF"/>
|
|
60
|
+
<text x="972" y="429" font-size="17" font-weight="600" fill="#6FB1FF">15/15(100.0%)</text>
|
|
61
|
+
<text x="240" y="464" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">事件排序</text>
|
|
62
|
+
<rect x="260" y="445" width="653" height="25" rx="6" fill="#6FB1FF"/>
|
|
63
|
+
<text x="925" y="464" font-size="17" font-weight="600" fill="#6FB1FF">14/15(93.3%)</text>
|
|
64
|
+
<text x="240" y="499" font-size="18" font-weight="400" fill="#C2CDDE" text-anchor="end">同义改写</text>
|
|
65
|
+
<rect x="260" y="480" width="700" height="25" rx="6" fill="#6FB1FF"/>
|
|
66
|
+
<text x="972" y="499" font-size="17" font-weight="600" fill="#6FB1FF">15/15(100.0%)</text>
|
|
42
67
|
|
|
43
|
-
<
|
|
44
|
-
|
|
45
|
-
<!-- ── 分题型行 ── -->
|
|
46
|
-
<!-- 抽取 -->
|
|
47
|
-
<text x="240" y="286" font-size="19" fill="#C2CDDE" text-anchor="end">抽取</text>
|
|
48
|
-
<rect x="260" y="254" width="700" height="18" rx="5" fill="#FFD15D"/>
|
|
49
|
-
<text x="974" y="269" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">45/45</text>
|
|
50
|
-
<rect x="260" y="278" width="47" height="18" rx="5" fill="#4E6DA8"/>
|
|
51
|
-
<text x="319" y="293" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">3/45</text>
|
|
52
|
-
|
|
53
|
-
<!-- 多跳 -->
|
|
54
|
-
<text x="240" y="348" font-size="19" fill="#C2CDDE" text-anchor="end">多跳</text>
|
|
55
|
-
<rect x="260" y="316" width="700" height="18" rx="5" fill="#FFD15D"/>
|
|
56
|
-
<text x="974" y="331" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">45/45</text>
|
|
57
|
-
<text x="272" y="355" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">0/45</text>
|
|
58
|
-
|
|
59
|
-
<!-- 时序 -->
|
|
60
|
-
<text x="240" y="410" font-size="19" fill="#C2CDDE" text-anchor="end">时序</text>
|
|
61
|
-
<rect x="260" y="378" width="669" height="18" rx="5" fill="#FFD15D"/>
|
|
62
|
-
<text x="943" y="393" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">43/45</text>
|
|
63
|
-
<text x="272" y="417" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">0/45</text>
|
|
64
|
-
|
|
65
|
-
<!-- 知识更新 -->
|
|
66
|
-
<text x="240" y="472" font-size="19" fill="#C2CDDE" text-anchor="end">知识更新 *</text>
|
|
67
|
-
<rect x="260" y="440" width="482" height="18" rx="5" fill="#FFD15D"/>
|
|
68
|
-
<text x="756" y="455" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">31/45</text>
|
|
69
|
-
<text x="272" y="479" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">0/45</text>
|
|
70
|
-
|
|
71
|
-
<!-- 场景回忆 -->
|
|
72
|
-
<text x="240" y="534" font-size="19" fill="#C2CDDE" text-anchor="end">场景回忆</text>
|
|
73
|
-
<rect x="260" y="502" width="638" height="18" rx="5" fill="#FFD15D"/>
|
|
74
|
-
<text x="912" y="517" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">41/45</text>
|
|
75
|
-
<text x="272" y="541" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">0/45</text>
|
|
76
|
-
|
|
77
|
-
<!-- 拒答 -->
|
|
78
|
-
<text x="240" y="596" font-size="19" fill="#C2CDDE" text-anchor="end">拒答 *</text>
|
|
79
|
-
<rect x="260" y="564" width="700" height="18" rx="5" fill="#FFD15D"/>
|
|
80
|
-
<text x="974" y="579" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">45/45</text>
|
|
81
|
-
<rect x="260" y="588" width="700" height="18" rx="5" fill="#4E6DA8"/>
|
|
82
|
-
<text x="974" y="603" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">45/45</text>
|
|
83
|
-
|
|
84
|
-
<!-- ── 脚注 ── -->
|
|
85
|
-
<text x="64" y="672" font-size="16" fill="#8296B3">* 知识更新:改口后仍答旧值记 0 分(直接考核 L1 去重更新);拒答:没发生过的事编造即 0 分(两组均 0 编造)。</text>
|
|
68
|
+
<text x="64" y="600" font-size="16" fill="#8296B3">DSH-MemBench 0.8.5 基线 · DeepSeek v4-flash · 3 次重复合并 · 场景库跨场景累积(越靠后的场景记忆干扰越大)</text>
|
|
86
69
|
</g>
|
|
87
70
|
</svg>
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
width="1200" height="660" viewBox="0 0 1200 660"
|
|
3
3
|
role="img" aria-labelledby="benchWorkflowTitle benchWorkflowDesc">
|
|
4
4
|
<title id="benchWorkflowTitle">DSH-MemBench 工作流赛道:A/B 两组完成度与成本对照</title>
|
|
5
|
-
<desc id="benchWorkflowDesc">真实工具沙箱(
|
|
5
|
+
<desc id="benchWorkflowDesc">真实工具沙箱(7 场景,A 组 3 轮 / B 组 1 轮)探针段合计:任务完成度 A 组 59/69(85.5%)对 B 组 10/23(43.5%);成本对比以 B 组为满格基准(每场景均值),A 组相对更省——步骤 24.3 对 41.4(B +70%)、工具调用 37.7 对 62.1(B +65%)、输入 token 266k 对 1.81M(B 6.8 倍);风格规范场景探针 A 12/12 对 B 0/4;向用户求助 A 组 3 轮共 1 次、B 组 1 轮 1 次。</desc>
|
|
6
6
|
|
|
7
7
|
<rect width="1200" height="660" rx="26" fill="#0D1526"/>
|
|
8
8
|
|
|
@@ -10,70 +10,70 @@
|
|
|
10
10
|
|
|
11
11
|
<!-- ── 标题与图例 ── -->
|
|
12
12
|
<text x="64" y="66" font-size="26" font-weight="700" fill="#EFF3FA">工作流赛道:真实工具沙箱 · 做得对,也更省</text>
|
|
13
|
-
<text x="64" y="94" font-size="18" fill="#8296B3">
|
|
13
|
+
<text x="64" y="94" font-size="18" fill="#8296B3">7 场景 · A 组 3 轮 / B 组 1 轮(成本护栏)· 探针段合计 · 成本行以 B 组为满格基准</text>
|
|
14
14
|
|
|
15
15
|
<rect x="842" y="50" width="18" height="18" rx="5" fill="#FFD15D"/>
|
|
16
16
|
<text x="868" y="65" font-size="18" fill="#C2CDDE">A 组 · 记忆开</text>
|
|
17
17
|
<rect x="842" y="76" width="18" height="18" rx="5" fill="#4E6DA8"/>
|
|
18
18
|
<text x="868" y="91" font-size="18" fill="#C2CDDE">B 组 · 记忆关</text>
|
|
19
19
|
|
|
20
|
-
<!-- ──
|
|
21
|
-
<text x="64" y="152" font-size="20" font-weight="700" fill="#EFF3FA"
|
|
22
|
-
<text x="
|
|
20
|
+
<!-- ── 探针段完成度 ── -->
|
|
21
|
+
<text x="64" y="152" font-size="20" font-weight="700" fill="#EFF3FA">探针段完成度</text>
|
|
22
|
+
<text x="216" y="152" font-size="17" fill="#8296B3">(新会话延续任务,产物程序化校验)</text>
|
|
23
23
|
|
|
24
|
-
<rect x="260" y="170" width="
|
|
25
|
-
<text x="
|
|
26
|
-
<rect x="260" y="206" width="
|
|
27
|
-
<text x="
|
|
24
|
+
<rect x="260" y="170" width="598" height="26" rx="6" fill="#FFD15D"/>
|
|
25
|
+
<text x="872" y="191" font-size="22" font-weight="700" fill="#FFD15D">59/69 · 85.5%</text>
|
|
26
|
+
<rect x="260" y="206" width="304" height="26" rx="6" fill="#4E6DA8"/>
|
|
27
|
+
<text x="578" y="227" font-size="22" font-weight="700" fill="#8FA6C6">10/23 · 43.5%</text>
|
|
28
28
|
|
|
29
29
|
<line x1="64" y1="258" x2="1136" y2="258" stroke="#263A5C" stroke-width="1.5"/>
|
|
30
30
|
|
|
31
31
|
<!-- ── 成本对比(B = 满格基准)── -->
|
|
32
32
|
<text x="64" y="290" font-size="20" font-weight="700" fill="#EFF3FA">成本对比</text>
|
|
33
|
-
<text x="166" y="290" font-size="17" fill="#8296B3"
|
|
33
|
+
<text x="166" y="290" font-size="17" fill="#8296B3">(每场景均值,B 组 = 满格基准,条越短越省)</text>
|
|
34
34
|
|
|
35
35
|
<!-- 步骤 -->
|
|
36
36
|
<text x="240" y="348" font-size="19" fill="#C2CDDE" text-anchor="end">步骤</text>
|
|
37
|
-
<rect x="260" y="318" width="
|
|
38
|
-
<text x="
|
|
37
|
+
<rect x="260" y="318" width="329" height="18" rx="5" fill="#FFD15D"/>
|
|
38
|
+
<text x="603" y="333" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">24.3</text>
|
|
39
39
|
<rect x="260" y="342" width="560" height="18" rx="5" fill="#4E6DA8"/>
|
|
40
|
-
<text x="834" y="357" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">
|
|
40
|
+
<text x="834" y="357" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">41.4</text>
|
|
41
41
|
<rect x="980" y="321" width="136" height="36" rx="10" fill="#315DA6"/>
|
|
42
|
-
<text x="1048" y="345" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +
|
|
42
|
+
<text x="1048" y="345" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +70%</text>
|
|
43
43
|
|
|
44
44
|
<!-- 工具调用 -->
|
|
45
45
|
<text x="240" y="430" font-size="19" fill="#C2CDDE" text-anchor="end">工具调用</text>
|
|
46
|
-
<rect x="260" y="400" width="
|
|
47
|
-
<text x="
|
|
46
|
+
<rect x="260" y="400" width="340" height="18" rx="5" fill="#FFD15D"/>
|
|
47
|
+
<text x="614" y="415" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">37.7</text>
|
|
48
48
|
<rect x="260" y="424" width="560" height="18" rx="5" fill="#4E6DA8"/>
|
|
49
|
-
<text x="834" y="439" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">
|
|
49
|
+
<text x="834" y="439" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">62.1</text>
|
|
50
50
|
<rect x="980" y="403" width="136" height="36" rx="10" fill="#315DA6"/>
|
|
51
|
-
<text x="1048" y="427" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +
|
|
51
|
+
<text x="1048" y="427" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +65%</text>
|
|
52
52
|
|
|
53
53
|
<!-- 输入 token -->
|
|
54
54
|
<text x="240" y="512" font-size="19" fill="#C2CDDE" text-anchor="end">输入 token</text>
|
|
55
|
-
<rect x="260" y="482" width="
|
|
56
|
-
<text x="
|
|
55
|
+
<rect x="260" y="482" width="82" height="18" rx="5" fill="#FFD15D"/>
|
|
56
|
+
<text x="356" y="497" font-size="18" font-weight="700" fill="#FFD15D" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">266k</text>
|
|
57
57
|
<rect x="260" y="506" width="560" height="18" rx="5" fill="#4E6DA8"/>
|
|
58
|
-
<text x="834" y="521" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">1.
|
|
58
|
+
<text x="834" y="521" font-size="18" font-weight="700" fill="#8FA6C6" font-family="ui-monospace,SFMono-Regular,Menlo,Consolas,monospace">1.81M</text>
|
|
59
59
|
<rect x="980" y="485" width="136" height="36" rx="10" fill="#315DA6"/>
|
|
60
|
-
<text x="1048" y="509" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +
|
|
60
|
+
<text x="1048" y="509" font-size="19" font-weight="700" fill="#FFD15D" text-anchor="middle">B +580%</text>
|
|
61
61
|
|
|
62
62
|
<!-- ── 底部特写卡片 ── -->
|
|
63
63
|
<rect x="64" y="544" width="512" height="72" rx="14" fill="#15223A" stroke="#263A5C" stroke-width="1.5"/>
|
|
64
|
-
<text x="88" y="574" font-size="17" fill="#8296B3"
|
|
64
|
+
<text x="88" y="574" font-size="17" fill="#8296B3">风格规范场景 · 探针段(约定只存记忆,沙箱探不出来)</text>
|
|
65
65
|
<text x="88" y="600" font-size="20" font-weight="700">
|
|
66
|
-
<tspan fill="#FFD15D">A
|
|
67
|
-
<tspan fill="#8FA6C6"> · B
|
|
66
|
+
<tspan fill="#FFD15D">A 12/12</tspan>
|
|
67
|
+
<tspan fill="#8FA6C6"> · B 0/4</tspan>
|
|
68
68
|
</text>
|
|
69
69
|
|
|
70
70
|
<rect x="624" y="544" width="512" height="72" rx="14" fill="#15223A" stroke="#263A5C" stroke-width="1.5"/>
|
|
71
|
-
<text x="648" y="574" font-size="17" fill="#8296B3"
|
|
71
|
+
<text x="648" y="574" font-size="17" fill="#8296B3">长任务成本 · 每场景输入 token(B 无记忆重新探索)</text>
|
|
72
72
|
<text x="648" y="600" font-size="20" font-weight="700">
|
|
73
|
-
<tspan fill="#FFD15D">A
|
|
74
|
-
<tspan fill="#8FA6C6"> · B
|
|
73
|
+
<tspan fill="#FFD15D">A 266k</tspan>
|
|
74
|
+
<tspan fill="#8FA6C6"> · B 1.81M</tspan>
|
|
75
75
|
</text>
|
|
76
76
|
<rect x="1040" y="578" width="72" height="26" rx="8" fill="#315DA6"/>
|
|
77
|
-
<text x="1076" y="597" font-size="18" font-weight="700" fill="#FFD15D" text-anchor="middle"
|
|
77
|
+
<text x="1076" y="597" font-size="18" font-weight="700" fill="#FFD15D" text-anchor="middle">6.8×</text>
|
|
78
78
|
</g>
|
|
79
79
|
</svg>
|