dsh-session-eval 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +41 -0
- package/README.md +40 -75
- package/README.zh.md +56 -26
- package/cordis.patch.yml +6 -0
- package/lib/index.d.ts +49 -2
- package/lib/index.js +132 -6
- package/package.json +10 -10
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.2.2 — 2026-10-02
|
|
4
|
+
|
|
5
|
+
Contributor-first docs release.
|
|
6
|
+
|
|
7
|
+
- Install command fixed to `dsh plugin --profile web add dsh-session-eval` (a global npm install does not mount the slash commands).
|
|
8
|
+
- README.zh.md language switch fixed (English link restored, toggle on top); sample outputs bumped to the shipped version.
|
|
9
|
+
- Skeleton: one-liner → install → try-once → toolchain table → 6-line contributor block.
|
|
10
|
+
- CONTRIBUTING.md (with source map), SECURITY.md, CODE_OF_CONDUCT.md added; GitHub topics set.
|
|
11
|
+
- Docs no longer hardcode test counts.
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
## 0.2.1 — 2026-09-26
|
|
15
|
+
|
|
16
|
+
- **Fix: ship the bundle overlay.** v0.2.0 declared `dsh.bundle.patch: ./cordis.patch.yml`
|
|
17
|
+
in `package.json` but the file was never committed, so the published tarball did not
|
|
18
|
+
contain it and `dsh web` failed at startup with
|
|
19
|
+
`failed to read overlay .../dsh-session-eval/cordis.patch.yml: ENOENT`.
|
|
20
|
+
The overlay (mounting `/eval`, `/eval-diff`, `/eval-history`) is now in the tarball;
|
|
21
|
+
verified with `npm pack --dry-run`.
|
|
22
|
+
- **Fix: peer ranges match the current harness.** `@deepseek-ai/*` peers and dev
|
|
23
|
+
dependencies moved from `^0.1.1-rc.2` to `^0.1.5-rc.3` (semver prerelease ranges do not
|
|
24
|
+
cross `0.1.1-rc.x` → `0.1.5-rc.3`, which let stale `0.1.1-rc.2` copies install into
|
|
25
|
+
profiles and shadow the running harness). `@deepseek-ai/cordis` pinned to `4.0.2`,
|
|
26
|
+
matching the official plugins at this harness level.
|
|
27
|
+
- Added this changelog; `files` always listed it but it was never written.
|
|
28
|
+
|
|
29
|
+
## 0.2.0 — 2026-09-25
|
|
30
|
+
|
|
31
|
+
- New `/eval-history [N] [--json]` command: lists the current workspace's recent sessions
|
|
32
|
+
(same `sessionQuery` seam as `/archive`), grades the last N, and diffs first-vs-last for
|
|
33
|
+
a trend verdict. Caps the manual id-picking that `/eval-diff` required.
|
|
34
|
+
- Version string single-sourced from `src/version.ts` (replaces four hardcoded `v0.1.0`
|
|
35
|
+
strings).
|
|
36
|
+
- 8 new tests (35 total) and `typecheck` script.
|
|
37
|
+
|
|
38
|
+
## 0.1.0 — 2026-09-24
|
|
39
|
+
|
|
40
|
+
- Initial release: `/eval` grade cards from persisted session logs (tool-error rate,
|
|
41
|
+
follow-up signals, tool load), `/eval-diff` regression comparison, no LLM judge.
|
package/README.md
CHANGED
|
@@ -11,41 +11,22 @@ English | [中文](README.zh.md)
|
|
|
11
11
|
|
|
12
12
|
No benchmark suites to author. No LLM judges. No new runs. Every grade is a fixed, documented threshold over persisted session logs — fully reproducible.
|
|
13
13
|
|
|
14
|
-
##
|
|
15
|
-
|
|
16
|
-
| Plugin | Layer | Answers |
|
|
17
|
-
|---|---|---|
|
|
18
|
-
| [`dsh-session-export`](https://www.npmjs.com/package/dsh-session-export) | Evidence | "What exactly happened in this session?" |
|
|
19
|
-
| [`dsh-session-recall`](https://www.npmjs.com/package/dsh-session-recall) | Memory | "What did I do before, and where is it?" |
|
|
20
|
-
| **`dsh-session-eval`** | **Measurement** | **"Was that session good? Is the trend improving?"** |
|
|
21
|
-
|
|
22
|
-
All three read through the same trusted `ctx.sessionQuery` seam, so any persistence backend (JSONL or SQLite) works without touching raw artifacts.
|
|
23
|
-
|
|
24
|
-
## How it differs from benchmark-style evaluation
|
|
25
|
-
|
|
26
|
-
| | `dsh-eval` (benchmark) | `dsh-session-eval` (this) |
|
|
27
|
-
|---|---|---|
|
|
28
|
-
| Evaluates | Runs you launch ahead of time | Sessions that already happened |
|
|
29
|
-
| Setup | Author benchmark YAML + cases | None — read existing logs |
|
|
30
|
-
| Judge | Metric folding over trials | Fixed thresholds, deterministic |
|
|
31
|
-
| Best at | Pre-deploy benchmarks | Regression tracking over real work |
|
|
32
|
-
|
|
33
|
-
Both are useful; they don't compete. Benchmarks test what you *predicted*; retrospective evaluation watches what *actually happened*.
|
|
34
|
-
|
|
35
|
-
## Quick Start
|
|
14
|
+
## Install
|
|
36
15
|
|
|
37
16
|
```bash
|
|
38
|
-
|
|
17
|
+
dsh plugin --profile web add dsh-session-eval
|
|
39
18
|
```
|
|
40
19
|
|
|
41
|
-
|
|
20
|
+
## Try it once
|
|
21
|
+
|
|
22
|
+
In DeepSeek Harness:
|
|
42
23
|
|
|
43
24
|
```text
|
|
44
25
|
/eval
|
|
45
26
|
```
|
|
46
27
|
|
|
47
28
|
```text
|
|
48
|
-
Session eval — a1b2c3d4… (dsh-session-eval v0.2.
|
|
29
|
+
Session eval — a1b2c3d4… (dsh-session-eval v0.2.1)
|
|
49
30
|
Overall: A
|
|
50
31
|
Reliability A 2/87 tool results errored (2.3%)
|
|
51
32
|
Re-ask A 1 re-ask signal(s) over 23 turns (0.4/10 turns)
|
|
@@ -54,45 +35,50 @@ Notes:
|
|
|
54
35
|
• Wall time 41.2 min over 23 turns.
|
|
55
36
|
```
|
|
56
37
|
|
|
57
|
-
|
|
38
|
+
## The measurement layer of the session toolchain
|
|
58
39
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
40
|
+
| Plugin | Layer | Answers |
|
|
41
|
+
|---|---|---|
|
|
42
|
+
| [`dsh-session-export`](https://www.npmjs.com/package/dsh-session-export) | Evidence | "What exactly happened in this session?" |
|
|
43
|
+
| [`dsh-session-recall`](https://www.npmjs.com/package/dsh-session-recall) | Memory | "What did I do before, and where is it?" |
|
|
44
|
+
| **`dsh-session-eval`** | **Measurement** | **"Was that session good? Is the trend improving?"** |
|
|
62
45
|
|
|
63
|
-
|
|
46
|
+
All three read through the same trusted `ctx.sessionQuery` seam, so any persistence backend (JSONL or SQLite) works without touching raw artifacts.
|
|
64
47
|
|
|
65
|
-
|
|
66
|
-
/eval-diff <beforeSessionId> <afterSessionId>
|
|
67
|
-
```
|
|
48
|
+
## Contributing
|
|
68
49
|
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
Re-ask 0.5 → 3.2 regressed
|
|
74
|
-
Tool load 3.8 → 4.1 flat
|
|
75
|
-
```
|
|
50
|
+
- **Local dev**: `npm ci && npm run typecheck && npm test && npm run bundle` (Node 20 or 22).
|
|
51
|
+
- **Start in the source**: [`src/grade.ts`](src/grade.ts) (thresholds, grade cards), [`src/metrics.ts`](src/metrics.ts) (metric extraction), [`src/trend.ts`](src/trend.ts) (`/eval-history` trend logic). See [CONTRIBUTING.md](CONTRIBUTING.md) for the source map and first-PR suggestions.
|
|
52
|
+
- **Open gaps**: threshold configurability, wider `/eval-history` filters, trend export — tracked in [issues labeled `good first issue`](https://github.com/kittimzhe/dsh-session-eval/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
|
|
53
|
+
- **Rules**: behavior changes need tests; doc changes must update `README.md` and `README.zh.md` in sync; version releases belong to the maintainer. Details: [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
76
54
|
|
|
77
|
-
|
|
55
|
+
## How it differs from benchmark-style evaluation
|
|
78
56
|
|
|
79
|
-
|
|
57
|
+
| | `dsh-eval` (benchmark) | `dsh-session-eval` (this) |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| Evaluates | Runs you launch ahead of time | Sessions that already happened |
|
|
60
|
+
| Setup | Author benchmark YAML + cases | None — read existing logs |
|
|
61
|
+
| Judge | Metric folding over trials | Fixed thresholds, deterministic |
|
|
62
|
+
| Best at | Pre-deploy benchmarks | Regression tracking over real work |
|
|
80
63
|
|
|
81
|
-
|
|
82
|
-
/eval-history # last 5 sessions in this workspace
|
|
83
|
-
/eval-history 12 # last 12
|
|
84
|
-
```
|
|
64
|
+
Both are useful; they don't compete. Benchmarks test what you *predicted*; retrospective evaluation watches what *actually happened*.
|
|
85
65
|
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
66
|
+
## Commands
|
|
67
|
+
|
|
68
|
+
| Command | Effect |
|
|
69
|
+
|---|---|
|
|
70
|
+
| `/eval` | Grade card for the current session |
|
|
71
|
+
| `/eval --id <sessionId>` | Grade a specific session |
|
|
72
|
+
| `/eval --json` | Machine-readable metrics + card |
|
|
73
|
+
| `/eval-diff <beforeId> <afterId>` | Regression comparison, noise-tolerant |
|
|
74
|
+
| `/eval-diff … --json` | Machine-readable trend report |
|
|
75
|
+
| `/eval-history [N]` | Grade the last N sessions in this workspace and show the trend |
|
|
76
|
+
| `/eval-history … --json` | Machine-readable history report |
|
|
93
77
|
|
|
94
78
|
`/eval-history` finds the sessions for you via the same `sessionQuery` seam `/archive` uses (scoped to the current workspace's `cwd`), grades each, and diffs the first against the last. `/eval-diff` remains the tool when you already know the two ids that matter.
|
|
95
79
|
|
|
80
|
+
Typical use: grade the ten sessions before and after a plugin or prompt change; a persistent `Reliability` regression is an early warning that the change hurt real work, not just benchmarks.
|
|
81
|
+
|
|
96
82
|
## Dimensions & thresholds
|
|
97
83
|
|
|
98
84
|
All grades derive from three deterministic metrics. `n/a` when the session lacks the relevant activity.
|
|
@@ -105,7 +91,7 @@ All grades derive from three deterministic metrics. `n/a` when the session lacks
|
|
|
105
91
|
|
|
106
92
|
¹ A *correction signal* is each extra user message inside a consecutive-user run — the human re-asking before any assistant reply, a behavioral proxy for "the agent didn't get it right the first time."
|
|
107
93
|
|
|
108
|
-
`Overall` is the **worst** graded dimension — conservative by design. Thresholds are constants in [`src/grade.ts`](src/grade.ts);
|
|
94
|
+
`Overall` is the **worst** graded dimension — conservative by design. Thresholds are constants in [`src/grade.ts`](src/grade.ts); making them configurable is on the roadmap (see issues).
|
|
109
95
|
|
|
110
96
|
Diff verdicts include a noise band (±2pp error rate, ±0.5 re-ask, ±1 tool load) so trivial jitter doesn't cry regression.
|
|
111
97
|
|
|
@@ -113,27 +99,6 @@ Diff verdicts include a noise band (±2pp error rate, ±0.5 re-ask, ±1 tool loa
|
|
|
113
99
|
|
|
114
100
|
Same session logs in → same grade card out, every time, forever. There is no model in the loop, no sampling, no clock dependence in the metrics (wall time is reported, never graded). This is what makes `/eval-diff` trustworthy as a regression signal.
|
|
115
101
|
|
|
116
|
-
## Commands
|
|
117
|
-
|
|
118
|
-
| Command | Effect |
|
|
119
|
-
|---|---|
|
|
120
|
-
| `/eval` | Grade card for the current session |
|
|
121
|
-
| `/eval --id <sessionId>` | Grade a specific session |
|
|
122
|
-
| `/eval --json` | Machine-readable metrics + card |
|
|
123
|
-
| `/eval-diff <beforeId> <afterId>` | Regression comparison, noise-tolerant |
|
|
124
|
-
| `/eval-diff … --json` | Machine-readable trend report |
|
|
125
|
-
| `/eval-history [N]` | Grade the last N sessions in this workspace and show the trend |
|
|
126
|
-
| `/eval-history … --json` | Machine-readable history report |
|
|
127
|
-
|
|
128
|
-
## Development
|
|
129
|
-
|
|
130
|
-
```bash
|
|
131
|
-
npm install
|
|
132
|
-
npm run typecheck # tsc --noEmit
|
|
133
|
-
npm test # vitest run — 24 tests
|
|
134
|
-
npm run bundle # tsdown → lib/
|
|
135
|
-
```
|
|
136
|
-
|
|
137
102
|
## License
|
|
138
103
|
|
|
139
104
|
[MIT](LICENSE)
|
package/README.zh.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# dsh-session-eval
|
|
2
2
|
|
|
3
|
+
[English](README.md) | 中文
|
|
4
|
+
|
|
3
5
|
[](https://www.npmjs.com/package/dsh-session-eval)
|
|
4
6
|
[](https://github.com/kittimzhe/dsh-session-eval/actions/workflows/test.yml)
|
|
5
7
|
[](LICENSE)
|
|
@@ -8,40 +10,73 @@
|
|
|
8
10
|
|
|
9
11
|
不需要编写基准测试集，没有 LLM 裁判，不需要跑新任务。所有等级 = 持久化会话日志上的固定公开阈值——完全可复现。
|
|
10
12
|
|
|
11
|
-
|
|
13
|
+
## 安装
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
dsh plugin --profile web add dsh-session-eval
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## 试一次
|
|
20
|
+
|
|
21
|
+
在 DeepSeek Harness 里：
|
|
22
|
+
|
|
23
|
+
```text
|
|
24
|
+
/eval
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
```text
|
|
28
|
+
Session eval — a1b2c3d4… (dsh-session-eval v0.2.1)
|
|
29
|
+
Overall: A
|
|
30
|
+
Reliability A 2/87 tool results errored (2.3%)
|
|
31
|
+
Re-ask A 1 re-ask signal(s) over 23 turns (0.4/10 turns)
|
|
32
|
+
Tool load B 3.8 tool calls per turn
|
|
33
|
+
Notes:
|
|
34
|
+
• Wall time 41.2 min over 23 turns.
|
|
35
|
+
```
|
|
12
36
|
|
|
13
37
|
## 会话工具链的评测层
|
|
14
38
|
|
|
15
39
|
| 插件 | 层 | 回答的问题 |
|
|
16
40
|
|---|---|---|
|
|
17
|
-
| `dsh-session-export` | 证据层 | "这个会话到底发生了什么？" |
|
|
18
|
-
| `dsh-session-recall` | 记忆层 | "我以前做过什么，在哪？" |
|
|
41
|
+
| [`dsh-session-export`](https://www.npmjs.com/package/dsh-session-export) | 证据层 | "这个会话到底发生了什么？" |
|
|
42
|
+
| [`dsh-session-recall`](https://www.npmjs.com/package/dsh-session-recall) | 记忆层 | "我以前做过什么，在哪？" |
|
|
19
43
|
| **`dsh-session-eval`** | **评测层** | **"刚才的会话好不好？趋势在变好吗？"** |
|
|
20
44
|
|
|
21
45
|
三者都通过同一个可信的 `ctx.sessionQuery` 缝读取，JSONL / SQLite 等任何持久化后端均可使用。
|
|
22
46
|
|
|
47
|
+
## 贡献
|
|
48
|
+
|
|
49
|
+
- **本地开发**：`npm ci && npm run typecheck && npm test && npm run bundle`（Node 20 或 22）。
|
|
50
|
+
- **源码入口**：[`src/grade.ts`](src/grade.ts)（阈值与等级卡）、[`src/metrics.ts`](src/metrics.ts)（指标提取）、[`src/trend.ts`](src/trend.ts)（`/eval-history` 趋势逻辑）。源码地图与第一次 PR 建议见 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
|
51
|
+
- **当前缺口**：阈值可配置、`/eval-history` 过滤条件扩展、趋势导出——见 [`good first issue` 标签的 issue](https://github.com/kittimzhe/dsh-session-eval/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)。
|
|
52
|
+
- **规矩**：行为变更必须带测试；文档必须 `README.md` 与 `README.zh.md` 同步改；版本发布由维护者执行。详见 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
|
53
|
+
|
|
23
54
|
## 与基准式评测（dsh-eval）的区别
|
|
24
55
|
|
|
25
|
-
- `dsh-eval
|
|
26
|
-
|
|
56
|
+
| | `dsh-eval`（基准） | `dsh-session-eval`（本插件） |
|
|
57
|
+
|---|---|---|
|
|
58
|
+
| 评什么 | 预先发起的运行 | 已经发生的会话 |
|
|
59
|
+
| 准备 | 编写基准 YAML + 用例 | 零——直接读存量日志 |
|
|
60
|
+
| 裁判 | 跨 trial 的指标折叠 | 固定阈值，确定性 |
|
|
61
|
+
| 擅长 | 上线前基准 | 真实工作的回归追踪 |
|
|
27
62
|
|
|
28
|
-
|
|
63
|
+
两者互补而非竞争：基准测你*预测到*的场景，回溯评测看*实际发生*的工作。
|
|
29
64
|
|
|
30
|
-
##
|
|
65
|
+
## 命令
|
|
31
66
|
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
67
|
+
| 命令 | 效果 |
|
|
68
|
+
|---|---|
|
|
69
|
+
| `/eval` | 当前会话等级卡 |
|
|
70
|
+
| `/eval --id <会话ID>` | 指定会话 |
|
|
71
|
+
| `/eval --json` | 机器可读指标 + 等级卡 |
|
|
72
|
+
| `/eval-diff <前ID> <后ID>` | 回归对比（带噪声带） |
|
|
73
|
+
| `/eval-diff … --json` | 机器可读趋势报告 |
|
|
74
|
+
| `/eval-history [N]` | 本工作区最近 N 个会话成绩单 + 首末趋势 |
|
|
75
|
+
| `/eval-history … --json` | 机器可读历史报告 |
|
|
35
76
|
|
|
36
|
-
|
|
37
|
-
/eval # 当前会话等级卡
|
|
38
|
-
/eval --id <会话ID> # 指定会话
|
|
39
|
-
/eval-diff <前> <后> # 回归对比（换插件/prompt 前后各评一次）
|
|
40
|
-
/eval-history # 本工作区最近 5 个会话的成绩单 + 首末趋势
|
|
41
|
-
/eval-history 12 # 最近 12 个
|
|
42
|
-
```
|
|
77
|
+
`/eval-history` 通过与 `/archive` 相同的 `sessionQuery` 接缝自动找到会话（按当前工作区 `cwd` 过滤），逐个评级，并把首尾两个会话做趋势对比——不用再手动挑 id。已经知道两个关键 id 时用 `/eval-diff`。
|
|
43
78
|
|
|
44
|
-
|
|
79
|
+
典型用法：换插件或 prompt 前后各评十个会话；`Reliability` 持续回归 = 改动伤害了真实工作的早期预警。
|
|
45
80
|
|
|
46
81
|
## 维度与阈值（确定性，无裁判）
|
|
47
82
|
|
|
@@ -53,19 +88,14 @@ npm install -g dsh-session-eval
|
|
|
53
88
|
|
|
54
89
|
¹ 纠正信号 = 连续用户消息中多出的那条（agent 还没回复用户就重问）——"第一次没做对"的行为学代理指标。
|
|
55
90
|
|
|
56
|
-
总评 =
|
|
91
|
+
总评 = 最差维度（保守设计）。阈值是 [`src/grade.ts`](src/grade.ts) 中的常量；可配置化在路线图上（见 issue）。
|
|
92
|
+
|
|
93
|
+
Diff 判定带噪声带（错误率 ±2pp、Re-ask ±0.5、工具负载 ±1），微小抖动不会误报回归。
|
|
57
94
|
|
|
58
95
|
## 确定性保证
|
|
59
96
|
|
|
60
97
|
同样的日志进 → 同样的等级出。没有模型在环、没有采样、指标不含时钟依赖（wall time 只报告不评级）。这是 `/eval-diff` 能作为回归信号被信任的原因。
|
|
61
98
|
|
|
62
|
-
## 开发
|
|
63
|
-
|
|
64
|
-
```bash
|
|
65
|
-
npm install && npm run typecheck && npm test # 24 个测试
|
|
66
|
-
npm run bundle
|
|
67
|
-
```
|
|
68
|
-
|
|
69
99
|
## 许可
|
|
70
100
|
|
|
71
101
|
[MIT](LICENSE)
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
# The dsh-session-eval bundle patch: mounts /eval, /eval-diff, and /eval-history.
|
|
2
|
+
# Applied as one insert; requires `commands` and `sessionQuery` services,
|
|
3
|
+
# which the shipped dsh-base / web bundles already provide.
|
|
4
|
+
- insert:
|
|
5
|
+
- id: session-eval
|
|
6
|
+
name: 'dsh-session-eval'
|
package/lib/index.d.ts
CHANGED
|
@@ -132,10 +132,57 @@ declare function executeEval(ctx: Context, invocation: CommandInvocation, seam?:
|
|
|
132
132
|
/** Execute /eval-diff against the session-query seam. */
|
|
133
133
|
declare function executeEvalDiff(ctx: Context, invocation: CommandInvocation, seam?: QuerySeam): Promise<CommandResult>;
|
|
134
134
|
//#endregion
|
|
135
|
+
//#region src/historyCommand.d.ts
|
|
136
|
+
declare const EVAL_HISTORY_USAGE = "Usage: /eval-history [N] [--json] (N = last N sessions, 1-20, default 5)";
|
|
137
|
+
/** Args for /eval-history. */
|
|
138
|
+
interface EvalHistoryArgs {
|
|
139
|
+
readonly count: number;
|
|
140
|
+
readonly json: boolean;
|
|
141
|
+
}
|
|
142
|
+
/** Parse /eval-history input; returns args or a usage-error string. */
|
|
143
|
+
declare function parseEvalHistoryArgs(rawInput: string): EvalHistoryArgs | string;
|
|
144
|
+
/** One graded session in a history report. */
|
|
145
|
+
interface HistoryEntry {
|
|
146
|
+
readonly sessionId: string;
|
|
147
|
+
readonly createdAt: number | null;
|
|
148
|
+
readonly metrics: SessionMetrics;
|
|
149
|
+
readonly card: GradeCard;
|
|
150
|
+
}
|
|
151
|
+
/** The full /eval-history report. */
|
|
152
|
+
interface HistoryReport {
|
|
153
|
+
readonly entries: readonly HistoryEntry[];
|
|
154
|
+
readonly trend: TrendReport | null;
|
|
155
|
+
}
|
|
156
|
+
/** The slice of sessionQuery this command consumes (structural, for testability). */
|
|
157
|
+
interface HistorySeam {
|
|
158
|
+
listSessions(): Promise<ReadonlyArray<{
|
|
159
|
+
header: {
|
|
160
|
+
id: unknown;
|
|
161
|
+
createdAt: unknown;
|
|
162
|
+
cwd?: unknown;
|
|
163
|
+
};
|
|
164
|
+
}>>;
|
|
165
|
+
readSession(id: ReturnType<typeof SessionId>): Promise<{
|
|
166
|
+
events: readonly unknown[];
|
|
167
|
+
}>;
|
|
168
|
+
}
|
|
169
|
+
/** Render a history report as terminal text. */
|
|
170
|
+
declare function renderHistory(report: HistoryReport): string;
|
|
171
|
+
/** Execute /eval-history against the session-query seam. */
|
|
172
|
+
declare function executeEvalHistory(ctx: Context, invocation: CommandInvocation, seam?: HistorySeam): Promise<CommandResult>;
|
|
173
|
+
//#endregion
|
|
174
|
+
//#region src/version.d.ts
|
|
175
|
+
/**
|
|
176
|
+
* Single source of truth for the plugin version string.
|
|
177
|
+
*
|
|
178
|
+
* @module dsh-session-eval/version
|
|
179
|
+
*/
|
|
180
|
+
declare const VERSION = "0.2.2";
|
|
181
|
+
//#endregion
|
|
135
182
|
//#region src/index.d.ts
|
|
136
183
|
declare const name = "session-eval";
|
|
137
184
|
declare const inject: string[];
|
|
138
|
-
/** Plugin entry: mount the /eval and /eval-
|
|
185
|
+
/** Plugin entry: mount the /eval, /eval-diff and /eval-history commands. */
|
|
139
186
|
declare function apply(ctx: Context): void;
|
|
140
187
|
//#endregion
|
|
141
|
-
export { type DimensionDelta, type DimensionGrade, EVAL_DIFF_USAGE, EVAL_USAGE, type EvalArgs, type EvalDiffArgs, type EvalEvent, type EvalEventKind, type Grade, type GradeCard, type SessionMetrics, type TrendReport, adaptEvents, apply, compareSessions, computeMetrics, executeEval, executeEvalDiff, gradeSession, id8, inject, name, parseEvalArgs, parseEvalDiffArgs, renderCard, renderTrend };
|
|
188
|
+
export { type DimensionDelta, type DimensionGrade, EVAL_DIFF_USAGE, EVAL_HISTORY_USAGE, EVAL_USAGE, type EvalArgs, type EvalDiffArgs, type EvalEvent, type EvalEventKind, type EvalHistoryArgs, type Grade, type GradeCard, type HistoryEntry, type HistoryReport, type HistorySeam, type SessionMetrics, type TrendReport, VERSION, adaptEvents, apply, compareSessions, computeMetrics, executeEval, executeEvalDiff, executeEvalHistory, gradeSession, id8, inject, name, parseEvalArgs, parseEvalDiffArgs, parseEvalHistoryArgs, renderCard, renderHistory, renderTrend };
|
package/lib/index.js
CHANGED
|
@@ -221,6 +221,14 @@ function compareSessions(before, after) {
|
|
|
221
221
|
};
|
|
222
222
|
}
|
|
223
223
|
//#endregion
|
|
224
|
+
//#region src/version.ts
|
|
225
|
+
/**
|
|
226
|
+
* Single source of truth for the plugin version string.
|
|
227
|
+
*
|
|
228
|
+
* @module dsh-session-eval/version
|
|
229
|
+
*/
|
|
230
|
+
const VERSION = "0.2.2";
|
|
231
|
+
//#endregion
|
|
224
232
|
//#region src/evalCommand.ts
|
|
225
233
|
const EVAL_USAGE = "Usage: /eval [--id <sessionId>] [--json]";
|
|
226
234
|
const EVAL_DIFF_USAGE = "Usage: /eval-diff <beforeSessionId> <afterSessionId> [--json]";
|
|
@@ -319,14 +327,14 @@ async function executeEval(ctx, invocation, seam = ctx.sessionQuery) {
|
|
|
319
327
|
if (parsed.json === true) return {
|
|
320
328
|
kind: "success",
|
|
321
329
|
text: JSON.stringify({
|
|
322
|
-
generator: "dsh-session-eval v0.
|
|
330
|
+
generator: "dsh-session-eval v0.2.2",
|
|
323
331
|
metrics,
|
|
324
332
|
card
|
|
325
333
|
}, null, 2)
|
|
326
334
|
};
|
|
327
335
|
return {
|
|
328
336
|
kind: "success",
|
|
329
|
-
text: renderCard(card, "dsh-session-eval v0.
|
|
337
|
+
text: renderCard(card, "dsh-session-eval v0.2.2")
|
|
330
338
|
};
|
|
331
339
|
}
|
|
332
340
|
/** Execute /eval-diff against the session-query seam. */
|
|
@@ -350,20 +358,133 @@ async function executeEvalDiff(ctx, invocation, seam = ctx.sessionQuery) {
|
|
|
350
358
|
if (parsed.json === true) return {
|
|
351
359
|
kind: "success",
|
|
352
360
|
text: JSON.stringify({
|
|
353
|
-
generator: "dsh-session-eval v0.
|
|
361
|
+
generator: "dsh-session-eval v0.2.2",
|
|
362
|
+
report
|
|
363
|
+
}, null, 2)
|
|
364
|
+
};
|
|
365
|
+
return {
|
|
366
|
+
kind: "success",
|
|
367
|
+
text: renderTrend(report, "dsh-session-eval v0.2.2")
|
|
368
|
+
};
|
|
369
|
+
}
|
|
370
|
+
//#endregion
|
|
371
|
+
//#region src/historyCommand.ts
|
|
372
|
+
const EVAL_HISTORY_USAGE = "Usage: /eval-history [N] [--json] (N = last N sessions, 1-20, default 5)";
|
|
373
|
+
/** Parse /eval-history input; returns args or a usage-error string. */
|
|
374
|
+
function parseEvalHistoryArgs(rawInput) {
|
|
375
|
+
const tokens = rawInput.trim().split(/\s+/).filter((t) => t.length > 0);
|
|
376
|
+
let count = 5;
|
|
377
|
+
let json = false;
|
|
378
|
+
let sawCount = false;
|
|
379
|
+
for (const token of tokens) {
|
|
380
|
+
if (token === "--json") {
|
|
381
|
+
if (json) return `--json may be given only once.\n${EVAL_HISTORY_USAGE}`;
|
|
382
|
+
json = true;
|
|
383
|
+
continue;
|
|
384
|
+
}
|
|
385
|
+
if (token.startsWith("--")) return `Unknown argument: ${token}\n${EVAL_HISTORY_USAGE}`;
|
|
386
|
+
if (sawCount) return `Expected at most one count argument.\n${EVAL_HISTORY_USAGE}`;
|
|
387
|
+
if (!/^\d+$/.test(token)) return `N must be a positive integer.\n${EVAL_HISTORY_USAGE}`;
|
|
388
|
+
const value = Number(token);
|
|
389
|
+
if (value < 1 || value > 20) return `N must be between 1 and 20.\n${EVAL_HISTORY_USAGE}`;
|
|
390
|
+
count = value;
|
|
391
|
+
sawCount = true;
|
|
392
|
+
}
|
|
393
|
+
return {
|
|
394
|
+
count,
|
|
395
|
+
json
|
|
396
|
+
};
|
|
397
|
+
}
|
|
398
|
+
function toMillis(createdAt) {
|
|
399
|
+
if (typeof createdAt === "number" && Number.isFinite(createdAt)) return createdAt;
|
|
400
|
+
if (typeof createdAt === "string") {
|
|
401
|
+
const parsed = Date.parse(createdAt);
|
|
402
|
+
if (Number.isFinite(parsed)) return parsed;
|
|
403
|
+
}
|
|
404
|
+
return null;
|
|
405
|
+
}
|
|
406
|
+
function stamp(ms) {
|
|
407
|
+
if (ms === null) return "?";
|
|
408
|
+
return new Date(ms).toISOString().slice(0, 16).replace("T", " ");
|
|
409
|
+
}
|
|
410
|
+
/** Render a history report as terminal text. */
|
|
411
|
+
function renderHistory(report) {
|
|
412
|
+
const lines = [];
|
|
413
|
+
if (report.entries.length === 0) return "No sessions found in this workspace; nothing to grade.";
|
|
414
|
+
lines.push(`Session history — last ${report.entries.length} in this workspace`);
|
|
415
|
+
for (const entry of report.entries) {
|
|
416
|
+
const m = entry.metrics;
|
|
417
|
+
lines.push(` ${stamp(entry.createdAt)} ${id8(entry.sessionId)} ${String(entry.card.overall).padEnd(2)} ${m.userTurns} turns, ${m.toolCalls} tools`);
|
|
418
|
+
}
|
|
419
|
+
const trend = report.trend;
|
|
420
|
+
if (trend !== null) lines.push(`Trend (first → last): ${trend.overall} — ${trend.summary}`);
|
|
421
|
+
return lines.join("\n");
|
|
422
|
+
}
|
|
423
|
+
/** Execute /eval-history against the session-query seam. */
|
|
424
|
+
async function executeEvalHistory(ctx, invocation, seam = ctx.sessionQuery) {
|
|
425
|
+
const parsed = parseEvalHistoryArgs(invocation.rawInput);
|
|
426
|
+
if (typeof parsed === "string") return {
|
|
427
|
+
kind: "error",
|
|
428
|
+
text: parsed
|
|
429
|
+
};
|
|
430
|
+
const ownCwd = invocation.agent.session.header?.cwd;
|
|
431
|
+
let records;
|
|
432
|
+
try {
|
|
433
|
+
records = await seam.listSessions();
|
|
434
|
+
} catch (error) {
|
|
435
|
+
return {
|
|
436
|
+
kind: "error",
|
|
437
|
+
text: `Could not list sessions: ${error instanceof Error ? error.message : String(error)}`
|
|
438
|
+
};
|
|
439
|
+
}
|
|
440
|
+
const recent = records.map((record) => record.header).filter((header) => header != null).filter((header) => ownCwd === void 0 || header.cwd === ownCwd).map((header) => ({
|
|
441
|
+
id: String(header.id),
|
|
442
|
+
createdAt: toMillis(header.createdAt)
|
|
443
|
+
})).sort((a, b) => (a.createdAt ?? 0) - (b.createdAt ?? 0)).slice(-parsed.count);
|
|
444
|
+
if (recent.length === 0) return {
|
|
445
|
+
kind: "success",
|
|
446
|
+
text: "No sessions found in this workspace; nothing to grade."
|
|
447
|
+
};
|
|
448
|
+
const entries = [];
|
|
449
|
+
for (const header of recent) {
|
|
450
|
+
let events;
|
|
451
|
+
try {
|
|
452
|
+
events = (await seam.readSession(SessionId(header.id))).events;
|
|
453
|
+
} catch (error) {
|
|
454
|
+
return {
|
|
455
|
+
kind: "error",
|
|
456
|
+
text: `Could not read session ${id8(header.id)}: ${error instanceof Error ? error.message : String(error)}`
|
|
457
|
+
};
|
|
458
|
+
}
|
|
459
|
+
const metrics = computeMetrics(header.id, adaptEvents(events));
|
|
460
|
+
entries.push({
|
|
461
|
+
sessionId: header.id,
|
|
462
|
+
createdAt: header.createdAt,
|
|
463
|
+
metrics,
|
|
464
|
+
card: gradeSession(metrics)
|
|
465
|
+
});
|
|
466
|
+
}
|
|
467
|
+
const report = {
|
|
468
|
+
entries,
|
|
469
|
+
trend: entries.length >= 2 ? compareSessions(entries[0].metrics, entries[entries.length - 1].metrics) : null
|
|
470
|
+
};
|
|
471
|
+
if (parsed.json === true) return {
|
|
472
|
+
kind: "success",
|
|
473
|
+
text: JSON.stringify({
|
|
474
|
+
generator: `dsh-session-eval v${VERSION}`,
|
|
354
475
|
report
|
|
355
476
|
}, null, 2)
|
|
356
477
|
};
|
|
357
478
|
return {
|
|
358
479
|
kind: "success",
|
|
359
|
-
text:
|
|
480
|
+
text: renderHistory(report)
|
|
360
481
|
};
|
|
361
482
|
}
|
|
362
483
|
//#endregion
|
|
363
484
|
//#region src/index.ts
|
|
364
485
|
const name = "session-eval";
|
|
365
486
|
const inject = ["commands", "sessionQuery"];
|
|
366
|
-
/** Plugin entry: mount the /eval and /eval-
|
|
487
|
+
/** Plugin entry: mount the /eval, /eval-diff and /eval-history commands. */
|
|
367
488
|
function apply(ctx) {
|
|
368
489
|
ctx.effect(function* () {
|
|
369
490
|
yield ctx.commands.register({
|
|
@@ -376,7 +497,12 @@ function apply(ctx) {
|
|
|
376
497
|
description: "Compare two sessions (before, after) and report which dimensions improved or regressed",
|
|
377
498
|
handler: (invocation) => executeEvalDiff(ctx, invocation)
|
|
378
499
|
});
|
|
500
|
+
yield ctx.commands.register({
|
|
501
|
+
name: "eval-history",
|
|
502
|
+
description: "Grade the last N sessions in this workspace (default 5) and show whether the trend is improving",
|
|
503
|
+
handler: (invocation) => executeEvalHistory(ctx, invocation)
|
|
504
|
+
});
|
|
379
505
|
}, "session-eval lifecycle");
|
|
380
506
|
}
|
|
381
507
|
//#endregion
|
|
382
|
-
export { EVAL_DIFF_USAGE, EVAL_USAGE, adaptEvents, apply, compareSessions, computeMetrics, executeEval, executeEvalDiff, gradeSession, id8, inject, name, parseEvalArgs, parseEvalDiffArgs, renderCard, renderTrend };
|
|
508
|
+
export { EVAL_DIFF_USAGE, EVAL_HISTORY_USAGE, EVAL_USAGE, VERSION, adaptEvents, apply, compareSessions, computeMetrics, executeEval, executeEvalDiff, executeEvalHistory, gradeSession, id8, inject, name, parseEvalArgs, parseEvalDiffArgs, parseEvalHistoryArgs, renderCard, renderHistory, renderTrend };
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-session-eval",
|
|
3
3
|
"description": "Retrospective agent-session evaluation for DeepSeek Harness: deterministic grade cards from persisted session logs, cross-session regression diffs, era comparison — no benchmark runs, no LLM judges.",
|
|
4
|
-
"version": "0.2.
|
|
4
|
+
"version": "0.2.2",
|
|
5
5
|
"publishConfig": {
|
|
6
6
|
"access": "public"
|
|
7
7
|
},
|
|
@@ -38,21 +38,21 @@
|
|
|
38
38
|
"regression"
|
|
39
39
|
],
|
|
40
40
|
"peerDependencies": {
|
|
41
|
-
"@deepseek-ai/cordis": "
|
|
42
|
-
"@deepseek-ai/dsh-commands": "^0.1.
|
|
43
|
-
"@deepseek-ai/dsh-llm": "^0.1.
|
|
44
|
-
"@deepseek-ai/dsh-session": "^0.1.
|
|
45
|
-
"@deepseek-ai/dsh-session-query": "^0.1.
|
|
46
|
-
"@deepseek-ai/dsh-tools": "^0.1.
|
|
41
|
+
"@deepseek-ai/cordis": "4.0.2",
|
|
42
|
+
"@deepseek-ai/dsh-commands": "^0.1.5-rc.3",
|
|
43
|
+
"@deepseek-ai/dsh-llm": "^0.1.5-rc.3",
|
|
44
|
+
"@deepseek-ai/dsh-session": "^0.1.5-rc.3",
|
|
45
|
+
"@deepseek-ai/dsh-session-query": "^0.1.5-rc.3",
|
|
46
|
+
"@deepseek-ai/dsh-tools": "^0.1.5-rc.3"
|
|
47
47
|
},
|
|
48
48
|
"devDependencies": {
|
|
49
49
|
"@types/node": "^26.2.0",
|
|
50
50
|
"tsdown": "^0.22.14",
|
|
51
51
|
"typescript": "^7.0.2",
|
|
52
52
|
"vitest": "^4.1.11",
|
|
53
|
-
"@deepseek-ai/cordis": "
|
|
54
|
-
"@deepseek-ai/dsh-commands": "^0.1.
|
|
55
|
-
"@deepseek-ai/dsh-session": "^0.1.
|
|
53
|
+
"@deepseek-ai/cordis": "4.0.2",
|
|
54
|
+
"@deepseek-ai/dsh-commands": "^0.1.5-rc.3",
|
|
55
|
+
"@deepseek-ai/dsh-session": "^0.1.5-rc.3"
|
|
56
56
|
},
|
|
57
57
|
"scripts": {
|
|
58
58
|
"bundle": "tsdown",
|