dsh-session-eval 0.2.2 → 0.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +11 -0
- package/README.md +31 -2
- package/README.zh.md +32 -2
- package/lib/index.d.ts +1 -1
- package/lib/index.js +5 -5
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,12 @@
|
|
|
1
|
+
## 0.2.3 — 2026-10-03
|
|
2
|
+
|
|
3
|
+
Docs polish round (no runtime changes).
|
|
4
|
+
|
|
5
|
+
- Install detail completed: requirements, GitHub install route, the `cordis.patch.yml` row.
|
|
6
|
+
- Contributor block links the open-gap issues directly; sample outputs aligned with the shipped version.
|
|
7
|
+
- Community section; bilingual issue templates (`.github/ISSUE_TEMPLATE/`).
|
|
8
|
+
- English/Chinese README sections kept in sync.
|
|
9
|
+
|
|
1
10
|
# Changelog
|
|
2
11
|
|
|
3
12
|
## 0.2.2 — 2026-10-02
|
|
@@ -9,6 +18,8 @@ Contributor-first docs release.
|
|
|
9
18
|
- Skeleton: one-liner → install → try-once → toolchain table → 6-line contributor block.
|
|
10
19
|
- CONTRIBUTING.md (with source map), SECURITY.md, CODE_OF_CONDUCT.md added; GitHub topics set.
|
|
11
20
|
- Docs no longer hardcode test counts.
|
|
21
|
+
- Install details (GitHub route + `cordis.patch.yml`), Development/Community footers, and issue links #1–#3 added for parity with the sibling plugins.
|
|
22
|
+
- GitHub issue templates (`bug_report`, `feature_request`) added.
|
|
12
23
|
|
|
13
24
|
|
|
14
25
|
## 0.2.1 — 2026-09-26
|
package/README.md
CHANGED
|
@@ -13,10 +13,26 @@ No benchmark suites to author. No LLM judges. No new runs. Every grade is a fixe
|
|
|
13
13
|
|
|
14
14
|
## Install
|
|
15
15
|
|
|
16
|
+
**Requirements**: Node.js 20 or 22 · a DeepSeek Harness profile that mounts the `commands` and `sessionQuery` services (the shipped `web` / `agent` profiles qualify).
|
|
17
|
+
|
|
16
18
|
```bash
|
|
17
19
|
dsh plugin --profile web add dsh-session-eval
|
|
18
20
|
```
|
|
19
21
|
|
|
22
|
+
Or from GitHub:
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
dsh plugin --profile web add github:kittimzhe/dsh-session-eval
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
The bundle overlay mounts `/eval`, `/eval-diff`, and `/eval-history`. If you maintain your own `cordis.patch.yml`, keep this row:
|
|
29
|
+
|
|
30
|
+
```yaml
|
|
31
|
+
- insert:
|
|
32
|
+
- id: session-eval
|
|
33
|
+
name: 'dsh-session-eval'
|
|
34
|
+
```
|
|
35
|
+
|
|
20
36
|
## Try it once
|
|
21
37
|
|
|
22
38
|
In DeepSeek Harness:
|
|
@@ -26,7 +42,7 @@ In DeepSeek Harness:
|
|
|
26
42
|
```
|
|
27
43
|
|
|
28
44
|
```text
|
|
29
|
-
Session eval — a1b2c3d4… (dsh-session-eval v0.2.
|
|
45
|
+
Session eval — a1b2c3d4… (dsh-session-eval v0.2.2)
|
|
30
46
|
Overall: A
|
|
31
47
|
Reliability A 2/87 tool results errored (2.3%)
|
|
32
48
|
Re-ask A 1 re-ask signal(s) over 23 turns (0.4/10 turns)
|
|
@@ -49,7 +65,7 @@ All three read through the same trusted `ctx.sessionQuery` seam, so any persiste
|
|
|
49
65
|
|
|
50
66
|
- **Local dev**: `npm ci && npm run typecheck && npm test && npm run bundle` (Node 20 or 22).
|
|
51
67
|
- **Start in the source**: [`src/grade.ts`](src/grade.ts) (thresholds, grade cards), [`src/metrics.ts`](src/metrics.ts) (metric extraction), [`src/trend.ts`](src/trend.ts) (`/eval-history` trend logic). See [CONTRIBUTING.md](CONTRIBUTING.md) for the source map and first-PR suggestions.
|
|
52
|
-
- **Open gaps**:
|
|
68
|
+
- **Open gaps**: [#1](https://github.com/kittimzhe/dsh-session-eval/issues/1) (configurable thresholds), [#2](https://github.com/kittimzhe/dsh-session-eval/issues/2) (`/eval-history --since`), [#3](https://github.com/kittimzhe/dsh-session-eval/issues/3) (`/eval-history --out`) — or browse [issues labeled `good first issue`](https://github.com/kittimzhe/dsh-session-eval/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
|
|
53
69
|
- **Rules**: behavior changes need tests; doc changes must update `README.md` and `README.zh.md` in sync; version releases belong to the maintainer. Details: [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
54
70
|
|
|
55
71
|
## How it differs from benchmark-style evaluation
|
|
@@ -99,6 +115,19 @@ Diff verdicts include a noise band (±2pp error rate, ±0.5 re-ask, ±1 tool loa
|
|
|
99
115
|
|
|
100
116
|
Same session logs in → same grade card out, every time, forever. There is no model in the loop, no sampling, no clock dependence in the metrics (wall time is reported, never graded). This is what makes `/eval-diff` trustworthy as a regression signal.
|
|
101
117
|
|
|
118
|
+
## Development
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
npm ci
|
|
122
|
+
npm run typecheck # tsc --noEmit
|
|
123
|
+
npm test # vitest run
|
|
124
|
+
npm run bundle # tsdown → lib/
|
|
125
|
+
```
|
|
126
|
+
|
|
102
127
|
## License
|
|
103
128
|
|
|
104
129
|
[MIT](LICENSE)
|
|
130
|
+
|
|
131
|
+
## Community
|
|
132
|
+
|
|
133
|
+
- [Contributing](CONTRIBUTING.md) · [Security policy](SECURITY.md) · [Code of Conduct](CODE_OF_CONDUCT.md)
|
package/README.zh.md
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
[English](README.md) | 中文
|
|
4
4
|
|
|
5
5
|
[](https://www.npmjs.com/package/dsh-session-eval)
|
|
6
|
+
[](https://www.npmjs.com/package/dsh-session-eval)
|
|
6
7
|
[](https://github.com/kittimzhe/dsh-session-eval/actions/workflows/test.yml)
|
|
7
8
|
[](LICENSE)
|
|
8
9
|
|
|
@@ -12,10 +13,26 @@
|
|
|
12
13
|
|
|
13
14
|
## 安装
|
|
14
15
|
|
|
16
|
+
**环境要求**:Node.js 20 或 22 · 挂载了 `commands` 与 `sessionQuery` 服务的 DeepSeek Harness profile(官方 `web` / `agent` profile 均满足)。
|
|
17
|
+
|
|
15
18
|
```bash
|
|
16
19
|
dsh plugin --profile web add dsh-session-eval
|
|
17
20
|
```
|
|
18
21
|
|
|
22
|
+
或从 GitHub:
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
dsh plugin --profile web add github:kittimzhe/dsh-session-eval
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
bundle 覆盖层会挂上 `/eval`、`/eval-diff`、`/eval-history`。若你维护自己的 `cordis.patch.yml`,请保留这一行:
|
|
29
|
+
|
|
30
|
+
```yaml
|
|
31
|
+
- insert:
|
|
32
|
+
- id: session-eval
|
|
33
|
+
name: 'dsh-session-eval'
|
|
34
|
+
```
|
|
35
|
+
|
|
19
36
|
## 试一次
|
|
20
37
|
|
|
21
38
|
在 DeepSeek Harness 里:
|
|
@@ -25,7 +42,7 @@ dsh plugin --profile web add dsh-session-eval
|
|
|
25
42
|
```
|
|
26
43
|
|
|
27
44
|
```text
|
|
28
|
-
Session eval — a1b2c3d4… (dsh-session-eval v0.2.
|
|
45
|
+
Session eval — a1b2c3d4… (dsh-session-eval v0.2.2)
|
|
29
46
|
Overall: A
|
|
30
47
|
Reliability A 2/87 tool results errored (2.3%)
|
|
31
48
|
Re-ask A 1 re-ask signal(s) over 23 turns (0.4/10 turns)
|
|
@@ -48,7 +65,7 @@ Notes:
|
|
|
48
65
|
|
|
49
66
|
- **本地开发**:`npm ci && npm run typecheck && npm test && npm run bundle`(Node 20 或 22)。
|
|
50
67
|
- **源码入口**:[`src/grade.ts`](src/grade.ts)(阈值与等级卡)、[`src/metrics.ts`](src/metrics.ts)(指标提取)、[`src/trend.ts`](src/trend.ts)(`/eval-history` 趋势逻辑)。源码地图与第一次 PR 建议见 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
|
51
|
-
-
|
|
68
|
+
- **当前缺口**:[#1](https://github.com/kittimzhe/dsh-session-eval/issues/1)(阈值可配置)、[#2](https://github.com/kittimzhe/dsh-session-eval/issues/2)(`/eval-history --since`)、[#3](https://github.com/kittimzhe/dsh-session-eval/issues/3)(`/eval-history --out`)——或浏览 [`good first issue` 标签](https://github.com/kittimzhe/dsh-session-eval/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22)。
|
|
52
69
|
- **规矩**:行为变更必须带测试;文档必须 `README.md` 与 `README.zh.md` 同步改;版本发布由维护者执行。详见 [CONTRIBUTING.md](CONTRIBUTING.md)。
|
|
53
70
|
|
|
54
71
|
## 与基准式评测(dsh-eval)的区别
|
|
@@ -96,6 +113,19 @@ Diff 判定带噪声带(错误率 ±2pp、Re-ask ±0.5、工具负载 ±1)
|
|
|
96
113
|
|
|
97
114
|
同样的日志进 → 同样的等级出。没有模型在环、没有采样、指标不含时钟依赖(wall time 只报告不评级)。这是 `/eval-diff` 能作为回归信号被信任的原因。
|
|
98
115
|
|
|
116
|
+
## 开发
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
npm ci
|
|
120
|
+
npm run typecheck # tsc --noEmit
|
|
121
|
+
npm test # vitest run
|
|
122
|
+
npm run bundle # tsdown → lib/
|
|
123
|
+
```
|
|
124
|
+
|
|
99
125
|
## 许可
|
|
100
126
|
|
|
101
127
|
[MIT](LICENSE)
|
|
128
|
+
|
|
129
|
+
## 社区
|
|
130
|
+
|
|
131
|
+
- [贡献指南](CONTRIBUTING.md) · [安全策略](SECURITY.md) · [行为准则](CODE_OF_CONDUCT.md)
|
package/lib/index.d.ts
CHANGED
|
@@ -177,7 +177,7 @@ declare function executeEvalHistory(ctx: Context, invocation: CommandInvocation,
|
|
|
177
177
|
*
|
|
178
178
|
* @module dsh-session-eval/version
|
|
179
179
|
*/
|
|
180
|
-
declare const VERSION = "0.2.
|
|
180
|
+
declare const VERSION = "0.2.3";
|
|
181
181
|
//#endregion
|
|
182
182
|
//#region src/index.d.ts
|
|
183
183
|
declare const name = "session-eval";
|
package/lib/index.js
CHANGED
|
@@ -227,7 +227,7 @@ function compareSessions(before, after) {
|
|
|
227
227
|
*
|
|
228
228
|
* @module dsh-session-eval/version
|
|
229
229
|
*/
|
|
230
|
-
const VERSION = "0.2.
|
|
230
|
+
const VERSION = "0.2.3";
|
|
231
231
|
//#endregion
|
|
232
232
|
//#region src/evalCommand.ts
|
|
233
233
|
const EVAL_USAGE = "Usage: /eval [--id <sessionId>] [--json]";
|
|
@@ -327,14 +327,14 @@ async function executeEval(ctx, invocation, seam = ctx.sessionQuery) {
|
|
|
327
327
|
if (parsed.json === true) return {
|
|
328
328
|
kind: "success",
|
|
329
329
|
text: JSON.stringify({
|
|
330
|
-
generator: "dsh-session-eval v0.2.
|
|
330
|
+
generator: "dsh-session-eval v0.2.3",
|
|
331
331
|
metrics,
|
|
332
332
|
card
|
|
333
333
|
}, null, 2)
|
|
334
334
|
};
|
|
335
335
|
return {
|
|
336
336
|
kind: "success",
|
|
337
|
-
text: renderCard(card, "dsh-session-eval v0.2.
|
|
337
|
+
text: renderCard(card, "dsh-session-eval v0.2.3")
|
|
338
338
|
};
|
|
339
339
|
}
|
|
340
340
|
/** Execute /eval-diff against the session-query seam. */
|
|
@@ -358,13 +358,13 @@ async function executeEvalDiff(ctx, invocation, seam = ctx.sessionQuery) {
|
|
|
358
358
|
if (parsed.json === true) return {
|
|
359
359
|
kind: "success",
|
|
360
360
|
text: JSON.stringify({
|
|
361
|
-
generator: "dsh-session-eval v0.2.
|
|
361
|
+
generator: "dsh-session-eval v0.2.3",
|
|
362
362
|
report
|
|
363
363
|
}, null, 2)
|
|
364
364
|
};
|
|
365
365
|
return {
|
|
366
366
|
kind: "success",
|
|
367
|
-
text: renderTrend(report, "dsh-session-eval v0.2.
|
|
367
|
+
text: renderTrend(report, "dsh-session-eval v0.2.3")
|
|
368
368
|
};
|
|
369
369
|
}
|
|
370
370
|
//#endregion
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-session-eval",
|
|
3
3
|
"description": "Retrospective agent-session evaluation for DeepSeek Harness: deterministic grade cards from persisted session logs, cross-session regression diffs, era comparison — no benchmark runs, no LLM judges.",
|
|
4
|
-
"version": "0.2.
|
|
4
|
+
"version": "0.2.3",
|
|
5
5
|
"publishConfig": {
|
|
6
6
|
"access": "public"
|
|
7
7
|
},
|