dsh-session-eval 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +21 -2
- package/README.zh.md +4 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -45,7 +45,7 @@ Then in DeepSeek Harness:
|
|
|
45
45
|
```
|
|
46
46
|
|
|
47
47
|
```text
|
|
48
|
-
Session eval — a1b2c3d4… (dsh-session-eval v0.
|
|
48
|
+
Session eval — a1b2c3d4… (dsh-session-eval v0.2.0)
|
|
49
49
|
Overall: A
|
|
50
50
|
Reliability A 2/87 tool results errored (2.3%)
|
|
51
51
|
Re-ask A 1 re-ask signal(s) over 23 turns (0.4/10 turns)
|
|
@@ -67,7 +67,7 @@ Compare two sessions — the regression workflow:
|
|
|
67
67
|
```
|
|
68
68
|
|
|
69
69
|
```text
|
|
70
|
-
Session diff — a1b2c3d4… → e5f6a7b8… (dsh-session-eval v0.
|
|
70
|
+
Session diff — a1b2c3d4… → e5f6a7b8… (dsh-session-eval v0.2.0)
|
|
71
71
|
Overall: regressed — 0 improved, 2 regressed, 1 flat.
|
|
72
72
|
Reliability 2.1% → 11.8% regressed
|
|
73
73
|
Re-ask 0.5 → 3.2 regressed
|
|
@@ -76,6 +76,23 @@ Overall: regressed — 0 improved, 2 regressed, 1 flat.
|
|
|
76
76
|
|
|
77
77
|
Typical use: grade the ten sessions before and after a plugin or prompt change; a persistent `Reliability` regression is an early warning that the change hurt real work, not just benchmarks.
|
|
78
78
|
|
|
79
|
+
See the whole trend — no ids to hunt for:
|
|
80
|
+
|
|
81
|
+
```text
|
|
82
|
+
/eval-history # last 5 sessions in this workspace
|
|
83
|
+
/eval-history 12 # last 12
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
```text
|
|
87
|
+
Session history — last 5 in this workspace
|
|
88
|
+
2026-09-20 14:02 a1b2c3d4… B 4 turns, 9 tools
|
|
89
|
+
2026-09-21 09:15 c3d4e5f6… A 5 turns, 11 tools
|
|
90
|
+
2026-09-22 17:44 e5f6a7b8… A 3 turns, 7 tools
|
|
91
|
+
Trend (first → last): improved — 2 improved, 0 regressed, 1 flat.
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
`/eval-history` finds the sessions for you via the same `sessionQuery` seam `/archive` uses (scoped to the current workspace's `cwd`), grades each, and diffs the first against the last. `/eval-diff` remains the tool when you already know the two ids that matter.
|
|
95
|
+
|
|
79
96
|
## Dimensions & thresholds
|
|
80
97
|
|
|
81
98
|
All grades derive from three deterministic metrics. `n/a` when the session lacks the relevant activity.
|
|
@@ -105,6 +122,8 @@ Same session logs in → same grade card out, every time, forever. There is no m
|
|
|
105
122
|
| `/eval --json` | Machine-readable metrics + card |
|
|
106
123
|
| `/eval-diff <beforeId> <afterId>` | Regression comparison, noise-tolerant |
|
|
107
124
|
| `/eval-diff … --json` | Machine-readable trend report |
|
|
125
|
+
| `/eval-history [N]` | Grade the last N sessions in this workspace and show the trend |
|
|
126
|
+
| `/eval-history … --json` | Machine-readable history report |
|
|
108
127
|
|
|
109
128
|
## Development
|
|
110
129
|
|
package/README.zh.md
CHANGED
|
@@ -37,8 +37,12 @@ npm install -g dsh-session-eval
|
|
|
37
37
|
/eval # 当前会话等级卡
|
|
38
38
|
/eval --id <会话ID> # 指定会话
|
|
39
39
|
/eval-diff <前> <后> # 回归对比（换插件/prompt 前后各评一次）
|
|
40
|
+
/eval-history # 本工作区最近 5 个会话的成绩单 + 首末趋势
|
|
41
|
+
/eval-history 12 # 最近 12 个
|
|
40
42
|
```
|
|
41
43
|
|
|
44
|
+
`/eval-history` 通过与 `/archive` 相同的 `sessionQuery` 接缝自动找到会话（按当前工作区 `cwd` 过滤），逐个评级，并把首尾两个会话做趋势对比——不用再手动挑 id。
|
|
45
|
+
|
|
42
46
|
## 维度与阈值（确定性，无裁判）
|
|
43
47
|
|
|
44
48
|
| 维度 | 指标 | A | B | C | D | E |
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-session-eval",
|
|
3
3
|
"description": "Retrospective agent-session evaluation for DeepSeek Harness: deterministic grade cards from persisted session logs, cross-session regression diffs, era comparison — no benchmark runs, no LLM judges.",
|
|
4
|
-
"version": "0.
|
|
4
|
+
"version": "0.2.0",
|
|
5
5
|
"publishConfig": {
|
|
6
6
|
"access": "public"
|
|
7
7
|
},
|