dsh-session-eval 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 kittimzhe
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,120 @@
+# dsh-session-eval
+
+English | [中文](README.zh.md)
+
+[![npm](https://img.shields.io/npm/v/dsh-session-eval)](https://www.npmjs.com/package/dsh-session-eval)
+[![npm downloads](https://img.shields.io/npm/dw/dsh-session-eval)](https://www.npmjs.com/package/dsh-session-eval)
+[![tests](https://github.com/kittimzhe/dsh-session-eval/actions/workflows/test.yml/badge.svg)](https://github.com/kittimzhe/dsh-session-eval/actions/workflows/test.yml)
+[![license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+**Retrospective evaluation for DeepSeek Harness agent sessions** — deterministic grade cards computed from sessions that already happened, plus cross-session regression diffs that answer *"did my agent get better or worse after that plugin / prompt / model change?"*
+
+No benchmark suites to author. No LLM judges. No new runs. Every grade is a fixed, documented threshold over persisted session logs — fully reproducible.
+
+## The measurement layer of the session toolchain
+
+| Plugin | Layer | Answers |
+|---|---|---|
+| [`dsh-session-export`](https://www.npmjs.com/package/dsh-session-export) | Evidence | "What exactly happened in this session?" |
+| [`dsh-session-recall`](https://www.npmjs.com/package/dsh-session-recall) | Memory | "What did I do before, and where is it?" |
+| **`dsh-session-eval`** | **Measurement** | **"Was that session good? Is the trend improving?"** |
+
+All three read through the same trusted `ctx.sessionQuery` seam, so any persistence backend (JSONL or SQLite) works without touching raw artifacts.
+
+## How it differs from benchmark-style evaluation
+
+| | `dsh-eval` (benchmark) | `dsh-session-eval` (this) |
+|---|---|---|
+| Evaluates | Runs you launch ahead of time | Sessions that already happened |
+| Setup | Author benchmark YAML + cases | None — read existing logs |
+| Judge | Metric folding over trials | Fixed thresholds, deterministic |
+| Best at | Pre-deploy benchmarks | Regression tracking over real work |
+
+Both are useful; they don't compete. Benchmarks test what you *predicted*; retrospective evaluation watches what *actually happened*.
+
+## Quick Start
+
+```bash
+npm install -g dsh-session-eval
+```
+
+Then in DeepSeek Harness:
+
+```text
+/eval
+```
+
+```text
+Session eval — a1b2c3d4… (dsh-session-eval v0.1.0)
+Overall: A
+  Reliability   A    2/87 tool results errored (2.3%)
+  Re-ask        A    1 re-ask signal(s) over 23 turns (0.4/10 turns)
+  Tool load     B    3.8 tool calls per turn
+Notes:
+  • Wall time 41.2 min over 23 turns.
+```
+
+Grade another session, or emit machine-readable JSON:
+
+```text
+/eval --id <sessionId> --json
+```
+
+Compare two sessions — the regression workflow:
+
+```text
+/eval-diff <beforeSessionId> <afterSessionId>
+```
+
+```text
+Session diff — a1b2c3d4… → e5f6a7b8… (dsh-session-eval v0.1.0)
+Overall: regressed — 0 improved, 2 regressed, 1 flat.
+  Reliability   2.1% → 11.8%   regressed
+  Re-ask        0.5 → 3.2      regressed
+  Tool load     3.8 → 4.1      flat
+```
+
+Typical use: grade the ten sessions before and after a plugin or prompt change; a persistent `Reliability` regression is an early warning that the change hurt real work, not just benchmarks.
+
+## Dimensions & thresholds
+
+All grades derive from three deterministic metrics. `n/a` when the session lacks the relevant activity.
+
+| Dimension | Metric | A | B | C | D | E |
+|---|---|---|---|---|---|---|
+| Reliability | tool error rate | ≤5% | ≤10% | ≤20% | ≤35% | >35% |
+| Re-ask | corrections per 10 turns¹ | 0 | <1 | <2 | <4 | ≥4 |
+| Tool load | tool calls per turn | 0.5–8 | 0.1–0.5 or just above 8 | <0.1 or 8–20 | >20 | — |
+
+¹ A *correction signal* is each extra user message inside a consecutive-user run — the human re-asking before any assistant reply, a behavioral proxy for "the agent didn't get it right the first time."
+
+`Overall` is the **worst** graded dimension — conservative by design. Thresholds are constants in [`src/grade.ts`](src/grade.ts); they will be configurable in a future release.
+
+Diff verdicts include a noise band (±2pp error rate, ±0.5 re-ask, ±1 tool load) so trivial jitter doesn't cry regression.
+
+## Determinism guarantee
+
+Same session logs in → same grade card out, every time, forever. There is no model in the loop, no sampling, no clock dependence in the metrics (wall time is reported, never graded). This is what makes `/eval-diff` trustworthy as a regression signal.
+
+## Commands
+
+| Command | Effect |
+|---|---|
+| `/eval` | Grade card for the current session |
+| `/eval --id <sessionId>` | Grade a specific session |
+| `/eval --json` | Machine-readable metrics + card |
+| `/eval-diff <beforeId> <afterId>` | Regression comparison, noise-tolerant |
+| `/eval-diff … --json` | Machine-readable trend report |
+
+## Development
+
+```bash
+npm install
+npm run typecheck   # tsc --noEmit
+npm test            # vitest run — 24 tests
+npm run bundle      # tsdown → lib/
+```
+
+## License
+
+[MIT](LICENSE)

package/README.zh.md

@@ -0,0 +1,67 @@
+# dsh-session-eval
+
+[![npm](https://img.shields.io/npm/v/dsh-session-eval)](https://www.npmjs.com/package/dsh-session-eval)
+[![tests](https://github.com/kittimzhe/dsh-session-eval/actions/workflows/test.yml/badge.svg)](https://github.com/kittimzhe/dsh-session-eval/actions/workflows/test.yml)
+[![license](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+**DeepSeek Harness 会话的回溯式评测**：对已经发生的会话做确定性等级评定，跨会话回归对比，回答"换了插件 / prompt / 模型之后，我的 agent 是变好了还是变差了？"
+
+不需要编写基准测试集，没有 LLM 裁判，不需要跑新任务。所有等级 = 持久化会话日志上的固定公开阈值——完全可复现。
+
+English | [中文](README.zh.md)
+
+## 会话工具链的评测层
+
+| 插件 | 层 | 回答的问题 |
+|---|---|---|
+| `dsh-session-export` | 证据层 | "这个会话到底发生了什么？" |
+| `dsh-session-recall` | 记忆层 | "我以前做过什么，在哪？" |
+| **`dsh-session-eval`** | **评测层** | **"刚才的会话好不好？趋势在变好吗？"** |
+
+三者都通过同一个可信的 `ctx.sessionQuery` 缝读取，JSONL / SQLite 等任何持久化后端均可使用。
+
+## 与基准式评测（dsh-eval）的区别
+
+- `dsh-eval`：**前瞻**——预先写好基准 YAML，跑无头评测，评的是"你预测到的场景"
+- 本插件：**回溯**——直接读存量日志，评的是"真实发生的工作"，零准备、零额外 token
+
+两者互补而非竞争。
+
+## 快速开始
+
+```bash
+npm install -g dsh-session-eval
+```
+
+```text
+/eval                 # 当前会话等级卡
+/eval --id <会话ID>    # 指定会话
+/eval-diff <前> <后>   # 回归对比（换插件/prompt 前后各评一次）
+```
+
+## 维度与阈值（确定性，无裁判）
+
+| 维度 | 指标 | A | B | C | D | E |
+|---|---|---|---|---|---|---|
+| Reliability | 工具错误率 | ≤5% | ≤10% | ≤20% | ≤35% | >35% |
+| Re-ask | 每 10 轮纠正信号¹ | 0 | <1 | <2 | <4 | ≥4 |
+| Tool load | 每轮工具调用 | 0.5–8 | 0.1–0.5 | <0.1 或 8–20 | >20 | — |
+
+¹ 纠正信号 = 连续用户消息中多出的那条（agent 还没回复用户就重问）——"第一次没做对"的行为学代理指标。
+
+总评 = 最差维度（保守设计）。Diff 判定带噪声带（错误率 ±2pp 等），微小抖动不会误报回归。
+
+## 确定性保证
+
+同样的日志进 → 同样的等级出。没有模型在环、没有采样、指标不含时钟依赖（wall time 只报告不评级）。这是 `/eval-diff` 能作为回归信号被信任的原因。
+
+## 开发
+
+```bash
+npm install && npm run typecheck && npm test   # 24 个测试
+npm run bundle
+```
+
+## 许可
+
+[MIT](LICENSE)

package/lib/index.d.ts

@@ -0,0 +1,141 @@
+import { SessionId } from "@deepseek-ai/dsh-session";
+import { Context } from "@deepseek-ai/cordis";
+import { CommandInvocation, CommandResult } from "@deepseek-ai/dsh-commands";
+//#region src/types.d.ts
+/**
+ * Core types for retrospective session evaluation.
+ *
+ * The plugin projects raw session events into a small event vocabulary once
+ * (see {@link adaptEvents}), then every metric, grade, and trend computation
+ * is pure over plain data — no LLM judges, no network, no nondeterminism.
+ *
+ * @module dsh-session-eval/types
+ */
+/** Event kinds the evaluation layer reasons about. */
+type EvalEventKind = 'user' | 'assistant' | 'tool_call' | 'tool_result';
+/** One projected session event. */
+interface EvalEvent {
+  /** Raw-log sequence number of the source event. */
+  readonly seq: number;
+  /** Event time in epoch milliseconds. */
+  readonly time: number;
+  /** Projected kind. */
+  readonly kind: EvalEventKind;
+  /** Tool name for `tool_call` / `tool_result` events. */
+  readonly tool?: string;
+  /** Whether a `tool_result` reported an error. */
+  readonly isError?: boolean;
+}
+/** Deterministic per-session metrics. */
+interface SessionMetrics {
+  readonly sessionId: string;
+  readonly events: number;
+  readonly userTurns: number;
+  readonly assistantMessages: number;
+  readonly toolCalls: number;
+  readonly toolErrors: number;
+  /** toolErrors / max(1, toolCalls). 0 when no tool calls. */
+  readonly toolErrorRate: number;
+  /** Extra user messages inside consecutive-user runs (re-ask signals). */
+  readonly corrections: number;
+  /** corrections per 10 user turns. */
+  readonly correctionRate: number;
+  /** toolCalls / max(1, userTurns). */
+  readonly toolLoad: number;
+  /** wall-clock span in ms (last.time - first.time); 0 for empty sessions. */
+  readonly wallMs: number;
+  readonly firstTime: number | null;
+  readonly lastTime: number | null;
+}
+/** Letter grade for one dimension. */
+type Grade = 'A' | 'B' | 'C' | 'D' | 'E';
+/** One graded dimension. */
+interface DimensionGrade {
+  readonly dimension: 'Reliability' | 'Re-ask' | 'Tool load';
+  readonly grade: Grade | 'n/a';
+  readonly detail: string;
+}
+/** Deterministic grade card for one session. */
+interface GradeCard {
+  readonly sessionId: string;
+  readonly dimensions: readonly DimensionGrade[];
+  /** Worst non-n/a dimension grade; 'n/a' when all dimensions are n/a. */
+  readonly overall: Grade | 'n/a';
+  readonly notes: readonly string[];
+}
+/** Verdict for one dimension between two sessions. */
+interface DimensionDelta {
+  readonly dimension: DimensionGrade['dimension'];
+  readonly before: string;
+  readonly after: string;
+  readonly verdict: 'improved' | 'regressed' | 'flat';
+}
+/** Cross-session regression report. */
+interface TrendReport {
+  readonly before: SessionMetrics;
+  readonly after: SessionMetrics;
+  readonly deltas: readonly DimensionDelta[];
+  readonly overall: 'improved' | 'regressed' | 'flat';
+  readonly summary: string;
+}
+//#endregion
+//#region src/metrics.d.ts
+/** Compute the deterministic metric set for one session. */
+declare function computeMetrics(sessionId: string, events: readonly EvalEvent[]): SessionMetrics;
+/**
+ * Project raw session-log events into the evaluation vocabulary.
+ *
+ * Defensive by design: the adapter matches the `user/message`,
+ * `assistant/message`, `tool/call`, `tool/result` event-type vocabulary
+ * (plus role-based fallbacks) and never throws — unknown shapes are skipped.
+ */
+declare function adaptEvents(raw: readonly unknown[]): EvalEvent[];
+//#endregion
+//#region src/grade.d.ts
+/** Grade one session deterministically. */
+declare function gradeSession(m: SessionMetrics): GradeCard;
+//#endregion
+//#region src/trend.d.ts
+/** Compare two sessions dimension by dimension. */
+declare function compareSessions(before: SessionMetrics, after: SessionMetrics): TrendReport;
+//#endregion
+//#region src/evalCommand.d.ts
+declare const EVAL_USAGE = "Usage: /eval [--id <sessionId>] [--json]";
+declare const EVAL_DIFF_USAGE = "Usage: /eval-diff <beforeSessionId> <afterSessionId> [--json]";
+declare function id8(sessionId: string): string;
+/** Args for /eval. */
+interface EvalArgs {
+  readonly sessionId?: string;
+  readonly json?: boolean;
+}
+/** Args for /eval-diff. */
+interface EvalDiffArgs {
+  readonly beforeId: string;
+  readonly afterId: string;
+  readonly json?: boolean;
+}
+/** Parse /eval input; returns args or a usage-error string. */
+declare function parseEvalArgs(rawInput: string): EvalArgs | string;
+/** Parse /eval-diff input; returns args or a usage-error string. */
+declare function parseEvalDiffArgs(rawInput: string): EvalDiffArgs | string;
+/** Render a grade card as terminal text. */
+declare function renderCard(card: GradeCard, generator: string): string;
+/** Render a trend report as terminal text. */
+declare function renderTrend(report: TrendReport, generator: string): string;
+interface QuerySeam {
+  readSession(id: ReturnType<typeof SessionId>): Promise<{
+    events: readonly unknown[];
+  }>;
+}
+/** Execute /eval against the session-query seam. */
+declare function executeEval(ctx: Context, invocation: CommandInvocation, seam?: QuerySeam): Promise<CommandResult>;
+/** Execute /eval-diff against the session-query seam. */
+declare function executeEvalDiff(ctx: Context, invocation: CommandInvocation, seam?: QuerySeam): Promise<CommandResult>;
+//#endregion
+//#region src/index.d.ts
+declare const name = "session-eval";
+declare const inject: string[];
+/** Plugin entry: mount the /eval and /eval-diff commands. */
+declare function apply(ctx: Context): void;
+//#endregion
+export { type DimensionDelta, type DimensionGrade, EVAL_DIFF_USAGE, EVAL_USAGE, type EvalArgs, type EvalDiffArgs, type EvalEvent, type EvalEventKind, type Grade, type GradeCard, type SessionMetrics, type TrendReport, adaptEvents, apply, compareSessions, computeMetrics, executeEval, executeEvalDiff, gradeSession, id8, inject, name, parseEvalArgs, parseEvalDiffArgs, renderCard, renderTrend };

package/lib/index.js

@@ -0,0 +1,382 @@
+import { SessionId } from "@deepseek-ai/dsh-session";
+//#region src/metrics.ts
+/** Count a correction signal for every extra user message inside a consecutive-user run. */
+function countCorrections(events) {
+	let corrections = 0;
+	let runLength = 0;
+	for (const event of events) if (event.kind === "user") {
+		runLength += 1;
+		if (runLength >= 2) corrections += 1;
+	} else runLength = 0;
+	return corrections;
+}
+/** Compute the deterministic metric set for one session. */
+function computeMetrics(sessionId, events) {
+	const userTurns = events.filter((e) => e.kind === "user").length;
+	const assistantMessages = events.filter((e) => e.kind === "assistant").length;
+	const toolEvents = events.filter((e) => e.kind === "tool_call");
+	const toolResults = events.filter((e) => e.kind === "tool_result");
+	const toolCalls = Math.max(toolEvents.length, toolResults.length);
+	const toolErrors = toolResults.filter((e) => e.isError === true).length;
+	const corrections = countCorrections(events);
+	const times = events.map((e) => e.time).filter((t) => Number.isFinite(t));
+	const firstTime = times.length > 0 ? Math.min(...times) : null;
+	const lastTime = times.length > 0 ? Math.max(...times) : null;
+	return {
+		sessionId,
+		events: events.length,
+		userTurns,
+		assistantMessages,
+		toolCalls,
+		toolErrors,
+		toolErrorRate: toolCalls > 0 ? toolErrors / toolCalls : 0,
+		corrections,
+		correctionRate: userTurns > 0 ? corrections / userTurns * 10 : 0,
+		toolLoad: userTurns > 0 ? toolCalls / userTurns : 0,
+		wallMs: firstTime !== null && lastTime !== null ? lastTime - firstTime : 0,
+		firstTime,
+		lastTime
+	};
+}
+/**
+* Project raw session-log events into the evaluation vocabulary.
+*
+* Defensive by design: the adapter matches the `user/message`,
+* `assistant/message`, `tool/call`, `tool/result` event-type vocabulary
+* (plus role-based fallbacks) and never throws — unknown shapes are skipped.
+*/
+function adaptEvents(raw) {
+	const out = [];
+	for (let i = 0; i < raw.length; i += 1) {
+		const event = raw[i];
+		if (event == null || typeof event !== "object") continue;
+		const seq = typeof event.seq === "number" ? event.seq : i;
+		const time = typeof event.time === "number" ? event.time : 0;
+		const message = event.message ?? event.payload;
+		const kind = classify(String(event.type ?? message?.type ?? ""), message);
+		if (kind === null) continue;
+		const isError = readBool(message, "isError") ?? readBool(message, "is_error") ?? readBool(event, "isError");
+		const tool = typeof message?.tool === "string" ? message.tool : typeof message?.name === "string" ? message.name : void 0;
+		out.push({
+			seq,
+			time,
+			kind,
+			tool: kind === "tool_call" || kind === "tool_result" ? tool : void 0,
+			isError: kind === "tool_result" ? isError === true : void 0
+		});
+	}
+	return out.sort((a, b) => a.seq - b.seq);
+}
+function classify(type, message) {
+	const t = type.toLowerCase();
+	if (t.includes("user/message") || t === "user") return "user";
+	if (t.includes("assistant/message") || t === "assistant") return "assistant";
+	if (t.includes("tool/call") || t.includes("tool_call")) return "tool_call";
+	if (t.includes("tool/result") || t.includes("tool_result")) return "tool_result";
+	const role = message?.role;
+	if (role === "user") return "user";
+	if (role === "assistant") return "assistant";
+	return null;
+}
+function readBool(source, key) {
+	const value = source?.[key];
+	return typeof value === "boolean" ? value : void 0;
+}
+//#endregion
+//#region src/grade.ts
+/** Reliability thresholds on tool error rate (fraction). */
+const RELIABILITY = [
+	["A", .05],
+	["B", .1],
+	["C", .2],
+	["D", .35]
+];
+/** Re-ask thresholds: corrections per 10 user turns. */
+const REASK = [
+	["A", 0],
+	["B", 1],
+	["C", 2],
+	["D", 4]
+];
+/** Tool-load thresholds: tool calls per user turn (both churn extremes graded). */
+const TOOL_LOAD_LOW_B = .5;
+const TOOL_LOAD_LOW_C = .1;
+const TOOL_LOAD_HIGH_C = 8;
+const TOOL_LOAD_HIGH_D = 20;
+function gradeBy(rate, table) {
+	for (const [grade, max] of table) if (rate <= max) return grade;
+	return "E";
+}
+const ORDER = [
+	"A",
+	"B",
+	"C",
+	"D",
+	"E"
+];
+function worst(a, b) {
+	return ORDER.indexOf(a) >= ORDER.indexOf(b) ? a : b;
+}
+function pct(rate) {
+	return `${(rate * 100).toFixed(1)}%`;
+}
+/** Grade one session deterministically. */
+function gradeSession(m) {
+	const dimensions = [];
+	const notes = [];
+	if (m.toolCalls === 0) {
+		dimensions.push({
+			dimension: "Reliability",
+			grade: "n/a",
+			detail: "no tool calls"
+		});
+		notes.push("Reliability is n/a — the session made no tool calls.");
+	} else {
+		const grade = gradeBy(m.toolErrorRate, RELIABILITY);
+		dimensions.push({
+			dimension: "Reliability",
+			grade,
+			detail: `${m.toolErrors}/${m.toolCalls} tool results errored (${pct(m.toolErrorRate)})`
+		});
+	}
+	if (m.userTurns === 0) dimensions.push({
+		dimension: "Re-ask",
+		grade: "n/a",
+		detail: "no user turns"
+	});
+	else {
+		const grade = gradeBy(m.correctionRate, REASK);
+		dimensions.push({
+			dimension: "Re-ask",
+			grade,
+			detail: `${m.corrections} re-ask signal(s) over ${m.userTurns} turns (${m.correctionRate.toFixed(1)}/10 turns)`
+		});
+	}
+	if (m.userTurns === 0 || m.toolCalls === 0) dimensions.push({
+		dimension: "Tool load",
+		grade: "n/a",
+		detail: "no tool activity"
+	});
+	else {
+		let grade = "A";
+		if (m.toolLoad < TOOL_LOAD_LOW_B) grade = m.toolLoad < TOOL_LOAD_LOW_C ? "C" : "B";
+		if (m.toolLoad > TOOL_LOAD_HIGH_C) grade = worst(grade, m.toolLoad > TOOL_LOAD_HIGH_D ? "D" : "C");
+		dimensions.push({
+			dimension: "Tool load",
+			grade,
+			detail: `${m.toolLoad.toFixed(1)} tool calls per turn`
+		});
+	}
+	if (m.events === 0) notes.push("Session is empty — all dimensions are provisional.");
+	if (m.wallMs > 0 && m.userTurns > 0) notes.push(`Wall time ${(m.wallMs / 6e4).toFixed(1)} min over ${m.userTurns} turns.`);
+	const graded = dimensions.filter((d) => d.grade !== "n/a");
+	const overall = graded.length === 0 ? "n/a" : graded.reduce((acc, d) => worst(acc, d.grade), "A");
+	return {
+		sessionId: m.sessionId,
+		dimensions,
+		overall,
+		notes
+	};
+}
+//#endregion
+//#region src/trend.ts
+/** Minimum absolute change treated as signal (not noise). */
+const ERROR_RATE_EPSILON = .02;
+const REASK_EPSILON = .5;
+const TOOL_LOAD_EPSILON = 1;
+function verdict(before, after, eps, lowerIsBetter) {
+	const delta = after - before;
+	if (Math.abs(delta) <= eps) return "flat";
+	return (lowerIsBetter ? delta < 0 : delta > 0) ? "improved" : "regressed";
+}
+/** Compare two sessions dimension by dimension. */
+function compareSessions(before, after) {
+	const deltas = [];
+	if (before.toolCalls > 0 || after.toolCalls > 0) deltas.push({
+		dimension: "Reliability",
+		before: `${(before.toolErrorRate * 100).toFixed(1)}%`,
+		after: `${(after.toolErrorRate * 100).toFixed(1)}%`,
+		verdict: verdict(before.toolErrorRate, after.toolErrorRate, ERROR_RATE_EPSILON, true)
+	});
+	if (before.userTurns > 0 || after.userTurns > 0) deltas.push({
+		dimension: "Re-ask",
+		before: before.correctionRate.toFixed(1),
+		after: after.correctionRate.toFixed(1),
+		verdict: verdict(before.correctionRate, after.correctionRate, REASK_EPSILON, true)
+	});
+	if (before.toolCalls > 0 || after.toolCalls > 0) deltas.push({
+		dimension: "Tool load",
+		before: before.toolLoad.toFixed(1),
+		after: after.toolLoad.toFixed(1),
+		verdict: verdict(before.toolLoad, after.toolLoad, TOOL_LOAD_EPSILON, false)
+	});
+	const improved = deltas.filter((d) => d.verdict === "improved").length;
+	const regressed = deltas.filter((d) => d.verdict === "regressed").length;
+	return {
+		before,
+		after,
+		deltas,
+		overall: regressed > improved ? "regressed" : improved > regressed ? "improved" : "flat",
+		summary: deltas.length === 0 ? "No comparable dimensions — both sessions lack tool or turn activity." : `${improved} improved, ${regressed} regressed, ${deltas.length - improved - regressed} flat.`
+	};
+}
+//#endregion
+//#region src/evalCommand.ts
+const EVAL_USAGE = "Usage: /eval [--id <sessionId>] [--json]";
+const EVAL_DIFF_USAGE = "Usage: /eval-diff <beforeSessionId> <afterSessionId> [--json]";
+function id8(sessionId) {
+	return sessionId.length > 8 ? `${sessionId.slice(0, 8)}…` : sessionId;
+}
+/** Parse /eval input; returns args or a usage-error string. */
+function parseEvalArgs(rawInput) {
+	const trimmed = rawInput.trim();
+	if (trimmed.length === 0) return {};
+	const tokens = trimmed.split(/\s+/);
+	const args = {};
+	let i = 0;
+	while (i < tokens.length) {
+		const token = tokens[i];
+		if (token === void 0) break;
+		if (token === "--id") {
+			const value = tokens[i + 1];
+			if (value === void 0 || value.startsWith("--")) return `--id requires a session id value.\n${EVAL_USAGE}`;
+			if (args.sessionId !== void 0) return `--id may be given only once.\n${EVAL_USAGE}`;
+			args.sessionId = value;
+			i += 2;
+			continue;
+		}
+		if (token === "--json") {
+			if (args.json === true) return `--json may be given only once.\n${EVAL_USAGE}`;
+			args.json = true;
+			i += 1;
+			continue;
+		}
+		return `Unknown argument: ${token}\n${EVAL_USAGE}`;
+	}
+	return args;
+}
+/** Parse /eval-diff input; returns args or a usage-error string. */
+function parseEvalDiffArgs(rawInput) {
+	const tokens = rawInput.trim().split(/\s+/).filter((t) => t.length > 0);
+	let json = false;
+	const positional = [];
+	for (const token of tokens) {
+		if (token === "--json") {
+			if (json) return `--json may be given only once.\n${EVAL_DIFF_USAGE}`;
+			json = true;
+			continue;
+		}
+		if (token.startsWith("--")) return `Unknown argument: ${token}\n${EVAL_DIFF_USAGE}`;
+		positional.push(token);
+	}
+	if (positional.length !== 2) return `Expected exactly two session ids (before, after).\n${EVAL_DIFF_USAGE}`;
+	const [beforeId, afterId] = positional;
+	if (beforeId === afterId) return `The two session ids must differ.\n${EVAL_DIFF_USAGE}`;
+	return {
+		beforeId,
+		afterId,
+		json
+	};
+}
+/** Render a grade card as terminal text. */
+function renderCard(card, generator) {
+	const lines = [`Session eval — ${id8(card.sessionId)} (${generator})`, `Overall: ${card.overall}`];
+	for (const dimension of card.dimensions) lines.push(`  ${dimension.dimension.padEnd(13)} ${String(dimension.grade).padEnd(3)} ${dimension.detail}`);
+	if (card.notes.length > 0) {
+		lines.push("Notes:");
+		for (const note of card.notes) lines.push(`  • ${note}`);
+	}
+	return lines.join("\n");
+}
+/** Render a trend report as terminal text. */
+function renderTrend(report, generator) {
+	const lines = [`Session diff — ${id8(report.before.sessionId)} → ${id8(report.after.sessionId)} (${generator})`, `Overall: ${report.overall} — ${report.summary}`];
+	for (const delta of report.deltas) lines.push(`  ${delta.dimension.padEnd(13)} ${delta.before} → ${delta.after}   ${delta.verdict}`);
+	return lines.join("\n");
+}
+async function readEvents(ctx, seam, sessionIdRaw) {
+	try {
+		return (await seam.readSession(SessionId(sessionIdRaw))).events;
+	} catch (error) {
+		return `Could not read session ${id8(sessionIdRaw)}: ${error instanceof Error ? error.message : String(error)}`;
+	}
+}
+/** Execute /eval against the session-query seam. */
+async function executeEval(ctx, invocation, seam = ctx.sessionQuery) {
+	const parsed = parseEvalArgs(invocation.rawInput);
+	if (typeof parsed === "string") return {
+		kind: "error",
+		text: parsed
+	};
+	const sessionIdRaw = parsed.sessionId ?? String(invocation.agent.session.id);
+	const events = await readEvents(ctx, seam, sessionIdRaw);
+	if (typeof events === "string") return {
+		kind: "error",
+		text: events
+	};
+	const metrics = computeMetrics(sessionIdRaw, adaptEvents(events));
+	const card = gradeSession(metrics);
+	if (parsed.json === true) return {
+		kind: "success",
+		text: JSON.stringify({
+			generator: "dsh-session-eval v0.1.0",
+			metrics,
+			card
+		}, null, 2)
+	};
+	return {
+		kind: "success",
+		text: renderCard(card, "dsh-session-eval v0.1.0")
+	};
+}
+/** Execute /eval-diff against the session-query seam. */
+async function executeEvalDiff(ctx, invocation, seam = ctx.sessionQuery) {
+	const parsed = parseEvalDiffArgs(invocation.rawInput);
+	if (typeof parsed === "string") return {
+		kind: "error",
+		text: parsed
+	};
+	const beforeEvents = await readEvents(ctx, seam, parsed.beforeId);
+	if (typeof beforeEvents === "string") return {
+		kind: "error",
+		text: beforeEvents
+	};
+	const afterEvents = await readEvents(ctx, seam, parsed.afterId);
+	if (typeof afterEvents === "string") return {
+		kind: "error",
+		text: afterEvents
+	};
+	const report = compareSessions(computeMetrics(parsed.beforeId, adaptEvents(beforeEvents)), computeMetrics(parsed.afterId, adaptEvents(afterEvents)));
+	if (parsed.json === true) return {
+		kind: "success",
+		text: JSON.stringify({
+			generator: "dsh-session-eval v0.1.0",
+			report
+		}, null, 2)
+	};
+	return {
+		kind: "success",
+		text: renderTrend(report, "dsh-session-eval v0.1.0")
+	};
+}
+//#endregion
+//#region src/index.ts
+const name = "session-eval";
+const inject = ["commands", "sessionQuery"];
+/** Plugin entry: mount the /eval and /eval-diff commands. */
+function apply(ctx) {
+	ctx.effect(function* () {
+		yield ctx.commands.register({
+			name: "eval",
+			description: "Print a deterministic grade card for this session (or another via --id): reliability, re-ask, tool load",
+			handler: (invocation) => executeEval(ctx, invocation)
+		});
+		yield ctx.commands.register({
+			name: "eval-diff",
+			description: "Compare two sessions (before, after) and report which dimensions improved or regressed",
+			handler: (invocation) => executeEvalDiff(ctx, invocation)
+		});
+	}, "session-eval lifecycle");
+}
+//#endregion
+export { EVAL_DIFF_USAGE, EVAL_USAGE, adaptEvents, apply, compareSessions, computeMetrics, executeEval, executeEvalDiff, gradeSession, id8, inject, name, parseEvalArgs, parseEvalDiffArgs, renderCard, renderTrend };

package/package.json

@@ -0,0 +1,75 @@
+{
+  "name": "dsh-session-eval",
+  "description": "Retrospective agent-session evaluation for DeepSeek Harness: deterministic grade cards from persisted session logs, cross-session regression diffs, era comparison — no benchmark runs, no LLM judges.",
+  "version": "0.1.0",
+  "publishConfig": {
+    "access": "public"
+  },
+  "type": "module",
+  "main": "lib/index.js",
+  "types": "lib/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./lib/index.d.ts",
+      "default": "./lib/index.js"
+    },
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "lib",
+    "README.md",
+    "README.zh.md",
+    "CHANGELOG.md",
+    "LICENSE",
+    "cordis.patch.yml"
+  ],
+  "dsh": {
+    "bundle": {
+      "patch": "./cordis.patch.yml"
+    }
+  },
+  "license": "MIT",
+  "keywords": [
+    "deepseek-harness",
+    "dsh",
+    "dsh-plugin",
+    "agent-evaluation",
+    "session-metrics",
+    "regression"
+  ],
+  "peerDependencies": {
+    "@deepseek-ai/cordis": "^4.0.1",
+    "@deepseek-ai/dsh-commands": "^0.1.1-rc.2",
+    "@deepseek-ai/dsh-llm": "^0.1.1-rc.2",
+    "@deepseek-ai/dsh-session": "^0.1.1-rc.2",
+    "@deepseek-ai/dsh-session-query": "^0.1.1-rc.2",
+    "@deepseek-ai/dsh-tools": "^0.1.1-rc.2"
+  },
+  "devDependencies": {
+    "@types/node": "^26.2.0",
+    "tsdown": "^0.22.14",
+    "typescript": "^7.0.2",
+    "vitest": "^4.1.11",
+    "@deepseek-ai/cordis": "^4.0.1",
+    "@deepseek-ai/dsh-commands": "^0.1.1-rc.2",
+    "@deepseek-ai/dsh-session": "^0.1.1-rc.2"
+  },
+  "scripts": {
+    "bundle": "tsdown",
+    "typecheck": "tsc --noEmit",
+    "test": "vitest run"
+  },
+  "peerDependenciesMeta": {
+    "@deepseek-ai/dsh-tools": {
+      "optional": true
+    }
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/kittimzhe/dsh-session-eval.git"
+  },
+  "homepage": "https://github.com/kittimzhe/dsh-session-eval#readme",
+  "bugs": {
+    "url": "https://github.com/kittimzhe/dsh-session-eval/issues"
+  }
+}