evaldrift 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 promptdrift contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,157 @@
+# promptdrift
+
+> 给 LLM prompt / agent 做**快照式回归测试**——改一句提示词，第一时间知道有没有把别的用例悄悄改坏了。
+>
+> **国产模型 & 中文优先**：DeepSeek / Kimi / 通义 / 豆包 / 智谱 / 本地 Ollama 开箱即用。
+
+调 prompt 最大的坑是：你为了修好 A 场景改了提示词，结果悄悄把 B、C 场景弄坏了，而你根本没发现，直到用户来投诉。`promptdrift` 把这件事变成像跑单元测试一样简单——锁一个**基线**，之后每次改提示词跑一下，它直接告诉你**哪些用例退化了**。
+
+```
+✓ 退款问题  ██████████ 100%  120ms
+✗ 营业时间  ░░░░░░░░░░   0%   98ms
+    ✗ [contains] 缺少关键词: 09:00
+✓ 情绪安抚  █████████░  85%  340ms
+────────────────────────────────────────
+对比基线：
+  ↓ 退化 营业时间  100% → 0% (-100%) [通过→失败]
+
+1 处退化 · 0 处改进
+```
+
+## 为什么用它（而不是 promptfoo / deepeval）
+
+那些工具很好，但都是英文世界的，**对国产模型和中文评测支持很弱**。`promptdrift` 专门补这个空档：
+
+- **国产模型一等公民**：一个 `baseUrl` 切到 DeepSeek / Kimi / 通义 / 豆包 / 智谱，不用折腾 SDK。
+- **中文评测**：内置的 `llm-judge` 裁判提示、报告、报错全是中文。
+- **聚焦"防退化"这一件事**：基线对比 + `通过→失败` 高亮 + CI 退出码，像 jest snapshot 一样直觉。
+- **零服务器、纯本地**：数据不出本机，`npx` 就能跑，能直接塞进 CI。
+- **离线可跑**：内置 mock 模型，没 API Key 也能先把用例和断言调通。
+
+## 快速开始
+
+```bash
+# 1. 生成配置（默认离线 mock，立即可跑）
+npx promptdrift init
+
+# 2. 跑测试
+npx promptdrift run
+
+# 3. 满意了就锁定基线
+npx promptdrift baseline
+
+# 4. 以后每次改完 prompt 再跑，自动对比基线、标出退化
+npx promptdrift run --html report.html
+```
+
+## 接真实模型
+
+把配置里的 `provider` 换成下面任意一个：
+
+```yaml
+# DeepSeek（OpenAI 兼容）
+provider:
+  type: openai
+  model: deepseek-chat
+  baseUrl: https://api.deepseek.com/v1
+  apiKeyEnv: DEEPSEEK_API_KEY     # 从环境变量读，别把密钥写进文件
+  temperature: 0
+```
+
+| 模型 | type | baseUrl |
+|---|---|---|
+| DeepSeek | `openai` | `https://api.deepseek.com/v1` |
+| Kimi (Moonshot) | `openai` | `https://api.moonshot.cn/v1` |
+| 通义千问 | `openai` | `https://dashscope.aliyuncs.com/compatible-mode/v1` |
+| 豆包 (火山方舟) | `openai` | `https://ark.cn-beijing.volces.com/api/v3` |
+| 智谱 GLM | `openai` | `https://open.bigmodel.cn/api/paas/v4` |
+| 本地 Ollama | `ollama` | `http://localhost:11434` |
+| Claude | `anthropic` | 默认官方 |
+
+## 配置说明
+
+```yaml
+provider:
+  type: openai            # openai | anthropic | ollama | mock
+  model: deepseek-chat
+  baseUrl: https://api.deepseek.com/v1
+  apiKeyEnv: DEEPSEEK_API_KEY
+
+judge:                    # 可选：llm-judge 用的裁判模型，缺省复用 provider
+  type: openai
+  model: deepseek-chat
+  baseUrl: https://api.deepseek.com/v1
+  apiKeyEnv: DEEPSEEK_API_KEY
+
+prompt:
+  system: "你是一个礼貌专业的中文客服。"
+  user: "{{question}}"    # {{var}} 会被 test.vars 替换
+
+defaultAssert:            # 可选：应用到每个用例的断言
+  - type: regex
+    value: "^[\\s\\S]{6,}$"
+
+tests:
+  - name: 退款问题
+    vars: { question: "我要退款怎么办" }
+    assert:
+      - type: contains
+        value: ["退款", "工作日"]   # 数组 = 必须全部包含
+```
+
+### 断言类型
+
+| type | 作用 | 关键字段 |
+|---|---|---|
+| `contains` | 输出必须包含关键词（数组则全含） | `value` |
+| `not-contains` | 输出不能出现这些词 | `value` |
+| `regex` | 匹配正则 | `value` |
+| `json-schema` | 输出是合法 JSON 且符合 schema | `schema` |
+| `llm-judge` | 用模型按中文评分标准打分 | `rubric` `threshold` |
+
+每条断言可加 `weight`（权重，默认 1），用例总分 = 加权平均。
+
+## 命令
+
+```
+promptdrift init                 生成配置模板
+promptdrift run [选项]           跑测试并对比基线
+promptdrift baseline [选项]      跑一次并锁定为基线
+
+选项:
+  -c, --config <path>     指定配置文件
+      --html <path>       输出自包含 HTML 报告
+      --json <path>       输出 JSON 结果
+      --update-baseline   run 跑完顺便更新基线
+      --fail-on <mode>    CI 退出码: any(默认) | regression | none
+```
+
+## 放进 CI
+
+```yaml
+# .github/workflows/promptdrift.yml
+- run: npx promptdrift run --fail-on regression
+  env:
+    DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
+```
+
+把 `.promptdrift/baseline.json` 提交进仓库，PR 改了提示词导致退化时 CI 直接红灯。
+
+## 工作原理
+
+```
+配置(provider + prompt + tests)
+        │
+        ▼
+   对每个用例调用模型 ──► 跑断言打分 ──► 加权得出用例分
+        │
+        ▼
+   和 .promptdrift/baseline.json 逐用例对比
+        │
+        ▼
+   终端表格 + HTML 报告 + CI 退出码（退化即非 0）
+```
+
+## License
+
+MIT

package/dist/baseline.js

@@ -0,0 +1,66 @@
+import { readFileSync, writeFileSync, existsSync, mkdirSync } from "node:fs";
+import { resolve, dirname } from "node:path";
+export const ARTIFACT_DIR = ".promptdrift";
+export const BASELINE_FILE = "baseline.json";
+export function baselinePath(cwd = process.cwd()) {
+    return resolve(cwd, ARTIFACT_DIR, BASELINE_FILE);
+}
+export function saveBaseline(run, cwd = process.cwd()) {
+    const p = baselinePath(cwd);
+    mkdirSync(dirname(p), { recursive: true });
+    writeFileSync(p, JSON.stringify(run, null, 2), "utf8");
+    return p;
+}
+export function loadBaseline(cwd = process.cwd()) {
+    const p = baselinePath(cwd);
+    if (!existsSync(p))
+        return undefined;
+    return JSON.parse(readFileSync(p, "utf8"));
+}
+const EPS = 0.01;
+export function diffAgainstBaseline(current, baseline) {
+    if (!baseline) {
+        return { hasBaseline: false, regressions: 0, improvements: 0, drifts: [] };
+    }
+    const baseMap = new Map(baseline.tests.map((t) => [t.name, t]));
+    const curMap = new Map(current.tests.map((t) => [t.name, t]));
+    const drifts = [];
+    for (const cur of current.tests) {
+        const base = baseMap.get(cur.name);
+        if (!base) {
+            drifts.push({ name: cur.name, kind: "new", curScore: cur.score, curPass: cur.pass, delta: 0 });
+            continue;
+        }
+        const delta = cur.score - base.score;
+        let kind = "same";
+        // pass->fail 永远算退化；fail->pass 算改进；否则看分数变化
+        if (base.pass && !cur.pass)
+            kind = "regression";
+        else if (!base.pass && cur.pass)
+            kind = "improvement";
+        else if (delta < -EPS)
+            kind = "regression";
+        else if (delta > EPS)
+            kind = "improvement";
+        drifts.push({
+            name: cur.name,
+            kind,
+            baseScore: base.score,
+            curScore: cur.score,
+            basePass: base.pass,
+            curPass: cur.pass,
+            delta,
+        });
+    }
+    for (const base of baseline.tests) {
+        if (!curMap.has(base.name)) {
+            drifts.push({ name: base.name, kind: "removed", baseScore: base.score, basePass: base.pass, delta: 0 });
+        }
+    }
+    return {
+        hasBaseline: true,
+        regressions: drifts.filter((d) => d.kind === "regression").length,
+        improvements: drifts.filter((d) => d.kind === "improvement").length,
+        drifts,
+    };
+}

package/dist/cli.js

@@ -0,0 +1,170 @@
+#!/usr/bin/env node
+import { parseArgs } from "node:util";
+import { writeFileSync, copyFileSync, existsSync } from "node:fs";
+import { resolve, dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
+import pc from "picocolors";
+import { findConfig, loadConfig, CONFIG_NAMES } from "./config.js";
+import { runConfig } from "./runner.js";
+import { saveBaseline, loadBaseline, diffAgainstBaseline } from "./baseline.js";
+import { printRun } from "./report/terminal.js";
+import { renderHtml } from "./report/html.js";
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const PKG_ROOT = resolve(__dirname, "..");
+const VERSION = "0.1.0";
+function help() {
+    console.log(`
+${pc.bold("promptdrift")} — 给 LLM prompt / agent 做快照式回归测试，改提示词时第一时间发现质量退化。
+国产模型 & 中文优先（DeepSeek / Kimi / 通义 / 豆包 / Ollama）。
+
+用法:
+  promptdrift init                 在当前目录生成可直接跑的配置模板
+  promptdrift run [选项]           跑测试并对比基线
+  promptdrift baseline [选项]      跑一次并把结果锁定为基线
+
+run / baseline 选项:
+  -c, --config <path>   指定配置文件（默认自动查找）
+      --html <path>     额外输出自包含 HTML 报告
+      --json <path>     额外输出 JSON 结果
+      --update-baseline run 跑完顺便更新基线
+      --fail-on <mode>  CI 退出码策略: any(默认) | regression | none
+
+其他:
+  -h, --help            显示帮助
+  -v, --version         显示版本
+`);
+}
+async function cmdInit() {
+    const target = resolve(process.cwd(), CONFIG_NAMES[0]);
+    if (existsSync(target)) {
+        console.log(pc.yellow(`已存在 ${CONFIG_NAMES[0]}，未覆盖。`));
+        return;
+    }
+    const tpl = join(PKG_ROOT, "templates", "promptdrift.config.yaml");
+    if (existsSync(tpl)) {
+        copyFileSync(tpl, target);
+    }
+    else {
+        writeFileSync(target, FALLBACK_TEMPLATE, "utf8");
+    }
+    console.log(pc.green(`已生成 ${CONFIG_NAMES[0]}`));
+    console.log(pc.dim("它默认用离线 mock 模型，直接 `promptdrift run` 就能看到效果。"));
+    console.log(pc.dim("改成真实模型：把 provider.type 换成 openai，填 baseUrl / model / apiKeyEnv。"));
+}
+async function cmdRun(opts, asBaseline) {
+    const configPath = findConfig(process.cwd(), opts.config);
+    const config = loadConfig(configPath);
+    const run = await runConfig(config);
+    if (asBaseline) {
+        const p = saveBaseline(run);
+        console.log(pc.green(`已把当前结果锁定为基线: ${p}`));
+        const empty = diffAgainstBaseline(run, undefined);
+        printRun(run, empty);
+        writeArtifacts(run, opts);
+        return 0;
+    }
+    const baseline = loadBaseline();
+    const drift = diffAgainstBaseline(run, baseline);
+    printRun(run, drift);
+    writeArtifacts(run, opts);
+    if (opts.updateBaseline) {
+        const p = saveBaseline(run);
+        console.log(pc.dim(`基线已更新: ${p}`));
+    }
+    if (opts.failOn === "none")
+        return 0;
+    if (opts.failOn === "regression")
+        return drift.regressions > 0 ? 1 : 0;
+    return run.summary.failed > 0 || drift.regressions > 0 ? 1 : 0;
+}
+function writeArtifacts(run, opts) {
+    if (opts.json) {
+        const p = resolve(process.cwd(), opts.json);
+        writeFileSync(p, JSON.stringify(run, null, 2), "utf8");
+        console.log(pc.dim(`JSON 已写入: ${p}`));
+    }
+    if (opts.html) {
+        const baseline = loadBaseline();
+        const drift = diffAgainstBaseline(run, baseline);
+        const p = resolve(process.cwd(), opts.html);
+        writeFileSync(p, renderHtml(run, drift), "utf8");
+        console.log(pc.dim(`HTML 报告已写入: ${p}`));
+    }
+}
+async function main() {
+    const argv = process.argv.slice(2);
+    const command = argv[0];
+    if (!command || command === "-h" || command === "--help" || command === "help") {
+        help();
+        return;
+    }
+    if (command === "-v" || command === "--version") {
+        console.log(VERSION);
+        return;
+    }
+    const { values } = parseArgs({
+        args: argv.slice(1),
+        options: {
+            config: { type: "string", short: "c" },
+            html: { type: "string" },
+            json: { type: "string" },
+            "update-baseline": { type: "boolean", default: false },
+            "fail-on": { type: "string", default: "any" },
+        },
+        allowPositionals: true,
+    });
+    const failOn = values["fail-on"] || "any";
+    if (!["any", "regression", "none"].includes(failOn)) {
+        console.error(pc.red(`--fail-on 只能是 any | regression | none`));
+        process.exit(2);
+    }
+    const opts = {
+        config: values.config,
+        html: values.html,
+        json: values.json,
+        updateBaseline: Boolean(values["update-baseline"]),
+        failOn: failOn,
+    };
+    switch (command) {
+        case "init":
+            await cmdInit();
+            break;
+        case "run":
+            process.exit(await cmdRun(opts, false));
+            break;
+        case "baseline":
+            process.exit(await cmdRun(opts, true));
+            break;
+        default:
+            console.error(pc.red(`未知命令: ${command}`));
+            help();
+            process.exit(2);
+    }
+}
+const FALLBACK_TEMPLATE = `provider:
+  type: mock
+  model: demo
+  mockResponses:
+    退款: "您好，关于退款：支持 7 天无理由退款，请在订单页点击申请，1-3 个工作日到账。"
+    几点: "我们每天 09:00-22:00 营业，节假日正常。"
+
+prompt:
+  system: "你是一个礼貌专业的中文客服。"
+  user: "{{question}}"
+
+tests:
+  - name: 退款问题
+    vars: { question: "我要退款怎么办" }
+    assert:
+      - type: contains
+        value: ["退款", "工作日"]
+  - name: 营业时间
+    vars: { question: "你们几点开门" }
+    assert:
+      - type: contains
+        value: "09:00"
+`;
+main().catch((e) => {
+    console.error(pc.red(e.message));
+    process.exit(1);
+});

package/dist/config.js

@@ -0,0 +1,54 @@
+import { readFileSync, existsSync } from "node:fs";
+import { resolve } from "node:path";
+import YAML from "yaml";
+export const CONFIG_NAMES = [
+    "promptdrift.config.yaml",
+    "promptdrift.config.yml",
+    "promptdrift.config.json",
+];
+export function findConfig(cwd = process.cwd(), explicit) {
+    if (explicit) {
+        const p = resolve(cwd, explicit);
+        if (!existsSync(p))
+            throw new Error(`找不到配置文件: ${p}`);
+        return p;
+    }
+    for (const name of CONFIG_NAMES) {
+        const p = resolve(cwd, name);
+        if (existsSync(p))
+            return p;
+    }
+    throw new Error(`当前目录没有配置文件（${CONFIG_NAMES.join(" / ")}）。先运行: promptdrift init`);
+}
+export function loadConfig(path) {
+    const raw = readFileSync(path, "utf8");
+    const data = path.endsWith(".json")
+        ? JSON.parse(raw)
+        : YAML.parse(raw);
+    return validate(data, path);
+}
+function validate(data, path) {
+    if (typeof data !== "object" || data === null) {
+        throw new Error(`配置文件格式错误: ${path}`);
+    }
+    const cfg = data;
+    if (!cfg.provider || typeof cfg.provider !== "object") {
+        throw new Error("配置缺少 provider 段");
+    }
+    if (!cfg.provider.type)
+        throw new Error("provider.type 不能为空");
+    if (!cfg.provider.model)
+        throw new Error("provider.model 不能为空");
+    if (!cfg.prompt || typeof cfg.prompt.user !== "string") {
+        throw new Error("配置缺少 prompt.user（用户提示词模板）");
+    }
+    if (!Array.isArray(cfg.tests) || cfg.tests.length === 0) {
+        throw new Error("配置缺少 tests（至少一个测试用例）");
+    }
+    for (const [i, t] of cfg.tests.entries()) {
+        if (!t || typeof t.name !== "string") {
+            throw new Error(`tests[${i}] 缺少 name`);
+        }
+    }
+    return cfg;
+}

package/dist/providers/anthropic.js

@@ -0,0 +1,39 @@
+import { readApiKey } from "../util.js";
+export function createAnthropicProvider(cfg) {
+    const baseUrl = (cfg.baseUrl || "https://api.anthropic.com").replace(/\/+$/, "");
+    const apiKey = readApiKey(cfg.apiKeyEnv);
+    return {
+        label: `anthropic(${cfg.model})`,
+        async complete(messages) {
+            if (!apiKey) {
+                throw new Error(`缺少 API Key：请设置环境变量 ${cfg.apiKeyEnv || "ANTHROPIC_API_KEY"}`);
+            }
+            const system = messages
+                .filter((m) => m.role === "system")
+                .map((m) => m.content)
+                .join("\n");
+            const chat = messages.filter((m) => m.role !== "system");
+            const res = await fetch(`${baseUrl}/v1/messages`, {
+                method: "POST",
+                headers: {
+                    "Content-Type": "application/json",
+                    "x-api-key": apiKey,
+                    "anthropic-version": "2023-06-01",
+                },
+                body: JSON.stringify({
+                    model: cfg.model,
+                    max_tokens: cfg.maxTokens ?? 1024,
+                    temperature: cfg.temperature ?? 0,
+                    ...(system ? { system } : {}),
+                    messages: chat.map((m) => ({ role: m.role, content: m.content })),
+                }),
+            });
+            if (!res.ok) {
+                const text = await res.text().catch(() => "");
+                throw new Error(`Anthropic 接口报错 ${res.status}: ${text.slice(0, 300)}`);
+            }
+            const data = (await res.json());
+            return data.content?.map((c) => c.text ?? "").join("") ?? "";
+        },
+    };
+}

package/dist/providers/index.js

@@ -0,0 +1,18 @@
+import { createOpenAIProvider } from "./openai.js";
+import { createAnthropicProvider } from "./anthropic.js";
+import { createOllamaProvider } from "./ollama.js";
+import { createMockProvider } from "./mock.js";
+export function createProvider(cfg) {
+    switch (cfg.type) {
+        case "openai":
+            return createOpenAIProvider(cfg);
+        case "anthropic":
+            return createAnthropicProvider(cfg);
+        case "ollama":
+            return createOllamaProvider(cfg);
+        case "mock":
+            return createMockProvider(cfg);
+        default:
+            throw new Error(`未知的 provider.type: ${cfg.type}`);
+    }
+}

package/dist/providers/mock.js

@@ -0,0 +1,22 @@
+/**
+ * 离线 mock provider —— 不需要任何 API Key，开箱即跑。
+ * 用于 example、CI、以及你想先把测试用例和断言调通、再接真实模型的场景。
+ *
+ * 匹配规则：把最后一条 user 消息和 mockResponses 的每个 key 比较，
+ * 只要 user 内容包含某个 key，就返回对应的预设回复；都不匹配则回显。
+ */
+export function createMockProvider(cfg) {
+    const table = cfg.mockResponses ?? {};
+    return {
+        label: `mock(${cfg.model})`,
+        async complete(messages) {
+            const lastUser = [...messages].reverse().find((m) => m.role === "user");
+            const content = lastUser?.content ?? "";
+            for (const [key, reply] of Object.entries(table)) {
+                if (key && content.includes(key))
+                    return reply;
+            }
+            return table["__default__"] ?? `（mock 回显）${content}`;
+        },
+    };
+}

package/dist/providers/ollama.js

@@ -0,0 +1,24 @@
+export function createOllamaProvider(cfg) {
+    const baseUrl = (cfg.baseUrl || "http://localhost:11434").replace(/\/+$/, "");
+    return {
+        label: `ollama(${cfg.model} @ ${baseUrl})`,
+        async complete(messages) {
+            const res = await fetch(`${baseUrl}/api/chat`, {
+                method: "POST",
+                headers: { "Content-Type": "application/json" },
+                body: JSON.stringify({
+                    model: cfg.model,
+                    messages,
+                    stream: false,
+                    options: { temperature: cfg.temperature ?? 0 },
+                }),
+            });
+            if (!res.ok) {
+                const text = await res.text().catch(() => "");
+                throw new Error(`Ollama 接口报错 ${res.status}: ${text.slice(0, 300)}`);
+            }
+            const data = (await res.json());
+            return data.message?.content ?? "";
+        },
+    };
+}

package/dist/providers/openai.js

@@ -0,0 +1,41 @@
+import { readApiKey } from "../util.js";
+/**
+ * OpenAI 兼容 provider。
+ * 同一套接口覆盖绝大多数国产模型，只要换 baseUrl：
+ *   DeepSeek  https://api.deepseek.com/v1
+ *   Kimi      https://api.moonshot.cn/v1
+ *   通义千问   https://dashscope.aliyuncs.com/compatible-mode/v1
+ *   豆包       https://ark.cn-beijing.volces.com/api/v3
+ *   智谱       https://open.bigmodel.cn/api/paas/v4
+ */
+export function createOpenAIProvider(cfg) {
+    const baseUrl = (cfg.baseUrl || "https://api.openai.com/v1").replace(/\/+$/, "");
+    const apiKey = readApiKey(cfg.apiKeyEnv);
+    return {
+        label: `openai(${cfg.model} @ ${baseUrl})`,
+        async complete(messages) {
+            if (!apiKey) {
+                throw new Error(`缺少 API Key：请设置环境变量 ${cfg.apiKeyEnv || "(provider.apiKeyEnv 未配置)"}`);
+            }
+            const res = await fetch(`${baseUrl}/chat/completions`, {
+                method: "POST",
+                headers: {
+                    "Content-Type": "application/json",
+                    Authorization: `Bearer ${apiKey}`,
+                },
+                body: JSON.stringify({
+                    model: cfg.model,
+                    messages,
+                    temperature: cfg.temperature ?? 0,
+                    ...(cfg.maxTokens ? { max_tokens: cfg.maxTokens } : {}),
+                }),
+            });
+            if (!res.ok) {
+                const text = await res.text().catch(() => "");
+                throw new Error(`OpenAI 兼容接口报错 ${res.status}: ${text.slice(0, 300)}`);
+            }
+            const data = (await res.json());
+            return data.choices?.[0]?.message?.content ?? "";
+        },
+    };
+}

package/dist/report/html.js

@@ -0,0 +1,106 @@
+function esc(s) {
+    return s
+        .replace(/&/g, "&")
+        .replace(/</g, "&lt;")
+        .replace(/>/g, "&gt;")
+        .replace(/"/g, "&quot;");
+}
+function driftBadge(d) {
+    if (!d)
+        return "";
+    const map = {
+        regression: ["退化", "reg"],
+        improvement: ["改进", "imp"],
+        new: ["新增", "new"],
+        removed: ["移除", "rm"],
+        same: ["", ""],
+    };
+    const [label, cls] = map[d.kind] ?? ["", ""];
+    if (!label)
+        return "";
+    const delta = d.curScore !== undefined && d.baseScore !== undefined
+        ? ` ${(d.delta * 100 > 0 ? "+" : "")}${(d.delta * 100).toFixed(0)}%`
+        : "";
+    return `<span class="badge ${cls}">${label}${delta}</span>`;
+}
+export function renderHtml(run, drift) {
+    const driftMap = new Map(drift.drifts.map((d) => [d.name, d]));
+    const s = run.summary;
+    const rows = run.tests
+        .map((t) => {
+        const d = driftMap.get(t.name);
+        const asserts = t.error
+            ? `<div class="err">错误: ${esc(t.error)}</div>`
+            : t.assertions
+                .map((a) => `<div class="assert ${a.pass ? "ok" : "bad"}"><span>${a.pass ? "✓" : "✗"}</span> <code>${a.type}</code> ${esc(a.message)}</div>`)
+                .join("") || `<div class="assert dim">无断言（仅冒烟）</div>`;
+        return `
+      <tr class="${t.pass ? "pass" : "fail"}">
+        <td class="mark">${t.pass ? "✓" : "✗"}</td>
+        <td class="name">${esc(t.name)} ${driftBadge(d)}</td>
+        <td class="score"><div class="sbar"><i style="width:${Math.round(t.score * 100)}%"></i></div>${(t.score * 100).toFixed(0)}%</td>
+        <td class="lat">${t.latencyMs}ms</td>
+      </tr>
+      <tr class="detail ${t.pass ? "pass" : "fail"}">
+        <td></td>
+        <td colspan="3">
+          ${asserts}
+          <details><summary>模型输出</summary><pre>${esc(t.output)}</pre></details>
+        </td>
+      </tr>`;
+    })
+        .join("");
+    const driftLine = drift.hasBaseline
+        ? `<span class="${drift.regressions > 0 ? "reg" : "imp"}">${drift.regressions} 处退化</span> · <span class="imp">${drift.improvements} 处改进</span>`
+        : `<span class="dim">无基线</span>`;
+    return `<!DOCTYPE html>
+<html lang="zh-CN"><head><meta charset="utf-8"/>
+<meta name="viewport" content="width=device-width, initial-scale=1"/>
+<title>promptdrift 报告 · ${esc(run.provider.model)}</title>
+<style>
+  :root { --ok:#16a34a; --bad:#dc2626; --ink:#0f172a; --muted:#64748b; --line:#e2e8f0; --bg:#f8fafc; }
+  *{box-sizing:border-box;margin:0;padding:0}
+  body{font-family:-apple-system,"PingFang SC","Microsoft YaHei",sans-serif;background:var(--bg);color:var(--ink);padding:32px;line-height:1.5}
+  .wrap{max-width:980px;margin:0 auto}
+  h1{font-size:20px;font-weight:800}
+  .meta{color:var(--muted);font-size:13px;margin-top:4px}
+  .cards{display:grid;grid-template-columns:repeat(4,1fr);gap:12px;margin:20px 0}
+  .card{background:#fff;border:1px solid var(--line);border-radius:12px;padding:14px 16px}
+  .card .n{font-size:26px;font-weight:800}
+  .card .l{font-size:12px;color:var(--muted);margin-top:2px}
+  table{width:100%;border-collapse:collapse;background:#fff;border:1px solid var(--line);border-radius:12px;overflow:hidden}
+  td{padding:10px 12px;border-bottom:1px solid var(--line);vertical-align:top}
+  tr.pass .mark{color:var(--ok)} tr.fail .mark{color:var(--bad)}
+  .mark{font-weight:800;width:28px;text-align:center}
+  .name{font-weight:700}
+  .score{width:140px;font-variant-numeric:tabular-nums;color:var(--muted)}
+  .sbar{height:6px;background:var(--line);border-radius:99px;overflow:hidden;margin-bottom:4px}
+  .sbar i{display:block;height:100%;background:var(--ok)}
+  tr.fail .sbar i{background:var(--bad)}
+  .lat{width:80px;color:var(--muted);font-size:13px}
+  tr.detail td{padding-top:0;border-bottom:1px solid var(--line)}
+  .assert{font-size:13px;margin:3px 0}
+  .assert.ok{color:var(--ok)} .assert.bad{color:var(--bad)} .assert.dim{color:var(--muted)}
+  .assert code{background:#f1f5f9;padding:1px 6px;border-radius:5px;color:var(--ink)}
+  details{margin-top:6px} summary{cursor:pointer;color:var(--muted);font-size:13px}
+  pre{background:#0f172a;color:#e2e8f0;padding:12px;border-radius:8px;margin-top:6px;white-space:pre-wrap;word-break:break-word;font-size:12px;max-height:320px;overflow:auto}
+  .err{color:var(--bad);font-size:13px}
+  .badge{font-size:11px;font-weight:700;padding:1px 7px;border-radius:99px;margin-left:6px}
+  .badge.reg{background:#fee2e2;color:#b91c1c} .badge.imp{background:#dcfce7;color:#15803d}
+  .badge.new{background:#e0f2fe;color:#0369a1} .badge.rm{background:#f1f5f9;color:#64748b}
+  .reg{color:var(--bad);font-weight:700} .imp{color:var(--ok);font-weight:700} .dim{color:var(--muted)}
+  footer{color:var(--muted);font-size:12px;text-align:center;margin-top:20px}
+</style></head>
+<body><div class="wrap">
+  <h1>promptdrift 报告</h1>
+  <div class="meta">${esc(run.provider.type)} / ${esc(run.provider.model)} · ${esc(run.timestamp)}</div>
+  <div class="cards">
+    <div class="card"><div class="n">${s.passed}/${s.total}</div><div class="l">通过</div></div>
+    <div class="card"><div class="n" style="color:${s.failed ? "var(--bad)" : "var(--ok)"}">${s.failed}</div><div class="l">失败</div></div>
+    <div class="card"><div class="n">${(s.avgScore * 100).toFixed(0)}%</div><div class="l">平均分</div></div>
+    <div class="card"><div class="n" style="font-size:15px;padding-top:8px">${driftLine}</div><div class="l">对比基线</div></div>
+  </div>
+  <table><tbody>${rows}</tbody></table>
+  <footer>Generated by promptdrift</footer>
+</div></body></html>`;
+}

package/dist/report/terminal.js

@@ -0,0 +1,62 @@
+import pc from "picocolors";
+function bar(score, width = 10) {
+    const filled = Math.round(score * width);
+    return "█".repeat(filled) + "░".repeat(width - filled);
+}
+export function printRun(run, drift) {
+    console.log("");
+    console.log(pc.bold(`promptdrift · ${run.provider.type}/${run.provider.model}`));
+    console.log(pc.dim("─".repeat(56)));
+    for (const t of run.tests) {
+        const mark = t.pass ? pc.green("✓") : pc.red("✗");
+        const scoreStr = t.error ? pc.red("ERR") : `${(t.score * 100).toFixed(0).padStart(3)}%`;
+        console.log(`${mark} ${pc.bold(t.name)}  ${pc.dim(bar(t.error ? 0 : t.score))} ${scoreStr}  ${pc.dim(t.latencyMs + "ms")}`);
+        if (t.error) {
+            console.log(pc.red(`    错误: ${t.error}`));
+            continue;
+        }
+        for (const a of t.assertions) {
+            if (!a.pass)
+                console.log(pc.red(`    ✗ [${a.type}] ${a.message}`));
+        }
+    }
+    console.log(pc.dim("─".repeat(56)));
+    const s = run.summary;
+    const head = s.failed === 0 ? pc.green(pc.bold("全部通过")) : pc.red(pc.bold(`${s.failed} 个失败`));
+    console.log(`${head}  ${s.passed}/${s.total} 通过 · 平均分 ${(s.avgScore * 100).toFixed(0)}%`);
+    if (drift.hasBaseline) {
+        console.log("");
+        if (drift.regressions === 0 && drift.improvements === 0) {
+            console.log(pc.dim("对比基线：无明显变化"));
+        }
+        else {
+            console.log(pc.bold("对比基线："));
+            for (const d of drift.drifts) {
+                if (d.kind === "regression") {
+                    console.log(pc.red(`  ↓ 退化 ${d.name}  ${fmtDelta(d)}`));
+                }
+                else if (d.kind === "improvement") {
+                    console.log(pc.green(`  ↑ 改进 ${d.name}  ${fmtDelta(d)}`));
+                }
+                else if (d.kind === "new") {
+                    console.log(pc.cyan(`  + 新增 ${d.name}`));
+                }
+                else if (d.kind === "removed") {
+                    console.log(pc.dim(`  - 移除 ${d.name}`));
+                }
+            }
+            console.log("");
+            console.log(`${drift.regressions > 0 ? pc.red(drift.regressions + " 处退化") : pc.green("0 退化")} · ${pc.green(drift.improvements + " 处改进")}`);
+        }
+    }
+    else {
+        console.log(pc.dim("（无基线，运行 `promptdrift baseline` 锁定当前结果作为基线）"));
+    }
+    console.log("");
+}
+function fmtDelta(d) {
+    const passInfo = d.basePass !== d.curPass ? ` [${d.basePass ? "通过" : "失败"}→${d.curPass ? "通过" : "失败"}]` : "";
+    const pct = (d.delta * 100).toFixed(0);
+    const sign = d.delta > 0 ? "+" : "";
+    return `${(d.baseScore * 100).toFixed(0)}% → ${(d.curScore * 100).toFixed(0)}% (${sign}${pct}%)${passInfo}`;
+}

package/dist/runner.js

@@ -0,0 +1,52 @@
+import { createProvider } from "./providers/index.js";
+import { runAssertion } from "./scorers/index.js";
+import { renderTemplate } from "./util.js";
+function weightedScore(results) {
+    if (results.length === 0)
+        return 1;
+    const totalW = results.reduce((s, r) => s + r.weight, 0) || 1;
+    return results.reduce((s, r) => s + r.score * r.weight, 0) / totalW;
+}
+export async function runConfig(config) {
+    const provider = createProvider(config.provider);
+    const judge = config.judge ? createProvider(config.judge) : provider;
+    const ctx = { judge };
+    const tests = [];
+    for (const tc of config.tests) {
+        const vars = tc.vars ?? {};
+        const messages = [];
+        if (config.prompt.system) {
+            messages.push({ role: "system", content: renderTemplate(config.prompt.system, vars) });
+        }
+        messages.push({ role: "user", content: renderTemplate(config.prompt.user, vars) });
+        const assertions = [...(config.defaultAssert ?? []), ...(tc.assert ?? [])];
+        const started = Date.now();
+        let output = "";
+        let error;
+        try {
+            output = await provider.complete(messages);
+        }
+        catch (e) {
+            error = e.message;
+        }
+        const latencyMs = Date.now() - started;
+        let assertionResults = [];
+        if (!error) {
+            for (const a of assertions) {
+                assertionResults.push(await runAssertion(output, a, ctx));
+            }
+        }
+        const pass = !error && assertionResults.every((r) => r.pass);
+        const score = error ? 0 : weightedScore(assertionResults);
+        tests.push({ name: tc.name, vars, output, latencyMs, error, assertions: assertionResults, pass, score });
+    }
+    const passed = tests.filter((t) => t.pass).length;
+    const avgScore = tests.length ? tests.reduce((s, t) => s + t.score, 0) / tests.length : 0;
+    return {
+        schemaVersion: 1,
+        timestamp: new Date().toISOString(),
+        provider: { type: config.provider.type, model: config.provider.model },
+        summary: { total: tests.length, passed, failed: tests.length - passed, avgScore },
+        tests,
+    };
+}

package/dist/scorers/index.js

@@ -0,0 +1,142 @@
+import AjvImport from "ajv";
+import { clamp01 } from "../util.js";
+// ajv 是 CommonJS 包，在 NodeNext ESM 下默认导出可能被包一层 .default
+const Ajv = (AjvImport.default ??
+    AjvImport);
+const ajv = new Ajv({ allErrors: true, strict: false });
+function toArray(v) {
+    if (v === undefined)
+        return [];
+    return Array.isArray(v) ? v : [v];
+}
+function tryExtractJson(text) {
+    try {
+        return JSON.parse(text);
+    }
+    catch {
+        // 从混合文本里抠出第一个 JSON 块
+        const match = text.match(/[{[][\s\S]*[}\]]/);
+        if (match) {
+            try {
+                return JSON.parse(match[0]);
+            }
+            catch {
+                return undefined;
+            }
+        }
+        return undefined;
+    }
+}
+async function scoreContains(output, a) {
+    const needles = toArray(a.value);
+    const missing = needles.filter((n) => !output.includes(n));
+    const pass = missing.length === 0;
+    return {
+        type: "contains",
+        pass,
+        score: pass ? 1 : 0,
+        weight: a.weight ?? 1,
+        message: pass
+            ? `包含全部关键词: ${needles.join(", ")}`
+            : `缺少关键词: ${missing.join(", ")}`,
+    };
+}
+async function scoreNotContains(output, a) {
+    const needles = toArray(a.value);
+    const found = needles.filter((n) => output.includes(n));
+    const pass = found.length === 0;
+    return {
+        type: "not-contains",
+        pass,
+        score: pass ? 1 : 0,
+        weight: a.weight ?? 1,
+        message: pass ? `未出现禁止词` : `出现了禁止词: ${found.join(", ")}`,
+    };
+}
+async function scoreRegex(output, a) {
+    const pattern = Array.isArray(a.value) ? a.value[0] : a.value;
+    if (!pattern) {
+        return failResult("regex", a, "regex 断言缺少 value（正则表达式）");
+    }
+    let re;
+    try {
+        re = new RegExp(pattern, "s");
+    }
+    catch (e) {
+        return failResult("regex", a, `正则无效: ${e.message}`);
+    }
+    const pass = re.test(output);
+    return {
+        type: "regex",
+        pass,
+        score: pass ? 1 : 0,
+        weight: a.weight ?? 1,
+        message: pass ? `匹配正则 /${pattern}/` : `不匹配正则 /${pattern}/`,
+    };
+}
+async function scoreJsonSchema(output, a) {
+    if (!a.schema)
+        return failResult("json-schema", a, "json-schema 断言缺少 schema");
+    const parsed = tryExtractJson(output);
+    if (parsed === undefined) {
+        return failResult("json-schema", a, "输出不是合法 JSON / 抠不出 JSON 块");
+    }
+    const validateFn = ajv.compile(a.schema);
+    const ok = validateFn(parsed);
+    return {
+        type: "json-schema",
+        pass: ok,
+        score: ok ? 1 : 0,
+        weight: a.weight ?? 1,
+        message: ok
+            ? "JSON 符合 schema"
+            : `JSON 校验失败: ${ajv.errorsText(validateFn.errors)}`,
+    };
+}
+async function scoreLlmJudge(output, a, ctx) {
+    if (!a.rubric)
+        return failResult("llm-judge", a, "llm-judge 断言缺少 rubric（评分标准）");
+    if (!ctx.judge)
+        return failResult("llm-judge", a, "没有可用的裁判模型（judge）");
+    const threshold = a.threshold ?? 0.6;
+    const raw = await ctx.judge.complete([
+        {
+            role: "system",
+            content: "你是一个严格、客观的中文评测员。根据【评分标准】给【待评测输出】打分。" +
+                '只输出一个 JSON，不要解释：{"score": 0到1之间的小数, "reason": "简短理由"}。',
+        },
+        {
+            role: "user",
+            content: `【评分标准】\n${a.rubric}\n\n【待评测输出】\n"""\n${output}\n"""`,
+        },
+    ]);
+    const parsed = tryExtractJson(raw);
+    const score = clamp01(Number(parsed?.score));
+    const pass = score >= threshold;
+    return {
+        type: "llm-judge",
+        pass,
+        score,
+        weight: a.weight ?? 1,
+        message: `裁判打分 ${score.toFixed(2)}（阈值 ${threshold}）${parsed?.reason ? " · " + parsed.reason : ""}`,
+    };
+}
+function failResult(type, a, message) {
+    return { type, pass: false, score: 0, weight: a.weight ?? 1, message };
+}
+export async function runAssertion(output, a, ctx) {
+    switch (a.type) {
+        case "contains":
+            return scoreContains(output, a);
+        case "not-contains":
+            return scoreNotContains(output, a);
+        case "regex":
+            return scoreRegex(output, a);
+        case "json-schema":
+            return scoreJsonSchema(output, a);
+        case "llm-judge":
+            return scoreLlmJudge(output, a, ctx);
+        default:
+            return failResult("contains", a, `未知断言类型: ${a.type}`);
+    }
+}

package/dist/types.js

@@ -0,0 +1 @@
+export {};

package/dist/util.js

@@ -0,0 +1,17 @@
+/** 把 "{{name}}" 占位符替换成 vars 里的值；缺失的占位符替换为空串。 */
+export function renderTemplate(tpl, vars = {}) {
+    return tpl.replace(/\{\{\s*([\w.-]+)\s*\}\}/g, (_, key) => {
+        const v = vars[key];
+        return v === undefined || v === null ? "" : String(v);
+    });
+}
+export function readApiKey(envName) {
+    if (!envName)
+        return undefined;
+    return process.env[envName];
+}
+export function clamp01(n) {
+    if (Number.isNaN(n))
+        return 0;
+    return Math.max(0, Math.min(1, n));
+}

package/package.json

@@ -0,0 +1,50 @@
+{
+  "name": "evaldrift",
+  "version": "0.1.0",
+  "description": "Snapshot-style regression testing for LLM prompts & agents — catch silent quality drift when you tweak a prompt. China-model & Chinese-first (DeepSeek / Kimi / Qwen / Doubao / Ollama).",
+  "type": "module",
+  "bin": {
+    "evaldrift": "dist/cli.js"
+  },
+  "files": [
+    "dist",
+    "templates",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "build": "tsc -p tsconfig.json",
+    "dev": "tsx src/cli.ts",
+    "prepublishOnly": "npm run build",
+    "test": "node --test"
+  },
+  "keywords": [
+    "llm",
+    "prompt",
+    "eval",
+    "evaluation",
+    "regression-testing",
+    "prompt-engineering",
+    "deepseek",
+    "qwen",
+    "kimi",
+    "ollama",
+    "ai-testing",
+    "snapshot-testing"
+  ],
+  "author": "",
+  "license": "MIT",
+  "engines": {
+    "node": ">=18"
+  },
+  "dependencies": {
+    "ajv": "^8.17.1",
+    "picocolors": "^1.1.1",
+    "yaml": "^2.6.1"
+  },
+  "devDependencies": {
+    "@types/node": "^22.10.2",
+    "tsx": "^4.19.2",
+    "typescript": "^5.7.2"
+  }
+}

package/templates/promptdrift.config.yaml

@@ -0,0 +1,66 @@
+# promptdrift 配置 —— 默认用离线 mock 模型，`promptdrift run` 立即可见效果。
+# 换成真实模型：把 provider 整段替换成下面任意一个注释示例即可。
+
+provider:
+  type: mock
+  model: demo
+  # mock 规则：模型“回复”里命中哪个关键词，就返回对应文本（仅用于离线演示/调通断言）
+  mockResponses:
+    # 这一条让离线 mock 也能扮演 llm-judge 裁判（收到评分请求时返回 JSON 分数）
+    "【待评测输出】": '{"score": 0.85, "reason": "回复礼貌并给出了下一步建议"}'
+    退款: "您好，关于退款政策：支持 7 天无理由退款，请在订单页点击申请，款项 1-3 个工作日原路退回。"
+    几点: "我们每天 09:00-22:00 营业，节假日正常营业，欢迎光临。"
+    __default__: "您好，非常抱歉让您久等了，我马上为您处理，请问具体是哪笔订单呢？"
+
+# ---- 接 DeepSeek（OpenAI 兼容）----
+# provider:
+#   type: openai
+#   model: deepseek-chat
+#   baseUrl: https://api.deepseek.com/v1
+#   apiKeyEnv: DEEPSEEK_API_KEY
+#   temperature: 0
+#
+# ---- 接 Kimi / 通义 / 豆包：同上，只改 model + baseUrl + apiKeyEnv ----
+# ---- 接本地 Ollama ----
+# provider:
+#   type: ollama
+#   model: qwen2.5
+#   baseUrl: http://localhost:11434
+
+# llm-judge 用的裁判模型（可选，缺省复用上面的 provider）
+# judge:
+#   type: openai
+#   model: deepseek-chat
+#   baseUrl: https://api.deepseek.com/v1
+#   apiKeyEnv: DEEPSEEK_API_KEY
+
+prompt:
+  system: "你是一个礼貌、专业、简洁的中文客服助手。"
+  user: "{{question}}"
+
+# 应用到每个用例的默认断言（可选）
+defaultAssert:
+  - type: regex
+    value: "^[\\s\\S]{8,}$"   # 至少有点内容，别空着
+
+tests:
+  - name: 退款问题
+    vars: { question: "我想退款，怎么操作？" }
+    assert:
+      - type: contains
+        value: ["退款", "工作日"]
+      - type: not-contains
+        value: "不支持退款"
+
+  - name: 营业时间
+    vars: { question: "你们几点开门？" }
+    assert:
+      - type: contains
+        value: "09:00"
+
+  - name: 礼貌性（裁判打分）
+    vars: { question: "你们怎么这么慢，气死我了" }
+    assert:
+      - type: llm-judge
+        rubric: "回复是否保持礼貌、安抚情绪、并给出下一步建议。礼貌且有建设性给高分，敷衍或冷漠给低分。"
+        threshold: 0.6