modelfunction-cli 0.5.4 → 0.6.0

package/README.md

@@ -5,7 +5,7 @@
 ## 安装
 
 ```bash
-npx -y modelfunction-cli@0.5.3 --version
+npx -y modelfunction-cli@0.6.0 --version
 # 或
 npm i -D modelfunction-cli
 npx mf --version
@@ -69,14 +69,14 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
   --models-file my-models.json --models internal-ds
 ```
 
-`pricing` 可省（省了就不输出成本）；要算成本请按自家价目填写，默认 registry 里的价格只是参考值。
+`pricing` 可省（省了就不输出成本）；要算成本请按自家价目填写，默认 registry 里的价格只是参考值。价格条目旁写 `"currency": "USD"`（默认 `CNY`）即可切换币种；token 用量不受此影响，始终是客观计量。
 
 ## 命令
 
 | 命令 | 作用 |
 | --- | --- |
 | `mf scan` | 扫描工具定义（function.json / OpenAPI / MCP 配置 / Python `@tool` / JS·TS 声明与 TypeBox·zod schema）；`--probe` 连 stdio 或 streamable-HTTP MCP server（运行时结果覆盖静态推断；headers/query 不写入报告）；`--from` 导入运行时工具数组；草稿按来源分组（`profiles`）并列出未覆盖的 skill |
-| `mf eval` | 跑评分卡；`--repeat` 多次取多数并报不稳定用例；`--fail-under` 阈值；`--baseline` + `--fail-on-regression` 回归门禁；`--profile` 按挂载组分别评分；`--max-cost` 按累计成本停机（退出 2）；`--progress` 逐用例进度（stderr） |
+| `mf eval` | 跑评分卡；`--repeat` 多次取多数并报不稳定用例；`--fail-under` 阈值；`--baseline` + `--fail-on-regression` 回归门禁；`--profile` 按挂载组分别评分；`--max-cost` 按累计成本停机（退出 2，币种随 registry）；`--max-tokens` 按累计 input+output token 停机（退出 2）；`--retry` 429/5xx 重试次数；`--min-interval` 请求最小间隔（压 RPM）；`--progress` 逐用例进度（stderr） |
 | `mf case add` | 写入用例（默认 `source=agent`、`reviewed=false`） |
 | `mf coverage` | 每工具用例数、参数覆盖、负例比例、疑似混淆对；有 `profiles` 时按组统计 |
 | `mf audit` | 一次静态审计：lint + 混淆族边界 + 用例覆盖 + 挂载检查，输出可转发的 Markdown/JSON；不调用模型 |
@@ -126,21 +126,19 @@ mf eval --suite mf.suite.json --models internal-ds \
   "mcpServers": {
     "modelfunction": {
       "command": "npx",
-      "args": ["-y", "modelfunction-cli@0.5.3", "mcp"]
+      "args": ["-y", "modelfunction-cli@0.6.0", "mcp"]
     }
   }
 }
 ```
 
-暴露：`scan_tools`、`list_suite`、`coverage`、`audit`、`lint`、`check`、`cluster`、`compare`、`add_cases`、`run_eval`（`run_eval` 支持 `profile`、`repeat`、`maxCost`、`progress`）。`lint`/`check`/`cluster`/`compare` 都是静态操作，不调用模型；`check` 就是安装门禁，Agent 可直接传入 manifest 与 installed。
+暴露：`scan_tools`、`list_suite`、`coverage`、`audit`、`lint`、`check`、`cluster`、`compare`、`add_cases`、`run_eval`（`run_eval` 支持 `profile`、`repeat`、`maxCost`、`maxTokens`、`retry`、`minInterval`、`progress`）。`lint`/`check`/`cluster`/`compare` 都是静态操作，不调用模型；`check` 就是安装门禁，Agent 可直接传入 manifest 与 installed。
 
 ## 契约与版本
 
 对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md)，自 0.4.3 起冻结为 v1：v1 内只做加法，重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
 
-`0.5.3` 已发布到 npm：含凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查、门禁严重度收敛、静态规则性能（真实 100 能力清单 `checkMany` 471ms→40ms）、`installed` 形状校验与 `profiles` 平台形态兼容。示例统一使用 `@0.5.3`。
-
-`0.5.4`（待发布）：用例新增 `excludeFromBase` / `excludeFromProfiles`（布尔，排除计分但不省调用）；评分卡新增行级 `errorKind` 与 `errors` / `errorsByKind` / `scoredTotal` / `selectionAccuracyExcludingErrors`；挂载对比不再把超时/HTTP 错误记成 jammed，分组表单列 `errors`；`mf audit` 分组表增加 `neg%`。契约见 `SCHEMA.md`。
+`0.6.0` 待 npm 发布：token 成为主口径（评分卡新增 `totalTokens`），模型注册表支持 `currency`（CNY/USD，默认 CNY），成本字段改为 `totalCost` / `avgCost` + `costCurrency`（`totalCostCny` / `avgCostCny` 仅在 CNY 时保留），新增 `mf eval --max-tokens <n>` 与 MCP `maxTokens`；`--fail-on-regression` 在基线模型缺失或本轮全部调用失败时改为 exit 2（无法判定 ≠ 无回归），自动选模型时在 stderr 打 note；本地门禁模板改用退出码判定、只在成功 eval 后更新 state，`.cmd` 固定 CRLF。`0.5.5` 含 eval `--retry <n>` / `--min-interval <ms>`、评分卡 `retries`、CI 模板 static 层 `cluster --json`、`templates/local/` 与 `docs/local-gate.md`（内网/非 GitHub 环境的 pre-push 静态门禁、变更门、定时回归）。`0.5.4` 含用例级 `excludeFromBase` / `excludeFromProfiles`、评分卡 `errorKind` / `errorsByKind` / `scoredTotal` / `selectionAccuracyExcludingErrors`、挂载对比错误不计 jammed、audit 分组 `neg%` 与分组 token 用量聚合。`0.5.3` 及更早的能力见下：凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查、门禁严重度收敛、静态规则性能（真实 100 能力清单 `checkMany` 471ms→40ms）、`installed` 形状校验与 `profiles` 平台形态兼容。示例统一使用 `@0.6.0`。
 
 ## 数据边界
 
@@ -169,7 +167,7 @@ mf feedback --out out.json  # 导出脱敏反馈包（聚合值，不含原始
 
 ```bash
 # 1. 把 mf 挂成 MCP server
-claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.3 mcp
+claude mcp add modelfunction -- npx -y modelfunction-cli@0.6.0 mcp
 ```
 
 ```text

package/SCHEMA.md

@@ -95,21 +95,26 @@ skills 只做静态检查，不参与模型评测。
   "models": [ { "model": "dsv4-flash", "total": 12, "reviewedTotal": 12, "excludedTotal": 0,
                "selectionAccuracy": 0.83, "selectionAccuracyExcludingErrors": 0.83,
                "argAccuracy": 0.9, "passRate": 0.75, "errors": 0, "errorsByKind": {},
-               "scoredTotal": 12, "p50LatencyMs": 900,
-               "p95LatencyMs": 1800, "avgCostCny": 0.001, "totalCostCny": 0.012,
+               "scoredTotal": 12, "retries": 0, "p50LatencyMs": 900,
+               "p95LatencyMs": 1800, "avgCost": 0.001, "totalCost": 0.012,
+               "costCurrency": "CNY", "avgCostCny": 0.001, "totalCostCny": 0.012,
                "inputTokens": 1000, "cachedTokens": 0, "outputTokens": 200,
+               "totalTokens": 1200,
                "rows": [ { "id": "c1", "expectedTool": "...", "actualTool": "...",
                            "selectionCorrect": true, "args": { "total": 1, "matched": 1, "failures": [] },
                            "passed": true, "reviewed": true, "source": "manual",
-                           "latencyMs": 900, "usage": { "input": 80, "output": 20 }, "error": null } ] } ],
+                           "latencyMs": 900, "usage": { "input": 80, "output": 20 }, "error": null,
+                           "retries": 1 } ] } ],
   "profiles": { "groups": [ ... ] }
 }
 ```
 
 - 正式指标只统计 `reviewed` 且未被 `excludeFromBase` 排除的行；未复核与排除结果单独给出。
 - `errors` 是调用层失败（超时、HTTP、网络）的数量，按 `errorsByKind` 分类；它们仍计入 `selectionAccuracy`（保守口径），同时给出排除错误后的 `selectionAccuracyExcludingErrors` 与可评分行数 `scoredTotal`。
+- `retries` 是本轮因 429/5xx/网络错误而重试的请求次数（模型级汇总；行级只在大于 0 时出现）。重试不改变判分：最终成功的请求按正常结果计分，最终失败的仍计入 `errors`。
+- `inputTokens` / `cachedTokens` / `outputTokens` / `totalTokens` 是客观计量（`totalTokens = inputTokens + outputTokens`，缓存 token 已含在输入内）。成本需要模型注册表的 `pricing`；`currency` 取模型注册表条目（CNY/USD，默认 CNY）。读 `totalCost` / `avgCost` 时一并读 `costCurrency`；`totalCostCny` / `avgCostCny` 只在 CNY 时出现，是兼容别名。
 - `--repeat` 时模型对象带 `stability`（翻转用例与多次结果分布）。
-- `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`、`errors`；错误行不参与挂载对比（不会被记成 jammed），每个组的 `reviewed` 是可评分行数。
+- `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`、`errors`，以及 `usage`（`inputTokens` / `cachedTokens` / `outputTokens` / `totalTokens` / `cost` / `costCurrency`；CNY 时另带 `costCny`）；错误行不参与挂载对比（不会被记成 jammed），每个组的 `reviewed` 是可评分行数。
 - `fingerprint` 由工具与用例内容计算；`mf compare` 用它判断两份报告是否可比。
 
 ## 3. 扫描输出（`mf scan`）
@@ -225,7 +230,7 @@ const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
 | `cluster` | `suite?`、`threshold?`、`maxSize?` |
 | `compare` | `before`、`after`（评分卡 JSON 路径） |
 | `add_cases` | `suite?`、`cases[]`、`source?`、`reviewed?`、`dryRun?` |
-| `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?`、`maxCost?`、`progress?` |
+| `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?`、`maxCost?`、`maxTokens?`、`retry?`、`minInterval?`、`progress?` |
 
 返回值均为 JSON 文本。
 
@@ -235,7 +240,7 @@ const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
 | --- | --- |
 | 0 | 成功（`mf check` 的 warn 也是 0） |
 | 1 | 输入/用法错误；`mf compare` 判定不可直接比较 |
-| 2 | `mf eval --fail-under` 未达标、`--fail-on-regression` 发现回退，或 `--max-cost` 超限 |
+| 2 | `mf eval --fail-under` 未达标、`--fail-on-regression` 发现回退或无法判定（基线模型缺失、本轮全部调用失败）、`--max-cost` / `--max-tokens` 超限 |
 | 3 | `mf exec` 被权限策略拒绝；`mf scan --json` 未找到工具或 skill |
 | 4 | `mf exec` 超时 |
 

package/fixtures/models.json

@@ -1,11 +1,12 @@
 {
-  "_note": "pricing 为人民币/百万 tokens 的参考值，运行前请按各家最新价目核对",
+  "_note": "pricing 为每百万 tokens 的参考价，currency 指定币种（CNY/USD，默认 CNY）；运行前请按各家最新价目核对",
   "mock": { "type": "mock" },
   "deepseek": {
     "type": "openai",
     "baseUrl": "https://api.deepseek.com/v1",
     "model": "deepseek-chat",
     "apiKeyEnv": "DEEPSEEK_API_KEY",
+    "currency": "CNY",
     "pricing": { "input": 2, "output": 8 }
   },
   "dsv4-flash": {
@@ -13,6 +14,7 @@
     "baseUrl": "https://api.siliconflow.cn/v1",
     "model": "deepseek-ai/DeepSeek-V4-Flash",
     "apiKeyEnv": "SILICONFLOW_API_KEY",
+    "currency": "CNY",
     "pricing": { "input": 1.0, "cachedInput": 0.02, "output": 2.0 }
   },
   "qwen": {
@@ -20,6 +22,7 @@
     "baseUrl": "https://dashscope.aliyuncs.com/compatible-mode/v1",
     "model": "qwen-plus",
     "apiKeyEnv": "DASHSCOPE_API_KEY",
+    "currency": "CNY",
     "pricing": { "input": 0.8, "output": 2 }
   },
   "glm": {
@@ -27,6 +30,7 @@
     "baseUrl": "https://open.bigmodel.cn/api/paas/v4",
     "model": "glm-4-air",
     "apiKeyEnv": "ZHIPU_API_KEY",
+    "currency": "CNY",
     "pricing": { "input": 0.5, "output": 0.5 }
   },
   "openai": {
@@ -34,6 +38,7 @@
     "baseUrl": "https://api.openai.com/v1",
     "model": "gpt-4o-mini",
     "apiKeyEnv": "OPENAI_API_KEY",
+    "currency": "CNY",
     "pricing": { "input": 1.1, "output": 4.3 }
   }
 }

package/lib/adapters/openai.js

@@ -1,5 +1,6 @@
 const TIMEOUT_MS = 60_000;
-const MAX_ATTEMPTS = 3;
+const DEFAULT_RETRIES = 2;
+const MAX_RETRY_DELAY_MS = 30_000;
 
 export function createOpenAiAdapter(config) {
   const apiKey = process.env[config.apiKeyEnv];
@@ -7,13 +8,25 @@ export function createOpenAiAdapter(config) {
     throw new Error(`${config.id}: environment variable ${config.apiKeyEnv} is not set`);
   }
   const endpoint = `${config.baseUrl.replace(/\/+$/, "")}/chat/completions`;
+  const retries = clampInt(config.retries, 0, 10, DEFAULT_RETRIES);
+  const minIntervalMs = clampInt(config.minIntervalMs, 0, 60_000, 0);
+  let lastRequestAt = 0;
+
+  async function throttle() {
+    if (minIntervalMs <= 0) return;
+    const wait = lastRequestAt + minIntervalMs - Date.now();
+    if (wait > 0) await sleep(wait);
+    lastRequestAt = Date.now();
+  }
 
   return {
     label: config.model,
     async call({ messages, tools }) {
       const started = performance.now();
       let lastError = "request failed";
-      for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt += 1) {
+      for (let attempt = 1; attempt <= retries + 1; attempt += 1) {
+        const retried = attempt - 1;
+        await throttle();
         try {
           const response = await fetch(endpoint, {
             method: "POST",
@@ -33,13 +46,16 @@ export function createOpenAiAdapter(config) {
 
           if (response.status === 429 || response.status >= 500) {
             lastError = `HTTP ${response.status}: ${(await response.text()).slice(0, 300)}`;
-            await sleep(attempt * 1000);
+            if (attempt <= retries) {
+              await sleep(retryDelayMs(response.headers.get("retry-after"), retried));
+            }
             continue;
           }
           if (!response.ok) {
             const body = await response.text();
             return {
               ok: false,
+              retries: retried,
               error: `HTTP ${response.status}: ${body.slice(0, 300)}`,
               toolCalls: [],
               usage: { input: 0, output: 0 },
@@ -55,6 +71,7 @@ export function createOpenAiAdapter(config) {
           }));
           return {
             ok: true,
+            retries: retried,
             toolCalls,
             usage: {
               input: data.usage?.prompt_tokens ?? 0,
@@ -66,11 +83,12 @@ export function createOpenAiAdapter(config) {
           };
         } catch (error) {
           lastError = error?.name === "TimeoutError" ? `timeout after ${TIMEOUT_MS}ms` : String(error?.message ?? error);
-          if (attempt < MAX_ATTEMPTS) await sleep(attempt * 1000);
+          if (attempt <= retries) await sleep(retryDelayMs(null, retried));
         }
       }
       return {
         ok: false,
+        retries,
         error: lastError,
         toolCalls: [],
         usage: { input: 0, output: 0 },
@@ -80,6 +98,22 @@ export function createOpenAiAdapter(config) {
   };
 }
 
+function retryDelayMs(retryAfter, retried) {
+  const header = retryAfter == null || String(retryAfter).trim() === "" ? NaN : Number(retryAfter);
+  if (Number.isFinite(header) && header >= 0) {
+    return Math.min(header * 1000, MAX_RETRY_DELAY_MS);
+  }
+  return Math.min(1000 * 2 ** retried, MAX_RETRY_DELAY_MS);
+}
+
+function clampInt(value, min, max, fallback) {
+  const parsed = Number(value);
+  if (!Number.isFinite(parsed)) return fallback;
+  const rounded = Math.floor(parsed);
+  if (rounded < min || rounded > max) return fallback;
+  return rounded;
+}
+
 function parseArguments(raw) {
   if (raw === undefined || raw === null || raw === "") return {};
   if (typeof raw === "object") return raw;

package/lib/budget.js

@@ -1,9 +1,9 @@
-// Running cost guard for evaluations. Pricing is known per model, but token
-// usage is only known after a call returns, so the limit is enforced after
-// each call: the run stops as soon as the budget is crossed.
+// Running guards for evaluations. Token and cost usage are only known after a
+// call returns, so limits are enforced after each call: the run stops as soon
+// as either limit is crossed.
 export class CostLimitError extends Error {
   constructor({ limit, spent, calls }) {
-    super(`cost limit reached: spent ¥${spent.toFixed(4)} of ¥${limit} after ${calls} call(s)`);
+    super(`cost limit reached: spent ${spent.toFixed(4)} of ${limit} after ${calls} call(s)`);
     this.name = "CostLimitError";
     this.limit = limit;
     this.spent = spent;
@@ -11,20 +11,42 @@ export class CostLimitError extends Error {
   }
 }
 
-export function createBudget(limit = null) {
+export class TokenLimitError extends Error {
+  constructor({ limit, spent, calls }) {
+    super(`token limit reached: used ${spent} of ${limit} tokens after ${calls} call(s)`);
+    this.name = "TokenLimitError";
+    this.limit = limit;
+    this.spent = spent;
+    this.calls = calls;
+  }
+}
+
+export function createBudget(limits = null) {
+  const { maxCost = null, maxTokens = null } =
+    typeof limits === "object" && limits !== null
+      ? limits
+      : { maxCost: limits, maxTokens: null };
   let spent = 0;
+  let tokens = 0;
   let calls = 0;
   return {
     get spent() {
       return spent;
     },
+    get tokens() {
+      return tokens;
+    },
     get calls() {
       return calls;
     },
-    spend(cost) {
+    spend(usage) {
+      const { cost, tokens: usedTokens } =
+        typeof usage === "number" ? { cost: usage, tokens: 0 } : (usage ?? {});
       calls += 1;
       if (Number.isFinite(cost) && cost > 0) spent += cost;
-      if (limit !== null && spent > limit) throw new CostLimitError({ limit, spent, calls });
+      if (Number.isFinite(usedTokens) && usedTokens > 0) tokens += usedTokens;
+      if (maxCost !== null && spent > maxCost) throw new CostLimitError({ limit: maxCost, spent, calls });
+      if (maxTokens !== null && tokens > maxTokens) throw new TokenLimitError({ limit: maxTokens, spent: tokens, calls });
     }
   };
 }

package/lib/compare.js

@@ -182,6 +182,46 @@ function compareProfiles(before, after) {
   };
 }
 
+export function regressionGate(comparison, summaries = []) {
+  const missing = comparison.models.filter((model) => model.reason === "模型不在新报告中");
+  if (missing.length > 0) {
+    return {
+      status: "fail",
+      message: `baseline model(s) not in this report: ${missing.map((model) => model.model).join(", ")}；无法判定回归（门禁失败）`
+    };
+  }
+
+  const errored = summaries.filter(
+    (summary) => summary.reviewedTotal > 0 && summary.errors === summary.reviewedTotal
+  );
+  if (errored.length > 0) {
+    const detail = errored
+      .map((summary) => {
+        const kinds = summary.errorsByKind && Object.keys(summary.errorsByKind).length
+          ? ` ${JSON.stringify(summary.errorsByKind)}`
+          : "";
+        return `${summary.model} (${summary.errors}/${summary.reviewedTotal}${kinds})`;
+      })
+      .join(", ");
+    return {
+      status: "fail",
+      message: `all calls failed for: ${detail}；基础设施故障，无法判定回归（门禁失败）`
+    };
+  }
+
+  const comparable = comparison.models.filter((model) => model.comparable && model.reviewed > 0);
+  if (comparable.length === 0) {
+    return { status: "skip", message: "定义或用例已变更，无法判定回归（门禁跳过）" };
+  }
+
+  return {
+    status: "ok",
+    regressions: comparable.flatMap((model) =>
+      model.regressions.map((item) => `${model.model}/${item.id}`)
+    )
+  };
+}
+
 export function formatCompare(report) {
   const lines = [];
   if (!report.beforeHash || !report.afterHash) {

package/lib/exec.js

@@ -1,4 +1,5 @@
 import { spawn } from "node:child_process";
+import { realpathSync } from "node:fs";
 import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { dirname, join, resolve } from "node:path";
@@ -13,18 +14,19 @@ const ENV_KEEP = ["PATH", "SystemRoot", "SYSTEMROOT", "TEMP", "TMP", "HOME", "US
 export async function executeTool(options) {
   const runtime = options.runtime ?? "node";
   const engine = runtime === "python" ? "python-audit" : "node-permission";
-  const entry = resolve(options.entry);
+  const entry = realpathAllow(options.entry);
   const nodeMajor = Number(process.versions.node.split(".")[0]);
   if (runtime === "node" && options.allowNet?.length && nodeMajor < 25) {
     throw new Error("--allow-net 需要 Node >= 25（Node 24 的权限模型不限制网络）");
   }
 
   let tempDir = null;
-  let inputPath = options.inputPath ? resolve(options.inputPath) : null;
+  let inputPath = options.inputPath ? realpathAllow(options.inputPath) : null;
   if (!inputPath) {
     tempDir = await mkdtemp(join(tmpdir(), "mf-exec-"));
     inputPath = join(tempDir, "input.json");
     await writeFile(inputPath, options.inputJson ?? "{}", "utf8");
+    inputPath = realpathAllow(inputPath);
   }
 
   try {
@@ -57,11 +59,11 @@ export async function executeTool(options) {
         dirname(inputPath),
         process.cwd(),
         ...(options.allowRead ?? [])
-      ].map((path) => resolve(path));
+      ].map((path) => realpathAllow(path));
       args = [permissionFlag()];
       for (const path of dedupe(readPaths)) args.push(`--allow-fs-read=${path}`);
       if (options.allowWrite?.length) {
-        for (const path of dedupe(options.allowWrite.map((item) => resolve(item)))) {
+        for (const path of dedupe(options.allowWrite.map((item) => realpathAllow(item)))) {
           args.push(`--allow-fs-write=${path}`);
         }
       }
@@ -264,3 +266,14 @@ function formatExec(entry, result) {
   if (result.permissionDenied) lines.push("  note:       权限模型拒绝了文件/网络访问，可用于推断沙箱需求");
   return lines.join("\n");
 }
+
+// Node's permission model matches real paths. On macOS /var is a symlink to
+// /private/var, so an unresolved tmpdir path in --allow-fs-read is rejected.
+function realpathAllow(path) {
+  const abs = resolve(path);
+  try {
+    return realpathSync.native ? realpathSync.native(abs) : realpathSync(abs);
+  } catch {
+    return abs;
+  }
+}

package/lib/mcp-server.js

@@ -247,11 +247,14 @@ export async function startMcpServer() {
           .optional()
           .describe("每条用例重复次数，取多数结果；建议正式基线用 3"),
         failUnder: z.number().optional().describe("选择准确率阈值，低于则退出码为 2"),
-        maxCost: z.number().positive().optional().describe("累计估算成本上限（CNY），超限退出 2 且不返回部分评分卡"),
+        maxCost: z.number().positive().optional().describe("累计估算成本上限（币种见模型注册表 currency），超限退出 2 且不返回部分评分卡"),
+        maxTokens: z.number().int().positive().optional().describe("累计 input+output token 上限，超限退出 2 且不返回部分评分卡"),
+        retry: z.number().int().min(0).max(10).optional().describe("单次请求遇到 429/5xx/网络错误的重试次数（默认 2，共 3 次尝试）"),
+        minInterval: z.number().int().min(0).max(60000).optional().describe("两次模型请求的最小间隔（毫秒），用于压住端点 RPM"),
         progress: z.boolean().optional().describe("把逐用例进度写到 stderr（stdout 仍为纯 JSON）")
       }
     },
-    async ({ suite, models, cases, profile, repeat, failUnder, maxCost, progress }) => {
+    async ({ suite, models, cases, profile, repeat, failUnder, maxCost, maxTokens, retry, minInterval, progress }) => {
       const args = ["eval", "--json", "--suite", suitePath(suite)];
       if (models) args.push("--models", models);
       if (cases) args.push("--case", cases);
@@ -259,6 +262,9 @@ export async function startMcpServer() {
       if (repeat !== undefined) args.push("--repeat", String(repeat));
       if (failUnder !== undefined) args.push("--fail-under", String(failUnder));
       if (maxCost !== undefined) args.push("--max-cost", String(maxCost));
+      if (maxTokens !== undefined) args.push("--max-tokens", String(maxTokens));
+      if (retry !== undefined) args.push("--retry", String(retry));
+      if (minInterval !== undefined) args.push("--min-interval", String(minInterval));
       if (progress) args.push("--progress");
 
       const result = await runCli(args);

package/lib/profile.js

@@ -146,7 +146,16 @@ export function modelTotals(entries) {
   let errors = 0;
   let mountedCorrect = 0;
   let allCorrect = 0;
-  const usage = { inputTokens: 0, cachedTokens: 0, outputTokens: 0, costCny: 0, estimated: false };
+  const usage = {
+    inputTokens: 0,
+    cachedTokens: 0,
+    outputTokens: 0,
+    totalTokens: 0,
+    cost: 0,
+    costCurrency: null,
+    costCny: 0,
+    estimated: false
+  };
   for (const entry of entries) {
     reviewed += entry.reviewed;
     errors += entry.errors ?? 0;
@@ -155,9 +164,13 @@ export function modelTotals(entries) {
     usage.inputTokens += entry.usage?.inputTokens ?? 0;
     usage.cachedTokens += entry.usage?.cachedTokens ?? 0;
     usage.outputTokens += entry.usage?.outputTokens ?? 0;
+    usage.totalTokens += entry.usage?.totalTokens ?? 0;
+    usage.cost += entry.usage?.cost ?? 0;
+    usage.costCurrency ||= entry.usage?.costCurrency ?? null;
     usage.costCny += entry.usage?.costCny ?? 0;
     usage.estimated ||= Boolean(entry.usage?.estimated);
   }
+  usage.costCurrency ??= "CNY";
   if (reviewed === 0) {
     return {
       reviewed: 0,

package/lib/report.js

@@ -1,10 +1,13 @@
 import { pct } from "./score.js";
 import { suiteFingerprint } from "./fingerprint.js";
 
-function money(value, estimated) {
-  if (value === 0) return "0";
+const CURRENCY_SYMBOLS = { CNY: "¥", USD: "$" };
+
+function money(value, estimated, currency = "CNY") {
+  if (!value) return "0";
   const text = value >= 0.01 ? value.toFixed(3) : value.toFixed(4);
-  return `${estimated ? "~" : ""}${text}`;
+  const symbol = CURRENCY_SYMBOLS[currency] ?? `${currency} `;
+  return `${estimated ? "~" : ""}${symbol}${text}`;
 }
 
 function isEstimated(summary) {
@@ -32,7 +35,7 @@ export function formatCaseLines(summary, cases) {
 }
 
 export function formatTable(summaries) {
-  const headers = ["model", "select", "args", "pass", "p50ms", "p95ms", "cny/call", "errors"];
+  const headers = ["model", "select", "args", "pass", "p50ms", "p95ms", "cost/call", "errors"];
   const rows = summaries.map((summary) => [
     summary.model,
     pct(summary.selectionAccuracy),
@@ -40,7 +43,7 @@ export function formatTable(summaries) {
     pct(summary.passRate),
     String(summary.p50LatencyMs),
     String(summary.p95LatencyMs),
-    money(summary.avgCostCny, isEstimated(summary)),
+    money(summary.avgCost ?? summary.avgCostCny, isEstimated(summary), summary.costCurrency ?? "CNY"),
     String(summary.errors)
   ]);
 
@@ -71,8 +74,8 @@ export function formatMarkdown(suite, cases, summaries) {
 
   const withExec = summaries.some((summary) => summary.exec);
   const headers = withExec
-    ? ["model", "select", "args", "pass", "exec", "run p50 ms", "max RSS MB", "p50 ms", "p95 ms", "CNY/call", "errors"]
-    : ["model", "select", "args", "pass", "p50 ms", "p95 ms", "CNY/call", "errors"];
+    ? ["model", "select", "args", "pass", "exec", "run p50 ms", "max RSS MB", "p50 ms", "p95 ms", "cost/call", "errors"]
+    : ["model", "select", "args", "pass", "p50 ms", "p95 ms", "cost/call", "errors"];
   lines.push(`| ${headers.join(" | ")} |`, `| ${headers.map(() => "---").join(" | ")} |`);
 
   for (const summary of summaries) {
@@ -90,7 +93,7 @@ export function formatMarkdown(suite, cases, summaries) {
     cells.push(
       String(summary.p50LatencyMs),
       String(summary.p95LatencyMs),
-      money(summary.avgCostCny, isEstimated(summary)),
+      money(summary.avgCost ?? summary.avgCostCny, isEstimated(summary), summary.costCurrency ?? "CNY"),
       String(summary.errors)
     );
     lines.push(`| ${cells.join(" | ")} |`);

package/lib/score.js

@@ -76,7 +76,11 @@ export function pickRepresentative(runs) {
   };
 }
 
-export function costCnyOf(usage, pricing) {
+export function tokensOf(usage) {
+  return (usage?.input ?? 0) + (usage?.output ?? 0);
+}
+
+export function costOf(usage, pricing) {
   if (!pricing) return 0;
   const input = usage?.input ?? 0;
   const cached = Math.min(input, usage?.cached ?? 0);
@@ -89,7 +93,7 @@ export function costCnyOf(usage, pricing) {
   );
 }
 
-export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
+export function scoreModel({ model, cases, results, pricing, scope = "base", currency = "CNY" }) {
   const rows = cases.map((testCase, index) => {
     const result = results[index];
     const actualTool = result.ok ? (result.toolCalls?.[0]?.name ?? null) : null;
@@ -112,6 +116,7 @@ export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
       source: testCase.source ?? "manual",
       latencyMs: result.latencyMs ?? 0,
       usage: result.usage ?? { input: 0, output: 0 },
+      ...(result.retries > 0 ? { retries: result.retries } : {}),
       error,
       ...(error ? { errorKind: classifyError(error) } : {})
     };
@@ -143,7 +148,9 @@ export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
     inputTokens,
     rows.reduce((sum, row) => sum + (row.usage.cached ?? 0), 0)
   );
-  const costCny = rows.reduce((sum, row) => sum + costCnyOf(row.usage, pricing), 0);
+  const totalCost = rows.reduce((sum, row) => sum + costOf(row.usage, pricing), 0);
+  const totalTokens = inputTokens + outputTokens;
+  const costCurrency = String(currency || "CNY").trim().toUpperCase();
   const reviewedTotal = reviewedRows.length;
 
   return {
@@ -162,13 +169,19 @@ export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
     errors,
     errorsByKind,
     scoredTotal,
+    retries: rows.reduce((sum, row) => sum + (row.retries ?? 0), 0),
     p50LatencyMs: percentile(latencies, 0.5),
     p95LatencyMs: percentile(latencies, 0.95),
-    avgCostCny: total ? costCny / total : 0,
-    totalCostCny: costCny,
+    avgCost: total ? totalCost / total : 0,
+    totalCost,
+    costCurrency,
+    ...(costCurrency === "CNY"
+      ? { avgCostCny: total ? totalCost / total : 0, totalCostCny: totalCost }
+      : {}),
     inputTokens,
     cachedTokens,
     outputTokens,
+    totalTokens,
     rows
   };
 }

package/mf.js

@@ -4,7 +4,7 @@ import { mkdir, readFile } from "node:fs/promises";
 import { basename, dirname, extname, relative, resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 import { loadSuite, toModelTools } from "./lib/suite.js";
-import { scoreModel, pickRepresentative, pct, costCnyOf } from "./lib/score.js";
+import { scoreModel, pickRepresentative, pct, costOf, tokensOf } from "./lib/score.js";
 import { formatTable, formatCaseLines, formatMarkdown } from "./lib/report.js";
 import { scanProject, buildSuiteDraft, formatScanSummary, addTools, redactReport } from "./lib/scan.js";
 import { probeStdioServer, probeHttpServer } from "./lib/mcp-probe.js";
@@ -16,7 +16,7 @@ import { auditSuite, formatAudit } from "./lib/audit.js";
 import { suiteFingerprint, hashText, hashOf } from "./lib/fingerprint.js";
 import { runExecutionChecks } from "./lib/eval-exec.js";
 import { checkSandbox, formatSandboxReport } from "./lib/sandbox-check.js";
-import { compareScorecards, formatCompare } from "./lib/compare.js";
+import { compareScorecards, formatCompare, regressionGate } from "./lib/compare.js";
 import { analyzeAblation, formatAblation } from "./lib/ablate.js";
 import { clusterTools, formatClusters } from "./lib/cluster.js";
 import {
@@ -31,7 +31,7 @@ import {
   formatProfileMarkdown
 } from "./lib/profile.js";
 import { runExec } from "./lib/exec.js";
-import { createBudget, CostLimitError } from "./lib/budget.js";
+import { createBudget, CostLimitError, TokenLimitError } from "./lib/budget.js";
 import { writeFileAtomic } from "./lib/fsutil.js";
 import {
   applyModelCredentials,
@@ -77,7 +77,7 @@ Usage:
   mf lint [--suite <file>] [--json]  # 静态检查 tools 与 skills，不调用模型
   mf check (--manifest <file> | --manifests <file>) [--installed <file>] [--json]  # 安装门禁：新能力 × 已装能力（--manifests 支持一包）
   mf compare <before.json> <after.json> [--json]
-  mf ablate --tool <name> [--suite <file>] [--profile <group>] [--models <ids>] [--max-cost <cny>] [--progress] [--json]
+  mf ablate --tool <name> [--suite <file>] [--profile <group>] [--models <ids>] [--max-cost <amount>] [--max-tokens <n>] [--progress] [--json]
   mf cluster [--suite <file>] [--threshold <0.5>] [--max-size <10>] [--out <file>] [--json]  # 建议挂载分组 + 同族候选
   mf exec --entry <file> (--input <json> | --input-json <json>) [--runtime node|python] [--python <cmd>] [--export run] [--allow-read <paths>] [--allow-write <paths>] [--allow-net <hosts>] [--timeout <ms>] [--warmup <n>] [--json]
   mf sandbox check [--json]   # probe available isolation engines on this machine
@@ -96,7 +96,10 @@ Eval options:
   --case <ids>          run only these case ids (comma-separated, repeatable)
   --profile <names>     score each mounted group separately ("all" or names from suite.profiles)
   --repeat <n>          run each case n times and score the majority; reports unstable cases (default 1)
-  --max-cost <cny>      stop after the estimated spend crosses this limit (exit 2; no partial scorecard)
+  --max-cost <amount>   stop after the estimated spend crosses this limit (exit 2; no partial scorecard)
+  --max-tokens <n>      stop after cumulative input+output tokens cross this limit (exit 2)
+  --retry <n>           retries per request on HTTP 429/5xx and network errors (default 2; 3 attempts total)
+  --min-interval <ms>   minimum spacing between model requests; use it to stay under an endpoint RPM
   --progress            print per-case progress to stderr (stdout stays clean for --json)
   --show-cases          print per-case results
   --json                print JSON instead of the table
@@ -136,14 +139,19 @@ export async function runEval(options, io = console) {
   }
   const suite = await loadSuite(options.suite);
   const cases = selectCases(suite.cases, options.cases);
-  const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io);
+  const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io, {
+    retries: options.retries,
+    minIntervalMs: options.minIntervalMs
+  });
   const tools = toModelTools(suite.tools);
 
-  const budget = createBudget(options.maxCost ?? null);
+  const budget = createBudget({ maxCost: options.maxCost ?? null, maxTokens: options.maxTokens ?? null });
   const runs = [];
   try {
     for (const adapter of adapters) {
-      const pricing = registry[adapter.id]?.pricing ?? null;
+      const config = registry[adapter.id];
+      const pricing = config?.pricing ?? null;
+      const currency = currencyOf(adapter.id, config);
       const evaluated = await evaluate(adapter, tools, cases, options.repeat ?? 1, {
         pricing,
         budget,
@@ -154,14 +162,15 @@ export async function runEval(options, io = console) {
         cases,
         results: evaluated.results,
         pricing,
-        scope: "base"
+        scope: "base",
+        currency
       });
       if (evaluated.stability) summary.stability = evaluated.stability;
       runs.push({ adapter, results: evaluated.results, summary });
     }
   } catch (error) {
-    if (error instanceof CostLimitError) {
-      io.error(costLimitMessage(error));
+    if (error instanceof CostLimitError || error instanceof TokenLimitError) {
+      io.error(budgetLimitMessage(error));
       return 2;
     }
     throw error;
@@ -208,8 +217,8 @@ export async function runEval(options, io = console) {
         io
       });
     } catch (error) {
-      if (error instanceof CostLimitError) {
-        io.error(costLimitMessage(error));
+      if (error instanceof CostLimitError || error instanceof TokenLimitError) {
+        io.error(budgetLimitMessage(error));
         return 2;
       }
       throw error;
@@ -306,13 +315,15 @@ export async function runEval(options, io = console) {
       io.log("");
       io.log(formatCompare(comparison));
     }
-    const comparable = comparison.models.filter((model) => model.comparable && model.reviewed > 0);
-    if (comparable.length === 0) {
-      io.error("定义或用例已变更，无法判定回归（门禁跳过）");
+    const gate = regressionGate(comparison, summaries);
+    if (gate.status === "fail") {
+      io.error(gate.message);
+      return 2;
+    }
+    if (gate.status === "skip") {
+      io.error(gate.message);
     } else {
-      regressions = comparable.flatMap((model) =>
-        model.regressions.map((item) => `${model.model}/${item.id}`)
-      );
+      regressions = gate.regressions;
       if (regressions.length > 0) io.error(`regressions: ${regressions.join(", ")}`);
     }
   }
@@ -393,16 +404,21 @@ export async function runAblate(options, io = console) {
     : selectedCases;
   const reducedCases = baseCases.filter((testCase) => testCase.expectedTool !== ablated.name);
   const droppedCases = baseCases.length - reducedCases.length;
-  const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io);
+  const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io, {
+    retries: options.retries,
+    minIntervalMs: options.minIntervalMs
+  });
 
   const fullTools = toModelTools(baseTools);
   const reducedTools = toModelTools(baseTools.filter((tool) => tool.name !== ablated.name));
   const models = [];
-  const budget = createBudget(options.maxCost ?? null);
+  const budget = createBudget({ maxCost: options.maxCost ?? null, maxTokens: options.maxTokens ?? null });
 
   try {
     for (const adapter of adapters) {
-      const pricing = registry[adapter.id]?.pricing ?? null;
+      const config = registry[adapter.id];
+      const pricing = config?.pricing ?? null;
+      const currency = currencyOf(adapter.id, config);
       const fullResults = (
         await evaluate(adapter, fullTools, baseCases, 1, {
           pricing,
@@ -410,7 +426,7 @@ export async function runAblate(options, io = console) {
           onProgress: makeProgress(options, io, `${adapter.id}/full`, baseCases.length)
         })
       ).results;
-      const full = scoreModel({ model: adapter.id, cases: baseCases, results: fullResults, pricing });
+      const full = scoreModel({ model: adapter.id, cases: baseCases, results: fullResults, pricing, currency });
       const reducedResults = (
         await evaluate(adapter, reducedTools, reducedCases, 1, {
           pricing,
@@ -418,12 +434,12 @@ export async function runAblate(options, io = console) {
           onProgress: makeProgress(options, io, `${adapter.id}/ablated`, reducedCases.length)
         })
       ).results;
-      const reduced = scoreModel({ model: adapter.id, cases: reducedCases, results: reducedResults, pricing });
+      const reduced = scoreModel({ model: adapter.id, cases: reducedCases, results: reducedResults, pricing, currency });
       models.push(analyzeAblation(adapter.id, full, reduced));
     }
   } catch (error) {
-    if (error instanceof CostLimitError) {
-      io.error(costLimitMessage(error));
+    if (error instanceof CostLimitError || error instanceof TokenLimitError) {
+      io.error(budgetLimitMessage(error));
       return 2;
     }
     throw error;
@@ -778,12 +794,26 @@ function selectCases(cases, ids) {
   return cases.filter((testCase) => wanted.has(testCase.id));
 }
 
-async function resolveModels(ids, modelsFile, io) {
+function currencyOf(id, config) {
+  const code = String(config?.currency ?? "CNY").trim().toUpperCase();
+  if (!/^[A-Z]{3}$/.test(code)) {
+    throw new Error(`model "${id}": currency must be a 3-letter code (got "${config.currency}")`);
+  }
+  return code;
+}
+
+async function resolveModels(ids, modelsFile, io, network = {}) {
   const registry = JSON.parse(await readFile(resolve(modelsFile), "utf8"));
   // Env wins; the credentials file only fills what is missing.
   await applyModelCredentials(registry);
   const explicit = Boolean(ids && ids.length > 0);
   const chosen = explicit ? ids : autoSelect(registry);
+  if (!explicit && chosen.length > 0) {
+    const detail = chosen
+      .map((id) => `${id} (${registry[id]?.apiKeyEnv ?? "?"})`)
+      .join(", ");
+    io.error(`note: auto-selected ${detail} from ${modelsFile}; pass --models or --models-file to pin the registry`);
+  }
   const adapters = [];
 
   for (const id of chosen) {
@@ -805,7 +835,15 @@ async function resolveModels(ids, modelsFile, io) {
       continue;
     }
     registerSecrets([apiKey]);
-    adapters.push({ id, ...createOpenAiAdapter({ id, ...config }) });
+    adapters.push({
+      id,
+      ...createOpenAiAdapter({
+        id,
+        ...config,
+        retries: network.retries,
+        minIntervalMs: network.minIntervalMs
+      })
+    });
   }
 
   if (adapters.length === 0) {
@@ -835,7 +873,7 @@ async function evaluate(
     const runs = [];
     for (let attempt = 0; attempt < repeat; attempt += 1) {
       const result = await callOnce(adapter, tools, testCase);
-      budget?.spend(costCnyOf(result.usage, pricing));
+      budget?.spend({ cost: costOf(result.usage, pricing), tokens: tokensOf(result.usage) });
       runs.push(result);
     }
     if (repeat === 1) {
@@ -859,8 +897,9 @@ function makeProgress(options, io, label, total) {
   };
 }
 
-function costLimitMessage(error) {
-  return `${error.message}；已发出的请求已计费，本次不计分。减少 --case/--models/--repeat 或提高 --max-cost 后重跑。`;
+function budgetLimitMessage(error) {
+  const hint = error instanceof TokenLimitError ? "--max-tokens" : "--max-cost";
+  return `${error.message}；已发出的请求已计费，本次不计分。减少 --case/--models/--repeat 或提高 ${hint} 后重跑。`;
 }
 
 async function callOnce(adapter, tools, testCase) {
@@ -949,7 +988,9 @@ async function evaluateProfiles({
 
   const models = [];
   for (const run of runs) {
-    const pricing = registry[run.adapter.id]?.pricing ?? null;
+    const config = registry[run.adapter.id];
+    const pricing = config?.pricing ?? null;
+    const currency = currencyOf(run.adapter.id, config);
     const entries = [];
     for (const [index, profile] of selected.entries()) {
       const profileCases = caseSets[index];
@@ -965,21 +1006,26 @@ async function evaluateProfiles({
         cases: profileCases,
         results: mountedEval.results,
         pricing,
-        scope: "profiles"
+        scope: "profiles",
+        currency
       });
       const allInOne = scoreModel({
         model: run.adapter.id,
         cases: profileCases,
         results: profileCases.map((testCase) => run.results[indexById.get(testCase.id)]),
         pricing,
-        scope: "profiles"
+        scope: "profiles",
+        currency
       });
       const entry = analyzeMount(profile, mounted, allInOne);
       entry.usage = {
         inputTokens: mounted.inputTokens,
         cachedTokens: mounted.cachedTokens,
         outputTokens: mounted.outputTokens,
-        costCny: mounted.totalCostCny,
+        totalTokens: mounted.totalTokens,
+        cost: mounted.totalCost,
+        costCurrency: mounted.costCurrency,
+        ...(mounted.costCurrency === "CNY" ? { costCny: mounted.totalCostCny } : {}),
         estimated: mounted.rows.some((row) => row.usage?.estimated)
       };
       if (mountedEval.stability) entry.stability = mountedEval.stability;
@@ -1032,6 +1078,9 @@ function parseArgs(argv) {
     profiles: null,
     repeat: 1,
     maxCost: null,
+    maxTokens: null,
+    retries: 2,
+    minIntervalMs: 0,
     progress: false,
     showCases: false,
     json: false,
@@ -1093,10 +1142,28 @@ function parseArgs(argv) {
       }
       case "--max-cost": {
         const raw = Number(value());
-        if (!Number.isFinite(raw) || raw <= 0) throw new Error("--max-cost expects a positive CNY amount");
+        if (!Number.isFinite(raw) || raw <= 0) throw new Error("--max-cost expects a positive amount");
         options.maxCost = raw;
         break;
       }
+      case "--max-tokens": {
+        const raw = Number(value());
+        if (!Number.isInteger(raw) || raw <= 0) throw new Error("--max-tokens expects a positive integer");
+        options.maxTokens = raw;
+        break;
+      }
+      case "--retry": {
+        const raw = Number(value());
+        if (!Number.isInteger(raw) || raw < 0 || raw > 10) throw new Error("--retry expects an integer between 0 and 10");
+        options.retries = raw;
+        break;
+      }
+      case "--min-interval": {
+        const raw = Number(value());
+        if (!Number.isInteger(raw) || raw < 0 || raw > 60_000) throw new Error("--min-interval expects an integer between 0 and 60000 (ms)");
+        options.minIntervalMs = raw;
+        break;
+      }
       case "--progress": options.progress = true; break;
       case "--show-cases": options.showCases = true; break;
       case "--json": options.json = true; break;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelfunction-cli",
-  "version": "0.5.4",
+  "version": "0.6.0",
   "type": "module",
   "license": "MIT",
   "description": "mf — function-calling accuracy scorecard for AI agents",