modelfunction-cli 0.5.3 → 0.5.5

package/README.md

@@ -5,7 +5,7 @@
 ## 安装
 
 ```bash
-npx -y modelfunction-cli@0.5.1 --version
+npx -y modelfunction-cli@0.5.5 --version
 # 或
 npm i -D modelfunction-cli
 npx mf --version
@@ -76,7 +76,7 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
 | 命令 | 作用 |
 | --- | --- |
 | `mf scan` | 扫描工具定义（function.json / OpenAPI / MCP 配置 / Python `@tool` / JS·TS 声明与 TypeBox·zod schema）；`--probe` 连 stdio 或 streamable-HTTP MCP server（运行时结果覆盖静态推断；headers/query 不写入报告）；`--from` 导入运行时工具数组；草稿按来源分组（`profiles`）并列出未覆盖的 skill |
-| `mf eval` | 跑评分卡；`--repeat` 多次取多数并报不稳定用例；`--fail-under` 阈值；`--baseline` + `--fail-on-regression` 回归门禁；`--profile` 按挂载组分别评分；`--max-cost` 按累计成本停机（退出 2）；`--progress` 逐用例进度（stderr） |
+| `mf eval` | 跑评分卡；`--repeat` 多次取多数并报不稳定用例；`--fail-under` 阈值；`--baseline` + `--fail-on-regression` 回归门禁；`--profile` 按挂载组分别评分；`--max-cost` 按累计成本停机（退出 2）；`--retry` 429/5xx 重试次数；`--min-interval` 请求最小间隔（压 RPM）；`--progress` 逐用例进度（stderr） |
 | `mf case add` | 写入用例（默认 `source=agent`、`reviewed=false`） |
 | `mf coverage` | 每工具用例数、参数覆盖、负例比例、疑似混淆对；有 `profiles` 时按组统计 |
 | `mf audit` | 一次静态审计：lint + 混淆族边界 + 用例覆盖 + 挂载检查，输出可转发的 Markdown/JSON；不调用模型 |
@@ -126,19 +126,19 @@ mf eval --suite mf.suite.json --models internal-ds \
   "mcpServers": {
     "modelfunction": {
       "command": "npx",
-      "args": ["-y", "modelfunction-cli@0.5.1", "mcp"]
+      "args": ["-y", "modelfunction-cli@0.5.5", "mcp"]
     }
   }
 }
 ```
 
-暴露：`scan_tools`、`list_suite`、`coverage`、`audit`、`lint`、`check`、`cluster`、`compare`、`add_cases`、`run_eval`（`run_eval` 支持 `profile`、`repeat`、`maxCost`、`progress`）。`lint`/`check`/`cluster`/`compare` 都是静态操作，不调用模型；`check` 就是安装门禁，Agent 可直接传入 manifest 与 installed。
+暴露：`scan_tools`、`list_suite`、`coverage`、`audit`、`lint`、`check`、`cluster`、`compare`、`add_cases`、`run_eval`（`run_eval` 支持 `profile`、`repeat`、`maxCost`、`retry`、`minInterval`、`progress`）。`lint`/`check`/`cluster`/`compare` 都是静态操作，不调用模型；`check` 就是安装门禁，Agent 可直接传入 manifest 与 installed。
 
 ## 契约与版本
 
 对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md)，自 0.4.3 起冻结为 v1：v1 内只做加法，重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
 
-`0.5.2` 已发布到 npm：含凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查，门禁 warn 只保留混淆对/必填来源/结构性错误。本仓库 `0.5.3` 待发布：静态规则性能（真实 100 能力清单 `checkMany` 471ms→40ms，550 能力 9.6s→2.1s，输出与 golden 一致），以及 `installed` 形状校验（接受 `mf scan --json` 输出，无法识别时报错而不是静默返回假 pass）和 `profiles` 兼容平台形态（对象或 `[{agentType, tools, skills}]` 数组）。示例统一使用 `@0.5.2`。
+`0.5.5`（待 npm 发布）：eval 新增 `--retry <n>`（429/5xx/网络错误指数退避，默认 2，共 3 次尝试；优先采用服务端 `Retry-After`）与 `--min-interval <ms>`（请求最小间隔，用于压住端点 RPM），评分卡新增 `retries` 计数；CI 模板 static 层补上 `cluster --json`，与 `action.yml` 对齐；新增 `templates/local/` 与 `docs/local-gate.md`（内网/非 GitHub 环境的 pre-push 静态门禁、变更门、定时回归）。`0.5.4` 含用例级 `excludeFromBase` / `excludeFromProfiles`、评分卡 `errorKind` / `errorsByKind` / `scoredTotal` / `selectionAccuracyExcludingErrors`、挂载对比错误不计 jammed、audit 分组 `neg%` 与分组 token 用量聚合。`0.5.3` 及更早的能力见下：凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查、门禁严重度收敛、静态规则性能（真实 100 能力清单 `checkMany` 471ms→40ms）、`installed` 形状校验与 `profiles` 平台形态兼容。示例统一使用 `@0.5.5`。
 
 ## 数据边界
 
@@ -167,7 +167,7 @@ mf feedback --out out.json  # 导出脱敏反馈包（聚合值，不含原始
 
 ```bash
 # 1. 把 mf 挂成 MCP server
-claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.1 mcp
+claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.5 mcp
 ```
 
 ```text

package/SCHEMA.md

@@ -51,6 +51,8 @@
   "note": "",
   "source": "manual",
   "reviewed": true,
+  "excludeFromBase": false,
+  "excludeFromProfiles": false,
   "tags": []
 }
 ```
@@ -59,6 +61,7 @@
 - `expectedArgs` 仅在有 `expectedTool` 时可用；值为断言，支持字面量、`$contains`、`$oneOf`、`$exists`。
 - `toolInput` + `expect` 用于 `--execute`：`expect` 支持 `ok`、`contains`、`equals`、`errorType`、`maxDurationMs`、`maxPeakRssMB`。
 - `reviewed` 缺省为 `true`（手写用例）；`mf case add` 缺省写入 `source=agent`、`reviewed=false`。
+- `excludeFromBase` / `excludeFromProfiles` 必须是布尔值。用例仍会实际调用（不省 token），只是不计入对应范围的正式指标；用于「该用例只对某一挂载形态有意义」的情况。
 
 ### skills[]
 
@@ -89,21 +92,26 @@ skills 只做静态检查，不参与模型评测。
     "fingerprint": "sha256:...",
     "execution": { "runtime": "node", "engine": "node-permission", "entry": "index.mjs", "hash": "..." }
   },
-  "models": [ { "model": "dsv4-flash", "total": 12, "reviewedTotal": 12, "selectionAccuracy": 0.83,
-               "argAccuracy": 0.9, "passRate": 0.75, "errors": 0, "p50LatencyMs": 900,
+  "models": [ { "model": "dsv4-flash", "total": 12, "reviewedTotal": 12, "excludedTotal": 0,
+               "selectionAccuracy": 0.83, "selectionAccuracyExcludingErrors": 0.83,
+               "argAccuracy": 0.9, "passRate": 0.75, "errors": 0, "errorsByKind": {},
+               "scoredTotal": 12, "retries": 0, "p50LatencyMs": 900,
                "p95LatencyMs": 1800, "avgCostCny": 0.001, "totalCostCny": 0.012,
                "inputTokens": 1000, "cachedTokens": 0, "outputTokens": 200,
                "rows": [ { "id": "c1", "expectedTool": "...", "actualTool": "...",
                            "selectionCorrect": true, "args": { "total": 1, "matched": 1, "failures": [] },
                            "passed": true, "reviewed": true, "source": "manual",
-                           "latencyMs": 900, "usage": { "input": 80, "output": 20 }, "error": null } ] } ],
+                           "latencyMs": 900, "usage": { "input": 80, "output": 20 }, "error": null,
+                           "retries": 1 } ] } ],
   "profiles": { "groups": [ ... ] }
 }
 ```
 
-- 正式指标只统计 `reviewed` 行；未复核结果单独给出。
+- 正式指标只统计 `reviewed` 且未被 `excludeFromBase` 排除的行；未复核与排除结果单独给出。
+- `errors` 是调用层失败（超时、HTTP、网络）的数量，按 `errorsByKind` 分类；它们仍计入 `selectionAccuracy`（保守口径），同时给出排除错误后的 `selectionAccuracyExcludingErrors` 与可评分行数 `scoredTotal`。
+- `retries` 是本轮因 429/5xx/网络错误而重试的请求次数（模型级汇总；行级只在大于 0 时出现）。重试不改变判分：最终成功的请求按正常结果计分，最终失败的仍计入 `errors`。
 - `--repeat` 时模型对象带 `stability`（翻转用例与多次结果分布）。
-- `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`。
+- `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`、`errors`；错误行不参与挂载对比（不会被记成 jammed），每个组的 `reviewed` 是可评分行数。
 - `fingerprint` 由工具与用例内容计算；`mf compare` 用它判断两份报告是否可比。
 
 ## 3. 扫描输出（`mf scan`）
@@ -338,4 +346,4 @@ const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
 }
 ```
 
-`profiles` 为 `null` 表示套件未定义分组。`clusters`/`coverage` 的明细与 §5/§4 同源；`actions` 是给人看的优先级列表。`mf audit` 恒退出 0（advisory），不阻断流程。
+`profiles` 为 `null` 表示套件未定义分组。`clusters`/`coverage` 的明细与 §5/§4 同源；`actions` 是给人看的优先级列表。`mf audit` 恒退出 0（advisory），不阻断流程；负例占比告警只按整 suite 计算，分组表里的 `negativeRatio` 是该组「正例 + 负例」内的比例，仅供参考。

package/lib/adapters/openai.js

@@ -1,5 +1,6 @@
 const TIMEOUT_MS = 60_000;
-const MAX_ATTEMPTS = 3;
+const DEFAULT_RETRIES = 2;
+const MAX_RETRY_DELAY_MS = 30_000;
 
 export function createOpenAiAdapter(config) {
   const apiKey = process.env[config.apiKeyEnv];
@@ -7,13 +8,25 @@ export function createOpenAiAdapter(config) {
     throw new Error(`${config.id}: environment variable ${config.apiKeyEnv} is not set`);
   }
   const endpoint = `${config.baseUrl.replace(/\/+$/, "")}/chat/completions`;
+  const retries = clampInt(config.retries, 0, 10, DEFAULT_RETRIES);
+  const minIntervalMs = clampInt(config.minIntervalMs, 0, 60_000, 0);
+  let lastRequestAt = 0;
+
+  async function throttle() {
+    if (minIntervalMs <= 0) return;
+    const wait = lastRequestAt + minIntervalMs - Date.now();
+    if (wait > 0) await sleep(wait);
+    lastRequestAt = Date.now();
+  }
 
   return {
     label: config.model,
     async call({ messages, tools }) {
       const started = performance.now();
       let lastError = "request failed";
-      for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt += 1) {
+      for (let attempt = 1; attempt <= retries + 1; attempt += 1) {
+        const retried = attempt - 1;
+        await throttle();
         try {
           const response = await fetch(endpoint, {
             method: "POST",
@@ -33,13 +46,16 @@ export function createOpenAiAdapter(config) {
 
           if (response.status === 429 || response.status >= 500) {
             lastError = `HTTP ${response.status}: ${(await response.text()).slice(0, 300)}`;
-            await sleep(attempt * 1000);
+            if (attempt <= retries) {
+              await sleep(retryDelayMs(response.headers.get("retry-after"), retried));
+            }
             continue;
           }
           if (!response.ok) {
             const body = await response.text();
             return {
               ok: false,
+              retries: retried,
               error: `HTTP ${response.status}: ${body.slice(0, 300)}`,
               toolCalls: [],
               usage: { input: 0, output: 0 },
@@ -55,6 +71,7 @@ export function createOpenAiAdapter(config) {
           }));
           return {
             ok: true,
+            retries: retried,
             toolCalls,
             usage: {
               input: data.usage?.prompt_tokens ?? 0,
@@ -66,11 +83,12 @@ export function createOpenAiAdapter(config) {
           };
         } catch (error) {
           lastError = error?.name === "TimeoutError" ? `timeout after ${TIMEOUT_MS}ms` : String(error?.message ?? error);
-          if (attempt < MAX_ATTEMPTS) await sleep(attempt * 1000);
+          if (attempt <= retries) await sleep(retryDelayMs(null, retried));
         }
       }
       return {
         ok: false,
+        retries,
         error: lastError,
         toolCalls: [],
         usage: { input: 0, output: 0 },
@@ -80,6 +98,22 @@ export function createOpenAiAdapter(config) {
   };
 }
 
+function retryDelayMs(retryAfter, retried) {
+  const header = retryAfter == null || String(retryAfter).trim() === "" ? NaN : Number(retryAfter);
+  if (Number.isFinite(header) && header >= 0) {
+    return Math.min(header * 1000, MAX_RETRY_DELAY_MS);
+  }
+  return Math.min(1000 * 2 ** retried, MAX_RETRY_DELAY_MS);
+}
+
+function clampInt(value, min, max, fallback) {
+  const parsed = Number(value);
+  if (!Number.isFinite(parsed)) return fallback;
+  const rounded = Math.floor(parsed);
+  if (rounded < min || rounded > max) return fallback;
+  return rounded;
+}
+
 function parseArguments(raw) {
   if (raw === undefined || raw === null || raw === "") return {};
   if (typeof raw === "object") return raw;

package/lib/audit.js

@@ -134,11 +134,11 @@ export function formatAudit(report) {
   if (!profiles) {
     lines.push("未定义 profiles（平台可先用 `mf cluster --out <suite>` 生成候选分组）。");
   } else {
-    lines.push("group                   tools  cases  negatives  tools-without-cases");
-    lines.push("----------------------  -----  -----  ---------  -------------------");
+    lines.push("group                   tools  cases  negatives  neg%  tools-without-cases");
+    lines.push("----------------------  -----  -----  ---------  ----  -------------------");
     for (const row of profiles.groups) {
       lines.push(
-        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(9)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
+        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(9)}  ${`${Math.round((row.negativeRatio ?? 0) * 100)}%`.padStart(4)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
       );
     }
     if (profiles.unmounted.length > 0) lines.push("", `未挂载工具：${profiles.unmounted.join(", ")}`);

package/lib/cases.js

@@ -185,12 +185,12 @@ export function formatCoverage(report) {
     );
   }
   if (report.profiles && report.profiles.length > 0) {
-    lines.push("", "profiles（分组后可评分的用例；负例每组都跑）:");
-    lines.push("group                   tools  cases  neg  tools-without-cases");
-    lines.push("----------------------  -----  -----  ---  -------------------");
+    lines.push("", "profiles（分组后可评分的用例；负例每组都跑，比例按组内正例+负例计）:");
+    lines.push("group                   tools  cases  neg  neg%  tools-without-cases");
+    lines.push("----------------------  -----  -----  ---  ----  -------------------");
     for (const row of report.profiles) {
       lines.push(
-        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(3)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
+        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(3)}  ${`${Math.round((row.negativeRatio ?? 0) * 100)}%`.padStart(4)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
       );
     }
   }

package/lib/exec.js

@@ -1,4 +1,5 @@
 import { spawn } from "node:child_process";
+import { realpathSync } from "node:fs";
 import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { dirname, join, resolve } from "node:path";
@@ -13,18 +14,19 @@ const ENV_KEEP = ["PATH", "SystemRoot", "SYSTEMROOT", "TEMP", "TMP", "HOME", "US
 export async function executeTool(options) {
   const runtime = options.runtime ?? "node";
   const engine = runtime === "python" ? "python-audit" : "node-permission";
-  const entry = resolve(options.entry);
+  const entry = realpathAllow(options.entry);
   const nodeMajor = Number(process.versions.node.split(".")[0]);
   if (runtime === "node" && options.allowNet?.length && nodeMajor < 25) {
     throw new Error("--allow-net 需要 Node >= 25（Node 24 的权限模型不限制网络）");
   }
 
   let tempDir = null;
-  let inputPath = options.inputPath ? resolve(options.inputPath) : null;
+  let inputPath = options.inputPath ? realpathAllow(options.inputPath) : null;
   if (!inputPath) {
     tempDir = await mkdtemp(join(tmpdir(), "mf-exec-"));
     inputPath = join(tempDir, "input.json");
     await writeFile(inputPath, options.inputJson ?? "{}", "utf8");
+    inputPath = realpathAllow(inputPath);
   }
 
   try {
@@ -57,11 +59,11 @@ export async function executeTool(options) {
         dirname(inputPath),
         process.cwd(),
         ...(options.allowRead ?? [])
-      ].map((path) => resolve(path));
+      ].map((path) => realpathAllow(path));
       args = [permissionFlag()];
       for (const path of dedupe(readPaths)) args.push(`--allow-fs-read=${path}`);
       if (options.allowWrite?.length) {
-        for (const path of dedupe(options.allowWrite.map((item) => resolve(item)))) {
+        for (const path of dedupe(options.allowWrite.map((item) => realpathAllow(item)))) {
           args.push(`--allow-fs-write=${path}`);
         }
       }
@@ -264,3 +266,14 @@ function formatExec(entry, result) {
   if (result.permissionDenied) lines.push("  note:       权限模型拒绝了文件/网络访问，可用于推断沙箱需求");
   return lines.join("\n");
 }
+
+// Node's permission model matches real paths. On macOS /var is a symlink to
+// /private/var, so an unresolved tmpdir path in --allow-fs-read is rejected.
+function realpathAllow(path) {
+  const abs = resolve(path);
+  try {
+    return realpathSync.native ? realpathSync.native(abs) : realpathSync(abs);
+  } catch {
+    return abs;
+  }
+}

package/lib/mcp-server.js

@@ -248,10 +248,12 @@ export async function startMcpServer() {
           .describe("每条用例重复次数，取多数结果；建议正式基线用 3"),
         failUnder: z.number().optional().describe("选择准确率阈值，低于则退出码为 2"),
         maxCost: z.number().positive().optional().describe("累计估算成本上限（CNY），超限退出 2 且不返回部分评分卡"),
+        retry: z.number().int().min(0).max(10).optional().describe("单次请求遇到 429/5xx/网络错误的重试次数（默认 2，共 3 次尝试）"),
+        minInterval: z.number().int().min(0).max(60000).optional().describe("两次模型请求的最小间隔（毫秒），用于压住端点 RPM"),
         progress: z.boolean().optional().describe("把逐用例进度写到 stderr（stdout 仍为纯 JSON）")
       }
     },
-    async ({ suite, models, cases, profile, repeat, failUnder, maxCost, progress }) => {
+    async ({ suite, models, cases, profile, repeat, failUnder, maxCost, retry, minInterval, progress }) => {
       const args = ["eval", "--json", "--suite", suitePath(suite)];
       if (models) args.push("--models", models);
       if (cases) args.push("--case", cases);
@@ -259,6 +261,8 @@ export async function startMcpServer() {
       if (repeat !== undefined) args.push("--repeat", String(repeat));
       if (failUnder !== undefined) args.push("--fail-under", String(failUnder));
       if (maxCost !== undefined) args.push("--max-cost", String(maxCost));
+      if (retry !== undefined) args.push("--retry", String(retry));
+      if (minInterval !== undefined) args.push("--min-interval", String(minInterval));
       if (progress) args.push("--progress");
 
       const result = await runCli(args);

package/lib/profile.js

@@ -54,7 +54,11 @@ export function selectProfiles(profiles, wanted) {
 // A case belongs to a group when its expected tool is mounted there. Negative
 // cases ("no tool applies") hold everywhere: the group's agent still gets them.
 export function selectProfileCases(cases, mountedTools) {
-  return cases.filter((testCase) => testCase.expectedTool === null || mountedTools.has(testCase.expectedTool));
+  return cases.filter(
+    (testCase) =>
+      testCase.excludeFromProfiles !== true &&
+      (testCase.expectedTool === null || mountedTools.has(testCase.expectedTool))
+  );
 }
 
 export function profileWarnings(profiles, suite) {
@@ -76,11 +80,12 @@ export function profileWarnings(profiles, suite) {
 }
 
 export function profileCaseCoverage(profiles, cases) {
-  const negatives = cases.filter((testCase) => testCase.expectedTool === null).length;
+  const considered = cases.filter((testCase) => testCase.excludeFromProfiles !== true);
+  const negatives = considered.filter((testCase) => testCase.expectedTool === null).length;
   return profiles.map((profile) => {
     const mounted = new Set(profile.tools);
     const counts = new Map(profile.tools.map((name) => [name, 0]));
-    for (const testCase of cases) {
+    for (const testCase of considered) {
       if (testCase.expectedTool !== null && mounted.has(testCase.expectedTool)) {
         counts.set(testCase.expectedTool, counts.get(testCase.expectedTool) + 1);
       }
@@ -88,7 +93,14 @@ export function profileCaseCoverage(profiles, cases) {
     const missing = [...counts.entries()].filter(([, count]) => count === 0).map(([name]) => name);
     let total = 0;
     for (const count of counts.values()) total += count;
-    return { name: profile.name, tools: profile.tools.length, cases: total, negatives, missing };
+    return {
+      name: profile.name,
+      tools: profile.tools.length,
+      cases: total,
+      negatives,
+      negativeRatio: total + negatives > 0 ? negatives / (total + negatives) : 0,
+      missing
+    };
   });
 }
 
@@ -99,18 +111,21 @@ export function analyzeMount(profile, mounted, allInOne) {
   for (const before of allInOne.rows) {
     const row = after.get(before.id);
     if (!row || before.reviewed === false || row.reviewed === false) continue;
+    // A provider error is not evidence about mounting; score it separately.
+    if (before.error || row.error) continue;
     if (!before.selectionCorrect && row.selectionCorrect) lifted.push(changeOf(before, row));
     else if (before.selectionCorrect && !row.selectionCorrect) jammed.push(changeOf(before, row));
   }
 
   const mountedMetrics = metricsOf(mounted);
   const allMetrics = metricsOf(allInOne);
-  const reviewed = mounted.reviewedTotal;
+  const reviewed = mountedMetrics.scored;
   return {
     profile: profile.name,
     mountedTools: profile.tools.length,
     cases: mounted.total,
     reviewed,
+    errors: mountedMetrics.errors,
     mounted: mountedMetrics,
     allInOne: allMetrics,
     delta: {
@@ -128,32 +143,44 @@ export function analyzeMount(profile, mounted, allInOne) {
 
 export function modelTotals(entries) {
   let reviewed = 0;
+  let errors = 0;
   let mountedCorrect = 0;
   let allCorrect = 0;
+  const usage = { inputTokens: 0, cachedTokens: 0, outputTokens: 0, costCny: 0, estimated: false };
   for (const entry of entries) {
     reviewed += entry.reviewed;
+    errors += entry.errors ?? 0;
     mountedCorrect += entry.mounted.correct;
     allCorrect += entry.allInOne.correct;
+    usage.inputTokens += entry.usage?.inputTokens ?? 0;
+    usage.cachedTokens += entry.usage?.cachedTokens ?? 0;
+    usage.outputTokens += entry.usage?.outputTokens ?? 0;
+    usage.costCny += entry.usage?.costCny ?? 0;
+    usage.estimated ||= Boolean(entry.usage?.estimated);
   }
   if (reviewed === 0) {
     return {
       reviewed: 0,
+      errors,
       mountedCorrect: 0,
       allCorrect: 0,
       mountedAccuracy: null,
       allAccuracy: null,
-      delta: null
+      delta: null,
+      usage
     };
   }
   const mountedAccuracy = mountedCorrect / reviewed;
   const allAccuracy = allCorrect / reviewed;
   return {
     reviewed,
+    errors,
     mountedCorrect,
     allCorrect,
     mountedAccuracy,
     allAccuracy,
-    delta: mountedAccuracy - allAccuracy
+    delta: mountedAccuracy - allAccuracy,
+    usage
   };
 }
 
@@ -169,15 +196,16 @@ export function formatProfileReport(report, { showCases = false } = {}) {
       entry.profile,
       String(entry.mountedTools),
       String(entry.cases),
+      String(entry.errors ?? 0),
       pct(entry.mounted.selectionAccuracy),
       pct(entry.allInOne.selectionAccuracy),
       signedPoints(entry.delta.selection)
     ]);
     lines.push(
       ...renderTable(
-        ["profile", "tools", "cases", "mounted", "all-in-one", "delta"],
+        ["profile", "tools", "cases", "errors", "mounted", "all-in-one", "delta"],
         rows,
-        [1, 2, 3, 4, 5]
+        [1, 2, 3, 4, 5, 6]
       )
     );
     const totals = model.totals;
@@ -185,7 +213,8 @@ export function formatProfileReport(report, { showCases = false } = {}) {
       lines.push(
         `  合计: 分组 ${pct(totals.mountedAccuracy)}（${totals.mountedCorrect}/${totals.reviewed}）` +
           ` / 全量 ${pct(totals.allAccuracy)}（${totals.allCorrect}/${totals.reviewed}）` +
-          `，${signedPoints(totals.delta)}`
+          `，${signedPoints(totals.delta)}` +
+          (totals.errors > 0 ? `；${totals.errors} 次调用错误不计入对比` : "")
       );
     }
     const unstable = unstableOf(model.profiles);
@@ -224,12 +253,12 @@ export function formatProfileMarkdown(report) {
     lines.push(
       `### ${model.model}`,
       "",
-      "| group | tools | cases | mounted | all-in-one | delta |",
-      "| --- | --- | --- | --- | --- | --- |"
+      "| group | tools | cases | errors | mounted | all-in-one | delta |",
+      "| --- | --- | --- | --- | --- | --- | --- |"
     );
     for (const entry of model.profiles) {
       lines.push(
-        `| ${entry.profile} | ${entry.mountedTools} | ${entry.cases} | ${pct(entry.mounted.selectionAccuracy)} | ${pct(entry.allInOne.selectionAccuracy)} | ${signedPoints(entry.delta.selection)} |`
+        `| ${entry.profile} | ${entry.mountedTools} | ${entry.cases} | ${entry.errors ?? 0} | ${pct(entry.mounted.selectionAccuracy)} | ${pct(entry.allInOne.selectionAccuracy)} | ${signedPoints(entry.delta.selection)} |`
       );
     }
     const totals = model.totals;
@@ -237,7 +266,8 @@ export function formatProfileMarkdown(report) {
     if (totals.reviewed > 0) {
       lines.push(
         `- 合计：分组 ${pct(totals.mountedAccuracy)}（${totals.mountedCorrect}/${totals.reviewed}），` +
-          `全量 ${pct(totals.allAccuracy)}（${totals.allCorrect}/${totals.reviewed}），${signedPoints(totals.delta)}`
+          `全量 ${pct(totals.allAccuracy)}（${totals.allCorrect}/${totals.reviewed}），${signedPoints(totals.delta)}` +
+          (totals.errors > 0 ? `；${totals.errors} 次调用错误不计入对比` : "")
       );
     } else {
       lines.push("- 合计：没有已复核用例，无法比较");
@@ -267,14 +297,18 @@ function changeOf(before, after) {
 
 function metricsOf(summary) {
   const reviewedRows = summary.rows.filter((row) => row.reviewed !== false);
-  const correct = reviewedRows.filter((row) => row.selectionCorrect).length;
+  const errorRows = reviewedRows.filter((row) => row.error);
+  const viableRows = reviewedRows.filter((row) => !row.error);
+  const correct = viableRows.filter((row) => row.selectionCorrect).length;
   return {
-    selectionAccuracy: reviewedRows.length ? correct / reviewedRows.length : null,
+    selectionAccuracy: viableRows.length ? correct / viableRows.length : null,
     argAccuracy: summary.argAccuracy,
-    passRate: reviewedRows.length
-      ? reviewedRows.filter((row) => row.passed).length / reviewedRows.length
+    passRate: viableRows.length
+      ? viableRows.filter((row) => row.passed).length / viableRows.length
       : null,
-    correct
+    correct,
+    errors: errorRows.length,
+    scored: viableRows.length
   };
 }
 

package/lib/score.js

@@ -89,7 +89,7 @@ export function costCnyOf(usage, pricing) {
   );
 }
 
-export function scoreModel({ model, cases, results, pricing }) {
+export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
   const rows = cases.map((testCase, index) => {
     const result = results[index];
     const actualTool = result.ok ? (result.toolCalls?.[0]?.name ?? null) : null;
@@ -98,6 +98,7 @@ export function scoreModel({ model, cases, results, pricing }) {
       ? matchArgs(result.toolCalls?.[0]?.arguments, testCase.expectedArgs)
       : { total: 0, matched: 0, failures: [] };
     const passed = Boolean(result.ok) && selectionCorrect && args.failures.length === 0;
+    const error = result.error ?? null;
     return {
       id: testCase.id,
       expectedTool: testCase.expectedTool,
@@ -106,21 +107,36 @@ export function scoreModel({ model, cases, results, pricing }) {
       args,
       passed,
       reviewed: testCase.reviewed !== false,
+      ...(testCase.excludeFromBase ? { excludeFromBase: true } : {}),
+      ...(testCase.excludeFromProfiles ? { excludeFromProfiles: true } : {}),
       source: testCase.source ?? "manual",
       latencyMs: result.latencyMs ?? 0,
       usage: result.usage ?? { input: 0, output: 0 },
-      error: result.error ?? null
+      ...(result.retries > 0 ? { retries: result.retries } : {}),
+      error,
+      ...(error ? { errorKind: classifyError(error) } : {})
     };
   });
 
   const total = rows.length;
-  const reviewedRows = rows.filter((row) => row.reviewed);
+  // An excluded case still runs (so profile / all-in-one views keep their rows);
+  // it just does not contribute to this scope's official metrics.
+  const excluded = (row) => (scope === "profiles" ? row.excludeFromProfiles : row.excludeFromBase);
+  const reviewedRows = rows.filter((row) => row.reviewed && !excluded(row));
   const unreviewedRows = rows.filter((row) => !row.reviewed);
+  const excludedRows = rows.filter((row) => row.reviewed && excluded(row));
   const selectionHits = reviewedRows.filter((row) => row.selectionCorrect).length;
   const argFields = reviewedRows.reduce((sum, row) => sum + row.args.total, 0);
   const argHits = reviewedRows.reduce((sum, row) => sum + row.args.matched, 0);
   const passed = reviewedRows.filter((row) => row.passed).length;
-  const errors = rows.filter((row) => row.error).length;
+  const errorRows = reviewedRows.filter((row) => row.error);
+  const errors = errorRows.length;
+  const errorsByKind = errorRows.reduce((counts, row) => {
+    counts[row.errorKind] = (counts[row.errorKind] ?? 0) + 1;
+    return counts;
+  }, {});
+  const scoredTotal = reviewedRows.length - errors;
+  const selectionHitsWithoutErrors = reviewedRows.filter((row) => !row.error && row.selectionCorrect).length;
   const latencies = rows.filter((row) => row.latencyMs > 0).map((row) => row.latencyMs);
   const inputTokens = rows.reduce((sum, row) => sum + (row.usage.input ?? 0), 0);
   const outputTokens = rows.reduce((sum, row) => sum + (row.usage.output ?? 0), 0);
@@ -135,14 +151,19 @@ export function scoreModel({ model, cases, results, pricing }) {
     model,
     total,
     reviewedTotal,
+    excludedTotal: excludedRows.length,
     unreviewedTotal: unreviewedRows.length,
     unreviewedSelectionAccuracy: unreviewedRows.length
       ? unreviewedRows.filter((row) => row.selectionCorrect).length / unreviewedRows.length
       : null,
     selectionAccuracy: reviewedTotal ? selectionHits / reviewedTotal : 0,
+    selectionAccuracyExcludingErrors: scoredTotal ? selectionHitsWithoutErrors / scoredTotal : null,
     argAccuracy: argFields ? argHits / argFields : null,
     passRate: reviewedTotal ? passed / reviewedTotal : 0,
     errors,
+    errorsByKind,
+    scoredTotal,
+    retries: rows.reduce((sum, row) => sum + (row.retries ?? 0), 0),
     p50LatencyMs: percentile(latencies, 0.5),
     p95LatencyMs: percentile(latencies, 0.95),
     avgCostCny: total ? costCny / total : 0,
@@ -154,6 +175,15 @@ export function scoreModel({ model, cases, results, pricing }) {
   };
 }
 
+export function classifyError(error) {
+  const text = String(error ?? "");
+  if (/timeout|timed out/i.test(text)) return "timeout";
+  if (/^HTTP 429\b/.test(text)) return "rate-limit";
+  if (/^HTTP \d{3}\b/.test(text)) return "http";
+  if (/ECONN|ENOTFOUND|EAI_AGAIN|fetch failed|network/i.test(text)) return "network";
+  return "other";
+}
+
 function percentile(values, ratio) {
   if (values.length === 0) return 0;
   const sorted = [...values].sort((a, b) => a - b);

package/lib/suite.js

@@ -95,6 +95,11 @@ function normalizeCase(entry, index, toolNames) {
   if (toolInput && expectedTool === null) {
     throw new Error(`case ${id}: toolInput requires an expectedTool`);
   }
+  for (const key of ["excludeFromBase", "excludeFromProfiles"]) {
+    if (entry[key] !== undefined && typeof entry[key] !== "boolean") {
+      throw new Error(`case ${id}: "${key}" must be a boolean`);
+    }
+  }
   return {
     id,
     input: entry.input,
@@ -103,6 +108,8 @@ function normalizeCase(entry, index, toolNames) {
     note: entry.note ?? "",
     source: typeof entry.source === "string" && entry.source ? entry.source : "manual",
     reviewed: entry.reviewed === undefined ? true : Boolean(entry.reviewed),
+    ...(entry.excludeFromBase ? { excludeFromBase: true } : {}),
+    ...(entry.excludeFromProfiles ? { excludeFromProfiles: true } : {}),
     tags: Array.isArray(entry.tags) ? entry.tags.map(String) : [],
     toolInput,
     expect:

package/mf.js

@@ -97,6 +97,8 @@ Eval options:
   --profile <names>     score each mounted group separately ("all" or names from suite.profiles)
   --repeat <n>          run each case n times and score the majority; reports unstable cases (default 1)
   --max-cost <cny>      stop after the estimated spend crosses this limit (exit 2; no partial scorecard)
+  --retry <n>           retries per request on HTTP 429/5xx and network errors (default 2; 3 attempts total)
+  --min-interval <ms>   minimum spacing between model requests; use it to stay under an endpoint RPM
   --progress            print per-case progress to stderr (stdout stays clean for --json)
   --show-cases          print per-case results
   --json                print JSON instead of the table
@@ -136,7 +138,10 @@ export async function runEval(options, io = console) {
   }
   const suite = await loadSuite(options.suite);
   const cases = selectCases(suite.cases, options.cases);
-  const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io);
+  const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io, {
+    retries: options.retries,
+    minIntervalMs: options.minIntervalMs
+  });
   const tools = toModelTools(suite.tools);
 
   const budget = createBudget(options.maxCost ?? null);
@@ -153,7 +158,8 @@ export async function runEval(options, io = console) {
         model: adapter.id,
         cases,
         results: evaluated.results,
-        pricing
+        pricing,
+        scope: "base"
       });
       if (evaluated.stability) summary.stability = evaluated.stability;
       runs.push({ adapter, results: evaluated.results, summary });
@@ -250,6 +256,15 @@ export async function runEval(options, io = console) {
       if (summary.unreviewedTotal > 0) {
         io.log(`note: ${summary.unreviewedTotal} unreviewed case(s) excluded from select/args/pass`);
       }
+      if (summary.excludedTotal > 0) {
+        io.log(`note: ${summary.excludedTotal} case(s) marked excludeFromBase, excluded from base score`);
+      }
+      if (summary.errors > 0) {
+        const kinds = Object.entries(summary.errorsByKind)
+          .map(([kind, count]) => `${kind} ${count}`)
+          .join(", ");
+        io.log(`note: ${summary.errors} call error(s) counted as misses (${kinds})`);
+      }
       if (summary.exec) {
         const parts = [
           `exec: ${summary.exec.passed}/${summary.exec.executed} passed`,
@@ -383,7 +398,10 @@ export async function runAblate(options, io = console) {
     : selectedCases;
   const reducedCases = baseCases.filter((testCase) => testCase.expectedTool !== ablated.name);
   const droppedCases = baseCases.length - reducedCases.length;
-  const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io);
+  const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io, {
+    retries: options.retries,
+    minIntervalMs: options.minIntervalMs
+  });
 
   const fullTools = toModelTools(baseTools);
   const reducedTools = toModelTools(baseTools.filter((tool) => tool.name !== ablated.name));
@@ -768,7 +786,7 @@ function selectCases(cases, ids) {
   return cases.filter((testCase) => wanted.has(testCase.id));
 }
 
-async function resolveModels(ids, modelsFile, io) {
+async function resolveModels(ids, modelsFile, io, network = {}) {
   const registry = JSON.parse(await readFile(resolve(modelsFile), "utf8"));
   // Env wins; the credentials file only fills what is missing.
   await applyModelCredentials(registry);
@@ -795,7 +813,15 @@ async function resolveModels(ids, modelsFile, io) {
       continue;
     }
     registerSecrets([apiKey]);
-    adapters.push({ id, ...createOpenAiAdapter({ id, ...config }) });
+    adapters.push({
+      id,
+      ...createOpenAiAdapter({
+        id,
+        ...config,
+        retries: network.retries,
+        minIntervalMs: network.minIntervalMs
+      })
+    });
   }
 
   if (adapters.length === 0) {
@@ -915,6 +941,14 @@ async function evaluateProfiles({
   const selected = selectProfiles(profiles, selection);
   const indexById = new Map(cases.map((testCase, index) => [testCase.id, index]));
   const warnings = profileWarnings(selected, suite);
+  const excludedFromProfiles = cases.filter((testCase) => testCase.excludeFromProfiles === true);
+  if (excludedFromProfiles.length > 0) {
+    warnings.push(
+      `${excludedFromProfiles.length} 条用例标记 excludeFromProfiles，不参与分组评分: ${excludedFromProfiles
+        .map((testCase) => testCase.id)
+        .join(", ")}`
+    );
+  }
 
   const groups = [];
   const caseSets = selected.map((profile) => {
@@ -946,15 +980,24 @@ async function evaluateProfiles({
         model: run.adapter.id,
         cases: profileCases,
         results: mountedEval.results,
-        pricing
+        pricing,
+        scope: "profiles"
       });
       const allInOne = scoreModel({
         model: run.adapter.id,
         cases: profileCases,
         results: profileCases.map((testCase) => run.results[indexById.get(testCase.id)]),
-        pricing
+        pricing,
+        scope: "profiles"
       });
       const entry = analyzeMount(profile, mounted, allInOne);
+      entry.usage = {
+        inputTokens: mounted.inputTokens,
+        cachedTokens: mounted.cachedTokens,
+        outputTokens: mounted.outputTokens,
+        costCny: mounted.totalCostCny,
+        estimated: mounted.rows.some((row) => row.usage?.estimated)
+      };
       if (mountedEval.stability) entry.stability = mountedEval.stability;
       entries.push(entry);
     }
@@ -1005,6 +1048,8 @@ function parseArgs(argv) {
     profiles: null,
     repeat: 1,
     maxCost: null,
+    retries: 2,
+    minIntervalMs: 0,
     progress: false,
     showCases: false,
     json: false,
@@ -1070,6 +1115,18 @@ function parseArgs(argv) {
         options.maxCost = raw;
         break;
       }
+      case "--retry": {
+        const raw = Number(value());
+        if (!Number.isInteger(raw) || raw < 0 || raw > 10) throw new Error("--retry expects an integer between 0 and 10");
+        options.retries = raw;
+        break;
+      }
+      case "--min-interval": {
+        const raw = Number(value());
+        if (!Number.isInteger(raw) || raw < 0 || raw > 60_000) throw new Error("--min-interval expects an integer between 0 and 60000 (ms)");
+        options.minIntervalMs = raw;
+        break;
+      }
       case "--progress": options.progress = true; break;
       case "--show-cases": options.showCases = true; break;
       case "--json": options.json = true; break;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelfunction-cli",
-  "version": "0.5.3",
+  "version": "0.5.5",
   "type": "module",
   "license": "MIT",
   "description": "mf — function-calling accuracy scorecard for AI agents",