modelfunction-cli 0.5.2 → 0.5.4

package/README.md

@@ -5,7 +5,7 @@
 ## 安装
 
 ```bash
-npx -y modelfunction-cli@0.5.1 --version
+npx -y modelfunction-cli@0.5.3 --version
 # 或
 npm i -D modelfunction-cli
 npx mf --version
@@ -126,7 +126,7 @@ mf eval --suite mf.suite.json --models internal-ds \
   "mcpServers": {
     "modelfunction": {
       "command": "npx",
-      "args": ["-y", "modelfunction-cli@0.5.1", "mcp"]
+      "args": ["-y", "modelfunction-cli@0.5.3", "mcp"]
     }
   }
 }
@@ -138,7 +138,9 @@ mf eval --suite mf.suite.json --models internal-ds \
 
 对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md)，自 0.4.3 起冻结为 v1：v1 内只做加法，重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
 
-`0.5.1` 已发布到 npm：含凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP。本仓库 `0.5.2` 待发布：MCP 补齐 `lint`/`check`/`cluster`/`compare`，安装门禁支持 `--manifests` 一包批量检查，门禁 warn 只保留混淆对/必填来源/结构性错误（描述类降 info）。示例统一使用 `@0.5.1`。
+`0.5.3` 已发布到 npm：含凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查、门禁严重度收敛、静态规则性能（真实 100 能力清单 `checkMany` 471ms→40ms）、`installed` 形状校验与 `profiles` 平台形态兼容。示例统一使用 `@0.5.3`。
+
+`0.5.4`（待发布）：用例新增 `excludeFromBase` / `excludeFromProfiles`（布尔，排除计分但不省调用）；评分卡新增行级 `errorKind` 与 `errors` / `errorsByKind` / `scoredTotal` / `selectionAccuracyExcludingErrors`；挂载对比不再把超时/HTTP 错误记成 jammed，分组表单列 `errors`；`mf audit` 分组表增加 `neg%`。契约见 `SCHEMA.md`。
 
 ## 数据边界
 
@@ -167,7 +169,7 @@ mf feedback --out out.json  # 导出脱敏反馈包（聚合值，不含原始
 
 ```bash
 # 1. 把 mf 挂成 MCP server
-claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.1 mcp
+claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.3 mcp
 ```
 
 ```text

package/SCHEMA.md

@@ -51,6 +51,8 @@
   "note": "",
   "source": "manual",
   "reviewed": true,
+  "excludeFromBase": false,
+  "excludeFromProfiles": false,
   "tags": []
 }
 ```
@@ -59,6 +61,7 @@
 - `expectedArgs` 仅在有 `expectedTool` 时可用；值为断言，支持字面量、`$contains`、`$oneOf`、`$exists`。
 - `toolInput` + `expect` 用于 `--execute`：`expect` 支持 `ok`、`contains`、`equals`、`errorType`、`maxDurationMs`、`maxPeakRssMB`。
 - `reviewed` 缺省为 `true`（手写用例）；`mf case add` 缺省写入 `source=agent`、`reviewed=false`。
+- `excludeFromBase` / `excludeFromProfiles` 必须是布尔值。用例仍会实际调用（不省 token），只是不计入对应范围的正式指标；用于「该用例只对某一挂载形态有意义」的情况。
 
 ### skills[]
 
@@ -73,7 +76,7 @@ skills 只做静态检查，不参与模型评测。
 
 ### profiles
 
-组名 → 工具名数组。所有工具名必须存在；组可重叠。`mf eval --profile` 与 `mf ablate --profile` 依赖它。
+组名 → 工具名数组（`{ "group": ["tool", ...] }`）。也接受平台常见的两种形态：对象 `{ "0": { agentType, tools, skills } }` 与数组 `[{ agentType, tools, skills }]`，组名依次取 `name` → `agentType` → 键/下标；`skills` 字段目前被忽略（profiles 只用于工具挂载与 eval 分组）。所有工具名必须存在；组可重叠，组名不可重复。`mf eval --profile` 与 `mf ablate --profile` 依赖它。
 
 ### execution
 
@@ -89,8 +92,10 @@ skills 只做静态检查，不参与模型评测。
     "fingerprint": "sha256:...",
     "execution": { "runtime": "node", "engine": "node-permission", "entry": "index.mjs", "hash": "..." }
   },
-  "models": [ { "model": "dsv4-flash", "total": 12, "reviewedTotal": 12, "selectionAccuracy": 0.83,
-               "argAccuracy": 0.9, "passRate": 0.75, "errors": 0, "p50LatencyMs": 900,
+  "models": [ { "model": "dsv4-flash", "total": 12, "reviewedTotal": 12, "excludedTotal": 0,
+               "selectionAccuracy": 0.83, "selectionAccuracyExcludingErrors": 0.83,
+               "argAccuracy": 0.9, "passRate": 0.75, "errors": 0, "errorsByKind": {},
+               "scoredTotal": 12, "p50LatencyMs": 900,
                "p95LatencyMs": 1800, "avgCostCny": 0.001, "totalCostCny": 0.012,
                "inputTokens": 1000, "cachedTokens": 0, "outputTokens": 200,
                "rows": [ { "id": "c1", "expectedTool": "...", "actualTool": "...",
@@ -101,9 +106,10 @@ skills 只做静态检查，不参与模型评测。
 }
 ```
 
-- 正式指标只统计 `reviewed` 行；未复核结果单独给出。
+- 正式指标只统计 `reviewed` 且未被 `excludeFromBase` 排除的行；未复核与排除结果单独给出。
+- `errors` 是调用层失败（超时、HTTP、网络）的数量，按 `errorsByKind` 分类；它们仍计入 `selectionAccuracy`（保守口径），同时给出排除错误后的 `selectionAccuracyExcludingErrors` 与可评分行数 `scoredTotal`。
 - `--repeat` 时模型对象带 `stability`（翻转用例与多次结果分布）。
-- `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`。
+- `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`、`errors`；错误行不参与挂载对比（不会被记成 jammed），每个组的 `reviewed` 是可评分行数。
 - `fingerprint` 由工具与用例内容计算；`mf compare` 用它判断两份报告是否可比。
 
 ## 3. 扫描输出（`mf scan`）
@@ -112,7 +118,7 @@ skills 只做静态检查，不参与模型评测。
 
 report：`root`、`tools[]`（`name` / `description` / `parameters` / `origin` / `confidence` / `source` / `note?` / `descriptionSource?`）、`skills[]`、`mcpServers[]`（env 只保留 `envKeys`，headers 只保留 `headerKeys`；URL 去掉 userinfo / query / hash）、`probes[]`（`server` / `ok` / `toolCount?` / `tools?` / `error?`）、`sources[]`、`warnings[]`。`--probe` 支持 stdio 与 streamable HTTP 两种传输。
 
-`scan --json` 在未找到任何工具时退出码为 3。
+`scan --json` 在工具与 skill 都未找到时退出码为 3（只有 skill 也算有结果，退出 0）。
 
 ## 4. lint JSON
 
@@ -159,7 +165,7 @@ report：`root`、`tools[]`（`name` / `description` / `parameters` / `origin` /
 import { check } from "modelfunction-cli";
 ```
 
-输入 `{ manifest, installed }`：`manifest` 是一个能力（`name` 必填，`description` 可选，`parameters` / `inputSchema` / `input` 任一为 JSON Schema；无 schema 视为 skill）。`installed` 可以是能力数组，或 `{ tools: [], skills: [] }`。
+输入 `{ manifest, installed }`：`manifest` 是一个能力（`name` 必填，`description` 可选，`parameters` / `inputSchema` / `input` 任一为 JSON Schema；无 schema 视为 skill）。`installed` 可以是能力数组、`{ tools: [], skills: [] }`，或直接传 `mf scan --json` 的输出（能力在 `report` 下）。空对象视为没有已装能力；其他无法识别的形状会抛错，避免静默返回假的 `pass`。
 
 返回：
 
@@ -175,8 +181,9 @@ import { check } from "modelfunction-cli";
 
 - `verdict` 只有 `pass` / `warn`，**没有 fail**；质量判定不阻断安装。
 - **严重度策略（门禁语境）**：只有会改变模型行为的规则保持原等级——`desc-missing`、`param-type-missing`（error），`required-unsourced`、`pair-no-boundary`、`skill-pair-no-boundary`（warn）；描述、命名、必填数量等质量规则降为 `info`，避免成熟工具集每次安装都告警。`mf lint` 不受影响，仍输出原始严重度。
-- 唯一抛错场景：`manifest.name` 缺失或为空。
+- 抛错场景：`manifest.name` 缺失或为空；`installed` 不是上述三种形状之一（空对象仍视为无已装能力）。
 - 同步、无网络、无文件系统、无状态。类型声明见 `lib/check.d.ts`。
+- 性能（0.5.3）：每个能力只分词一次，且只比较涉及新能力的对。真实 100 能力清单（53 新 + 47 已装）约 40–50ms，47 工具对 53 skill 约 25ms；550 个高度同质的合成最坏情况约 2s。
 
 批量（一次安装一包）：
 
@@ -229,7 +236,7 @@ const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
 | 0 | 成功（`mf check` 的 warn 也是 0） |
 | 1 | 输入/用法错误；`mf compare` 判定不可直接比较 |
 | 2 | `mf eval --fail-under` 未达标、`--fail-on-regression` 发现回退，或 `--max-cost` 超限 |
-| 3 | `mf exec` 被权限策略拒绝；`mf scan --json` 未找到工具 |
+| 3 | `mf exec` 被权限策略拒绝；`mf scan --json` 未找到工具或 skill |
 | 4 | `mf exec` 超时 |
 
 ## 9. 本地文件（0.5.0 起）
@@ -337,4 +344,4 @@ const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
 }
 ```
 
-`profiles` 为 `null` 表示套件未定义分组。`clusters`/`coverage` 的明细与 §5/§4 同源；`actions` 是给人看的优先级列表。`mf audit` 恒退出 0（advisory），不阻断流程。
+`profiles` 为 `null` 表示套件未定义分组。`clusters`/`coverage` 的明细与 §5/§4 同源；`actions` 是给人看的优先级列表。`mf audit` 恒退出 0（advisory），不阻断流程；负例占比告警只按整 suite 计算，分组表里的 `negativeRatio` 是该组「正例 + 负例」内的比例，仅供参考。

package/lib/audit.js

@@ -134,11 +134,11 @@ export function formatAudit(report) {
   if (!profiles) {
     lines.push("未定义 profiles（平台可先用 `mf cluster --out <suite>` 生成候选分组）。");
   } else {
-    lines.push("group                   tools  cases  negatives  tools-without-cases");
-    lines.push("----------------------  -----  -----  ---------  -------------------");
+    lines.push("group                   tools  cases  negatives  neg%  tools-without-cases");
+    lines.push("----------------------  -----  -----  ---------  ----  -------------------");
     for (const row of profiles.groups) {
       lines.push(
-        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(9)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
+        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(9)}  ${`${Math.round((row.negativeRatio ?? 0) * 100)}%`.padStart(4)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
       );
     }
     if (profiles.unmounted.length > 0) lines.push("", `未挂载工具：${profiles.unmounted.join(", ")}`);

package/lib/cases.js

@@ -185,12 +185,12 @@ export function formatCoverage(report) {
     );
   }
   if (report.profiles && report.profiles.length > 0) {
-    lines.push("", "profiles（分组后可评分的用例；负例每组都跑）:");
-    lines.push("group                   tools  cases  neg  tools-without-cases");
-    lines.push("----------------------  -----  -----  ---  -------------------");
+    lines.push("", "profiles（分组后可评分的用例；负例每组都跑，比例按组内正例+负例计）:");
+    lines.push("group                   tools  cases  neg  neg%  tools-without-cases");
+    lines.push("----------------------  -----  -----  ---  ----  -------------------");
     for (const row of report.profiles) {
       lines.push(
-        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(3)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
+        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(3)}  ${`${Math.round((row.negativeRatio ?? 0) * 100)}%`.padStart(4)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
       );
     }
   }
@@ -207,38 +207,52 @@ export function formatCoverage(report) {
   return lines.join("\n");
 }
 
-export function pairSimilarity(x, y, { extraStop } = {}) {
-  const a = tokens(`${x.name} ${x.description}`, extraStop);
-  const b = tokens(`${y.name} ${y.description}`, extraStop);
-  const union = new Set([...a, ...b]);
-  let lexical = 0;
-  if (union.size > 0) {
-    let shared = 0;
-    for (const token of union) if (a.has(token) && b.has(token)) shared += 1;
-    lexical = shared / union.size;
-  }
-  const structural = structuralScore(x, y);
+// O(n²) callers prepare each capability once; the pair loop then only compares
+// sets. Tokenizing inside the loop made a 550-capability gate take ~14s.
+export function prepareItem(item, { extraStop } = {}) {
+  const names = nameTokens(item.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
+  return {
+    name: item.name,
+    tokens: tokens(`${item.name} ${item.description}`, extraStop),
+    nameTokens: names,
+    nameTokenSet: new Set(names),
+    params: paramNames(item),
+    required: new Set(item.parameters?.required ?? [])
+  };
+}
+
+export function pairSimilarityPrepared(a, b) {
+  const shared = intersectionSize(a.tokens, b.tokens);
+  const union = a.tokens.size + b.tokens.size - shared;
+  const lexical = union > 0 ? shared / union : 0;
+  const structural = intersects(a.nameTokenSet, b.nameTokenSet) ? structuralScorePrepared(a, b) : 0;
   return {
     similarity: Math.max(lexical, structural),
     basis: structural > lexical ? "structural" : "lexical"
   };
 }
 
+export function pairSimilarity(x, y, { extraStop } = {}) {
+  return pairSimilarityPrepared(prepareItem(x, { extraStop }), prepareItem(y, { extraStop }));
+}
+
 // Clustering wants "same family" (alarm_query / alarm_ack belong to one agent),
 // which is looser than "confusable" (lint's question). Shared name token plus
 // any parameter overlap is enough to belong together; two shared tokens are
 // enough on their own (fault_alarm_query / fault_alarm_ack take different ids).
 // One shared token with disjoint params is not: cross-domain tools often share
 // a bare verb (query / list) and nothing else.
-export function familyScore(x, y) {
-  const left = nameTokens(x.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
-  const right = nameTokens(y.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
-  const shared = left.filter((token) => right.includes(token));
-  if (shared.length === 0) return 0;
-  const paramOverlap = jaccard(paramNames(x), paramNames(y));
+export function familyScorePrepared(x, y) {
+  const shared = intersectionSize(x.nameTokenSet, y.nameTokenSet);
+  if (shared === 0) return 0;
+  const paramOverlap = jaccard(x.params, y.params);
   if (paramOverlap > 0) return 0.8;
-  if (shared.length >= 2) return 0.7;
-  return paramNames(x).size === 0 && paramNames(y).size === 0 ? 0.6 : 0.2;
+  if (shared >= 2) return 0.7;
+  return x.params.size === 0 && y.params.size === 0 ? 0.6 : 0.2;
+}
+
+export function familyScore(x, y) {
+  return familyScorePrepared(prepareItem(x), prepareItem(y));
 }
 
 // Turn a confusable pair into an authoring instruction. We do not invent the
@@ -280,12 +294,13 @@ export function boundaryHint(x, y) {
 }
 
 export function suspectedPairs(tools, { threshold = PAIR_SIMILARITY, extraStop } = {}) {
+  const prepared = tools.map((tool) => prepareItem(tool, { extraStop }));
   const pairs = [];
-  for (let i = 0; i < tools.length; i += 1) {
-    for (let j = i + 1; j < tools.length; j += 1) {
-      const { similarity, basis } = pairSimilarity(tools[i], tools[j], { extraStop });
+  for (let i = 0; i < prepared.length; i += 1) {
+    for (let j = i + 1; j < prepared.length; j += 1) {
+      const { similarity, basis } = pairSimilarityPrepared(prepared[i], prepared[j]);
       if (similarity >= threshold) {
-        pairs.push({ a: tools[i].name, b: tools[j].name, similarity, basis });
+        pairs.push({ a: prepared[i].name, b: prepared[j].name, similarity, basis });
       }
     }
   }
@@ -296,20 +311,13 @@ export function suspectedPairs(tools, { threshold = PAIR_SIMILARITY, extraStop }
 // parameter shape confuse the model even when the wording differs, which the
 // lexical score alone misses. Name order is unreliable: `tavily_search` leads
 // with the vendor, `search_docs` with the action.
-function structuralScore(x, y) {
-  const left = nameTokens(x.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
-  const right = nameTokens(y.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
-  const sharedName = left.some((token) => right.includes(token));
+function structuralScorePrepared(x, y) {
+  const sharedName = x.nameTokens.some((token) => y.nameTokens.includes(token));
   if (!sharedName) return 0;
-  const params = paramNames(x);
-  const otherParams = paramNames(y);
-  if (params.size === 0 && otherParams.size === 0) return 0;
-  const paramOverlap = jaccard(params, otherParams);
+  if (x.params.size === 0 && y.params.size === 0) return 0;
+  const paramOverlap = jaccard(x.params, y.params);
   if (paramOverlap < 0.34) return 0;
-  const requiredOverlap = jaccard(
-    new Set(x.parameters?.required ?? []),
-    new Set(y.parameters?.required ?? [])
-  );
+  const requiredOverlap = jaccard(x.required, y.required);
   return 0.5 + 0.5 * (0.6 * paramOverlap + 0.4 * requiredOverlap);
 }
 
@@ -340,10 +348,26 @@ function paramNames(tool) {
 
 function jaccard(a, b) {
   if (a.size === 0 && b.size === 0) return 1;
-  const union = new Set([...a, ...b]);
+  const shared = intersectionSize(a, b);
+  return shared / (a.size + b.size - shared);
+}
+
+// Iterate the smaller set; avoids allocating a union Set for every pair in the
+// O(n²) loops.
+function intersectionSize(a, b) {
+  const small = a.size <= b.size ? a : b;
+  const large = small === a ? b : a;
   let shared = 0;
-  for (const item of union) if (a.has(item) && b.has(item)) shared += 1;
-  return shared / union.size;
+  for (const item of small) if (large.has(item)) shared += 1;
+  return shared;
+}
+
+function intersects(a, b) {
+  if (a.size === 0 || b.size === 0) return false;
+  const small = a.size <= b.size ? a : b;
+  const large = small === a ? b : a;
+  for (const item of small) if (large.has(item)) return true;
+  return false;
 }
 
 function tokens(text, extraStop) {

package/lib/check.d.ts

@@ -17,9 +17,14 @@ export interface InstalledCapabilities {
   skills?: Capability[];
 }
 
+/** `mf scan --json` output; its capabilities live under `report`. */
+export interface ScanReport {
+  report: InstalledCapabilities;
+}
+
 export interface CheckInput {
   manifest: Capability;
-  installed?: Capability[] | InstalledCapabilities;
+  installed?: Capability[] | InstalledCapabilities | ScanReport;
 }
 
 export interface CheckFinding {
@@ -48,7 +53,7 @@ export interface CheckResult {
 
 export interface CheckManyInput {
   manifests: Capability[];
-  installed?: Capability[] | InstalledCapabilities;
+  installed?: Capability[] | InstalledCapabilities | ScanReport;
 }
 
 export interface CheckItemResult extends CheckResult {
@@ -66,14 +71,16 @@ export interface CheckManyResult {
 /**
  * Static install gate: judge one new capability against installed ones.
  * No model call, no network, no file system, no state.
- * @throws only when `manifest.name` is missing or empty.
+ * @throws when `manifest.name` is missing or empty, or `installed` is neither
+ * an array, `{ tools, skills }`, nor an `mf scan --json` report.
  */
 export declare function check(input: CheckInput): CheckResult;
 
 /**
  * Batch install gate: judge a pack of new capabilities against installed ones
  * and against each other.
- * @throws when manifests is empty, an item lacks a name, or names are duplicated.
+ * @throws when manifests is empty, an item lacks a name, names are duplicated,
+ * or `installed` is not an accepted shape.
  */
 export declare function checkMany(input: CheckManyInput): CheckManyResult;
 

package/lib/check.js

@@ -40,12 +40,32 @@ export function checkMany({ manifests, installed = [] } = {}) {
   };
 }
 
-function evaluateTargets(targets, installed) {
-  const pool = [];
-  if (Array.isArray(installed)) pool.push(...installed);
-  else if (installed && typeof installed === "object") {
-    pool.push(...(installed.tools ?? []), ...(installed.skills ?? []));
+// A gate that silently ignores a mis-shaped `installed` returns a false pass —
+// the worst failure mode for an install check. Accept the array and
+// {tools,skills} forms, plus `mf scan --json` output (wrapped in `report`);
+// anything else non-empty is an input error.
+function installedPool(installed) {
+  if (installed == null) return [];
+  if (Array.isArray(installed)) return installed;
+  if (typeof installed !== "object") {
+    throw new Error("installed 需要能力数组、{ tools, skills } 或 mf scan --json 的 report");
+  }
+  if (Object.keys(installed).length === 0) return [];
+  const source =
+    installed.report && typeof installed.report === "object" ? installed.report : installed;
+  const tools = source.tools ?? [];
+  const skills = source.skills ?? [];
+  if (!Array.isArray(tools) || !Array.isArray(skills)) {
+    throw new Error("installed.tools / installed.skills 必须是数组");
+  }
+  if (!Array.isArray(source.tools) && !Array.isArray(source.skills)) {
+    throw new Error("installed 需要能力数组、{ tools, skills } 或 mf scan --json 的 report");
   }
+  return [...tools, ...skills];
+}
+
+function evaluateTargets(targets, installed) {
+  const pool = installedPool(installed);
   const entries = pool.map(normalizeEntry).filter(Boolean);
 
   const tools = [];
@@ -53,7 +73,10 @@ function evaluateTargets(targets, installed) {
   for (const entry of entries) (isToolLike(entry) ? tools : skills).push(entry);
   for (const target of targets) (isToolLike(target) ? tools : skills).push(target);
 
-  const report = lintSuite({ tools, skills });
+  // The gate only reports findings touching the new capabilities; skipping
+  // installed × installed pairs keeps a 500-capability pool linear-ish.
+  const pairScope = new Set(targets.map((target) => target.name));
+  const report = lintSuite({ tools, skills }, { pairScope });
   const pairFindings = report.findings.filter((finding) => finding.pair);
   const results = targets.map((target) => {
     const findings = report.findings
@@ -102,6 +125,11 @@ function normalizeEntry(entry) {
   return {
     name,
     description: typeof entry.description === "string" ? entry.description : "",
+    // Scan marks descriptions assembled at runtime; without this the gate
+    // treats every Claude-Code-style tool as "description missing".
+    ...(typeof entry.descriptionSource === "string"
+      ? { descriptionSource: entry.descriptionSource }
+      : {}),
     parameters: parameters && typeof parameters === "object" ? parameters : null
   };
 }

package/lib/cluster.js

@@ -1,4 +1,10 @@
-import { pairSimilarity, familyScore, nameTokens, GENERIC_NAME_TOKENS } from "./cases.js";
+import {
+  prepareItem,
+  pairSimilarityPrepared,
+  familyScorePrepared,
+  nameTokens,
+  GENERIC_NAME_TOKENS
+} from "./cases.js";
 
 // A shared domain noun ("session", "goal", "tool") says these tools serve one
 // purpose even when their verbs and parameter shapes differ — exactly what
@@ -21,12 +27,13 @@ const BOUNDARY_FAMILY = 0.6;
 
 export function clusterTools(tools, { threshold = 0.5, maxSize = 10 } = {}) {
   const scores = tools.map(() => new Array(tools.length).fill(0));
+  const prepared = tools.map((tool) => prepareItem(tool));
   const pairs = [];
   const edges = [];
   for (let i = 0; i < tools.length; i += 1) {
     for (let j = i + 1; j < tools.length; j += 1) {
-      const { similarity, basis } = pairSimilarity(tools[i], tools[j]);
-      const family = familyScore(tools[i], tools[j]);
+      const { similarity, basis } = pairSimilarityPrepared(prepared[i], prepared[j]);
+      const family = familyScorePrepared(prepared[i], prepared[j]);
       const link = Math.max(similarity, family);
       scores[i][j] = link;
       scores[j][i] = link;

package/lib/lint.js

@@ -57,7 +57,12 @@ const BOUNDARY_HINT = /(不适用|不要用于|不用于|不能用于|而非|区
 // Chinese descriptions are compared as bigrams, so mid-0.3s similarity is noise.
 const NO_BOUNDARY_SIMILARITY = 0.5;
 
-export function lintSuite(suite) {
+// `pairScope` (a Set of names) is for incremental gates: only emit pair
+// findings that touch one of the new capabilities. Installed × installed pairs
+// are computed and then discarded by the caller, which dominates large sets.
+export function lintSuite(suite, { pairScope } = {}) {
+  const inScope = (pair) => !pairScope || pairScope.has(pair.a) || pairScope.has(pair.b);
+  const toolByName = new Map(suite.tools.map((tool) => [tool.name, tool]));
   const findings = [];
   const add = (severity, code, tool, message, extra = {}) =>
     findings.push({ severity, code, kind: "tool", tool, message, ...extra });
@@ -205,9 +210,10 @@ export function lintSuite(suite) {
   }
 
   for (const pair of suspectedPairs(suite.tools)) {
+    if (!inScope(pair)) continue;
     const similarity = pair.similarity.toFixed(2);
     const declaresBoundary = [pair.a, pair.b].some((name) => {
-      const tool = suite.tools.find((item) => item.name === name);
+      const tool = toolByName.get(name);
       return BOUNDARY_HINT.test(tool?.description ?? "");
     });
     if (declaresBoundary) {
@@ -223,8 +229,8 @@ export function lintSuite(suite) {
         }
       }
       const reason = pair.basis === "structural" ? "名字与参数形状接近" : "描述相近";
-      const toolA = suite.tools.find((tool) => tool.name === pair.a);
-      const toolB = suite.tools.find((tool) => tool.name === pair.b);
+      const toolA = toolByName.get(pair.a);
+      const toolB = toolByName.get(pair.b);
       add("warn", "pair-no-boundary", pair.a, `与 ${pair.b} ${reason}（相似度 ${similarity}），且两边都没写“不适用于”边界，模型只能靠相似度硬猜`, {
         pair: [pair.a, pair.b],
         suggestion: boundaryHint(toolA, toolB).suggestion
@@ -236,6 +242,7 @@ export function lintSuite(suite) {
   // question is narrower than for tools: is the description specific enough to
   // be triggered, and does it collide with another installed skill?
   const skills = suite.skills ?? [];
+  const skillByName = new Map(skills.map((skill) => [skill.name, skill]));
   for (const skill of skills) {
     const description = String(skill.description ?? "").trim();
     if (!description) {
@@ -258,9 +265,10 @@ export function lintSuite(suite) {
     threshold: SKILL_PAIR_SIMILARITY,
     extraStop: SKILL_BOILERPLATE
   })) {
+    if (!inScope(pair)) continue;
     const similarity = pair.similarity.toFixed(2);
     const declaresBoundary = [pair.a, pair.b].some((name) => {
-      const skill = skills.find((item) => item.name === name);
+      const skill = skillByName.get(name);
       return BOUNDARY_HINT.test(skill?.description ?? "");
     });
     // Same-vendor siblings ("feishu-doc" / "feishu-wiki") share the vendor token

package/lib/profile.js

@@ -2,17 +2,28 @@ import { pct } from "./score.js";
 
 export function normalizeProfiles(raw, toolNames) {
   if (raw === undefined || raw === null) return null;
-  if (typeof raw !== "object" || Array.isArray(raw)) {
-    throw new Error('suite "profiles" must be an object of { "group": ["tool", ...] }');
+  if (typeof raw !== "object") {
+    throw new Error('suite "profiles" must be an object or an array of groups');
   }
   const known = new Set(toolNames);
   const profiles = [];
-  for (const [key, list] of Object.entries(raw)) {
-    const name = String(key).trim();
+  const seen = new Set();
+  // Platform registries often hand us an array of
+  // { agentType, tools, skills } rather than { "group": ["tool"] }.
+  const entries = Array.isArray(raw)
+    ? raw.map((value, index) => [String(index), value])
+    : Object.entries(raw);
+  for (const [key, value] of entries) {
+    // Array form: { "group": ["tool", ...] }. Object form (platform registries
+    // often look like this): { "0": { agentType, tools, skills } }.
+    const list = Array.isArray(value) ? value : value?.tools;
+    const name = String(Array.isArray(value) ? key : value?.name ?? value?.agentType ?? key).trim();
     if (!name) throw new Error('suite "profiles": group name must be non-empty');
     if (!Array.isArray(list) || list.length === 0) {
       throw new Error(`profile "${name}": must be a non-empty array of tool names`);
     }
+    if (seen.has(name)) throw new Error(`profile "${name}": duplicate group name`);
+    seen.add(name);
     const tools = [];
     for (const item of list) {
       if (typeof item !== "string" || item.trim() === "") {
@@ -43,7 +54,11 @@ export function selectProfiles(profiles, wanted) {
 // A case belongs to a group when its expected tool is mounted there. Negative
 // cases ("no tool applies") hold everywhere: the group's agent still gets them.
 export function selectProfileCases(cases, mountedTools) {
-  return cases.filter((testCase) => testCase.expectedTool === null || mountedTools.has(testCase.expectedTool));
+  return cases.filter(
+    (testCase) =>
+      testCase.excludeFromProfiles !== true &&
+      (testCase.expectedTool === null || mountedTools.has(testCase.expectedTool))
+  );
 }
 
 export function profileWarnings(profiles, suite) {
@@ -65,11 +80,12 @@ export function profileWarnings(profiles, suite) {
 }
 
 export function profileCaseCoverage(profiles, cases) {
-  const negatives = cases.filter((testCase) => testCase.expectedTool === null).length;
+  const considered = cases.filter((testCase) => testCase.excludeFromProfiles !== true);
+  const negatives = considered.filter((testCase) => testCase.expectedTool === null).length;
   return profiles.map((profile) => {
     const mounted = new Set(profile.tools);
     const counts = new Map(profile.tools.map((name) => [name, 0]));
-    for (const testCase of cases) {
+    for (const testCase of considered) {
       if (testCase.expectedTool !== null && mounted.has(testCase.expectedTool)) {
         counts.set(testCase.expectedTool, counts.get(testCase.expectedTool) + 1);
       }
@@ -77,7 +93,14 @@ export function profileCaseCoverage(profiles, cases) {
     const missing = [...counts.entries()].filter(([, count]) => count === 0).map(([name]) => name);
     let total = 0;
     for (const count of counts.values()) total += count;
-    return { name: profile.name, tools: profile.tools.length, cases: total, negatives, missing };
+    return {
+      name: profile.name,
+      tools: profile.tools.length,
+      cases: total,
+      negatives,
+      negativeRatio: total + negatives > 0 ? negatives / (total + negatives) : 0,
+      missing
+    };
   });
 }
 
@@ -88,18 +111,21 @@ export function analyzeMount(profile, mounted, allInOne) {
   for (const before of allInOne.rows) {
     const row = after.get(before.id);
     if (!row || before.reviewed === false || row.reviewed === false) continue;
+    // A provider error is not evidence about mounting; score it separately.
+    if (before.error || row.error) continue;
     if (!before.selectionCorrect && row.selectionCorrect) lifted.push(changeOf(before, row));
     else if (before.selectionCorrect && !row.selectionCorrect) jammed.push(changeOf(before, row));
   }
 
   const mountedMetrics = metricsOf(mounted);
   const allMetrics = metricsOf(allInOne);
-  const reviewed = mounted.reviewedTotal;
+  const reviewed = mountedMetrics.scored;
   return {
     profile: profile.name,
     mountedTools: profile.tools.length,
     cases: mounted.total,
     reviewed,
+    errors: mountedMetrics.errors,
     mounted: mountedMetrics,
     allInOne: allMetrics,
     delta: {
@@ -117,32 +143,44 @@ export function analyzeMount(profile, mounted, allInOne) {
 
 export function modelTotals(entries) {
   let reviewed = 0;
+  let errors = 0;
   let mountedCorrect = 0;
   let allCorrect = 0;
+  const usage = { inputTokens: 0, cachedTokens: 0, outputTokens: 0, costCny: 0, estimated: false };
   for (const entry of entries) {
     reviewed += entry.reviewed;
+    errors += entry.errors ?? 0;
     mountedCorrect += entry.mounted.correct;
     allCorrect += entry.allInOne.correct;
+    usage.inputTokens += entry.usage?.inputTokens ?? 0;
+    usage.cachedTokens += entry.usage?.cachedTokens ?? 0;
+    usage.outputTokens += entry.usage?.outputTokens ?? 0;
+    usage.costCny += entry.usage?.costCny ?? 0;
+    usage.estimated ||= Boolean(entry.usage?.estimated);
   }
   if (reviewed === 0) {
     return {
       reviewed: 0,
+      errors,
       mountedCorrect: 0,
       allCorrect: 0,
       mountedAccuracy: null,
       allAccuracy: null,
-      delta: null
+      delta: null,
+      usage
     };
   }
   const mountedAccuracy = mountedCorrect / reviewed;
   const allAccuracy = allCorrect / reviewed;
   return {
     reviewed,
+    errors,
     mountedCorrect,
     allCorrect,
     mountedAccuracy,
     allAccuracy,
-    delta: mountedAccuracy - allAccuracy
+    delta: mountedAccuracy - allAccuracy,
+    usage
   };
 }
 
@@ -158,15 +196,16 @@ export function formatProfileReport(report, { showCases = false } = {}) {
       entry.profile,
       String(entry.mountedTools),
       String(entry.cases),
+      String(entry.errors ?? 0),
       pct(entry.mounted.selectionAccuracy),
       pct(entry.allInOne.selectionAccuracy),
       signedPoints(entry.delta.selection)
     ]);
     lines.push(
       ...renderTable(
-        ["profile", "tools", "cases", "mounted", "all-in-one", "delta"],
+        ["profile", "tools", "cases", "errors", "mounted", "all-in-one", "delta"],
         rows,
-        [1, 2, 3, 4, 5]
+        [1, 2, 3, 4, 5, 6]
       )
     );
     const totals = model.totals;
@@ -174,7 +213,8 @@ export function formatProfileReport(report, { showCases = false } = {}) {
       lines.push(
         `  合计: 分组 ${pct(totals.mountedAccuracy)}（${totals.mountedCorrect}/${totals.reviewed}）` +
           ` / 全量 ${pct(totals.allAccuracy)}（${totals.allCorrect}/${totals.reviewed}）` +
-          `，${signedPoints(totals.delta)}`
+          `，${signedPoints(totals.delta)}` +
+          (totals.errors > 0 ? `；${totals.errors} 次调用错误不计入对比` : "")
       );
     }
     const unstable = unstableOf(model.profiles);
@@ -213,12 +253,12 @@ export function formatProfileMarkdown(report) {
     lines.push(
       `### ${model.model}`,
       "",
-      "| group | tools | cases | mounted | all-in-one | delta |",
-      "| --- | --- | --- | --- | --- | --- |"
+      "| group | tools | cases | errors | mounted | all-in-one | delta |",
+      "| --- | --- | --- | --- | --- | --- | --- |"
     );
     for (const entry of model.profiles) {
       lines.push(
-        `| ${entry.profile} | ${entry.mountedTools} | ${entry.cases} | ${pct(entry.mounted.selectionAccuracy)} | ${pct(entry.allInOne.selectionAccuracy)} | ${signedPoints(entry.delta.selection)} |`
+        `| ${entry.profile} | ${entry.mountedTools} | ${entry.cases} | ${entry.errors ?? 0} | ${pct(entry.mounted.selectionAccuracy)} | ${pct(entry.allInOne.selectionAccuracy)} | ${signedPoints(entry.delta.selection)} |`
       );
     }
     const totals = model.totals;
@@ -226,7 +266,8 @@ export function formatProfileMarkdown(report) {
     if (totals.reviewed > 0) {
       lines.push(
         `- 合计：分组 ${pct(totals.mountedAccuracy)}（${totals.mountedCorrect}/${totals.reviewed}），` +
-          `全量 ${pct(totals.allAccuracy)}（${totals.allCorrect}/${totals.reviewed}），${signedPoints(totals.delta)}`
+          `全量 ${pct(totals.allAccuracy)}（${totals.allCorrect}/${totals.reviewed}），${signedPoints(totals.delta)}` +
+          (totals.errors > 0 ? `；${totals.errors} 次调用错误不计入对比` : "")
       );
     } else {
       lines.push("- 合计：没有已复核用例，无法比较");
@@ -256,14 +297,18 @@ function changeOf(before, after) {
 
 function metricsOf(summary) {
   const reviewedRows = summary.rows.filter((row) => row.reviewed !== false);
-  const correct = reviewedRows.filter((row) => row.selectionCorrect).length;
+  const errorRows = reviewedRows.filter((row) => row.error);
+  const viableRows = reviewedRows.filter((row) => !row.error);
+  const correct = viableRows.filter((row) => row.selectionCorrect).length;
   return {
-    selectionAccuracy: reviewedRows.length ? correct / reviewedRows.length : null,
+    selectionAccuracy: viableRows.length ? correct / viableRows.length : null,
     argAccuracy: summary.argAccuracy,
-    passRate: reviewedRows.length
-      ? reviewedRows.filter((row) => row.passed).length / reviewedRows.length
+    passRate: viableRows.length
+      ? viableRows.filter((row) => row.passed).length / viableRows.length
       : null,
-    correct
+    correct,
+    errors: errorRows.length,
+    scored: viableRows.length
   };
 }
 

package/lib/score.js

@@ -89,7 +89,7 @@ export function costCnyOf(usage, pricing) {
   );
 }
 
-export function scoreModel({ model, cases, results, pricing }) {
+export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
   const rows = cases.map((testCase, index) => {
     const result = results[index];
     const actualTool = result.ok ? (result.toolCalls?.[0]?.name ?? null) : null;
@@ -98,6 +98,7 @@ export function scoreModel({ model, cases, results, pricing }) {
       ? matchArgs(result.toolCalls?.[0]?.arguments, testCase.expectedArgs)
       : { total: 0, matched: 0, failures: [] };
     const passed = Boolean(result.ok) && selectionCorrect && args.failures.length === 0;
+    const error = result.error ?? null;
     return {
       id: testCase.id,
       expectedTool: testCase.expectedTool,
@@ -106,21 +107,35 @@ export function scoreModel({ model, cases, results, pricing }) {
       args,
       passed,
       reviewed: testCase.reviewed !== false,
+      ...(testCase.excludeFromBase ? { excludeFromBase: true } : {}),
+      ...(testCase.excludeFromProfiles ? { excludeFromProfiles: true } : {}),
       source: testCase.source ?? "manual",
       latencyMs: result.latencyMs ?? 0,
       usage: result.usage ?? { input: 0, output: 0 },
-      error: result.error ?? null
+      error,
+      ...(error ? { errorKind: classifyError(error) } : {})
     };
   });
 
   const total = rows.length;
-  const reviewedRows = rows.filter((row) => row.reviewed);
+  // An excluded case still runs (so profile / all-in-one views keep their rows);
+  // it just does not contribute to this scope's official metrics.
+  const excluded = (row) => (scope === "profiles" ? row.excludeFromProfiles : row.excludeFromBase);
+  const reviewedRows = rows.filter((row) => row.reviewed && !excluded(row));
   const unreviewedRows = rows.filter((row) => !row.reviewed);
+  const excludedRows = rows.filter((row) => row.reviewed && excluded(row));
   const selectionHits = reviewedRows.filter((row) => row.selectionCorrect).length;
   const argFields = reviewedRows.reduce((sum, row) => sum + row.args.total, 0);
   const argHits = reviewedRows.reduce((sum, row) => sum + row.args.matched, 0);
   const passed = reviewedRows.filter((row) => row.passed).length;
-  const errors = rows.filter((row) => row.error).length;
+  const errorRows = reviewedRows.filter((row) => row.error);
+  const errors = errorRows.length;
+  const errorsByKind = errorRows.reduce((counts, row) => {
+    counts[row.errorKind] = (counts[row.errorKind] ?? 0) + 1;
+    return counts;
+  }, {});
+  const scoredTotal = reviewedRows.length - errors;
+  const selectionHitsWithoutErrors = reviewedRows.filter((row) => !row.error && row.selectionCorrect).length;
   const latencies = rows.filter((row) => row.latencyMs > 0).map((row) => row.latencyMs);
   const inputTokens = rows.reduce((sum, row) => sum + (row.usage.input ?? 0), 0);
   const outputTokens = rows.reduce((sum, row) => sum + (row.usage.output ?? 0), 0);
@@ -135,14 +150,18 @@ export function scoreModel({ model, cases, results, pricing }) {
     model,
     total,
     reviewedTotal,
+    excludedTotal: excludedRows.length,
     unreviewedTotal: unreviewedRows.length,
     unreviewedSelectionAccuracy: unreviewedRows.length
       ? unreviewedRows.filter((row) => row.selectionCorrect).length / unreviewedRows.length
       : null,
     selectionAccuracy: reviewedTotal ? selectionHits / reviewedTotal : 0,
+    selectionAccuracyExcludingErrors: scoredTotal ? selectionHitsWithoutErrors / scoredTotal : null,
     argAccuracy: argFields ? argHits / argFields : null,
     passRate: reviewedTotal ? passed / reviewedTotal : 0,
     errors,
+    errorsByKind,
+    scoredTotal,
     p50LatencyMs: percentile(latencies, 0.5),
     p95LatencyMs: percentile(latencies, 0.95),
     avgCostCny: total ? costCny / total : 0,
@@ -154,6 +173,15 @@ export function scoreModel({ model, cases, results, pricing }) {
   };
 }
 
+export function classifyError(error) {
+  const text = String(error ?? "");
+  if (/timeout|timed out/i.test(text)) return "timeout";
+  if (/^HTTP 429\b/.test(text)) return "rate-limit";
+  if (/^HTTP \d{3}\b/.test(text)) return "http";
+  if (/ECONN|ENOTFOUND|EAI_AGAIN|fetch failed|network/i.test(text)) return "network";
+  return "other";
+}
+
 function percentile(values, ratio) {
   if (values.length === 0) return 0;
   const sorted = [...values].sort((a, b) => a - b);

package/lib/suite.js

@@ -95,6 +95,11 @@ function normalizeCase(entry, index, toolNames) {
   if (toolInput && expectedTool === null) {
     throw new Error(`case ${id}: toolInput requires an expectedTool`);
   }
+  for (const key of ["excludeFromBase", "excludeFromProfiles"]) {
+    if (entry[key] !== undefined && typeof entry[key] !== "boolean") {
+      throw new Error(`case ${id}: "${key}" must be a boolean`);
+    }
+  }
   return {
     id,
     input: entry.input,
@@ -103,6 +108,8 @@ function normalizeCase(entry, index, toolNames) {
     note: entry.note ?? "",
     source: typeof entry.source === "string" && entry.source ? entry.source : "manual",
     reviewed: entry.reviewed === undefined ? true : Boolean(entry.reviewed),
+    ...(entry.excludeFromBase ? { excludeFromBase: true } : {}),
+    ...(entry.excludeFromProfiles ? { excludeFromProfiles: true } : {}),
     tags: Array.isArray(entry.tags) ? entry.tags.map(String) : [],
     toolInput,
     expect:

package/mf.js

@@ -153,7 +153,8 @@ export async function runEval(options, io = console) {
         model: adapter.id,
         cases,
         results: evaluated.results,
-        pricing
+        pricing,
+        scope: "base"
       });
       if (evaluated.stability) summary.stability = evaluated.stability;
       runs.push({ adapter, results: evaluated.results, summary });
@@ -250,6 +251,15 @@ export async function runEval(options, io = console) {
       if (summary.unreviewedTotal > 0) {
         io.log(`note: ${summary.unreviewedTotal} unreviewed case(s) excluded from select/args/pass`);
       }
+      if (summary.excludedTotal > 0) {
+        io.log(`note: ${summary.excludedTotal} case(s) marked excludeFromBase, excluded from base score`);
+      }
+      if (summary.errors > 0) {
+        const kinds = Object.entries(summary.errorsByKind)
+          .map(([kind, count]) => `${kind} ${count}`)
+          .join(", ");
+        io.log(`note: ${summary.errors} call error(s) counted as misses (${kinds})`);
+      }
       if (summary.exec) {
         const parts = [
           `exec: ${summary.exec.passed}/${summary.exec.executed} passed`,
@@ -482,7 +492,7 @@ export async function runScan(options, io = console) {
 
   if (options.json) {
     io.log(JSON.stringify({ report: redactReport(report), draft }, null, 2));
-    return report.tools.length > 0 ? 0 : 3;
+    return report.tools.length > 0 || report.skills.length > 0 ? 0 : 3;
   }
 
   io.log(formatScanSummary(report));
@@ -497,7 +507,7 @@ export async function runScan(options, io = console) {
     io.log(`wrote ${reportPath}`);
   }
 
-  return report.tools.length > 0 ? 0 : 3;
+  return report.tools.length > 0 || report.skills.length > 0 ? 0 : 3;
 }
 
 export async function runCaseAdd(options, io = console) {
@@ -915,6 +925,14 @@ async function evaluateProfiles({
   const selected = selectProfiles(profiles, selection);
   const indexById = new Map(cases.map((testCase, index) => [testCase.id, index]));
   const warnings = profileWarnings(selected, suite);
+  const excludedFromProfiles = cases.filter((testCase) => testCase.excludeFromProfiles === true);
+  if (excludedFromProfiles.length > 0) {
+    warnings.push(
+      `${excludedFromProfiles.length} 条用例标记 excludeFromProfiles，不参与分组评分: ${excludedFromProfiles
+        .map((testCase) => testCase.id)
+        .join(", ")}`
+    );
+  }
 
   const groups = [];
   const caseSets = selected.map((profile) => {
@@ -946,15 +964,24 @@ async function evaluateProfiles({
         model: run.adapter.id,
         cases: profileCases,
         results: mountedEval.results,
-        pricing
+        pricing,
+        scope: "profiles"
       });
       const allInOne = scoreModel({
         model: run.adapter.id,
         cases: profileCases,
         results: profileCases.map((testCase) => run.results[indexById.get(testCase.id)]),
-        pricing
+        pricing,
+        scope: "profiles"
       });
       const entry = analyzeMount(profile, mounted, allInOne);
+      entry.usage = {
+        inputTokens: mounted.inputTokens,
+        cachedTokens: mounted.cachedTokens,
+        outputTokens: mounted.outputTokens,
+        costCny: mounted.totalCostCny,
+        estimated: mounted.rows.some((row) => row.usage?.estimated)
+      };
       if (mountedEval.stability) entry.stability = mountedEval.stability;
       entries.push(entry);
     }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelfunction-cli",
-  "version": "0.5.2",
+  "version": "0.5.4",
   "type": "module",
   "license": "MIT",
   "description": "mf — function-calling accuracy scorecard for AI agents",