modelfunction-cli 0.5.0 → 0.5.1

package/README.md

@@ -5,7 +5,7 @@
 ## 安装
 
 ```bash
-npx -y modelfunction-cli@0.4.4 --version
+npx -y modelfunction-cli@0.5.0 --version
 # 或
 npm i -D modelfunction-cli
 npx mf --version
@@ -26,6 +26,7 @@ mf scan --root . --probe              # 连 stdio MCP server 拉 tools/list
 mf coverage --suite mf.suite.json     # 看每个工具还缺多少用例
 mf eval --suite mf.suite.json --models deepseek,qwen --show-cases
 mf compare before.json after.json     # 跨版本比较（内容指纹判定可比性）
+mf audit --suite mf.suite.json --out audit.md  # 一次静态体检：lint + 边界 + 覆盖 + 挂载
 ```
 
 被测模型用环境变量提供 Key：
@@ -74,10 +75,11 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
 
 | 命令 | 作用 |
 | --- | --- |
-| `mf scan` | 扫描工具定义（function.json / OpenAPI / MCP 配置 / Python `@tool` / JS·TS 声明与 TypeBox·zod schema）；`--probe` 连 MCP server（运行时结果覆盖静态推断）；`--from` 导入运行时工具数组；草稿按来源分组（`profiles`）并列出未覆盖的 skill |
-| `mf eval` | 跑评分卡；`--repeat` 多次取多数并报不稳定用例；`--fail-under` 阈值；`--baseline` + `--fail-on-regression` 回归门禁；`--profile` 按挂载组分别评分 |
+| `mf scan` | 扫描工具定义（function.json / OpenAPI / MCP 配置 / Python `@tool` / JS·TS 声明与 TypeBox·zod schema）；`--probe` 连 stdio 或 streamable-HTTP MCP server（运行时结果覆盖静态推断；headers/query 不写入报告）；`--from` 导入运行时工具数组；草稿按来源分组（`profiles`）并列出未覆盖的 skill |
+| `mf eval` | 跑评分卡；`--repeat` 多次取多数并报不稳定用例；`--fail-under` 阈值；`--baseline` + `--fail-on-regression` 回归门禁；`--profile` 按挂载组分别评分；`--max-cost` 按累计成本停机（退出 2）；`--progress` 逐用例进度（stderr） |
 | `mf case add` | 写入用例（默认 `source=agent`、`reviewed=false`） |
 | `mf coverage` | 每工具用例数、参数覆盖、负例比例、疑似混淆对；有 `profiles` 时按组统计 |
+| `mf audit` | 一次静态审计：lint + 混淆族边界 + 用例覆盖 + 挂载检查，输出可转发的 Markdown/JSON；不调用模型 |
 | `mf lint` | 静态检查工具描述与参数 schema，以及 skills 的 name/description（不调用模型）；重复规则自动合并成一条；有 `profiles` 时给出「这些组不要合并挂载」 |
 | `mf check` | 安装门禁：一个新能力 × 已装能力（`--manifest` + `--installed`），纯静态、返回 pass/warn、不阻断；也可 `import { check } from "modelfunction-cli"` |
 | `mf compare` | 跨版本比较、干扰归因、混淆矩阵；两份报告都有分组时按组对比 |
@@ -87,7 +89,7 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
 | `mf eval --execute` | 一次跑完选择 + 参数 + 运行断言（`toolInput` / `expect`） |
 | `mf sandbox check` | 探测本机可用隔离引擎与降级建议 |
 | `mf config set-key` | 把模型 Key 写入 `~/.modelfunction/credentials.json`（隐藏输入）；`remove-key` 删除；`list` 只显示掩码，`path` 打印路径 |
-| `mf stats` | 本地使用摘要（命令次数、模型、通过率区间、lint 规则命中）；不联网 |
+| `mf stats` | 本地使用摘要（命令次数、本地/CI 区分、模型、通过率区间、lint 规则命中）；不联网 |
 | `mf feedback` | 导出脱敏反馈包（仅计数与比率），发不发由你决定 |
 | `mf mcp` | stdio MCP server，供 Claude Code / Cursor / Kimi Code 调用 |
 
@@ -124,7 +126,7 @@ mf eval --suite mf.suite.json --models internal-ds \
   "mcpServers": {
     "modelfunction": {
       "command": "npx",
-      "args": ["-y", "modelfunction-cli@0.4.4", "mcp"]
+      "args": ["-y", "modelfunction-cli@0.5.0", "mcp"]
     }
   }
 }
@@ -136,7 +138,7 @@ mf eval --suite mf.suite.json --models internal-ds \
 
 对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md)，自 0.4.3 起冻结为 v1：v1 内只做加法，重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
 
-本仓库当前版本 `0.5.0`（凭证与本地反馈渠道，待发布）；上文的 `@0.4.4` 是 npm 上已发布的版本，保证示例可复制。发布后把示例里的版本号统一升级即可。
+npm 已发布 `0.5.0`（凭证与本地反馈渠道）；`0.5.1` 待发布，合并开发改动：usage `ci` 标记与 local/ci 统计、eval `--progress`/`--max-cost`、`mf audit` 组合审计、`scan --probe` 支持 streamable HTTP。示例固定 `@0.5.0`，发布后统一升级。
 
 ## 数据边界
 
@@ -146,7 +148,7 @@ mf eval --suite mf.suite.json --models internal-ds \
 
 ## 本地使用记录与反馈（默认不上传）
 
-每条命令会在 `~/.modelfunction/usage/YYYY-MM.jsonl` 追加一行**计数记录**：命令名、退出码、耗时、mf/Node/OS 版本、模型 id、用例/工具数量、通过率区间、lint 规则命中数。**不记录**用例原文、工具描述、源码、文件路径或密钥。
+每条命令会在 `~/.modelfunction/usage/YYYY-MM.jsonl` 追加一行**计数记录**：命令名、退出码、耗时、mf/Node/OS 版本、模型 id、用例/工具数量、通过率区间、lint 规则命中数、是否 CI。**不记录**用例原文、工具描述、源码、文件路径或密钥。`mf stats` 会把本地运行与 CI 运行分开统计。
 
 ```bash
 mf stats                    # 看本机记录；--json 给脚本
@@ -165,7 +167,7 @@ mf feedback --out out.json  # 导出脱敏反馈包（聚合值，不含原始
 
 ```bash
 # 1. 把 mf 挂成 MCP server
-claude mcp add modelfunction -- npx -y modelfunction-cli@0.4.4 mcp
+claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.0 mcp
 ```
 
 ```text

package/SCHEMA.md

@@ -1,6 +1,6 @@
 # Schemas v1
 
-自 `modelfunction-cli@0.4.3` 起冻结为 v1；`0.4.4` 是同一 v1 的补丁；`0.5.0` 增加 §9 本地文件，仍属 v1。这里描述 CLI 与库对外的稳定契约。人类可读文本（表格、提示语）不属于契约；**JSON 字段**属于。
+自 `modelfunction-cli@0.4.3` 起冻结为 v1；`0.4.4` 是同一 v1 的补丁；`0.5.0` 增加 §9 本地文件、`0.5.1` 增加 §10 审计 JSON，仍属 v1。这里描述 CLI 与库对外的稳定契约。人类可读文本（表格、提示语）不属于契约；**JSON 字段**属于。
 
 ## 兼容政策
 
@@ -110,7 +110,7 @@ skills 只做静态检查，不参与模型评测。
 
 `--json` 输出 `{ "report": {...}, "draft": {...} }`；默认写 `mf.suite.json`（draft）与 `mf.scan.json`（report）。
 
-report：`root`、`tools[]`（`name` / `description` / `parameters` / `origin` / `confidence` / `source` / `note?` / `descriptionSource?`）、`skills[]`、`mcpServers[]`（env 只保留 `envKeys`）、`probes[]`、`sources[]`、`warnings[]`。
+report：`root`、`tools[]`（`name` / `description` / `parameters` / `origin` / `confidence` / `source` / `note?` / `descriptionSource?`）、`skills[]`、`mcpServers[]`（env 只保留 `envKeys`，headers 只保留 `headerKeys`；URL 去掉 userinfo / query / hash）、`probes[]`（`server` / `ok` / `toolCount?` / `tools?` / `error?`）、`sources[]`、`warnings[]`。`--probe` 支持 stdio 与 streamable HTTP 两种传输。
 
 `scan --json` 在未找到任何工具时退出码为 3。
 
@@ -184,8 +184,9 @@ import { check } from "modelfunction-cli";
 | `scan_tools` | `root?`、`probe?`、`timeoutMs?`、`writeSuite?` |
 | `list_suite` | `suite?` |
 | `coverage` | `suite?` |
+| `audit` | `suite?` |
 | `add_cases` | `suite?`、`cases[]`、`source?`、`reviewed?`、`dryRun?` |
-| `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?` |
+| `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?`、`maxCost?`、`progress?` |
 
 返回值均为 JSON 文本。
 
@@ -195,7 +196,7 @@ import { check } from "modelfunction-cli";
 | --- | --- |
 | 0 | 成功（`mf check` 的 warn 也是 0） |
 | 1 | 输入/用法错误；`mf compare` 判定不可直接比较 |
-| 2 | `mf eval --fail-under` 未达标，或 `--fail-on-regression` 发现回退 |
+| 2 | `mf eval --fail-under` 未达标、`--fail-on-regression` 发现回退，或 `--max-cost` 超限 |
 | 3 | `mf exec` 被权限策略拒绝；`mf scan --json` 未找到工具 |
 | 4 | `mf exec` 超时 |
 
@@ -226,6 +227,7 @@ import { check } from "modelfunction-cli";
   "platform": "win32",
   "arch": "x64",
   "node": "24.6.0",
+  "ci": false,
   "models": ["mock"],
   "caseCount": 13,
   "toolCount": 4,
@@ -241,7 +243,7 @@ import { check } from "modelfunction-cli";
 }
 ```
 
-字段为白名单，只含计数、比率与 id；**不含**用例原文、工具描述、源码、文件路径或密钥。新增字段属于 v1 加法，消费方可忽略未知字段。`MF_NO_USAGE=1` 关闭写入；删除 `usage/` 即清空。
+字段为白名单，只含计数、比率与 id；**不含**用例原文、工具描述、源码、文件路径或密钥。`ci` 表示检测到 CI 环境（`CI=true/1` 或 `GITHUB_ACTIONS` / `GITLAB_CI` / `CIRCLECI` / `TRAVIS` / `BUILDKITE` / `TF_BUILD` / `JENKINS_URL` / `CNB_BUILD_ID`），用于把机器运行与本地运行分开；`mf stats` 分别统计。新增字段属于 v1 加法，消费方可忽略未知字段。`MF_NO_USAGE=1` 关闭写入；删除 `usage/` 即清空。
 
 ### `mf feedback` 包
 
@@ -260,3 +262,47 @@ import { check } from "modelfunction-cli";
 ```
 
 `usage` 是 `mf stats` 的聚合结果，不含原始事件、路径或密钥。是否发送由用户决定。
+
+## 10. 工具集审计（`mf audit --json`，0.5.1 起）
+
+```json
+{
+  "suite": { "name": "…", "path": "…", "tools": 4, "skills": 0, "cases": 13, "reviewed": 13 },
+  "lint": {
+    "tools": 4,
+    "skills": 0,
+    "counts": { "error": 0, "warn": 6, "info": 0 },
+    "findings": [],
+    "profilePairs": []
+  },
+  "clusters": {
+    "threshold": 0.5,
+    "total": 4,
+    "clusters": [],
+    "riskyPairs": [],
+    "familyHints": [],
+    "boundaries": []
+  },
+  "coverage": {
+    "total": 13,
+    "negative": 1,
+    "negativeRatio": 0.08,
+    "unreviewed": 0,
+    "tools": [],
+    "pairs": [],
+    "profiles": [],
+    "warnings": []
+  },
+  "profiles": { "groups": [], "warnings": [], "unmounted": [] },
+  "actions": [
+    {
+      "kind": "boundary|param-source|cases|args|review|mount",
+      "subject": "…",
+      "message": "…",
+      "detail": "…"
+    }
+  ]
+}
+```
+
+`profiles` 为 `null` 表示套件未定义分组。`clusters`/`coverage` 的明细与 §5/§4 同源；`actions` 是给人看的优先级列表。`mf audit` 恒退出 0（advisory），不阻断流程。

package/lib/audit.js

@@ -0,0 +1,149 @@
+// One static pass over a whole tool set: lint + cluster boundaries + case
+// coverage + mount checks. No model calls, no execution, no network.
+// The markdown output is meant to be forwarded as a trial deliverable.
+import { lintSuite, formatLint } from "./lint.js";
+import { clusterTools, formatClusters } from "./cluster.js";
+import { computeCoverage, formatCoverage } from "./cases.js";
+import { normalizeProfiles, profileWarnings, profileCaseCoverage } from "./profile.js";
+
+export function auditSuite(suite) {
+  const lint = lintSuite(suite);
+  const clusters = clusterTools(suite.tools);
+  const coverage = computeCoverage(suite);
+  const profiles = normalizeProfiles(suite.profiles, suite.tools.map((tool) => tool.name));
+  const mounted = profiles ? new Set(profiles.flatMap((profile) => profile.tools)) : null;
+  const profile = profiles
+    ? {
+        groups: profileCaseCoverage(profiles, suite.cases),
+        warnings: profileWarnings(profiles, suite),
+        unmounted: suite.tools.map((tool) => tool.name).filter((name) => !mounted.has(name))
+      }
+    : null;
+
+  return {
+    suite: {
+      name: suite.name,
+      path: suite.path ?? null,
+      tools: suite.tools.length,
+      skills: suite.skills?.length ?? 0,
+      cases: suite.cases.length,
+      reviewed: suite.cases.filter((testCase) => testCase.reviewed !== false).length
+    },
+    lint: {
+      tools: lint.tools,
+      skills: lint.skills,
+      counts: lint.counts,
+      findings: lint.findings,
+      profilePairs: lint.profilePairs ?? []
+    },
+    clusters,
+    coverage,
+    profiles: profile,
+    actions: buildActions({ lint, clusters, coverage, profile })
+  };
+}
+
+function buildActions({ lint, clusters, coverage, profile }) {
+  const actions = [];
+
+  const families = (clusters.boundaries ?? [])
+    .filter((family) => family.size > 1)
+    .sort((a, b) => b.pairs.length - a.pairs.length)
+    .slice(0, 3);
+  for (const family of families) {
+    actions.push({
+      kind: "boundary",
+      subject: family.label,
+      message: `为「${family.label}」补选择边界（${family.pairs.length} 对互抢）`,
+      detail: family.suggestion
+    });
+  }
+
+  const unsourced = lint.findings.filter((finding) => finding.code === "required-unsourced");
+  for (const finding of unsourced.slice(0, 5)) {
+    actions.push({
+      kind: "param-source",
+      subject: `${finding.tool}.${finding.param}`,
+      message: `补参数来源：${finding.tool}.${finding.param}`,
+      detail: finding.message
+    });
+  }
+
+  const empty = coverage.tools.filter((tool) => tool.total === 0);
+  if (empty.length > 0) {
+    actions.push({
+      kind: "cases",
+      subject: `${empty.length} tools`,
+      message: `为 ${empty.length} 个工具补用例`,
+      detail: empty.map((tool) => tool.name).join(", ")
+    });
+  }
+  const noArgs = coverage.tools.filter((tool) => tool.total > 0 && tool.withArgs === 0);
+  if (noArgs.length > 0) {
+    actions.push({
+      kind: "args",
+      subject: `${noArgs.length} tools`,
+      message: `为 ${noArgs.length} 个工具补带参数的用例`,
+      detail: noArgs.map((tool) => tool.name).join(", ")
+    });
+  }
+  if (coverage.unreviewed > 0) {
+    actions.push({
+      kind: "review",
+      subject: `${coverage.unreviewed} cases`,
+      message: `复核 ${coverage.unreviewed} 条未复核用例（复核后才计入正式分数）`,
+      detail: "把认可的用例 reviewed 改为 true"
+    });
+  }
+  if (profile && profile.unmounted.length > 0) {
+    actions.push({
+      kind: "mount",
+      subject: `${profile.unmounted.length} tools`,
+      message: `${profile.unmounted.length} 个工具不在任何挂载组`,
+      detail: profile.unmounted.join(", ")
+    });
+  }
+
+  return actions;
+}
+
+export function formatAudit(report) {
+  const { suite, lint, clusters, coverage, profiles, actions } = report;
+  const lines = [
+    `# ${suite.name} — 工具集审计`,
+    "",
+    `- 工具 ${suite.tools} · skills ${suite.skills} · 用例 ${suite.cases}（已复核 ${suite.reviewed}）`,
+    `- 文件：${suite.path ?? "(内存)"}`,
+    `- 纯静态：不调用模型、不执行工具、无网络`,
+    "",
+    "## 建议动作"
+  ];
+  if (actions.length === 0) {
+    lines.push("暂无必须处理项。");
+  } else {
+    actions.forEach((action, index) => {
+      lines.push(`${index + 1}. ${action.message}`);
+      if (action.detail) lines.push(`   ${action.detail}`);
+    });
+  }
+
+  lines.push("", "## 描述与参数（lint）", "", formatLint(lint));
+  lines.push("", "## 混淆与分组（cluster）", "", formatClusters(clusters));
+  lines.push("", "## 用例覆盖（coverage）", "", formatCoverage(coverage));
+  lines.push("", "## 挂载分组（profiles）", "");
+  if (!profiles) {
+    lines.push("未定义 profiles（平台可先用 `mf cluster --out <suite>` 生成候选分组）。");
+  } else {
+    lines.push("group                   tools  cases  negatives  tools-without-cases");
+    lines.push("----------------------  -----  -----  ---------  -------------------");
+    for (const row of profiles.groups) {
+      lines.push(
+        `${row.name.padEnd(22)}  ${String(row.tools).padStart(5)}  ${String(row.cases).padStart(5)}  ${String(row.negatives).padStart(9)}  ${row.missing.length ? row.missing.join(", ") : "-"}`
+      );
+    }
+    if (profiles.unmounted.length > 0) lines.push("", `未挂载工具：${profiles.unmounted.join(", ")}`);
+    for (const warning of profiles.warnings) lines.push(`- ${warning}`);
+  }
+
+  return lines.join("\n");
+}

package/lib/budget.js

@@ -0,0 +1,30 @@
+// Running cost guard for evaluations. Pricing is known per model, but token
+// usage is only known after a call returns, so the limit is enforced after
+// each call: the run stops as soon as the budget is crossed.
+export class CostLimitError extends Error {
+  constructor({ limit, spent, calls }) {
+    super(`cost limit reached: spent ¥${spent.toFixed(4)} of ¥${limit} after ${calls} call(s)`);
+    this.name = "CostLimitError";
+    this.limit = limit;
+    this.spent = spent;
+    this.calls = calls;
+  }
+}
+
+export function createBudget(limit = null) {
+  let spent = 0;
+  let calls = 0;
+  return {
+    get spent() {
+      return spent;
+    },
+    get calls() {
+      return calls;
+    },
+    spend(cost) {
+      calls += 1;
+      if (Number.isFinite(cost) && cost > 0) spent += cost;
+      if (limit !== null && spent > limit) throw new CostLimitError({ limit, spent, calls });
+    }
+  };
+}

package/lib/mcp-probe.js

@@ -1,10 +1,67 @@
 import { spawn } from "node:child_process";
+import { readFileSync } from "node:fs";
+import { Client } from "@modelcontextprotocol/sdk/client/index.js";
+import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js";
 
 const MAX_BYTES = 1_000_000;
 const MAX_STDERR = 4_000;
-const CLIENT_INFO = { name: "mf", version: "0.1.0" };
+const PACKAGE = JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8"));
+const CLIENT_INFO = { name: "mf", version: PACKAGE.version };
 const PROTOCOL_VERSION = "2025-06-18";
 
+// Streamable HTTP is the current remote MCP transport. Headers come from the
+// user's own config and are never copied into reports or logs.
+export async function probeHttpServer(server, { timeoutMs = 5_000 } = {}) {
+  if (!server?.url) return { ok: false, error: "missing url" };
+  let url;
+  try {
+    url = new URL(server.url);
+  } catch {
+    return { ok: false, error: "invalid url" };
+  }
+
+  const client = new Client(CLIENT_INFO);
+  const headers = server.headers && Object.keys(server.headers).length > 0 ? server.headers : null;
+  const transport = new StreamableHTTPClientTransport(url, {
+    requestInit: headers ? { headers } : undefined
+  });
+
+  try {
+    await withTimeout(client.connect(transport), timeoutMs, "connect");
+    const result = await withTimeout(client.listTools(), timeoutMs, "tools/list");
+    return { ok: true, tools: normalizeTools(result?.tools ?? []) };
+  } catch (error) {
+    return { ok: false, error: `http: ${scrubUrl(error?.message ?? error, url)}` };
+  } finally {
+    await client.close().catch(() => {});
+  }
+}
+
+function withTimeout(promise, ms, label) {
+  let id;
+  const timeout = new Promise((_, reject) => {
+    id = setTimeout(() => reject(new Error(`${label} timeout after ${ms}ms`)), ms);
+  });
+  return Promise.race([promise, timeout]).finally(() => clearTimeout(id));
+}
+
+function scrubUrl(message, url) {
+  let text = String(message ?? "");
+  text = text.split(url.href).join(`${url.origin}${url.pathname}`);
+  if (url.search) text = text.split(url.search).join("");
+  return text;
+}
+
+function normalizeTools(tools) {
+  return tools
+    .filter((tool) => typeof tool?.name === "string" && tool.name.trim() !== "")
+    .map((tool) => ({
+      name: tool.name.trim(),
+      description: tool.description ?? "",
+      parameters: tool.inputSchema ?? { type: "object", properties: {} }
+    }));
+}
+
 export async function probeStdioServer(server, { timeoutMs = 5_000, cwd } = {}) {
   if (!server?.command) {
     return { ok: false, error: "missing command" };
@@ -129,14 +186,7 @@ export async function probeStdioServer(server, { timeoutMs = 5_000, cwd } = {})
     });
     send({ jsonrpc: "2.0", method: "notifications/initialized" });
     const result = await request("tools/list", {});
-    const tools = (result?.tools ?? [])
-      .filter((tool) => typeof tool?.name === "string" && tool.name.trim() !== "")
-      .map((tool) => ({
-        name: tool.name.trim(),
-        description: tool.description ?? "",
-        parameters: tool.inputSchema ?? { type: "object", properties: {} }
-      }));
-    return { ok: true, tools };
+    return { ok: true, tools: normalizeTools(result?.tools ?? []) };
   } catch (error) {
     return {
       ok: false,

package/lib/mcp-server.js

@@ -8,8 +8,9 @@ import { z } from "zod";
 import { loadSuite } from "./suite.js";
 import { scanProject, addTools, buildSuiteDraft, redactReport } from "./scan.js";
 import { writeFileAtomic } from "./fsutil.js";
-import { probeStdioServer } from "./mcp-probe.js";
+import { probeStdioServer, probeHttpServer } from "./mcp-probe.js";
 import { addCases, computeCoverage } from "./cases.js";
+import { auditSuite } from "./audit.js";
 
 const MF_PATH = fileURLToPath(new URL("../mf.js", import.meta.url));
 const PACKAGE_JSON = new URL("../package.json", import.meta.url);
@@ -32,7 +33,7 @@ export async function startMcpServer() {
     {
       title: "Scan tools",
       description:
-        "扫描仓库里的 function.json / OpenAPI / MCP 配置 / SKILL.md / 代码内定义。probe=true 时连接 stdio MCP server 拉取 tools/list。不执行工具。默认只读；传 writeSuite 时在该路径写出套件草稿（cases 为空，需补用例并复核；来源 ≥2 时草稿带按来源分组的 profiles，并列出不参与评测的 skills）。",
+        "扫描仓库里的 function.json / OpenAPI / MCP 配置 / SKILL.md / 代码内定义。probe=true 时连接 stdio 或 streamable-HTTP MCP server 拉取 tools/list（HTTP headers 只用于连接，不写入报告）。不执行工具。默认只读；传 writeSuite 时在该路径写出套件草稿（cases 为空，需补用例并复核；来源 ≥2 时草稿带按来源分组的 profiles，并列出不参与评测的 skills）。",
       inputSchema: {
         root: z.string().optional().describe("仓库目录，默认进程当前目录"),
         probe: z.boolean().optional().describe("是否连接 MCP server 拉取工具清单"),
@@ -44,11 +45,10 @@ export async function startMcpServer() {
       const report = await scanProject(root ?? process.cwd());
       if (probe) {
         for (const mcpServer of [...report.mcpServers]) {
-          if (mcpServer.transport !== "stdio") {
-            report.probes.push({ server: mcpServer.name, ok: false, error: "http transport 暂不支持" });
-            continue;
-          }
-          const result = await probeStdioServer(mcpServer, { timeoutMs: timeoutMs ?? 5000 });
+          const result =
+            mcpServer.transport === "stdio"
+              ? await probeStdioServer(mcpServer, { timeoutMs: timeoutMs ?? 5000 })
+              : await probeHttpServer(mcpServer, { timeoutMs: timeoutMs ?? 5000 });
           if (!result.ok) {
             report.probes.push({ server: mcpServer.name, ok: false, error: result.error });
             continue;
@@ -109,6 +109,22 @@ export async function startMcpServer() {
     async ({ suite }) => json(computeCoverage(await loadSuite(suitePath(suite))))
   );
 
+  server.registerTool(
+    "audit",
+    {
+      title: "Audit tool set",
+      description:
+        "一次静态审计整个工具集：lint 发现、混淆族与边界建议、用例覆盖、挂载检查。不调用模型、不执行工具、无网络。返回 JSON。",
+      inputSchema: { suite: z.string().optional().describe("套件路径，默认 ./mf.suite.json") }
+    },
+    async ({ suite }) =>
+      json(
+        auditSuite(
+          await loadSuite(suitePath(suite), { allowEmptyCases: true, allowEmptyTools: true })
+        )
+      )
+  );
+
   server.registerTool(
     "add_cases",
     {
@@ -154,16 +170,20 @@ export async function startMcpServer() {
           .max(10)
           .optional()
           .describe("每条用例重复次数，取多数结果；建议正式基线用 3"),
-        failUnder: z.number().optional().describe("选择准确率阈值，低于则退出码为 2")
+        failUnder: z.number().optional().describe("选择准确率阈值，低于则退出码为 2"),
+        maxCost: z.number().positive().optional().describe("累计估算成本上限（CNY），超限退出 2 且不返回部分评分卡"),
+        progress: z.boolean().optional().describe("把逐用例进度写到 stderr（stdout 仍为纯 JSON）")
       }
     },
-    async ({ suite, models, cases, profile, repeat, failUnder }) => {
+    async ({ suite, models, cases, profile, repeat, failUnder, maxCost, progress }) => {
       const args = ["eval", "--json", "--suite", suitePath(suite)];
       if (models) args.push("--models", models);
       if (cases) args.push("--case", cases);
       if (profile) args.push("--profile", profile);
       if (repeat !== undefined) args.push("--repeat", String(repeat));
       if (failUnder !== undefined) args.push("--fail-under", String(failUnder));
+      if (maxCost !== undefined) args.push("--max-cost", String(maxCost));
+      if (progress) args.push("--progress");
 
       const result = await runCli(args);
       let scorecard = null;

package/lib/scan.js

@@ -337,9 +337,11 @@ export function addTools(
 export function redactReport(report) {
   return {
     ...report,
-    mcpServers: report.mcpServers.map(({ env, ...rest }) => ({
+    mcpServers: report.mcpServers.map(({ env, headers, ...rest }) => ({
       ...rest,
-      envKeys: env ? Object.keys(env) : []
+      url: rest.url ? sanitizeUrl(rest.url) : null,
+      envKeys: env ? Object.keys(env) : [],
+      headerKeys: headers ? Object.keys(headers).sort() : []
     }))
   };
 }
@@ -526,10 +528,26 @@ function mcpServersFrom(config) {
     command: value.command ?? null,
     args: Array.isArray(value.args) ? value.args : [],
     env: value.env && typeof value.env === "object" ? value.env : null,
+    headers: value.headers && typeof value.headers === "object" ? value.headers : null,
     url: value.url ?? null
   }));
 }
 
+// Remote MCP configs often carry tokens in the query string; keep the report
+// safe to share while the live server object still probes the full URL.
+function sanitizeUrl(raw) {
+  try {
+    const url = new URL(raw);
+    url.username = "";
+    url.password = "";
+    url.search = "";
+    url.hash = "";
+    return url.toString();
+  } catch {
+    return null;
+  }
+}
+
 function toolsFromOpenApi(spec) {
   const tools = [];
   const paths = spec.paths ?? {};

package/lib/score.js

@@ -76,6 +76,19 @@ export function pickRepresentative(runs) {
   };
 }
 
+export function costCnyOf(usage, pricing) {
+  if (!pricing) return 0;
+  const input = usage?.input ?? 0;
+  const cached = Math.min(input, usage?.cached ?? 0);
+  const output = usage?.output ?? 0;
+  return (
+    ((input - cached) * (pricing.input ?? 0) +
+      cached * (pricing.cachedInput ?? pricing.input ?? 0) +
+      output * (pricing.output ?? 0)) /
+    1e6
+  );
+}
+
 export function scoreModel({ model, cases, results, pricing }) {
   const rows = cases.map((testCase, index) => {
     const result = results[index];
@@ -115,13 +128,7 @@ export function scoreModel({ model, cases, results, pricing }) {
     inputTokens,
     rows.reduce((sum, row) => sum + (row.usage.cached ?? 0), 0)
   );
-  const uncachedTokens = inputTokens - cachedTokens;
-  const costCny = pricing
-    ? (uncachedTokens * (pricing.input ?? 0) +
-        cachedTokens * (pricing.cachedInput ?? pricing.input ?? 0) +
-        outputTokens * (pricing.output ?? 0)) /
-      1e6
-    : 0;
+  const costCny = rows.reduce((sum, row) => sum + costCnyOf(row.usage, pricing), 0);
   const reviewedTotal = reviewedRows.length;
 
   return {

package/lib/usage.js

@@ -6,6 +6,16 @@ import { configDir } from "./credentials.js";
 // Events are redacted by construction: only counts, ids and ratios leave the
 // process. Never add free-form text (inputs, descriptions, paths, errors).
 const OPT_OUT_ENV = "MF_NO_USAGE";
+const CI_ENV_KEYS = [
+  "GITHUB_ACTIONS",
+  "GITLAB_CI",
+  "CIRCLECI",
+  "TRAVIS",
+  "BUILDKITE",
+  "TF_BUILD",
+  "JENKINS_URL",
+  "CNB_BUILD_ID"
+];
 
 let pendingDetails = null;
 
@@ -75,10 +85,16 @@ export function buildRecord({ command, exitCode, durationMs, version }) {
     platform: process.platform,
     arch: process.arch,
     node: process.versions.node,
+    ci: detectCi(),
     ...(pendingDetails ?? {})
   };
 }
 
+function detectCi() {
+  if (process.env.CI === "true" || process.env.CI === "1") return true;
+  return CI_ENV_KEYS.some((key) => Boolean(process.env[key]));
+}
+
 export async function readUsageEvents() {
   let files = [];
   try {
@@ -112,6 +128,8 @@ export function summarizeUsage(events) {
     firstRun: events[0]?.ts ?? null,
     lastRun: events.at(-1)?.ts ?? null,
     activeDays: new Set(events.map((event) => event.ts.slice(0, 10))).size,
+    ciRuns: 0,
+    localRuns: 0,
     exitCodes: {},
     commands: {},
     platforms: {},
@@ -128,6 +146,8 @@ export function summarizeUsage(events) {
   const args = [];
 
   for (const event of events) {
+    if (event.ci) summary.ciRuns += 1;
+    else summary.localRuns += 1;
     bump(summary.exitCodes, String(event.exitCode ?? 0));
     bump(summary.commands, event.command ?? "unknown");
     if (event.platform) bump(summary.platforms, event.platform);
@@ -198,7 +218,7 @@ export function formatStats(summary, { events = 0, files = 0, skipped = 0, path
   }
   const lines = [
     `local usage: ${path} (${events} events, ${files} file(s)${skipped > 0 ? `, ${skipped} unreadable line(s)` : ""})`,
-    `runs: ${summary.runs}   first: ${summary.firstRun}   last: ${summary.lastRun}   active days: ${summary.activeDays}`,
+    `runs: ${summary.runs} (local ${summary.localRuns}, ci ${summary.ciRuns})   first: ${summary.firstRun}   last: ${summary.lastRun}   active days: ${summary.activeDays}`,
     `exit codes: ${pairs(summary.exitCodes)}`,
     `commands:   ${pairs(summary.commands)}`
   ];

package/mf.js

@@ -4,14 +4,15 @@ import { mkdir, readFile } from "node:fs/promises";
 import { basename, dirname, extname, relative, resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 import { loadSuite, toModelTools } from "./lib/suite.js";
-import { scoreModel, pickRepresentative, pct } from "./lib/score.js";
+import { scoreModel, pickRepresentative, pct, costCnyOf } from "./lib/score.js";
 import { formatTable, formatCaseLines, formatMarkdown } from "./lib/report.js";
 import { scanProject, buildSuiteDraft, formatScanSummary, addTools, redactReport } from "./lib/scan.js";
-import { probeStdioServer } from "./lib/mcp-probe.js";
+import { probeStdioServer, probeHttpServer } from "./lib/mcp-probe.js";
 import { loadAdapterTools } from "./lib/adapter-import.js";
 import { addCases, computeCoverage, formatCoverage } from "./lib/cases.js";
 import { lintSuite, formatLint } from "./lib/lint.js";
 import { check, formatCheck } from "./lib/check.js";
+import { auditSuite, formatAudit } from "./lib/audit.js";
 import { suiteFingerprint, hashText, hashOf } from "./lib/fingerprint.js";
 import { runExecutionChecks } from "./lib/eval-exec.js";
 import { checkSandbox, formatSandboxReport } from "./lib/sandbox-check.js";
@@ -30,6 +31,7 @@ import {
   formatProfileMarkdown
 } from "./lib/profile.js";
 import { runExec } from "./lib/exec.js";
+import { createBudget, CostLimitError } from "./lib/budget.js";
 import { writeFileAtomic } from "./lib/fsutil.js";
 import {
   applyModelCredentials,
@@ -71,10 +73,11 @@ Usage:
   mf scan [options]
   mf case add --suite <file> (--input <file> | stdin) [--source agent] [--reviewed] [--dry-run]
   mf coverage [--suite <file>] [--json]
+  mf audit [--suite <file>] [--out <file>] [--json]  # 静态审计：lint + 分组边界 + 用例覆盖 + 挂载
   mf lint [--suite <file>] [--json]  # 静态检查 tools 与 skills，不调用模型
   mf check --manifest <file> [--installed <file>] [--json]  # 安装门禁：新能力 × 已装能力
   mf compare <before.json> <after.json> [--json]
-  mf ablate --tool <name> [--suite <file>] [--profile <group>] [--models <ids>] [--json]
+  mf ablate --tool <name> [--suite <file>] [--profile <group>] [--models <ids>] [--max-cost <cny>] [--progress] [--json]
   mf cluster [--suite <file>] [--threshold <0.5>] [--max-size <10>] [--out <file>] [--json]  # 建议挂载分组 + 同族候选
   mf exec --entry <file> (--input <json> | --input-json <json>) [--runtime node|python] [--python <cmd>] [--export run] [--allow-read <paths>] [--allow-write <paths>] [--allow-net <hosts>] [--timeout <ms>] [--warmup <n>] [--json]
   mf sandbox check [--json]   # probe available isolation engines on this machine
@@ -93,6 +96,8 @@ Eval options:
   --case <ids>          run only these case ids (comma-separated, repeatable)
   --profile <names>     score each mounted group separately ("all" or names from suite.profiles)
   --repeat <n>          run each case n times and score the majority; reports unstable cases (default 1)
+  --max-cost <cny>      stop after the estimated spend crosses this limit (exit 2; no partial scorecard)
+  --progress            print per-case progress to stderr (stdout stays clean for --json)
   --show-cases          print per-case results
   --json                print JSON instead of the table
   --out <file>          write scorecard (.md => markdown, otherwise JSON)
@@ -106,7 +111,7 @@ Eval options:
 Scan options:
   --root <dir>          repository to scan (default: .)
   --from <file>         import tools from a module (export default [...] or export const tools)
-  --probe               start stdio MCP servers and read tools/list
+  --probe               connect stdio or streamable-HTTP MCP servers and read tools/list
   --timeout <ms>        probe timeout per server (default: 5000)
   --out <file>          suite draft to write (default: mf.suite.json)
   --report <file>       scan report to write (default: mf.scan.json)
@@ -134,17 +139,31 @@ export async function runEval(options, io = console) {
   const { registry, adapters } = await resolveModels(options.models, options.modelsFile, io);
   const tools = toModelTools(suite.tools);
 
+  const budget = createBudget(options.maxCost ?? null);
   const runs = [];
-  for (const adapter of adapters) {
-    const evaluated = await evaluate(adapter, tools, cases, options.repeat ?? 1);
-    const summary = scoreModel({
-      model: adapter.id,
-      cases,
-      results: evaluated.results,
-      pricing: registry[adapter.id]?.pricing ?? null
-    });
-    if (evaluated.stability) summary.stability = evaluated.stability;
-    runs.push({ adapter, results: evaluated.results, summary });
+  try {
+    for (const adapter of adapters) {
+      const pricing = registry[adapter.id]?.pricing ?? null;
+      const evaluated = await evaluate(adapter, tools, cases, options.repeat ?? 1, {
+        pricing,
+        budget,
+        onProgress: makeProgress(options, io, adapter.id, cases.length)
+      });
+      const summary = scoreModel({
+        model: adapter.id,
+        cases,
+        results: evaluated.results,
+        pricing
+      });
+      if (evaluated.stability) summary.stability = evaluated.stability;
+      runs.push({ adapter, results: evaluated.results, summary });
+    }
+  } catch (error) {
+    if (error instanceof CostLimitError) {
+      io.error(costLimitMessage(error));
+      return 2;
+    }
+    throw error;
   }
   const summaries = runs.map((run) => run.summary);
 
@@ -173,16 +192,28 @@ export async function runEval(options, io = console) {
     };
   }
 
-  const profileReport = options.profiles
-    ? await evaluateProfiles({
+  let profileReport = null;
+  if (options.profiles) {
+    try {
+      profileReport = await evaluateProfiles({
         suite,
         cases,
         runs,
         registry,
         selection: options.profiles,
-        repeat: options.repeat ?? 1
-      })
-    : null;
+        repeat: options.repeat ?? 1,
+        budget,
+        progress: options.progress,
+        io
+      });
+    } catch (error) {
+      if (error instanceof CostLimitError) {
+        io.error(costLimitMessage(error));
+        return 2;
+      }
+      throw error;
+    }
+  }
 
   if (options.out) {
     const target = resolve(options.out);
@@ -357,14 +388,35 @@ export async function runAblate(options, io = console) {
   const fullTools = toModelTools(baseTools);
   const reducedTools = toModelTools(baseTools.filter((tool) => tool.name !== ablated.name));
   const models = [];
+  const budget = createBudget(options.maxCost ?? null);
 
-  for (const adapter of adapters) {
-    const pricing = registry[adapter.id]?.pricing ?? null;
-    const fullResults = (await evaluate(adapter, fullTools, baseCases)).results;
-    const full = scoreModel({ model: adapter.id, cases: baseCases, results: fullResults, pricing });
-    const reducedResults = (await evaluate(adapter, reducedTools, reducedCases)).results;
-    const reduced = scoreModel({ model: adapter.id, cases: reducedCases, results: reducedResults, pricing });
-    models.push(analyzeAblation(adapter.id, full, reduced));
+  try {
+    for (const adapter of adapters) {
+      const pricing = registry[adapter.id]?.pricing ?? null;
+      const fullResults = (
+        await evaluate(adapter, fullTools, baseCases, 1, {
+          pricing,
+          budget,
+          onProgress: makeProgress(options, io, `${adapter.id}/full`, baseCases.length)
+        })
+      ).results;
+      const full = scoreModel({ model: adapter.id, cases: baseCases, results: fullResults, pricing });
+      const reducedResults = (
+        await evaluate(adapter, reducedTools, reducedCases, 1, {
+          pricing,
+          budget,
+          onProgress: makeProgress(options, io, `${adapter.id}/ablated`, reducedCases.length)
+        })
+      ).results;
+      const reduced = scoreModel({ model: adapter.id, cases: reducedCases, results: reducedResults, pricing });
+      models.push(analyzeAblation(adapter.id, full, reduced));
+    }
+  } catch (error) {
+    if (error instanceof CostLimitError) {
+      io.error(costLimitMessage(error));
+      return 2;
+    }
+    throw error;
   }
 
   const payload = {
@@ -399,11 +451,10 @@ export async function runScan(options, io = console) {
 
   if (options.probe) {
     for (const server of [...report.mcpServers]) {
-      if (server.transport !== "stdio") {
-        report.probes.push({ server: server.name, ok: false, error: "http transport 暂不支持" });
-        continue;
-      }
-      const result = await probeStdioServer(server, { timeoutMs: options.timeoutMs });
+      const result =
+        server.transport === "stdio"
+          ? await probeStdioServer(server, { timeoutMs: options.timeoutMs })
+          : await probeHttpServer(server, { timeoutMs: options.timeoutMs });
       if (!result.ok) {
         report.probes.push({ server: server.name, ok: false, error: result.error, stderr: result.stderr });
         continue;
@@ -495,6 +546,35 @@ export async function runCoverage(options, io = console) {
   return 0;
 }
 
+export async function runAudit(options, io = console) {
+  const suite = await loadSuite(resolve(options.suite ?? DEFAULT_SUITE), {
+    allowEmptyCases: true,
+    allowEmptyTools: true
+  });
+  const report = auditSuite(suite);
+  const rules = {};
+  for (const finding of report.lint.findings) rules[finding.code] = (rules[finding.code] ?? 0) + 1;
+  noteUsage({
+    toolCount: report.suite.tools,
+    caseCount: report.suite.cases,
+    lintErrors: report.lint.counts.error,
+    lintWarns: report.lint.counts.warn,
+    lintInfos: report.lint.counts.info,
+    lintRules: rules,
+    clusters: report.clusters.clusters.length,
+    profileGroups: report.profiles?.groups.length ?? 0
+  });
+
+  if (options.out) {
+    const target = resolve(options.out);
+    const markdown = extname(target).toLowerCase() === ".md";
+    await writeFileAtomic(target, markdown ? `${formatAudit(report)}\n` : `${JSON.stringify(report, null, 2)}\n`);
+    io.log(`wrote ${target}`);
+  }
+  io.log(options.json ? JSON.stringify(report, null, 2) : formatAudit(report));
+  return 0;
+}
+
 export async function runSandboxCheck(options, io = console) {
   const report = checkSandbox();
   io.log(options.json ? JSON.stringify(report, null, 2) : formatSandboxReport(report));
@@ -719,25 +799,47 @@ function autoSelect(registry) {
     .map(([id]) => id);
 }
 
-async function evaluate(adapter, tools, cases, repeat = 1) {
+async function evaluate(
+  adapter,
+  tools,
+  cases,
+  repeat = 1,
+  { pricing = null, budget = null, onProgress = null } = {}
+) {
   const results = [];
   const unstable = [];
-  for (const testCase of cases) {
+  for (const [index, testCase] of cases.entries()) {
     const runs = [];
     for (let attempt = 0; attempt < repeat; attempt += 1) {
-      runs.push(await callOnce(adapter, tools, testCase));
+      const result = await callOnce(adapter, tools, testCase);
+      budget?.spend(costCnyOf(result.usage, pricing));
+      runs.push(result);
     }
     if (repeat === 1) {
       results.push(runs[0]);
-      continue;
+    } else {
+      const picked = pickRepresentative(runs);
+      results.push(picked.result);
+      if (picked.unstable) unstable.push({ id: testCase.id, outcomes: picked.outcomes });
     }
-    const picked = pickRepresentative(runs);
-    results.push(picked.result);
-    if (picked.unstable) unstable.push({ id: testCase.id, outcomes: picked.outcomes });
+    onProgress?.(index + 1, cases.length);
   }
   return { results, stability: repeat > 1 ? { runs: repeat, unstable } : null };
 }
 
+function makeProgress(options, io, label, total) {
+  if (!options.progress) return null;
+  const step = Math.max(1, Math.ceil(total / 10));
+  return (done, count) => {
+    if (done !== count && done % step !== 0) return;
+    io.error(`[${label}] ${done}/${count}`);
+  };
+}
+
+function costLimitMessage(error) {
+  return `${error.message}；已发出的请求已计费，本次不计分。减少 --case/--models/--repeat 或提高 --max-cost 后重跑。`;
+}
+
 async function callOnce(adapter, tools, testCase) {
   const messages = [
     { role: "system", content: SYSTEM_PROMPT },
@@ -782,7 +884,17 @@ function ablationScope(suite, names) {
   return { name: profile.name, tools: suite.tools.filter((tool) => mounted.has(tool.name)) };
 }
 
-async function evaluateProfiles({ suite, cases, runs, registry, selection, repeat = 1 }) {
+async function evaluateProfiles({
+  suite,
+  cases,
+  runs,
+  registry,
+  selection,
+  repeat = 1,
+  budget = null,
+  progress = false,
+  io = console
+}) {
   const profiles = normalizeProfiles(suite.profiles, suite.tools.map((tool) => tool.name));
   if (!profiles) {
     throw new Error(`suite ${suite.name} 没有 profiles；先运行 mf cluster --out <suite> 生成分组`);
@@ -812,7 +924,11 @@ async function evaluateProfiles({ suite, cases, runs, registry, selection, repea
       const profileCases = caseSets[index];
       if (profileCases.length === 0) continue;
       const mountedTools = suite.tools.filter((tool) => profile.tools.includes(tool.name));
-      const mountedEval = await evaluate(run.adapter, toModelTools(mountedTools), profileCases, repeat);
+      const mountedEval = await evaluate(run.adapter, toModelTools(mountedTools), profileCases, repeat, {
+        pricing,
+        budget,
+        onProgress: makeProgress({ progress }, io, `${run.adapter.id}/${profile.name}`, profileCases.length)
+      });
       const mounted = scoreModel({
         model: run.adapter.id,
         cases: profileCases,
@@ -875,6 +991,8 @@ function parseArgs(argv) {
     cases: null,
     profiles: null,
     repeat: 1,
+    maxCost: null,
+    progress: false,
     showCases: false,
     json: false,
     out: null,
@@ -932,6 +1050,13 @@ function parseArgs(argv) {
         options.repeat = raw;
         break;
       }
+      case "--max-cost": {
+        const raw = Number(value());
+        if (!Number.isFinite(raw) || raw <= 0) throw new Error("--max-cost expects a positive CNY amount");
+        options.maxCost = raw;
+        break;
+      }
+      case "--progress": options.progress = true; break;
       case "--show-cases": options.showCases = true; break;
       case "--json": options.json = true; break;
       case "--out": options.out = value(); break;
@@ -1104,7 +1229,7 @@ if (isMain) {
     }
   } else if (argv.length === 0 || argv[0] === "help" || argv[0] === "-h" || argv[0] === "--help") {
     process.stdout.write(USAGE);
-  } else if (!["eval", "scan", "case", "coverage", "lint", "check", "ablate", "cluster", "exec"].includes(argv[0])) {
+  } else if (!["eval", "scan", "case", "coverage", "audit", "lint", "check", "ablate", "cluster", "exec"].includes(argv[0])) {
     process.stderr.write(`unknown command "${argv[0]}"\n\n${USAGE}`);
     process.exit(1);
   } else if (argv[0] === "case" && argv[1] !== "add") {
@@ -1126,6 +1251,8 @@ if (isMain) {
             ? await runCaseAdd(options)
             : argv[0] === "coverage"
               ? await runCoverage(options)
+              : argv[0] === "audit"
+                ? await runAudit(options)
               : argv[0] === "lint"
                 ? await runLint(options)
                 : argv[0] === "check"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelfunction-cli",
-  "version": "0.5.0",
+  "version": "0.5.1",
   "type": "module",
   "license": "MIT",
   "description": "mf — function-calling accuracy scorecard for AI agents",