modelfunction-cli 0.5.5 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -5,7 +5,7 @@
5
5
  ## 安装
6
6
 
7
7
  ```bash
8
- npx -y modelfunction-cli@0.5.5 --version
8
+ npx -y modelfunction-cli@0.6.1 --version
9
9
  # 或
10
10
  npm i -D modelfunction-cli
11
11
  npx mf --version
@@ -69,18 +69,18 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
69
69
  --models-file my-models.json --models internal-ds
70
70
  ```
71
71
 
72
- `pricing` 可省(省了就不输出成本);要算成本请按自家价目填写,默认 registry 里的价格只是参考值。
72
+ `pricing` 可省(省了就不输出成本);要算成本请按自家价目填写,默认 registry 里的价格只是参考值。价格条目旁写 `"currency": "USD"`(默认 `CNY`)即可切换币种;token 用量不受此影响,始终是客观计量。
73
73
 
74
74
  ## 命令
75
75
 
76
76
  | 命令 | 作用 |
77
77
  | --- | --- |
78
78
  | `mf scan` | 扫描工具定义(function.json / OpenAPI / MCP 配置 / Python `@tool` / JS·TS 声明与 TypeBox·zod schema);`--probe` 连 stdio 或 streamable-HTTP MCP server(运行时结果覆盖静态推断;headers/query 不写入报告);`--from` 导入运行时工具数组;草稿按来源分组(`profiles`)并列出未覆盖的 skill |
79
- | `mf eval` | 跑评分卡;`--repeat` 多次取多数并报不稳定用例;`--fail-under` 阈值;`--baseline` + `--fail-on-regression` 回归门禁;`--profile` 按挂载组分别评分;`--max-cost` 按累计成本停机(退出 2);`--retry` 429/5xx 重试次数;`--min-interval` 请求最小间隔(压 RPM);`--progress` 逐用例进度(stderr) |
79
+ | `mf eval` | 跑评分卡;`--repeat` 多次取多数并报不稳定用例;`--fail-under` 阈值;`--baseline` + `--fail-on-regression` 回归门禁;`--profile` 按挂载组分别评分;`--max-cost` 按累计成本停机(退出 2,币种随 registry);`--max-tokens` 按累计 input+output token 停机(退出 2);`--retry` 429/5xx 重试次数;`--min-interval` 请求最小间隔(压 RPM);`--progress` 逐用例进度(stderr) |
80
80
  | `mf case add` | 写入用例(默认 `source=agent`、`reviewed=false`) |
81
81
  | `mf coverage` | 每工具用例数、参数覆盖、负例比例、疑似混淆对;有 `profiles` 时按组统计 |
82
82
  | `mf audit` | 一次静态审计:lint + 混淆族边界 + 用例覆盖 + 挂载检查,输出可转发的 Markdown/JSON;不调用模型 |
83
- | `mf lint` | 静态检查工具描述与参数 schema,以及 skills 的 name/description(不调用模型);重复规则自动合并成一条;有 `profiles` 时给出「这些组不要合并挂载」 |
83
+ | `mf lint` | 静态检查工具描述与参数 schema,以及 skills 的 name/description(不调用模型);0.6.1 起含「可选参数祈使句」与「悬空工具名引用」两条 info 规则;重复规则自动合并成一条;有 `profiles` 时给出「这些组不要合并挂载」 |
84
84
  | `mf check` | 安装门禁:一个或一包新能力 × 已装能力(`--manifest` / `--manifests` + `--installed`),纯静态、返回 pass/warn、不阻断;门禁语境只对混淆对/必填来源/结构性错误报 warn,其余降 info;也可 `import { check, checkMany } from "modelfunction-cli"` |
85
85
  | `mf compare` | 跨版本比较、干扰归因、混淆矩阵;两份报告都有分组时按组对比 |
86
86
  | `mf ablate` | 留一法:去掉某个工具再跑一遍,量化它是否干扰其他工具;`--profile` 限定在某个挂载组内 |
@@ -126,19 +126,19 @@ mf eval --suite mf.suite.json --models internal-ds \
126
126
  "mcpServers": {
127
127
  "modelfunction": {
128
128
  "command": "npx",
129
- "args": ["-y", "modelfunction-cli@0.5.5", "mcp"]
129
+ "args": ["-y", "modelfunction-cli@0.6.1", "mcp"]
130
130
  }
131
131
  }
132
132
  }
133
133
  ```
134
134
 
135
- 暴露:`scan_tools`、`list_suite`、`coverage`、`audit`、`lint`、`check`、`cluster`、`compare`、`add_cases`、`run_eval`(`run_eval` 支持 `profile`、`repeat`、`maxCost`、`retry`、`minInterval`、`progress`)。`lint`/`check`/`cluster`/`compare` 都是静态操作,不调用模型;`check` 就是安装门禁,Agent 可直接传入 manifest 与 installed。
135
+ 暴露:`scan_tools`、`list_suite`、`coverage`、`audit`、`lint`、`check`、`cluster`、`compare`、`add_cases`、`run_eval`(`run_eval` 支持 `profile`、`repeat`、`maxCost`、`maxTokens`、`retry`、`minInterval`、`progress`)。`lint`/`check`/`cluster`/`compare` 都是静态操作,不调用模型;`check` 就是安装门禁,Agent 可直接传入 manifest 与 installed。
136
136
 
137
137
  ## 契约与版本
138
138
 
139
139
  对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md),自 0.4.3 起冻结为 v1:v1 内只做加法,重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
140
140
 
141
- `0.5.5`(待 npm 发布):eval 新增 `--retry <n>`(429/5xx/网络错误指数退避,默认 2,共 3 次尝试;优先采用服务端 `Retry-After`)与 `--min-interval <ms>`(请求最小间隔,用于压住端点 RPM),评分卡新增 `retries` 计数;CI 模板 static 层补上 `cluster --json`,与 `action.yml` 对齐;新增 `templates/local/` 与 `docs/local-gate.md`(内网/非 GitHub 环境的 pre-push 静态门禁、变更门、定时回归)。`0.5.4` 含用例级 `excludeFromBase` / `excludeFromProfiles`、评分卡 `errorKind` / `errorsByKind` / `scoredTotal` / `selectionAccuracyExcludingErrors`、挂载对比错误不计 jammed、audit 分组 `neg%` 与分组 token 用量聚合。`0.5.3` 及更早的能力见下:凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查、门禁严重度收敛、静态规则性能(真实 100 能力清单 `checkMany` 471ms→40ms)、`installed` 形状校验与 `profiles` 平台形态兼容。示例统一使用 `@0.5.5`。
141
+ `0.6.1` 待 npm 发布:新增 `param-optional-imperative` 与 `desc-dangling-tool` 两条 info lint,并把「具名否定 / 泛化否定 / 可选参数祈使句 / 改名引用 / 话术范围」写进 `skills/mf-cases/SKILL.md`。`0.6.0` 已发布到 npm(latest):token 成为主口径(评分卡新增 `totalTokens`),模型注册表支持 `currency`(CNY/USD,默认 CNY),成本字段改为 `totalCost` / `avgCost` + `costCurrency`(`totalCostCny` / `avgCostCny` 仅在 CNY 时保留),新增 `mf eval --max-tokens <n>` 与 MCP `maxTokens`;`--fail-on-regression` 在基线模型缺失或本轮全部调用失败时改为 exit 2(无法判定 ≠ 无回归),自动选模型时在 stderr 打 note;本地门禁模板改用退出码判定、只在成功 eval 后更新 state,`.cmd` 固定 CRLF。`0.5.5` 含 eval `--retry <n>` / `--min-interval <ms>`、评分卡 `retries`、CI 模板 static 层 `cluster --json`、`templates/local/` 与 `docs/local-gate.md`(内网/非 GitHub 环境的 pre-push 静态门禁、变更门、定时回归)。`0.5.4` 含用例级 `excludeFromBase` / `excludeFromProfiles`、评分卡 `errorKind` / `errorsByKind` / `scoredTotal` / `selectionAccuracyExcludingErrors`、挂载对比错误不计 jammed、audit 分组 `neg%` 与分组 token 用量聚合。`0.5.3` 及更早的能力见下:凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查、门禁严重度收敛、静态规则性能(真实 100 能力清单 `checkMany` 471ms→40ms)、`installed` 形状校验与 `profiles` 平台形态兼容。示例统一使用 `@0.6.0`。
142
142
 
143
143
  ## 数据边界
144
144
 
@@ -167,7 +167,7 @@ mf feedback --out out.json # 导出脱敏反馈包(聚合值,不含原始
167
167
 
168
168
  ```bash
169
169
  # 1. 把 mf 挂成 MCP server
170
- claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.5 mcp
170
+ claude mcp add modelfunction -- npx -y modelfunction-cli@0.6.1 mcp
171
171
  ```
172
172
 
173
173
  ```text
package/SCHEMA.md CHANGED
@@ -96,8 +96,10 @@ skills 只做静态检查,不参与模型评测。
96
96
  "selectionAccuracy": 0.83, "selectionAccuracyExcludingErrors": 0.83,
97
97
  "argAccuracy": 0.9, "passRate": 0.75, "errors": 0, "errorsByKind": {},
98
98
  "scoredTotal": 12, "retries": 0, "p50LatencyMs": 900,
99
- "p95LatencyMs": 1800, "avgCostCny": 0.001, "totalCostCny": 0.012,
99
+ "p95LatencyMs": 1800, "avgCost": 0.001, "totalCost": 0.012,
100
+ "costCurrency": "CNY", "avgCostCny": 0.001, "totalCostCny": 0.012,
100
101
  "inputTokens": 1000, "cachedTokens": 0, "outputTokens": 200,
102
+ "totalTokens": 1200,
101
103
  "rows": [ { "id": "c1", "expectedTool": "...", "actualTool": "...",
102
104
  "selectionCorrect": true, "args": { "total": 1, "matched": 1, "failures": [] },
103
105
  "passed": true, "reviewed": true, "source": "manual",
@@ -110,8 +112,9 @@ skills 只做静态检查,不参与模型评测。
110
112
  - 正式指标只统计 `reviewed` 且未被 `excludeFromBase` 排除的行;未复核与排除结果单独给出。
111
113
  - `errors` 是调用层失败(超时、HTTP、网络)的数量,按 `errorsByKind` 分类;它们仍计入 `selectionAccuracy`(保守口径),同时给出排除错误后的 `selectionAccuracyExcludingErrors` 与可评分行数 `scoredTotal`。
112
114
  - `retries` 是本轮因 429/5xx/网络错误而重试的请求次数(模型级汇总;行级只在大于 0 时出现)。重试不改变判分:最终成功的请求按正常结果计分,最终失败的仍计入 `errors`。
115
+ - `inputTokens` / `cachedTokens` / `outputTokens` / `totalTokens` 是客观计量(`totalTokens = inputTokens + outputTokens`,缓存 token 已含在输入内)。成本需要模型注册表的 `pricing`;`currency` 取模型注册表条目(CNY/USD,默认 CNY)。读 `totalCost` / `avgCost` 时一并读 `costCurrency`;`totalCostCny` / `avgCostCny` 只在 CNY 时出现,是兼容别名。
113
116
  - `--repeat` 时模型对象带 `stability`(翻转用例与多次结果分布)。
114
- - `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`、`errors`;错误行不参与挂载对比(不会被记成 jammed),每个组的 `reviewed` 是可评分行数。
117
+ - `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`、`errors`,以及 `usage`(`inputTokens` / `cachedTokens` / `outputTokens` / `totalTokens` / `cost` / `costCurrency`;CNY 时另带 `costCny`);错误行不参与挂载对比(不会被记成 jammed),每个组的 `reviewed` 是可评分行数。
115
118
  - `fingerprint` 由工具与用例内容计算;`mf compare` 用它判断两份报告是否可比。
116
119
 
117
120
  ## 3. 扫描输出(`mf scan`)
@@ -138,8 +141,11 @@ report:`root`、`tools[]`(`name` / `description` / `parameters` / `origin` /
138
141
  ```
139
142
 
140
143
  - `kind: "skill"` 时用 `skill` 字段代替 `tool`。
141
- - `param` 出现在参数级发现上;`pair` 出现在工具对/技能对发现上。
144
+ - `param` 出现在参数级发现上;`pair` 出现在工具对/技能对发现上;`ref` / `similarTo` 出现在悬空工具名引用上。
142
145
  - `profilePairs` 只在套件含 `profiles` 时出现。
146
+ - 0.6.1 新增两条 advisory(均为 `info`,不影响 `check()` 的 pass/warn):
147
+ - `param-optional-imperative`:可选参数的描述用了祈使句;用户输入没提供该参数时可能抑制调用。
148
+ - `desc-dangling-tool`:描述引用了套件中不存在的工具名;`similarTo` 给出共享领域词的现有工具,通常来自改名后遗留的引用。
143
149
 
144
150
  ## 5. cluster JSON(`mf cluster --json`)
145
151
 
@@ -227,7 +233,7 @@ const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
227
233
  | `cluster` | `suite?`、`threshold?`、`maxSize?` |
228
234
  | `compare` | `before`、`after`(评分卡 JSON 路径) |
229
235
  | `add_cases` | `suite?`、`cases[]`、`source?`、`reviewed?`、`dryRun?` |
230
- | `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?`、`maxCost?`、`progress?` |
236
+ | `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?`、`maxCost?`、`maxTokens?`、`retry?`、`minInterval?`、`progress?` |
231
237
 
232
238
  返回值均为 JSON 文本。
233
239
 
@@ -237,7 +243,7 @@ const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
237
243
  | --- | --- |
238
244
  | 0 | 成功(`mf check` 的 warn 也是 0) |
239
245
  | 1 | 输入/用法错误;`mf compare` 判定不可直接比较 |
240
- | 2 | `mf eval --fail-under` 未达标、`--fail-on-regression` 发现回退,或 `--max-cost` 超限 |
246
+ | 2 | `mf eval --fail-under` 未达标、`--fail-on-regression` 发现回退或无法判定(基线模型缺失、本轮全部调用失败)、`--max-cost` / `--max-tokens` 超限 |
241
247
  | 3 | `mf exec` 被权限策略拒绝;`mf scan --json` 未找到工具或 skill |
242
248
  | 4 | `mf exec` 超时 |
243
249
 
@@ -1,11 +1,12 @@
1
1
  {
2
- "_note": "pricing 为人民币/百万 tokens 的参考值,运行前请按各家最新价目核对",
2
+ "_note": "pricing 为每百万 tokens 的参考价,currency 指定币种(CNY/USD,默认 CNY);运行前请按各家最新价目核对",
3
3
  "mock": { "type": "mock" },
4
4
  "deepseek": {
5
5
  "type": "openai",
6
6
  "baseUrl": "https://api.deepseek.com/v1",
7
7
  "model": "deepseek-chat",
8
8
  "apiKeyEnv": "DEEPSEEK_API_KEY",
9
+ "currency": "CNY",
9
10
  "pricing": { "input": 2, "output": 8 }
10
11
  },
11
12
  "dsv4-flash": {
@@ -13,6 +14,7 @@
13
14
  "baseUrl": "https://api.siliconflow.cn/v1",
14
15
  "model": "deepseek-ai/DeepSeek-V4-Flash",
15
16
  "apiKeyEnv": "SILICONFLOW_API_KEY",
17
+ "currency": "CNY",
16
18
  "pricing": { "input": 1.0, "cachedInput": 0.02, "output": 2.0 }
17
19
  },
18
20
  "qwen": {
@@ -20,6 +22,7 @@
20
22
  "baseUrl": "https://dashscope.aliyuncs.com/compatible-mode/v1",
21
23
  "model": "qwen-plus",
22
24
  "apiKeyEnv": "DASHSCOPE_API_KEY",
25
+ "currency": "CNY",
23
26
  "pricing": { "input": 0.8, "output": 2 }
24
27
  },
25
28
  "glm": {
@@ -27,6 +30,7 @@
27
30
  "baseUrl": "https://open.bigmodel.cn/api/paas/v4",
28
31
  "model": "glm-4-air",
29
32
  "apiKeyEnv": "ZHIPU_API_KEY",
33
+ "currency": "CNY",
30
34
  "pricing": { "input": 0.5, "output": 0.5 }
31
35
  },
32
36
  "openai": {
@@ -34,6 +38,7 @@
34
38
  "baseUrl": "https://api.openai.com/v1",
35
39
  "model": "gpt-4o-mini",
36
40
  "apiKeyEnv": "OPENAI_API_KEY",
41
+ "currency": "CNY",
37
42
  "pricing": { "input": 1.1, "output": 4.3 }
38
43
  }
39
44
  }
package/lib/budget.js CHANGED
@@ -1,9 +1,9 @@
1
- // Running cost guard for evaluations. Pricing is known per model, but token
2
- // usage is only known after a call returns, so the limit is enforced after
3
- // each call: the run stops as soon as the budget is crossed.
1
+ // Running guards for evaluations. Token and cost usage are only known after a
2
+ // call returns, so limits are enforced after each call: the run stops as soon
3
+ // as either limit is crossed.
4
4
  export class CostLimitError extends Error {
5
5
  constructor({ limit, spent, calls }) {
6
- super(`cost limit reached: spent ¥${spent.toFixed(4)} of ¥${limit} after ${calls} call(s)`);
6
+ super(`cost limit reached: spent ${spent.toFixed(4)} of ${limit} after ${calls} call(s)`);
7
7
  this.name = "CostLimitError";
8
8
  this.limit = limit;
9
9
  this.spent = spent;
@@ -11,20 +11,42 @@ export class CostLimitError extends Error {
11
11
  }
12
12
  }
13
13
 
14
- export function createBudget(limit = null) {
14
+ export class TokenLimitError extends Error {
15
+ constructor({ limit, spent, calls }) {
16
+ super(`token limit reached: used ${spent} of ${limit} tokens after ${calls} call(s)`);
17
+ this.name = "TokenLimitError";
18
+ this.limit = limit;
19
+ this.spent = spent;
20
+ this.calls = calls;
21
+ }
22
+ }
23
+
24
+ export function createBudget(limits = null) {
25
+ const { maxCost = null, maxTokens = null } =
26
+ typeof limits === "object" && limits !== null
27
+ ? limits
28
+ : { maxCost: limits, maxTokens: null };
15
29
  let spent = 0;
30
+ let tokens = 0;
16
31
  let calls = 0;
17
32
  return {
18
33
  get spent() {
19
34
  return spent;
20
35
  },
36
+ get tokens() {
37
+ return tokens;
38
+ },
21
39
  get calls() {
22
40
  return calls;
23
41
  },
24
- spend(cost) {
42
+ spend(usage) {
43
+ const { cost, tokens: usedTokens } =
44
+ typeof usage === "number" ? { cost: usage, tokens: 0 } : (usage ?? {});
25
45
  calls += 1;
26
46
  if (Number.isFinite(cost) && cost > 0) spent += cost;
27
- if (limit !== null && spent > limit) throw new CostLimitError({ limit, spent, calls });
47
+ if (Number.isFinite(usedTokens) && usedTokens > 0) tokens += usedTokens;
48
+ if (maxCost !== null && spent > maxCost) throw new CostLimitError({ limit: maxCost, spent, calls });
49
+ if (maxTokens !== null && tokens > maxTokens) throw new TokenLimitError({ limit: maxTokens, spent: tokens, calls });
28
50
  }
29
51
  };
30
52
  }
package/lib/compare.js CHANGED
@@ -182,6 +182,46 @@ function compareProfiles(before, after) {
182
182
  };
183
183
  }
184
184
 
185
+ export function regressionGate(comparison, summaries = []) {
186
+ const missing = comparison.models.filter((model) => model.reason === "模型不在新报告中");
187
+ if (missing.length > 0) {
188
+ return {
189
+ status: "fail",
190
+ message: `baseline model(s) not in this report: ${missing.map((model) => model.model).join(", ")};无法判定回归(门禁失败)`
191
+ };
192
+ }
193
+
194
+ const errored = summaries.filter(
195
+ (summary) => summary.reviewedTotal > 0 && summary.errors === summary.reviewedTotal
196
+ );
197
+ if (errored.length > 0) {
198
+ const detail = errored
199
+ .map((summary) => {
200
+ const kinds = summary.errorsByKind && Object.keys(summary.errorsByKind).length
201
+ ? ` ${JSON.stringify(summary.errorsByKind)}`
202
+ : "";
203
+ return `${summary.model} (${summary.errors}/${summary.reviewedTotal}${kinds})`;
204
+ })
205
+ .join(", ");
206
+ return {
207
+ status: "fail",
208
+ message: `all calls failed for: ${detail};基础设施故障,无法判定回归(门禁失败)`
209
+ };
210
+ }
211
+
212
+ const comparable = comparison.models.filter((model) => model.comparable && model.reviewed > 0);
213
+ if (comparable.length === 0) {
214
+ return { status: "skip", message: "定义或用例已变更,无法判定回归(门禁跳过)" };
215
+ }
216
+
217
+ return {
218
+ status: "ok",
219
+ regressions: comparable.flatMap((model) =>
220
+ model.regressions.map((item) => `${model.model}/${item.id}`)
221
+ )
222
+ };
223
+ }
224
+
185
225
  export function formatCompare(report) {
186
226
  const lines = [];
187
227
  if (!report.beforeHash || !report.afterHash) {
package/lib/lint.js CHANGED
@@ -56,6 +56,70 @@ function idStrength(name) {
56
56
  const BOUNDARY_HINT = /(不适用|不要用于|不用于|不能用于|而非|区别于|not for|do not use)/i;
57
57
  // Chinese descriptions are compared as bigrams, so mid-0.3s similarity is noise.
58
58
  const NO_BOUNDARY_SIMILARITY = 0.5;
59
+ // Two advisory checks from real description experiments: imperatives aimed at
60
+ // optional params can suppress the call entirely, and renamed tools leave stale
61
+ // names in sibling descriptions. Both are info-only; they depend on wording.
62
+ const OPTIONAL_HINT =
63
+ /(可选|optional|非必填|如有|如果需要|可提供|if [^.\n]{0,30}\b(provided|available|present|omitted)\b|when [^.\n]{0,30}\b(provided|available|present)\b)/i;
64
+ const IMPERATIVE_EN = /\b(pass|provide|supply|include|set|specify|add|send|put)\b/i;
65
+ const IMPERATIVE_ZH = /(传入|传|提供|带上|设置|指定|填写|附上|注明)/;
66
+ const IMPERATIVE_START_EN = /^(pass|provide|supply|include|set|specify|add|send|put)\b/i;
67
+ const IMPERATIVE_START_ZH = /^(传入|传|提供|带上|设置|指定|填写|附上|注明)/;
68
+ const BACKTICK_ID = /`([a-z][a-z0-9_-]{2,})`/g;
69
+ const SNAKE_ID = /\b([a-z][a-z0-9]*(?:_[a-z0-9]+)+)\b/g;
70
+ const NON_TOOL_SUFFIX =
71
+ /_(id|ids|key|token|path|name|url|uri|type|schema|count|size|format|length|text|json|yaml|yml|md|txt|html|htm|pdf|csv)$/;
72
+ const NON_TOOL_NAMES = new Set([
73
+ "api_key", "access_token", "base_url", "content_type", "status_code", "error_code",
74
+ "http_status", "input_schema", "output_schema", "json_schema", "file_path", "file_name",
75
+ "user_id", "session_id", "tenant_id", "request_id", "trace_id", "tool_name", "mcp_server"
76
+ ]);
77
+ const ACTION_TOKENS = new Set([
78
+ "get", "set", "list", "read", "write", "search", "create", "delete", "update", "send",
79
+ "fetch", "query", "run", "exec", "execute", "open", "close", "add", "remove", "check",
80
+ "scan", "parse", "score", "match", "generate", "render", "save", "load", "find", "lookup",
81
+ "call", "invoke", "publish", "edit", "rewrite", "analyze", "analyse", "summarize",
82
+ "summarise", "translate", "tool", "tools", "info", "data", "result", "results", "item",
83
+ "items", "skill", "skills", "mcp"
84
+ ]);
85
+
86
+ function escapeRegExp(value) {
87
+ return String(value).replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
88
+ }
89
+
90
+ function mentionsImperatively(text, name) {
91
+ if (!text || !name) return false;
92
+ const index = text.indexOf(name);
93
+ if (index === -1) return false;
94
+ const around = text.slice(Math.max(0, index - 40), Math.min(text.length, index + name.length + 40));
95
+ if (OPTIONAL_HINT.test(around)) return false;
96
+ return IMPERATIVE_EN.test(around) || IMPERATIVE_ZH.test(around);
97
+ }
98
+
99
+ function startsImperatively(text) {
100
+ const trimmed = String(text ?? "").trim();
101
+ if (!trimmed || OPTIONAL_HINT.test(trimmed)) return false;
102
+ return IMPERATIVE_START_EN.test(trimmed) || IMPERATIVE_START_ZH.test(trimmed);
103
+ }
104
+
105
+ function referencedNames(text) {
106
+ const found = new Set();
107
+ for (const match of String(text ?? "").matchAll(BACKTICK_ID)) found.add(match[1]);
108
+ for (const match of String(text ?? "").matchAll(SNAKE_ID)) found.add(match[1]);
109
+ return found;
110
+ }
111
+
112
+ function sharedDomainToken(candidate, toolNames) {
113
+ const candidateTokens = new Set(nameTokens(candidate).filter((token) => !ACTION_TOKENS.has(token)));
114
+ if (candidateTokens.size === 0) return null;
115
+ for (const known of toolNames) {
116
+ const knownTokens = new Set(nameTokens(known).filter((token) => !ACTION_TOKENS.has(token)));
117
+ for (const token of candidateTokens) {
118
+ if (knownTokens.has(token)) return { name: known, token };
119
+ }
120
+ }
121
+ return null;
122
+ }
59
123
 
60
124
  // `pairScope` (a Set of names) is for incremental gates: only emit pair
61
125
  // findings that touch one of the new capabilities. Installed × installed pairs
@@ -63,6 +127,12 @@ const NO_BOUNDARY_SIMILARITY = 0.5;
63
127
  export function lintSuite(suite, { pairScope } = {}) {
64
128
  const inScope = (pair) => !pairScope || pairScope.has(pair.a) || pairScope.has(pair.b);
65
129
  const toolByName = new Map(suite.tools.map((tool) => [tool.name, tool]));
130
+ const toolNamesList = suite.tools.map((tool) => tool.name);
131
+ const knownNames = new Set([
132
+ ...toolNamesList,
133
+ ...(suite.skills ?? []).map((skill) => skill.name),
134
+ ...suite.tools.flatMap((tool) => Object.keys(tool.parameters?.properties ?? {}))
135
+ ]);
66
136
  const findings = [];
67
137
  const add = (severity, code, tool, message, extra = {}) =>
68
138
  findings.push({ severity, code, kind: "tool", tool, message, ...extra });
@@ -207,6 +277,41 @@ export function lintSuite(suite, { pairScope } = {}) {
207
277
  add("warn", "param-array-no-items", tool.name, `参数 ${name} 是数组但未定义 items`, { param: name });
208
278
  }
209
279
  }
280
+
281
+ for (const name of names) {
282
+ if (required.includes(name)) continue;
283
+ const schema = properties[name] ?? {};
284
+ if (mentionsImperatively(description, name) || startsImperatively(schema.description)) {
285
+ add(
286
+ "info",
287
+ "param-optional-imperative",
288
+ tool.name,
289
+ `可选参数 ${name} 用了祈使句提示;用户输入没提供它时,模型可能因此干脆不调用`,
290
+ {
291
+ param: name,
292
+ suggestion: `改成条件式(例如「如有${name}可传入」),或把 ${name} 改为必填`
293
+ }
294
+ );
295
+ }
296
+ }
297
+
298
+ for (const ref of referencedNames(description)) {
299
+ if (ref === tool.name || knownNames.has(ref) || NON_TOOL_NAMES.has(ref) || NON_TOOL_SUFFIX.test(ref)) {
300
+ continue;
301
+ }
302
+ const similar = sharedDomainToken(ref, toolNamesList);
303
+ const backticked = new RegExp(`\`${escapeRegExp(ref)}\``).test(description);
304
+ if (!similar && !backticked) continue;
305
+ add(
306
+ "info",
307
+ "desc-dangling-tool",
308
+ tool.name,
309
+ similar
310
+ ? `description 提到 ${ref},但套件里没有这个工具;最接近的是 ${similar.name}(共享 ${similar.token})。若是改名,请同步描述引用`
311
+ : `description 用代码格式提到 ${ref},但套件里没有这个工具;确认不是改名后遗留的悬空引用`,
312
+ { ref, ...(similar ? { similarTo: similar.name } : {}) }
313
+ );
314
+ }
210
315
  }
211
316
 
212
317
  for (const pair of suspectedPairs(suite.tools)) {
package/lib/mcp-server.js CHANGED
@@ -247,13 +247,14 @@ export async function startMcpServer() {
247
247
  .optional()
248
248
  .describe("每条用例重复次数,取多数结果;建议正式基线用 3"),
249
249
  failUnder: z.number().optional().describe("选择准确率阈值,低于则退出码为 2"),
250
- maxCost: z.number().positive().optional().describe("累计估算成本上限(CNY),超限退出 2 且不返回部分评分卡"),
250
+ maxCost: z.number().positive().optional().describe("累计估算成本上限(币种见模型注册表 currency),超限退出 2 且不返回部分评分卡"),
251
+ maxTokens: z.number().int().positive().optional().describe("累计 input+output token 上限,超限退出 2 且不返回部分评分卡"),
251
252
  retry: z.number().int().min(0).max(10).optional().describe("单次请求遇到 429/5xx/网络错误的重试次数(默认 2,共 3 次尝试)"),
252
253
  minInterval: z.number().int().min(0).max(60000).optional().describe("两次模型请求的最小间隔(毫秒),用于压住端点 RPM"),
253
254
  progress: z.boolean().optional().describe("把逐用例进度写到 stderr(stdout 仍为纯 JSON)")
254
255
  }
255
256
  },
256
- async ({ suite, models, cases, profile, repeat, failUnder, maxCost, retry, minInterval, progress }) => {
257
+ async ({ suite, models, cases, profile, repeat, failUnder, maxCost, maxTokens, retry, minInterval, progress }) => {
257
258
  const args = ["eval", "--json", "--suite", suitePath(suite)];
258
259
  if (models) args.push("--models", models);
259
260
  if (cases) args.push("--case", cases);
@@ -261,6 +262,7 @@ export async function startMcpServer() {
261
262
  if (repeat !== undefined) args.push("--repeat", String(repeat));
262
263
  if (failUnder !== undefined) args.push("--fail-under", String(failUnder));
263
264
  if (maxCost !== undefined) args.push("--max-cost", String(maxCost));
265
+ if (maxTokens !== undefined) args.push("--max-tokens", String(maxTokens));
264
266
  if (retry !== undefined) args.push("--retry", String(retry));
265
267
  if (minInterval !== undefined) args.push("--min-interval", String(minInterval));
266
268
  if (progress) args.push("--progress");
package/lib/profile.js CHANGED
@@ -146,7 +146,16 @@ export function modelTotals(entries) {
146
146
  let errors = 0;
147
147
  let mountedCorrect = 0;
148
148
  let allCorrect = 0;
149
- const usage = { inputTokens: 0, cachedTokens: 0, outputTokens: 0, costCny: 0, estimated: false };
149
+ const usage = {
150
+ inputTokens: 0,
151
+ cachedTokens: 0,
152
+ outputTokens: 0,
153
+ totalTokens: 0,
154
+ cost: 0,
155
+ costCurrency: null,
156
+ costCny: 0,
157
+ estimated: false
158
+ };
150
159
  for (const entry of entries) {
151
160
  reviewed += entry.reviewed;
152
161
  errors += entry.errors ?? 0;
@@ -155,9 +164,13 @@ export function modelTotals(entries) {
155
164
  usage.inputTokens += entry.usage?.inputTokens ?? 0;
156
165
  usage.cachedTokens += entry.usage?.cachedTokens ?? 0;
157
166
  usage.outputTokens += entry.usage?.outputTokens ?? 0;
167
+ usage.totalTokens += entry.usage?.totalTokens ?? 0;
168
+ usage.cost += entry.usage?.cost ?? 0;
169
+ usage.costCurrency ||= entry.usage?.costCurrency ?? null;
158
170
  usage.costCny += entry.usage?.costCny ?? 0;
159
171
  usage.estimated ||= Boolean(entry.usage?.estimated);
160
172
  }
173
+ usage.costCurrency ??= "CNY";
161
174
  if (reviewed === 0) {
162
175
  return {
163
176
  reviewed: 0,
package/lib/report.js CHANGED
@@ -1,10 +1,13 @@
1
1
  import { pct } from "./score.js";
2
2
  import { suiteFingerprint } from "./fingerprint.js";
3
3
 
4
- function money(value, estimated) {
5
- if (value === 0) return "0";
4
+ const CURRENCY_SYMBOLS = { CNY: "¥", USD: "$" };
5
+
6
+ function money(value, estimated, currency = "CNY") {
7
+ if (!value) return "0";
6
8
  const text = value >= 0.01 ? value.toFixed(3) : value.toFixed(4);
7
- return `${estimated ? "~" : ""}${text}`;
9
+ const symbol = CURRENCY_SYMBOLS[currency] ?? `${currency} `;
10
+ return `${estimated ? "~" : ""}${symbol}${text}`;
8
11
  }
9
12
 
10
13
  function isEstimated(summary) {
@@ -32,7 +35,7 @@ export function formatCaseLines(summary, cases) {
32
35
  }
33
36
 
34
37
  export function formatTable(summaries) {
35
- const headers = ["model", "select", "args", "pass", "p50ms", "p95ms", "cny/call", "errors"];
38
+ const headers = ["model", "select", "args", "pass", "p50ms", "p95ms", "cost/call", "errors"];
36
39
  const rows = summaries.map((summary) => [
37
40
  summary.model,
38
41
  pct(summary.selectionAccuracy),
@@ -40,7 +43,7 @@ export function formatTable(summaries) {
40
43
  pct(summary.passRate),
41
44
  String(summary.p50LatencyMs),
42
45
  String(summary.p95LatencyMs),
43
- money(summary.avgCostCny, isEstimated(summary)),
46
+ money(summary.avgCost ?? summary.avgCostCny, isEstimated(summary), summary.costCurrency ?? "CNY"),
44
47
  String(summary.errors)
45
48
  ]);
46
49
 
@@ -71,8 +74,8 @@ export function formatMarkdown(suite, cases, summaries) {
71
74
 
72
75
  const withExec = summaries.some((summary) => summary.exec);
73
76
  const headers = withExec
74
- ? ["model", "select", "args", "pass", "exec", "run p50 ms", "max RSS MB", "p50 ms", "p95 ms", "CNY/call", "errors"]
75
- : ["model", "select", "args", "pass", "p50 ms", "p95 ms", "CNY/call", "errors"];
77
+ ? ["model", "select", "args", "pass", "exec", "run p50 ms", "max RSS MB", "p50 ms", "p95 ms", "cost/call", "errors"]
78
+ : ["model", "select", "args", "pass", "p50 ms", "p95 ms", "cost/call", "errors"];
76
79
  lines.push(`| ${headers.join(" | ")} |`, `| ${headers.map(() => "---").join(" | ")} |`);
77
80
 
78
81
  for (const summary of summaries) {
@@ -90,7 +93,7 @@ export function formatMarkdown(suite, cases, summaries) {
90
93
  cells.push(
91
94
  String(summary.p50LatencyMs),
92
95
  String(summary.p95LatencyMs),
93
- money(summary.avgCostCny, isEstimated(summary)),
96
+ money(summary.avgCost ?? summary.avgCostCny, isEstimated(summary), summary.costCurrency ?? "CNY"),
94
97
  String(summary.errors)
95
98
  );
96
99
  lines.push(`| ${cells.join(" | ")} |`);
package/lib/score.js CHANGED
@@ -76,7 +76,11 @@ export function pickRepresentative(runs) {
76
76
  };
77
77
  }
78
78
 
79
- export function costCnyOf(usage, pricing) {
79
+ export function tokensOf(usage) {
80
+ return (usage?.input ?? 0) + (usage?.output ?? 0);
81
+ }
82
+
83
+ export function costOf(usage, pricing) {
80
84
  if (!pricing) return 0;
81
85
  const input = usage?.input ?? 0;
82
86
  const cached = Math.min(input, usage?.cached ?? 0);
@@ -89,7 +93,7 @@ export function costCnyOf(usage, pricing) {
89
93
  );
90
94
  }
91
95
 
92
- export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
96
+ export function scoreModel({ model, cases, results, pricing, scope = "base", currency = "CNY" }) {
93
97
  const rows = cases.map((testCase, index) => {
94
98
  const result = results[index];
95
99
  const actualTool = result.ok ? (result.toolCalls?.[0]?.name ?? null) : null;
@@ -144,7 +148,9 @@ export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
144
148
  inputTokens,
145
149
  rows.reduce((sum, row) => sum + (row.usage.cached ?? 0), 0)
146
150
  );
147
- const costCny = rows.reduce((sum, row) => sum + costCnyOf(row.usage, pricing), 0);
151
+ const totalCost = rows.reduce((sum, row) => sum + costOf(row.usage, pricing), 0);
152
+ const totalTokens = inputTokens + outputTokens;
153
+ const costCurrency = String(currency || "CNY").trim().toUpperCase();
148
154
  const reviewedTotal = reviewedRows.length;
149
155
 
150
156
  return {
@@ -166,11 +172,16 @@ export function scoreModel({ model, cases, results, pricing, scope = "base" }) {
166
172
  retries: rows.reduce((sum, row) => sum + (row.retries ?? 0), 0),
167
173
  p50LatencyMs: percentile(latencies, 0.5),
168
174
  p95LatencyMs: percentile(latencies, 0.95),
169
- avgCostCny: total ? costCny / total : 0,
170
- totalCostCny: costCny,
175
+ avgCost: total ? totalCost / total : 0,
176
+ totalCost,
177
+ costCurrency,
178
+ ...(costCurrency === "CNY"
179
+ ? { avgCostCny: total ? totalCost / total : 0, totalCostCny: totalCost }
180
+ : {}),
171
181
  inputTokens,
172
182
  cachedTokens,
173
183
  outputTokens,
184
+ totalTokens,
174
185
  rows
175
186
  };
176
187
  }
package/mf.js CHANGED
@@ -4,7 +4,7 @@ import { mkdir, readFile } from "node:fs/promises";
4
4
  import { basename, dirname, extname, relative, resolve } from "node:path";
5
5
  import { fileURLToPath } from "node:url";
6
6
  import { loadSuite, toModelTools } from "./lib/suite.js";
7
- import { scoreModel, pickRepresentative, pct, costCnyOf } from "./lib/score.js";
7
+ import { scoreModel, pickRepresentative, pct, costOf, tokensOf } from "./lib/score.js";
8
8
  import { formatTable, formatCaseLines, formatMarkdown } from "./lib/report.js";
9
9
  import { scanProject, buildSuiteDraft, formatScanSummary, addTools, redactReport } from "./lib/scan.js";
10
10
  import { probeStdioServer, probeHttpServer } from "./lib/mcp-probe.js";
@@ -16,7 +16,7 @@ import { auditSuite, formatAudit } from "./lib/audit.js";
16
16
  import { suiteFingerprint, hashText, hashOf } from "./lib/fingerprint.js";
17
17
  import { runExecutionChecks } from "./lib/eval-exec.js";
18
18
  import { checkSandbox, formatSandboxReport } from "./lib/sandbox-check.js";
19
- import { compareScorecards, formatCompare } from "./lib/compare.js";
19
+ import { compareScorecards, formatCompare, regressionGate } from "./lib/compare.js";
20
20
  import { analyzeAblation, formatAblation } from "./lib/ablate.js";
21
21
  import { clusterTools, formatClusters } from "./lib/cluster.js";
22
22
  import {
@@ -31,7 +31,7 @@ import {
31
31
  formatProfileMarkdown
32
32
  } from "./lib/profile.js";
33
33
  import { runExec } from "./lib/exec.js";
34
- import { createBudget, CostLimitError } from "./lib/budget.js";
34
+ import { createBudget, CostLimitError, TokenLimitError } from "./lib/budget.js";
35
35
  import { writeFileAtomic } from "./lib/fsutil.js";
36
36
  import {
37
37
  applyModelCredentials,
@@ -77,7 +77,7 @@ Usage:
77
77
  mf lint [--suite <file>] [--json] # 静态检查 tools 与 skills,不调用模型
78
78
  mf check (--manifest <file> | --manifests <file>) [--installed <file>] [--json] # 安装门禁:新能力 × 已装能力(--manifests 支持一包)
79
79
  mf compare <before.json> <after.json> [--json]
80
- mf ablate --tool <name> [--suite <file>] [--profile <group>] [--models <ids>] [--max-cost <cny>] [--progress] [--json]
80
+ mf ablate --tool <name> [--suite <file>] [--profile <group>] [--models <ids>] [--max-cost <amount>] [--max-tokens <n>] [--progress] [--json]
81
81
  mf cluster [--suite <file>] [--threshold <0.5>] [--max-size <10>] [--out <file>] [--json] # 建议挂载分组 + 同族候选
82
82
  mf exec --entry <file> (--input <json> | --input-json <json>) [--runtime node|python] [--python <cmd>] [--export run] [--allow-read <paths>] [--allow-write <paths>] [--allow-net <hosts>] [--timeout <ms>] [--warmup <n>] [--json]
83
83
  mf sandbox check [--json] # probe available isolation engines on this machine
@@ -96,7 +96,8 @@ Eval options:
96
96
  --case <ids> run only these case ids (comma-separated, repeatable)
97
97
  --profile <names> score each mounted group separately ("all" or names from suite.profiles)
98
98
  --repeat <n> run each case n times and score the majority; reports unstable cases (default 1)
99
- --max-cost <cny> stop after the estimated spend crosses this limit (exit 2; no partial scorecard)
99
+ --max-cost <amount> stop after the estimated spend crosses this limit (exit 2; no partial scorecard)
100
+ --max-tokens <n> stop after cumulative input+output tokens cross this limit (exit 2)
100
101
  --retry <n> retries per request on HTTP 429/5xx and network errors (default 2; 3 attempts total)
101
102
  --min-interval <ms> minimum spacing between model requests; use it to stay under an endpoint RPM
102
103
  --progress print per-case progress to stderr (stdout stays clean for --json)
@@ -144,11 +145,13 @@ export async function runEval(options, io = console) {
144
145
  });
145
146
  const tools = toModelTools(suite.tools);
146
147
 
147
- const budget = createBudget(options.maxCost ?? null);
148
+ const budget = createBudget({ maxCost: options.maxCost ?? null, maxTokens: options.maxTokens ?? null });
148
149
  const runs = [];
149
150
  try {
150
151
  for (const adapter of adapters) {
151
- const pricing = registry[adapter.id]?.pricing ?? null;
152
+ const config = registry[adapter.id];
153
+ const pricing = config?.pricing ?? null;
154
+ const currency = currencyOf(adapter.id, config);
152
155
  const evaluated = await evaluate(adapter, tools, cases, options.repeat ?? 1, {
153
156
  pricing,
154
157
  budget,
@@ -159,14 +162,15 @@ export async function runEval(options, io = console) {
159
162
  cases,
160
163
  results: evaluated.results,
161
164
  pricing,
162
- scope: "base"
165
+ scope: "base",
166
+ currency
163
167
  });
164
168
  if (evaluated.stability) summary.stability = evaluated.stability;
165
169
  runs.push({ adapter, results: evaluated.results, summary });
166
170
  }
167
171
  } catch (error) {
168
- if (error instanceof CostLimitError) {
169
- io.error(costLimitMessage(error));
172
+ if (error instanceof CostLimitError || error instanceof TokenLimitError) {
173
+ io.error(budgetLimitMessage(error));
170
174
  return 2;
171
175
  }
172
176
  throw error;
@@ -213,8 +217,8 @@ export async function runEval(options, io = console) {
213
217
  io
214
218
  });
215
219
  } catch (error) {
216
- if (error instanceof CostLimitError) {
217
- io.error(costLimitMessage(error));
220
+ if (error instanceof CostLimitError || error instanceof TokenLimitError) {
221
+ io.error(budgetLimitMessage(error));
218
222
  return 2;
219
223
  }
220
224
  throw error;
@@ -311,13 +315,15 @@ export async function runEval(options, io = console) {
311
315
  io.log("");
312
316
  io.log(formatCompare(comparison));
313
317
  }
314
- const comparable = comparison.models.filter((model) => model.comparable && model.reviewed > 0);
315
- if (comparable.length === 0) {
316
- io.error("定义或用例已变更,无法判定回归(门禁跳过)");
318
+ const gate = regressionGate(comparison, summaries);
319
+ if (gate.status === "fail") {
320
+ io.error(gate.message);
321
+ return 2;
322
+ }
323
+ if (gate.status === "skip") {
324
+ io.error(gate.message);
317
325
  } else {
318
- regressions = comparable.flatMap((model) =>
319
- model.regressions.map((item) => `${model.model}/${item.id}`)
320
- );
326
+ regressions = gate.regressions;
321
327
  if (regressions.length > 0) io.error(`regressions: ${regressions.join(", ")}`);
322
328
  }
323
329
  }
@@ -406,11 +412,13 @@ export async function runAblate(options, io = console) {
406
412
  const fullTools = toModelTools(baseTools);
407
413
  const reducedTools = toModelTools(baseTools.filter((tool) => tool.name !== ablated.name));
408
414
  const models = [];
409
- const budget = createBudget(options.maxCost ?? null);
415
+ const budget = createBudget({ maxCost: options.maxCost ?? null, maxTokens: options.maxTokens ?? null });
410
416
 
411
417
  try {
412
418
  for (const adapter of adapters) {
413
- const pricing = registry[adapter.id]?.pricing ?? null;
419
+ const config = registry[adapter.id];
420
+ const pricing = config?.pricing ?? null;
421
+ const currency = currencyOf(adapter.id, config);
414
422
  const fullResults = (
415
423
  await evaluate(adapter, fullTools, baseCases, 1, {
416
424
  pricing,
@@ -418,7 +426,7 @@ export async function runAblate(options, io = console) {
418
426
  onProgress: makeProgress(options, io, `${adapter.id}/full`, baseCases.length)
419
427
  })
420
428
  ).results;
421
- const full = scoreModel({ model: adapter.id, cases: baseCases, results: fullResults, pricing });
429
+ const full = scoreModel({ model: adapter.id, cases: baseCases, results: fullResults, pricing, currency });
422
430
  const reducedResults = (
423
431
  await evaluate(adapter, reducedTools, reducedCases, 1, {
424
432
  pricing,
@@ -426,12 +434,12 @@ export async function runAblate(options, io = console) {
426
434
  onProgress: makeProgress(options, io, `${adapter.id}/ablated`, reducedCases.length)
427
435
  })
428
436
  ).results;
429
- const reduced = scoreModel({ model: adapter.id, cases: reducedCases, results: reducedResults, pricing });
437
+ const reduced = scoreModel({ model: adapter.id, cases: reducedCases, results: reducedResults, pricing, currency });
430
438
  models.push(analyzeAblation(adapter.id, full, reduced));
431
439
  }
432
440
  } catch (error) {
433
- if (error instanceof CostLimitError) {
434
- io.error(costLimitMessage(error));
441
+ if (error instanceof CostLimitError || error instanceof TokenLimitError) {
442
+ io.error(budgetLimitMessage(error));
435
443
  return 2;
436
444
  }
437
445
  throw error;
@@ -786,12 +794,26 @@ function selectCases(cases, ids) {
786
794
  return cases.filter((testCase) => wanted.has(testCase.id));
787
795
  }
788
796
 
797
+ function currencyOf(id, config) {
798
+ const code = String(config?.currency ?? "CNY").trim().toUpperCase();
799
+ if (!/^[A-Z]{3}$/.test(code)) {
800
+ throw new Error(`model "${id}": currency must be a 3-letter code (got "${config.currency}")`);
801
+ }
802
+ return code;
803
+ }
804
+
789
805
  async function resolveModels(ids, modelsFile, io, network = {}) {
790
806
  const registry = JSON.parse(await readFile(resolve(modelsFile), "utf8"));
791
807
  // Env wins; the credentials file only fills what is missing.
792
808
  await applyModelCredentials(registry);
793
809
  const explicit = Boolean(ids && ids.length > 0);
794
810
  const chosen = explicit ? ids : autoSelect(registry);
811
+ if (!explicit && chosen.length > 0) {
812
+ const detail = chosen
813
+ .map((id) => `${id} (${registry[id]?.apiKeyEnv ?? "?"})`)
814
+ .join(", ");
815
+ io.error(`note: auto-selected ${detail} from ${modelsFile}; pass --models or --models-file to pin the registry`);
816
+ }
795
817
  const adapters = [];
796
818
 
797
819
  for (const id of chosen) {
@@ -851,7 +873,7 @@ async function evaluate(
851
873
  const runs = [];
852
874
  for (let attempt = 0; attempt < repeat; attempt += 1) {
853
875
  const result = await callOnce(adapter, tools, testCase);
854
- budget?.spend(costCnyOf(result.usage, pricing));
876
+ budget?.spend({ cost: costOf(result.usage, pricing), tokens: tokensOf(result.usage) });
855
877
  runs.push(result);
856
878
  }
857
879
  if (repeat === 1) {
@@ -875,8 +897,9 @@ function makeProgress(options, io, label, total) {
875
897
  };
876
898
  }
877
899
 
878
- function costLimitMessage(error) {
879
- return `${error.message};已发出的请求已计费,本次不计分。减少 --case/--models/--repeat 或提高 --max-cost 后重跑。`;
900
+ function budgetLimitMessage(error) {
901
+ const hint = error instanceof TokenLimitError ? "--max-tokens" : "--max-cost";
902
+ return `${error.message};已发出的请求已计费,本次不计分。减少 --case/--models/--repeat 或提高 ${hint} 后重跑。`;
880
903
  }
881
904
 
882
905
  async function callOnce(adapter, tools, testCase) {
@@ -965,7 +988,9 @@ async function evaluateProfiles({
965
988
 
966
989
  const models = [];
967
990
  for (const run of runs) {
968
- const pricing = registry[run.adapter.id]?.pricing ?? null;
991
+ const config = registry[run.adapter.id];
992
+ const pricing = config?.pricing ?? null;
993
+ const currency = currencyOf(run.adapter.id, config);
969
994
  const entries = [];
970
995
  for (const [index, profile] of selected.entries()) {
971
996
  const profileCases = caseSets[index];
@@ -981,21 +1006,26 @@ async function evaluateProfiles({
981
1006
  cases: profileCases,
982
1007
  results: mountedEval.results,
983
1008
  pricing,
984
- scope: "profiles"
1009
+ scope: "profiles",
1010
+ currency
985
1011
  });
986
1012
  const allInOne = scoreModel({
987
1013
  model: run.adapter.id,
988
1014
  cases: profileCases,
989
1015
  results: profileCases.map((testCase) => run.results[indexById.get(testCase.id)]),
990
1016
  pricing,
991
- scope: "profiles"
1017
+ scope: "profiles",
1018
+ currency
992
1019
  });
993
1020
  const entry = analyzeMount(profile, mounted, allInOne);
994
1021
  entry.usage = {
995
1022
  inputTokens: mounted.inputTokens,
996
1023
  cachedTokens: mounted.cachedTokens,
997
1024
  outputTokens: mounted.outputTokens,
998
- costCny: mounted.totalCostCny,
1025
+ totalTokens: mounted.totalTokens,
1026
+ cost: mounted.totalCost,
1027
+ costCurrency: mounted.costCurrency,
1028
+ ...(mounted.costCurrency === "CNY" ? { costCny: mounted.totalCostCny } : {}),
999
1029
  estimated: mounted.rows.some((row) => row.usage?.estimated)
1000
1030
  };
1001
1031
  if (mountedEval.stability) entry.stability = mountedEval.stability;
@@ -1048,6 +1078,7 @@ function parseArgs(argv) {
1048
1078
  profiles: null,
1049
1079
  repeat: 1,
1050
1080
  maxCost: null,
1081
+ maxTokens: null,
1051
1082
  retries: 2,
1052
1083
  minIntervalMs: 0,
1053
1084
  progress: false,
@@ -1111,10 +1142,16 @@ function parseArgs(argv) {
1111
1142
  }
1112
1143
  case "--max-cost": {
1113
1144
  const raw = Number(value());
1114
- if (!Number.isFinite(raw) || raw <= 0) throw new Error("--max-cost expects a positive CNY amount");
1145
+ if (!Number.isFinite(raw) || raw <= 0) throw new Error("--max-cost expects a positive amount");
1115
1146
  options.maxCost = raw;
1116
1147
  break;
1117
1148
  }
1149
+ case "--max-tokens": {
1150
+ const raw = Number(value());
1151
+ if (!Number.isInteger(raw) || raw <= 0) throw new Error("--max-tokens expects a positive integer");
1152
+ options.maxTokens = raw;
1153
+ break;
1154
+ }
1118
1155
  case "--retry": {
1119
1156
  const raw = Number(value());
1120
1157
  if (!Number.isInteger(raw) || raw < 0 || raw > 10) throw new Error("--retry expects an integer between 0 and 10");
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "modelfunction-cli",
3
- "version": "0.5.5",
3
+ "version": "0.6.1",
4
4
  "type": "module",
5
5
  "license": "MIT",
6
6
  "description": "mf — function-calling accuracy scorecard for AI agents",
@@ -31,6 +31,13 @@ description: 为 ModelFunction 评测套件撰写 function-calling 用例。当
31
31
  - `{ "$exists": true|false }`(只关心有没有)
32
32
  6. **不要用被测模型生成用例**,也不要让被测模型参与出题。
33
33
 
34
+ # 描述边界写法(实测)
35
+
36
+ - 排除具体竞争者时**点名**:`do NOT call file_read` 有效;泛化否定(`no separate file reads`)几乎压不住任何工具。
37
+ - 可选参数不要用祈使句(“传一个 title”);用户没给该参数时,模型可能干脆不调用。改成条件式:“如有 title 可传入”。
38
+ - 改名工具后,检查所有 description 是否还引用旧名;悬空引用会继续误导模型。
39
+ - 用例话术必须提交范围(公网 / 本地 / 会话历史);“上周那个记录”这类歧义无法靠描述修复,只能改话术。
40
+
34
41
  # 输出格式
35
42
 
36
43
  ```json