modelfunction-cli 0.5.1 → 0.5.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -5,7 +5,7 @@
5
5
  ## 安装
6
6
 
7
7
  ```bash
8
- npx -y modelfunction-cli@0.5.0 --version
8
+ npx -y modelfunction-cli@0.5.1 --version
9
9
  # 或
10
10
  npm i -D modelfunction-cli
11
11
  npx mf --version
@@ -81,7 +81,7 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
81
81
  | `mf coverage` | 每工具用例数、参数覆盖、负例比例、疑似混淆对;有 `profiles` 时按组统计 |
82
82
  | `mf audit` | 一次静态审计:lint + 混淆族边界 + 用例覆盖 + 挂载检查,输出可转发的 Markdown/JSON;不调用模型 |
83
83
  | `mf lint` | 静态检查工具描述与参数 schema,以及 skills 的 name/description(不调用模型);重复规则自动合并成一条;有 `profiles` 时给出「这些组不要合并挂载」 |
84
- | `mf check` | 安装门禁:一个新能力 × 已装能力(`--manifest` + `--installed`),纯静态、返回 pass/warn、不阻断;也可 `import { check } from "modelfunction-cli"` |
84
+ | `mf check` | 安装门禁:一个或一包新能力 × 已装能力(`--manifest` / `--manifests` + `--installed`),纯静态、返回 pass/warn、不阻断;门禁语境只对混淆对/必填来源/结构性错误报 warn,其余降 info;也可 `import { check, checkMany } from "modelfunction-cli"` |
85
85
  | `mf compare` | 跨版本比较、干扰归因、混淆矩阵;两份报告都有分组时按组对比 |
86
86
  | `mf ablate` | 留一法:去掉某个工具再跑一遍,量化它是否干扰其他工具;`--profile` 限定在某个挂载组内 |
87
87
  | `mf cluster` | 聚类工具,建议挂载分组;参数形状拆散的同一族(如 `sessions_*`)列为「同族候选」;并输出「族内边界清单」(每族会互抢的对 + 每个工具可用的区分词);`--out` 把分组写成套件的 `profiles` |
@@ -126,19 +126,19 @@ mf eval --suite mf.suite.json --models internal-ds \
126
126
  "mcpServers": {
127
127
  "modelfunction": {
128
128
  "command": "npx",
129
- "args": ["-y", "modelfunction-cli@0.5.0", "mcp"]
129
+ "args": ["-y", "modelfunction-cli@0.5.1", "mcp"]
130
130
  }
131
131
  }
132
132
  }
133
133
  ```
134
134
 
135
- 暴露:`scan_tools`、`list_suite`、`coverage`、`add_cases`、`run_eval`(`run_eval` 支持 `profile` 与 `repeat` 参数:传 `all` 或组名按挂载组评分,`repeat` 取多数结果)。
135
+ 暴露:`scan_tools`、`list_suite`、`coverage`、`audit`、`lint`、`check`、`cluster`、`compare`、`add_cases`、`run_eval`(`run_eval` 支持 `profile`、`repeat`、`maxCost`、`progress`)。`lint`/`check`/`cluster`/`compare` 都是静态操作,不调用模型;`check` 就是安装门禁,Agent 可直接传入 manifest 与 installed。
136
136
 
137
137
  ## 契约与版本
138
138
 
139
139
  对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md),自 0.4.3 起冻结为 v1:v1 内只做加法,重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
140
140
 
141
- npm 已发布 `0.5.0`(凭证与本地反馈渠道);`0.5.1` 待发布,合并开发改动:usage `ci` 标记与 local/ci 统计、eval `--progress`/`--max-cost`、`mf audit` 组合审计、`scan --probe` 支持 streamable HTTP。示例固定 `@0.5.0`,发布后统一升级。
141
+ `0.5.2` 已发布到 npm:含凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查,门禁 warn 只保留混淆对/必填来源/结构性错误。本仓库 `0.5.3` 待发布:静态规则性能(真实 100 能力清单 `checkMany` 471ms→40ms,550 能力 9.6s→2.1s,输出与 golden 一致),以及 `installed` 形状校验(接受 `mf scan --json` 输出,无法识别时报错而不是静默返回假 pass)和 `profiles` 兼容平台形态(对象或 `[{agentType, tools, skills}]` 数组)。示例统一使用 `@0.5.2`。
142
142
 
143
143
  ## 数据边界
144
144
 
@@ -167,7 +167,7 @@ mf feedback --out out.json # 导出脱敏反馈包(聚合值,不含原始
167
167
 
168
168
  ```bash
169
169
  # 1. 把 mf 挂成 MCP server
170
- claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.0 mcp
170
+ claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.1 mcp
171
171
  ```
172
172
 
173
173
  ```text
package/SCHEMA.md CHANGED
@@ -73,7 +73,7 @@ skills 只做静态检查,不参与模型评测。
73
73
 
74
74
  ### profiles
75
75
 
76
- 组名 → 工具名数组。所有工具名必须存在;组可重叠。`mf eval --profile` 与 `mf ablate --profile` 依赖它。
76
+ 组名 → 工具名数组(`{ "group": ["tool", ...] }`)。也接受平台常见的两种形态:对象 `{ "0": { agentType, tools, skills } }` 与数组 `[{ agentType, tools, skills }]`,组名依次取 `name` → `agentType` → 键/下标;`skills` 字段目前被忽略(profiles 只用于工具挂载与 eval 分组)。所有工具名必须存在;组可重叠,组名不可重复。`mf eval --profile` 与 `mf ablate --profile` 依赖它。
77
77
 
78
78
  ### execution
79
79
 
@@ -112,7 +112,7 @@ skills 只做静态检查,不参与模型评测。
112
112
 
113
113
  report:`root`、`tools[]`(`name` / `description` / `parameters` / `origin` / `confidence` / `source` / `note?` / `descriptionSource?`)、`skills[]`、`mcpServers[]`(env 只保留 `envKeys`,headers 只保留 `headerKeys`;URL 去掉 userinfo / query / hash)、`probes[]`(`server` / `ok` / `toolCount?` / `tools?` / `error?`)、`sources[]`、`warnings[]`。`--probe` 支持 stdio 与 streamable HTTP 两种传输。
114
114
 
115
- `scan --json` 在未找到任何工具时退出码为 3。
115
+ `scan --json` 在工具与 skill 都未找到时退出码为 3(只有 skill 也算有结果,退出 0)。
116
116
 
117
117
  ## 4. lint JSON
118
118
 
@@ -159,7 +159,7 @@ report:`root`、`tools[]`(`name` / `description` / `parameters` / `origin` /
159
159
  import { check } from "modelfunction-cli";
160
160
  ```
161
161
 
162
- 输入 `{ manifest, installed }`:`manifest` 是一个能力(`name` 必填,`description` 可选,`parameters` / `inputSchema` / `input` 任一为 JSON Schema;无 schema 视为 skill)。`installed` 可以是能力数组,或 `{ tools: [], skills: [] }`。
162
+ 输入 `{ manifest, installed }`:`manifest` 是一个能力(`name` 必填,`description` 可选,`parameters` / `inputSchema` / `input` 任一为 JSON Schema;无 schema 视为 skill)。`installed` 可以是能力数组、`{ tools: [], skills: [] }`,或直接传 `mf scan --json` 的输出(能力在 `report` 下)。空对象视为没有已装能力;其他无法识别的形状会抛错,避免静默返回假的 `pass`。
163
163
 
164
164
  返回:
165
165
 
@@ -174,8 +174,37 @@ import { check } from "modelfunction-cli";
174
174
  ```
175
175
 
176
176
  - `verdict` 只有 `pass` / `warn`,**没有 fail**;质量判定不阻断安装。
177
- - 唯一抛错场景:`manifest.name` 缺失或为空。
177
+ - **严重度策略(门禁语境)**:只有会改变模型行为的规则保持原等级——`desc-missing`、`param-type-missing`(error),`required-unsourced`、`pair-no-boundary`、`skill-pair-no-boundary`(warn);描述、命名、必填数量等质量规则降为 `info`,避免成熟工具集每次安装都告警。`mf lint` 不受影响,仍输出原始严重度。
178
+ - 抛错场景:`manifest.name` 缺失或为空;`installed` 不是上述三种形状之一(空对象仍视为无已装能力)。
178
179
  - 同步、无网络、无文件系统、无状态。类型声明见 `lib/check.d.ts`。
180
+ - 性能(0.5.3):每个能力只分词一次,且只比较涉及新能力的对。真实 100 能力清单(53 新 + 47 已装)约 40–50ms,47 工具对 53 skill 约 25ms;550 个高度同质的合成最坏情况约 2s。
181
+
182
+ 批量(一次安装一包):
183
+
184
+ ```js
185
+ import { checkMany } from "modelfunction-cli";
186
+ const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
187
+ ```
188
+
189
+ 返回 `{ verdict, summary, items, results[] }`:
190
+
191
+ ```json
192
+ {
193
+ "verdict": "warn",
194
+ "summary": { "error": 0, "warn": 1, "info": 0 },
195
+ "items": 2,
196
+ "results": [
197
+ { "name": "gitcrawl", "verdict": "warn", "summary": { "error": 0, "warn": 1, "info": 0 },
198
+ "findings": [ { "severity": "warn", "code": "skill-pair-no-boundary", "subject": "gitcrawl",
199
+ "with": "tag-duplicate-prs-issues", "message": "...", "suggestion": "..." } ],
200
+ "family": ["gitcrawl", "tag-duplicate-prs-issues"] },
201
+ { "name": "tag-duplicate-prs-issues", "verdict": "warn", "summary": { "error": 0, "warn": 1, "info": 0 },
202
+ "findings": [ /* 同一对,subject 为自身 */ ], "family": ["gitcrawl", "tag-duplicate-prs-issues"] }
203
+ ]
204
+ }
205
+ ```
206
+
207
+ `summary` 按 subject/with 去重(同一对只计一次);`results[]` 保留与该能力相关的全部发现(pair 会出现在双方)。`manifests` 为空、某项缺 name、名称重复时抛错。CLI:`mf check --manifests pack.json [--installed installed.json] [--json]`。
179
208
 
180
209
  ## 7. MCP 工具
181
210
 
@@ -185,6 +214,10 @@ import { check } from "modelfunction-cli";
185
214
  | `list_suite` | `suite?` |
186
215
  | `coverage` | `suite?` |
187
216
  | `audit` | `suite?` |
217
+ | `lint` | `suite?` |
218
+ | `check` | `manifest`(对象)、`installed?`(数组或 `{tools,skills}`) |
219
+ | `cluster` | `suite?`、`threshold?`、`maxSize?` |
220
+ | `compare` | `before`、`after`(评分卡 JSON 路径) |
188
221
  | `add_cases` | `suite?`、`cases[]`、`source?`、`reviewed?`、`dryRun?` |
189
222
  | `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?`、`maxCost?`、`progress?` |
190
223
 
@@ -197,7 +230,7 @@ import { check } from "modelfunction-cli";
197
230
  | 0 | 成功(`mf check` 的 warn 也是 0) |
198
231
  | 1 | 输入/用法错误;`mf compare` 判定不可直接比较 |
199
232
  | 2 | `mf eval --fail-under` 未达标、`--fail-on-regression` 发现回退,或 `--max-cost` 超限 |
200
- | 3 | `mf exec` 被权限策略拒绝;`mf scan --json` 未找到工具 |
233
+ | 3 | `mf exec` 被权限策略拒绝;`mf scan --json` 未找到工具或 skill |
201
234
  | 4 | `mf exec` 超时 |
202
235
 
203
236
  ## 9. 本地文件(0.5.0 起)
package/lib/cases.js CHANGED
@@ -207,38 +207,52 @@ export function formatCoverage(report) {
207
207
  return lines.join("\n");
208
208
  }
209
209
 
210
- export function pairSimilarity(x, y, { extraStop } = {}) {
211
- const a = tokens(`${x.name} ${x.description}`, extraStop);
212
- const b = tokens(`${y.name} ${y.description}`, extraStop);
213
- const union = new Set([...a, ...b]);
214
- let lexical = 0;
215
- if (union.size > 0) {
216
- let shared = 0;
217
- for (const token of union) if (a.has(token) && b.has(token)) shared += 1;
218
- lexical = shared / union.size;
219
- }
220
- const structural = structuralScore(x, y);
210
+ // O(n²) callers prepare each capability once; the pair loop then only compares
211
+ // sets. Tokenizing inside the loop made a 550-capability gate take ~14s.
212
+ export function prepareItem(item, { extraStop } = {}) {
213
+ const names = nameTokens(item.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
214
+ return {
215
+ name: item.name,
216
+ tokens: tokens(`${item.name} ${item.description}`, extraStop),
217
+ nameTokens: names,
218
+ nameTokenSet: new Set(names),
219
+ params: paramNames(item),
220
+ required: new Set(item.parameters?.required ?? [])
221
+ };
222
+ }
223
+
224
+ export function pairSimilarityPrepared(a, b) {
225
+ const shared = intersectionSize(a.tokens, b.tokens);
226
+ const union = a.tokens.size + b.tokens.size - shared;
227
+ const lexical = union > 0 ? shared / union : 0;
228
+ const structural = intersects(a.nameTokenSet, b.nameTokenSet) ? structuralScorePrepared(a, b) : 0;
221
229
  return {
222
230
  similarity: Math.max(lexical, structural),
223
231
  basis: structural > lexical ? "structural" : "lexical"
224
232
  };
225
233
  }
226
234
 
235
+ export function pairSimilarity(x, y, { extraStop } = {}) {
236
+ return pairSimilarityPrepared(prepareItem(x, { extraStop }), prepareItem(y, { extraStop }));
237
+ }
238
+
227
239
  // Clustering wants "same family" (alarm_query / alarm_ack belong to one agent),
228
240
  // which is looser than "confusable" (lint's question). Shared name token plus
229
241
  // any parameter overlap is enough to belong together; two shared tokens are
230
242
  // enough on their own (fault_alarm_query / fault_alarm_ack take different ids).
231
243
  // One shared token with disjoint params is not: cross-domain tools often share
232
244
  // a bare verb (query / list) and nothing else.
233
- export function familyScore(x, y) {
234
- const left = nameTokens(x.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
235
- const right = nameTokens(y.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
236
- const shared = left.filter((token) => right.includes(token));
237
- if (shared.length === 0) return 0;
238
- const paramOverlap = jaccard(paramNames(x), paramNames(y));
245
+ export function familyScorePrepared(x, y) {
246
+ const shared = intersectionSize(x.nameTokenSet, y.nameTokenSet);
247
+ if (shared === 0) return 0;
248
+ const paramOverlap = jaccard(x.params, y.params);
239
249
  if (paramOverlap > 0) return 0.8;
240
- if (shared.length >= 2) return 0.7;
241
- return paramNames(x).size === 0 && paramNames(y).size === 0 ? 0.6 : 0.2;
250
+ if (shared >= 2) return 0.7;
251
+ return x.params.size === 0 && y.params.size === 0 ? 0.6 : 0.2;
252
+ }
253
+
254
+ export function familyScore(x, y) {
255
+ return familyScorePrepared(prepareItem(x), prepareItem(y));
242
256
  }
243
257
 
244
258
  // Turn a confusable pair into an authoring instruction. We do not invent the
@@ -280,12 +294,13 @@ export function boundaryHint(x, y) {
280
294
  }
281
295
 
282
296
  export function suspectedPairs(tools, { threshold = PAIR_SIMILARITY, extraStop } = {}) {
297
+ const prepared = tools.map((tool) => prepareItem(tool, { extraStop }));
283
298
  const pairs = [];
284
- for (let i = 0; i < tools.length; i += 1) {
285
- for (let j = i + 1; j < tools.length; j += 1) {
286
- const { similarity, basis } = pairSimilarity(tools[i], tools[j], { extraStop });
299
+ for (let i = 0; i < prepared.length; i += 1) {
300
+ for (let j = i + 1; j < prepared.length; j += 1) {
301
+ const { similarity, basis } = pairSimilarityPrepared(prepared[i], prepared[j]);
287
302
  if (similarity >= threshold) {
288
- pairs.push({ a: tools[i].name, b: tools[j].name, similarity, basis });
303
+ pairs.push({ a: prepared[i].name, b: prepared[j].name, similarity, basis });
289
304
  }
290
305
  }
291
306
  }
@@ -296,20 +311,13 @@ export function suspectedPairs(tools, { threshold = PAIR_SIMILARITY, extraStop }
296
311
  // parameter shape confuse the model even when the wording differs, which the
297
312
  // lexical score alone misses. Name order is unreliable: `tavily_search` leads
298
313
  // with the vendor, `search_docs` with the action.
299
- function structuralScore(x, y) {
300
- const left = nameTokens(x.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
301
- const right = nameTokens(y.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
302
- const sharedName = left.some((token) => right.includes(token));
314
+ function structuralScorePrepared(x, y) {
315
+ const sharedName = x.nameTokens.some((token) => y.nameTokens.includes(token));
303
316
  if (!sharedName) return 0;
304
- const params = paramNames(x);
305
- const otherParams = paramNames(y);
306
- if (params.size === 0 && otherParams.size === 0) return 0;
307
- const paramOverlap = jaccard(params, otherParams);
317
+ if (x.params.size === 0 && y.params.size === 0) return 0;
318
+ const paramOverlap = jaccard(x.params, y.params);
308
319
  if (paramOverlap < 0.34) return 0;
309
- const requiredOverlap = jaccard(
310
- new Set(x.parameters?.required ?? []),
311
- new Set(y.parameters?.required ?? [])
312
- );
320
+ const requiredOverlap = jaccard(x.required, y.required);
313
321
  return 0.5 + 0.5 * (0.6 * paramOverlap + 0.4 * requiredOverlap);
314
322
  }
315
323
 
@@ -340,10 +348,26 @@ function paramNames(tool) {
340
348
 
341
349
  function jaccard(a, b) {
342
350
  if (a.size === 0 && b.size === 0) return 1;
343
- const union = new Set([...a, ...b]);
351
+ const shared = intersectionSize(a, b);
352
+ return shared / (a.size + b.size - shared);
353
+ }
354
+
355
+ // Iterate the smaller set; avoids allocating a union Set for every pair in the
356
+ // O(n²) loops.
357
+ function intersectionSize(a, b) {
358
+ const small = a.size <= b.size ? a : b;
359
+ const large = small === a ? b : a;
344
360
  let shared = 0;
345
- for (const item of union) if (a.has(item) && b.has(item)) shared += 1;
346
- return shared / union.size;
361
+ for (const item of small) if (large.has(item)) shared += 1;
362
+ return shared;
363
+ }
364
+
365
+ function intersects(a, b) {
366
+ if (a.size === 0 || b.size === 0) return false;
367
+ const small = a.size <= b.size ? a : b;
368
+ const large = small === a ? b : a;
369
+ for (const item of small) if (large.has(item)) return true;
370
+ return false;
347
371
  }
348
372
 
349
373
  function tokens(text, extraStop) {
package/lib/check.d.ts CHANGED
@@ -17,12 +17,22 @@ export interface InstalledCapabilities {
17
17
  skills?: Capability[];
18
18
  }
19
19
 
20
+ /** `mf scan --json` output; its capabilities live under `report`. */
21
+ export interface ScanReport {
22
+ report: InstalledCapabilities;
23
+ }
24
+
20
25
  export interface CheckInput {
21
26
  manifest: Capability;
22
- installed?: Capability[] | InstalledCapabilities;
27
+ installed?: Capability[] | InstalledCapabilities | ScanReport;
23
28
  }
24
29
 
25
30
  export interface CheckFinding {
31
+ /**
32
+ * Install-gate view: only behavior-changing findings keep warn/error
33
+ * (selection conflicts, unsourced required params, schema errors).
34
+ * Description and naming rules are reported as info so mature tools pass.
35
+ */
26
36
  severity: Severity;
27
37
  code: string;
28
38
  subject: string;
@@ -41,11 +51,39 @@ export interface CheckResult {
41
51
  family?: string[];
42
52
  }
43
53
 
54
+ export interface CheckManyInput {
55
+ manifests: Capability[];
56
+ installed?: Capability[] | InstalledCapabilities | ScanReport;
57
+ }
58
+
59
+ export interface CheckItemResult extends CheckResult {
60
+ name: string;
61
+ }
62
+
63
+ export interface CheckManyResult {
64
+ verdict: Verdict;
65
+ /** Deduplicated by subject/with; per-item summaries are in results[]. */
66
+ summary: { error: number; warn: number; info: number };
67
+ items: number;
68
+ results: CheckItemResult[];
69
+ }
70
+
44
71
  /**
45
72
  * Static install gate: judge one new capability against installed ones.
46
73
  * No model call, no network, no file system, no state.
47
- * @throws only when `manifest.name` is missing or empty.
74
+ * @throws when `manifest.name` is missing or empty, or `installed` is neither
75
+ * an array, `{ tools, skills }`, nor an `mf scan --json` report.
48
76
  */
49
77
  export declare function check(input: CheckInput): CheckResult;
50
78
 
79
+ /**
80
+ * Batch install gate: judge a pack of new capabilities against installed ones
81
+ * and against each other.
82
+ * @throws when manifests is empty, an item lacks a name, names are duplicated,
83
+ * or `installed` is not an accepted shape.
84
+ */
85
+ export declare function checkMany(input: CheckManyInput): CheckManyResult;
86
+
51
87
  export declare function formatCheck(result: CheckResult): string;
88
+
89
+ export declare function formatCheckMany(result: CheckManyResult): string;
package/lib/check.js CHANGED
@@ -1,4 +1,4 @@
1
- // Install-gate entry point: judge one new capability against what is already
1
+ // Install-gate entry point: judge new capabilities against what is already
2
2
  // installed. Static only — no model call, no network, no file system, no state.
3
3
  // Quality findings are advisory: the verdict is pass/warn, never "block".
4
4
  import { lintSuite } from "./lint.js";
@@ -6,32 +6,115 @@ import { lintSuite } from "./lint.js";
6
6
  export function check({ manifest, installed = [] } = {}) {
7
7
  const target = normalizeEntry(manifest);
8
8
  if (!target) throw new Error('check: manifest 需要非空 "name"');
9
- const pool = [];
10
- if (Array.isArray(installed)) pool.push(...installed);
11
- else if (installed && typeof installed === "object") {
12
- pool.push(...(installed.tools ?? []), ...(installed.skills ?? []));
9
+ const { results } = evaluateTargets([target], installed);
10
+ const [result] = results;
11
+ return {
12
+ verdict: result.verdict,
13
+ summary: result.summary,
14
+ findings: result.findings,
15
+ ...(result.family ? { family: result.family } : {})
16
+ };
17
+ }
18
+
19
+ // Installers usually apply a pack at once: new skills must be checked against
20
+ // installed ones AND against each other.
21
+ export function checkMany({ manifests, installed = [] } = {}) {
22
+ if (!Array.isArray(manifests) || manifests.length === 0) {
23
+ throw new Error("checkMany: manifests 需要非空数组");
24
+ }
25
+ const targets = manifests.map((entry) => normalizeEntry(entry));
26
+ if (targets.some((target) => target === null)) {
27
+ throw new Error('checkMany: 每个 manifest 都需要非空 "name"');
13
28
  }
29
+ const names = new Set();
30
+ for (const target of targets) {
31
+ if (names.has(target.name)) throw new Error(`checkMany: duplicate name "${target.name}"`);
32
+ names.add(target.name);
33
+ }
34
+ const { results } = evaluateTargets(targets, installed);
35
+ return {
36
+ verdict: results.some((item) => item.verdict === "warn") ? "warn" : "pass",
37
+ summary: uniqueSummary(results),
38
+ items: results.length,
39
+ results
40
+ };
41
+ }
42
+
43
+ // A gate that silently ignores a mis-shaped `installed` returns a false pass —
44
+ // the worst failure mode for an install check. Accept the array and
45
+ // {tools,skills} forms, plus `mf scan --json` output (wrapped in `report`);
46
+ // anything else non-empty is an input error.
47
+ function installedPool(installed) {
48
+ if (installed == null) return [];
49
+ if (Array.isArray(installed)) return installed;
50
+ if (typeof installed !== "object") {
51
+ throw new Error("installed 需要能力数组、{ tools, skills } 或 mf scan --json 的 report");
52
+ }
53
+ if (Object.keys(installed).length === 0) return [];
54
+ const source =
55
+ installed.report && typeof installed.report === "object" ? installed.report : installed;
56
+ const tools = source.tools ?? [];
57
+ const skills = source.skills ?? [];
58
+ if (!Array.isArray(tools) || !Array.isArray(skills)) {
59
+ throw new Error("installed.tools / installed.skills 必须是数组");
60
+ }
61
+ if (!Array.isArray(source.tools) && !Array.isArray(source.skills)) {
62
+ throw new Error("installed 需要能力数组、{ tools, skills } 或 mf scan --json 的 report");
63
+ }
64
+ return [...tools, ...skills];
65
+ }
66
+
67
+ function evaluateTargets(targets, installed) {
68
+ const pool = installedPool(installed);
14
69
  const entries = pool.map(normalizeEntry).filter(Boolean);
15
70
 
16
71
  const tools = [];
17
72
  const skills = [];
18
73
  for (const entry of entries) (isToolLike(entry) ? tools : skills).push(entry);
19
- (isToolLike(target) ? tools : skills).push(target);
74
+ for (const target of targets) (isToolLike(target) ? tools : skills).push(target);
20
75
 
21
- const report = lintSuite({ tools, skills });
22
- const findings = report.findings
23
- .filter((finding) => touches(finding, target.name))
24
- .map((finding) => shape(finding, target.name));
25
- const summary = { error: 0, warn: 0, info: 0 };
26
- for (const finding of findings) summary[finding.severity] += 1;
27
- const family = familyOf(report.findings.filter((finding) => finding.pair), target.name);
76
+ // The gate only reports findings touching the new capabilities; skipping
77
+ // installed × installed pairs keeps a 500-capability pool linear-ish.
78
+ const pairScope = new Set(targets.map((target) => target.name));
79
+ const report = lintSuite({ tools, skills }, { pairScope });
80
+ const pairFindings = report.findings.filter((finding) => finding.pair);
81
+ const results = targets.map((target) => {
82
+ const findings = report.findings
83
+ .filter((finding) => touches(finding, target.name))
84
+ .map((finding) => shape(finding, target.name));
85
+ const summary = { error: 0, warn: 0, info: 0 };
86
+ for (const finding of findings) summary[finding.severity] += 1;
87
+ const family = familyOf(pairFindings, target.name);
88
+ return {
89
+ name: target.name,
90
+ verdict: summary.error + summary.warn > 0 ? "warn" : "pass",
91
+ summary,
92
+ findings,
93
+ ...(family.length > 1 ? { family } : {})
94
+ };
95
+ });
96
+ return { results };
97
+ }
28
98
 
29
- return {
30
- verdict: summary.error + summary.warn > 0 ? "warn" : "pass",
31
- summary,
32
- findings,
33
- ...(family.length > 1 ? { family } : {})
34
- };
99
+ // A pair finding touches both items, so the batch summary is deduped by
100
+ // subject/with; per-item summaries keep the same finding under each item.
101
+ function uniqueSummary(results) {
102
+ const seen = new Map();
103
+ for (const result of results) {
104
+ for (const finding of result.findings) {
105
+ const key = [
106
+ finding.code,
107
+ finding.severity,
108
+ finding.subject,
109
+ finding.with ?? "",
110
+ finding.param ?? ""
111
+ ].join("|");
112
+ if (!seen.has(key)) seen.set(key, finding);
113
+ }
114
+ }
115
+ const summary = { error: 0, warn: 0, info: 0 };
116
+ for (const finding of seen.values()) summary[finding.severity] += 1;
117
+ return summary;
35
118
  }
36
119
 
37
120
  function normalizeEntry(entry) {
@@ -42,6 +125,11 @@ function normalizeEntry(entry) {
42
125
  return {
43
126
  name,
44
127
  description: typeof entry.description === "string" ? entry.description : "",
128
+ // Scan marks descriptions assembled at runtime; without this the gate
129
+ // treats every Claude-Code-style tool as "description missing".
130
+ ...(typeof entry.descriptionSource === "string"
131
+ ? { descriptionSource: entry.descriptionSource }
132
+ : {}),
45
133
  parameters: parameters && typeof parameters === "object" ? parameters : null
46
134
  };
47
135
  }
@@ -61,10 +149,22 @@ function touches(finding, name) {
61
149
  return Array.isArray(finding.pair) && finding.pair.includes(name);
62
150
  }
63
151
 
152
+ // An install gate should only warn about things that change model behavior:
153
+ // selection conflicts, unsourceable required params, and hard schema errors.
154
+ // Description and naming rules stay informative — otherwise every mature
155
+ // gateway tool warns on install and the signal becomes noise.
156
+ const GATE_CRITICAL_CODES = new Set([
157
+ "desc-missing",
158
+ "param-type-missing",
159
+ "required-unsourced",
160
+ "pair-no-boundary",
161
+ "skill-pair-no-boundary"
162
+ ]);
163
+
64
164
  function shape(finding, name) {
65
165
  const other = (finding.pair ?? []).find((item) => item !== name);
66
166
  return {
67
- severity: finding.severity,
167
+ severity: GATE_CRITICAL_CODES.has(finding.code) ? finding.severity : "info",
68
168
  code: finding.code,
69
169
  subject: name,
70
170
  ...(finding.param ? { param: finding.param } : {}),
@@ -101,18 +201,41 @@ function familyOf(pairs, name) {
101
201
  return [...seen].sort();
102
202
  }
103
203
 
204
+ function findingLines(finding) {
205
+ const param = finding.param ? `.${finding.param}` : "";
206
+ const withWhat = finding.with ? ` × ${finding.with}` : "";
207
+ const lines = [
208
+ `[${finding.severity}] ${finding.code} ${finding.subject}${param}${withWhat}`,
209
+ ` ${finding.message}`
210
+ ];
211
+ if (finding.suggestion) lines.push(` ${finding.suggestion}`);
212
+ return lines;
213
+ }
214
+
104
215
  export function formatCheck(result) {
105
216
  const lines = [
106
217
  `verdict: ${result.verdict}(error ${result.summary.error} / warn ${result.summary.warn} / info ${result.summary.info})`
107
218
  ];
108
219
  if (result.findings.length === 0) lines.push("没有发现问题");
109
- for (const finding of result.findings) {
110
- const param = finding.param ? `.${finding.param}` : "";
111
- const withWhat = finding.with ? ` × ${finding.with}` : "";
112
- lines.push(`[${finding.severity}] ${finding.code} ${finding.subject}${param}${withWhat}`);
113
- lines.push(` ${finding.message}`);
114
- if (finding.suggestion) lines.push(` ${finding.suggestion}`);
115
- }
220
+ for (const finding of result.findings) lines.push(...findingLines(finding));
116
221
  if (result.family) lines.push("", `族:${result.family.join(", ")}`);
117
222
  return lines.join("\n");
118
223
  }
224
+
225
+ export function formatCheckMany(result) {
226
+ const lines = [
227
+ `verdict: ${result.verdict}(${result.items} 个新能力;去重后 error ${result.summary.error} / warn ${result.summary.warn} / info ${result.summary.info})`
228
+ ];
229
+ result.results.forEach((item, index) => {
230
+ lines.push(
231
+ "",
232
+ `[${index + 1}] ${item.name} ${item.verdict}(error ${item.summary.error} / warn ${item.summary.warn} / info ${item.summary.info})`
233
+ );
234
+ if (item.findings.length === 0) lines.push(" 没有发现问题");
235
+ for (const finding of item.findings) {
236
+ for (const line of findingLines(finding)) lines.push(` ${line}`);
237
+ }
238
+ if (item.family) lines.push(` 族:${item.family.join(", ")}`);
239
+ });
240
+ return lines.join("\n");
241
+ }
package/lib/cluster.js CHANGED
@@ -1,4 +1,10 @@
1
- import { pairSimilarity, familyScore, nameTokens, GENERIC_NAME_TOKENS } from "./cases.js";
1
+ import {
2
+ prepareItem,
3
+ pairSimilarityPrepared,
4
+ familyScorePrepared,
5
+ nameTokens,
6
+ GENERIC_NAME_TOKENS
7
+ } from "./cases.js";
2
8
 
3
9
  // A shared domain noun ("session", "goal", "tool") says these tools serve one
4
10
  // purpose even when their verbs and parameter shapes differ — exactly what
@@ -21,12 +27,13 @@ const BOUNDARY_FAMILY = 0.6;
21
27
 
22
28
  export function clusterTools(tools, { threshold = 0.5, maxSize = 10 } = {}) {
23
29
  const scores = tools.map(() => new Array(tools.length).fill(0));
30
+ const prepared = tools.map((tool) => prepareItem(tool));
24
31
  const pairs = [];
25
32
  const edges = [];
26
33
  for (let i = 0; i < tools.length; i += 1) {
27
34
  for (let j = i + 1; j < tools.length; j += 1) {
28
- const { similarity, basis } = pairSimilarity(tools[i], tools[j]);
29
- const family = familyScore(tools[i], tools[j]);
35
+ const { similarity, basis } = pairSimilarityPrepared(prepared[i], prepared[j]);
36
+ const family = familyScorePrepared(prepared[i], prepared[j]);
30
37
  const link = Math.max(similarity, family);
31
38
  scores[i][j] = link;
32
39
  scores[j][i] = link;
package/lib/lint.js CHANGED
@@ -57,7 +57,12 @@ const BOUNDARY_HINT = /(不适用|不要用于|不用于|不能用于|而非|区
57
57
  // Chinese descriptions are compared as bigrams, so mid-0.3s similarity is noise.
58
58
  const NO_BOUNDARY_SIMILARITY = 0.5;
59
59
 
60
- export function lintSuite(suite) {
60
+ // `pairScope` (a Set of names) is for incremental gates: only emit pair
61
+ // findings that touch one of the new capabilities. Installed × installed pairs
62
+ // are computed and then discarded by the caller, which dominates large sets.
63
+ export function lintSuite(suite, { pairScope } = {}) {
64
+ const inScope = (pair) => !pairScope || pairScope.has(pair.a) || pairScope.has(pair.b);
65
+ const toolByName = new Map(suite.tools.map((tool) => [tool.name, tool]));
61
66
  const findings = [];
62
67
  const add = (severity, code, tool, message, extra = {}) =>
63
68
  findings.push({ severity, code, kind: "tool", tool, message, ...extra });
@@ -205,9 +210,10 @@ export function lintSuite(suite) {
205
210
  }
206
211
 
207
212
  for (const pair of suspectedPairs(suite.tools)) {
213
+ if (!inScope(pair)) continue;
208
214
  const similarity = pair.similarity.toFixed(2);
209
215
  const declaresBoundary = [pair.a, pair.b].some((name) => {
210
- const tool = suite.tools.find((item) => item.name === name);
216
+ const tool = toolByName.get(name);
211
217
  return BOUNDARY_HINT.test(tool?.description ?? "");
212
218
  });
213
219
  if (declaresBoundary) {
@@ -223,8 +229,8 @@ export function lintSuite(suite) {
223
229
  }
224
230
  }
225
231
  const reason = pair.basis === "structural" ? "名字与参数形状接近" : "描述相近";
226
- const toolA = suite.tools.find((tool) => tool.name === pair.a);
227
- const toolB = suite.tools.find((tool) => tool.name === pair.b);
232
+ const toolA = toolByName.get(pair.a);
233
+ const toolB = toolByName.get(pair.b);
228
234
  add("warn", "pair-no-boundary", pair.a, `与 ${pair.b} ${reason}(相似度 ${similarity}),且两边都没写“不适用于”边界,模型只能靠相似度硬猜`, {
229
235
  pair: [pair.a, pair.b],
230
236
  suggestion: boundaryHint(toolA, toolB).suggestion
@@ -236,6 +242,7 @@ export function lintSuite(suite) {
236
242
  // question is narrower than for tools: is the description specific enough to
237
243
  // be triggered, and does it collide with another installed skill?
238
244
  const skills = suite.skills ?? [];
245
+ const skillByName = new Map(skills.map((skill) => [skill.name, skill]));
239
246
  for (const skill of skills) {
240
247
  const description = String(skill.description ?? "").trim();
241
248
  if (!description) {
@@ -258,9 +265,10 @@ export function lintSuite(suite) {
258
265
  threshold: SKILL_PAIR_SIMILARITY,
259
266
  extraStop: SKILL_BOILERPLATE
260
267
  })) {
268
+ if (!inScope(pair)) continue;
261
269
  const similarity = pair.similarity.toFixed(2);
262
270
  const declaresBoundary = [pair.a, pair.b].some((name) => {
263
- const skill = skills.find((item) => item.name === name);
271
+ const skill = skillByName.get(name);
264
272
  return BOUNDARY_HINT.test(skill?.description ?? "");
265
273
  });
266
274
  // Same-vendor siblings ("feishu-doc" / "feishu-wiki") share the vendor token
package/lib/mcp-server.js CHANGED
@@ -11,6 +11,10 @@ import { writeFileAtomic } from "./fsutil.js";
11
11
  import { probeStdioServer, probeHttpServer } from "./mcp-probe.js";
12
12
  import { addCases, computeCoverage } from "./cases.js";
13
13
  import { auditSuite } from "./audit.js";
14
+ import { check, checkMany } from "./check.js";
15
+ import { lintSuite } from "./lint.js";
16
+ import { clusterTools } from "./cluster.js";
17
+ import { compareScorecards } from "./compare.js";
14
18
 
15
19
  const MF_PATH = fileURLToPath(new URL("../mf.js", import.meta.url));
16
20
  const PACKAGE_JSON = new URL("../package.json", import.meta.url);
@@ -125,6 +129,78 @@ export async function startMcpServer() {
125
129
  )
126
130
  );
127
131
 
132
+ server.registerTool(
133
+ "lint",
134
+ {
135
+ title: "Lint suite",
136
+ description:
137
+ "静态检查 tools 与 skills 的描述、参数与工具对边界(不调用模型)。返回 counts / findings / profilePairs。",
138
+ inputSchema: { suite: z.string().optional().describe("套件路径,默认 ./mf.suite.json") }
139
+ },
140
+ async ({ suite }) =>
141
+ json(
142
+ lintSuite(
143
+ await loadSuite(suitePath(suite), { allowEmptyCases: true, allowEmptyTools: true })
144
+ )
145
+ )
146
+ );
147
+
148
+ server.registerTool(
149
+ "check",
150
+ {
151
+ title: "Install gate",
152
+ description:
153
+ "安装门禁:把新能力与已装能力做静态比较,返回 pass/warn、findings 与竞争族。传 manifest 检查单个;传 manifests 检查一包(新能力之间也会互比)。不阻断、不调用模型、无网络。",
154
+ inputSchema: {
155
+ manifest: z.any().optional().describe("新能力 {name, description, parameters?} 或 tool schema"),
156
+ manifests: z.array(z.any()).optional().describe("一次安装一包能力时的数组"),
157
+ installed: z.any().optional().describe("已装能力数组,或 {tools: [], skills: []}")
158
+ }
159
+ },
160
+ async ({ manifest, manifests, installed }) => {
161
+ if (manifests) return json(checkMany({ manifests, installed }));
162
+ return json(check({ manifest, installed }));
163
+ }
164
+ );
165
+
166
+ server.registerTool(
167
+ "cluster",
168
+ {
169
+ title: "Cluster tools",
170
+ description:
171
+ "聚类工具,给出挂载分组、跨组高风险与族内边界清单(每族要补哪句选择条件)。不调用模型。",
172
+ inputSchema: {
173
+ suite: z.string().optional().describe("套件路径,默认 ./mf.suite.json"),
174
+ threshold: z.number().min(0).max(1).optional().describe("聚类阈值,默认 0.5"),
175
+ maxSize: z.number().int().positive().optional().describe("单组最大工具数,默认 10")
176
+ }
177
+ },
178
+ async ({ suite, threshold, maxSize }) => {
179
+ const loaded = await loadSuite(suitePath(suite), { allowEmptyCases: true });
180
+ return json(clusterTools(loaded.tools, { threshold, maxSize }));
181
+ }
182
+ );
183
+
184
+ server.registerTool(
185
+ "compare",
186
+ {
187
+ title: "Compare scorecards",
188
+ description:
189
+ "比较两份评分卡 JSON:可比性、升降、干扰归因与混淆矩阵。只读文件,不调用模型。",
190
+ inputSchema: {
191
+ before: z.string().describe("旧评分卡 JSON 路径"),
192
+ after: z.string().describe("新评分卡 JSON 路径")
193
+ }
194
+ },
195
+ async ({ before, after }) => {
196
+ const [oldText, newText] = await Promise.all([
197
+ readFile(resolve(before), "utf8"),
198
+ readFile(resolve(after), "utf8")
199
+ ]);
200
+ return json(compareScorecards(JSON.parse(oldText), JSON.parse(newText)));
201
+ }
202
+ );
203
+
128
204
  server.registerTool(
129
205
  "add_cases",
130
206
  {
package/lib/profile.js CHANGED
@@ -2,17 +2,28 @@ import { pct } from "./score.js";
2
2
 
3
3
  export function normalizeProfiles(raw, toolNames) {
4
4
  if (raw === undefined || raw === null) return null;
5
- if (typeof raw !== "object" || Array.isArray(raw)) {
6
- throw new Error('suite "profiles" must be an object of { "group": ["tool", ...] }');
5
+ if (typeof raw !== "object") {
6
+ throw new Error('suite "profiles" must be an object or an array of groups');
7
7
  }
8
8
  const known = new Set(toolNames);
9
9
  const profiles = [];
10
- for (const [key, list] of Object.entries(raw)) {
11
- const name = String(key).trim();
10
+ const seen = new Set();
11
+ // Platform registries often hand us an array of
12
+ // { agentType, tools, skills } rather than { "group": ["tool"] }.
13
+ const entries = Array.isArray(raw)
14
+ ? raw.map((value, index) => [String(index), value])
15
+ : Object.entries(raw);
16
+ for (const [key, value] of entries) {
17
+ // Array form: { "group": ["tool", ...] }. Object form (platform registries
18
+ // often look like this): { "0": { agentType, tools, skills } }.
19
+ const list = Array.isArray(value) ? value : value?.tools;
20
+ const name = String(Array.isArray(value) ? key : value?.name ?? value?.agentType ?? key).trim();
12
21
  if (!name) throw new Error('suite "profiles": group name must be non-empty');
13
22
  if (!Array.isArray(list) || list.length === 0) {
14
23
  throw new Error(`profile "${name}": must be a non-empty array of tool names`);
15
24
  }
25
+ if (seen.has(name)) throw new Error(`profile "${name}": duplicate group name`);
26
+ seen.add(name);
16
27
  const tools = [];
17
28
  for (const item of list) {
18
29
  if (typeof item !== "string" || item.trim() === "") {
package/mf.js CHANGED
@@ -11,7 +11,7 @@ import { probeStdioServer, probeHttpServer } from "./lib/mcp-probe.js";
11
11
  import { loadAdapterTools } from "./lib/adapter-import.js";
12
12
  import { addCases, computeCoverage, formatCoverage } from "./lib/cases.js";
13
13
  import { lintSuite, formatLint } from "./lib/lint.js";
14
- import { check, formatCheck } from "./lib/check.js";
14
+ import { check, checkMany, formatCheck, formatCheckMany } from "./lib/check.js";
15
15
  import { auditSuite, formatAudit } from "./lib/audit.js";
16
16
  import { suiteFingerprint, hashText, hashOf } from "./lib/fingerprint.js";
17
17
  import { runExecutionChecks } from "./lib/eval-exec.js";
@@ -75,7 +75,7 @@ Usage:
75
75
  mf coverage [--suite <file>] [--json]
76
76
  mf audit [--suite <file>] [--out <file>] [--json] # 静态审计:lint + 分组边界 + 用例覆盖 + 挂载
77
77
  mf lint [--suite <file>] [--json] # 静态检查 tools 与 skills,不调用模型
78
- mf check --manifest <file> [--installed <file>] [--json] # 安装门禁:新能力 × 已装能力
78
+ mf check (--manifest <file> | --manifests <file>) [--installed <file>] [--json] # 安装门禁:新能力 × 已装能力(--manifests 支持一包)
79
79
  mf compare <before.json> <after.json> [--json]
80
80
  mf ablate --tool <name> [--suite <file>] [--profile <group>] [--models <ids>] [--max-cost <cny>] [--progress] [--json]
81
81
  mf cluster [--suite <file>] [--threshold <0.5>] [--max-size <10>] [--out <file>] [--json] # 建议挂载分组 + 同族候选
@@ -482,7 +482,7 @@ export async function runScan(options, io = console) {
482
482
 
483
483
  if (options.json) {
484
484
  io.log(JSON.stringify({ report: redactReport(report), draft }, null, 2));
485
- return report.tools.length > 0 ? 0 : 3;
485
+ return report.tools.length > 0 || report.skills.length > 0 ? 0 : 3;
486
486
  }
487
487
 
488
488
  io.log(formatScanSummary(report));
@@ -497,7 +497,7 @@ export async function runScan(options, io = console) {
497
497
  io.log(`wrote ${reportPath}`);
498
498
  }
499
499
 
500
- return report.tools.length > 0 ? 0 : 3;
500
+ return report.tools.length > 0 || report.skills.length > 0 ? 0 : 3;
501
501
  }
502
502
 
503
503
  export async function runCaseAdd(options, io = console) {
@@ -584,11 +584,24 @@ export async function runSandboxCheck(options, io = console) {
584
584
  // Advisory install gate: static only, never blocks. Exit code stays 0 for a
585
585
  // warn verdict so a platform can log it without failing the install.
586
586
  export async function runCheck(options, io = console) {
587
- if (!options.manifest) throw new Error("--manifest <file> 必填(新能力的 [name, description, parameters?])");
588
- const manifest = JSON.parse(await readFile(resolve(options.manifest), "utf8"));
589
587
  const installed = options.installed
590
588
  ? JSON.parse(await readFile(resolve(options.installed), "utf8"))
591
589
  : [];
590
+
591
+ if (options.manifests) {
592
+ const manifests = JSON.parse(await readFile(resolve(options.manifests), "utf8"));
593
+ const result = checkMany({ manifests, installed });
594
+ noteUsage({
595
+ checkVerdict: result.verdict,
596
+ checkFindings: Object.values(result.summary).reduce((sum, count) => sum + count, 0)
597
+ });
598
+ io.log(options.json ? JSON.stringify(result, null, 2) : formatCheckMany(result));
599
+ return 0;
600
+ }
601
+ if (!options.manifest) {
602
+ throw new Error("--manifest 或 --manifests <file> 必填([name, description, parameters?])");
603
+ }
604
+ const manifest = JSON.parse(await readFile(resolve(options.manifest), "utf8"));
592
605
  const result = check({ manifest, installed });
593
606
  noteUsage({ checkVerdict: result.verdict, checkFindings: result.findings.length });
594
607
  io.log(options.json ? JSON.stringify(result, null, 2) : formatCheck(result));
@@ -1009,6 +1022,7 @@ function parseArgs(argv) {
1009
1022
  dryRun: false,
1010
1023
  tool: null,
1011
1024
  manifest: null,
1025
+ manifests: null,
1012
1026
  installed: null,
1013
1027
  threshold: 0.5,
1014
1028
  maxSize: 10,
@@ -1077,6 +1091,7 @@ function parseArgs(argv) {
1077
1091
  case "--dry-run": options.dryRun = true; break;
1078
1092
  case "--tool": options.tool = value(); break;
1079
1093
  case "--manifest": options.manifest = resolve(value()); break;
1094
+ case "--manifests": options.manifests = resolve(value()); break;
1080
1095
  case "--installed": options.installed = resolve(value()); break;
1081
1096
  case "--threshold": {
1082
1097
  const raw = Number(value());
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "modelfunction-cli",
3
- "version": "0.5.1",
3
+ "version": "0.5.3",
4
4
  "type": "module",
5
5
  "license": "MIT",
6
6
  "description": "mf — function-calling accuracy scorecard for AI agents",