modelfunction-cli 0.5.1 → 0.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -6
- package/SCHEMA.md +38 -5
- package/lib/cases.js +62 -38
- package/lib/check.d.ts +40 -2
- package/lib/check.js +150 -27
- package/lib/cluster.js +10 -3
- package/lib/lint.js +13 -5
- package/lib/mcp-server.js +76 -0
- package/lib/profile.js +15 -4
- package/mf.js +21 -6
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
## 安装
|
|
6
6
|
|
|
7
7
|
```bash
|
|
8
|
-
npx -y modelfunction-cli@0.5.
|
|
8
|
+
npx -y modelfunction-cli@0.5.1 --version
|
|
9
9
|
# 或
|
|
10
10
|
npm i -D modelfunction-cli
|
|
11
11
|
npx mf --version
|
|
@@ -81,7 +81,7 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
|
|
|
81
81
|
| `mf coverage` | 每工具用例数、参数覆盖、负例比例、疑似混淆对;有 `profiles` 时按组统计 |
|
|
82
82
|
| `mf audit` | 一次静态审计:lint + 混淆族边界 + 用例覆盖 + 挂载检查,输出可转发的 Markdown/JSON;不调用模型 |
|
|
83
83
|
| `mf lint` | 静态检查工具描述与参数 schema,以及 skills 的 name/description(不调用模型);重复规则自动合并成一条;有 `profiles` 时给出「这些组不要合并挂载」 |
|
|
84
|
-
| `mf check` |
|
|
84
|
+
| `mf check` | 安装门禁:一个或一包新能力 × 已装能力(`--manifest` / `--manifests` + `--installed`),纯静态、返回 pass/warn、不阻断;门禁语境只对混淆对/必填来源/结构性错误报 warn,其余降 info;也可 `import { check, checkMany } from "modelfunction-cli"` |
|
|
85
85
|
| `mf compare` | 跨版本比较、干扰归因、混淆矩阵;两份报告都有分组时按组对比 |
|
|
86
86
|
| `mf ablate` | 留一法:去掉某个工具再跑一遍,量化它是否干扰其他工具;`--profile` 限定在某个挂载组内 |
|
|
87
87
|
| `mf cluster` | 聚类工具,建议挂载分组;参数形状拆散的同一族(如 `sessions_*`)列为「同族候选」;并输出「族内边界清单」(每族会互抢的对 + 每个工具可用的区分词);`--out` 把分组写成套件的 `profiles` |
|
|
@@ -126,19 +126,19 @@ mf eval --suite mf.suite.json --models internal-ds \
|
|
|
126
126
|
"mcpServers": {
|
|
127
127
|
"modelfunction": {
|
|
128
128
|
"command": "npx",
|
|
129
|
-
"args": ["-y", "modelfunction-cli@0.5.
|
|
129
|
+
"args": ["-y", "modelfunction-cli@0.5.1", "mcp"]
|
|
130
130
|
}
|
|
131
131
|
}
|
|
132
132
|
}
|
|
133
133
|
```
|
|
134
134
|
|
|
135
|
-
暴露:`scan_tools`、`list_suite`、`coverage`、`add_cases`、`run_eval`(`run_eval` 支持 `profile`
|
|
135
|
+
暴露:`scan_tools`、`list_suite`、`coverage`、`audit`、`lint`、`check`、`cluster`、`compare`、`add_cases`、`run_eval`(`run_eval` 支持 `profile`、`repeat`、`maxCost`、`progress`)。`lint`/`check`/`cluster`/`compare` 都是静态操作,不调用模型;`check` 就是安装门禁,Agent 可直接传入 manifest 与 installed。
|
|
136
136
|
|
|
137
137
|
## 契约与版本
|
|
138
138
|
|
|
139
139
|
对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md),自 0.4.3 起冻结为 v1:v1 内只做加法,重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
|
|
140
140
|
|
|
141
|
-
|
|
141
|
+
`0.5.2` 已发布到 npm:含凭证与本地反馈、usage `ci` 标记、eval `--progress`/`--max-cost`、`mf audit`、`scan --probe` streamable HTTP、MCP 静态工具 `lint`/`check`/`cluster`/`compare`、`checkMany` 一包批量检查,门禁 warn 只保留混淆对/必填来源/结构性错误。本仓库 `0.5.3` 待发布:静态规则性能(真实 100 能力清单 `checkMany` 471ms→40ms,550 能力 9.6s→2.1s,输出与 golden 一致),以及 `installed` 形状校验(接受 `mf scan --json` 输出,无法识别时报错而不是静默返回假 pass)和 `profiles` 兼容平台形态(对象或 `[{agentType, tools, skills}]` 数组)。示例统一使用 `@0.5.2`。
|
|
142
142
|
|
|
143
143
|
## 数据边界
|
|
144
144
|
|
|
@@ -167,7 +167,7 @@ mf feedback --out out.json # 导出脱敏反馈包(聚合值,不含原始
|
|
|
167
167
|
|
|
168
168
|
```bash
|
|
169
169
|
# 1. 把 mf 挂成 MCP server
|
|
170
|
-
claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.
|
|
170
|
+
claude mcp add modelfunction -- npx -y modelfunction-cli@0.5.1 mcp
|
|
171
171
|
```
|
|
172
172
|
|
|
173
173
|
```text
|
package/SCHEMA.md
CHANGED
|
@@ -73,7 +73,7 @@ skills 只做静态检查,不参与模型评测。
|
|
|
73
73
|
|
|
74
74
|
### profiles
|
|
75
75
|
|
|
76
|
-
组名 →
|
|
76
|
+
组名 → 工具名数组(`{ "group": ["tool", ...] }`)。也接受平台常见的两种形态:对象 `{ "0": { agentType, tools, skills } }` 与数组 `[{ agentType, tools, skills }]`,组名依次取 `name` → `agentType` → 键/下标;`skills` 字段目前被忽略(profiles 只用于工具挂载与 eval 分组)。所有工具名必须存在;组可重叠,组名不可重复。`mf eval --profile` 与 `mf ablate --profile` 依赖它。
|
|
77
77
|
|
|
78
78
|
### execution
|
|
79
79
|
|
|
@@ -112,7 +112,7 @@ skills 只做静态检查,不参与模型评测。
|
|
|
112
112
|
|
|
113
113
|
report:`root`、`tools[]`(`name` / `description` / `parameters` / `origin` / `confidence` / `source` / `note?` / `descriptionSource?`)、`skills[]`、`mcpServers[]`(env 只保留 `envKeys`,headers 只保留 `headerKeys`;URL 去掉 userinfo / query / hash)、`probes[]`(`server` / `ok` / `toolCount?` / `tools?` / `error?`)、`sources[]`、`warnings[]`。`--probe` 支持 stdio 与 streamable HTTP 两种传输。
|
|
114
114
|
|
|
115
|
-
`scan --json`
|
|
115
|
+
`scan --json` 在工具与 skill 都未找到时退出码为 3(只有 skill 也算有结果,退出 0)。
|
|
116
116
|
|
|
117
117
|
## 4. lint JSON
|
|
118
118
|
|
|
@@ -159,7 +159,7 @@ report:`root`、`tools[]`(`name` / `description` / `parameters` / `origin` /
|
|
|
159
159
|
import { check } from "modelfunction-cli";
|
|
160
160
|
```
|
|
161
161
|
|
|
162
|
-
输入 `{ manifest, installed }`:`manifest` 是一个能力(`name` 必填,`description` 可选,`parameters` / `inputSchema` / `input` 任一为 JSON Schema;无 schema 视为 skill)。`installed`
|
|
162
|
+
输入 `{ manifest, installed }`:`manifest` 是一个能力(`name` 必填,`description` 可选,`parameters` / `inputSchema` / `input` 任一为 JSON Schema;无 schema 视为 skill)。`installed` 可以是能力数组、`{ tools: [], skills: [] }`,或直接传 `mf scan --json` 的输出(能力在 `report` 下)。空对象视为没有已装能力;其他无法识别的形状会抛错,避免静默返回假的 `pass`。
|
|
163
163
|
|
|
164
164
|
返回:
|
|
165
165
|
|
|
@@ -174,8 +174,37 @@ import { check } from "modelfunction-cli";
|
|
|
174
174
|
```
|
|
175
175
|
|
|
176
176
|
- `verdict` 只有 `pass` / `warn`,**没有 fail**;质量判定不阻断安装。
|
|
177
|
-
-
|
|
177
|
+
- **严重度策略(门禁语境)**:只有会改变模型行为的规则保持原等级——`desc-missing`、`param-type-missing`(error),`required-unsourced`、`pair-no-boundary`、`skill-pair-no-boundary`(warn);描述、命名、必填数量等质量规则降为 `info`,避免成熟工具集每次安装都告警。`mf lint` 不受影响,仍输出原始严重度。
|
|
178
|
+
- 抛错场景:`manifest.name` 缺失或为空;`installed` 不是上述三种形状之一(空对象仍视为无已装能力)。
|
|
178
179
|
- 同步、无网络、无文件系统、无状态。类型声明见 `lib/check.d.ts`。
|
|
180
|
+
- 性能(0.5.3):每个能力只分词一次,且只比较涉及新能力的对。真实 100 能力清单(53 新 + 47 已装)约 40–50ms,47 工具对 53 skill 约 25ms;550 个高度同质的合成最坏情况约 2s。
|
|
181
|
+
|
|
182
|
+
批量(一次安装一包):
|
|
183
|
+
|
|
184
|
+
```js
|
|
185
|
+
import { checkMany } from "modelfunction-cli";
|
|
186
|
+
const result = checkMany({ manifests: [/* ... */], installed: [/* ... */] });
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
返回 `{ verdict, summary, items, results[] }`:
|
|
190
|
+
|
|
191
|
+
```json
|
|
192
|
+
{
|
|
193
|
+
"verdict": "warn",
|
|
194
|
+
"summary": { "error": 0, "warn": 1, "info": 0 },
|
|
195
|
+
"items": 2,
|
|
196
|
+
"results": [
|
|
197
|
+
{ "name": "gitcrawl", "verdict": "warn", "summary": { "error": 0, "warn": 1, "info": 0 },
|
|
198
|
+
"findings": [ { "severity": "warn", "code": "skill-pair-no-boundary", "subject": "gitcrawl",
|
|
199
|
+
"with": "tag-duplicate-prs-issues", "message": "...", "suggestion": "..." } ],
|
|
200
|
+
"family": ["gitcrawl", "tag-duplicate-prs-issues"] },
|
|
201
|
+
{ "name": "tag-duplicate-prs-issues", "verdict": "warn", "summary": { "error": 0, "warn": 1, "info": 0 },
|
|
202
|
+
"findings": [ /* 同一对,subject 为自身 */ ], "family": ["gitcrawl", "tag-duplicate-prs-issues"] }
|
|
203
|
+
]
|
|
204
|
+
}
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
`summary` 按 subject/with 去重(同一对只计一次);`results[]` 保留与该能力相关的全部发现(pair 会出现在双方)。`manifests` 为空、某项缺 name、名称重复时抛错。CLI:`mf check --manifests pack.json [--installed installed.json] [--json]`。
|
|
179
208
|
|
|
180
209
|
## 7. MCP 工具
|
|
181
210
|
|
|
@@ -185,6 +214,10 @@ import { check } from "modelfunction-cli";
|
|
|
185
214
|
| `list_suite` | `suite?` |
|
|
186
215
|
| `coverage` | `suite?` |
|
|
187
216
|
| `audit` | `suite?` |
|
|
217
|
+
| `lint` | `suite?` |
|
|
218
|
+
| `check` | `manifest`(对象)、`installed?`(数组或 `{tools,skills}`) |
|
|
219
|
+
| `cluster` | `suite?`、`threshold?`、`maxSize?` |
|
|
220
|
+
| `compare` | `before`、`after`(评分卡 JSON 路径) |
|
|
188
221
|
| `add_cases` | `suite?`、`cases[]`、`source?`、`reviewed?`、`dryRun?` |
|
|
189
222
|
| `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?`、`maxCost?`、`progress?` |
|
|
190
223
|
|
|
@@ -197,7 +230,7 @@ import { check } from "modelfunction-cli";
|
|
|
197
230
|
| 0 | 成功(`mf check` 的 warn 也是 0) |
|
|
198
231
|
| 1 | 输入/用法错误;`mf compare` 判定不可直接比较 |
|
|
199
232
|
| 2 | `mf eval --fail-under` 未达标、`--fail-on-regression` 发现回退,或 `--max-cost` 超限 |
|
|
200
|
-
| 3 | `mf exec` 被权限策略拒绝;`mf scan --json`
|
|
233
|
+
| 3 | `mf exec` 被权限策略拒绝;`mf scan --json` 未找到工具或 skill |
|
|
201
234
|
| 4 | `mf exec` 超时 |
|
|
202
235
|
|
|
203
236
|
## 9. 本地文件(0.5.0 起)
|
package/lib/cases.js
CHANGED
|
@@ -207,38 +207,52 @@ export function formatCoverage(report) {
|
|
|
207
207
|
return lines.join("\n");
|
|
208
208
|
}
|
|
209
209
|
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
const
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
210
|
+
// O(n²) callers prepare each capability once; the pair loop then only compares
|
|
211
|
+
// sets. Tokenizing inside the loop made a 550-capability gate take ~14s.
|
|
212
|
+
export function prepareItem(item, { extraStop } = {}) {
|
|
213
|
+
const names = nameTokens(item.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
|
|
214
|
+
return {
|
|
215
|
+
name: item.name,
|
|
216
|
+
tokens: tokens(`${item.name} ${item.description}`, extraStop),
|
|
217
|
+
nameTokens: names,
|
|
218
|
+
nameTokenSet: new Set(names),
|
|
219
|
+
params: paramNames(item),
|
|
220
|
+
required: new Set(item.parameters?.required ?? [])
|
|
221
|
+
};
|
|
222
|
+
}
|
|
223
|
+
|
|
224
|
+
export function pairSimilarityPrepared(a, b) {
|
|
225
|
+
const shared = intersectionSize(a.tokens, b.tokens);
|
|
226
|
+
const union = a.tokens.size + b.tokens.size - shared;
|
|
227
|
+
const lexical = union > 0 ? shared / union : 0;
|
|
228
|
+
const structural = intersects(a.nameTokenSet, b.nameTokenSet) ? structuralScorePrepared(a, b) : 0;
|
|
221
229
|
return {
|
|
222
230
|
similarity: Math.max(lexical, structural),
|
|
223
231
|
basis: structural > lexical ? "structural" : "lexical"
|
|
224
232
|
};
|
|
225
233
|
}
|
|
226
234
|
|
|
235
|
+
export function pairSimilarity(x, y, { extraStop } = {}) {
|
|
236
|
+
return pairSimilarityPrepared(prepareItem(x, { extraStop }), prepareItem(y, { extraStop }));
|
|
237
|
+
}
|
|
238
|
+
|
|
227
239
|
// Clustering wants "same family" (alarm_query / alarm_ack belong to one agent),
|
|
228
240
|
// which is looser than "confusable" (lint's question). Shared name token plus
|
|
229
241
|
// any parameter overlap is enough to belong together; two shared tokens are
|
|
230
242
|
// enough on their own (fault_alarm_query / fault_alarm_ack take different ids).
|
|
231
243
|
// One shared token with disjoint params is not: cross-domain tools often share
|
|
232
244
|
// a bare verb (query / list) and nothing else.
|
|
233
|
-
export function
|
|
234
|
-
const
|
|
235
|
-
|
|
236
|
-
const
|
|
237
|
-
if (shared.length === 0) return 0;
|
|
238
|
-
const paramOverlap = jaccard(paramNames(x), paramNames(y));
|
|
245
|
+
export function familyScorePrepared(x, y) {
|
|
246
|
+
const shared = intersectionSize(x.nameTokenSet, y.nameTokenSet);
|
|
247
|
+
if (shared === 0) return 0;
|
|
248
|
+
const paramOverlap = jaccard(x.params, y.params);
|
|
239
249
|
if (paramOverlap > 0) return 0.8;
|
|
240
|
-
if (shared
|
|
241
|
-
return
|
|
250
|
+
if (shared >= 2) return 0.7;
|
|
251
|
+
return x.params.size === 0 && y.params.size === 0 ? 0.6 : 0.2;
|
|
252
|
+
}
|
|
253
|
+
|
|
254
|
+
export function familyScore(x, y) {
|
|
255
|
+
return familyScorePrepared(prepareItem(x), prepareItem(y));
|
|
242
256
|
}
|
|
243
257
|
|
|
244
258
|
// Turn a confusable pair into an authoring instruction. We do not invent the
|
|
@@ -280,12 +294,13 @@ export function boundaryHint(x, y) {
|
|
|
280
294
|
}
|
|
281
295
|
|
|
282
296
|
export function suspectedPairs(tools, { threshold = PAIR_SIMILARITY, extraStop } = {}) {
|
|
297
|
+
const prepared = tools.map((tool) => prepareItem(tool, { extraStop }));
|
|
283
298
|
const pairs = [];
|
|
284
|
-
for (let i = 0; i <
|
|
285
|
-
for (let j = i + 1; j <
|
|
286
|
-
const { similarity, basis } =
|
|
299
|
+
for (let i = 0; i < prepared.length; i += 1) {
|
|
300
|
+
for (let j = i + 1; j < prepared.length; j += 1) {
|
|
301
|
+
const { similarity, basis } = pairSimilarityPrepared(prepared[i], prepared[j]);
|
|
287
302
|
if (similarity >= threshold) {
|
|
288
|
-
pairs.push({ a:
|
|
303
|
+
pairs.push({ a: prepared[i].name, b: prepared[j].name, similarity, basis });
|
|
289
304
|
}
|
|
290
305
|
}
|
|
291
306
|
}
|
|
@@ -296,20 +311,13 @@ export function suspectedPairs(tools, { threshold = PAIR_SIMILARITY, extraStop }
|
|
|
296
311
|
// parameter shape confuse the model even when the wording differs, which the
|
|
297
312
|
// lexical score alone misses. Name order is unreliable: `tavily_search` leads
|
|
298
313
|
// with the vendor, `search_docs` with the action.
|
|
299
|
-
function
|
|
300
|
-
const
|
|
301
|
-
const right = nameTokens(y.name).filter((token) => !GENERIC_NAME_TOKENS.has(token));
|
|
302
|
-
const sharedName = left.some((token) => right.includes(token));
|
|
314
|
+
function structuralScorePrepared(x, y) {
|
|
315
|
+
const sharedName = x.nameTokens.some((token) => y.nameTokens.includes(token));
|
|
303
316
|
if (!sharedName) return 0;
|
|
304
|
-
|
|
305
|
-
const
|
|
306
|
-
if (params.size === 0 && otherParams.size === 0) return 0;
|
|
307
|
-
const paramOverlap = jaccard(params, otherParams);
|
|
317
|
+
if (x.params.size === 0 && y.params.size === 0) return 0;
|
|
318
|
+
const paramOverlap = jaccard(x.params, y.params);
|
|
308
319
|
if (paramOverlap < 0.34) return 0;
|
|
309
|
-
const requiredOverlap = jaccard(
|
|
310
|
-
new Set(x.parameters?.required ?? []),
|
|
311
|
-
new Set(y.parameters?.required ?? [])
|
|
312
|
-
);
|
|
320
|
+
const requiredOverlap = jaccard(x.required, y.required);
|
|
313
321
|
return 0.5 + 0.5 * (0.6 * paramOverlap + 0.4 * requiredOverlap);
|
|
314
322
|
}
|
|
315
323
|
|
|
@@ -340,10 +348,26 @@ function paramNames(tool) {
|
|
|
340
348
|
|
|
341
349
|
function jaccard(a, b) {
|
|
342
350
|
if (a.size === 0 && b.size === 0) return 1;
|
|
343
|
-
const
|
|
351
|
+
const shared = intersectionSize(a, b);
|
|
352
|
+
return shared / (a.size + b.size - shared);
|
|
353
|
+
}
|
|
354
|
+
|
|
355
|
+
// Iterate the smaller set; avoids allocating a union Set for every pair in the
|
|
356
|
+
// O(n²) loops.
|
|
357
|
+
function intersectionSize(a, b) {
|
|
358
|
+
const small = a.size <= b.size ? a : b;
|
|
359
|
+
const large = small === a ? b : a;
|
|
344
360
|
let shared = 0;
|
|
345
|
-
for (const item of
|
|
346
|
-
return shared
|
|
361
|
+
for (const item of small) if (large.has(item)) shared += 1;
|
|
362
|
+
return shared;
|
|
363
|
+
}
|
|
364
|
+
|
|
365
|
+
function intersects(a, b) {
|
|
366
|
+
if (a.size === 0 || b.size === 0) return false;
|
|
367
|
+
const small = a.size <= b.size ? a : b;
|
|
368
|
+
const large = small === a ? b : a;
|
|
369
|
+
for (const item of small) if (large.has(item)) return true;
|
|
370
|
+
return false;
|
|
347
371
|
}
|
|
348
372
|
|
|
349
373
|
function tokens(text, extraStop) {
|
package/lib/check.d.ts
CHANGED
|
@@ -17,12 +17,22 @@ export interface InstalledCapabilities {
|
|
|
17
17
|
skills?: Capability[];
|
|
18
18
|
}
|
|
19
19
|
|
|
20
|
+
/** `mf scan --json` output; its capabilities live under `report`. */
|
|
21
|
+
export interface ScanReport {
|
|
22
|
+
report: InstalledCapabilities;
|
|
23
|
+
}
|
|
24
|
+
|
|
20
25
|
export interface CheckInput {
|
|
21
26
|
manifest: Capability;
|
|
22
|
-
installed?: Capability[] | InstalledCapabilities;
|
|
27
|
+
installed?: Capability[] | InstalledCapabilities | ScanReport;
|
|
23
28
|
}
|
|
24
29
|
|
|
25
30
|
export interface CheckFinding {
|
|
31
|
+
/**
|
|
32
|
+
* Install-gate view: only behavior-changing findings keep warn/error
|
|
33
|
+
* (selection conflicts, unsourced required params, schema errors).
|
|
34
|
+
* Description and naming rules are reported as info so mature tools pass.
|
|
35
|
+
*/
|
|
26
36
|
severity: Severity;
|
|
27
37
|
code: string;
|
|
28
38
|
subject: string;
|
|
@@ -41,11 +51,39 @@ export interface CheckResult {
|
|
|
41
51
|
family?: string[];
|
|
42
52
|
}
|
|
43
53
|
|
|
54
|
+
export interface CheckManyInput {
|
|
55
|
+
manifests: Capability[];
|
|
56
|
+
installed?: Capability[] | InstalledCapabilities | ScanReport;
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
export interface CheckItemResult extends CheckResult {
|
|
60
|
+
name: string;
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
export interface CheckManyResult {
|
|
64
|
+
verdict: Verdict;
|
|
65
|
+
/** Deduplicated by subject/with; per-item summaries are in results[]. */
|
|
66
|
+
summary: { error: number; warn: number; info: number };
|
|
67
|
+
items: number;
|
|
68
|
+
results: CheckItemResult[];
|
|
69
|
+
}
|
|
70
|
+
|
|
44
71
|
/**
|
|
45
72
|
* Static install gate: judge one new capability against installed ones.
|
|
46
73
|
* No model call, no network, no file system, no state.
|
|
47
|
-
* @throws
|
|
74
|
+
* @throws when `manifest.name` is missing or empty, or `installed` is neither
|
|
75
|
+
* an array, `{ tools, skills }`, nor an `mf scan --json` report.
|
|
48
76
|
*/
|
|
49
77
|
export declare function check(input: CheckInput): CheckResult;
|
|
50
78
|
|
|
79
|
+
/**
|
|
80
|
+
* Batch install gate: judge a pack of new capabilities against installed ones
|
|
81
|
+
* and against each other.
|
|
82
|
+
* @throws when manifests is empty, an item lacks a name, names are duplicated,
|
|
83
|
+
* or `installed` is not an accepted shape.
|
|
84
|
+
*/
|
|
85
|
+
export declare function checkMany(input: CheckManyInput): CheckManyResult;
|
|
86
|
+
|
|
51
87
|
export declare function formatCheck(result: CheckResult): string;
|
|
88
|
+
|
|
89
|
+
export declare function formatCheckMany(result: CheckManyResult): string;
|
package/lib/check.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
// Install-gate entry point: judge
|
|
1
|
+
// Install-gate entry point: judge new capabilities against what is already
|
|
2
2
|
// installed. Static only — no model call, no network, no file system, no state.
|
|
3
3
|
// Quality findings are advisory: the verdict is pass/warn, never "block".
|
|
4
4
|
import { lintSuite } from "./lint.js";
|
|
@@ -6,32 +6,115 @@ import { lintSuite } from "./lint.js";
|
|
|
6
6
|
export function check({ manifest, installed = [] } = {}) {
|
|
7
7
|
const target = normalizeEntry(manifest);
|
|
8
8
|
if (!target) throw new Error('check: manifest 需要非空 "name"');
|
|
9
|
-
const
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
9
|
+
const { results } = evaluateTargets([target], installed);
|
|
10
|
+
const [result] = results;
|
|
11
|
+
return {
|
|
12
|
+
verdict: result.verdict,
|
|
13
|
+
summary: result.summary,
|
|
14
|
+
findings: result.findings,
|
|
15
|
+
...(result.family ? { family: result.family } : {})
|
|
16
|
+
};
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
// Installers usually apply a pack at once: new skills must be checked against
|
|
20
|
+
// installed ones AND against each other.
|
|
21
|
+
export function checkMany({ manifests, installed = [] } = {}) {
|
|
22
|
+
if (!Array.isArray(manifests) || manifests.length === 0) {
|
|
23
|
+
throw new Error("checkMany: manifests 需要非空数组");
|
|
24
|
+
}
|
|
25
|
+
const targets = manifests.map((entry) => normalizeEntry(entry));
|
|
26
|
+
if (targets.some((target) => target === null)) {
|
|
27
|
+
throw new Error('checkMany: 每个 manifest 都需要非空 "name"');
|
|
13
28
|
}
|
|
29
|
+
const names = new Set();
|
|
30
|
+
for (const target of targets) {
|
|
31
|
+
if (names.has(target.name)) throw new Error(`checkMany: duplicate name "${target.name}"`);
|
|
32
|
+
names.add(target.name);
|
|
33
|
+
}
|
|
34
|
+
const { results } = evaluateTargets(targets, installed);
|
|
35
|
+
return {
|
|
36
|
+
verdict: results.some((item) => item.verdict === "warn") ? "warn" : "pass",
|
|
37
|
+
summary: uniqueSummary(results),
|
|
38
|
+
items: results.length,
|
|
39
|
+
results
|
|
40
|
+
};
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
// A gate that silently ignores a mis-shaped `installed` returns a false pass —
|
|
44
|
+
// the worst failure mode for an install check. Accept the array and
|
|
45
|
+
// {tools,skills} forms, plus `mf scan --json` output (wrapped in `report`);
|
|
46
|
+
// anything else non-empty is an input error.
|
|
47
|
+
function installedPool(installed) {
|
|
48
|
+
if (installed == null) return [];
|
|
49
|
+
if (Array.isArray(installed)) return installed;
|
|
50
|
+
if (typeof installed !== "object") {
|
|
51
|
+
throw new Error("installed 需要能力数组、{ tools, skills } 或 mf scan --json 的 report");
|
|
52
|
+
}
|
|
53
|
+
if (Object.keys(installed).length === 0) return [];
|
|
54
|
+
const source =
|
|
55
|
+
installed.report && typeof installed.report === "object" ? installed.report : installed;
|
|
56
|
+
const tools = source.tools ?? [];
|
|
57
|
+
const skills = source.skills ?? [];
|
|
58
|
+
if (!Array.isArray(tools) || !Array.isArray(skills)) {
|
|
59
|
+
throw new Error("installed.tools / installed.skills 必须是数组");
|
|
60
|
+
}
|
|
61
|
+
if (!Array.isArray(source.tools) && !Array.isArray(source.skills)) {
|
|
62
|
+
throw new Error("installed 需要能力数组、{ tools, skills } 或 mf scan --json 的 report");
|
|
63
|
+
}
|
|
64
|
+
return [...tools, ...skills];
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
function evaluateTargets(targets, installed) {
|
|
68
|
+
const pool = installedPool(installed);
|
|
14
69
|
const entries = pool.map(normalizeEntry).filter(Boolean);
|
|
15
70
|
|
|
16
71
|
const tools = [];
|
|
17
72
|
const skills = [];
|
|
18
73
|
for (const entry of entries) (isToolLike(entry) ? tools : skills).push(entry);
|
|
19
|
-
(isToolLike(target) ? tools : skills).push(target);
|
|
74
|
+
for (const target of targets) (isToolLike(target) ? tools : skills).push(target);
|
|
20
75
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
const
|
|
26
|
-
|
|
27
|
-
|
|
76
|
+
// The gate only reports findings touching the new capabilities; skipping
|
|
77
|
+
// installed × installed pairs keeps a 500-capability pool linear-ish.
|
|
78
|
+
const pairScope = new Set(targets.map((target) => target.name));
|
|
79
|
+
const report = lintSuite({ tools, skills }, { pairScope });
|
|
80
|
+
const pairFindings = report.findings.filter((finding) => finding.pair);
|
|
81
|
+
const results = targets.map((target) => {
|
|
82
|
+
const findings = report.findings
|
|
83
|
+
.filter((finding) => touches(finding, target.name))
|
|
84
|
+
.map((finding) => shape(finding, target.name));
|
|
85
|
+
const summary = { error: 0, warn: 0, info: 0 };
|
|
86
|
+
for (const finding of findings) summary[finding.severity] += 1;
|
|
87
|
+
const family = familyOf(pairFindings, target.name);
|
|
88
|
+
return {
|
|
89
|
+
name: target.name,
|
|
90
|
+
verdict: summary.error + summary.warn > 0 ? "warn" : "pass",
|
|
91
|
+
summary,
|
|
92
|
+
findings,
|
|
93
|
+
...(family.length > 1 ? { family } : {})
|
|
94
|
+
};
|
|
95
|
+
});
|
|
96
|
+
return { results };
|
|
97
|
+
}
|
|
28
98
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
99
|
+
// A pair finding touches both items, so the batch summary is deduped by
|
|
100
|
+
// subject/with; per-item summaries keep the same finding under each item.
|
|
101
|
+
function uniqueSummary(results) {
|
|
102
|
+
const seen = new Map();
|
|
103
|
+
for (const result of results) {
|
|
104
|
+
for (const finding of result.findings) {
|
|
105
|
+
const key = [
|
|
106
|
+
finding.code,
|
|
107
|
+
finding.severity,
|
|
108
|
+
finding.subject,
|
|
109
|
+
finding.with ?? "",
|
|
110
|
+
finding.param ?? ""
|
|
111
|
+
].join("|");
|
|
112
|
+
if (!seen.has(key)) seen.set(key, finding);
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
const summary = { error: 0, warn: 0, info: 0 };
|
|
116
|
+
for (const finding of seen.values()) summary[finding.severity] += 1;
|
|
117
|
+
return summary;
|
|
35
118
|
}
|
|
36
119
|
|
|
37
120
|
function normalizeEntry(entry) {
|
|
@@ -42,6 +125,11 @@ function normalizeEntry(entry) {
|
|
|
42
125
|
return {
|
|
43
126
|
name,
|
|
44
127
|
description: typeof entry.description === "string" ? entry.description : "",
|
|
128
|
+
// Scan marks descriptions assembled at runtime; without this the gate
|
|
129
|
+
// treats every Claude-Code-style tool as "description missing".
|
|
130
|
+
...(typeof entry.descriptionSource === "string"
|
|
131
|
+
? { descriptionSource: entry.descriptionSource }
|
|
132
|
+
: {}),
|
|
45
133
|
parameters: parameters && typeof parameters === "object" ? parameters : null
|
|
46
134
|
};
|
|
47
135
|
}
|
|
@@ -61,10 +149,22 @@ function touches(finding, name) {
|
|
|
61
149
|
return Array.isArray(finding.pair) && finding.pair.includes(name);
|
|
62
150
|
}
|
|
63
151
|
|
|
152
|
+
// An install gate should only warn about things that change model behavior:
|
|
153
|
+
// selection conflicts, unsourceable required params, and hard schema errors.
|
|
154
|
+
// Description and naming rules stay informative — otherwise every mature
|
|
155
|
+
// gateway tool warns on install and the signal becomes noise.
|
|
156
|
+
const GATE_CRITICAL_CODES = new Set([
|
|
157
|
+
"desc-missing",
|
|
158
|
+
"param-type-missing",
|
|
159
|
+
"required-unsourced",
|
|
160
|
+
"pair-no-boundary",
|
|
161
|
+
"skill-pair-no-boundary"
|
|
162
|
+
]);
|
|
163
|
+
|
|
64
164
|
function shape(finding, name) {
|
|
65
165
|
const other = (finding.pair ?? []).find((item) => item !== name);
|
|
66
166
|
return {
|
|
67
|
-
severity: finding.severity,
|
|
167
|
+
severity: GATE_CRITICAL_CODES.has(finding.code) ? finding.severity : "info",
|
|
68
168
|
code: finding.code,
|
|
69
169
|
subject: name,
|
|
70
170
|
...(finding.param ? { param: finding.param } : {}),
|
|
@@ -101,18 +201,41 @@ function familyOf(pairs, name) {
|
|
|
101
201
|
return [...seen].sort();
|
|
102
202
|
}
|
|
103
203
|
|
|
204
|
+
function findingLines(finding) {
|
|
205
|
+
const param = finding.param ? `.${finding.param}` : "";
|
|
206
|
+
const withWhat = finding.with ? ` × ${finding.with}` : "";
|
|
207
|
+
const lines = [
|
|
208
|
+
`[${finding.severity}] ${finding.code} ${finding.subject}${param}${withWhat}`,
|
|
209
|
+
` ${finding.message}`
|
|
210
|
+
];
|
|
211
|
+
if (finding.suggestion) lines.push(` ${finding.suggestion}`);
|
|
212
|
+
return lines;
|
|
213
|
+
}
|
|
214
|
+
|
|
104
215
|
export function formatCheck(result) {
|
|
105
216
|
const lines = [
|
|
106
217
|
`verdict: ${result.verdict}(error ${result.summary.error} / warn ${result.summary.warn} / info ${result.summary.info})`
|
|
107
218
|
];
|
|
108
219
|
if (result.findings.length === 0) lines.push("没有发现问题");
|
|
109
|
-
for (const finding of result.findings)
|
|
110
|
-
const param = finding.param ? `.${finding.param}` : "";
|
|
111
|
-
const withWhat = finding.with ? ` × ${finding.with}` : "";
|
|
112
|
-
lines.push(`[${finding.severity}] ${finding.code} ${finding.subject}${param}${withWhat}`);
|
|
113
|
-
lines.push(` ${finding.message}`);
|
|
114
|
-
if (finding.suggestion) lines.push(` ${finding.suggestion}`);
|
|
115
|
-
}
|
|
220
|
+
for (const finding of result.findings) lines.push(...findingLines(finding));
|
|
116
221
|
if (result.family) lines.push("", `族:${result.family.join(", ")}`);
|
|
117
222
|
return lines.join("\n");
|
|
118
223
|
}
|
|
224
|
+
|
|
225
|
+
export function formatCheckMany(result) {
|
|
226
|
+
const lines = [
|
|
227
|
+
`verdict: ${result.verdict}(${result.items} 个新能力;去重后 error ${result.summary.error} / warn ${result.summary.warn} / info ${result.summary.info})`
|
|
228
|
+
];
|
|
229
|
+
result.results.forEach((item, index) => {
|
|
230
|
+
lines.push(
|
|
231
|
+
"",
|
|
232
|
+
`[${index + 1}] ${item.name} ${item.verdict}(error ${item.summary.error} / warn ${item.summary.warn} / info ${item.summary.info})`
|
|
233
|
+
);
|
|
234
|
+
if (item.findings.length === 0) lines.push(" 没有发现问题");
|
|
235
|
+
for (const finding of item.findings) {
|
|
236
|
+
for (const line of findingLines(finding)) lines.push(` ${line}`);
|
|
237
|
+
}
|
|
238
|
+
if (item.family) lines.push(` 族:${item.family.join(", ")}`);
|
|
239
|
+
});
|
|
240
|
+
return lines.join("\n");
|
|
241
|
+
}
|
package/lib/cluster.js
CHANGED
|
@@ -1,4 +1,10 @@
|
|
|
1
|
-
import {
|
|
1
|
+
import {
|
|
2
|
+
prepareItem,
|
|
3
|
+
pairSimilarityPrepared,
|
|
4
|
+
familyScorePrepared,
|
|
5
|
+
nameTokens,
|
|
6
|
+
GENERIC_NAME_TOKENS
|
|
7
|
+
} from "./cases.js";
|
|
2
8
|
|
|
3
9
|
// A shared domain noun ("session", "goal", "tool") says these tools serve one
|
|
4
10
|
// purpose even when their verbs and parameter shapes differ — exactly what
|
|
@@ -21,12 +27,13 @@ const BOUNDARY_FAMILY = 0.6;
|
|
|
21
27
|
|
|
22
28
|
export function clusterTools(tools, { threshold = 0.5, maxSize = 10 } = {}) {
|
|
23
29
|
const scores = tools.map(() => new Array(tools.length).fill(0));
|
|
30
|
+
const prepared = tools.map((tool) => prepareItem(tool));
|
|
24
31
|
const pairs = [];
|
|
25
32
|
const edges = [];
|
|
26
33
|
for (let i = 0; i < tools.length; i += 1) {
|
|
27
34
|
for (let j = i + 1; j < tools.length; j += 1) {
|
|
28
|
-
const { similarity, basis } =
|
|
29
|
-
const family =
|
|
35
|
+
const { similarity, basis } = pairSimilarityPrepared(prepared[i], prepared[j]);
|
|
36
|
+
const family = familyScorePrepared(prepared[i], prepared[j]);
|
|
30
37
|
const link = Math.max(similarity, family);
|
|
31
38
|
scores[i][j] = link;
|
|
32
39
|
scores[j][i] = link;
|
package/lib/lint.js
CHANGED
|
@@ -57,7 +57,12 @@ const BOUNDARY_HINT = /(不适用|不要用于|不用于|不能用于|而非|区
|
|
|
57
57
|
// Chinese descriptions are compared as bigrams, so mid-0.3s similarity is noise.
|
|
58
58
|
const NO_BOUNDARY_SIMILARITY = 0.5;
|
|
59
59
|
|
|
60
|
-
|
|
60
|
+
// `pairScope` (a Set of names) is for incremental gates: only emit pair
|
|
61
|
+
// findings that touch one of the new capabilities. Installed × installed pairs
|
|
62
|
+
// are computed and then discarded by the caller, which dominates large sets.
|
|
63
|
+
export function lintSuite(suite, { pairScope } = {}) {
|
|
64
|
+
const inScope = (pair) => !pairScope || pairScope.has(pair.a) || pairScope.has(pair.b);
|
|
65
|
+
const toolByName = new Map(suite.tools.map((tool) => [tool.name, tool]));
|
|
61
66
|
const findings = [];
|
|
62
67
|
const add = (severity, code, tool, message, extra = {}) =>
|
|
63
68
|
findings.push({ severity, code, kind: "tool", tool, message, ...extra });
|
|
@@ -205,9 +210,10 @@ export function lintSuite(suite) {
|
|
|
205
210
|
}
|
|
206
211
|
|
|
207
212
|
for (const pair of suspectedPairs(suite.tools)) {
|
|
213
|
+
if (!inScope(pair)) continue;
|
|
208
214
|
const similarity = pair.similarity.toFixed(2);
|
|
209
215
|
const declaresBoundary = [pair.a, pair.b].some((name) => {
|
|
210
|
-
const tool =
|
|
216
|
+
const tool = toolByName.get(name);
|
|
211
217
|
return BOUNDARY_HINT.test(tool?.description ?? "");
|
|
212
218
|
});
|
|
213
219
|
if (declaresBoundary) {
|
|
@@ -223,8 +229,8 @@ export function lintSuite(suite) {
|
|
|
223
229
|
}
|
|
224
230
|
}
|
|
225
231
|
const reason = pair.basis === "structural" ? "名字与参数形状接近" : "描述相近";
|
|
226
|
-
const toolA =
|
|
227
|
-
const toolB =
|
|
232
|
+
const toolA = toolByName.get(pair.a);
|
|
233
|
+
const toolB = toolByName.get(pair.b);
|
|
228
234
|
add("warn", "pair-no-boundary", pair.a, `与 ${pair.b} ${reason}(相似度 ${similarity}),且两边都没写“不适用于”边界,模型只能靠相似度硬猜`, {
|
|
229
235
|
pair: [pair.a, pair.b],
|
|
230
236
|
suggestion: boundaryHint(toolA, toolB).suggestion
|
|
@@ -236,6 +242,7 @@ export function lintSuite(suite) {
|
|
|
236
242
|
// question is narrower than for tools: is the description specific enough to
|
|
237
243
|
// be triggered, and does it collide with another installed skill?
|
|
238
244
|
const skills = suite.skills ?? [];
|
|
245
|
+
const skillByName = new Map(skills.map((skill) => [skill.name, skill]));
|
|
239
246
|
for (const skill of skills) {
|
|
240
247
|
const description = String(skill.description ?? "").trim();
|
|
241
248
|
if (!description) {
|
|
@@ -258,9 +265,10 @@ export function lintSuite(suite) {
|
|
|
258
265
|
threshold: SKILL_PAIR_SIMILARITY,
|
|
259
266
|
extraStop: SKILL_BOILERPLATE
|
|
260
267
|
})) {
|
|
268
|
+
if (!inScope(pair)) continue;
|
|
261
269
|
const similarity = pair.similarity.toFixed(2);
|
|
262
270
|
const declaresBoundary = [pair.a, pair.b].some((name) => {
|
|
263
|
-
const skill =
|
|
271
|
+
const skill = skillByName.get(name);
|
|
264
272
|
return BOUNDARY_HINT.test(skill?.description ?? "");
|
|
265
273
|
});
|
|
266
274
|
// Same-vendor siblings ("feishu-doc" / "feishu-wiki") share the vendor token
|
package/lib/mcp-server.js
CHANGED
|
@@ -11,6 +11,10 @@ import { writeFileAtomic } from "./fsutil.js";
|
|
|
11
11
|
import { probeStdioServer, probeHttpServer } from "./mcp-probe.js";
|
|
12
12
|
import { addCases, computeCoverage } from "./cases.js";
|
|
13
13
|
import { auditSuite } from "./audit.js";
|
|
14
|
+
import { check, checkMany } from "./check.js";
|
|
15
|
+
import { lintSuite } from "./lint.js";
|
|
16
|
+
import { clusterTools } from "./cluster.js";
|
|
17
|
+
import { compareScorecards } from "./compare.js";
|
|
14
18
|
|
|
15
19
|
const MF_PATH = fileURLToPath(new URL("../mf.js", import.meta.url));
|
|
16
20
|
const PACKAGE_JSON = new URL("../package.json", import.meta.url);
|
|
@@ -125,6 +129,78 @@ export async function startMcpServer() {
|
|
|
125
129
|
)
|
|
126
130
|
);
|
|
127
131
|
|
|
132
|
+
server.registerTool(
|
|
133
|
+
"lint",
|
|
134
|
+
{
|
|
135
|
+
title: "Lint suite",
|
|
136
|
+
description:
|
|
137
|
+
"静态检查 tools 与 skills 的描述、参数与工具对边界(不调用模型)。返回 counts / findings / profilePairs。",
|
|
138
|
+
inputSchema: { suite: z.string().optional().describe("套件路径,默认 ./mf.suite.json") }
|
|
139
|
+
},
|
|
140
|
+
async ({ suite }) =>
|
|
141
|
+
json(
|
|
142
|
+
lintSuite(
|
|
143
|
+
await loadSuite(suitePath(suite), { allowEmptyCases: true, allowEmptyTools: true })
|
|
144
|
+
)
|
|
145
|
+
)
|
|
146
|
+
);
|
|
147
|
+
|
|
148
|
+
server.registerTool(
|
|
149
|
+
"check",
|
|
150
|
+
{
|
|
151
|
+
title: "Install gate",
|
|
152
|
+
description:
|
|
153
|
+
"安装门禁:把新能力与已装能力做静态比较,返回 pass/warn、findings 与竞争族。传 manifest 检查单个;传 manifests 检查一包(新能力之间也会互比)。不阻断、不调用模型、无网络。",
|
|
154
|
+
inputSchema: {
|
|
155
|
+
manifest: z.any().optional().describe("新能力 {name, description, parameters?} 或 tool schema"),
|
|
156
|
+
manifests: z.array(z.any()).optional().describe("一次安装一包能力时的数组"),
|
|
157
|
+
installed: z.any().optional().describe("已装能力数组,或 {tools: [], skills: []}")
|
|
158
|
+
}
|
|
159
|
+
},
|
|
160
|
+
async ({ manifest, manifests, installed }) => {
|
|
161
|
+
if (manifests) return json(checkMany({ manifests, installed }));
|
|
162
|
+
return json(check({ manifest, installed }));
|
|
163
|
+
}
|
|
164
|
+
);
|
|
165
|
+
|
|
166
|
+
server.registerTool(
|
|
167
|
+
"cluster",
|
|
168
|
+
{
|
|
169
|
+
title: "Cluster tools",
|
|
170
|
+
description:
|
|
171
|
+
"聚类工具,给出挂载分组、跨组高风险与族内边界清单(每族要补哪句选择条件)。不调用模型。",
|
|
172
|
+
inputSchema: {
|
|
173
|
+
suite: z.string().optional().describe("套件路径,默认 ./mf.suite.json"),
|
|
174
|
+
threshold: z.number().min(0).max(1).optional().describe("聚类阈值,默认 0.5"),
|
|
175
|
+
maxSize: z.number().int().positive().optional().describe("单组最大工具数,默认 10")
|
|
176
|
+
}
|
|
177
|
+
},
|
|
178
|
+
async ({ suite, threshold, maxSize }) => {
|
|
179
|
+
const loaded = await loadSuite(suitePath(suite), { allowEmptyCases: true });
|
|
180
|
+
return json(clusterTools(loaded.tools, { threshold, maxSize }));
|
|
181
|
+
}
|
|
182
|
+
);
|
|
183
|
+
|
|
184
|
+
server.registerTool(
|
|
185
|
+
"compare",
|
|
186
|
+
{
|
|
187
|
+
title: "Compare scorecards",
|
|
188
|
+
description:
|
|
189
|
+
"比较两份评分卡 JSON:可比性、升降、干扰归因与混淆矩阵。只读文件,不调用模型。",
|
|
190
|
+
inputSchema: {
|
|
191
|
+
before: z.string().describe("旧评分卡 JSON 路径"),
|
|
192
|
+
after: z.string().describe("新评分卡 JSON 路径")
|
|
193
|
+
}
|
|
194
|
+
},
|
|
195
|
+
async ({ before, after }) => {
|
|
196
|
+
const [oldText, newText] = await Promise.all([
|
|
197
|
+
readFile(resolve(before), "utf8"),
|
|
198
|
+
readFile(resolve(after), "utf8")
|
|
199
|
+
]);
|
|
200
|
+
return json(compareScorecards(JSON.parse(oldText), JSON.parse(newText)));
|
|
201
|
+
}
|
|
202
|
+
);
|
|
203
|
+
|
|
128
204
|
server.registerTool(
|
|
129
205
|
"add_cases",
|
|
130
206
|
{
|
package/lib/profile.js
CHANGED
|
@@ -2,17 +2,28 @@ import { pct } from "./score.js";
|
|
|
2
2
|
|
|
3
3
|
export function normalizeProfiles(raw, toolNames) {
|
|
4
4
|
if (raw === undefined || raw === null) return null;
|
|
5
|
-
if (typeof raw !== "object"
|
|
6
|
-
throw new Error('suite "profiles" must be an object
|
|
5
|
+
if (typeof raw !== "object") {
|
|
6
|
+
throw new Error('suite "profiles" must be an object or an array of groups');
|
|
7
7
|
}
|
|
8
8
|
const known = new Set(toolNames);
|
|
9
9
|
const profiles = [];
|
|
10
|
-
|
|
11
|
-
|
|
10
|
+
const seen = new Set();
|
|
11
|
+
// Platform registries often hand us an array of
|
|
12
|
+
// { agentType, tools, skills } rather than { "group": ["tool"] }.
|
|
13
|
+
const entries = Array.isArray(raw)
|
|
14
|
+
? raw.map((value, index) => [String(index), value])
|
|
15
|
+
: Object.entries(raw);
|
|
16
|
+
for (const [key, value] of entries) {
|
|
17
|
+
// Array form: { "group": ["tool", ...] }. Object form (platform registries
|
|
18
|
+
// often look like this): { "0": { agentType, tools, skills } }.
|
|
19
|
+
const list = Array.isArray(value) ? value : value?.tools;
|
|
20
|
+
const name = String(Array.isArray(value) ? key : value?.name ?? value?.agentType ?? key).trim();
|
|
12
21
|
if (!name) throw new Error('suite "profiles": group name must be non-empty');
|
|
13
22
|
if (!Array.isArray(list) || list.length === 0) {
|
|
14
23
|
throw new Error(`profile "${name}": must be a non-empty array of tool names`);
|
|
15
24
|
}
|
|
25
|
+
if (seen.has(name)) throw new Error(`profile "${name}": duplicate group name`);
|
|
26
|
+
seen.add(name);
|
|
16
27
|
const tools = [];
|
|
17
28
|
for (const item of list) {
|
|
18
29
|
if (typeof item !== "string" || item.trim() === "") {
|
package/mf.js
CHANGED
|
@@ -11,7 +11,7 @@ import { probeStdioServer, probeHttpServer } from "./lib/mcp-probe.js";
|
|
|
11
11
|
import { loadAdapterTools } from "./lib/adapter-import.js";
|
|
12
12
|
import { addCases, computeCoverage, formatCoverage } from "./lib/cases.js";
|
|
13
13
|
import { lintSuite, formatLint } from "./lib/lint.js";
|
|
14
|
-
import { check, formatCheck } from "./lib/check.js";
|
|
14
|
+
import { check, checkMany, formatCheck, formatCheckMany } from "./lib/check.js";
|
|
15
15
|
import { auditSuite, formatAudit } from "./lib/audit.js";
|
|
16
16
|
import { suiteFingerprint, hashText, hashOf } from "./lib/fingerprint.js";
|
|
17
17
|
import { runExecutionChecks } from "./lib/eval-exec.js";
|
|
@@ -75,7 +75,7 @@ Usage:
|
|
|
75
75
|
mf coverage [--suite <file>] [--json]
|
|
76
76
|
mf audit [--suite <file>] [--out <file>] [--json] # 静态审计:lint + 分组边界 + 用例覆盖 + 挂载
|
|
77
77
|
mf lint [--suite <file>] [--json] # 静态检查 tools 与 skills,不调用模型
|
|
78
|
-
mf check --manifest <file> [--installed <file>] [--json] # 安装门禁:新能力 ×
|
|
78
|
+
mf check (--manifest <file> | --manifests <file>) [--installed <file>] [--json] # 安装门禁:新能力 × 已装能力(--manifests 支持一包)
|
|
79
79
|
mf compare <before.json> <after.json> [--json]
|
|
80
80
|
mf ablate --tool <name> [--suite <file>] [--profile <group>] [--models <ids>] [--max-cost <cny>] [--progress] [--json]
|
|
81
81
|
mf cluster [--suite <file>] [--threshold <0.5>] [--max-size <10>] [--out <file>] [--json] # 建议挂载分组 + 同族候选
|
|
@@ -482,7 +482,7 @@ export async function runScan(options, io = console) {
|
|
|
482
482
|
|
|
483
483
|
if (options.json) {
|
|
484
484
|
io.log(JSON.stringify({ report: redactReport(report), draft }, null, 2));
|
|
485
|
-
return report.tools.length > 0 ? 0 : 3;
|
|
485
|
+
return report.tools.length > 0 || report.skills.length > 0 ? 0 : 3;
|
|
486
486
|
}
|
|
487
487
|
|
|
488
488
|
io.log(formatScanSummary(report));
|
|
@@ -497,7 +497,7 @@ export async function runScan(options, io = console) {
|
|
|
497
497
|
io.log(`wrote ${reportPath}`);
|
|
498
498
|
}
|
|
499
499
|
|
|
500
|
-
return report.tools.length > 0 ? 0 : 3;
|
|
500
|
+
return report.tools.length > 0 || report.skills.length > 0 ? 0 : 3;
|
|
501
501
|
}
|
|
502
502
|
|
|
503
503
|
export async function runCaseAdd(options, io = console) {
|
|
@@ -584,11 +584,24 @@ export async function runSandboxCheck(options, io = console) {
|
|
|
584
584
|
// Advisory install gate: static only, never blocks. Exit code stays 0 for a
|
|
585
585
|
// warn verdict so a platform can log it without failing the install.
|
|
586
586
|
export async function runCheck(options, io = console) {
|
|
587
|
-
if (!options.manifest) throw new Error("--manifest <file> 必填(新能力的 [name, description, parameters?])");
|
|
588
|
-
const manifest = JSON.parse(await readFile(resolve(options.manifest), "utf8"));
|
|
589
587
|
const installed = options.installed
|
|
590
588
|
? JSON.parse(await readFile(resolve(options.installed), "utf8"))
|
|
591
589
|
: [];
|
|
590
|
+
|
|
591
|
+
if (options.manifests) {
|
|
592
|
+
const manifests = JSON.parse(await readFile(resolve(options.manifests), "utf8"));
|
|
593
|
+
const result = checkMany({ manifests, installed });
|
|
594
|
+
noteUsage({
|
|
595
|
+
checkVerdict: result.verdict,
|
|
596
|
+
checkFindings: Object.values(result.summary).reduce((sum, count) => sum + count, 0)
|
|
597
|
+
});
|
|
598
|
+
io.log(options.json ? JSON.stringify(result, null, 2) : formatCheckMany(result));
|
|
599
|
+
return 0;
|
|
600
|
+
}
|
|
601
|
+
if (!options.manifest) {
|
|
602
|
+
throw new Error("--manifest 或 --manifests <file> 必填([name, description, parameters?])");
|
|
603
|
+
}
|
|
604
|
+
const manifest = JSON.parse(await readFile(resolve(options.manifest), "utf8"));
|
|
592
605
|
const result = check({ manifest, installed });
|
|
593
606
|
noteUsage({ checkVerdict: result.verdict, checkFindings: result.findings.length });
|
|
594
607
|
io.log(options.json ? JSON.stringify(result, null, 2) : formatCheck(result));
|
|
@@ -1009,6 +1022,7 @@ function parseArgs(argv) {
|
|
|
1009
1022
|
dryRun: false,
|
|
1010
1023
|
tool: null,
|
|
1011
1024
|
manifest: null,
|
|
1025
|
+
manifests: null,
|
|
1012
1026
|
installed: null,
|
|
1013
1027
|
threshold: 0.5,
|
|
1014
1028
|
maxSize: 10,
|
|
@@ -1077,6 +1091,7 @@ function parseArgs(argv) {
|
|
|
1077
1091
|
case "--dry-run": options.dryRun = true; break;
|
|
1078
1092
|
case "--tool": options.tool = value(); break;
|
|
1079
1093
|
case "--manifest": options.manifest = resolve(value()); break;
|
|
1094
|
+
case "--manifests": options.manifests = resolve(value()); break;
|
|
1080
1095
|
case "--installed": options.installed = resolve(value()); break;
|
|
1081
1096
|
case "--threshold": {
|
|
1082
1097
|
const raw = Number(value());
|