modelfunction-cli 0.4.2 → 0.4.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -5,7 +5,7 @@
5
5
  ## 安装
6
6
 
7
7
  ```bash
8
- npx -y modelfunction-cli@0.2.0 --version
8
+ npx -y modelfunction-cli@0.4.3 --version
9
9
  # 或
10
10
  npm i -D modelfunction-cli
11
11
  npx mf --version
@@ -72,7 +72,7 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
72
72
  | `mf check` | 安装门禁:一个新能力 × 已装能力(`--manifest` + `--installed`),纯静态、返回 pass/warn、不阻断;也可 `import { check } from "modelfunction-cli"` |
73
73
  | `mf compare` | 跨版本比较、干扰归因、混淆矩阵;两份报告都有分组时按组对比 |
74
74
  | `mf ablate` | 留一法:去掉某个工具再跑一遍,量化它是否干扰其他工具;`--profile` 限定在某个挂载组内 |
75
- | `mf cluster` | 聚类工具,建议挂载分组;参数形状拆散的同一族(如 `sessions_*`)列为「同族候选」;`--out` 把分组写成套件的 `profiles` |
75
+ | `mf cluster` | 聚类工具,建议挂载分组;参数形状拆散的同一族(如 `sessions_*`)列为「同族候选」;并输出「族内边界清单」(每族会互抢的对 + 每个工具可用的区分词);`--out` 把分组写成套件的 `profiles` |
76
76
  | `mf exec` | 单工具执行 + 指标(Node / Python),支持 `--allow-read/write/net`、超时、sideEffects 准入 |
77
77
  | `mf eval --execute` | 一次跑完选择 + 参数 + 运行断言(`toolInput` / `expect`) |
78
78
  | `mf sandbox check` | 探测本机可用隔离引擎与降级建议 |
@@ -102,6 +102,8 @@ mf eval --suite mf.suite.json --models internal-ds \
102
102
 
103
103
  配套的三个检查:`mf lint` 列出组间高风险对(相似工具分属不同组,合并挂载会互抢);`mf coverage` 按组看谁还缺用例;两次 `--profile` 报告之间用 `mf compare` 看各组升降(分组定义变了会明确标注不可比)。组内还有干扰时,用 `mf ablate --profile <组> --tool <工具>` 做组内留一法,看拆掉谁最划算。
104
104
 
105
+ `mf cluster` 还会输出**族内边界清单**:把会互抢的工具对按连通关系分成「族」,每族列出各工具名称里的独有词(名称没有独有词时退到独有参数)作为区分轴,并标出共享参数(不能当判据)。这是写给作者的动作项——每个工具补一句"何时用我、何时用对方"即可,不需要先跑模型。`--json` 的 `boundaries[]` 是该清单的结构化版本(`axes` / `sharedParams` / `pairs` / `suggestion`),平台方可以直接消费。
106
+
105
107
  ## MCP 配置
106
108
 
107
109
  ```json
@@ -109,7 +111,7 @@ mf eval --suite mf.suite.json --models internal-ds \
109
111
  "mcpServers": {
110
112
  "modelfunction": {
111
113
  "command": "npx",
112
- "args": ["-y", "modelfunction-cli@0.2.0", "mcp"]
114
+ "args": ["-y", "modelfunction-cli@0.4.3", "mcp"]
113
115
  }
114
116
  }
115
117
  }
@@ -117,6 +119,10 @@ mf eval --suite mf.suite.json --models internal-ds \
117
119
 
118
120
  暴露:`scan_tools`、`list_suite`、`coverage`、`add_cases`、`run_eval`(`run_eval` 支持 `profile` 与 `repeat` 参数:传 `all` 或组名按挂载组评分,`repeat` 取多数结果)。
119
121
 
122
+ ## 契约与版本
123
+
124
+ 对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md),自 0.4.3 起冻结为 v1:v1 内只做加法,重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
125
+
120
126
  ## 数据边界
121
127
 
122
128
  - 本地运行,无遥测、无服务端地址
@@ -131,7 +137,7 @@ mf eval --suite mf.suite.json --models internal-ds \
131
137
 
132
138
  ```bash
133
139
  # 1. 把 mf 挂成 MCP server
134
- claude mcp add modelfunction -- npx -y modelfunction-cli@0.2.0 mcp
140
+ claude mcp add modelfunction -- npx -y modelfunction-cli@0.4.3 mcp
135
141
  ```
136
142
 
137
143
  ```text
package/SCHEMA.md ADDED
@@ -0,0 +1,200 @@
1
+ # Schemas v1
2
+
3
+ Frozen at `modelfunction-cli@0.4.3`。这里描述 CLI 与库对外的稳定契约。人类可读文本(表格、提示语)不属于契约;**JSON 字段**属于。
4
+
5
+ ## 兼容政策
6
+
7
+ - v1 内只做**加法**:新增字段、新增命令、新增可选参数。消费方必须忽略未知字段。
8
+ - 重命名、删除、单位/语义变化 → major 版本;至少提前 **2 个 minor 或 90 天**标记弃用。
9
+ - `mf --version` 与包版本是唯一版本锚点;报告里不重复写 schema 版本。
10
+ - 未复核用例(`reviewed: false`)永远不计入正式分数,这条不受版本影响。
11
+
12
+ ## 1. suite.json v1
13
+
14
+ ```json
15
+ {
16
+ "name": "my-agent",
17
+ "description": "可选",
18
+ "tools": [ ... ],
19
+ "cases": [ ... ],
20
+ "skills": [ ... ],
21
+ "profiles": { "group-a": ["tool_x", "tool_y"] },
22
+ "execution": { "runtime": "node", "entry": "tools/index.mjs" }
23
+ }
24
+ ```
25
+
26
+ ### tools[]
27
+
28
+ 两种写法:
29
+
30
+ ```json
31
+ { "name": "search_docs", "description": "…", "parameters": { "type": "object", "properties": {} } }
32
+ { "ref": "fixtures/tools/search_docs.json" }
33
+ ```
34
+
35
+ - `name` 必填、非空、同套件内唯一。
36
+ - `parameters` 可用 `input` 代替(function.json 兼容)。缺省为空对象 schema。
37
+ - `descriptionSource: "runtime"`:描述由代码运行时生成,静态检查只报 info。
38
+ - `sideEffects`: `"none" | "read" | "write" | "external"`,执行准入用。
39
+ - `ref` 指向 function.json;读取 `name` / `description` / `input` 或 `parameters`。
40
+
41
+ ### cases[]
42
+
43
+ ```json
44
+ {
45
+ "id": "c1",
46
+ "input": "帮我把这份文档转成 PDF",
47
+ "expectedTool": "convert_pdf",
48
+ "expectedArgs": { "path": "a.docx" },
49
+ "toolInput": { "path": "a.docx" },
50
+ "expect": { "ok": true, "contains": "pdf", "maxDurationMs": 2000 },
51
+ "note": "",
52
+ "source": "manual",
53
+ "reviewed": true,
54
+ "tags": []
55
+ }
56
+ ```
57
+
58
+ - `input` 必填;`expectedTool: null` 表示负例(不该调用任何工具)。
59
+ - `expectedArgs` 仅在有 `expectedTool` 时可用;值为断言,支持字面量、`$contains`、`$oneOf`、`$exists`。
60
+ - `toolInput` + `expect` 用于 `--execute`:`expect` 支持 `ok`、`contains`、`equals`、`errorType`、`maxDurationMs`、`maxPeakRssMB`。
61
+ - `reviewed` 缺省为 `true`(手写用例);`mf case add` 缺省写入 `source=agent`、`reviewed=false`。
62
+
63
+ ### skills[]
64
+
65
+ 字符串或对象:
66
+
67
+ ```json
68
+ "pdf-report"
69
+ { "name": "pdf-report", "description": "生成 PDF 报告" }
70
+ ```
71
+
72
+ skills 只做静态检查,不参与模型评测。
73
+
74
+ ### profiles
75
+
76
+ 组名 → 工具名数组。所有工具名必须存在;组可重叠。`mf eval --profile` 与 `mf ablate --profile` 依赖它。
77
+
78
+ ### execution
79
+
80
+ `runtime`(`node` | `python`)、`entry`、`python`、`export`、`timeoutMs`、`allowRead`、`allowWrite`、`allowNet`。命令行参数优先于文件。
81
+
82
+ ## 2. 评分卡 JSON(`mf eval --json` / `--out report.json`)
83
+
84
+ ```json
85
+ {
86
+ "suite": {
87
+ "name": "...", "path": "...",
88
+ "tools": ["..."], "cases": 12,
89
+ "fingerprint": "sha256:...",
90
+ "execution": { "runtime": "node", "engine": "node-permission", "entry": "index.mjs", "hash": "..." }
91
+ },
92
+ "models": [ { "model": "dsv4-flash", "total": 12, "reviewedTotal": 12, "selectionAccuracy": 0.83,
93
+ "argAccuracy": 0.9, "passRate": 0.75, "errors": 0, "p50LatencyMs": 900,
94
+ "p95LatencyMs": 1800, "avgCostCny": 0.001, "totalCostCny": 0.012,
95
+ "inputTokens": 1000, "cachedTokens": 0, "outputTokens": 200,
96
+ "rows": [ { "id": "c1", "expectedTool": "...", "actualTool": "...",
97
+ "selectionCorrect": true, "args": { "total": 1, "matched": 1, "failures": [] },
98
+ "passed": true, "reviewed": true, "source": "manual",
99
+ "latencyMs": 900, "usage": { "input": 80, "output": 20 }, "error": null } ] } ],
100
+ "profiles": { "groups": [ ... ] }
101
+ }
102
+ ```
103
+
104
+ - 正式指标只统计 `reviewed` 行;未复核结果单独给出。
105
+ - `--repeat` 时模型对象带 `stability`(翻转用例与多次结果分布)。
106
+ - `--profile` 时每个组含 `profile`、`mounted`、`allInOne`、`delta`、`lifted`、`jammed`。
107
+ - `fingerprint` 由工具与用例内容计算;`mf compare` 用它判断两份报告是否可比。
108
+
109
+ ## 3. 扫描输出(`mf scan`)
110
+
111
+ `--json` 输出 `{ "report": {...}, "draft": {...} }`;默认写 `mf.suite.json`(draft)与 `mf.scan.json`(report)。
112
+
113
+ report:`root`、`tools[]`(`name` / `description` / `parameters` / `origin` / `confidence` / `source` / `note?` / `descriptionSource?`)、`skills[]`、`mcpServers[]`(env 只保留 `envKeys`)、`probes[]`、`sources[]`、`warnings[]`。
114
+
115
+ `scan --json` 在未找到任何工具时退出码为 3。
116
+
117
+ ## 4. lint JSON
118
+
119
+ ```json
120
+ {
121
+ "tools": 47, "skills": 53,
122
+ "counts": { "error": 0, "warn": 5, "info": 12 },
123
+ "findings": [
124
+ { "severity": "warn", "code": "pair-no-boundary", "kind": "tool",
125
+ "tool": "tavily_search", "pair": ["tavily_search", "google_search"],
126
+ "message": "...", "suggestion": "..." }
127
+ ],
128
+ "profilePairs": [ { "a": "group-a", "b": "group-b", "pairs": [ ... ] } ]
129
+ }
130
+ ```
131
+
132
+ - `kind: "skill"` 时用 `skill` 字段代替 `tool`。
133
+ - `param` 出现在参数级发现上;`pair` 出现在工具对/技能对发现上。
134
+ - `profilePairs` 只在套件含 `profiles` 时出现。
135
+
136
+ ## 5. cluster JSON(`mf cluster --json`)
137
+
138
+ ```json
139
+ {
140
+ "threshold": 0.5, "total": 47,
141
+ "clusters": [ { "label": "session", "tools": ["..."] } ],
142
+ "riskyPairs": [ { "a": "...", "b": "...", "similarity": 0.85, "basis": "structural" } ],
143
+ "familyHints": [ { "token": "session", "size": 6, "merged": ["..."], "loose": ["..."] } ],
144
+ "boundaries": [
145
+ { "label": "task", "size": 6, "tools": ["..."],
146
+ "axes": [ { "tool": "task_get", "distinct": ["get"], "paramsOnly": [] } ],
147
+ "sharedParams": ["taskId"],
148
+ "pairs": [ { "a": "task_get", "b": "task_output", "similarity": 0.65, "basis": "structural" } ],
149
+ "suggestion": "..." }
150
+ ]
151
+ }
152
+ ```
153
+
154
+ `boundaries[]` 是族内边界清单的结构化版本;`basis` 为 `lexical` / `structural` / `family`。
155
+
156
+ ## 6. `check()`(安装门禁)
157
+
158
+ ```js
159
+ import { check } from "modelfunction-cli";
160
+ ```
161
+
162
+ 输入 `{ manifest, installed }`:`manifest` 是一个能力(`name` 必填,`description` 可选,`parameters` / `inputSchema` / `input` 任一为 JSON Schema;无 schema 视为 skill)。`installed` 可以是能力数组,或 `{ tools: [], skills: [] }`。
163
+
164
+ 返回:
165
+
166
+ ```json
167
+ {
168
+ "verdict": "pass",
169
+ "summary": { "error": 0, "warn": 0, "info": 1 },
170
+ "findings": [ { "severity": "info", "code": "skill-pair-sibling", "subject": "feishu-wiki",
171
+ "with": "feishu-doc", "message": "...", "suggestion": "..." } ],
172
+ "family": ["feishu-doc", "feishu-wiki"]
173
+ }
174
+ ```
175
+
176
+ - `verdict` 只有 `pass` / `warn`,**没有 fail**;质量判定不阻断安装。
177
+ - 唯一抛错场景:`manifest.name` 缺失或为空。
178
+ - 同步、无网络、无文件系统、无状态。类型声明见 `lib/check.d.ts`。
179
+
180
+ ## 7. MCP 工具
181
+
182
+ | 工具 | 参数 |
183
+ | --- | --- |
184
+ | `scan_tools` | `root?`、`probe?`、`timeoutMs?`、`writeSuite?` |
185
+ | `list_suite` | `suite?` |
186
+ | `coverage` | `suite?` |
187
+ | `add_cases` | `suite?`、`cases[]`、`source?`、`reviewed?`、`dryRun?` |
188
+ | `run_eval` | `suite?`、`models?`、`cases?`、`profile?`、`repeat?`、`failUnder?` |
189
+
190
+ 返回值均为 JSON 文本。
191
+
192
+ ## 8. 退出码
193
+
194
+ | 码 | 含义 |
195
+ | --- | --- |
196
+ | 0 | 成功(`mf check` 的 warn 也是 0) |
197
+ | 1 | 输入/用法错误;`mf compare` 判定不可直接比较 |
198
+ | 2 | `mf eval --fail-under` 未达标,或 `--fail-on-regression` 发现回退 |
199
+ | 3 | `mf exec` 被权限策略拒绝;`mf scan --json` 未找到工具 |
200
+ | 4 | `mf exec` 超时 |
package/lib/cases.js CHANGED
@@ -241,6 +241,44 @@ export function familyScore(x, y) {
241
241
  return paramNames(x).size === 0 && paramNames(y).size === 0 ? 0.6 : 0.2;
242
242
  }
243
243
 
244
+ // Turn a confusable pair into an authoring instruction. We do not invent the
245
+ // semantic rule (which one wins when); we point at the dimensions the author
246
+ // controls — the words unique to each name and the parameters only one side
247
+ // takes — so the boundary sentence can be written without guessing.
248
+ export function boundaryHint(x, y) {
249
+ const left = [...new Set(nameTokens(x.name))].filter((token) => !GENERIC_NAME_TOKENS.has(token));
250
+ const right = [...new Set(nameTokens(y.name))].filter((token) => !GENERIC_NAME_TOKENS.has(token));
251
+ const sharedName = left.filter((token) => right.includes(token)).sort();
252
+ const distinctName = {
253
+ a: left.filter((token) => !right.includes(token)).sort(),
254
+ b: right.filter((token) => !left.includes(token)).sort()
255
+ };
256
+ const aParams = Object.keys(x.parameters?.properties ?? {}).sort();
257
+ const bParams = Object.keys(y.parameters?.properties ?? {}).sort();
258
+ const sharedParams = aParams.filter((param) => bParams.includes(param));
259
+ const paramsOnly = {
260
+ a: aParams.filter((param) => !bParams.includes(param)),
261
+ b: bParams.filter((param) => !aParams.includes(param))
262
+ };
263
+
264
+ let axis;
265
+ if (distinctName.a.length > 0 || distinctName.b.length > 0) {
266
+ axis = `${x.name} 独有 ${distinctName.a.join("/") || "无"},${y.name} 独有 ${distinctName.b.join("/") || "无"}`;
267
+ } else if (paramsOnly.a.length > 0 || paramsOnly.b.length > 0) {
268
+ axis = `${x.name} 要传 ${paramsOnly.a.join("/") || "无"},${y.name} 要传 ${paramsOnly.b.join("/") || "无"}`;
269
+ } else {
270
+ axis = "各自的触发场景";
271
+ }
272
+ const shared = sharedParams.length > 0 ? `;共享参数 ${sharedParams.join("、")} 不能当判据` : "";
273
+ return {
274
+ sharedName,
275
+ distinctName,
276
+ sharedParams,
277
+ paramsOnly,
278
+ suggestion: `在 ${x.name} 和 ${y.name} 的 description 里各补一句选择条件(区分维度:${axis}${shared}),再补一条歧义用例`
279
+ };
280
+ }
281
+
244
282
  export function suspectedPairs(tools, { threshold = PAIR_SIMILARITY, extraStop } = {}) {
245
283
  const pairs = [];
246
284
  for (let i = 0; i < tools.length; i += 1) {
package/lib/check.d.ts ADDED
@@ -0,0 +1,51 @@
1
+ export type Severity = "error" | "warn" | "info";
2
+ export type Verdict = "pass" | "warn";
3
+
4
+ export interface Capability {
5
+ name: string;
6
+ description?: string;
7
+ /** JSON Schema. A capability without one is treated as a prompt-injected skill. */
8
+ parameters?: Record<string, unknown>;
9
+ /** Accepted alias for parameters, matching MCP tool shape. */
10
+ inputSchema?: Record<string, unknown>;
11
+ /** Accepted alias for parameters, matching function.json shape. */
12
+ input?: Record<string, unknown>;
13
+ }
14
+
15
+ export interface InstalledCapabilities {
16
+ tools?: Capability[];
17
+ skills?: Capability[];
18
+ }
19
+
20
+ export interface CheckInput {
21
+ manifest: Capability;
22
+ installed?: Capability[] | InstalledCapabilities;
23
+ }
24
+
25
+ export interface CheckFinding {
26
+ severity: Severity;
27
+ code: string;
28
+ subject: string;
29
+ param?: string;
30
+ /** The installed capability this pair finding is about. */
31
+ with?: string;
32
+ message: string;
33
+ suggestion?: string;
34
+ }
35
+
36
+ export interface CheckResult {
37
+ verdict: Verdict;
38
+ summary: { error: number; warn: number; info: number };
39
+ findings: CheckFinding[];
40
+ /** Connected family of competing installed capabilities, sorted. */
41
+ family?: string[];
42
+ }
43
+
44
+ /**
45
+ * Static install gate: judge one new capability against installed ones.
46
+ * No model call, no network, no file system, no state.
47
+ * @throws only when `manifest.name` is missing or empty.
48
+ */
49
+ export declare function check(input: CheckInput): CheckResult;
50
+
51
+ export declare function formatCheck(result: CheckResult): string;
package/lib/cluster.js CHANGED
@@ -12,9 +12,17 @@ const FAMILY_HINT_STOP = new Set([
12
12
  "save", "load", "merge", "sync", "scan", "check", "compare", "exec", "probe"
13
13
  ]);
14
14
 
15
+ // A pair below the clustering link but above these marks is still confusable
16
+ // enough to need a boundary sentence: same lint threshold for wording/shape,
17
+ // plus the stronger family scores (shared distinctive token with parameter
18
+ // overlap, two shared tokens, or two zero-parameter tools).
19
+ const BOUNDARY_SIMILARITY = 0.35;
20
+ const BOUNDARY_FAMILY = 0.6;
21
+
15
22
  export function clusterTools(tools, { threshold = 0.5, maxSize = 10 } = {}) {
16
23
  const scores = tools.map(() => new Array(tools.length).fill(0));
17
24
  const pairs = [];
25
+ const edges = [];
18
26
  for (let i = 0; i < tools.length; i += 1) {
19
27
  for (let j = i + 1; j < tools.length; j += 1) {
20
28
  const { similarity, basis } = pairSimilarity(tools[i], tools[j]);
@@ -25,6 +33,9 @@ export function clusterTools(tools, { threshold = 0.5, maxSize = 10 } = {}) {
25
33
  if (link >= threshold) {
26
34
  pairs.push({ i, j, similarity, basis: family > similarity ? "family" : basis });
27
35
  }
36
+ if (similarity >= BOUNDARY_SIMILARITY || family >= BOUNDARY_FAMILY) {
37
+ edges.push({ i, j, similarity, basis, family });
38
+ }
28
39
  }
29
40
  }
30
41
 
@@ -70,7 +81,119 @@ export function clusterTools(tools, { threshold = 0.5, maxSize = 10 } = {}) {
70
81
  }))
71
82
  .sort((x, y) => y.similarity - x.similarity);
72
83
 
73
- return { threshold, total: tools.length, clusters, riskyPairs, familyHints };
84
+ const boundaries = buildBoundaries(tools, edges);
85
+
86
+ return { threshold, total: tools.length, clusters, riskyPairs, familyHints, boundaries };
87
+ }
88
+
89
+ // Confusable pairs form connected components: that is the unit a platform can
90
+ // hand to an author ("these N tools compete; here is what each must say").
91
+ // Overlap with clusters is intentional — one asks where tools mount, the other
92
+ // asks what still needs a sentence.
93
+ function buildBoundaries(tools, edges) {
94
+ const parent = tools.map((_, index) => index);
95
+ const find = (index) => {
96
+ let root = index;
97
+ while (parent[root] !== root) root = parent[root];
98
+ while (parent[index] !== root) {
99
+ const next = parent[index];
100
+ parent[index] = root;
101
+ index = next;
102
+ }
103
+ return root;
104
+ };
105
+ const union = (a, b) => {
106
+ const rootA = find(a);
107
+ const rootB = find(b);
108
+ if (rootA !== rootB) parent[rootB] = rootA;
109
+ };
110
+ for (const edge of edges) union(edge.i, edge.j);
111
+
112
+ const groups = new Map();
113
+ for (const edge of edges) {
114
+ const root = find(edge.i);
115
+ const list = groups.get(root) ?? [];
116
+ list.push(edge);
117
+ groups.set(root, list);
118
+ }
119
+
120
+ const usedLabels = new Set();
121
+ const boundaries = [];
122
+ let position = 0;
123
+ for (const list of groups.values()) {
124
+ const memberSet = new Set();
125
+ for (const edge of list) {
126
+ memberSet.add(edge.i);
127
+ memberSet.add(edge.j);
128
+ }
129
+ const members = [...memberSet].sort((a, b) => tools[a].name.localeCompare(tools[b].name));
130
+ const label = uniqueLabel(labelFor(members.map((index) => tools[index].name)), usedLabels, position, "family");
131
+ position += 1;
132
+ const pairs = list
133
+ .map((edge) => ({
134
+ a: tools[edge.i].name,
135
+ b: tools[edge.j].name,
136
+ similarity: Number(edge.similarity.toFixed(2)),
137
+ basis: edge.similarity >= BOUNDARY_SIMILARITY ? (edge.basis === "structural" ? "structural" : "lexical") : "family"
138
+ }))
139
+ .sort((x, y) => y.similarity - x.similarity || x.a.localeCompare(y.a));
140
+ const { axes, sharedParams } = familyAxes(tools, members);
141
+ boundaries.push({
142
+ label,
143
+ size: members.length,
144
+ tools: members.map((index) => tools[index].name),
145
+ axes,
146
+ sharedParams,
147
+ pairs,
148
+ suggestion: familySuggestion(axes, sharedParams)
149
+ });
150
+ }
151
+
152
+ return boundaries.sort(
153
+ (x, y) => y.pairs.length - x.pairs.length || y.size - x.size || x.label.localeCompare(y.label)
154
+ );
155
+ }
156
+
157
+ // The dimension each tool can claim inside its family: a name word no sibling
158
+ // uses, or failing that a parameter only it takes. A tool with neither has to
159
+ // describe its trigger scenario — no static signal can write that sentence.
160
+ function familyAxes(tools, members) {
161
+ const tokenSets = new Map();
162
+ const paramSets = new Map();
163
+ const tokenCounts = new Map();
164
+ const paramCounts = new Map();
165
+ for (const index of members) {
166
+ const tokens = new Set(nameTokens(tools[index].name));
167
+ tokenSets.set(index, tokens);
168
+ for (const token of tokens) tokenCounts.set(token, (tokenCounts.get(token) ?? 0) + 1);
169
+ const params = new Set(Object.keys(tools[index].parameters?.properties ?? {}));
170
+ paramSets.set(index, params);
171
+ for (const param of params) paramCounts.set(param, (paramCounts.get(param) ?? 0) + 1);
172
+ }
173
+ const axes = members.map((index) => ({
174
+ tool: tools[index].name,
175
+ distinct: [...tokenSets.get(index)].filter((token) => tokenCounts.get(token) === 1).sort(),
176
+ paramsOnly: [...paramSets.get(index)].filter((param) => paramCounts.get(param) === 1).sort()
177
+ }));
178
+ const sharedParams = [...paramCounts.entries()]
179
+ .filter(([, count]) => count >= 2)
180
+ .map(([param]) => param)
181
+ .sort();
182
+ return { axes, sharedParams };
183
+ }
184
+
185
+ function familySuggestion(axes, sharedParams) {
186
+ const named = axes.filter((axis) => axis.distinct.length > 0);
187
+ const unnamed = axes.filter((axis) => axis.distinct.length === 0);
188
+ const parts = [];
189
+ if (named.length > 0) {
190
+ parts.push(`用名称里的独有词作判据:${named.map((axis) => `${axis.tool}=${axis.distinct.join("/")}`).join(";")}`);
191
+ }
192
+ if (unnamed.length > 0) {
193
+ parts.push(`${unnamed.map((axis) => axis.tool).join("、")} 名称里没有独有词,需写明各自的触发场景`);
194
+ }
195
+ const shared = sharedParams.length > 0 ? `;共享参数 ${sharedParams.slice(0, 8).join("、")} 不能当判据` : "";
196
+ return `${parts.join(";")}${shared}。在每个工具的 description 里各补一句“何时用我、何时用对方”,再补一条歧义用例`;
74
197
  }
75
198
 
76
199
  function buildFamilyHints(tools, clusterOf, clusterSize) {
@@ -108,7 +231,7 @@ function buildFamilyHints(tools, clusterOf, clusterSize) {
108
231
  function labelFor(names) {
109
232
  const counts = new Map();
110
233
  for (const name of names) {
111
- for (const token of String(name).toLowerCase().split(/[^a-z0-9]+/)) {
234
+ for (const token of nameTokens(name)) {
112
235
  if (!token || GENERIC_NAME_TOKENS.has(token)) continue;
113
236
  counts.set(token, (counts.get(token) ?? 0) + 1);
114
237
  }
@@ -120,11 +243,11 @@ function labelFor(names) {
120
243
  return best && best.count > 1 ? best.token : "";
121
244
  }
122
245
 
123
- function uniqueLabel(base, used, position) {
124
- let label = base || `cluster-${position + 1}`;
246
+ function uniqueLabel(base, used, position, prefix = "cluster") {
247
+ let label = base || `${prefix}-${position + 1}`;
125
248
  let suffix = 2;
126
249
  while (used.has(label)) {
127
- label = `${base || `cluster-${position + 1}`}-${suffix}`;
250
+ label = `${base || `${prefix}-${position + 1}`}-${suffix}`;
128
251
  suffix += 1;
129
252
  }
130
253
  used.add(label);
@@ -154,5 +277,28 @@ export function formatClusters(result) {
154
277
  lines.push(` [${hint.token}] ${hint.size} 个:${parts.join(" | ")}`);
155
278
  }
156
279
  }
280
+ if ((result.boundaries ?? []).length > 0) {
281
+ lines.push("", "族内边界清单(同族互抢的对,以及每个工具要补的区分词):");
282
+ for (const family of result.boundaries) {
283
+ lines.push(` [${family.label}] ${family.size} 个工具,${family.pairs.length} 对`);
284
+ const axes = family.axes.map((axis) => {
285
+ const label = axis.distinct.length > 0 ? axis.distinct.join("/") : axis.paramsOnly.length > 0 ? `参数 ${axis.paramsOnly.join("/")}` : "(无独有词)";
286
+ return `${axis.tool}=${label}`;
287
+ });
288
+ lines.push(` 区分轴:${axes.join(";")}`);
289
+ if (family.sharedParams.length > 0) {
290
+ const shown = family.sharedParams.slice(0, 8).join("、");
291
+ const more = family.sharedParams.length > 8 ? ` 等 ${family.sharedParams.length} 个` : "";
292
+ lines.push(` 共享参数:${shown}${more}(不能当判据)`);
293
+ }
294
+ const top = family.pairs.slice(0, 5);
295
+ const pairText = top
296
+ .map((pair) => `${pair.a} × ${pair.b} ${pair.similarity.toFixed(2)}${pair.basis === "family" ? "(同族)" : ""}`)
297
+ .join(";");
298
+ const rest = family.pairs.length > top.length ? `;…共 ${family.pairs.length} 对(--json 看全部)` : "";
299
+ lines.push(` 互抢对:${pairText}${rest}`);
300
+ lines.push(" → 在各自 description 里补一句“何时用我、何时用对方”(判据见区分轴),再补一条歧义用例");
301
+ }
302
+ }
157
303
  return lines.join("\n");
158
304
  }
package/lib/exec.js CHANGED
@@ -3,6 +3,7 @@ import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
3
3
  import { tmpdir } from "node:os";
4
4
  import { dirname, join, resolve } from "node:path";
5
5
  import { fileURLToPath } from "node:url";
6
+ import { permissionFlag } from "./node-permission.js";
6
7
 
7
8
  const HARNESS = fileURLToPath(new URL("./runtime/harness.mjs", import.meta.url));
8
9
  const HARNESS_PY = fileURLToPath(new URL("./runtime/harness.py", import.meta.url));
@@ -57,7 +58,7 @@ export async function executeTool(options) {
57
58
  process.cwd(),
58
59
  ...(options.allowRead ?? [])
59
60
  ].map((path) => resolve(path));
60
- args = ["--permission"];
61
+ args = [permissionFlag()];
61
62
  for (const path of dedupe(readPaths)) args.push(`--allow-fs-read=${path}`);
62
63
  if (options.allowWrite?.length) {
63
64
  for (const path of dedupe(options.allowWrite.map((item) => resolve(item)))) {
package/lib/lint.js CHANGED
@@ -1,4 +1,4 @@
1
- import { suspectedPairs, nameTokens } from "./cases.js";
1
+ import { suspectedPairs, nameTokens, boundaryHint } from "./cases.js";
2
2
  import { normalizeProfiles } from "./profile.js";
3
3
 
4
4
  const GENERIC_NAMES = new Set([
@@ -223,9 +223,11 @@ export function lintSuite(suite) {
223
223
  }
224
224
  }
225
225
  const reason = pair.basis === "structural" ? "名字与参数形状接近" : "描述相近";
226
+ const toolA = suite.tools.find((tool) => tool.name === pair.a);
227
+ const toolB = suite.tools.find((tool) => tool.name === pair.b);
226
228
  add("warn", "pair-no-boundary", pair.a, `与 ${pair.b} ${reason}(相似度 ${similarity}),且两边都没写“不适用于”边界,模型只能靠相似度硬猜`, {
227
229
  pair: [pair.a, pair.b],
228
- suggestion: `在 ${pair.a} 和 ${pair.b} 的 description 里各写一句“不适用于什么”,并补一条歧义用例`
230
+ suggestion: boundaryHint(toolA, toolB).suggestion
229
231
  });
230
232
  }
231
233
  }
@@ -0,0 +1,7 @@
1
+ // Permission Model flag renamed across Node versions: --experimental-permission
2
+ // in 20–23.4, --permission from 23.5. Using the wrong one makes exec fail with
3
+ // "bad option" on Node 20/22 even though the model itself is available.
4
+ export function permissionFlag(version = process.versions.node) {
5
+ const [major, minor] = String(version).split(".").map(Number);
6
+ return major > 23 || (major === 23 && minor >= 5) ? "--permission" : "--experimental-permission";
7
+ }
@@ -1,4 +1,5 @@
1
1
  import { spawnSync } from "node:child_process";
2
+ import { permissionFlag } from "./node-permission.js";
2
3
 
3
4
  function run(command, args, timeout = 4000) {
4
5
  const result = spawnSync(command, args, { encoding: "utf8", timeout, windowsHide: true });
@@ -20,7 +21,7 @@ export function checkSandbox() {
20
21
  const platform = process.platform;
21
22
  const nodeVersion = process.versions.node;
22
23
  const nodeMajor = Number(nodeVersion.split(".")[0]);
23
- const permission = run(process.execPath, ["--permission", "-e", "0"]);
24
+ const permission = run(process.execPath, [permissionFlag(), "-e", "0"]);
24
25
 
25
26
  const pythonProbe = platform === "win32" ? run("python", ["--version"]) : run("python3", ["--version"]);
26
27
  const pythonFallback = pythonProbe.ok ? pythonProbe : run("python", ["--version"]);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "modelfunction-cli",
3
- "version": "0.4.2",
3
+ "version": "0.4.3",
4
4
  "type": "module",
5
5
  "license": "MIT",
6
6
  "description": "mf — function-calling accuracy scorecard for AI agents",
@@ -22,17 +22,25 @@
22
22
  "fixtures",
23
23
  "skills",
24
24
  "README.md",
25
+ "SCHEMA.md",
25
26
  "LICENSE"
26
27
  ],
27
28
  "scripts": {
28
- "test": "node --test --test-concurrency=1 \"test/**/*.test.js\"",
29
+ "test": "node --test --test-concurrency=1",
30
+ "golden": "node scripts/golden.js",
31
+ "golden:update": "node scripts/golden.js --update",
32
+ "verify": "npm test && npm pack --dry-run",
29
33
  "eval": "node mf.js eval",
30
34
  "eval:mock": "node mf.js eval --models mock --show-cases",
31
35
  "pack:check": "npm pack --dry-run",
32
36
  "prepublishOnly": "npm test"
33
37
  },
38
+ "types": "./lib/check.d.ts",
34
39
  "exports": {
35
- ".": "./lib/check.js"
40
+ ".": {
41
+ "types": "./lib/check.d.ts",
42
+ "default": "./lib/check.js"
43
+ }
36
44
  },
37
45
  "engines": {
38
46
  "node": ">=20"