modelfunction-cli 0.4.3 → 0.5.0

package/README.md

@@ -5,7 +5,7 @@
 ## 安装
 
 ```bash
-npx -y modelfunction-cli@0.4.3 --version
+npx -y modelfunction-cli@0.4.4 --version
 # 或
 npm i -D modelfunction-cli
 npx mf --version
@@ -38,9 +38,19 @@ SILICONFLOW_API_KEY=... mf eval --models dsv4-flash
 OPENAI_API_KEY=... mf eval --models openai
 ```
 
+也可以把 Key 存进用户级凭证文件，输入时隐藏、不进 shell history：
+
+```bash
+mf config set-key DEEPSEEK_API_KEY   # 粘贴 Key，回车；只显示掩码
+mf config list                       # DEEPSEEK_API_KEY  sk-…abcd
+mf config path                       # ~/.modelfunction/credentials.json（0600）
+```
+
+加载优先级：**进程环境变量 > 凭证文件**（只补缺失项，不覆盖）。CI 与企业环境继续用环境变量或密钥管理，不要落盘。
+
 ### 自建 / 内网网关
 
-企业内网基本都是 OpenAI 兼容协议，写一个 models-file 即可，Key 只走环境变量、不落盘：
+企业内网基本都是 OpenAI 兼容协议，写一个 models-file 即可（Key 用环境变量，或用 `mf config set-key` 存在本机）：
 
 ```json
 {
@@ -76,6 +86,9 @@ INTERNAL_LLM_KEY=... mf eval --suite mf.suite.json \
 | `mf exec` | 单工具执行 + 指标（Node / Python），支持 `--allow-read/write/net`、超时、sideEffects 准入 |
 | `mf eval --execute` | 一次跑完选择 + 参数 + 运行断言（`toolInput` / `expect`） |
 | `mf sandbox check` | 探测本机可用隔离引擎与降级建议 |
+| `mf config set-key` | 把模型 Key 写入 `~/.modelfunction/credentials.json`（隐藏输入）；`remove-key` 删除；`list` 只显示掩码，`path` 打印路径 |
+| `mf stats` | 本地使用摘要（命令次数、模型、通过率区间、lint 规则命中）；不联网 |
+| `mf feedback` | 导出脱敏反馈包（仅计数与比率），发不发由你决定 |
 | `mf mcp` | stdio MCP server，供 Claude Code / Cursor / Kimi Code 调用 |
 
 ## 单次结果会抖：用 `--repeat` 看稳定性
@@ -111,7 +124,7 @@ mf eval --suite mf.suite.json --models internal-ds \
   "mcpServers": {
     "modelfunction": {
       "command": "npx",
-      "args": ["-y", "modelfunction-cli@0.4.3", "mcp"]
+      "args": ["-y", "modelfunction-cli@0.4.4", "mcp"]
     }
   }
 }
@@ -123,12 +136,27 @@ mf eval --suite mf.suite.json --models internal-ds \
 
 对外 JSON 字段、`check()` 返回结构、MCP 参数与退出码见 [SCHEMA.md](SCHEMA.md)，自 0.4.3 起冻结为 v1：v1 内只做加法，重命名或删除字段需 major 并提前两个 minor 或 90 天弃用。`check()` 的类型声明随包发布。
 
+本仓库当前版本 `0.5.0`（凭证与本地反馈渠道，待发布）；上文的 `@0.4.4` 是 npm 上已发布的版本，保证示例可复制。发布后把示例里的版本号统一升级即可。
+
 ## 数据边界
 
 - 本地运行，无遥测、无服务端地址
 - 唯一出站：你配置的模型端点，以及你自己的 MCP server（`--probe` 时）
 - 评测会把用例输入与工具描述发给被测模型供应商——敏感数据请选境内供应商或本地模型
 
+## 本地使用记录与反馈（默认不上传）
+
+每条命令会在 `~/.modelfunction/usage/YYYY-MM.jsonl` 追加一行**计数记录**：命令名、退出码、耗时、mf/Node/OS 版本、模型 id、用例/工具数量、通过率区间、lint 规则命中数。**不记录**用例原文、工具描述、源码、文件路径或密钥。
+
+```bash
+mf stats                    # 看本机记录；--json 给脚本
+mf feedback --out out.json  # 导出脱敏反馈包（聚合值，不含原始事件）
+```
+
+- 默认无网络、无遥测；`MF_NO_USAGE=1` 关闭记录；直接删除 `~/.modelfunction/usage/` 即清空。
+- 反馈包发不发由你决定；我们想知道的是「哪些命令在用、评测通过率大概什么水平、lint 命中哪些规则」，不需要项目细节。
+- `MF_CONFIG_DIR` 可改整个配置目录位置（CI/容器里常用）。
+
 ## 用例从哪来
 
 由你或你的编码 Agent 撰写，规则模板见包内 `skills/mf-cases/SKILL.md`。未复核用例（`reviewed:false`）不计入正式分数。
@@ -137,7 +165,7 @@ mf eval --suite mf.suite.json --models internal-ds \
 
 ```bash
 # 1. 把 mf 挂成 MCP server
-claude mcp add modelfunction -- npx -y modelfunction-cli@0.4.3 mcp
+claude mcp add modelfunction -- npx -y modelfunction-cli@0.4.4 mcp
 ```
 
 ```text

package/SCHEMA.md

@@ -1,6 +1,6 @@
 # Schemas v1
 
-Frozen at `modelfunction-cli@0.4.3`。这里描述 CLI 与库对外的稳定契约。人类可读文本（表格、提示语）不属于契约；**JSON 字段**属于。
+自 `modelfunction-cli@0.4.3` 起冻结为 v1；`0.4.4` 是同一 v1 的补丁；`0.5.0` 增加 §9 本地文件，仍属 v1。这里描述 CLI 与库对外的稳定契约。人类可读文本（表格、提示语）不属于契约；**JSON 字段**属于。
 
 ## 兼容政策
 
@@ -198,3 +198,65 @@ import { check } from "modelfunction-cli";
 | 2 | `mf eval --fail-under` 未达标，或 `--fail-on-regression` 发现回退 |
 | 3 | `mf exec` 被权限策略拒绝；`mf scan --json` 未找到工具 |
 | 4 | `mf exec` 超时 |
+
+## 9. 本地文件（0.5.0 起）
+
+### `~/.modelfunction/credentials.json`
+
+`MF_CONFIG_DIR` 可覆盖目录。扁平 JSON 对象，键为环境变量名、值为字符串：
+
+```json
+{ "DEEPSEEK_API_KEY": "sk-…" }
+```
+
+权限 0600（Windows 尽力）。加载规则：进程环境变量优先；凭证文件只补模型 registry 中声明的 `apiKeyEnv` 缺失项，不会向进程注入任意变量。不要提交到仓库；CI/企业用环境变量或密钥管理。
+
+### `~/.modelfunction/usage/YYYY-MM.jsonl`
+
+每行一个 JSON 事件（UTF-8，按 UTC 月份分文件；`ts` 为 ISO 8601）：
+
+```json
+{
+  "schema": 1,
+  "ts": "2026-11-01T01:26:33.000Z",
+  "command": "eval",
+  "exitCode": 0,
+  "durationMs": 1200,
+  "mf": "0.5.0",
+  "platform": "win32",
+  "arch": "x64",
+  "node": "24.6.0",
+  "models": ["mock"],
+  "caseCount": 13,
+  "toolCount": 4,
+  "reviewedCount": 13,
+  "repeat": 1,
+  "profileGroups": 0,
+  "selectionAccuracy": 0.92,
+  "argAccuracy": 1,
+  "passRate": 0.92,
+  "unstableCases": 0,
+  "failedThreshold": false,
+  "regressions": false
+}
+```
+
+字段为白名单，只含计数、比率与 id；**不含**用例原文、工具描述、源码、文件路径或密钥。新增字段属于 v1 加法，消费方可忽略未知字段。`MF_NO_USAGE=1` 关闭写入；删除 `usage/` 即清空。
+
+### `mf feedback` 包
+
+默认写到 `MF_CONFIG_DIR/feedback/mf-feedback-YYYYMMDD-HHmmss.json`（UTC）：
+
+```json
+{
+  "schema": 1,
+  "kind": "modelfunction-feedback",
+  "generatedAt": "2026-11-01T01:26:33.000Z",
+  "mf": { "version": "0.5.0", "node": "24.6.0", "platform": "win32", "arch": "x64" },
+  "usage": { "runs": 1, "commands": { "eval": 1 } },
+  "privacy": "…",
+  "request": "…"
+}
+```
+
+`usage` 是 `mf stats` 的聚合结果，不含原始事件、路径或密钥。是否发送由用户决定。

package/lib/credentials.js

@@ -0,0 +1,121 @@
+import { mkdir, readFile } from "node:fs/promises";
+import { homedir } from "node:os";
+import { join, resolve } from "node:path";
+import { writeFileAtomic } from "./fsutil.js";
+
+const ENV_NAME = /^[A-Z][A-Z0-9_]*$/;
+const FILE_MODE = 0o600;
+const DIR_MODE = 0o700;
+const SECRET_PATTERNS = [
+  { pattern: /(Bearer\s+)[A-Za-z0-9._~+/-]+=*/gi, replacement: "$1***" },
+  { pattern: /\bsk-[A-Za-z0-9_-]{8,}/g, replacement: "***" }
+];
+
+const registeredSecrets = new Set();
+
+export function configDir() {
+  const override = process.env.MF_CONFIG_DIR;
+  return override && override.trim() ? resolve(override.trim()) : join(homedir(), ".modelfunction");
+}
+
+export function credentialsPath() {
+  return join(configDir(), "credentials.json");
+}
+
+export async function readCredentials() {
+  let raw;
+  try {
+    raw = await readFile(credentialsPath(), "utf8");
+  } catch (error) {
+    if (error.code === "ENOENT") return {};
+    throw error;
+  }
+  let parsed;
+  try {
+    parsed = JSON.parse(raw);
+  } catch {
+    // Never echo the file content: it holds secrets.
+    throw new Error(`credentials file is not valid JSON: ${credentialsPath()}`);
+  }
+  if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
+    throw new Error(`credentials file must be a JSON object: ${credentialsPath()}`);
+  }
+  const out = {};
+  for (const [name, value] of Object.entries(parsed)) {
+    if (typeof value === "string" && value.length > 0) out[name] = value;
+  }
+  registerSecrets(Object.values(out));
+  return out;
+}
+
+export async function setCredential(name, value) {
+  if (!ENV_NAME.test(name)) {
+    throw new Error(`invalid environment variable name "${name}" (expected e.g. DEEPSEEK_API_KEY)`);
+  }
+  const trimmed = String(value ?? "").trim();
+  if (!trimmed) throw new Error("empty value; nothing written");
+  if (/[\r\n]/.test(trimmed)) throw new Error("value must be a single line");
+  const existing = await readCredentials();
+  existing[name] = trimmed;
+  await mkdir(configDir(), { recursive: true, mode: DIR_MODE });
+  await writeFileAtomic(credentialsPath(), `${JSON.stringify(existing, null, 2)}\n`, { mode: FILE_MODE });
+  registerSecrets([trimmed]);
+  return { name, path: credentialsPath() };
+}
+
+export async function removeCredential(name) {
+  if (!ENV_NAME.test(name)) {
+    throw new Error(`invalid environment variable name "${name}" (expected e.g. DEEPSEEK_API_KEY)`);
+  }
+  const existing = await readCredentials();
+  if (!(name in existing)) return { name, removed: false, path: credentialsPath() };
+  delete existing[name];
+  await mkdir(configDir(), { recursive: true, mode: DIR_MODE });
+  await writeFileAtomic(credentialsPath(), `${JSON.stringify(existing, null, 2)}\n`, { mode: FILE_MODE });
+  return { name, removed: true, path: credentialsPath() };
+}
+
+export async function listCredentials() {
+  const stored = await readCredentials();
+  return Object.entries(stored)
+    .map(([name, value]) => ({ name, masked: maskSecret(value) }))
+    .sort((a, b) => a.name.localeCompare(b.name));
+}
+
+// Only fill keys the model registry actually reads; a credentials file must
+// not be able to inject arbitrary environment variables into the process.
+export async function applyModelCredentials(registry) {
+  const stored = await readCredentials();
+  const names = new Set(
+    Object.values(registry)
+      .map((config) => config?.apiKeyEnv)
+      .filter((name) => typeof name === "string" && name.length > 0)
+  );
+  const filled = [];
+  for (const name of names) {
+    if (!process.env[name] && stored[name]) {
+      process.env[name] = stored[name];
+      filled.push(name);
+    }
+  }
+  return filled;
+}
+
+export function maskSecret(value) {
+  const text = String(value ?? "");
+  if (text.length <= 8) return "********";
+  return `${text.slice(0, 3)}…${text.slice(-4)}`;
+}
+
+export function registerSecrets(values) {
+  for (const value of values ?? []) {
+    if (typeof value === "string" && value.length >= 8) registeredSecrets.add(value);
+  }
+}
+
+export function redactText(text) {
+  let out = String(text ?? "");
+  for (const secret of registeredSecrets) out = out.split(secret).join("***");
+  for (const { pattern, replacement } of SECRET_PATTERNS) out = out.replace(pattern, replacement);
+  return out;
+}

package/lib/fsutil.js

@@ -1,7 +1,9 @@
-import { rename, writeFile } from "node:fs/promises";
+import { chmod, rename, writeFile } from "node:fs/promises";
 
-export async function writeFileAtomic(path, data) {
+export async function writeFileAtomic(path, data, { mode } = {}) {
   const temp = `${path}.${process.pid}.tmp`;
-  await writeFile(temp, data, "utf8");
+  await writeFile(temp, data, { encoding: "utf8", ...(mode ? { mode } : {}) });
   await rename(temp, path);
+  // Windows maps chmod to the read-only bit only; this is best-effort there.
+  if (mode) await chmod(path, mode).catch(() => {});
 }

package/lib/sandbox-check.js

@@ -1,7 +1,9 @@
 import { spawnSync } from "node:child_process";
 import { permissionFlag } from "./node-permission.js";
 
-function run(command, args, timeout = 4000) {
+// Shared CI runners can be slow to start a process; a 4s probe that times out
+// under load turns into a false "engine unavailable" and flakes the suite.
+function run(command, args, timeout = 10_000) {
   const result = spawnSync(command, args, { encoding: "utf8", timeout, windowsHide: true });
   return {
     ok: result.status === 0,

package/lib/usage.js

@@ -0,0 +1,321 @@
+import { appendFile, mkdir, readdir, readFile } from "node:fs/promises";
+import { appendFileSync, mkdirSync } from "node:fs";
+import { join } from "node:path";
+import { configDir } from "./credentials.js";
+
+// Events are redacted by construction: only counts, ids and ratios leave the
+// process. Never add free-form text (inputs, descriptions, paths, errors).
+const OPT_OUT_ENV = "MF_NO_USAGE";
+
+let pendingDetails = null;
+
+export function usageDir() {
+  return join(configDir(), "usage");
+}
+
+export function usageFile(date = new Date()) {
+  const month = date.toISOString().slice(0, 7);
+  return join(usageDir(), `${month}.jsonl`);
+}
+
+export function feedbackDir() {
+  return join(configDir(), "feedback");
+}
+
+export function defaultFeedbackPath(date = new Date()) {
+  const stamp = date.toISOString().replace(/[-:]/g, "").replace("T", "-").slice(0, 15);
+  return join(feedbackDir(), `mf-feedback-${stamp}.json`);
+}
+
+export function noteUsage(details) {
+  pendingDetails = { ...(pendingDetails ?? {}), ...sanitizeDetails(details) };
+}
+
+export async function recordRun({ command, exitCode = 0, durationMs = 0, version }) {
+  if (process.env[OPT_OUT_ENV] === "1") {
+    pendingDetails = null;
+    return;
+  }
+  const record = buildRecord({ command, exitCode, durationMs, version });
+  try {
+    await mkdir(usageDir(), { recursive: true, mode: 0o700 });
+    await appendFile(usageFile(), `${JSON.stringify(record)}\n`, { encoding: "utf8", mode: 0o600 });
+  } catch {
+    // Usage logging must never fail a command.
+  } finally {
+    pendingDetails = null;
+  }
+}
+
+// The process "exit" hook is synchronous, so the CLI uses this variant.
+export function recordRunSync({ command, exitCode = 0, durationMs = 0, version }) {
+  if (process.env[OPT_OUT_ENV] === "1") {
+    pendingDetails = null;
+    return;
+  }
+  const record = buildRecord({ command, exitCode, durationMs, version });
+  try {
+    mkdirSync(usageDir(), { recursive: true, mode: 0o700 });
+    appendFileSync(usageFile(), `${JSON.stringify(record)}\n`, { encoding: "utf8", mode: 0o600 });
+  } catch {
+    // ignore
+  } finally {
+    pendingDetails = null;
+  }
+}
+
+export function buildRecord({ command, exitCode, durationMs, version }) {
+  return {
+    schema: 1,
+    ts: new Date().toISOString(),
+    command: typeof command === "string" ? command.slice(0, 32) : "unknown",
+    exitCode: Number.isInteger(exitCode) ? exitCode : 1,
+    durationMs: Math.max(0, Math.round(Number(durationMs) || 0)),
+    mf: typeof version === "string" ? version : "unknown",
+    platform: process.platform,
+    arch: process.arch,
+    node: process.versions.node,
+    ...(pendingDetails ?? {})
+  };
+}
+
+export async function readUsageEvents() {
+  let files = [];
+  try {
+    files = (await readdir(usageDir())).filter((name) => /^\d{4}-\d{2}\.jsonl$/.test(name)).sort();
+  } catch (error) {
+    if (error.code === "ENOENT") return { events: [], files: [], skipped: 0 };
+    throw error;
+  }
+  const events = [];
+  let skipped = 0;
+  for (const file of files) {
+    const text = await readFile(join(usageDir(), file), "utf8");
+    for (const line of text.split(/\r?\n/)) {
+      if (!line.trim()) continue;
+      try {
+        const parsed = JSON.parse(line);
+        if (parsed && typeof parsed === "object" && typeof parsed.ts === "string") events.push(parsed);
+        else skipped += 1;
+      } catch {
+        skipped += 1;
+      }
+    }
+  }
+  events.sort((a, b) => a.ts.localeCompare(b.ts));
+  return { events, files, skipped };
+}
+
+export function summarizeUsage(events) {
+  const summary = {
+    runs: events.length,
+    firstRun: events[0]?.ts ?? null,
+    lastRun: events.at(-1)?.ts ?? null,
+    activeDays: new Set(events.map((event) => event.ts.slice(0, 10))).size,
+    exitCodes: {},
+    commands: {},
+    platforms: {},
+    nodeVersions: {},
+    mfVersions: {},
+    models: {},
+    eval: { runs: 0, executed: 0, meanSelection: null, meanPass: null, meanArgs: null, unstableCases: 0 },
+    lint: { runs: 0, errors: 0, warns: 0, infos: 0, rules: {} },
+    check: { runs: 0, verdicts: {} },
+    scan: { runs: 0, tools: 0, skills: 0 }
+  };
+  const selection = [];
+  const pass = [];
+  const args = [];
+
+  for (const event of events) {
+    bump(summary.exitCodes, String(event.exitCode ?? 0));
+    bump(summary.commands, event.command ?? "unknown");
+    if (event.platform) bump(summary.platforms, event.platform);
+    if (event.node) bump(summary.nodeVersions, event.node);
+    if (event.mf) bump(summary.mfVersions, event.mf);
+    for (const model of event.models ?? []) bump(summary.models, model);
+
+    if (event.command === "eval") {
+      summary.eval.runs += 1;
+      if (event.execEngine) summary.eval.executed += 1;
+      if (event.unstableCases) summary.eval.unstableCases += event.unstableCases;
+      if (typeof event.selectionAccuracy === "number") selection.push(event.selectionAccuracy);
+      if (typeof event.passRate === "number") pass.push(event.passRate);
+      if (typeof event.argAccuracy === "number") args.push(event.argAccuracy);
+    }
+    if (event.command === "lint") {
+      summary.lint.runs += 1;
+      summary.lint.errors += event.lintErrors ?? 0;
+      summary.lint.warns += event.lintWarns ?? 0;
+      summary.lint.infos += event.lintInfos ?? 0;
+      for (const [rule, count] of Object.entries(event.lintRules ?? {})) {
+        summary.lint.rules[rule] = (summary.lint.rules[rule] ?? 0) + count;
+      }
+    }
+    if (event.command === "check") {
+      summary.check.runs += 1;
+      bump(summary.check.verdicts, event.checkVerdict ?? "unknown");
+    }
+    if (event.command === "scan") {
+      summary.scan.runs += 1;
+      summary.scan.tools += event.scanTools ?? 0;
+      summary.scan.skills += event.scanSkills ?? 0;
+    }
+  }
+
+  summary.eval.meanSelection = mean(selection);
+  summary.eval.meanPass = mean(pass);
+  summary.eval.meanArgs = mean(args);
+  return summary;
+}
+
+export function buildFeedbackBundle({ events, version, generatedAt = new Date() }) {
+  return {
+    schema: 1,
+    kind: "modelfunction-feedback",
+    generatedAt: generatedAt.toISOString(),
+    mf: {
+      version,
+      node: process.versions.node,
+      platform: process.platform,
+      arch: process.arch
+    },
+    usage: summarizeUsage(events),
+    privacy:
+      "仅包含命令名、次数、模型 id、计数与比率。不含用例原文、工具描述、源码、文件路径或密钥。",
+    request:
+      "我们想了解：哪些命令被真实使用、在哪些平台上跑、评测通过率大概什么水平、lint/check 主要命中哪些规则。用于决定下一步做什么，不需要你的项目细节。"
+  };
+}
+
+export function formatStats(summary, { events = 0, files = 0, skipped = 0, path = usageDir() } = {}) {
+  if (summary.runs === 0) {
+    return [
+      `local usage: ${path}`,
+      "还没有本地记录。运行任意 mf 命令后会自动记录（仅计数，不上传）。",
+      "opt out: set MF_NO_USAGE=1"
+    ].join("\n");
+  }
+  const lines = [
+    `local usage: ${path} (${events} events, ${files} file(s)${skipped > 0 ? `, ${skipped} unreadable line(s)` : ""})`,
+    `runs: ${summary.runs}   first: ${summary.firstRun}   last: ${summary.lastRun}   active days: ${summary.activeDays}`,
+    `exit codes: ${pairs(summary.exitCodes)}`,
+    `commands:   ${pairs(summary.commands)}`
+  ];
+  if (Object.keys(summary.models).length > 0) lines.push(`models:     ${pairs(summary.models)}`);
+  if (summary.eval.runs > 0) {
+    const parts = [`${summary.eval.runs} run(s)`];
+    if (summary.eval.meanSelection !== null) parts.push(`select ${percent(summary.eval.meanSelection)}`);
+    if (summary.eval.meanPass !== null) parts.push(`pass ${percent(summary.eval.meanPass)}`);
+    if (summary.eval.executed > 0) parts.push(`${summary.eval.executed} with execution`);
+    if (summary.eval.unstableCases > 0) parts.push(`${summary.eval.unstableCases} unstable case(s)`);
+    lines.push(`eval:       ${parts.join(", ")}`);
+  }
+  if (summary.lint.runs > 0) {
+    const top = Object.entries(summary.lint.rules)
+      .sort((a, b) => b[1] - a[1])
+      .slice(0, 5)
+      .map(([rule, count]) => `${rule} ${count}`);
+    lines.push(
+      `lint:       ${summary.lint.runs} run(s), error ${summary.lint.errors} / warn ${summary.lint.warns} / info ${summary.lint.infos}` +
+        (top.length > 0 ? `; top: ${top.join(", ")}` : "")
+    );
+  }
+  if (summary.check.runs > 0) lines.push(`check:      ${summary.check.runs} run(s), ${pairs(summary.check.verdicts)}`);
+  if (summary.scan.runs > 0) lines.push(`scan:       ${summary.scan.runs} run(s), ${summary.scan.tools} tools, ${summary.scan.skills} skills`);
+  lines.push(`versions:   mf ${keys(summary.mfVersions)}   node ${keys(summary.nodeVersions)}   os ${keys(summary.platforms)}`);
+  lines.push(`no network, no telemetry. opt out: set MF_NO_USAGE=1. clear: delete ${path}`);
+  return lines.join("\n");
+}
+
+function bump(target, key) {
+  target[key] = (target[key] ?? 0) + 1;
+}
+
+function mean(values) {
+  if (values.length === 0) return null;
+  return Math.round((values.reduce((sum, value) => sum + value, 0) / values.length) * 10000) / 10000;
+}
+
+function percent(value) {
+  return `${(value * 100).toFixed(1)}%`;
+}
+
+function keys(counts) {
+  return Object.keys(counts).sort().slice(0, 5).join(", ") || "-";
+}
+
+function pairs(counts) {
+  return (
+    Object.entries(counts)
+      .sort((a, b) => b[1] - a[1])
+      .map(([key, value]) => `${key} ${value}`)
+      .join(", ") || "-"
+  );
+}
+
+const INT_FIELDS = new Set([
+  "caseCount",
+  "toolCount",
+  "reviewedCount",
+  "profileGroups",
+  "repeat",
+  "unstableCases",
+  "execExecuted",
+  "execPassed",
+  "execBlocked",
+  "execSkipped",
+  "scanTools",
+  "scanSkills",
+  "scanMcpServers",
+  "scanProbes",
+  "lintErrors",
+  "lintWarns",
+  "lintInfos",
+  "checkFindings",
+  "clusters"
+]);
+const RATIO_FIELDS = new Set(["selectionAccuracy", "argAccuracy", "passRate"]);
+const BOOL_FIELDS = new Set(["failedThreshold", "regressions"]);
+const STRING_FIELDS = new Set(["execEngine", "checkVerdict"]);
+const STRING_ARRAY_FIELDS = new Set(["models"]);
+const COUNT_OBJECT_FIELDS = new Set(["lintRules"]);
+
+export function sanitizeDetails(details) {
+  if (!details || typeof details !== "object") return {};
+  const out = {};
+  for (const [key, value] of Object.entries(details)) {
+    if (INT_FIELDS.has(key)) {
+      if (Number.isInteger(value) && value >= 0) out[key] = value;
+      continue;
+    }
+    if (RATIO_FIELDS.has(key)) {
+      if (typeof value === "number" && Number.isFinite(value)) out[key] = Math.round(value * 10000) / 10000;
+      continue;
+    }
+    if (BOOL_FIELDS.has(key)) {
+      if (typeof value === "boolean") out[key] = value;
+      continue;
+    }
+    if (STRING_FIELDS.has(key)) {
+      if (typeof value === "string" && /^[A-Za-z0-9_-]{1,32}$/.test(value)) out[key] = value;
+      continue;
+    }
+    if (STRING_ARRAY_FIELDS.has(key)) {
+      if (Array.isArray(value)) {
+        out[key] = value.filter((item) => typeof item === "string" && item.length <= 64).slice(0, 20);
+      }
+      continue;
+    }
+    if (COUNT_OBJECT_FIELDS.has(key)) {
+      if (value && typeof value === "object" && !Array.isArray(value)) {
+        out[key] = Object.fromEntries(
+          Object.entries(value)
+            .filter(([rule, count]) => /^[a-z0-9-]{1,40}$/.test(rule) && Number.isInteger(count) && count >= 0)
+            .slice(0, 40)
+        );
+      }
+    }
+  }
+  return out;
+}

package/mf.js

@@ -1,5 +1,6 @@
 #!/usr/bin/env node
-import { readFile } from "node:fs/promises";
+import { realpathSync } from "node:fs";
+import { mkdir, readFile } from "node:fs/promises";
 import { basename, dirname, extname, relative, resolve } from "node:path";
 import { fileURLToPath } from "node:url";
 import { loadSuite, toModelTools } from "./lib/suite.js";
@@ -30,6 +31,25 @@ import {
 } from "./lib/profile.js";
 import { runExec } from "./lib/exec.js";
 import { writeFileAtomic } from "./lib/fsutil.js";
+import {
+  applyModelCredentials,
+  credentialsPath,
+  listCredentials,
+  redactText,
+  registerSecrets,
+  removeCredential,
+  setCredential
+} from "./lib/credentials.js";
+import {
+  buildFeedbackBundle,
+  defaultFeedbackPath,
+  formatStats,
+  noteUsage,
+  readUsageEvents,
+  recordRunSync,
+  summarizeUsage,
+  usageDir
+} from "./lib/usage.js";
 import { createMockAdapter } from "./lib/adapters/mock.js";
 import { createOpenAiAdapter } from "./lib/adapters/openai.js";
 
@@ -58,6 +78,12 @@ Usage:
   mf cluster [--suite <file>] [--threshold <0.5>] [--max-size <10>] [--out <file>] [--json]  # 建议挂载分组 + 同族候选
   mf exec --entry <file> (--input <json> | --input-json <json>) [--runtime node|python] [--python <cmd>] [--export run] [--allow-read <paths>] [--allow-write <paths>] [--allow-net <hosts>] [--timeout <ms>] [--warmup <n>] [--json]
   mf sandbox check [--json]   # probe available isolation engines on this machine
+  mf config set-key <ENV_NAME>  # store a model key in ~/.modelfunction/credentials.json (hidden input)
+  mf config remove-key <ENV_NAME>  # delete a stored key
+  mf config list [--json]       # names + masked values only
+  mf config path                # print the credentials file path
+  mf stats [--json]             # local usage summary (never uploaded)
+  mf feedback [--out <file>] [--json]  # export a redacted feedback bundle; you decide whether to send it
   mf mcp                # start stdio MCP server for coding agents
 
 Eval options:
@@ -87,6 +113,8 @@ Scan options:
   --no-write            print only, do not write files
 
 Env keys: DEEPSEEK_API_KEY, DASHSCOPE_API_KEY, ZHIPU_API_KEY, OPENAI_API_KEY, SILICONFLOW_API_KEY
+Credentials: env vars win; missing model keys are read from ~/.modelfunction/credentials.json (MF_CONFIG_DIR overrides).
+Usage: recorded locally under ~/.modelfunction/usage (counts only, never uploaded); MF_NO_USAGE=1 opts out.
 
 Examples:
   mf eval --models mock --show-cases
@@ -251,6 +279,35 @@ export async function runEval(options, io = console) {
   const failed = options.failUnder === null
     ? []
     : summaries.filter((summary) => summary.selectionAccuracy < options.failUnder);
+  const profileBelow =
+    profileReport && options.failUnder !== null
+      ? profileReport.models.flatMap((model) =>
+          model.profiles
+            .filter((entry) => entry.reviewed > 0 && entry.mounted.selectionAccuracy < options.failUnder)
+            .map((entry) => `${model.model}/${entry.profile} (${pct(entry.mounted.selectionAccuracy)})`)
+        )
+      : [];
+
+  noteUsage({
+    models: summaries.map((summary) => summary.model),
+    caseCount: cases.length,
+    toolCount: suite.tools.length,
+    reviewedCount: cases.filter((testCase) => testCase.reviewed !== false).length,
+    repeat: options.repeat ?? 1,
+    profileGroups: profileReport ? profileReport.groups.length : 0,
+    selectionAccuracy: average(summaries.map((summary) => summary.selectionAccuracy)),
+    argAccuracy: average(summaries.map((summary) => summary.argAccuracy)),
+    passRate: average(summaries.map((summary) => summary.passRate)),
+    unstableCases: summaries.reduce((total, summary) => total + (summary.stability?.unstable.length ?? 0), 0),
+    execEngine: executionInfo?.engine,
+    execExecuted: sumExec(summaries, "executed"),
+    execPassed: sumExec(summaries, "passed"),
+    execBlocked: sumExec(summaries, "blocked"),
+    execSkipped: sumExec(summaries, "skipped"),
+    failedThreshold: failed.length > 0 || profileBelow.length > 0,
+    regressions: regressions.length > 0
+  });
+
   if (failed.length > 0) {
     io.error(
       `below threshold ${pct(options.failUnder)}: ${failed
@@ -259,21 +316,24 @@ export async function runEval(options, io = console) {
     );
     return 2;
   }
-  if (profileReport && options.failUnder !== null) {
-    const below = profileReport.models.flatMap((model) =>
-      model.profiles
-        .filter((entry) => entry.reviewed > 0 && entry.mounted.selectionAccuracy < options.failUnder)
-        .map((entry) => `${model.model}/${entry.profile} (${pct(entry.mounted.selectionAccuracy)})`)
-    );
-    if (below.length > 0) {
-      io.error(`profile below threshold ${pct(options.failUnder)}: ${below.join(", ")}`);
-      return 2;
-    }
+  if (profileBelow.length > 0) {
+    io.error(`profile below threshold ${pct(options.failUnder)}: ${profileBelow.join(", ")}`);
+    return 2;
   }
   if (regressions.length > 0) return 2;
   return 0;
 }
 
+function average(values) {
+  const usable = values.filter((value) => typeof value === "number" && Number.isFinite(value));
+  if (usable.length === 0) return undefined;
+  return usable.reduce((sum, value) => sum + value, 0) / usable.length;
+}
+
+function sumExec(summaries, field) {
+  return summaries.reduce((total, summary) => total + (summary.exec?.[field] ?? 0), 0);
+}
+
 export async function runAblate(options, io = console) {
   const suite = await loadSuite(resolve(options.suite ?? DEFAULT_SUITE));
   if (!options.tool) throw new Error("--tool is required");
@@ -314,6 +374,12 @@ export async function runAblate(options, io = console) {
     models,
     ...(scope ? { profile: { name: scope.name, tools: baseTools.length } } : {})
   };
+  noteUsage({
+    models: models.map((model) => model.model),
+    caseCount: baseCases.length,
+    toolCount: baseTools.length,
+    profileGroups: scope ? 1 : 0
+  });
   io.log(options.json ? JSON.stringify(payload, null, 2) : formatAblation(payload));
   return 0;
 }
@@ -354,6 +420,13 @@ export async function runScan(options, io = console) {
     }
   }
 
+  noteUsage({
+    scanTools: report.tools.length,
+    scanSkills: report.skills.length,
+    scanMcpServers: report.mcpServers.length,
+    scanProbes: report.probes.length
+  });
+
   const draft = buildSuiteDraft(report);
 
   if (options.json) {
@@ -417,6 +490,7 @@ export async function runCoverage(options, io = console) {
     allowEmptyTools: true
   });
   const report = computeCoverage(suite);
+  noteUsage({ toolCount: suite.tools.length, caseCount: suite.cases.length });
   io.log(options.json ? JSON.stringify(report, null, 2) : formatCoverage(report));
   return 0;
 }
@@ -436,6 +510,7 @@ export async function runCheck(options, io = console) {
     ? JSON.parse(await readFile(resolve(options.installed), "utf8"))
     : [];
   const result = check({ manifest, installed });
+  noteUsage({ checkVerdict: result.verdict, checkFindings: result.findings.length });
   io.log(options.json ? JSON.stringify(result, null, 2) : formatCheck(result));
   return 0;
 }
@@ -446,6 +521,15 @@ export async function runLint(options, io = console) {
     allowEmptyTools: true
   });
   const report = lintSuite(suite);
+  const rules = {};
+  for (const finding of report.findings) rules[finding.code] = (rules[finding.code] ?? 0) + 1;
+  noteUsage({
+    toolCount: report.tools,
+    lintErrors: report.counts.error,
+    lintWarns: report.counts.warn,
+    lintInfos: report.counts.info,
+    lintRules: rules
+  });
   io.log(options.json ? JSON.stringify(report, null, 2) : formatLint(report));
   return report.counts.error > 0 ? 1 : 0;
 }
@@ -460,6 +544,123 @@ export async function runCompare(options, io = console) {
   return changed ? 1 : 0;
 }
 
+export async function runConfig(options, io = console) {
+  const [sub, name] = options.args ?? [];
+  if (sub === "set-key") {
+    if (!name) throw new Error("usage: mf config set-key <ENV_NAME>");
+    const value = await (options.readSecret ?? readSecret)(`${name}: `);
+    const result = await setCredential(name, value);
+    io.log(`saved ${result.name} → ${result.path}`);
+    io.log("permissions: 0600 where supported. env vars still win; CI should use env or a secret manager.");
+    return 0;
+  }
+  if (sub === "remove-key") {
+    if (!name) throw new Error("usage: mf config remove-key <ENV_NAME>");
+    const result = await removeCredential(name);
+    io.log(
+      result.removed
+        ? `removed ${result.name} from ${result.path}`
+        : `${result.name} is not stored (${result.path})`
+    );
+    return 0;
+  }
+  if (sub === "list") {
+    const entries = await listCredentials();
+    if (options.json) {
+      io.log(JSON.stringify({ path: credentialsPath(), keys: entries }, null, 2));
+      return 0;
+    }
+    if (entries.length === 0) {
+      io.log(`no stored keys (${credentialsPath()})`);
+      return 0;
+    }
+    for (const entry of entries) io.log(`${entry.name}  ${entry.masked}`);
+    io.log(`file: ${credentialsPath()}`);
+    return 0;
+  }
+  if (sub === "path") {
+    io.log(credentialsPath());
+    return 0;
+  }
+  throw new Error("usage: mf config <set-key <ENV_NAME> | remove-key <ENV_NAME> | list | path>");
+}
+
+export async function runStats(options, io = console) {
+  const { events, files, skipped } = await readUsageEvents();
+  const summary = summarizeUsage(events);
+  if (options.json) {
+    io.log(JSON.stringify({ path: usageDir(), files: files.length, skipped, ...summary }, null, 2));
+  } else {
+    io.log(formatStats(summary, { events: events.length, files: files.length, skipped, path: usageDir() }));
+  }
+  return 0;
+}
+
+export async function runFeedback(options, io = console) {
+  const { events } = await readUsageEvents();
+  const version = await readVersion();
+  const bundle = buildFeedbackBundle({ events, version });
+  const target = resolve(options.out ?? defaultFeedbackPath());
+  await mkdir(dirname(target), { recursive: true, mode: 0o700 });
+  await writeFileAtomic(target, `${JSON.stringify(bundle, null, 2)}\n`, { mode: 0o600 });
+  if (options.json) io.log(JSON.stringify(bundle, null, 2));
+  io.log(`wrote ${target}`);
+  io.log(bundle.privacy);
+  io.log("send it to hi@yunzerui.com if you want us to look（可选，不发送不影响使用）。");
+  return 0;
+}
+
+// Hidden input: a key must never appear in argv, shell history, or logs.
+async function readSecret(prompt) {
+  const stdin = process.stdin;
+  if (!stdin.isTTY) {
+    const chunks = [];
+    for await (const chunk of stdin) chunks.push(chunk);
+    const value = Buffer.concat(chunks).toString("utf8").trim();
+    if (!value) throw new Error("empty input; nothing written");
+    return value;
+  }
+  process.stdout.write(prompt);
+  return new Promise((resolvePromise, reject) => {
+    let value = "";
+    const cleanup = () => {
+      stdin.removeListener("data", onData);
+      stdin.setRawMode(false);
+      stdin.pause();
+    };
+    const onData = (chunk) => {
+      for (const char of String(chunk)) {
+        if (char === "\u0003") {
+          cleanup();
+          process.stdout.write("\n");
+          reject(new Error("cancelled"));
+          return;
+        }
+        if (char === "\r" || char === "\n") {
+          cleanup();
+          process.stdout.write("\n");
+          resolvePromise(value.trim());
+          return;
+        }
+        if (char === "\u007f" || char === "\b") {
+          value = value.slice(0, -1);
+          continue;
+        }
+        value += char;
+      }
+    };
+    stdin.setRawMode(true);
+    stdin.setEncoding("utf8");
+    stdin.resume();
+    stdin.on("data", onData);
+  });
+}
+
+async function readVersion() {
+  const pkg = JSON.parse(await readFile(new URL("./package.json", import.meta.url), "utf8"));
+  return pkg.version;
+}
+
 async function readStdin() {
   const chunks = [];
   for await (const chunk of process.stdin) chunks.push(chunk);
@@ -476,6 +677,8 @@ function selectCases(cases, ids) {
 
 async function resolveModels(ids, modelsFile, io) {
   const registry = JSON.parse(await readFile(resolve(modelsFile), "utf8"));
+  // Env wins; the credentials file only fills what is missing.
+  await applyModelCredentials(registry);
   const explicit = Boolean(ids && ids.length > 0);
   const chosen = explicit ? ids : autoSelect(registry);
   const adapters = [];
@@ -492,11 +695,13 @@ async function resolveModels(ids, modelsFile, io) {
     if (config.type !== "openai") {
       throw new Error(`model "${id}": unsupported type "${config.type}"`);
     }
-    if (!process.env[config.apiKeyEnv]) {
+    const apiKey = process.env[config.apiKeyEnv];
+    if (!apiKey) {
       if (explicit) throw new Error(`model "${id}": environment variable ${config.apiKeyEnv} is not set`);
       io.error(`skip ${id}: ${config.apiKeyEnv} is not set`);
       continue;
     }
+    registerSecrets([apiKey]);
     adapters.push({ id, ...createOpenAiAdapter({ id, ...config }) });
   }
 
@@ -643,6 +848,7 @@ async function evaluateProfiles({ suite, cases, runs, registry, selection, repea
 export async function runCluster(options, io = console) {
   const suite = await loadSuite(options.suite, { allowEmptyCases: true });
   const result = clusterTools(suite.tools, { threshold: options.threshold, maxSize: options.maxSize });
+  noteUsage({ toolCount: suite.tools.length, clusters: result.clusters.length });
   if (options.out) {
     const target = resolve(options.out);
     const profiles = Object.fromEntries(result.clusters.map((cluster) => [cluster.label, cluster.tools]));
@@ -795,26 +1001,51 @@ function splitList(value) {
   return value.split(",").map((item) => item.trim()).filter(Boolean);
 }
 
-const isMain = process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url);
+// pnpm installs packages through symlinks; Node resolves ESM to the real
+// path, so a plain resolve(argv[1]) mismatch makes the CLI exit silently.
+const isMain = (() => {
+  if (!process.argv[1]) return false;
+  try {
+    return realpathSync(resolve(process.argv[1])) === realpathSync(fileURLToPath(import.meta.url));
+  } catch {
+    return false;
+  }
+})();
 
 if (isMain) {
   let argv = process.argv.slice(2);
   // `npx modelfunction-cli mf scan` passes the bin name through; accept it.
   if (argv[0] === "mf") argv = argv.slice(1);
+
+  const version = await readVersion();
+  const startedAt = Date.now();
+  let commandName = null;
+  const skipUsage = new Set(["mcp", "config", "stats", "feedback"]);
+  // Usage is local and count-only; the exit hook keeps every branch covered.
+  process.on("exit", (code) => {
+    if (!commandName || skipUsage.has(commandName)) return;
+    recordRunSync({
+      command: commandName,
+      exitCode: typeof code === "number" ? code : 0,
+      durationMs: Date.now() - startedAt,
+      version
+    });
+  });
+
   if (argv[0] === "--version" || argv[0] === "-v") {
-    const pkg = JSON.parse(await readFile(new URL("./package.json", import.meta.url), "utf8"));
-    process.stdout.write(`mf ${pkg.version}\n`);
+    process.stdout.write(`mf ${version}\n`);
     process.exit(0);
   } else if (argv[0] === "sandbox") {
     if (argv[1] !== "check") {
       process.stderr.write("usage: mf sandbox check [--json]\n");
       process.exit(1);
     }
+    commandName = "sandbox";
     try {
       const code = await runSandboxCheck({ json: argv.includes("--json") });
       process.exit(code);
     } catch (error) {
-      process.stderr.write(`${error?.message ?? error}\n`);
+      process.stderr.write(`${redactText(error?.message ?? error)}\n`);
       process.exit(1);
     }
   } else if (argv[0] === "compare") {
@@ -823,11 +1054,44 @@ if (isMain) {
       process.stderr.write("usage: mf compare <before.json> <after.json> [--json]\n");
       process.exit(1);
     }
+    commandName = "compare";
     try {
       const code = await runCompare({ paths, json: argv.includes("--json") });
       process.exit(code);
     } catch (error) {
-      process.stderr.write(`${error?.message ?? error}\n`);
+      process.stderr.write(`${redactText(error?.message ?? error)}\n`);
+      process.exit(1);
+    }
+  } else if (argv[0] === "config") {
+    try {
+      const code = await runConfig({ args: argv.slice(1), json: argv.includes("--json") });
+      process.exit(code);
+    } catch (error) {
+      process.stderr.write(`${redactText(error?.message ?? error)}\n`);
+      process.exit(1);
+    }
+  } else if (argv[0] === "stats") {
+    try {
+      const code = await runStats({ json: argv.includes("--json") });
+      process.exit(code);
+    } catch (error) {
+      process.stderr.write(`${redactText(error?.message ?? error)}\n`);
+      process.exit(1);
+    }
+  } else if (argv[0] === "feedback") {
+    try {
+      const outIndex = argv.indexOf("--out");
+      if (outIndex >= 0 && !argv[outIndex + 1]) {
+        process.stderr.write("usage: mf feedback [--out <file>] [--json]\n");
+        process.exit(1);
+      }
+      const code = await runFeedback({
+        out: outIndex >= 0 ? argv[outIndex + 1] : null,
+        json: argv.includes("--json")
+      });
+      process.exit(code);
+    } catch (error) {
+      process.stderr.write(`${redactText(error?.message ?? error)}\n`);
       process.exit(1);
     }
   } else if (argv[0] === "mcp") {
@@ -835,7 +1099,7 @@ if (isMain) {
       const { startMcpServer } = await import("./lib/mcp-server.js");
       await startMcpServer();
     } catch (error) {
-      process.stderr.write(`${error?.message ?? error}\n`);
+      process.stderr.write(`${redactText(error?.message ?? error)}\n`);
       process.exit(1);
     }
   } else if (argv.length === 0 || argv[0] === "help" || argv[0] === "-h" || argv[0] === "--help") {
@@ -854,6 +1118,7 @@ if (isMain) {
         process.stdout.write(USAGE);
         process.exit(0);
       }
+      commandName = argv[0] === "case" ? "case" : argv[0];
       const code =
         argv[0] === "scan"
           ? await runScan(options)
@@ -874,7 +1139,7 @@ if (isMain) {
                     : await runEval(options);
       process.exit(code);
     } catch (error) {
-      process.stderr.write(`${error?.message ?? error}\n`);
+      process.stderr.write(`${redactText(error?.message ?? error)}\n`);
       process.exit(1);
     }
   }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelfunction-cli",
-  "version": "0.4.3",
+  "version": "0.5.0",
   "type": "module",
   "license": "MIT",
   "description": "mf — function-calling accuracy scorecard for AI agents",