oh-my-knowledge 0.34.0 → 0.36.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -0
- package/README.zh.md +3 -0
- package/dist/assets/agent-skills/omk/SKILL.md +2 -2
- package/dist/assets/agent-skills/omk/references/commands.md +45 -0
- package/dist/authoring/evolver.js +13 -4
- package/dist/cli/commands/doctor.js +2 -1
- package/dist/cli/commands/eval/index.d.ts +1 -0
- package/dist/cli/commands/eval/index.js +40 -5
- package/dist/cli/commands/install.d.ts +2 -0
- package/dist/cli/commands/install.js +45 -20
- package/dist/cli/commands/list.d.ts +46 -0
- package/dist/cli/commands/list.js +243 -0
- package/dist/cli/commands/sample.d.ts +1 -1
- package/dist/cli/commands/sample.js +25 -12
- package/dist/cli/lib/cmd-flags.d.ts +1 -0
- package/dist/cli/lib/i18n-dict/install.d.ts +1 -1
- package/dist/cli/lib/i18n-dict/install.js +20 -0
- package/dist/cli/lib/i18n-dict/list.d.ts +3 -0
- package/dist/cli/lib/i18n-dict/list.js +32 -0
- package/dist/cli/lib/i18n-dict/run.d.ts +1 -1
- package/dist/cli/lib/i18n-dict/run.js +8 -0
- package/dist/cli/lib/i18n-dict.d.ts +2 -1
- package/dist/cli/lib/i18n-dict.js +2 -0
- package/dist/cli/lib/parse-run-config/variant-resolution.js +7 -0
- package/dist/cli/lib/run-tally.js +1 -1
- package/dist/cli/lib/shared.js +1 -1
- package/dist/doctor/index.js +6 -6
- package/dist/eval-core/cache.d.ts +11 -3
- package/dist/eval-core/cache.js +13 -5
- package/dist/eval-core/dependency-checker.js +5 -3
- package/dist/eval-core/evaluation-execution.js +4 -1
- package/dist/eval-core/evaluation-reporting.d.ts +1 -0
- package/dist/eval-core/evaluation-reporting.js +20 -9
- package/dist/eval-core/execution-strategy.js +6 -4
- package/dist/eval-core/task-planner.js +2 -1
- package/dist/eval-workflows/batch-evaluation-workflow.js +3 -2
- package/dist/eval-workflows/evaluation-preparation.js +3 -1
- package/dist/eval-workflows/run-evaluation.js +2 -2
- package/dist/inputs/content-hash.d.ts +28 -0
- package/dist/inputs/content-hash.js +106 -0
- package/dist/inputs/eval-config.js +44 -11
- package/dist/inputs/materialize-copy.d.ts +37 -0
- package/dist/inputs/materialize-copy.js +193 -0
- package/dist/inputs/skill-loader.d.ts +65 -2
- package/dist/inputs/skill-loader.js +314 -16
- package/dist/inputs/source-resolver.d.ts +16 -9
- package/dist/inputs/source-resolver.js +57 -77
- package/dist/managed/evidence.d.ts +22 -0
- package/dist/managed/evidence.js +143 -0
- package/dist/managed/index.d.ts +2 -0
- package/dist/managed/index.js +2 -0
- package/dist/managed/list-view.d.ts +55 -0
- package/dist/managed/list-view.js +66 -0
- package/dist/managed/store.d.ts +16 -20
- package/dist/managed/store.js +85 -84
- package/dist/observability/experience.d.ts +2 -0
- package/dist/observability/experience.js +31 -2
- package/dist/observability/inbox.js +31 -5
- package/dist/observability/review-state.js +22 -11
- package/dist/observability/soft-standards/llm-extractor.js +3 -3
- package/dist/observability/soft-standards/skill-standards-store.d.ts +1 -0
- package/dist/observability/soft-standards/skill-standards-store.js +77 -14
- package/dist/observability/soft-standards/types.d.ts +2 -2
- package/dist/renderer/html-renderer.js +3 -3
- package/dist/renderer/layout.js +2 -2
- package/dist/server/report-server.js +1 -1
- package/dist/server/report-store.d.ts +1 -1
- package/dist/server/report-store.js +16 -15
- package/dist/server/skill-index.js +5 -4
- package/dist/types/doctor.d.ts +2 -2
- package/dist/types/doctor.js +1 -1
- package/dist/types/eval.d.ts +12 -1
- package/dist/types/managed.d.ts +29 -4
- package/dist/types/observability.d.ts +6 -6
- package/dist/types/report.d.ts +24 -5
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -100,6 +100,8 @@ RAG-specific evals: see RAGAS (separate niche, complementary to omk). Full compa
|
|
|
100
100
|
| **Production observability** | parse Claude Code session JSONL traces; measure per-skill failure rate / latency / cost / knowledge-gap signals |
|
|
101
101
|
| **Knowledge-gap detection** | severity-weighted signals quantify risk exposure instead of claiming completeness |
|
|
102
102
|
| **Construct-validity isolation** | `--strict-baseline` (default ON) cuts three contamination channels so baseline doesn't silently see the skill it's being compared against |
|
|
103
|
+
| **Git & remote sources** | install / eval from a local git ref or a remote git URL (`--git-url`); directory-skills run in a content-addressed **isolated copy** so `references/` assets are real measured input, not just `SKILL.md` |
|
|
104
|
+
| **Evidence-gated management** | `omk install` registers a managed record; `omk eval` auto-writes evidence bound by content fingerprint, moving a skill `installed → measurable`; `omk list` surfaces each managed skill's status (installed / measurable / stale). [spec →](docs/specs/evidence-gated-management.md) |
|
|
103
105
|
| **Sample design science** | sample schema with `capability` / `difficulty` / `construct` / `provenance` metadata (HF Dataset Cards style); studio surfaces coverage breakdown plus `rubric_clarity_low` / `capability_thin` flags. [docs/specs/sample-design-spec.md](docs/specs/sample-design-spec.md) |
|
|
104
106
|
| **Multi-judge ensemble** | `--judge-models claude:opus,openai:gpt-4o` cross-vendor scoring + agreement metrics |
|
|
105
107
|
| **Blind A/B** | `--blind` hides variant names; HTML report has a reveal button |
|
|
@@ -122,6 +124,7 @@ The full docs are published at **[oh-my-knowledge.pages.dev](https://oh-my-knowl
|
|
|
122
124
|
- **[Sample design spec](docs/specs/sample-design-spec.md)** — capability / construct / provenance metadata; industry-gap mapping
|
|
123
125
|
- **[Statistical rigor](docs/explanation/statistical-rigor.md)** — why bootstrap CI / α / length-debias / saturation matter
|
|
124
126
|
- **[Comparison with 7 tools](docs/reference/comparison.md)** — 25+ dimensions across promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai
|
|
127
|
+
- **[Evidence-gated management](docs/specs/evidence-gated-management.md)** — managed records, lifecycle states (installed / measurable / stale), install → eval → measurable
|
|
125
128
|
|
|
126
129
|
## Environment variables
|
|
127
130
|
|
package/README.zh.md
CHANGED
|
@@ -100,6 +100,8 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
|
|
|
100
100
|
| **线上 session 观测** | 解析 Claude Code session JSONL,测量各 skill 的失败率、耗时、token 成本、知识缺口信号 |
|
|
101
101
|
| **知识缺口识别** | 严重度加权的信号量化风险敞口,不宣称完备性 |
|
|
102
102
|
| **用例隔离 (construct validity)** | `--strict-baseline`(默认开)三堵 baseline 拿到被测 skill 的污染路径 |
|
|
103
|
+
| **Git / 远端源** | install / eval 支持本地 git ref 或远端 git URL(`--git-url`);目录-skill 在内容寻址**隔离副本**里执行,`references/` 资产是真实测量输入,不只是 `SKILL.md` |
|
|
104
|
+
| **证据门控管理** | `omk install` 登记受管记录;`omk eval` 按内容指纹自动写入证据,把 skill 从 `installed` 推到 `measurable`;`omk list` 查看各受管 skill 的状态(installed / measurable / stale)。[规范 →](docs/zh/specs/evidence-gated-management.md) |
|
|
103
105
|
| **用例设计科学性** | Sample schema 加 `capability` / `difficulty` / `construct` / `provenance` 元数据字段(HF Dataset Cards 风),studio 输出 coverage 分桶 + `rubric_clarity_low` / `capability_thin` issue。[docs/zh/specs/sample-design-spec.md](docs/zh/specs/sample-design-spec.md) |
|
|
104
106
|
| **多评委 ensemble** | `--judge-models claude:opus,openai:gpt-4o` 跨厂商评分 + agreement 度量 |
|
|
105
107
|
| **盲测 A/B** | `--blind` 隐藏变体名称,HTML 报告有揭晓按钮 |
|
|
@@ -122,6 +124,7 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
|
|
|
122
124
|
- **[用例设计规范](docs/zh/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据;行业 gap 映射
|
|
123
125
|
- **[统计严谨性](docs/zh/explanation/statistical-rigor.md)** —— 为什么 Bootstrap CI / α / 长度去偏 / 饱和曲线重要
|
|
124
126
|
- **[7 工具对比](docs/zh/reference/comparison.md)** —— promptfoo / DeepEval / RAGAS / OpenAI Evals / LangSmith / lm-eval-harness / inspect-ai 等 25+ 维度横评
|
|
127
|
+
- **[证据门控管理](docs/zh/specs/evidence-gated-management.md)** —— 受管记录、生命周期状态(installed / measurable / stale)、install → eval → measurable
|
|
125
128
|
|
|
126
129
|
## 环境变量
|
|
127
130
|
|
|
@@ -4,7 +4,7 @@ description: |
|
|
|
4
4
|
oh-my-knowledge 知识载体评测工具的智能代理。评测 skill(系统提示词)质量,对比不同版本效果,自动迭代改进。
|
|
5
5
|
Use when: 用户提到"评测"、"测评"、"eval"、"benchmark"、"对比 skill"、"改进 skill"、"evolve"、"生成测试用例"、"gen-samples"、"omk"。
|
|
6
6
|
user-invocable: true
|
|
7
|
-
argument-hint: "<doctor|eval|evolve|init|install|observe|sample|studio> [options]"
|
|
7
|
+
argument-hint: "<doctor|eval|evolve|init|install|list|observe|sample|studio> [options]"
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# OMK — 知识载体评测
|
|
@@ -19,7 +19,7 @@ argument-hint: "<doctor|eval|evolve|init|install|observe|sample|studio> [options
|
|
|
19
19
|
npm i oh-my-knowledge -g
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
omk CLI 顶层命令包括:`init` / `install` / `doctor` / `eval` / `observe` / `evolve` / `sample` / `studio`。没有 `bench` / `improve` / `gen-samples` 这些旧子命令名 —— 如果你在历史 SKILL / 文档里看到了,那是 v0.30 命令树重构之前的写法。
|
|
22
|
+
omk CLI 顶层命令包括:`init` / `install` / `list` / `doctor` / `eval` / `observe` / `evolve` / `sample` / `studio`。没有 `bench` / `improve` / `gen-samples` 这些旧子命令名 —— 如果你在历史 SKILL / 文档里看到了,那是 v0.30 命令树重构之前的写法。
|
|
23
23
|
|
|
24
24
|
## 第二步:理解用户意图
|
|
25
25
|
|
|
@@ -89,6 +89,7 @@ omk eval [flags]
|
|
|
89
89
|
- `--no-cache` `boolean`:跳过 executor cache
|
|
90
90
|
- `--no-debias-length` `boolean`:关 length-debias(默认开)
|
|
91
91
|
- `--no-diagnostic` `boolean`:关闭 diagnostic 诊断 LLM 调用(默认开,给 failed sample 出「哪错了 + 怎么改」建议)。
|
|
92
|
+
- `--no-evidence` `boolean`:不把本次评测写成证据追加进受管记录(默认会为已 install 的 skill 自动写)。
|
|
92
93
|
- `--no-gate` `boolean`:关 verdict gate
|
|
93
94
|
- `--no-judge` `boolean`:跳过 LLM judge
|
|
94
95
|
- `--no-serve` `boolean`:不启 report server
|
|
@@ -303,6 +304,8 @@ omk install <input> [flags]
|
|
|
303
304
|
- `--dest` `option`:自定义 skill 根目录;skill 安装到 <dir>/<name>(内置 omk-agent-skill 为 <dir>/omk)。
|
|
304
305
|
- `--dry-run` `boolean`:只打印安装目标,不写文件。
|
|
305
306
|
- `--force` `boolean`:覆盖目标位置已存在的 skill。
|
|
307
|
+
- `--git-ref` `option`:远端 git 的 ref(分支 / tag / SHA),默认 HEAD。仅配合 --git-url 使用。
|
|
308
|
+
- `--git-url` `option`:远端 git 仓库 URL(https / ssh / git@host:path)。给了它时,位置参数当作仓库内 skill 路径(spec)。
|
|
306
309
|
- `--kind` `skill|prompt|agent|workflow`:用户 artifact 的 kind(对齐 Artifact.kind)。可省:命中 SKILL.md 自动推导,当前仅支持 skill。
|
|
307
310
|
- `--lang` `option` (默认 `zh`):输出语言 zh|en,优先级 CLI > OMK_LANG env > zh。
|
|
308
311
|
- `--to` `option` (默认 `auto`):安装目标:auto(默认,本机已检测目标) / codex / claude / all。
|
|
@@ -339,6 +342,48 @@ omk install ./skills/review
|
|
|
339
342
|
omk install git:main:skills/review
|
|
340
343
|
```
|
|
341
344
|
|
|
345
|
+
> 从远端 git 仓库安装 skill(位置参数是仓库内路径;认证用本机 git 凭证;记录钉实际 SHA)
|
|
346
|
+
|
|
347
|
+
```bash
|
|
348
|
+
omk install --git-url https://github.com/org/repo.git --git-ref v1.0.0 skills/review
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
## omk list
|
|
352
|
+
|
|
353
|
+
列出受管 skill 及其证据状态:生命周期(installed / measurable / stale)、最新 verdict、证据数、源。
|
|
354
|
+
|
|
355
|
+
**用法:**
|
|
356
|
+
|
|
357
|
+
```bash
|
|
358
|
+
omk list [flags]
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
**Flags:**
|
|
362
|
+
|
|
363
|
+
- `--global` `boolean`:看全局受管目录(~/.oh-my-knowledge/managed)而非项目 .omk/managed
|
|
364
|
+
- `--json` `boolean`:输出 JSON(含完整可比性 marker),供脚本消费
|
|
365
|
+
- `--lang` `option` (默认 `zh`):输出语言 zh|en,优先级 CLI > OMK_LANG env > zh。
|
|
366
|
+
|
|
367
|
+
**示例:**
|
|
368
|
+
|
|
369
|
+
> 列出当前项目的受管 skill
|
|
370
|
+
|
|
371
|
+
```bash
|
|
372
|
+
omk list
|
|
373
|
+
```
|
|
374
|
+
|
|
375
|
+
> 列出全局受管 skill
|
|
376
|
+
|
|
377
|
+
```bash
|
|
378
|
+
omk list --global
|
|
379
|
+
```
|
|
380
|
+
|
|
381
|
+
> 机器可读 JSON 输出
|
|
382
|
+
|
|
383
|
+
```bash
|
|
384
|
+
omk list --json
|
|
385
|
+
```
|
|
386
|
+
|
|
342
387
|
## omk observe
|
|
343
388
|
|
|
344
389
|
分析 sessions 目录的 skill 调用健康度(默认行为)。子命令:ingest / inbox / show。
|
|
@@ -6,6 +6,7 @@ import { persistReport, DEFAULT_OUTPUT_DIR, generateRunId, hashString } from '..
|
|
|
6
6
|
import { createFileStore } from '../server/report-store.js';
|
|
7
7
|
import { analyzeResults } from '../analysis/report-diagnostics.js';
|
|
8
8
|
import { loadSamples } from '../inputs/load-samples.js';
|
|
9
|
+
import { hashArtifactSource } from '../inputs/content-hash.js';
|
|
9
10
|
import { buildVariantSummary } from '../eval-core/schema.js';
|
|
10
11
|
import { bootstrapDiffCI, DEFAULT_BOOTSTRAP_ALPHA, DEFAULT_BOOTSTRAP_SAMPLES } from '../eval-core/bootstrap.js';
|
|
11
12
|
import { fixSamples } from './sample-fixer.js';
|
|
@@ -88,9 +89,13 @@ function singleVariantReport(report, variantKey) {
|
|
|
88
89
|
async function findReusableBaselineReport(opts) {
|
|
89
90
|
const store = createFileStore(DEFAULT_OUTPUT_DIR);
|
|
90
91
|
const { samples } = loadSamples(opts.samplesPath);
|
|
91
|
-
const artifactHash =
|
|
92
|
+
const artifactHash = opts.artifactHash;
|
|
92
93
|
const reports = await store.findByArtifactHash(artifactHash);
|
|
93
94
|
for (const report of reports) {
|
|
95
|
+
// schemaVersion < 2 的报告 artifactHashes 是旧文本哈,与当前树哈不同空间:即便值偶合也不该复用
|
|
96
|
+
// (口径不同会让 lineage 串错身份)。直接跳过,让旧 baseline 重跑出树哈报告。
|
|
97
|
+
if ((report.meta.schemaVersion ?? 0) < 2)
|
|
98
|
+
continue;
|
|
94
99
|
if (report.meta.model !== opts.model || report.meta.executor !== opts.executorName)
|
|
95
100
|
continue;
|
|
96
101
|
if ((report.meta.effort ?? undefined) !== (opts.effort ?? undefined))
|
|
@@ -541,7 +546,7 @@ export function mergeEvolveReports(roundReports, skillName, totalCostUSD, sample
|
|
|
541
546
|
}
|
|
542
547
|
const runId = `evolve-${skillName}-${generateRunId([skillName]).split('-').slice(-2).join('-')}`;
|
|
543
548
|
const report = {
|
|
544
|
-
|
|
549
|
+
kind: 'evaluation',
|
|
545
550
|
id: runId,
|
|
546
551
|
meta: {
|
|
547
552
|
...firstReport.meta,
|
|
@@ -669,10 +674,14 @@ export async function evolveSkill({ skillPath, samplesPath, rounds = 5, target =
|
|
|
669
674
|
let stopReason = 'rounds';
|
|
670
675
|
// 给定一个 report 看任一 variant 的 exec/judge cost 是否未报告
|
|
671
676
|
const reportHasUnreportedCost = (rep) => Object.values(rep.summary).some((v) => v.execCostReported === false || v.judgeCostReported === false);
|
|
672
|
-
// Round 0: baseline evaluation
|
|
677
|
+
// Round 0: baseline evaluation。复用查询键走整树哈,与 eval 报告口径一致:round 0 的 currentBest 即
|
|
678
|
+
// 磁盘上原始 skill,树哈取自磁盘(dir-skill 哈整目录、单文件 .md 哈单文件)。evolve 只改 SKILL.md 正文、
|
|
679
|
+
// 不动 references/ 资产,故磁盘树哈即该 baseline 的权威指纹。
|
|
680
|
+
const baselineIsDirSkill = basename(absSkillPath) === 'SKILL.md';
|
|
681
|
+
const baselineArtifactHash = hashArtifactSource(baselineIsDirSkill ? skillDir : absSkillPath, baselineIsDirSkill);
|
|
673
682
|
let baselineReport = reuseLatestEval
|
|
674
683
|
? await findReusableBaselineReport({
|
|
675
|
-
|
|
684
|
+
artifactHash: baselineArtifactHash,
|
|
676
685
|
samplesPath: absSamplesPath,
|
|
677
686
|
model,
|
|
678
687
|
executorName,
|
|
@@ -300,7 +300,8 @@ export function pruneDoctorHistory(dir, skillName, maxKeep) {
|
|
|
300
300
|
continue;
|
|
301
301
|
try {
|
|
302
302
|
const data = JSON.parse(readFileSync(join(dir, file), 'utf-8'));
|
|
303
|
-
|
|
303
|
+
const kind = data?.kind === 'doctor' ? data.kind : null;
|
|
304
|
+
if (!kind || !Array.isArray(data.skills) || data.skills.length !== 1)
|
|
304
305
|
continue;
|
|
305
306
|
if (data.skills[0].skillName !== skillName)
|
|
306
307
|
continue;
|
|
@@ -50,6 +50,7 @@ export default class Eval extends BaseCommand {
|
|
|
50
50
|
'trivial-diff': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
|
|
51
51
|
'report-only': import("@oclif/core/interfaces").BooleanFlag<boolean>;
|
|
52
52
|
'no-gate': import("@oclif/core/interfaces").BooleanFlag<boolean>;
|
|
53
|
+
'no-evidence': import("@oclif/core/interfaces").BooleanFlag<boolean>;
|
|
53
54
|
};
|
|
54
55
|
run(): Promise<void>;
|
|
55
56
|
}
|
|
@@ -8,6 +8,7 @@ import { parseRunConfig } from '../../lib/parse-run-config.js';
|
|
|
8
8
|
import { makeOnProgress } from '../../lib/progress.js';
|
|
9
9
|
import { computeRunTally } from '../../lib/run-tally.js';
|
|
10
10
|
import { DEFAULT_BOOTSTRAP_SAMPLES } from '../../../eval-core/bootstrap.js';
|
|
11
|
+
import { EVALUATION_REPORT_SCHEMA_VERSION } from '../../../eval-core/evaluation-reporting.js';
|
|
11
12
|
function isDryRunReport(report) {
|
|
12
13
|
return Boolean(report && typeof report === 'object' && report.dryRun === true);
|
|
13
14
|
}
|
|
@@ -37,21 +38,44 @@ function applyGateExitCode(code, values, lang) {
|
|
|
37
38
|
process.stderr.write(tCli('cli.run.report_only_gate_skipped', lang));
|
|
38
39
|
return 0;
|
|
39
40
|
}
|
|
40
|
-
async function emitEvaluationVerdict(report, values) {
|
|
41
|
+
async function emitEvaluationVerdict(report, values, lang) {
|
|
41
42
|
const { computeVerdict, formatVerdictText } = await import('../../../eval-core/verdict.js');
|
|
42
43
|
const result = computeVerdict(report, verdictOptions(values));
|
|
43
44
|
console.log(formatVerdictText(result, { verbose: true }));
|
|
45
|
+
await recordEvidenceSafely(report, result.level, values, lang);
|
|
44
46
|
return verdictPasses(result.level, result.headline) ? 0 : 1;
|
|
45
47
|
}
|
|
48
|
+
/**
|
|
49
|
+
* 把本次评测写成证据追加进**已纳管**记录(让 install 过的 skill 走到 measurable)。
|
|
50
|
+
* 永不致命:管理是 eval 的旁路,写入失败 / 无匹配记录都不影响 verdict 与 exit code。
|
|
51
|
+
* `--no-evidence` 关闭。仅对实际写入的记录打一行提示(无匹配则全静默)。
|
|
52
|
+
*/
|
|
53
|
+
async function recordEvidenceSafely(report, verdict, values, lang) {
|
|
54
|
+
if (values['no-evidence'] === true)
|
|
55
|
+
return;
|
|
56
|
+
try {
|
|
57
|
+
const { recordEvalEvidence } = await import('../../../managed/index.js');
|
|
58
|
+
const written = recordEvalEvidence(report, verdict, new Date().toISOString());
|
|
59
|
+
for (const w of written) {
|
|
60
|
+
process.stderr.write(tCli(w.bound ? 'cli.run.evidence_recorded' : 'cli.run.evidence_recorded_unbound', lang, { name: w.name }));
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
catch {
|
|
64
|
+
// 证据写入是旁路,任何异常都不该让评测失败
|
|
65
|
+
}
|
|
66
|
+
}
|
|
46
67
|
function batchItemFallbackReport(batch, item) {
|
|
47
68
|
return {
|
|
48
|
-
|
|
69
|
+
kind: 'evaluation',
|
|
49
70
|
id: item.reportId,
|
|
50
71
|
meta: {
|
|
51
72
|
...batch.meta,
|
|
52
73
|
variants: ['baseline', item.name],
|
|
53
74
|
sampleCount: item.sampleCount,
|
|
54
75
|
totalCostUSD: item.totalCostUSD,
|
|
76
|
+
// item.artifactHash 来自子报告(走 aggregateReport 的整树哈),故 fallback 与之一致标当前 eval
|
|
77
|
+
// report schemaVersion,避免「树哈 artifactHashes + 错位 schemaVersion」的错配。
|
|
78
|
+
schemaVersion: EVALUATION_REPORT_SCHEMA_VERSION,
|
|
55
79
|
artifactHashes: item.artifactHash ? { [item.name]: item.artifactHash } : {},
|
|
56
80
|
},
|
|
57
81
|
summary: item.summary,
|
|
@@ -65,7 +89,7 @@ async function loadBatchChildReports(batch, reportsDir, lang) {
|
|
|
65
89
|
const reports = [];
|
|
66
90
|
for (const item of batch.items) {
|
|
67
91
|
const loaded = await store.get(item.reportId);
|
|
68
|
-
if (loaded?.
|
|
92
|
+
if (loaded?.kind === 'evaluation') {
|
|
69
93
|
reports.push(loaded);
|
|
70
94
|
}
|
|
71
95
|
else {
|
|
@@ -83,6 +107,11 @@ async function emitBatchVerdict(report, reportsDir, values, lang) {
|
|
|
83
107
|
treatment: child.meta.variants[1] ?? child.id,
|
|
84
108
|
verdict: computeVerdict(child, verdictOptions(values)),
|
|
85
109
|
}));
|
|
110
|
+
// batch 每个子报告各自是一份独立 skill 的评测 → 各自写证据。
|
|
111
|
+
for (const child of childReports) {
|
|
112
|
+
const v = results.find((r) => r.id === child.id)?.verdict.level ?? 'SOLO';
|
|
113
|
+
await recordEvidenceSafely(child, v, values, lang);
|
|
114
|
+
}
|
|
86
115
|
const passed = results.filter((r) => verdictPasses(r.verdict.level, r.verdict.headline)).length;
|
|
87
116
|
const failed = results.length - passed;
|
|
88
117
|
const status = lang === 'zh'
|
|
@@ -100,7 +129,7 @@ async function emitBatchVerdict(report, reportsDir, values, lang) {
|
|
|
100
129
|
}
|
|
101
130
|
async function announceSavedReport({ report, filePath, reportsDir, values, lang, }) {
|
|
102
131
|
const tally = computeRunTally(report);
|
|
103
|
-
process.stderr.write(tCli(report.
|
|
132
|
+
process.stderr.write(tCli(report.kind === 'batch-evaluation' ? 'cli.run.batch_complete' : 'cli.run.eval_complete', lang));
|
|
104
133
|
process.stderr.write(tCli('cli.run.tally', lang, tally));
|
|
105
134
|
process.stderr.write(tCli('cli.run.report_saved', lang, { path: filePath }));
|
|
106
135
|
if (!values['no-serve'] && process.stdout.isTTY) {
|
|
@@ -255,7 +284,7 @@ async function runEval(_args, flags, lang) {
|
|
|
255
284
|
if (filePath) {
|
|
256
285
|
await announceSavedReport({ report, filePath, reportsDir: config.outputDir, values, lang });
|
|
257
286
|
}
|
|
258
|
-
const exitCode = await emitEvaluationVerdict(report, values);
|
|
287
|
+
const exitCode = await emitEvaluationVerdict(report, values, lang);
|
|
259
288
|
throw new CliExit(applyGateExitCode(exitCode, values, lang));
|
|
260
289
|
}
|
|
261
290
|
catch (err) {
|
|
@@ -466,6 +495,12 @@ export default class Eval extends BaseCommand {
|
|
|
466
495
|
'no-gate': Flags.boolean({
|
|
467
496
|
description: bilingual({ zh: '关 verdict gate', en: 'Disable verdict gate' }),
|
|
468
497
|
}),
|
|
498
|
+
'no-evidence': Flags.boolean({
|
|
499
|
+
description: bilingual({
|
|
500
|
+
zh: '不把本次评测写成证据追加进受管记录(默认会为已 install 的 skill 自动写)。',
|
|
501
|
+
en: 'Do not append this run as evidence to managed records (auto-written for installed skills by default).',
|
|
502
|
+
}),
|
|
503
|
+
}),
|
|
469
504
|
};
|
|
470
505
|
async run() {
|
|
471
506
|
const { args, flags } = await this.parse(Eval);
|
|
@@ -15,6 +15,8 @@ export default class Install extends BaseCommand {
|
|
|
15
15
|
dest: import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
|
|
16
16
|
force: import("@oclif/core/interfaces").BooleanFlag<boolean>;
|
|
17
17
|
'dry-run': import("@oclif/core/interfaces").BooleanFlag<boolean>;
|
|
18
|
+
'git-url': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
|
|
19
|
+
'git-ref': import("@oclif/core/interfaces").OptionFlag<string | undefined, import("@oclif/core/interfaces").CustomOptions>;
|
|
18
20
|
};
|
|
19
21
|
run(): Promise<void>;
|
|
20
22
|
private installBuiltinAgentSkill;
|
|
@@ -1,13 +1,13 @@
|
|
|
1
|
-
import { copyFileSync, cpSync, existsSync,
|
|
1
|
+
import { copyFileSync, cpSync, existsSync, mkdirSync, realpathSync, rmSync, statSync } from 'node:fs';
|
|
2
2
|
import { homedir } from 'node:os';
|
|
3
|
-
import { basename, dirname,
|
|
3
|
+
import { basename, dirname, resolve, join, sep } from 'node:path';
|
|
4
4
|
import { fileURLToPath } from 'node:url';
|
|
5
5
|
import { Args, Flags } from '@oclif/core';
|
|
6
6
|
import { LANG_FLAG, bilingual } from '../oclif/i18n.js';
|
|
7
7
|
import { BaseCommand } from '../oclif/base-command.js';
|
|
8
8
|
import { tCli } from '../lib/i18n.js';
|
|
9
|
-
import { resolveInstallSource, SourceResolveError } from '../../inputs/source-resolver.js';
|
|
10
|
-
import { buildManagedArtifactRecord, hashArtifactSource,
|
|
9
|
+
import { resolveInstallSource, resolveRemoteGitSource, SourceResolveError } from '../../inputs/source-resolver.js';
|
|
10
|
+
import { buildManagedArtifactRecord, hashArtifactSource, distributableCopyFilter, managedDir, recordManagedArtifact } from '../../managed/index.js';
|
|
11
11
|
const BUILTIN_OMK_AGENT_SKILL_ID = 'omk-agent-skill';
|
|
12
12
|
const INSTALLABLE_KINDS = ['skill', 'prompt', 'agent', 'workflow'];
|
|
13
13
|
const TARGET_ORDER = ['codex', 'claude'];
|
|
@@ -163,17 +163,10 @@ function copyArtifactToTarget(params) {
|
|
|
163
163
|
if (params.isDirectorySkill) {
|
|
164
164
|
cpSync(params.source, params.targetPath, {
|
|
165
165
|
recursive: true,
|
|
166
|
-
// 与 hashArtifactSource
|
|
167
|
-
//
|
|
168
|
-
//
|
|
169
|
-
filter: (
|
|
170
|
-
const rel = relative(params.source, src);
|
|
171
|
-
if (rel === '')
|
|
172
|
-
return true;
|
|
173
|
-
if (lstatSync(src).isSymbolicLink())
|
|
174
|
-
return false;
|
|
175
|
-
return isDistributablePath(rel.split(sep));
|
|
176
|
-
},
|
|
166
|
+
// 与 hashArtifactSource / eval 隔离副本共用同一处过滤(distributableCopyFilter),保证
|
|
167
|
+
// "分发出去的 == 算进 hash 的 == 测量的副本":源根永远拷;软链跳过;evolve 仅源根第一层排除、
|
|
168
|
+
// .omk/.git/node_modules 任意层级排除。
|
|
169
|
+
filter: distributableCopyFilter(params.source),
|
|
177
170
|
});
|
|
178
171
|
}
|
|
179
172
|
else {
|
|
@@ -253,6 +246,13 @@ export default class Install extends BaseCommand {
|
|
|
253
246
|
}),
|
|
254
247
|
command: '<%= config.bin %> install git:main:skills/review',
|
|
255
248
|
},
|
|
249
|
+
{
|
|
250
|
+
description: bilingual({
|
|
251
|
+
zh: '从远端 git 仓库安装 skill(位置参数是仓库内路径;认证用本机 git 凭证;记录钉实际 SHA)',
|
|
252
|
+
en: 'Install a skill from a remote git repo (positional arg is the in-repo path; auth via local git credentials; record pins the actual SHA)',
|
|
253
|
+
}),
|
|
254
|
+
command: '<%= config.bin %> install --git-url https://github.com/org/repo.git --git-ref v1.0.0 skills/review',
|
|
255
|
+
},
|
|
256
256
|
];
|
|
257
257
|
static args = {
|
|
258
258
|
input: Args.string({
|
|
@@ -299,11 +299,35 @@ export default class Install extends BaseCommand {
|
|
|
299
299
|
}),
|
|
300
300
|
default: false,
|
|
301
301
|
}),
|
|
302
|
+
'git-url': Flags.string({
|
|
303
|
+
description: bilingual({
|
|
304
|
+
zh: '远端 git 仓库 URL(https / ssh / git@host:path)。给了它时,位置参数当作仓库内 skill 路径(spec)。',
|
|
305
|
+
en: 'Remote git repository URL (https / ssh / git@host:path). When set, the positional arg is the in-repo skill path (spec).',
|
|
306
|
+
}),
|
|
307
|
+
}),
|
|
308
|
+
'git-ref': Flags.string({
|
|
309
|
+
description: bilingual({
|
|
310
|
+
zh: '远端 git 的 ref(分支 / tag / SHA),默认 HEAD。仅配合 --git-url 使用。',
|
|
311
|
+
en: 'Remote git ref (branch / tag / SHA), default HEAD. Only with --git-url.',
|
|
312
|
+
}),
|
|
313
|
+
}),
|
|
302
314
|
};
|
|
303
315
|
async run() {
|
|
304
316
|
const { args, flags } = await this.parse(Install);
|
|
305
317
|
const lang = this.lang;
|
|
306
318
|
await this.runWithCliExit(async () => {
|
|
319
|
+
// --git-ref 必须配 --git-url(否则静默丢弃、误把 spec 当本地路径解析,报错令人困惑)。
|
|
320
|
+
if (flags['git-ref'] && !flags['git-url']) {
|
|
321
|
+
throw new Error(tCli('cli.install.git_ref_needs_url', lang));
|
|
322
|
+
}
|
|
323
|
+
// 远端 git:--git-url 在场时,位置参数是仓库内 spec(repo 相对路径),先于其它分支判定。
|
|
324
|
+
if (flags['git-url']) {
|
|
325
|
+
this.installManagedSkill(args.input, flags.kind, flags, lang, {
|
|
326
|
+
url: flags['git-url'],
|
|
327
|
+
ref: flags['git-ref'] || 'HEAD',
|
|
328
|
+
});
|
|
329
|
+
return;
|
|
330
|
+
}
|
|
307
331
|
if (args.input === BUILTIN_OMK_AGENT_SKILL_ID) {
|
|
308
332
|
this.installBuiltinAgentSkill(flags, lang);
|
|
309
333
|
return;
|
|
@@ -330,17 +354,18 @@ export default class Install extends BaseCommand {
|
|
|
330
354
|
if (!flags['dry-run'])
|
|
331
355
|
console.log(tCli('cli.install.next_hint', lang));
|
|
332
356
|
}
|
|
333
|
-
installManagedSkill(input, kindFlag, flags, lang) {
|
|
357
|
+
installManagedSkill(input, kindFlag, flags, lang, remote) {
|
|
334
358
|
// kind 推导:--kind 显式优先;否则缺省 skill(Phase 1 仅 skill)。
|
|
335
359
|
const kind = kindFlag ?? 'skill';
|
|
336
360
|
if (kind !== 'skill') {
|
|
337
361
|
throw new Error(tCli('cli.install.kind_unsupported', lang, { kind }));
|
|
338
362
|
}
|
|
339
|
-
// 源解析委托给 source-resolver(file / git
|
|
363
|
+
// 源解析委托给 source-resolver(file / 本地 git / 远端 git),install 主干源无关。
|
|
364
|
+
// 远端走结构化 resolveRemoteGitSource(url/ref/spec 分字段,URL 不经任何字符串切分);
|
|
340
365
|
// resolver 不依赖 CLI,错误以 SourceResolveError(messageKey) 抛出,这里映射成本地化文案。
|
|
341
366
|
let src;
|
|
342
367
|
try {
|
|
343
|
-
src = resolveInstallSource(input);
|
|
368
|
+
src = remote ? resolveRemoteGitSource(remote.url, remote.ref, input) : resolveInstallSource(input);
|
|
344
369
|
}
|
|
345
370
|
catch (err) {
|
|
346
371
|
if (err instanceof SourceResolveError) {
|
|
@@ -349,7 +374,7 @@ export default class Install extends BaseCommand {
|
|
|
349
374
|
throw err;
|
|
350
375
|
}
|
|
351
376
|
try {
|
|
352
|
-
const { localRoot, name, isDirectorySkill, sourceKind, locator, ref } = src;
|
|
377
|
+
const { localRoot, name, isDirectorySkill, sourceKind, locator, ref, url } = src;
|
|
353
378
|
// 目录-skill 哈整棵可分发树(排除 .omk/.git/evolve);git 源哈的是物化后的临时树。
|
|
354
379
|
const contentHash = hashArtifactSource(localRoot, isDirectorySkill);
|
|
355
380
|
const targets = resolveInstallTargets({ to: flags.to, dest: flags.dest, lang });
|
|
@@ -371,7 +396,7 @@ export default class Install extends BaseCommand {
|
|
|
371
396
|
const record = buildManagedArtifactRecord({
|
|
372
397
|
name,
|
|
373
398
|
kind,
|
|
374
|
-
source: { sourceKind, locator, ...(ref ? { ref } : {}), isDirectorySkill },
|
|
399
|
+
source: { sourceKind, locator, ...(ref ? { ref } : {}), ...(url ? { url } : {}), isDirectorySkill },
|
|
375
400
|
contentHash,
|
|
376
401
|
installedAt: now,
|
|
377
402
|
distribution,
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
import { BaseCommand } from '../oclif/base-command.js';
|
|
2
|
+
import { type ManagedListRow, type SourceProbe } from '../../managed/index.js';
|
|
3
|
+
import type { ManagedArtifactRecord } from '../../types/index.js';
|
|
4
|
+
import type { CliLang } from '../lib/i18n.js';
|
|
5
|
+
/**
|
|
6
|
+
* 探测一条受管记录**当前源**的状态(三态,喂 buildManagedListRow 判 drift / 生命周期)。
|
|
7
|
+
* - 远端 git(带 url):源身份钉不可变 SHA(install 时 pin),内容恒定 → 直接取 record.contentHash,
|
|
8
|
+
* 不联网(list 是快读命令,不为 drift 检查发网络请求)。reachable。
|
|
9
|
+
* - 本地 git:locator `git:<ref>:<spec>` 复用 resolveInstallSource 在**仓库对象库**内重物化重哈
|
|
10
|
+
* (读取受 git 边界约束,无任意文件读 DoS)。解析不到(常因 `omk list` 的 cwd 与 install 时不同、
|
|
11
|
+
* spec 随 cwd 漂)→ 抛错 → **reachable:false(未核,不当 stale)**,避免对未改动的 skill 误报漂移。
|
|
12
|
+
* - 本地 file:locator 是绝对路径,但受管 JSON **用户可手改 / 随仓库分发**(无 install、无 opt-in 即被
|
|
13
|
+
* loadAllManagedRecords 读到)。只读命令绝不盲读任意路径:**拒软链、只读常规文件 / 真目录、单文件
|
|
14
|
+
* 设 size cap**(挡 `evil.md → /dev/zero` 这类 readFileSync 无界 DoS 与项目外任意读)。守卫不过 →
|
|
15
|
+
* reachable:false。目录-skill 整树哈本就跳软链(hashArtifactSource 用 isFile() 过滤)。
|
|
16
|
+
*/
|
|
17
|
+
export declare function probeSourceState(record: ManagedArtifactRecord): SourceProbe;
|
|
18
|
+
/** CJK 全角字符按 2 列计宽,使含中文表头的列也能对齐。 */
|
|
19
|
+
export declare function dispWidth(s: string): number;
|
|
20
|
+
/** 按**显示宽度**截断(不是 code unit):逐码点累加 dispWidth,绝不切断 surrogate 对、CJK 也不溢出列。 */
|
|
21
|
+
export declare function truncate(s: string, max: number): string;
|
|
22
|
+
/** 洗不可信显示字符:managed JSON 可随仓库分发,name / sourceLabel / verdict 等字段塞进表格前必须先洗,
|
|
23
|
+
* 否则会破坏表格甚至伪造终端输出。一律映射到可见 U+FFFD(--json 路径保留原值给脚本)。
|
|
24
|
+
* 用 Unicode **属性类**而非手列码点 —— 手列清单天然有缺口(BiDi、U+2028 / 2029、Tags 块都曾漏一轮补一轮),
|
|
25
|
+
* 属性类一次覆盖整类、新码点自动纳入:
|
|
26
|
+
* - `\p{Cc}` 控制符(C0 / C1 / DEL,含 ESC / 换行 / 回车 / TAB)→ 杀 ANSI / OSC 转义与终端控制;
|
|
27
|
+
* - `\p{Cf}` 格式符(BiDi 重排 / 隔离、零宽、joiners、BOM、Tags、interlinear)→ 防 Trojan-Source 视觉伪造与零宽隐藏 / 分割;
|
|
28
|
+
* - `\p{Zl}` / `\p{Zp}` 行 / 段分隔(U+2028 / 2029)→ LF 的 Unicode 孪生,防换行伪造表格行;
|
|
29
|
+
* - `\p{Mn}` / `\p{Me}` 非间距 / 封闭组合附加符 → 变可见,避免零前进宽度令 dispWidth 与终端列错位
|
|
30
|
+
* (间距组合符 `\p{Mc}` 合法占 1 列,保留);
|
|
31
|
+
* - Hangul filler(U+115F / U+1160 / U+3164,属 Lo 不在上述任何类)→ 零宽显示诡计,补列。 */
|
|
32
|
+
export declare function sanitizeCell(s: string): string;
|
|
33
|
+
export declare function renderTable(rows: ManagedListRow[], lang: CliLang): string;
|
|
34
|
+
export default class List extends BaseCommand {
|
|
35
|
+
static description: string;
|
|
36
|
+
static examples: {
|
|
37
|
+
description: string;
|
|
38
|
+
command: string;
|
|
39
|
+
}[];
|
|
40
|
+
static flags: {
|
|
41
|
+
lang: import("@oclif/core/interfaces").OptionFlag<string, import("@oclif/core/interfaces").CustomOptions>;
|
|
42
|
+
global: import("@oclif/core/interfaces").BooleanFlag<boolean>;
|
|
43
|
+
json: import("@oclif/core/interfaces").BooleanFlag<boolean>;
|
|
44
|
+
};
|
|
45
|
+
run(): Promise<void>;
|
|
46
|
+
}
|