oh-my-knowledge 0.30.0 → 0.32.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +33 -672
- package/README.zh.md +45 -684
- package/dist/analysis/coverage-analyzer.d.ts.map +1 -0
- package/dist/analysis/coverage-analyzer.js.map +1 -0
- package/dist/analysis/failure-clusterer.d.ts.map +1 -0
- package/dist/analysis/failure-clusterer.js.map +1 -0
- package/dist/analysis/gap-analyzer.d.ts +121 -0
- package/dist/analysis/gap-analyzer.d.ts.map +1 -0
- package/dist/analysis/gap-analyzer.js +471 -0
- package/dist/analysis/gap-analyzer.js.map +1 -0
- package/dist/analysis/hedging-classifier.d.ts.map +1 -0
- package/dist/analysis/hedging-classifier.js.map +1 -0
- package/dist/analysis/report-diagnostics.d.ts +34 -0
- package/dist/analysis/report-diagnostics.d.ts.map +1 -0
- package/dist/analysis/report-diagnostics.js +753 -0
- package/dist/analysis/report-diagnostics.js.map +1 -0
- package/dist/analysis/sample-diagnostics.d.ts.map +1 -0
- package/dist/analysis/sample-diagnostics.js.map +1 -0
- package/dist/analysis/saturation.d.ts.map +1 -0
- package/dist/analysis/saturation.js.map +1 -0
- package/dist/authoring/evolver.d.ts +102 -0
- package/dist/authoring/evolver.d.ts.map +1 -0
- package/dist/authoring/evolver.js +681 -0
- package/dist/authoring/evolver.js.map +1 -0
- package/dist/authoring/generator.d.ts +38 -0
- package/dist/authoring/generator.d.ts.map +1 -0
- package/dist/authoring/generator.js +720 -0
- package/dist/authoring/generator.js.map +1 -0
- package/dist/authoring/sample-fixer.d.ts +51 -0
- package/dist/authoring/sample-fixer.d.ts.map +1 -0
- package/dist/authoring/sample-fixer.js +213 -0
- package/dist/authoring/sample-fixer.js.map +1 -0
- package/dist/cli/commands/doctor.d.ts +26 -0
- package/dist/cli/commands/doctor.d.ts.map +1 -0
- package/dist/cli/commands/doctor.js +302 -0
- package/dist/cli/commands/doctor.js.map +1 -0
- package/dist/cli/commands/eval/gold/compare.d.ts +17 -0
- package/dist/cli/commands/eval/gold/compare.d.ts.map +1 -0
- package/dist/cli/commands/eval/gold/compare.js +91 -0
- package/dist/cli/commands/eval/gold/compare.js.map +1 -0
- package/dist/cli/commands/eval/gold/index.d.ts +9 -0
- package/dist/cli/commands/eval/gold/index.d.ts.map +1 -0
- package/dist/cli/commands/eval/gold/index.js +34 -0
- package/dist/cli/commands/eval/gold/index.js.map +1 -0
- package/dist/cli/commands/eval/gold/init.d.ts +11 -0
- package/dist/cli/commands/eval/gold/init.d.ts.map +1 -0
- package/dist/cli/commands/eval/gold/init.js +51 -0
- package/dist/cli/commands/eval/gold/init.js.map +1 -0
- package/dist/cli/commands/eval/gold/validate.d.ts +12 -0
- package/dist/cli/commands/eval/gold/validate.d.ts.map +1 -0
- package/dist/cli/commands/eval/gold/validate.js +46 -0
- package/dist/cli/commands/eval/gold/validate.js.map +1 -0
- package/dist/cli/commands/eval/index.d.ts +54 -0
- package/dist/cli/commands/eval/index.d.ts.map +1 -0
- package/dist/cli/commands/eval/index.js +465 -0
- package/dist/cli/commands/eval/index.js.map +1 -0
- package/dist/cli/commands/evolve.d.ts +37 -0
- package/dist/cli/commands/evolve.d.ts.map +1 -0
- package/dist/cli/commands/evolve.js +283 -0
- package/dist/cli/commands/evolve.js.map +1 -0
- package/dist/cli/commands/init.d.ts +16 -0
- package/dist/cli/commands/init.d.ts.map +1 -0
- package/dist/cli/commands/init.js +152 -0
- package/dist/cli/commands/init.js.map +1 -0
- package/dist/cli/commands/observe/inbox.d.ts +23 -0
- package/dist/cli/commands/observe/inbox.d.ts.map +1 -0
- package/dist/cli/commands/observe/inbox.js +260 -0
- package/dist/cli/commands/observe/inbox.js.map +1 -0
- package/dist/cli/commands/observe/index.d.ts +22 -0
- package/dist/cli/commands/observe/index.d.ts.map +1 -0
- package/dist/cli/commands/observe/index.js +118 -0
- package/dist/cli/commands/observe/index.js.map +1 -0
- package/dist/cli/commands/observe/ingest.d.ts +13 -0
- package/dist/cli/commands/observe/ingest.d.ts.map +1 -0
- package/dist/cli/commands/observe/ingest.js +71 -0
- package/dist/cli/commands/observe/ingest.js.map +1 -0
- package/dist/cli/commands/observe/show.d.ts +13 -0
- package/dist/cli/commands/observe/show.d.ts.map +1 -0
- package/dist/cli/commands/observe/show.js +49 -0
- package/dist/cli/commands/observe/show.js.map +1 -0
- package/dist/cli/commands/sample.d.ts +38 -0
- package/dist/cli/commands/sample.d.ts.map +1 -0
- package/dist/cli/commands/sample.js +487 -0
- package/dist/cli/commands/sample.js.map +1 -0
- package/dist/cli/commands/studio.d.ts +23 -0
- package/dist/cli/commands/studio.d.ts.map +1 -0
- package/dist/cli/commands/studio.js +158 -0
- package/dist/cli/commands/studio.js.map +1 -0
- package/dist/cli/index.d.ts.map +1 -0
- package/dist/cli/index.js +29 -0
- package/dist/cli/index.js.map +1 -0
- package/dist/cli/lib/cli-exit.d.ts +16 -0
- package/dist/cli/lib/cli-exit.d.ts.map +1 -0
- package/dist/cli/lib/cli-exit.js +20 -0
- package/dist/cli/lib/cli-exit.js.map +1 -0
- package/dist/cli/lib/cmd-flags.d.ts +224 -0
- package/dist/cli/lib/cmd-flags.d.ts.map +1 -0
- package/dist/cli/lib/cmd-flags.js +46 -0
- package/dist/cli/lib/cmd-flags.js.map +1 -0
- package/dist/cli/lib/i18n-dict/common.d.ts +4 -0
- package/dist/cli/lib/i18n-dict/common.d.ts.map +1 -0
- package/dist/cli/lib/i18n-dict/common.js +67 -0
- package/dist/cli/lib/i18n-dict/common.js.map +1 -0
- package/dist/cli/lib/i18n-dict/evolve.d.ts +4 -0
- package/dist/cli/lib/i18n-dict/evolve.d.ts.map +1 -0
- package/dist/cli/lib/i18n-dict/evolve.js +43 -0
- package/dist/cli/lib/i18n-dict/evolve.js.map +1 -0
- package/dist/cli/lib/i18n-dict/gen.d.ts +4 -0
- package/dist/cli/lib/i18n-dict/gen.d.ts.map +1 -0
- package/dist/cli/lib/i18n-dict/gen.js +63 -0
- package/dist/cli/lib/i18n-dict/gen.js.map +1 -0
- package/dist/cli/lib/i18n-dict/help.d.ts +4 -0
- package/dist/cli/lib/i18n-dict/help.d.ts.map +1 -0
- package/dist/cli/lib/i18n-dict/help.js +281 -0
- package/dist/cli/lib/i18n-dict/help.js.map +1 -0
- package/dist/cli/lib/i18n-dict/init.d.ts +4 -0
- package/dist/cli/lib/i18n-dict/init.d.ts.map +1 -0
- package/dist/cli/lib/i18n-dict/init.js +27 -0
- package/dist/cli/lib/i18n-dict/init.js.map +1 -0
- package/dist/cli/lib/i18n-dict/run.d.ts +4 -0
- package/dist/cli/lib/i18n-dict/run.d.ts.map +1 -0
- package/dist/cli/lib/i18n-dict/run.js +143 -0
- package/dist/cli/lib/i18n-dict/run.js.map +1 -0
- package/dist/cli/lib/i18n-dict/types.d.ts +5 -0
- package/dist/cli/lib/i18n-dict/types.d.ts.map +1 -0
- package/dist/cli/lib/i18n-dict/types.js +2 -0
- package/dist/cli/lib/i18n-dict/types.js.map +1 -0
- package/dist/cli/lib/i18n-dict.d.ts +63 -0
- package/dist/cli/lib/i18n-dict.d.ts.map +1 -0
- package/dist/cli/lib/i18n-dict.js +67 -0
- package/dist/cli/lib/i18n-dict.js.map +1 -0
- package/dist/cli/lib/i18n.d.ts +29 -0
- package/dist/cli/lib/i18n.d.ts.map +1 -0
- package/dist/cli/lib/i18n.js +58 -0
- package/dist/cli/lib/i18n.js.map +1 -0
- package/dist/cli/lib/parse-run-config/judge-models.d.ts +24 -0
- package/dist/cli/lib/parse-run-config/judge-models.d.ts.map +1 -0
- package/dist/cli/lib/parse-run-config/judge-models.js +55 -0
- package/dist/cli/lib/parse-run-config/judge-models.js.map +1 -0
- package/dist/cli/lib/parse-run-config/samples-discovery.d.ts +19 -0
- package/dist/cli/lib/parse-run-config/samples-discovery.d.ts.map +1 -0
- package/dist/cli/lib/parse-run-config/samples-discovery.js +53 -0
- package/dist/cli/lib/parse-run-config/samples-discovery.js.map +1 -0
- package/dist/cli/lib/parse-run-config/variant-resolution.d.ts +18 -0
- package/dist/cli/lib/parse-run-config/variant-resolution.d.ts.map +1 -0
- package/dist/cli/lib/parse-run-config/variant-resolution.js +59 -0
- package/dist/cli/lib/parse-run-config/variant-resolution.js.map +1 -0
- package/dist/cli/lib/parse-run-config.d.ts +93 -0
- package/dist/cli/lib/parse-run-config.d.ts.map +1 -0
- package/dist/cli/lib/parse-run-config.js +157 -0
- package/dist/cli/lib/parse-run-config.js.map +1 -0
- package/dist/cli/lib/progress.d.ts.map +1 -0
- package/dist/cli/lib/progress.js.map +1 -0
- package/dist/cli/lib/resolve-skill-input.d.ts +8 -0
- package/dist/cli/lib/resolve-skill-input.d.ts.map +1 -0
- package/dist/cli/lib/resolve-skill-input.js +38 -0
- package/dist/cli/lib/resolve-skill-input.js.map +1 -0
- package/dist/cli/lib/run-tally.d.ts +18 -0
- package/dist/cli/lib/run-tally.d.ts.map +1 -0
- package/dist/cli/lib/run-tally.js.map +1 -0
- package/dist/cli/lib/shared.d.ts +12 -0
- package/dist/cli/lib/shared.d.ts.map +1 -0
- package/dist/cli/lib/shared.js +26 -0
- package/dist/cli/lib/shared.js.map +1 -0
- package/dist/cli/lib/update-check.d.ts +9 -0
- package/dist/cli/lib/update-check.d.ts.map +1 -0
- package/dist/cli/lib/update-check.js +113 -0
- package/dist/cli/lib/update-check.js.map +1 -0
- package/dist/cli/oclif/base-command.d.ts +8 -0
- package/dist/cli/oclif/base-command.d.ts.map +1 -0
- package/dist/cli/oclif/base-command.js +27 -0
- package/dist/cli/oclif/base-command.js.map +1 -0
- package/dist/cli/oclif/help.d.ts +16 -0
- package/dist/cli/oclif/help.d.ts.map +1 -0
- package/dist/cli/oclif/help.js +33 -0
- package/dist/cli/oclif/help.js.map +1 -0
- package/dist/cli/oclif/i18n.d.ts +16 -0
- package/dist/cli/oclif/i18n.d.ts.map +1 -0
- package/dist/cli/oclif/i18n.js +62 -0
- package/dist/cli/oclif/i18n.js.map +1 -0
- package/dist/cli/oclif/parsers.d.ts +10 -0
- package/dist/cli/oclif/parsers.d.ts.map +1 -0
- package/dist/cli/oclif/parsers.js +102 -0
- package/dist/cli/oclif/parsers.js.map +1 -0
- package/dist/cli/oclif/projection.d.ts +23 -0
- package/dist/cli/oclif/projection.d.ts.map +1 -0
- package/dist/cli/oclif/projection.js +52 -0
- package/dist/cli/oclif/projection.js.map +1 -0
- package/dist/cli/oclif/run.d.ts +2 -0
- package/dist/cli/oclif/run.d.ts.map +1 -0
- package/dist/cli/oclif/run.js +11 -0
- package/dist/cli/oclif/run.js.map +1 -0
- package/dist/diagnosis/observe-mapper.d.ts +68 -0
- package/dist/diagnosis/observe-mapper.d.ts.map +1 -0
- package/dist/diagnosis/observe-mapper.js +276 -0
- package/dist/diagnosis/observe-mapper.js.map +1 -0
- package/dist/diagnosis/observe-producer.d.ts +9 -0
- package/dist/diagnosis/observe-producer.d.ts.map +1 -0
- package/dist/diagnosis/observe-producer.js +199 -0
- package/dist/diagnosis/observe-producer.js.map +1 -0
- package/dist/diagnosis/studio-projection.d.ts +7 -0
- package/dist/diagnosis/studio-projection.d.ts.map +1 -0
- package/dist/diagnosis/studio-projection.js +84 -0
- package/dist/diagnosis/studio-projection.js.map +1 -0
- package/dist/diagnosis/types.d.ts +4 -0
- package/dist/diagnosis/types.d.ts.map +1 -0
- package/dist/diagnosis/types.js +20 -0
- package/dist/diagnosis/types.js.map +1 -0
- package/dist/doctor/fixer.d.ts +12 -0
- package/dist/doctor/fixer.d.ts.map +1 -0
- package/dist/doctor/fixer.js +326 -0
- package/dist/doctor/fixer.js.map +1 -0
- package/dist/doctor/health/builtin-dimensions.d.ts.map +1 -0
- package/dist/doctor/health/builtin-dimensions.js.map +1 -0
- package/dist/doctor/health/composer.d.ts.map +1 -0
- package/dist/doctor/health/composer.js +296 -0
- package/dist/doctor/health/composer.js.map +1 -0
- package/dist/doctor/health/dimension-registry.d.ts.map +1 -0
- package/dist/doctor/health/dimension-registry.js.map +1 -0
- package/dist/doctor/health/dimension-spec.d.ts +47 -0
- package/dist/doctor/health/dimension-spec.d.ts.map +1 -0
- package/dist/doctor/health/dimension-spec.js.map +1 -0
- package/dist/doctor/health/parser.d.ts +26 -0
- package/dist/doctor/health/parser.d.ts.map +1 -0
- package/dist/doctor/health/parser.js +228 -0
- package/dist/doctor/health/parser.js.map +1 -0
- package/dist/doctor/health/prompt-builder.d.ts +3 -0
- package/dist/doctor/health/prompt-builder.d.ts.map +1 -0
- package/dist/doctor/health/prompt-builder.js +2 -0
- package/dist/doctor/health/prompt-builder.js.map +1 -0
- package/dist/doctor/health/register.d.ts.map +1 -0
- package/dist/doctor/health/register.js.map +1 -0
- package/dist/doctor/index.d.ts.map +1 -0
- package/dist/doctor/index.js +246 -0
- package/dist/doctor/index.js.map +1 -0
- package/dist/doctor/messages.d.ts +21 -0
- package/dist/doctor/messages.d.ts.map +1 -0
- package/dist/doctor/messages.js +217 -0
- package/dist/doctor/messages.js.map +1 -0
- package/dist/doctor/preflight.d.ts.map +1 -0
- package/dist/doctor/preflight.js.map +1 -0
- package/dist/doctor/renderer.d.ts +17 -0
- package/dist/doctor/renderer.d.ts.map +1 -0
- package/dist/doctor/renderer.js +122 -0
- package/dist/doctor/renderer.js.map +1 -0
- package/dist/doctor/rules.d.ts.map +1 -0
- package/dist/doctor/rules.js +289 -0
- package/dist/doctor/rules.js.map +1 -0
- package/dist/eval-core/bootstrap.d.ts.map +1 -0
- package/dist/eval-core/bootstrap.js.map +1 -0
- package/dist/eval-core/cache.d.ts.map +1 -0
- package/dist/eval-core/cache.js.map +1 -0
- package/dist/eval-core/comparability.d.ts.map +1 -0
- package/dist/eval-core/comparability.js.map +1 -0
- package/dist/eval-core/dependency-checker.d.ts +37 -0
- package/dist/eval-core/dependency-checker.d.ts.map +1 -0
- package/dist/eval-core/dependency-checker.js.map +1 -0
- package/dist/eval-core/evaluation-execution.d.ts.map +1 -0
- package/dist/eval-core/evaluation-execution.js.map +1 -0
- package/dist/eval-core/evaluation-job.d.ts.map +1 -0
- package/dist/eval-core/evaluation-job.js.map +1 -0
- package/dist/eval-core/evaluation-reporting.d.ts.map +1 -0
- package/dist/eval-core/evaluation-reporting.js.map +1 -0
- package/dist/eval-core/execution-strategy.d.ts.map +1 -0
- package/dist/eval-core/execution-strategy.js +165 -0
- package/dist/eval-core/execution-strategy.js.map +1 -0
- package/dist/eval-core/fact-checker.d.ts.map +1 -0
- package/dist/eval-core/fact-checker.js.map +1 -0
- package/dist/eval-core/layer-gates.d.ts.map +1 -0
- package/dist/eval-core/layer-gates.js.map +1 -0
- package/dist/eval-core/mock-hook.cjs +215 -0
- package/dist/eval-core/mocks-runtime.d.ts +100 -0
- package/dist/eval-core/mocks-runtime.d.ts.map +1 -0
- package/dist/eval-core/mocks-runtime.js +559 -0
- package/dist/eval-core/mocks-runtime.js.map +1 -0
- package/dist/eval-core/schema.d.ts.map +1 -0
- package/dist/eval-core/schema.js.map +1 -0
- package/dist/eval-core/statistics.d.ts.map +1 -0
- package/dist/eval-core/statistics.js.map +1 -0
- package/dist/eval-core/task-planner.d.ts.map +1 -0
- package/dist/eval-core/task-planner.js.map +1 -0
- package/dist/eval-core/verdict.d.ts.map +1 -0
- package/dist/eval-core/verdict.js.map +1 -0
- package/dist/eval-workflows/batch-evaluation-workflow.d.ts.map +1 -0
- package/dist/eval-workflows/batch-evaluation-workflow.js.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.d.ts +22 -0
- package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.d.ts.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.js +67 -0
- package/dist/eval-workflows/evaluation-pipeline/preflight-warnings.js.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline/report-finalize.d.ts +32 -0
- package/dist/eval-workflows/evaluation-pipeline/report-finalize.d.ts.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline/report-finalize.js +57 -0
- package/dist/eval-workflows/evaluation-pipeline/report-finalize.js.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline/run-state.d.ts +60 -0
- package/dist/eval-workflows/evaluation-pipeline/run-state.d.ts.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline/run-state.js +88 -0
- package/dist/eval-workflows/evaluation-pipeline/run-state.js.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline/test-set-hash.d.ts +21 -0
- package/dist/eval-workflows/evaluation-pipeline/test-set-hash.d.ts.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline/test-set-hash.js +46 -0
- package/dist/eval-workflows/evaluation-pipeline/test-set-hash.js.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline.d.ts +97 -0
- package/dist/eval-workflows/evaluation-pipeline.d.ts.map +1 -0
- package/dist/eval-workflows/evaluation-pipeline.js +179 -0
- package/dist/eval-workflows/evaluation-pipeline.js.map +1 -0
- package/dist/eval-workflows/evaluation-preparation.d.ts.map +1 -0
- package/dist/eval-workflows/evaluation-preparation.js.map +1 -0
- package/dist/eval-workflows/messages.d.ts +4 -0
- package/dist/eval-workflows/messages.d.ts.map +1 -0
- package/dist/eval-workflows/messages.js +30 -0
- package/dist/eval-workflows/messages.js.map +1 -0
- package/dist/eval-workflows/run-evaluation.d.ts.map +1 -0
- package/dist/eval-workflows/run-evaluation.js +528 -0
- package/dist/eval-workflows/run-evaluation.js.map +1 -0
- package/dist/executors/anthropic-api.d.ts.map +1 -0
- package/dist/executors/anthropic-api.js.map +1 -0
- package/dist/executors/claude-cli.d.ts.map +1 -0
- package/dist/executors/claude-cli.js +181 -0
- package/dist/executors/claude-cli.js.map +1 -0
- package/dist/executors/claude-sdk-trace.d.ts.map +1 -0
- package/dist/executors/claude-sdk-trace.js.map +1 -0
- package/dist/executors/claude-sdk.d.ts.map +1 -0
- package/dist/executors/claude-sdk.js.map +1 -0
- package/dist/executors/codex-cli-trace.d.ts.map +1 -0
- package/dist/executors/codex-cli-trace.js.map +1 -0
- package/dist/executors/codex-cli.d.ts.map +1 -0
- package/dist/executors/codex-cli.js.map +1 -0
- package/dist/executors/codex-sdk.d.ts.map +1 -0
- package/dist/executors/codex-sdk.js.map +1 -0
- package/dist/executors/gemini.d.ts.map +1 -0
- package/dist/executors/gemini.js.map +1 -0
- package/dist/executors/index.d.ts.map +1 -0
- package/dist/executors/index.js.map +1 -0
- package/dist/executors/openai-api.d.ts.map +1 -0
- package/dist/executors/openai-api.js.map +1 -0
- package/dist/executors/runtime-fingerprint.d.ts.map +1 -0
- package/dist/executors/runtime-fingerprint.js.map +1 -0
- package/dist/executors/script.d.ts.map +1 -0
- package/dist/executors/script.js.map +1 -0
- package/dist/executors/shared.d.ts +194 -0
- package/dist/executors/shared.d.ts.map +1 -0
- package/dist/executors/shared.js +253 -0
- package/dist/executors/shared.js.map +1 -0
- package/dist/grading/assertions.d.ts.map +1 -0
- package/dist/grading/assertions.js.map +1 -0
- package/dist/grading/debias-validate.d.ts.map +1 -0
- package/dist/grading/debias-validate.js.map +1 -0
- package/dist/grading/diagnostic.d.ts.map +1 -0
- package/dist/grading/diagnostic.js.map +1 -0
- package/dist/grading/gold-cli.d.ts.map +1 -0
- package/dist/grading/gold-cli.js.map +1 -0
- package/dist/grading/gold-dataset.d.ts.map +1 -0
- package/dist/grading/gold-dataset.js.map +1 -0
- package/dist/grading/human-gold.d.ts.map +1 -0
- package/dist/grading/human-gold.js.map +1 -0
- package/dist/grading/index.d.ts.map +1 -0
- package/dist/grading/index.js.map +1 -0
- package/dist/grading/judge.d.ts.map +1 -0
- package/dist/grading/judge.js.map +1 -0
- package/dist/grading/layered-scores.d.ts.map +1 -0
- package/dist/grading/layered-scores.js.map +1 -0
- package/dist/inputs/eval-config.d.ts.map +1 -0
- package/dist/inputs/eval-config.js.map +1 -0
- package/dist/inputs/load-samples.d.ts.map +1 -0
- package/dist/inputs/load-samples.js.map +1 -0
- package/dist/inputs/mcp-resolver.d.ts.map +1 -0
- package/dist/inputs/mcp-resolver.js.map +1 -0
- package/dist/inputs/skill-loader.d.ts.map +1 -0
- package/dist/inputs/skill-loader.js +261 -0
- package/dist/inputs/skill-loader.js.map +1 -0
- package/dist/inputs/url-fetcher.d.ts.map +1 -0
- package/dist/inputs/url-fetcher.js.map +1 -0
- package/dist/observability/experience-frontmatter.d.ts +37 -0
- package/dist/observability/experience-frontmatter.d.ts.map +1 -0
- package/dist/observability/experience-frontmatter.js +100 -0
- package/dist/observability/experience-frontmatter.js.map +1 -0
- package/dist/observability/experience.d.ts +23 -0
- package/dist/observability/experience.d.ts.map +1 -0
- package/dist/observability/experience.js +3438 -0
- package/dist/observability/experience.js.map +1 -0
- package/dist/observability/feedback-matchers.d.ts +37 -0
- package/dist/observability/feedback-matchers.d.ts.map +1 -0
- package/dist/observability/feedback-matchers.js +232 -0
- package/dist/observability/feedback-matchers.js.map +1 -0
- package/dist/observability/feedback-projection.d.ts +21 -0
- package/dist/observability/feedback-projection.d.ts.map +1 -0
- package/dist/observability/feedback-projection.js +20 -0
- package/dist/observability/feedback-projection.js.map +1 -0
- package/dist/observability/inbox-view-model.d.ts +43 -0
- package/dist/observability/inbox-view-model.d.ts.map +1 -0
- package/dist/observability/inbox-view-model.js +180 -0
- package/dist/observability/inbox-view-model.js.map +1 -0
- package/dist/observability/inbox.d.ts +28 -0
- package/dist/observability/inbox.d.ts.map +1 -0
- package/dist/observability/inbox.js +765 -0
- package/dist/observability/inbox.js.map +1 -0
- package/dist/observability/problem-patterns.d.ts +12 -0
- package/dist/observability/problem-patterns.d.ts.map +1 -0
- package/dist/observability/problem-patterns.js +244 -0
- package/dist/observability/problem-patterns.js.map +1 -0
- package/dist/observability/prompts/llm-enhanced-review.prompt.md +224 -0
- package/dist/observability/resolved-review.d.ts +37 -0
- package/dist/observability/resolved-review.d.ts.map +1 -0
- package/dist/observability/resolved-review.js +552 -0
- package/dist/observability/resolved-review.js.map +1 -0
- package/dist/observability/review-state.d.ts +20 -0
- package/dist/observability/review-state.d.ts.map +1 -0
- package/dist/observability/review-state.js +169 -0
- package/dist/observability/review-state.js.map +1 -0
- package/dist/observability/skill-chain-advisories.d.ts +16 -0
- package/dist/observability/skill-chain-advisories.d.ts.map +1 -0
- package/dist/observability/skill-chain-advisories.js +69 -0
- package/dist/observability/skill-chain-advisories.js.map +1 -0
- package/dist/observability/skill-chain.d.ts +7 -0
- package/dist/observability/skill-chain.d.ts.map +1 -0
- package/dist/observability/skill-chain.js +474 -0
- package/dist/observability/skill-chain.js.map +1 -0
- package/dist/observability/skill-health-analyzer.d.ts.map +1 -0
- package/dist/observability/skill-health-analyzer.js.map +1 -0
- package/dist/observability/soft-standards/constants.d.ts +5 -0
- package/dist/observability/soft-standards/constants.d.ts.map +1 -0
- package/dist/observability/soft-standards/constants.js +10 -0
- package/dist/observability/soft-standards/constants.js.map +1 -0
- package/dist/observability/soft-standards/index.d.ts +16 -0
- package/dist/observability/soft-standards/index.d.ts.map +1 -0
- package/dist/observability/soft-standards/index.js +15 -0
- package/dist/observability/soft-standards/index.js.map +1 -0
- package/dist/observability/soft-standards/llm-extractor.d.ts +6 -0
- package/dist/observability/soft-standards/llm-extractor.d.ts.map +1 -0
- package/dist/observability/soft-standards/llm-extractor.js +692 -0
- package/dist/observability/soft-standards/llm-extractor.js.map +1 -0
- package/dist/observability/soft-standards/runtime-evaluator.d.ts +12 -0
- package/dist/observability/soft-standards/runtime-evaluator.d.ts.map +1 -0
- package/dist/observability/soft-standards/runtime-evaluator.js +385 -0
- package/dist/observability/soft-standards/runtime-evaluator.js.map +1 -0
- package/dist/observability/soft-standards/skill-standards-store.d.ts +11 -0
- package/dist/observability/soft-standards/skill-standards-store.d.ts.map +1 -0
- package/dist/observability/soft-standards/skill-standards-store.js +158 -0
- package/dist/observability/soft-standards/skill-standards-store.js.map +1 -0
- package/dist/observability/soft-standards/types.d.ts +212 -0
- package/dist/observability/soft-standards/types.d.ts.map +1 -0
- package/dist/observability/soft-standards/types.js +2 -0
- package/dist/observability/soft-standards/types.js.map +1 -0
- package/dist/observability/text-signals.d.ts +14 -0
- package/dist/observability/text-signals.d.ts.map +1 -0
- package/dist/observability/text-signals.js +116 -0
- package/dist/observability/text-signals.js.map +1 -0
- package/dist/observability/trace-adapter.d.ts.map +1 -0
- package/dist/observability/trace-adapter.js.map +1 -0
- package/dist/observability/trace-attribution.d.ts +56 -0
- package/dist/observability/trace-attribution.d.ts.map +1 -0
- package/dist/observability/trace-attribution.js +200 -0
- package/dist/observability/trace-attribution.js.map +1 -0
- package/dist/observability/trace-segmenter.d.ts +52 -0
- package/dist/observability/trace-segmenter.d.ts.map +1 -0
- package/dist/observability/trace-segmenter.js +322 -0
- package/dist/observability/trace-segmenter.js.map +1 -0
- package/dist/observability/trace-source.d.ts +92 -0
- package/dist/observability/trace-source.d.ts.map +1 -0
- package/dist/observability/trace-source.js +502 -0
- package/dist/observability/trace-source.js.map +1 -0
- package/dist/renderer/html-renderer.d.ts.map +1 -0
- package/dist/renderer/html-renderer.js.map +1 -0
- package/dist/renderer/layout.d.ts.map +1 -0
- package/dist/renderer/layout.js.map +1 -0
- package/dist/renderer/observation-inbox/helpers.d.ts +14 -0
- package/dist/renderer/observation-inbox/helpers.d.ts.map +1 -0
- package/dist/renderer/observation-inbox/helpers.js +75 -0
- package/dist/renderer/observation-inbox/helpers.js.map +1 -0
- package/dist/renderer/observation-inbox/styles.d.ts +2 -0
- package/dist/renderer/observation-inbox/styles.d.ts.map +1 -0
- package/dist/renderer/observation-inbox/styles.js +5184 -0
- package/dist/renderer/observation-inbox/styles.js.map +1 -0
- package/dist/renderer/observation-inbox-renderer.d.ts +5 -0
- package/dist/renderer/observation-inbox-renderer.d.ts.map +1 -0
- package/dist/renderer/observation-inbox-renderer.js +7182 -0
- package/dist/renderer/observation-inbox-renderer.js.map +1 -0
- package/dist/renderer/skill-detail-renderer.d.ts +13 -0
- package/dist/renderer/skill-detail-renderer.d.ts.map +1 -0
- package/dist/renderer/skill-detail-renderer.js +1244 -0
- package/dist/renderer/skill-detail-renderer.js.map +1 -0
- package/dist/renderer/skill-health-renderer.d.ts.map +1 -0
- package/dist/renderer/skill-health-renderer.js.map +1 -0
- package/dist/renderer/skill-list-renderer.d.ts +3 -0
- package/dist/renderer/skill-list-renderer.d.ts.map +1 -0
- package/dist/renderer/skill-list-renderer.js +334 -0
- package/dist/renderer/skill-list-renderer.js.map +1 -0
- package/dist/renderer/summary.d.ts +68 -0
- package/dist/renderer/summary.d.ts.map +1 -0
- package/dist/renderer/summary.js +1896 -0
- package/dist/renderer/summary.js.map +1 -0
- package/dist/renderer/table.d.ts.map +1 -0
- package/dist/renderer/table.js.map +1 -0
- package/dist/renderer/test-view.d.ts.map +1 -0
- package/dist/renderer/test-view.js +905 -0
- package/dist/renderer/test-view.js.map +1 -0
- package/dist/renderer/trends.d.ts.map +1 -0
- package/dist/renderer/trends.js.map +1 -0
- package/dist/server/job-store.d.ts.map +1 -0
- package/dist/server/job-store.js.map +1 -0
- package/dist/server/report-server.d.ts.map +1 -0
- package/dist/server/report-server.js +892 -0
- package/dist/server/report-server.js.map +1 -0
- package/dist/server/report-store.d.ts.map +1 -0
- package/dist/server/report-store.js.map +1 -0
- package/dist/server/skill-index.d.ts +14 -0
- package/dist/server/skill-index.d.ts.map +1 -0
- package/dist/server/skill-index.js +381 -0
- package/dist/server/skill-index.js.map +1 -0
- package/dist/server/skill-insights.d.ts +32 -0
- package/dist/server/skill-insights.d.ts.map +1 -0
- package/dist/server/skill-insights.js +788 -0
- package/dist/server/skill-insights.js.map +1 -0
- package/dist/shared/hard-rules.d.ts +42 -0
- package/dist/shared/hard-rules.d.ts.map +1 -0
- package/dist/shared/hard-rules.js +242 -0
- package/dist/shared/hard-rules.js.map +1 -0
- package/dist/shared/llm-prompts/index.d.ts +14 -0
- package/dist/shared/llm-prompts/index.d.ts.map +1 -0
- package/dist/shared/llm-prompts/index.js +22 -0
- package/dist/shared/llm-prompts/index.js.map +1 -0
- package/dist/shared/llm-prompts/skill-health.d.ts +22 -0
- package/dist/shared/llm-prompts/skill-health.d.ts.map +1 -0
- package/dist/shared/llm-prompts/skill-health.js +170 -0
- package/dist/shared/llm-prompts/skill-health.js.map +1 -0
- package/dist/shared/time.d.ts.map +1 -0
- package/dist/shared/time.js.map +1 -0
- package/dist/shared/tool-search.d.ts.map +1 -0
- package/dist/shared/tool-search.js.map +1 -0
- package/dist/types/dependencies.d.ts +23 -0
- package/dist/types/dependencies.d.ts.map +1 -0
- package/dist/types/dependencies.js +2 -0
- package/dist/types/dependencies.js.map +1 -0
- package/dist/types/diagnosis.d.ts +88 -0
- package/dist/types/diagnosis.d.ts.map +1 -0
- package/dist/types/diagnosis.js +2 -0
- package/dist/types/diagnosis.js.map +1 -0
- package/dist/types/doctor.d.ts +157 -0
- package/dist/types/doctor.d.ts.map +1 -0
- package/dist/types/doctor.js.map +1 -0
- package/dist/types/eval.d.ts +377 -0
- package/dist/types/eval.d.ts.map +1 -0
- package/dist/types/eval.js.map +1 -0
- package/dist/types/executor.d.ts.map +1 -0
- package/dist/types/executor.js.map +1 -0
- package/dist/types/index.d.ts +12 -0
- package/dist/types/index.d.ts.map +1 -0
- package/dist/types/index.js +12 -0
- package/dist/types/index.js.map +1 -0
- package/dist/types/judge.d.ts.map +1 -0
- package/dist/types/judge.js.map +1 -0
- package/dist/types/observability.d.ts +865 -0
- package/dist/types/observability.d.ts.map +1 -0
- package/dist/types/observability.js +14 -0
- package/dist/types/observability.js.map +1 -0
- package/dist/types/report.d.ts +560 -0
- package/dist/types/report.d.ts.map +1 -0
- package/dist/types/report.js.map +1 -0
- package/dist/types/shared.d.ts.map +1 -0
- package/dist/types/shared.js.map +1 -0
- package/dist/types/skill-index.d.ts +142 -0
- package/dist/types/skill-index.d.ts.map +1 -0
- package/dist/types/skill-index.js +2 -0
- package/dist/types/skill-index.js.map +1 -0
- package/dist/types/storage.d.ts.map +1 -0
- package/dist/types/storage.js.map +1 -0
- package/dist/util/safe-slice.d.ts +23 -0
- package/dist/util/safe-slice.d.ts.map +1 -0
- package/dist/util/safe-slice.js +33 -0
- package/dist/util/safe-slice.js.map +1 -0
- package/package.json +40 -8
- package/dist/src/analysis/coverage-analyzer.d.ts.map +0 -1
- package/dist/src/analysis/coverage-analyzer.js.map +0 -1
- package/dist/src/analysis/failure-clusterer.d.ts.map +0 -1
- package/dist/src/analysis/failure-clusterer.js.map +0 -1
- package/dist/src/analysis/gap-analyzer.d.ts +0 -121
- package/dist/src/analysis/gap-analyzer.d.ts.map +0 -1
- package/dist/src/analysis/gap-analyzer.js +0 -471
- package/dist/src/analysis/gap-analyzer.js.map +0 -1
- package/dist/src/analysis/hedging-classifier.d.ts.map +0 -1
- package/dist/src/analysis/hedging-classifier.js.map +0 -1
- package/dist/src/analysis/report-diagnostics.d.ts +0 -34
- package/dist/src/analysis/report-diagnostics.d.ts.map +0 -1
- package/dist/src/analysis/report-diagnostics.js +0 -753
- package/dist/src/analysis/report-diagnostics.js.map +0 -1
- package/dist/src/analysis/sample-diagnostics.d.ts.map +0 -1
- package/dist/src/analysis/sample-diagnostics.js.map +0 -1
- package/dist/src/analysis/saturation.d.ts.map +0 -1
- package/dist/src/analysis/saturation.js.map +0 -1
- package/dist/src/authoring/evolver.d.ts +0 -78
- package/dist/src/authoring/evolver.d.ts.map +0 -1
- package/dist/src/authoring/evolver.js +0 -320
- package/dist/src/authoring/evolver.js.map +0 -1
- package/dist/src/authoring/generator.d.ts +0 -35
- package/dist/src/authoring/generator.d.ts.map +0 -1
- package/dist/src/authoring/generator.js +0 -704
- package/dist/src/authoring/generator.js.map +0 -1
- package/dist/src/authoring/sample-fixer.d.ts +0 -45
- package/dist/src/authoring/sample-fixer.d.ts.map +0 -1
- package/dist/src/authoring/sample-fixer.js +0 -186
- package/dist/src/authoring/sample-fixer.js.map +0 -1
- package/dist/src/cli/cli-exit.d.ts +0 -15
- package/dist/src/cli/cli-exit.d.ts.map +0 -1
- package/dist/src/cli/cli-exit.js +0 -19
- package/dist/src/cli/cli-exit.js.map +0 -1
- package/dist/src/cli/commands/_shared.d.ts +0 -12
- package/dist/src/cli/commands/_shared.d.ts.map +0 -1
- package/dist/src/cli/commands/_shared.js +0 -26
- package/dist/src/cli/commands/_shared.js.map +0 -1
- package/dist/src/cli/commands/doctor.d.ts +0 -2
- package/dist/src/cli/commands/doctor.d.ts.map +0 -1
- package/dist/src/cli/commands/doctor.js +0 -174
- package/dist/src/cli/commands/doctor.js.map +0 -1
- package/dist/src/cli/commands/eval-gold.d.ts +0 -2
- package/dist/src/cli/commands/eval-gold.d.ts.map +0 -1
- package/dist/src/cli/commands/eval-gold.js +0 -137
- package/dist/src/cli/commands/eval-gold.js.map +0 -1
- package/dist/src/cli/commands/eval-runner.d.ts +0 -2
- package/dist/src/cli/commands/eval-runner.d.ts.map +0 -1
- package/dist/src/cli/commands/eval-runner.js +0 -299
- package/dist/src/cli/commands/eval-runner.js.map +0 -1
- package/dist/src/cli/commands/eval.d.ts +0 -2
- package/dist/src/cli/commands/eval.d.ts.map +0 -1
- package/dist/src/cli/commands/eval.js +0 -11
- package/dist/src/cli/commands/eval.js.map +0 -1
- package/dist/src/cli/commands/evolve.d.ts +0 -2
- package/dist/src/cli/commands/evolve.d.ts.map +0 -1
- package/dist/src/cli/commands/evolve.js +0 -132
- package/dist/src/cli/commands/evolve.js.map +0 -1
- package/dist/src/cli/commands/init.d.ts +0 -2
- package/dist/src/cli/commands/init.d.ts.map +0 -1
- package/dist/src/cli/commands/init.js +0 -110
- package/dist/src/cli/commands/init.js.map +0 -1
- package/dist/src/cli/commands/observe.d.ts +0 -2
- package/dist/src/cli/commands/observe.d.ts.map +0 -1
- package/dist/src/cli/commands/observe.js +0 -247
- package/dist/src/cli/commands/observe.js.map +0 -1
- package/dist/src/cli/commands/registry.d.ts +0 -11
- package/dist/src/cli/commands/registry.d.ts.map +0 -1
- package/dist/src/cli/commands/registry.js +0 -32
- package/dist/src/cli/commands/registry.js.map +0 -1
- package/dist/src/cli/commands/sample.d.ts +0 -15
- package/dist/src/cli/commands/sample.d.ts.map +0 -1
- package/dist/src/cli/commands/sample.js +0 -415
- package/dist/src/cli/commands/sample.js.map +0 -1
- package/dist/src/cli/commands/studio.d.ts +0 -2
- package/dist/src/cli/commands/studio.d.ts.map +0 -1
- package/dist/src/cli/commands/studio.js +0 -84
- package/dist/src/cli/commands/studio.js.map +0 -1
- package/dist/src/cli/i18n-dict.d.ts +0 -54
- package/dist/src/cli/i18n-dict.d.ts.map +0 -1
- package/dist/src/cli/i18n-dict.js +0 -1125
- package/dist/src/cli/i18n-dict.js.map +0 -1
- package/dist/src/cli/i18n.d.ts +0 -24
- package/dist/src/cli/i18n.d.ts.map +0 -1
- package/dist/src/cli/i18n.js +0 -53
- package/dist/src/cli/i18n.js.map +0 -1
- package/dist/src/cli/index.d.ts.map +0 -1
- package/dist/src/cli/index.js +0 -61
- package/dist/src/cli/index.js.map +0 -1
- package/dist/src/cli/parse-run-config.d.ts +0 -90
- package/dist/src/cli/parse-run-config.d.ts.map +0 -1
- package/dist/src/cli/parse-run-config.js +0 -302
- package/dist/src/cli/parse-run-config.js.map +0 -1
- package/dist/src/cli/parse-strict.d.ts +0 -7
- package/dist/src/cli/parse-strict.d.ts.map +0 -1
- package/dist/src/cli/parse-strict.js +0 -26
- package/dist/src/cli/parse-strict.js.map +0 -1
- package/dist/src/cli/progress.d.ts.map +0 -1
- package/dist/src/cli/progress.js.map +0 -1
- package/dist/src/cli/run-tally.d.ts +0 -18
- package/dist/src/cli/run-tally.d.ts.map +0 -1
- package/dist/src/cli/run-tally.js.map +0 -1
- package/dist/src/cli/update-check.d.ts +0 -3
- package/dist/src/cli/update-check.d.ts.map +0 -1
- package/dist/src/cli/update-check.js +0 -37
- package/dist/src/cli/update-check.js.map +0 -1
- package/dist/src/doctor/health/builtin-dimensions.d.ts.map +0 -1
- package/dist/src/doctor/health/builtin-dimensions.js.map +0 -1
- package/dist/src/doctor/health/composer.d.ts.map +0 -1
- package/dist/src/doctor/health/composer.js +0 -293
- package/dist/src/doctor/health/composer.js.map +0 -1
- package/dist/src/doctor/health/dimension-registry.d.ts.map +0 -1
- package/dist/src/doctor/health/dimension-registry.js.map +0 -1
- package/dist/src/doctor/health/dimension-spec.d.ts +0 -47
- package/dist/src/doctor/health/dimension-spec.d.ts.map +0 -1
- package/dist/src/doctor/health/dimension-spec.js.map +0 -1
- package/dist/src/doctor/health/parser.d.ts +0 -27
- package/dist/src/doctor/health/parser.d.ts.map +0 -1
- package/dist/src/doctor/health/parser.js +0 -191
- package/dist/src/doctor/health/parser.js.map +0 -1
- package/dist/src/doctor/health/prompt-builder.d.ts +0 -22
- package/dist/src/doctor/health/prompt-builder.d.ts.map +0 -1
- package/dist/src/doctor/health/prompt-builder.js +0 -170
- package/dist/src/doctor/health/prompt-builder.js.map +0 -1
- package/dist/src/doctor/health/register.d.ts.map +0 -1
- package/dist/src/doctor/health/register.js.map +0 -1
- package/dist/src/doctor/html-renderer.d.ts +0 -20
- package/dist/src/doctor/html-renderer.d.ts.map +0 -1
- package/dist/src/doctor/html-renderer.js +0 -376
- package/dist/src/doctor/html-renderer.js.map +0 -1
- package/dist/src/doctor/index.d.ts.map +0 -1
- package/dist/src/doctor/index.js +0 -245
- package/dist/src/doctor/index.js.map +0 -1
- package/dist/src/doctor/preflight.d.ts.map +0 -1
- package/dist/src/doctor/preflight.js.map +0 -1
- package/dist/src/doctor/renderer.d.ts +0 -17
- package/dist/src/doctor/renderer.d.ts.map +0 -1
- package/dist/src/doctor/renderer.js +0 -123
- package/dist/src/doctor/renderer.js.map +0 -1
- package/dist/src/doctor/rules.d.ts.map +0 -1
- package/dist/src/doctor/rules.js +0 -289
- package/dist/src/doctor/rules.js.map +0 -1
- package/dist/src/eval-core/bootstrap.d.ts.map +0 -1
- package/dist/src/eval-core/bootstrap.js.map +0 -1
- package/dist/src/eval-core/cache.d.ts.map +0 -1
- package/dist/src/eval-core/cache.js.map +0 -1
- package/dist/src/eval-core/comparability.d.ts.map +0 -1
- package/dist/src/eval-core/comparability.js.map +0 -1
- package/dist/src/eval-core/dependency-checker.d.ts +0 -58
- package/dist/src/eval-core/dependency-checker.d.ts.map +0 -1
- package/dist/src/eval-core/dependency-checker.js.map +0 -1
- package/dist/src/eval-core/evaluation-execution.d.ts.map +0 -1
- package/dist/src/eval-core/evaluation-execution.js.map +0 -1
- package/dist/src/eval-core/evaluation-job.d.ts.map +0 -1
- package/dist/src/eval-core/evaluation-job.js.map +0 -1
- package/dist/src/eval-core/evaluation-reporting.d.ts.map +0 -1
- package/dist/src/eval-core/evaluation-reporting.js.map +0 -1
- package/dist/src/eval-core/execution-strategy.d.ts.map +0 -1
- package/dist/src/eval-core/execution-strategy.js +0 -165
- package/dist/src/eval-core/execution-strategy.js.map +0 -1
- package/dist/src/eval-core/fact-checker.d.ts.map +0 -1
- package/dist/src/eval-core/fact-checker.js.map +0 -1
- package/dist/src/eval-core/layer-gates.d.ts.map +0 -1
- package/dist/src/eval-core/layer-gates.js.map +0 -1
- package/dist/src/eval-core/mock-hook.cjs +0 -203
- package/dist/src/eval-core/mocks-runtime.d.ts +0 -96
- package/dist/src/eval-core/mocks-runtime.d.ts.map +0 -1
- package/dist/src/eval-core/mocks-runtime.js +0 -323
- package/dist/src/eval-core/mocks-runtime.js.map +0 -1
- package/dist/src/eval-core/schema.d.ts.map +0 -1
- package/dist/src/eval-core/schema.js.map +0 -1
- package/dist/src/eval-core/statistics.d.ts.map +0 -1
- package/dist/src/eval-core/statistics.js.map +0 -1
- package/dist/src/eval-core/task-planner.d.ts.map +0 -1
- package/dist/src/eval-core/task-planner.js.map +0 -1
- package/dist/src/eval-core/verdict.d.ts.map +0 -1
- package/dist/src/eval-core/verdict.js.map +0 -1
- package/dist/src/eval-workflows/batch-evaluation-workflow.d.ts.map +0 -1
- package/dist/src/eval-workflows/batch-evaluation-workflow.js.map +0 -1
- package/dist/src/eval-workflows/evaluation-pipeline.d.ts +0 -109
- package/dist/src/eval-workflows/evaluation-pipeline.d.ts.map +0 -1
- package/dist/src/eval-workflows/evaluation-pipeline.js +0 -373
- package/dist/src/eval-workflows/evaluation-pipeline.js.map +0 -1
- package/dist/src/eval-workflows/evaluation-preparation.d.ts.map +0 -1
- package/dist/src/eval-workflows/evaluation-preparation.js.map +0 -1
- package/dist/src/eval-workflows/run-evaluation.d.ts.map +0 -1
- package/dist/src/eval-workflows/run-evaluation.js +0 -528
- package/dist/src/eval-workflows/run-evaluation.js.map +0 -1
- package/dist/src/executors/anthropic-api.d.ts.map +0 -1
- package/dist/src/executors/anthropic-api.js.map +0 -1
- package/dist/src/executors/claude-cli.d.ts.map +0 -1
- package/dist/src/executors/claude-cli.js +0 -179
- package/dist/src/executors/claude-cli.js.map +0 -1
- package/dist/src/executors/claude-sdk-trace.d.ts.map +0 -1
- package/dist/src/executors/claude-sdk-trace.js.map +0 -1
- package/dist/src/executors/claude-sdk.d.ts.map +0 -1
- package/dist/src/executors/claude-sdk.js.map +0 -1
- package/dist/src/executors/codex-cli-trace.d.ts.map +0 -1
- package/dist/src/executors/codex-cli-trace.js.map +0 -1
- package/dist/src/executors/codex-cli.d.ts.map +0 -1
- package/dist/src/executors/codex-cli.js.map +0 -1
- package/dist/src/executors/codex-sdk.d.ts.map +0 -1
- package/dist/src/executors/codex-sdk.js.map +0 -1
- package/dist/src/executors/gemini.d.ts.map +0 -1
- package/dist/src/executors/gemini.js.map +0 -1
- package/dist/src/executors/index.d.ts.map +0 -1
- package/dist/src/executors/index.js.map +0 -1
- package/dist/src/executors/openai-api.d.ts.map +0 -1
- package/dist/src/executors/openai-api.js.map +0 -1
- package/dist/src/executors/runtime-fingerprint.d.ts.map +0 -1
- package/dist/src/executors/runtime-fingerprint.js.map +0 -1
- package/dist/src/executors/script.d.ts.map +0 -1
- package/dist/src/executors/script.js.map +0 -1
- package/dist/src/executors/shared.d.ts +0 -194
- package/dist/src/executors/shared.d.ts.map +0 -1
- package/dist/src/executors/shared.js +0 -256
- package/dist/src/executors/shared.js.map +0 -1
- package/dist/src/grading/assertions.d.ts.map +0 -1
- package/dist/src/grading/assertions.js.map +0 -1
- package/dist/src/grading/debias-validate.d.ts.map +0 -1
- package/dist/src/grading/debias-validate.js.map +0 -1
- package/dist/src/grading/diagnostic.d.ts.map +0 -1
- package/dist/src/grading/diagnostic.js.map +0 -1
- package/dist/src/grading/gold-cli.d.ts.map +0 -1
- package/dist/src/grading/gold-cli.js.map +0 -1
- package/dist/src/grading/gold-dataset.d.ts.map +0 -1
- package/dist/src/grading/gold-dataset.js.map +0 -1
- package/dist/src/grading/human-gold.d.ts.map +0 -1
- package/dist/src/grading/human-gold.js.map +0 -1
- package/dist/src/grading/index.d.ts.map +0 -1
- package/dist/src/grading/index.js.map +0 -1
- package/dist/src/grading/judge.d.ts.map +0 -1
- package/dist/src/grading/judge.js.map +0 -1
- package/dist/src/grading/layered-scores.d.ts.map +0 -1
- package/dist/src/grading/layered-scores.js.map +0 -1
- package/dist/src/inputs/eval-config.d.ts.map +0 -1
- package/dist/src/inputs/eval-config.js.map +0 -1
- package/dist/src/inputs/load-samples.d.ts.map +0 -1
- package/dist/src/inputs/load-samples.js.map +0 -1
- package/dist/src/inputs/mcp-resolver.d.ts.map +0 -1
- package/dist/src/inputs/mcp-resolver.js.map +0 -1
- package/dist/src/inputs/skill-loader.d.ts.map +0 -1
- package/dist/src/inputs/skill-loader.js +0 -253
- package/dist/src/inputs/skill-loader.js.map +0 -1
- package/dist/src/inputs/url-fetcher.d.ts.map +0 -1
- package/dist/src/inputs/url-fetcher.js.map +0 -1
- package/dist/src/observability/experience.d.ts +0 -260
- package/dist/src/observability/experience.d.ts.map +0 -1
- package/dist/src/observability/experience.js +0 -1233
- package/dist/src/observability/experience.js.map +0 -1
- package/dist/src/observability/inbox-view-model.d.ts +0 -27
- package/dist/src/observability/inbox-view-model.d.ts.map +0 -1
- package/dist/src/observability/inbox-view-model.js +0 -135
- package/dist/src/observability/inbox-view-model.js.map +0 -1
- package/dist/src/observability/inbox.d.ts +0 -125
- package/dist/src/observability/inbox.d.ts.map +0 -1
- package/dist/src/observability/inbox.js +0 -741
- package/dist/src/observability/inbox.js.map +0 -1
- package/dist/src/observability/problem-patterns.d.ts +0 -51
- package/dist/src/observability/problem-patterns.d.ts.map +0 -1
- package/dist/src/observability/problem-patterns.js +0 -205
- package/dist/src/observability/problem-patterns.js.map +0 -1
- package/dist/src/observability/review-state.d.ts +0 -54
- package/dist/src/observability/review-state.d.ts.map +0 -1
- package/dist/src/observability/review-state.js +0 -131
- package/dist/src/observability/review-state.js.map +0 -1
- package/dist/src/observability/skill-chain-advisories.d.ts +0 -25
- package/dist/src/observability/skill-chain-advisories.d.ts.map +0 -1
- package/dist/src/observability/skill-chain-advisories.js +0 -69
- package/dist/src/observability/skill-chain-advisories.js.map +0 -1
- package/dist/src/observability/skill-chain.d.ts +0 -61
- package/dist/src/observability/skill-chain.d.ts.map +0 -1
- package/dist/src/observability/skill-chain.js +0 -233
- package/dist/src/observability/skill-chain.js.map +0 -1
- package/dist/src/observability/skill-health-analyzer.d.ts.map +0 -1
- package/dist/src/observability/skill-health-analyzer.js.map +0 -1
- package/dist/src/observability/text-signals.d.ts +0 -7
- package/dist/src/observability/text-signals.d.ts.map +0 -1
- package/dist/src/observability/text-signals.js +0 -22
- package/dist/src/observability/text-signals.js.map +0 -1
- package/dist/src/observability/trace-adapter.d.ts.map +0 -1
- package/dist/src/observability/trace-adapter.js.map +0 -1
- package/dist/src/observability/trace-attribution.d.ts +0 -56
- package/dist/src/observability/trace-attribution.d.ts.map +0 -1
- package/dist/src/observability/trace-attribution.js +0 -200
- package/dist/src/observability/trace-attribution.js.map +0 -1
- package/dist/src/observability/trace-segmenter.d.ts +0 -52
- package/dist/src/observability/trace-segmenter.d.ts.map +0 -1
- package/dist/src/observability/trace-segmenter.js +0 -320
- package/dist/src/observability/trace-segmenter.js.map +0 -1
- package/dist/src/observability/trace-source.d.ts +0 -99
- package/dist/src/observability/trace-source.d.ts.map +0 -1
- package/dist/src/observability/trace-source.js +0 -492
- package/dist/src/observability/trace-source.js.map +0 -1
- package/dist/src/renderer/html-renderer.d.ts.map +0 -1
- package/dist/src/renderer/html-renderer.js.map +0 -1
- package/dist/src/renderer/layout.d.ts.map +0 -1
- package/dist/src/renderer/layout.js.map +0 -1
- package/dist/src/renderer/observation-inbox-renderer.d.ts +0 -4
- package/dist/src/renderer/observation-inbox-renderer.d.ts.map +0 -1
- package/dist/src/renderer/observation-inbox-renderer.js +0 -5376
- package/dist/src/renderer/observation-inbox-renderer.js.map +0 -1
- package/dist/src/renderer/skill-detail-renderer.d.ts +0 -15
- package/dist/src/renderer/skill-detail-renderer.d.ts.map +0 -1
- package/dist/src/renderer/skill-detail-renderer.js +0 -1234
- package/dist/src/renderer/skill-detail-renderer.js.map +0 -1
- package/dist/src/renderer/skill-health-renderer.d.ts.map +0 -1
- package/dist/src/renderer/skill-health-renderer.js.map +0 -1
- package/dist/src/renderer/skill-list-renderer.d.ts +0 -4
- package/dist/src/renderer/skill-list-renderer.d.ts.map +0 -1
- package/dist/src/renderer/skill-list-renderer.js +0 -327
- package/dist/src/renderer/skill-list-renderer.js.map +0 -1
- package/dist/src/renderer/summary.d.ts +0 -68
- package/dist/src/renderer/summary.d.ts.map +0 -1
- package/dist/src/renderer/summary.js +0 -1896
- package/dist/src/renderer/summary.js.map +0 -1
- package/dist/src/renderer/table.d.ts.map +0 -1
- package/dist/src/renderer/table.js.map +0 -1
- package/dist/src/renderer/test-view.d.ts.map +0 -1
- package/dist/src/renderer/test-view.js +0 -905
- package/dist/src/renderer/test-view.js.map +0 -1
- package/dist/src/renderer/trends.d.ts.map +0 -1
- package/dist/src/renderer/trends.js.map +0 -1
- package/dist/src/server/job-store.d.ts.map +0 -1
- package/dist/src/server/job-store.js.map +0 -1
- package/dist/src/server/report-server.d.ts.map +0 -1
- package/dist/src/server/report-server.js +0 -801
- package/dist/src/server/report-server.js.map +0 -1
- package/dist/src/server/report-store.d.ts.map +0 -1
- package/dist/src/server/report-store.js.map +0 -1
- package/dist/src/server/skill-index.d.ts +0 -77
- package/dist/src/server/skill-index.d.ts.map +0 -1
- package/dist/src/server/skill-index.js +0 -331
- package/dist/src/server/skill-index.js.map +0 -1
- package/dist/src/server/skill-insights.d.ts +0 -92
- package/dist/src/server/skill-insights.d.ts.map +0 -1
- package/dist/src/server/skill-insights.js +0 -676
- package/dist/src/server/skill-insights.js.map +0 -1
- package/dist/src/shared/hard-rules.d.ts +0 -40
- package/dist/src/shared/hard-rules.d.ts.map +0 -1
- package/dist/src/shared/hard-rules.js +0 -200
- package/dist/src/shared/hard-rules.js.map +0 -1
- package/dist/src/shared/time.d.ts.map +0 -1
- package/dist/src/shared/time.js.map +0 -1
- package/dist/src/shared/tool-search.d.ts.map +0 -1
- package/dist/src/shared/tool-search.js.map +0 -1
- package/dist/src/types/doctor.d.ts +0 -155
- package/dist/src/types/doctor.d.ts.map +0 -1
- package/dist/src/types/doctor.js.map +0 -1
- package/dist/src/types/eval.d.ts +0 -372
- package/dist/src/types/eval.d.ts.map +0 -1
- package/dist/src/types/eval.js.map +0 -1
- package/dist/src/types/executor.d.ts.map +0 -1
- package/dist/src/types/executor.js.map +0 -1
- package/dist/src/types/index.d.ts +0 -8
- package/dist/src/types/index.d.ts.map +0 -1
- package/dist/src/types/index.js +0 -8
- package/dist/src/types/index.js.map +0 -1
- package/dist/src/types/judge.d.ts.map +0 -1
- package/dist/src/types/judge.js.map +0 -1
- package/dist/src/types/report.d.ts +0 -551
- package/dist/src/types/report.d.ts.map +0 -1
- package/dist/src/types/report.js.map +0 -1
- package/dist/src/types/shared.d.ts.map +0 -1
- package/dist/src/types/shared.js.map +0 -1
- package/dist/src/types/storage.d.ts.map +0 -1
- package/dist/src/types/storage.js.map +0 -1
- package/dist/src/util/safe-slice.d.ts +0 -23
- package/dist/src/util/safe-slice.d.ts.map +0 -1
- package/dist/src/util/safe-slice.js +0 -33
- package/dist/src/util/safe-slice.js.map +0 -1
- /package/dist/{src/analysis → analysis}/coverage-analyzer.d.ts +0 -0
- /package/dist/{src/analysis → analysis}/coverage-analyzer.js +0 -0
- /package/dist/{src/analysis → analysis}/failure-clusterer.d.ts +0 -0
- /package/dist/{src/analysis → analysis}/failure-clusterer.js +0 -0
- /package/dist/{src/analysis → analysis}/hedging-classifier.d.ts +0 -0
- /package/dist/{src/analysis → analysis}/hedging-classifier.js +0 -0
- /package/dist/{src/analysis → analysis}/sample-diagnostics.d.ts +0 -0
- /package/dist/{src/analysis → analysis}/sample-diagnostics.js +0 -0
- /package/dist/{src/analysis → analysis}/saturation.d.ts +0 -0
- /package/dist/{src/analysis → analysis}/saturation.js +0 -0
- /package/dist/{src/cli → cli}/index.d.ts +0 -0
- /package/dist/{src/cli → cli/lib}/progress.d.ts +0 -0
- /package/dist/{src/cli → cli/lib}/progress.js +0 -0
- /package/dist/{src/cli → cli/lib}/run-tally.js +0 -0
- /package/dist/{src/doctor → doctor}/health/builtin-dimensions.d.ts +0 -0
- /package/dist/{src/doctor → doctor}/health/builtin-dimensions.js +0 -0
- /package/dist/{src/doctor → doctor}/health/composer.d.ts +0 -0
- /package/dist/{src/doctor → doctor}/health/dimension-registry.d.ts +0 -0
- /package/dist/{src/doctor → doctor}/health/dimension-registry.js +0 -0
- /package/dist/{src/doctor → doctor}/health/dimension-spec.js +0 -0
- /package/dist/{src/doctor → doctor}/health/register.d.ts +0 -0
- /package/dist/{src/doctor → doctor}/health/register.js +0 -0
- /package/dist/{src/doctor → doctor}/index.d.ts +0 -0
- /package/dist/{src/doctor → doctor}/preflight.d.ts +0 -0
- /package/dist/{src/doctor → doctor}/preflight.js +0 -0
- /package/dist/{src/doctor → doctor}/rules.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/bootstrap.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/bootstrap.js +0 -0
- /package/dist/{src/eval-core → eval-core}/cache.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/cache.js +0 -0
- /package/dist/{src/eval-core → eval-core}/comparability.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/comparability.js +0 -0
- /package/dist/{src/eval-core → eval-core}/dependency-checker.js +0 -0
- /package/dist/{src/eval-core → eval-core}/evaluation-execution.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/evaluation-execution.js +0 -0
- /package/dist/{src/eval-core → eval-core}/evaluation-job.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/evaluation-job.js +0 -0
- /package/dist/{src/eval-core → eval-core}/evaluation-reporting.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/evaluation-reporting.js +0 -0
- /package/dist/{src/eval-core → eval-core}/execution-strategy.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/fact-checker.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/fact-checker.js +0 -0
- /package/dist/{src/eval-core → eval-core}/layer-gates.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/layer-gates.js +0 -0
- /package/dist/{src/eval-core → eval-core}/schema.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/schema.js +0 -0
- /package/dist/{src/eval-core → eval-core}/statistics.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/statistics.js +0 -0
- /package/dist/{src/eval-core → eval-core}/task-planner.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/task-planner.js +0 -0
- /package/dist/{src/eval-core → eval-core}/verdict.d.ts +0 -0
- /package/dist/{src/eval-core → eval-core}/verdict.js +0 -0
- /package/dist/{src/eval-workflows → eval-workflows}/batch-evaluation-workflow.d.ts +0 -0
- /package/dist/{src/eval-workflows → eval-workflows}/batch-evaluation-workflow.js +0 -0
- /package/dist/{src/eval-workflows → eval-workflows}/evaluation-preparation.d.ts +0 -0
- /package/dist/{src/eval-workflows → eval-workflows}/evaluation-preparation.js +0 -0
- /package/dist/{src/eval-workflows → eval-workflows}/run-evaluation.d.ts +0 -0
- /package/dist/{src/executors → executors}/anthropic-api.d.ts +0 -0
- /package/dist/{src/executors → executors}/anthropic-api.js +0 -0
- /package/dist/{src/executors → executors}/claude-cli.d.ts +0 -0
- /package/dist/{src/executors → executors}/claude-sdk-trace.d.ts +0 -0
- /package/dist/{src/executors → executors}/claude-sdk-trace.js +0 -0
- /package/dist/{src/executors → executors}/claude-sdk.d.ts +0 -0
- /package/dist/{src/executors → executors}/claude-sdk.js +0 -0
- /package/dist/{src/executors → executors}/codex-cli-trace.d.ts +0 -0
- /package/dist/{src/executors → executors}/codex-cli-trace.js +0 -0
- /package/dist/{src/executors → executors}/codex-cli.d.ts +0 -0
- /package/dist/{src/executors → executors}/codex-cli.js +0 -0
- /package/dist/{src/executors → executors}/codex-sdk.d.ts +0 -0
- /package/dist/{src/executors → executors}/codex-sdk.js +0 -0
- /package/dist/{src/executors → executors}/gemini.d.ts +0 -0
- /package/dist/{src/executors → executors}/gemini.js +0 -0
- /package/dist/{src/executors → executors}/index.d.ts +0 -0
- /package/dist/{src/executors → executors}/index.js +0 -0
- /package/dist/{src/executors → executors}/openai-api.d.ts +0 -0
- /package/dist/{src/executors → executors}/openai-api.js +0 -0
- /package/dist/{src/executors → executors}/runtime-fingerprint.d.ts +0 -0
- /package/dist/{src/executors → executors}/runtime-fingerprint.js +0 -0
- /package/dist/{src/executors → executors}/script.d.ts +0 -0
- /package/dist/{src/executors → executors}/script.js +0 -0
- /package/dist/{src/grading → grading}/assertions.d.ts +0 -0
- /package/dist/{src/grading → grading}/assertions.js +0 -0
- /package/dist/{src/grading → grading}/debias-validate.d.ts +0 -0
- /package/dist/{src/grading → grading}/debias-validate.js +0 -0
- /package/dist/{src/grading → grading}/diagnostic.d.ts +0 -0
- /package/dist/{src/grading → grading}/diagnostic.js +0 -0
- /package/dist/{src/grading → grading}/gold-cli.d.ts +0 -0
- /package/dist/{src/grading → grading}/gold-cli.js +0 -0
- /package/dist/{src/grading → grading}/gold-dataset.d.ts +0 -0
- /package/dist/{src/grading → grading}/gold-dataset.js +0 -0
- /package/dist/{src/grading → grading}/human-gold.d.ts +0 -0
- /package/dist/{src/grading → grading}/human-gold.js +0 -0
- /package/dist/{src/grading → grading}/index.d.ts +0 -0
- /package/dist/{src/grading → grading}/index.js +0 -0
- /package/dist/{src/grading → grading}/judge.d.ts +0 -0
- /package/dist/{src/grading → grading}/judge.js +0 -0
- /package/dist/{src/grading → grading}/layered-scores.d.ts +0 -0
- /package/dist/{src/grading → grading}/layered-scores.js +0 -0
- /package/dist/{src/inputs → inputs}/eval-config.d.ts +0 -0
- /package/dist/{src/inputs → inputs}/eval-config.js +0 -0
- /package/dist/{src/inputs → inputs}/load-samples.d.ts +0 -0
- /package/dist/{src/inputs → inputs}/load-samples.js +0 -0
- /package/dist/{src/inputs → inputs}/mcp-resolver.d.ts +0 -0
- /package/dist/{src/inputs → inputs}/mcp-resolver.js +0 -0
- /package/dist/{src/inputs → inputs}/skill-loader.d.ts +0 -0
- /package/dist/{src/inputs → inputs}/url-fetcher.d.ts +0 -0
- /package/dist/{src/inputs → inputs}/url-fetcher.js +0 -0
- /package/dist/{src/observability → observability}/skill-health-analyzer.d.ts +0 -0
- /package/dist/{src/observability → observability}/skill-health-analyzer.js +0 -0
- /package/dist/{src/observability → observability}/trace-adapter.d.ts +0 -0
- /package/dist/{src/observability → observability}/trace-adapter.js +0 -0
- /package/dist/{src/renderer → renderer}/html-renderer.d.ts +0 -0
- /package/dist/{src/renderer → renderer}/html-renderer.js +0 -0
- /package/dist/{src/renderer → renderer}/layout.d.ts +0 -0
- /package/dist/{src/renderer → renderer}/layout.js +0 -0
- /package/dist/{src/renderer → renderer}/skill-health-renderer.d.ts +0 -0
- /package/dist/{src/renderer → renderer}/skill-health-renderer.js +0 -0
- /package/dist/{src/renderer → renderer}/table.d.ts +0 -0
- /package/dist/{src/renderer → renderer}/table.js +0 -0
- /package/dist/{src/renderer → renderer}/test-view.d.ts +0 -0
- /package/dist/{src/renderer → renderer}/trends.d.ts +0 -0
- /package/dist/{src/renderer → renderer}/trends.js +0 -0
- /package/dist/{src/server → server}/job-store.d.ts +0 -0
- /package/dist/{src/server → server}/job-store.js +0 -0
- /package/dist/{src/server → server}/report-server.d.ts +0 -0
- /package/dist/{src/server → server}/report-store.d.ts +0 -0
- /package/dist/{src/server → server}/report-store.js +0 -0
- /package/dist/{src/shared → shared}/time.d.ts +0 -0
- /package/dist/{src/shared → shared}/time.js +0 -0
- /package/dist/{src/shared → shared}/tool-search.d.ts +0 -0
- /package/dist/{src/shared → shared}/tool-search.js +0 -0
- /package/dist/{src/types → types}/doctor.js +0 -0
- /package/dist/{src/types → types}/eval.js +0 -0
- /package/dist/{src/types → types}/executor.d.ts +0 -0
- /package/dist/{src/types → types}/executor.js +0 -0
- /package/dist/{src/types → types}/judge.d.ts +0 -0
- /package/dist/{src/types → types}/judge.js +0 -0
- /package/dist/{src/types → types}/report.js +0 -0
- /package/dist/{src/types → types}/shared.d.ts +0 -0
- /package/dist/{src/types → types}/shared.js +0 -0
- /package/dist/{src/types → types}/storage.d.ts +0 -0
- /package/dist/{src/types → types}/storage.js +0 -0
package/README.zh.md
CHANGED
|
@@ -8,26 +8,24 @@
|
|
|
8
8
|
|
|
9
9
|
[English](./README.md) | **简体中文**
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
|
|
11
|
+
**你改完 prompt,真的变好了吗?**
|
|
12
|
+
用统计严谨性 A/B 测试你的 prompt 和 skill —— Bootstrap 置信区间、Krippendorff α、长度去偏,全部默认开。
|
|
13
13
|
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
<a id="statistical-rigor"></a>
|
|
17
|
-
> 默认带:Bootstrap 置信区间 · Krippendorff α(评委 ↔ 人工)· 长度去偏 · 饱和曲线 · 用例隔离(construct validity)。[这些为什么重要 →](docs/zh/statistical-rigor.md)
|
|
18
|
-
|
|
19
|
-

|
|
14
|
+

|
|
20
15
|
|
|
21
16
|
## 快速开始
|
|
22
17
|
|
|
23
18
|
```bash
|
|
24
|
-
npm i oh-my-knowledge
|
|
25
|
-
omk init
|
|
26
|
-
|
|
27
|
-
omk eval --control code-review-v1 --treatment code-review-v2 # → 5 分钟出 HTML 报告 + verdict
|
|
19
|
+
npm i -g oh-my-knowledge
|
|
20
|
+
omk init demo && cd demo
|
|
21
|
+
omk eval --control code-review-v1 --treatment code-review-v2
|
|
28
22
|
```
|
|
29
23
|
|
|
30
|
-
|
|
24
|
+
不用改任何文件 —— `omk init` 帮你脚手架两版 skill 和三条评测用例;`omk eval` 跑控制变量 A/B,5 分钟内出 HTML 报告 + 一行 verdict。
|
|
25
|
+
|
|
26
|
+
手把手教程:[5 分钟快速上手](docs/zh/quickstart-skill-eval.md)(推荐第一次跑评测的用户)。
|
|
27
|
+
|
|
28
|
+
深入:[CLI 参考](docs/zh/reference/cli.md) · [工作原理](docs/zh/explanation/architecture.md) · [评测样本格式](docs/zh/reference/eval-sample-format.md) · [执行器与 artifact 布局](docs/zh/reference/executors.md)
|
|
31
29
|
|
|
32
30
|
## 在 AI Coding Agent 中使用
|
|
33
31
|
|
|
@@ -41,7 +39,7 @@ omk eval --control code-review-v1 --treatment code-review-v2 # → 5 分钟
|
|
|
41
39
|
/omk sample # 生成或补齐评测用例
|
|
42
40
|
```
|
|
43
41
|
|
|
44
|
-
这些 slash command 是自然语言入口 —— agent 会从对话上下文里推断要操作哪个 skill
|
|
42
|
+
这些 slash command 是自然语言入口 —— agent 会从对话上下文里推断要操作哪个 skill。也可以直接说「帮我评测 v1 和 v2 的差异」、「改进一下这个 artifact」,omk 会自动理解意图并调用对应命令。
|
|
45
43
|
|
|
46
44
|
### 在 Codex 中使用
|
|
47
45
|
|
|
@@ -53,21 +51,11 @@ omk evolve skills/my-skill.md
|
|
|
53
51
|
omk sample skills/my-skill.md
|
|
54
52
|
```
|
|
55
53
|
|
|
56
|
-
|
|
54
|
+
也可以直接用自然语言描述目标,例如「比较 v1 和 v2 的评测差异」、「为这个 skill 生成评测用例」。
|
|
57
55
|
|
|
58
56
|
## 为什么需要这个工具
|
|
59
57
|
|
|
60
|
-
做知识工程的团队会产出大量知识载体(当前常见是 skill,也包括 prompt、agent、workflow
|
|
61
|
-
|
|
62
|
-
## 核心能力
|
|
63
|
-
|
|
64
|
-
- **LLM 健康度审计** — `omk doctor` 用单次 LLM 会话产出多维度健康度报告,7 个内置维度(触发与边界 / 文档清晰 / 指令精确性 / 依赖检查 / 工具规范 / 安全与合规 / 示例完备)独立给「健康 / 亚健康 / 不健康 / 不适用」+ findings + 改进建议;维度可扩展,`--html` 产可视化报告。无 LLM 环境(CI 节点 / 断网调试)可加 `--static-only` 跑纯静态检查(可读性 / 元数据 / 依赖 / samples 契约)。`omk eval` 内部仍跑静态可读性、元数据、依赖 gate 把关评测质量(角色分离:doctor=审计,eval=评测)
|
|
65
|
-
- **控制变量离线评测** — 固定模型和用例,只变知识载体;兼容 Claude Code skill、CLAUDE.md prompt、RAG 知识库等任何 markdown 形式的指令
|
|
66
|
-
- **六维独立打分** — Fact / Behavior / LLM-judge / Cost / Efficiency / Stability 分别出信号,单一维度的回退不会被其他维度的收益掩盖
|
|
67
|
-
- **线上 session 观测** — 解析 Claude Code session JSONL,在真实用户会话上测量各 skill 的失败率、耗时、token 成本和知识缺口信号
|
|
68
|
-
- **知识缺口识别** — 严重度加权的信号(显式标记 / 搜索失败 / hedging 用语 / 反复失败)量化风险敞口,不宣称完备性
|
|
69
|
-
- **合并前 CI 门** — `omk eval` 强制三层 all-pass(fact + behavior + llm-judge),抓复合分掩盖的单层回退
|
|
70
|
-
- **一行 ship/no-ship 结论** — `omk eval` 聚合 bootstrap CI / 三层 ci-gate / saturation / human α,给六档 verdict(PROGRESS / CAUTIOUS / REGRESS / NOISE / UNDERPOWERED / SOLO)+ 行动建议;exit code 反映是否可 ship
|
|
58
|
+
做知识工程的团队会产出大量知识载体(当前常见是 skill,也包括 prompt、agent、workflow 等)。当被问到「v2 比 v1 好在哪」时,需要客观数据而非主观判断。`oh-my-knowledge` 通过控制变量实验解决这个问题:**相同模型、相同评测用例,只改变知识载体。**
|
|
71
59
|
|
|
72
60
|
## 为什么选 omk
|
|
73
61
|
|
|
@@ -82,9 +70,9 @@ omk sample skills/my-skill.md
|
|
|
82
70
|
| 原生 Claude Code skill | ✓ | ✗ | ✗ | ✗ |
|
|
83
71
|
| 托管 SaaS 看板 | ✗ | ✗ | ✓ | ✓ |
|
|
84
72
|
|
|
85
|
-
omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 评委 ↔ 人工 α / 长度去偏不是 advanced flag,是默认行为。其他工具让你**手动**接置信区间;omk 让你**默认无法忽略**它。需要 SaaS 看板?选 LangSmith。要快速 prompt 迭代不要统计层?选 promptfoo
|
|
73
|
+
omk 的护城河是 **default-on 安全网** —— Bootstrap CI / 评委 ↔ 人工 α / 长度去偏不是 advanced flag,是默认行为。其他工具让你**手动**接置信区间;omk 让你**默认无法忽略**它。需要 SaaS 看板?选 LangSmith。要快速 prompt 迭代不要统计层?选 promptfoo。**要发到生产且会被问「为什么应该相信这个数字」?选 omk。**
|
|
86
74
|
|
|
87
|
-
RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比(7 个工具 × 25+ 维度): [docs/zh/comparison.md](docs/zh/comparison.md)
|
|
75
|
+
RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比(7 个工具 × 25+ 维度): [docs/zh/reference/comparison.md](docs/zh/reference/comparison.md)
|
|
88
76
|
|
|
89
77
|
## 特性
|
|
90
78
|
|
|
@@ -94,687 +82,60 @@ RAG 专项评测请看 RAGAS(独立 niche,跟 omk 互补)。完整对比
|
|
|
94
82
|
| **六维评估** | 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性独立展示 |
|
|
95
83
|
| **多执行器** | 支持 Claude CLI / Claude SDK / Codex CLI / Codex SDK / OpenAI / Gemini 及自定义命令 |
|
|
96
84
|
| **21+ 种断言** | 包含子串、正则、JSON Schema、ROUGE/BLEU/Levenshtein 相似度、Agent 工具调用、语义相似度、自定义函数等 |
|
|
97
|
-
| **统计严谨性** | Bootstrap CI / Krippendorff α / 长度去偏 / 饱和曲线 —— 全部默认开。[详情 →](docs/zh/statistical-rigor.md) |
|
|
98
|
-
| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context
|
|
99
|
-
|
|
|
100
|
-
|
|
|
101
|
-
|
|
|
85
|
+
| **统计严谨性** | Bootstrap CI / Krippendorff α / 长度去偏 / 饱和曲线 —— 全部默认开。[详情 →](docs/zh/explanation/statistical-rigor.md) |
|
|
86
|
+
| **RAG metrics** | `faithfulness` / `answer_relevancy` / `context_recall` 三 metric — 反幻觉 + 切题度 + context 覆盖 |
|
|
87
|
+
| **LLM 健康度审计** | `omk doctor` 给 7 个内置维度独立打分;`--static-only` 可离线无 LLM 调用 |
|
|
88
|
+
| **线上 session 观测** | 解析 Claude Code session JSONL,测量各 skill 的失败率、耗时、token 成本、知识缺口信号 |
|
|
89
|
+
| **知识缺口识别** | 严重度加权的信号量化风险敞口,不宣称完备性 |
|
|
90
|
+
| **用例隔离 (construct validity)** | `--strict-baseline`(默认开)三堵 baseline 拿到被测 skill 的污染路径 |
|
|
91
|
+
| **用例设计科学性** | Sample schema 加 `capability` / `difficulty` / `construct` / `provenance` 元数据字段(HF Dataset Cards 风),studio 输出 coverage 分桶 + `rubric_clarity_low` / `capability_thin` issue。[docs/specs/sample-design-spec.md](docs/specs/sample-design-spec.md) |
|
|
102
92
|
| **多评委 ensemble** | `--judge-models claude:opus,openai:gpt-4o` 跨厂商评分 + agreement 度量 |
|
|
103
|
-
| **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容(SSO 保护的知识库等) |
|
|
104
93
|
| **盲测 A/B** | `--blind` 隐藏变体名称,HTML 报告有揭晓按钮 |
|
|
105
94
|
| **多轮方差分析** | `--repeat N` 重复 N 次,计算均值/标准差/置信区间/t 检验 |
|
|
106
|
-
|
|
|
107
|
-
| **断言取反 + 组合** | 通用 `not: true` 字段 + `assert-set` (any/all) 任意嵌套 |
|
|
95
|
+
| **MCP URL 获取** | 通过 MCP Server 获取私有文档 URL 内容(SSO 保护的知识库等) |
|
|
108
96
|
| **自动分析** | 检测低区分度断言、均匀分数、全通过/全失败、高成本用例 |
|
|
109
97
|
| **可追溯性** | 报告含 CLI 版本、Node 版本、artifact 版本指纹、judge prompt hash |
|
|
110
98
|
| **中英切换** | HTML 报告右上角一键切换语言 |
|
|
111
99
|
|
|
112
|
-
##
|
|
113
|
-
|
|
114
|
-
核心思路:**固定模型 + 固定样本,只变 artifact 和 runtime context**,通过交错调度消除时间漂移,用断言 + LLM 评委双通道评分,再叠加知识缺口信号量化风险敞口。
|
|
115
|
-
|
|
116
|
-
```mermaid
|
|
117
|
-
flowchart TD
|
|
118
|
-
subgraph Input["① 输入"]
|
|
119
|
-
S["eval-samples<br/>(JSON / YAML)"]
|
|
120
|
-
A["artifacts<br/>skills/*.md · SKILL.md<br/>baseline · git:name · @cwd"]
|
|
121
|
-
end
|
|
122
|
-
|
|
123
|
-
subgraph Prep["② 预处理(解析与抓取)"]
|
|
124
|
-
V["变体解析<br/>variant → artifact + runtime context<br/>(cwd / 项目级 CLAUDE.md / 本地 skills)"]
|
|
125
|
-
U["URL 抓取<br/>prompt / context 中的 URL<br/>MCP Server(私有文档) → HTTP"]
|
|
126
|
-
end
|
|
127
|
-
|
|
128
|
-
subgraph Schedule["③ 交错调度 + 并发"]
|
|
129
|
-
Q["s1-v1 → s1-v2 → s2-v1 → s2-v2 …<br/>--concurrency N · --repeat N"]
|
|
130
|
-
end
|
|
131
|
-
|
|
132
|
-
subgraph Exec["④ 执行器(固定模型)"]
|
|
133
|
-
E["claude / claude-sdk / codex / openai / gemini<br/>anthropic-api / openai-api / 自定义命令"]
|
|
134
|
-
T["claude-sdk 抽取<br/>turns / toolCalls trace"]
|
|
135
|
-
E -.-> T
|
|
136
|
-
end
|
|
137
|
-
|
|
138
|
-
subgraph Score["⑤ 双通道评分"]
|
|
139
|
-
AS["断言(18 种)<br/>内容 / 结构 / 成本 / 延迟<br/>agent: tools_called · turns_min …"]
|
|
140
|
-
LS["LLM 评委<br/>rubric · dimensions(多维独立打分)"]
|
|
141
|
-
CS["综合分数<br/>断言 & LLM 有则均值"]
|
|
142
|
-
AS --> CS
|
|
143
|
-
LS --> CS
|
|
144
|
-
end
|
|
145
|
-
|
|
146
|
-
subgraph Analyze["⑥ 自动分析 + 知识缺口"]
|
|
147
|
-
D["低区分度断言 / 均匀分 / 全通过全失败<br/>高成本样本 · 方差 · t 检验"]
|
|
148
|
-
G["知识缺口信号<br/>(风险敞口量化, 不证明完备)"]
|
|
149
|
-
end
|
|
150
|
-
|
|
151
|
-
subgraph Report["⑦ 报告"]
|
|
152
|
-
R["六维: 事实 / 行为 / LLM 评价 / 成本 / 效率 / 稳定性<br/>JSON + HTML · 顶部 verdict pill · 盲测揭晓<br/>CLI/Node/版本指纹可追溯"]
|
|
153
|
-
end
|
|
154
|
-
|
|
155
|
-
S --> U
|
|
156
|
-
A --> V
|
|
157
|
-
V --> Q
|
|
158
|
-
U --> Q
|
|
159
|
-
Q --> E
|
|
160
|
-
T --> AS
|
|
161
|
-
E --> AS
|
|
162
|
-
E --> LS
|
|
163
|
-
CS --> D
|
|
164
|
-
CS --> G
|
|
165
|
-
D --> R
|
|
166
|
-
G --> R
|
|
167
|
-
```
|
|
168
|
-
|
|
169
|
-
**关键设计:**
|
|
170
|
-
|
|
171
|
-
- **交错调度**消除时间漂移:同一样本的不同 variant 交替发出,而非 v1 全跑完再跑 v2,避免模型负载/网络波动被错误归因给 artifact。
|
|
172
|
-
- **variant = artifact + runtime context**:`name@cwd` 让对照组可以显式声明"项目目录"这个隐性输入,把"项目级沉淀"和"显式 artifact 注入"拆开测。
|
|
173
|
-
- **双通道评分互补**:断言抓确定性缺陷(必须调用某工具/必须包含某字段),LLM 评委抓主观质量(可读性/完整性),两者都存在时取均值。
|
|
174
|
-
- **知识缺口信号**不是评分的一部分,而是一个独立追踪项:它告诉你"这次评测覆盖了多少风险敞口",用于追踪收敛,而非断言知识"完备"。
|
|
175
|
-
|
|
176
|
-
## 评测样本格式
|
|
177
|
-
|
|
178
|
-
支持 JSON 和 YAML(`eval-samples.json`、`eval-samples.yaml`、`eval-samples.yml`)。
|
|
179
|
-
|
|
180
|
-
```json
|
|
181
|
-
[
|
|
182
|
-
{
|
|
183
|
-
"sample_id": "s001",
|
|
184
|
-
"prompt": "审查这段代码的安全性",
|
|
185
|
-
"context": "function auth(u, p) { db.query('SELECT * FROM users WHERE name=' + u); }",
|
|
186
|
-
"rubric": "应识别 SQL 注入风险并建议参数化查询",
|
|
187
|
-
"assertions": [
|
|
188
|
-
{ "type": "contains", "value": "SQL 注入", "weight": 1 },
|
|
189
|
-
{ "type": "contains", "value": "参数化", "weight": 1 },
|
|
190
|
-
{ "type": "not_contains", "value": "没有问题", "weight": 0.5 }
|
|
191
|
-
],
|
|
192
|
-
"dimensions": {
|
|
193
|
-
"security": "是否识别出注入漏洞",
|
|
194
|
-
"actionability": "是否给出可直接使用的修复代码"
|
|
195
|
-
}
|
|
196
|
-
}
|
|
197
|
-
]
|
|
198
|
-
```
|
|
199
|
-
|
|
200
|
-
### 字段说明
|
|
201
|
-
|
|
202
|
-
| 字段 | 类型 | 必填 | 说明 |
|
|
203
|
-
|------|------|------|------|
|
|
204
|
-
| `sample_id` | `string` | **是** | 样本唯一标识 |
|
|
205
|
-
| `prompt` | `string` | **是** | 发送给模型的用户提示词 |
|
|
206
|
-
| `context` | `string` | 否 | 附加上下文(代码片段等),会被包裹在代码块中拼接到 prompt 后。也支持 URL,运行时自动抓取内容 |
|
|
207
|
-
| `rubric` | `string` | 否 | LLM 评委的评分标准(1-5 分) |
|
|
208
|
-
| `assertions` | `array` | 否 | 断言检查列表,详见[断言类型](#断言类型) |
|
|
209
|
-
| `assertions[].type` | `string` | **是** | 断言类型 |
|
|
210
|
-
| `assertions[].value` | `string\|number` | 视类型 | 检查值(`contains`、`min_length`、`cost_max` 等必填) |
|
|
211
|
-
| `assertions[].values` | `array` | 视类型 | 字符串数组(`contains_all`、`contains_any` 必填) |
|
|
212
|
-
| `assertions[].pattern` | `string` | 视类型 | 正则表达式(`regex` 必填) |
|
|
213
|
-
| `assertions[].flags` | `string` | 否 | 正则标志(默认 `"i"`) |
|
|
214
|
-
| `assertions[].schema` | `object` | 视类型 | JSON Schema 对象(`json_schema` 必填,基于 [ajv](https://ajv.js.org/)) |
|
|
215
|
-
| `assertions[].reference` | `string` | 视类型 | 参考文本(`semantic_similarity` 必填) |
|
|
216
|
-
| `assertions[].threshold` | `number` | 否 | 语义相似度通过阈值(默认 3) |
|
|
217
|
-
| `assertions[].fn` | `string` | 视类型 | 自定义断言 JS 文件路径(`custom` 必填) |
|
|
218
|
-
| `assertions[].weight` | `number` | 否 | 权重(默认 1) |
|
|
219
|
-
| `dimensions` | `object` | 否 | 多维度评分,key 为维度名,value 为评分标准文本 |
|
|
220
|
-
|
|
221
|
-
### URL 自动抓取
|
|
222
|
-
|
|
223
|
-
`prompt` 和 `context` 中的 URL 会在评测前自动抓取内容并内联到文本中。适用于引用在线文档、API 文档等场景:
|
|
224
|
-
|
|
225
|
-
```json
|
|
226
|
-
{
|
|
227
|
-
"sample_id": "s001",
|
|
228
|
-
"prompt": "请根据以下 PRD 文档生成评测用例:https://wiki.example.com/prd/feature-x"
|
|
229
|
-
}
|
|
230
|
-
```
|
|
231
|
-
|
|
232
|
-
运行时,URL 会被替换为实际文档内容。获取顺序:先通过 MCP Server 获取匹配的 URL(如 SSO 保护的私有文档),再通过 HTTP 获取剩余 URL。MCP 已成功的 URL 不会重复 HTTP 抓取。
|
|
233
|
-
|
|
234
|
-
**私有文档 URL**:在项目目录放一个 `.mcp.json` 配置文件,或通过 `--mcp-config` 指定路径:
|
|
235
|
-
|
|
236
|
-
```json
|
|
237
|
-
{
|
|
238
|
-
"mcpServers": {
|
|
239
|
-
"docs": {
|
|
240
|
-
"command": "npx",
|
|
241
|
-
"args": ["@example/docs-mcp-server"],
|
|
242
|
-
"env": { "DOCS_API_TOKEN": "xxx" },
|
|
243
|
-
"urlPatterns": ["docs.example.com"],
|
|
244
|
-
"fetchTool": {
|
|
245
|
-
"name": "fetch_doc",
|
|
246
|
-
"urlTransform": {
|
|
247
|
-
"regex": "docs\\.example\\.com/([^/]+/[^/]+)/([^/?#]+)",
|
|
248
|
-
"params": { "namespace": "$1", "slug": "$2" }
|
|
249
|
-
},
|
|
250
|
-
"contentExtract": "data.body"
|
|
251
|
-
}
|
|
252
|
-
}
|
|
253
|
-
}
|
|
254
|
-
}
|
|
255
|
-
```
|
|
256
|
-
|
|
257
|
-
**公网 URL**:直接 HTTP 获取,如果需要认证请确保命令行环境已配置好网络访问(VPN、代理等)。
|
|
258
|
-
|
|
259
|
-
### 评分策略
|
|
260
|
-
|
|
261
|
-
#### 1. 断言评分
|
|
262
|
-
|
|
263
|
-
基于规则的本地检查,每个断言产生通过/失败结果。
|
|
264
|
-
|
|
265
|
-
**计算方式:**
|
|
266
|
-
|
|
267
|
-
- 通过率 = 通过断言的权重之和 / 总权重(0~1)
|
|
268
|
-
- 分数 = 1 + 通过率 × 4(映射到 1~5 分)
|
|
269
|
-
- 示例:3 个断言(权重各 1),2 个通过 → 通过率 = 2/3 → 分数 = 1 + 0.67 × 4 = **3.67**
|
|
270
|
-
|
|
271
|
-
#### 2. Rubric / Dimensions 评分
|
|
272
|
-
|
|
273
|
-
评委模型(默认 `haiku`)按标准打 1-5 分。`dimensions` 模式下各维度独立评分后取平均。
|
|
274
|
-
|
|
275
|
-
#### 3. 综合分数
|
|
276
|
-
|
|
277
|
-
| 条件 | 公式 |
|
|
278
|
-
|------|------|
|
|
279
|
-
| 仅断言 | `assertionScore` |
|
|
280
|
-
| 仅 LLM | `llmScore` |
|
|
281
|
-
| 两者都有 | `(assertionScore + llmScore) / 2` |
|
|
282
|
-
| 都没有 | `0` |
|
|
283
|
-
|
|
284
|
-
### 断言类型
|
|
285
|
-
|
|
286
|
-
**确定性断言(21+ 种):**
|
|
287
|
-
|
|
288
|
-
| 类型 | 说明 |
|
|
289
|
-
|------|------|
|
|
290
|
-
| `contains` / `not_contains` | 包含/不包含子串 |
|
|
291
|
-
| `regex` | 正则匹配 |
|
|
292
|
-
| `min_length` / `max_length` | 长度范围 |
|
|
293
|
-
| `json_valid` / `json_schema` | JSON 校验 |
|
|
294
|
-
| `starts_with` / `ends_with` | 前缀/后缀匹配 |
|
|
295
|
-
| `equals` / `not_equals` | 精确匹配 |
|
|
296
|
-
| `word_count_min` / `word_count_max` | 词数范围 |
|
|
297
|
-
| `contains_all` / `contains_any` | 多值匹配 |
|
|
298
|
-
| `cost_max` / `latency_max` | 成本/延迟限制 |
|
|
299
|
-
| `tools_called` / `tools_not_called` / `tools_count_min` / `tools_count_max` | Agent 工具调用断言 |
|
|
300
|
-
| `tool_output_contains` / `tool_input_contains` | 工具输入/输出内容匹配 |
|
|
301
|
-
| `turns_min` / `turns_max` | 多轮对话轮数限制 |
|
|
302
|
-
| `rouge_n_min` | ROUGE-N recall ≥ threshold(`reference` 字段填参考答案,`n` 默认 1,`threshold` 默认 0.5) |
|
|
303
|
-
| `levenshtein_max` | 编辑距离 ≤ value(用于"输出跟参考几乎一致"场景) |
|
|
304
|
-
| `bleu_min` | BLEU-4 ≥ threshold(unsmoothed,短文本会塌陷到 0) |
|
|
305
|
-
| `faithfulness` | 输出是否被 `sample.context` 支持(反幻觉);LLM judge 1-5 评分,threshold 默认 3 |
|
|
306
|
-
| `answer_relevancy` | 输出是否切题回答 `sample.prompt`;能抓住跑题、回避、冗余;threshold 默认 3 |
|
|
307
|
-
| `context_recall` | `sample.context` 关键事实在输出中的覆盖率;`reference` 可显式指定 gold facts;threshold 默认 3 |
|
|
308
|
-
| `semantic_similarity` | LLM 语义相似度(与 reference 的整体相似度,与 RAG 三 metric 互补) |
|
|
309
|
-
| `custom` | 自定义 JS 函数(30s 超时) |
|
|
310
|
-
|
|
311
|
-
**通用修饰:**
|
|
312
|
-
|
|
313
|
-
任何断言加 `not: true` 即反向(替代 `not_contains` / `not_equals` 等成对类型;老类型保留作 alias):
|
|
314
|
-
|
|
315
|
-
```yaml
|
|
316
|
-
- type: regex
|
|
317
|
-
pattern: "TODO|FIXME"
|
|
318
|
-
not: true # 必须不含 TODO/FIXME
|
|
319
|
-
```
|
|
320
|
-
|
|
321
|
-
**断言组合(— assert-set):**
|
|
322
|
-
|
|
323
|
-
`assert-set` 类型让多个断言以 `any`(OR)或 `all`(AND)逻辑组合,可嵌套:
|
|
324
|
-
|
|
325
|
-
```yaml
|
|
326
|
-
- type: assert-set
|
|
327
|
-
mode: any # 任一通过即过 (mode: 'all' 则需全部通过)
|
|
328
|
-
children:
|
|
329
|
-
- { type: contains, value: "参数化" }
|
|
330
|
-
- { type: contains, value: "prepared statement" }
|
|
331
|
-
- { type: regex, pattern: "bind\\(.*\\?" }
|
|
332
|
-
```
|
|
333
|
-
|
|
334
|
-
子断言可独立带 `not: true`;嵌套 assert-set 可表达任意布尔逻辑。
|
|
335
|
-
|
|
336
|
-
### 自定义断言
|
|
337
|
-
|
|
338
|
-
```js
|
|
339
|
-
// my-assertion.mjs
|
|
340
|
-
export default function(output, { sample, assertion }) {
|
|
341
|
-
return { pass: output.includes('SQL'), message: '检查了 SQL 关键字' };
|
|
342
|
-
}
|
|
343
|
-
```
|
|
344
|
-
|
|
345
|
-
## 六维评估指标
|
|
346
|
-
|
|
347
|
-
评测报告从六个维度独立展示结果。其中评分三层(事实 / 行为 / LLM 评价)分开展示,让你看到**是哪一层拉胯**,而不是只看到一个合成分:
|
|
348
|
-
|
|
349
|
-
| 维度 | 指标 | 说明 |
|
|
350
|
-
|------|------|------|
|
|
351
|
-
| 📋 **事实** | 事实类断言通过率 | `contains` / `json_schema` / `fact_check` 等规则可验证断言的 1-5 分映射 |
|
|
352
|
-
| 🛠️ **行为** | 行为类断言通过率 | `tools_called` / `tool_output_contains` / `turns_max` 等执行合规类断言 |
|
|
353
|
-
| 💬 **LLM 评价** | rubric 评分 | 由评委模型按预先写好的评分规则(rubric)打的 1-5 分,主观但能抓规则断言之外的"整体好不好" |
|
|
354
|
-
| 💰 **成本** | 总成本、输入/输出 Token 数 | 基于 Token 消耗和模型定价的 API 费用 |
|
|
355
|
-
| ⚡ **效率** | 平均延迟 (ms) | 从发送请求到收到完整响应的端到端耗时 |
|
|
356
|
-
| 🛡️ **稳定性** | CV(变异系数) | 跨重复运行(`--repeat ≥ 2`)分数一致性;单轮评测显示 `—`,**诚实交代测不到什么** |
|
|
357
|
-
|
|
358
|
-
## CLI 参考
|
|
359
|
-
|
|
360
|
-
omk 的公开 CLI 由 7 个顶层命令构成完整闭环:`init`(脚手架)·`doctor`(静态检查)·`eval`(离线 A/B 评测)·`observe`(线上 trace 观测)·`evolve`(多轮自动迭代 skill)·`sample`(生成或补齐评测用例)·`studio`(本地 Web 工作台,看报告 / 分析)。
|
|
361
|
-
|
|
362
|
-
### `omk init`
|
|
363
|
-
|
|
364
|
-
```bash
|
|
365
|
-
omk init [目录]
|
|
366
|
-
```
|
|
367
|
-
|
|
368
|
-
生成一个评测项目脚手架,包含两版 starter skill 和 `eval-samples.json`。
|
|
369
|
-
|
|
370
|
-
### `omk doctor`
|
|
371
|
-
|
|
372
|
-
```bash
|
|
373
|
-
omk doctor # 体检当前目录或 ./skills
|
|
374
|
-
omk doctor skills/v1.md # 体检单个 skill
|
|
375
|
-
omk doctor skills/ --html report.html # 产 HTML 可视化报告
|
|
376
|
-
omk doctor skills/ --json > r.json # JSON 给 CI / 外部工具消费
|
|
377
|
-
omk doctor --gate; echo $? # 静默门禁,fatal 问题 exit 1,警告不阻断
|
|
378
|
-
omk doctor --static-only # 离线模式:只跑静态检查,不调 LLM
|
|
379
|
-
```
|
|
380
|
-
|
|
381
|
-
LLM 健康度审计:单次 LLM 会话产出 7 个内置维度的健康度评分 + findings + 改进建议;HTML 报告按 fail→warn→pass→skipped 排序,错误 finding 优先。维度可扩展(在自己代码里调 `registerHealthDimension`,自动并入同一次 LLM 调用的 prompt 与报告,顺序 = 注册顺序)。
|
|
382
|
-
|
|
383
|
-
离线静态模式(`--static-only`):CI 节点没装 claude / codex、本地断网调试等场景下跑 4 条静态 rule(可读性 / 元数据 / 依赖 / samples 契约),零 LLM 调用、零成本。结果同样进 `DoctorReport`,可与 `--json` / `--gate` / `--html` 组合。
|
|
384
|
-
|
|
385
|
-
`omk eval` 内部继续跑静态 readability / metadata / dependency / samples 契约 gate 保护评测质量,这条路径与用户入口的 `omk doctor` 角色分离,互不干扰。
|
|
386
|
-
|
|
387
|
-
### `omk eval`
|
|
388
|
-
|
|
389
|
-
```bash
|
|
390
|
-
omk eval --control baseline --treatment my-skill # 单 skill 必要性测试(baseline 是保留 variant,代表「不注入 skill」)
|
|
391
|
-
omk eval --control code-review-v1 --treatment code-review-v2 # 多版本 A/B
|
|
392
|
-
omk eval --config eval.yaml
|
|
393
|
-
omk eval --batch
|
|
394
|
-
omk eval gold compare <report-id> --gold-dir gold-dataset
|
|
395
|
-
```
|
|
396
|
-
|
|
397
|
-
运行离线评测,应用 verdict gate,持久化报告,并用 exit code 表示 ship/no-ship。这个工作流默认开启 bootstrap CI。
|
|
398
|
-
|
|
399
|
-
常用选项:
|
|
400
|
-
|
|
401
|
-
```text
|
|
402
|
-
--samples <路径> 样本文件(默认:eval-samples.json,也自动检测 .yaml/.yml;--skill-dir 下的 <skill>/.omk/samples.json 会被自动发现)
|
|
403
|
-
--skill-dir <路径> artifact 目录(默认:skills)
|
|
404
|
-
--control <expr> 对照组 variant 表达式
|
|
405
|
-
--treatment <v1,v2> 实验组 variants,逗号分隔
|
|
406
|
-
--config <路径> YAML/JSON 评测配置
|
|
407
|
-
--model <名称> 任务执行模型(默认:opus alias)
|
|
408
|
-
--effort <level> 执行模型扩展思考预算 low/medium/high/xhigh/max(默认:low;高 effort 跨报告不可严格比较)
|
|
409
|
-
--judge-models <list> 评委配置,例如 claude:haiku 或 claude:opus,openai:gpt-4o
|
|
410
|
-
--executor <名称> claude / claude-sdk / codex / codex-sdk / openai-api / gemini / custom
|
|
411
|
-
--no-judge 跳过 LLM 评委主观评分
|
|
412
|
-
--no-diagnostic 关闭 diagnostic 诊断 LLM 调用(默认开启,给 failed sample 出"哪错了 + 怎么改"建议)
|
|
413
|
-
--skip-doctor escape hatch:跳过 doctor 健康检查门禁(默认强制启用)。沙箱 mock 提供依赖时绕开 doctor 物理路径误报;garbage-in 风险由调用方承担
|
|
414
|
-
--dry-run 仅预览
|
|
415
|
-
--blind 盲测模式
|
|
416
|
-
--concurrency <n> 并行任务数
|
|
417
|
-
--timeout <秒> 单任务超时
|
|
418
|
-
--repeat <n> 重复 N 次做方差分析
|
|
419
|
-
--batch 每个 artifact 独立和 baseline 对比
|
|
420
|
-
--bootstrap-samples N bootstrap 重采样次数(默认 1000)
|
|
421
|
-
--threshold <number> verdict 分层门禁阈值(默认 3.5)
|
|
422
|
-
--trivial-diff <num> 实际可忽略 diff 阈值(默认 0.1)
|
|
423
|
-
--report-only 生成报告并打印 verdict,但始终 exit 0
|
|
424
|
-
--no-gate --report-only 的别名
|
|
425
|
-
--skip-connectivity 跳过模型连通性检查(内部静态 gate 仍强制执行)
|
|
426
|
-
--no-serve 评测后不自动启动报告服务
|
|
427
|
-
```
|
|
428
|
-
|
|
429
|
-
HTML 报告有两个 tab:
|
|
430
|
-
- **📊 评分视角** — verdict 驱动的 A/B 对比(事实/行为/judge 三层、bootstrap CI、length-debias)。
|
|
431
|
-
- **✅ 功能视角** — 每条 sample 当一条单测看:用例设计(prompt / rubric / 工具调用 mock / environment)+ 执行轨迹 + 断言结果 + 可操作的 diagnostic 建议。诊断给出归因(skill 文档模糊 / LLM 误读 / sample 设计 bug / 诱错样本 / ...)、工作流校验(rubric 每步 ✓/✗ + 证据)和失败模式标签(工作流跳步 / 硬编码值 / 幻觉输出 / 工具误用 / 环境拦截 / 误读约束 / 其他)。沙箱 mock 字段语义(`mocks` / `environment` / `tripwire` / `mocksStrict`)见 [docs/sample-design-spec.md §三](./docs/sample-design-spec.md)。
|
|
432
|
-
|
|
433
|
-
### `omk observe`
|
|
434
|
-
|
|
435
|
-
omk observe 提供两条工作流:默认的 skill 健康度报告,以及 reviewer 闭环用的 observe inbox。
|
|
436
|
-
|
|
437
|
-
#### A. skill 健康度报告(默认)
|
|
438
|
-
|
|
439
|
-
```bash
|
|
440
|
-
omk observe ~/.claude/projects/-Users-you-Documents-my-project
|
|
441
|
-
omk observe ~/.claude/projects/my-project --last 7d
|
|
442
|
-
omk observe ~/.claude/projects/my-project --from 2026-04-01T00:00:00Z --to 2026-04-15T23:59:59Z
|
|
443
|
-
omk observe ~/.claude/projects/my-project --skills audit,polish
|
|
444
|
-
omk observe ~/.claude/projects/my-project --kb /path/to/project
|
|
445
|
-
```
|
|
446
|
-
|
|
447
|
-
把真实 Claude Code session trace 转成 skill 健康度报告:知识使用、gap 信号、执行稳定性、token 和耗时。这是生产观测,不是生产评分。
|
|
448
|
-
|
|
449
|
-
#### B. observe inbox:reviewer 闭环
|
|
450
|
-
|
|
451
|
-
把真实 session trace 解析、聚合、降噪,输出可逐条 review 的 observation 列表。整条链路纯本地、零 LLM。
|
|
452
|
-
|
|
453
|
-
```bash
|
|
454
|
-
# 1. 把 trace 解析、聚合、落盘到 .omk/observations/
|
|
455
|
-
omk observe ingest ~/.claude/projects/my-project
|
|
456
|
-
omk observe ingest ~/.claude/projects/my-project --output-dir ./custom-dir
|
|
457
|
-
|
|
458
|
-
# 2. 看 inbox(默认 top 20,按 severity / confidence / lastSeen 排序)
|
|
459
|
-
omk observe inbox
|
|
460
|
-
omk observe inbox --limit 50
|
|
461
|
-
omk observe inbox --skill audit # 只看某个 skill
|
|
462
|
-
omk observe inbox --by-skill # 按 skill 资产视图
|
|
463
|
-
omk observe inbox --explore 10 # 从 medium / low 桶抽 10 条长尾
|
|
464
|
-
omk observe inbox --explore 10 --include-noise # 显式包含 noise 桶
|
|
465
|
-
omk observe inbox --json # JSON 输出,便于自动化消费
|
|
466
|
-
|
|
467
|
-
# 3. 反向查单条 observation 的事件三元组(前后 message 上下文)
|
|
468
|
-
omk observe show <inbox_id>
|
|
469
|
-
```
|
|
470
|
-
|
|
471
|
-
每条 observation 自带:
|
|
472
|
-
|
|
473
|
-
+ `confidence` 与 `attributionConfidence`:信号可信度 + skill 归因可信度,并列展示
|
|
474
|
-
+ `severityReasonCode`:判断为该 severity 的稳定结构化原因;人类可读说明由 CLI / studio 渲染时生成
|
|
475
|
-
+ `messageWindow`:前 3 条 / 触发点 / 后 3 条 message 上下文 + `resolutionAfter`(后续是否解决)
|
|
476
|
-
+ `evidence.{messageIndex,messageUuid,toolUseId}`:可反向回到原始 jsonl 的锚点
|
|
477
|
-
|
|
478
|
-
支持 trace 格式:Claude Code session JSONL(`.jsonl`)、OpenClaw session JSONL(`.jsonl`)、markdown 对话日志(`.log`)。
|
|
479
|
-
|
|
480
|
-
### `omk evolve`
|
|
481
|
-
|
|
482
|
-
```bash
|
|
483
|
-
omk evolve <skill> # 多轮自动迭代 skill
|
|
484
|
-
omk evolve skills/foo.md --rounds 10 --target 4.5
|
|
485
|
-
```
|
|
486
|
-
|
|
487
|
-
让 skill 跑 eval → judge → 改写 SKILL.md 的多轮闭环,直到达到 `--target` 或 `--rounds` 上限。耗时按 `轮数 × 样本 × 变体` 累加,几分钟到几十分钟级别。原始 skill 文件版本保存在 `skills/evolve/*.r0.md`。
|
|
488
|
-
|
|
489
|
-
### `omk sample`
|
|
490
|
-
|
|
491
|
-
```bash
|
|
492
|
-
omk sample <skill> # 为单个 skill 生成或补齐评测用例
|
|
493
|
-
omk sample --batch # 为目录下缺评测集的 skill 批量生成
|
|
494
|
-
```
|
|
495
|
-
|
|
496
|
-
一次性生成。自动给生成的用例打 `provenance`。生成的 assertions 使用英文 / 数字 / 代码 token,便于跨中英文输出对比。
|
|
497
|
-
|
|
498
|
-
### `omk studio`
|
|
499
|
-
|
|
500
|
-
```bash
|
|
501
|
-
omk studio
|
|
502
|
-
omk studio --port 7799
|
|
503
|
-
omk studio --host 0.0.0.0 # 局域网访问(默认 127.0.0.1)
|
|
504
|
-
omk studio --reports-dir ~/.oh-my-knowledge/reports
|
|
505
|
-
omk studio --observations-dir .omk/observations # observe inbox 数据目录
|
|
506
|
-
omk studio --no-open
|
|
507
|
-
```
|
|
508
|
-
|
|
509
|
-
启动本地知识工作台浏览报告。verdict、样本回退、跨样本 diff、饱和曲线、单样本 drill-down 全部在 studio UI 里 —— omk 不提供 CLI 导出 / 分析子命令。CI gate 用 `omk eval` 的 exit code(PROGRESS 退 0、其他非 0),需要文字摘要自己 `jq` report JSON。
|
|
510
|
-
|
|
511
|
-
Studio 是 skill-centric 信息架构 — 列表页(`/`)按 skill 卡片展示健康等级 / 0-100 参考分 / 待优化数 / 趋势,详情页(`/skills/<name>`)左栏列关键问题清单(skill 优化 / 用例优化 / 工具反馈三档),右栏画 chart.js 健康趋势 + 三个紧凑阶段卡(doctor / eval / observe),细节走 modal。旧的 run 列表挪到 `/runs`。访问 `/observations/inbox` 查看 observe inbox 看板:按 skill 资产视图(rollup)+ reviewer 待办建议 + 当前可观测漏斗 + 单 observation 详情面板(含事件三元组)。
|
|
512
|
-
|
|
513
|
-
## 执行器
|
|
514
|
-
|
|
515
|
-
### 内置执行器
|
|
516
|
-
|
|
517
|
-
| 执行器 | 适用场景 | 说明 |
|
|
518
|
-
|--------|----------|------|
|
|
519
|
-
| `claude` | 默认 | 通过 `claude -p` 调用 Claude CLI |
|
|
520
|
-
| `claude-sdk` | 结构化输出 | 通过 Claude Agent SDK 调用,无 stdout 解析,避免 buffer 截断 |
|
|
521
|
-
| `codex` | OpenAI agent CLI | 通过 `codex exec --json` 调用,需本地装好登录的 codex(`@openai/codex`);best-effort tool trace,**costUSD 不报**(codex 自身不输出 USD,需外部账单核算) |
|
|
522
|
-
| `codex-sdk` | OpenAI agent SDK | 通过 `@openai/codex-sdk` 调用其自带的 `@openai/codex` binary 和 SDK 事件流;**costUSD 不报** |
|
|
523
|
-
| `gemini` | 跨厂商对比 | 通过 `gemini` CLI 调用 |
|
|
524
|
-
| `anthropic-api` | 无需 CLI | 直接调用 Anthropic HTTP API(需 `ANTHROPIC_API_KEY`) |
|
|
525
|
-
| `openai-api` | 无需 CLI | 直接调用 OpenAI HTTP API(需 `OPENAI_API_KEY`) |
|
|
526
|
-
|
|
527
|
-
API 直调执行器支持通过环境变量自定义 Base URL:`ANTHROPIC_BASE_URL`、`OPENAI_BASE_URL`。
|
|
528
|
-
|
|
529
|
-
Codex construct-validity 说明:(1)`codex` 使用 `PATH` 上找到的 `codex` binary;`codex-sdk` 使用 `@openai/codex-sdk` 解析到的自带 `@openai/codex` binary。报告会持久化 per-variant `meta.executorRuntimes`、`meta.executorRuntime`,以及每个评委的 `meta.judgeModels[].runtime` 指纹(binary 或 SDK 版本 + 能力快照),strict comparability checks 会在 runtime 指纹无法审计时提示。runtime 指纹不一致时,结果应解释为 executor runtime 对比,而不只是 prompt/template 行为对比。(2)两个 executor 都隔离用户级 config:`codex` 传 `--ephemeral` + `--ignore-user-config`,`codex-sdk` 把 `$CODEX_HOME` 重定向到 per-process tmp 目录(auth.json 通过 symlink 透传)。用户的 `~/.codex/config.toml` 不会渗入任意一个 executor 的 eval。
|
|
530
|
-
|
|
531
|
-
### 自定义执行器
|
|
532
|
-
|
|
533
|
-
任何 shell 命令都可以作为执行器,通过 stdin/stdout JSON 协议通信:
|
|
534
|
-
|
|
535
|
-
```bash
|
|
536
|
-
omk eval --executor "python my_provider.py"
|
|
537
|
-
omk eval --executor "./my-executor.sh"
|
|
538
|
-
```
|
|
539
|
-
|
|
540
|
-
**协议约定:**
|
|
541
|
-
|
|
542
|
-
- **输入**(stdin):JSON `{"model":"...","system":"...","prompt":"..."}`
|
|
543
|
-
- **输出**(stdout):JSON `{"output":"模型回复","inputTokens":0,"outputTokens":0,"costUSD":0}`
|
|
544
|
-
- stdout 中只需返回有值的字段,其余默认为 0;也可以直接输出纯文本(不解析 token/成本)
|
|
545
|
-
- 非零退出码视为执行失败
|
|
546
|
-
|
|
547
|
-
### Artifact 目录结构
|
|
548
|
-
|
|
549
|
-
默认执行器(claude/openai/gemini)支持两种 artifact 布局,同一次评测中可混用:
|
|
550
|
-
|
|
551
|
-
```
|
|
552
|
-
skills/
|
|
553
|
-
├── v1.md # 方式一:直接放 .md 文件
|
|
554
|
-
└── my-skill/ # 方式二:完整 artifact 目录
|
|
555
|
-
├── SKILL.md # 工具自动读取此文件作为 system prompt
|
|
556
|
-
├── config.json # 其他文件不参与评测,仅保留完整性
|
|
557
|
-
└── scripts/
|
|
558
|
-
```
|
|
559
|
-
|
|
560
|
-
**Variant 解析规则:**
|
|
561
|
-
|
|
562
|
-
`variant` 是实验分组表达式。解析之后,OMK 会得到一个 `artifact` 与可选的 `runtime context`(当前主要是 `cwd`)。
|
|
563
|
-
|
|
564
|
-
| 格式 | 含义 |
|
|
565
|
-
|------|------|
|
|
566
|
-
| `name` | 从 artifact 目录查找 `name.md` 或 `name/SKILL.md`,解析为一个 artifact |
|
|
567
|
-
| `baseline` | 空 artifact,不使用 system prompt;可直接理解为“什么都没有” |
|
|
568
|
-
| `project-env@/path/to/project` | 空 artifact,但在指定项目目录运行,用于单独观察项目级 runtime context |
|
|
569
|
-
| `git:name` | 从 git HEAD 读取一个 artifact 的上次提交版本 |
|
|
570
|
-
| `git:ref:name` | 从 git 指定 commit 读取一个 artifact |
|
|
571
|
-
| `./path/to/file.md` | 含 `/` 的路径,直接读取文件作为 artifact |
|
|
572
|
-
| `variant@/path/to/project` | 给任意变体附加运行目录,支持 `name@cwd`、`git:name@cwd`、`/file.md@cwd` |
|
|
573
|
-
|
|
574
|
-
`--control` 和 `--treatment` 都不传时,用 `--config eval.yaml` 或 `--batch`。`--batch` 模式下会自动用 `baseline` 作对照组,每个被发现的 artifact 作实验组。
|
|
575
|
-
|
|
576
|
-
```bash
|
|
577
|
-
# 显式:一个 control,一个或多个 treatment
|
|
578
|
-
omk eval --control v1 --treatment v2
|
|
579
|
-
omk eval --control baseline --treatment v1,v2,v3
|
|
580
|
-
|
|
581
|
-
# 对比空 artifact 和显式 artifact 的效果差异
|
|
582
|
-
omk eval --control baseline --treatment my-skill
|
|
583
|
-
|
|
584
|
-
# 单独观察项目级 runtime context 的影响(用自描述标签)
|
|
585
|
-
omk eval --control baseline --treatment project-env@/path/to/target-project
|
|
586
|
-
|
|
587
|
-
# 对比"项目级 runtime context"与"显式 artifact 注入"
|
|
588
|
-
omk eval \
|
|
589
|
-
--control project-env@/path/to/target-project \
|
|
590
|
-
--treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
|
|
591
|
-
|
|
592
|
-
# 对比修改前后(旧版本从 git 历史读取)
|
|
593
|
-
omk eval --control git:my-skill --treatment my-skill
|
|
594
|
-
|
|
595
|
-
# 直接指定文件路径
|
|
596
|
-
omk eval --control ./old-skill.md --treatment ./new-skill.md
|
|
597
|
-
|
|
598
|
-
# 配置文件驱动(evaluation-as-code)
|
|
599
|
-
omk eval --config eval.yaml
|
|
600
|
-
```
|
|
601
|
-
|
|
602
|
-
**前置要求:**
|
|
603
|
-
|
|
604
|
-
- **claude**:安装 [Claude Code](https://claude.ai/code) 并认证
|
|
605
|
-
- **claude-sdk**:安装 [Claude Code](https://claude.ai/code) 并认证(使用 Agent SDK,无需 CLI stdout 解析)
|
|
606
|
-
- **anthropic-api**:设置 `ANTHROPIC_API_KEY` 环境变量
|
|
607
|
-
- **openai**:`pip install openai` 并设置 `OPENAI_API_KEY`
|
|
608
|
-
- **openai-api**:设置 `OPENAI_API_KEY` 环境变量
|
|
609
|
-
- **gemini**:`npm i -g @google/gemini-cli` 并认证
|
|
610
|
-
|
|
611
|
-
### Agent 评测与项目级 Runtime Context
|
|
612
|
-
|
|
613
|
-
当执行器使用 `claude-sdk` 时,OMK 现在已经支持第一版 agent-aware evaluation。
|
|
614
|
-
|
|
615
|
-
这里建议把几个概念分开理解:
|
|
616
|
-
|
|
617
|
-
- `artifact`:被评测对象,例如 baseline、skill、prompt、agent
|
|
618
|
-
- `variant`:CLI 里的实验分组表达式
|
|
619
|
-
- `runtime context`:运行时上下文,当前主要是 `cwd`;在项目型 agent 场景下,它就包含项目目录、`CLAUDE.md`、本地 skills 等会影响行为的环境因素
|
|
620
|
-
|
|
621
|
-
在 OMK 里,`agent` 不是所有对象的总称,`skill` 也不是所有对象的总称。更稳妥的说法是:你在比较不同 artifact 在不同 runtime context 下的表现。
|
|
622
|
-
|
|
623
|
-
- 自动抽取 turns / toolCalls trace
|
|
624
|
-
- 支持基于工具调用行为的断言
|
|
625
|
-
- 支持在指定 `cwd` 下运行,让 Claude Code 自动加载项目内的 `CLAUDE.md`、skills 和本地 runtime context
|
|
626
|
-
|
|
627
|
-
#### 推荐执行器
|
|
628
|
-
|
|
629
|
-
```bash
|
|
630
|
-
omk eval --executor claude-sdk
|
|
631
|
-
```
|
|
632
|
-
|
|
633
|
-
#### 支持的 agent 相关断言
|
|
634
|
-
|
|
635
|
-
| 断言 | 含义 |
|
|
636
|
-
|------|------|
|
|
637
|
-
| `tools_called` | 必须调用指定工具 |
|
|
638
|
-
| `tools_not_called` | 禁止调用指定工具 |
|
|
639
|
-
| `tools_count_min` / `tools_count_max` | 工具调用次数上下界 |
|
|
640
|
-
| `tool_output_contains` | 指定工具输出必须包含关键内容 |
|
|
641
|
-
| `turns_min` / `turns_max` | 交互轮次上下界 |
|
|
642
|
-
|
|
643
|
-
#### 三种常见对照组
|
|
644
|
-
|
|
645
|
-
**1. 裸模型 baseline**
|
|
646
|
-
|
|
647
|
-
不注入 system prompt,也不进入带知识的项目目录。至少需要一个 treatment 做对比:
|
|
648
|
-
|
|
649
|
-
```bash
|
|
650
|
-
omk eval \
|
|
651
|
-
--executor claude-sdk \
|
|
652
|
-
--control baseline \
|
|
653
|
-
--treatment my-skill
|
|
654
|
-
```
|
|
655
|
-
|
|
656
|
-
**2. 空 artifact + 项目级 runtime context**
|
|
657
|
-
|
|
658
|
-
不注入 system prompt,但在项目目录运行。它不是严格意义上的"裸 baseline",而是"空 artifact + 项目级 runtime context"。
|
|
659
|
-
|
|
660
|
-
```bash
|
|
661
|
-
omk eval \
|
|
662
|
-
--executor claude-sdk \
|
|
663
|
-
--control baseline \
|
|
664
|
-
--treatment project-env@/path/to/target-project
|
|
665
|
-
```
|
|
666
|
-
|
|
667
|
-
**3. 显式 artifact 注入**
|
|
668
|
-
|
|
669
|
-
直接把某个外部 `SKILL.md` 作为 artifact 注入,同时保留项目目录上下文。适合对比"项目级 runtime context"与"显式单 artifact 注入"之间的差异。
|
|
670
|
-
|
|
671
|
-
```bash
|
|
672
|
-
omk eval \
|
|
673
|
-
--executor claude-sdk \
|
|
674
|
-
--control project-env@/path/to/target-project \
|
|
675
|
-
--treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
|
|
676
|
-
```
|
|
677
|
-
|
|
678
|
-
#### 推荐的第一轮对照设计
|
|
679
|
-
|
|
680
|
-
对于 PRD / 复杂业务知识场景,建议从下面开始:
|
|
681
|
-
|
|
682
|
-
```bash
|
|
683
|
-
omk eval \
|
|
684
|
-
--executor claude-sdk \
|
|
685
|
-
--samples skills/evaluate-review/eval-samples.yaml \
|
|
686
|
-
--control baseline \
|
|
687
|
-
--treatment /path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
|
|
688
|
-
```
|
|
689
|
-
|
|
690
|
-
如果你想证明"项目目录中的知识沉淀本身"是否有效,加第二个 treatment:
|
|
691
|
-
|
|
692
|
-
```bash
|
|
693
|
-
omk eval \
|
|
694
|
-
--executor claude-sdk \
|
|
695
|
-
--samples skills/evaluate-review/eval-samples.yaml \
|
|
696
|
-
--control baseline \
|
|
697
|
-
--treatment project-env@/path/to/target-project,/path/to/target-project/.claude/skills/prd/SKILL.md@/path/to/target-project
|
|
698
|
-
```
|
|
699
|
-
|
|
700
|
-
#### 设计建议
|
|
701
|
-
|
|
702
|
-
- **先用 `--dry-run`**:确认样本、variant 和 `cwd` 被正确解析
|
|
703
|
-
- **项目级对照必须区分 `cwd`**:相同 prompt 在不同项目目录下会走不同 runtime context
|
|
704
|
-
- **优先先跑 PRD 场景**:相比 Coding,更容易验证知识完整性、影响面识别和业务正确性
|
|
705
|
-
|
|
706
|
-
### 常见模型配置示例
|
|
707
|
-
|
|
708
|
-
**没有 Claude?** 大多数国产模型(GLM、通义千问、Moonshot、DeepSeek 等)都兼容 OpenAI API 格式,可以直接使用 `openai-api` 执行器:
|
|
709
|
-
|
|
710
|
-
```bash
|
|
711
|
-
# GLM(智谱)
|
|
712
|
-
export OPENAI_API_KEY="你的智谱 API Key"
|
|
713
|
-
export OPENAI_BASE_URL="https://open.bigmodel.cn/api/paas/v4"
|
|
714
|
-
omk eval --executor openai-api --model glm-4-plus \
|
|
715
|
-
--judge-models openai-api:glm-4-plus --no-cache
|
|
716
|
-
|
|
717
|
-
# 通义千问
|
|
718
|
-
export OPENAI_API_KEY="你的通义 API Key"
|
|
719
|
-
export OPENAI_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"
|
|
720
|
-
omk eval --executor openai-api --model qwen-plus \
|
|
721
|
-
--judge-models openai-api:qwen-plus
|
|
722
|
-
|
|
723
|
-
# DeepSeek
|
|
724
|
-
export OPENAI_API_KEY="你的 DeepSeek API Key"
|
|
725
|
-
export OPENAI_BASE_URL="https://api.deepseek.com"
|
|
726
|
-
omk eval --executor openai-api --model deepseek-chat \
|
|
727
|
-
--judge-models openai-api:deepseek-chat
|
|
728
|
-
|
|
729
|
-
# Moonshot(Kimi)
|
|
730
|
-
export OPENAI_API_KEY="你的 Moonshot API Key"
|
|
731
|
-
export OPENAI_BASE_URL="https://api.moonshot.cn/v1"
|
|
732
|
-
omk eval --executor openai-api --model moonshot-v1-8k \
|
|
733
|
-
--judge-models openai-api:moonshot-v1-8k
|
|
734
|
-
```
|
|
735
|
-
|
|
736
|
-
**Ollama 本地模型:**
|
|
737
|
-
|
|
738
|
-
```bash
|
|
739
|
-
omk eval --executor "python examples/custom-executor/ollama-executor.py" \
|
|
740
|
-
--model llama3 --no-judge
|
|
741
|
-
```
|
|
742
|
-
|
|
743
|
-
**关于评委:**
|
|
100
|
+
## 文档
|
|
744
101
|
|
|
745
|
-
-
|
|
746
|
-
-
|
|
747
|
-
-
|
|
748
|
-
-
|
|
102
|
+
- **[工作原理](docs/zh/explanation/architecture.md)** —— 交错调度、variant 解析、双通道评分、六维报告
|
|
103
|
+
- **[评测样本格式](docs/zh/reference/eval-sample-format.md)** —— sample schema、评分公式、21+ 断言类型、自定义 JS 断言
|
|
104
|
+
- **[CLI 参考](docs/zh/reference/cli.md)** —— 7 个命令的 bash 示例和 flag 表
|
|
105
|
+
- **[执行器与 artifact 布局](docs/zh/reference/executors.md)** —— 内置 / 自定义执行器、Agent 评测、常见模型配置(Claude / OpenAI / GLM / 通义 / DeepSeek / Moonshot / Ollama)
|
|
106
|
+
- **[快速上手](docs/zh/quickstart-skill-eval.md)** —— 第一次跑评测的 5 分钟教程
|
|
107
|
+
- **[用例设计规范](docs/specs/sample-design-spec.md)** —— capability / construct / provenance 元数据;行业 gap 映射
|
|
108
|
+
- **[统计严谨性](docs/zh/explanation/statistical-rigor.md)** —— 为什么 Bootstrap CI / α / 长度去偏 / 饱和曲线重要
|
|
109
|
+
- **[7 工具对比](docs/zh/reference/comparison.md)** —— promptfoo / DeepEval / LangSmith / Langfuse / Braintrust 等 25+ 维度横评
|
|
749
110
|
|
|
750
111
|
## 环境变量
|
|
751
112
|
|
|
752
113
|
| 变量 | 说明 |
|
|
753
114
|
|------|------|
|
|
754
|
-
| `CCV_PROXY_URL` |
|
|
755
|
-
| `OMK_REPORT_PORT` |
|
|
115
|
+
| `CCV_PROXY_URL` | 通过 cc-viewer 代理请求,实时可视化评测流量 |
|
|
116
|
+
| `OMK_REPORT_PORT` | 报告服务端口(默认 7799) |
|
|
756
117
|
|
|
757
118
|
## 系统要求
|
|
758
119
|
|
|
759
120
|
- Node.js >= 20
|
|
760
|
-
- `claude` CLI
|
|
761
|
-
-
|
|
121
|
+
- `claude` CLI(默认执行器和 LLM 评委需要,参考 [Claude Code](https://claude.ai/code))
|
|
122
|
+
- 如果使用其它执行器(openai / gemini)+ `--no-judge` 则可不需要
|
|
762
123
|
|
|
763
124
|
## 安全说明
|
|
764
125
|
|
|
765
126
|
本工具设计用于**本地可信环境**(开发机、CI 流水线)。以下功能会执行本地代码,请确保输入来源可信:
|
|
766
127
|
|
|
767
|
-
| 功能 |
|
|
768
|
-
|
|
769
|
-
|
|
|
770
|
-
| **eval-samples.json** |
|
|
128
|
+
| 功能 | 风险 | 适用场景 |
|
|
129
|
+
|------|------|----------|
|
|
130
|
+
| **自定义断言**(`custom`) | 动态加载并执行用户指定的 `.mjs` 文件 | 仅使用自己编写或审查过的断言文件 |
|
|
131
|
+
| **eval-samples.json** | 断言配置可引用外部文件路径 | 不要使用来源不明的样本文件 |
|
|
771
132
|
|
|
772
133
|
**建议:**
|
|
773
134
|
|
|
774
|
-
-
|
|
775
|
-
-
|
|
776
|
-
- 自定义断言有 30
|
|
135
|
+
- 不要将本地报告服务暴露到公网(无身份认证)
|
|
136
|
+
- 不使用未经审查的第三方 eval-samples
|
|
137
|
+
- 自定义断言有 30 秒超时,但无沙箱隔离
|
|
777
138
|
|
|
778
139
|
---
|
|
779
140
|
|
|
780
|
-
|
|
141
|
+
发布日志见 [GitHub Releases](https://github.com/lizhiyao/oh-my-knowledge/releases)。欢迎贡献 —— 见 [CONTRIBUTING](./CONTRIBUTING.md)。
|