dsh-omni-router 2.0.0 → 2.2.0

package/README.md

@@ -71,7 +71,9 @@
 - **Evidence Engine / Write Locks**：`src/evidence-engine.mjs` 统一证据协议与存储；DAG scheduler 通过 `selectReadyBatch` 避免 write overlap 并行冲突。
 - **Stage 3/5**：Recursive Project Index、contextBudget 接入、Per-task Context、Missing capability handling、OmniBench v2 prompt generator。
 - **Stage 4（部分）**：`compileDagToWorkflow` 将 Mission DAG 编译为 DSH-native workflow outline。
-- **P1/P3 能力**：Real capability sandbox（已接入 `omni_mission_run`，按 role + capability 生成 toolFilter）、Capability Resolver v2（risk/reliability 综合打分）、OmniBench v2 scaffold（manifest schema + runner stub + prepare script + DAG-to-plan compile）。
+- **P1/P3 能力**：Real capability sandbox（已接入 `omni_mission_run`，按 role + capability 生成 toolFilter）、Capability Resolver v2（risk/reliability 综合打分）、OmniBench v2 runner（manifest schema + prepare + prompt generation + `--exec` 本地执行/结果采集 + DAG-to-plan compile）。
+- **2.1 收敛增强**：Mission Resume（`omni_mission_resume` 跨 session 继续 DAG，`runDagLoop` 支持 `onProgress` 周期保存）、Harness evidence extraction（`extractHarnessEvidence` 从 commands/tests/toolCalls/EVIDENCE_JSON 提取机器可验证证据）、context token budget（`maxContextTokens` 真正限制检索输出）、semantic failure retrieval + evidence-backed skill distillation、OmniBench v2 real runner。
+- **2.2 Capability Auto-Provisioning（优化7）**：Capability Auditor（baseline/task-time audit）、Gap Detection、Plugin/Skill Quality Scoring、Minimal Capability Set Solver、Discovery Adapters（marketplace/hub/static registry）、Trusted Auto Provisioning（recommend/auto-trusted/manual）、Post-install probe、Install rollback transaction、Capability performance learning / redundancy detection。
 - **TaskDecision**：`createTaskDecision` 生成唯一决策对象，Policy/Runtime 统一消费，避免重复 classify 分叉。
 - **Evidence Protocol**：`src/evidence.mjs` 用结构化 command/file/test/finding 证据判定 PASS/FAIL；`isQaPass` 已支持从 QA 输出解析 JSON evidence，不再只信文本。
 - **Task Compiler**：`compileTask` 生成 objective / constraints / non-goals / acceptance / hidden assumptions / ambiguities / invariants / risk / artifacts；`compileTaskWithLLM` 可用 LLM 增强。
@@ -153,7 +155,7 @@ Then restart DSH and select **Omni Router** in a new session.
 - `/omni direct` — enter direct mode.
 - `/omni mode spec|react|balanced` — set thinking mode.
 - `/omni reroute plan|direct` — adaptively reroute current task.
-- Model tools: `omni_status`, `omni_plan`, `omni_direct`, `omni_mode`, `omni_reroute`, `omni_delegate`, `omni_memory`, `omni_benchmark`, `omni_mission_run`, `omni_visual_check`.
+- Model tools: `omni_status`, `omni_plan`, `omni_direct`, `omni_mode`, `omni_reroute`, `omni_delegate`, `omni_memory`, `omni_benchmark`, `omni_mission_run`, `omni_mission_resume`, `omni_capability_audit`, `omni_capability_provision`, `omni_capability_probe`, `omni_visual_check`.
 
 ## Configuration
 

package/README.zh-CN.md

@@ -71,7 +71,9 @@
 - **Evidence Engine / Write Locks**：`src/evidence-engine.mjs` 统一证据协议与存储；DAG scheduler 通过 `selectReadyBatch` 避免 write overlap 并行冲突。
 - **Stage 3/5**：Recursive Project Index、contextBudget 接入、Per-task Context、Missing capability handling、OmniBench v2 prompt generator。
 - **Stage 4（部分）**：`compileDagToWorkflow` 将 Mission DAG 编译为 DSH-native workflow outline。
-- **P1/P3 能力**：Real capability sandbox（已接入 `omni_mission_run`，按 role + capability 生成 toolFilter）、Capability Resolver v2（risk/reliability 综合打分）、OmniBench v2 scaffold（manifest schema + runner stub + prepare script + DAG-to-plan compile）。
+- **P1/P3 能力**：Real capability sandbox（已接入 `omni_mission_run`，按 role + capability 生成 toolFilter）、Capability Resolver v2（risk/reliability 综合打分）、OmniBench v2 runner（manifest schema + prepare + prompt generation + `--exec` 本地执行/结果采集 + DAG-to-plan compile）。
+- **2.1 收敛增强**：Mission Resume（`omni_mission_resume` 跨 session 继续 DAG，`runDagLoop` 支持 `onProgress` 周期保存）、Harness evidence extraction（`extractHarnessEvidence` 从 commands/tests/toolCalls/EVIDENCE_JSON 提取机器可验证证据）、context token budget（`maxContextTokens` 真正限制检索输出）、semantic failure retrieval + evidence-backed skill distillation、OmniBench v2 real runner。
+- **2.2 Capability Auto-Provisioning（优化7）**：Capability Auditor（baseline/task-time audit）、Gap Detection、Plugin/Skill Quality Scoring、Minimal Capability Set Solver、Discovery Adapters（marketplace/hub/static registry）、Trusted Auto Provisioning（recommend/auto-trusted/manual）、Post-install probe、Install rollback transaction、Capability performance learning / redundancy detection。
 - **TaskDecision**：`createTaskDecision` 生成唯一决策对象，Policy/Runtime 统一消费，避免重复 classify 分叉。
 - **Evidence Protocol**：`src/evidence.mjs` 用结构化 command/file/test/finding 证据判定 PASS/FAIL；`isQaPass` 已支持从 QA 输出解析 JSON evidence，不再只信文本。
 - **Task Compiler**：`compileTask` 生成 objective / constraints / non-goals / acceptance / hidden assumptions / ambiguities / invariants / risk / artifacts；`compileTaskWithLLM` 可用 LLM 增强。
@@ -153,7 +155,7 @@ node scripts/install-preset.mjs
 - `/omni direct` — 进入直接执行模式。
 - `/omni mode spec|react|balanced` — 设置思维模式。
 - `/omni reroute plan|direct` — 动态切换当前任务路由。
-- 模型工具：`omni_status` / `omni_plan` / `omni_direct` / `omni_mode` / `omni_reroute` / `omni_delegate` / `omni_memory` / `omni_benchmark` / `omni_mission_run` / `omni_visual_check`。
+- 模型工具：`omni_status` / `omni_plan` / `omni_direct` / `omni_mode` / `omni_reroute` / `omni_delegate` / `omni_memory` / `omni_benchmark` / `omni_mission_run` / `omni_mission_resume` / `omni_capability_audit` / `omni_capability_provision` / `omni_capability_probe` / `omni_visual_check`。
 
 ## 配置
 

package/benchmark/omnibench-v2/README.md

@@ -1,8 +1,8 @@
-# OmniBench v2 (scaffold)
+# OmniBench v2
 
 Goal: move from 10 small fixture tasks to a reproducible multi-repo benchmark.
 
-## Requirements (from 优化5.md)
+## Requirements (from 优化5.md / 优化6.md)
 
 - 30~100 real GitHub repos
 - fixed starting commit
@@ -25,18 +25,18 @@ Goal: move from 10 small fixture tasks to a reproducible multi-repo benchmark.
   "task": "Fix intermittent login timeout under concurrent session refresh.",
   "acceptance": ["regression test added", "all tests pass"],
   "hiddenTests": ["tests/hidden/session-concurrent.test.js"],
+  "setupCommand": "npm ci",
+  "baselineCommand": "npm test",
+  "agentCommand": "node path/to/dsh-agent-runner.mjs",
+  "verifyCommand": "npm run test:hidden",
   "timeoutMs": 7200000,
   "runs": 5
 }
 ```
 
-See `manifest.example.json`.
-
-## Runner stub
-
-`node benchmark/omnibench-v2/runner-stub.mjs <manifest.json>` validates the
-manifest and prints a run plan. Actual execution will be driven by DSH
-subagents in a future integration.
+`setupCommand`, `baselineCommand`, `agentCommand`, and `verifyCommand` are
+optional. `agentCommand` receives the generated prompt as a single quoted
+argument; when present, `--exec` executes it inside the checked-out repo.
 
 ## Generate DSH prompts
 
@@ -46,7 +46,29 @@ npm run omnibench:v2:generate -- benchmark/omnibench-v2/manifest.local.example.j
 
 This writes one prompt per repo/arm/run under `benchmark/omnibench-v2/prompts/`.
 Open each prompt in a DSH Desktop session with the Omni Router preset to
-execute, then collect the results.
+execute manually, or wire `agentCommand` to your DSH CLI/runner for automation.
+
+## Run locally (automated)
+
+```bash
+# Plan + prompt generation only (default)
+node benchmark/omnibench-v2/run.mjs benchmark/omnibench-v2/manifest.local.example.json
+
+# Execute each run with the manifest's agentCommand
+node benchmark/omnibench-v2/run.mjs benchmark/omnibench-v2/manifest.local.example.json --exec
+
+# Override the agent command for every run
+node benchmark/omnibench-v2/run.mjs benchmark/omnibench-v2/manifest.local.example.json --exec --agent-command "node agent.mjs"
+```
+
+The runner clones/checks out each repo, optionally runs setup/baseline/verify
+commands, captures exit codes and output, and writes a timestamped JSON result
+file to `benchmark/omnibench-v2/results/`.
+
+## Runner stub
+
+`node benchmark/omnibench-v2/runner-stub.mjs <manifest.json>` still validates
+the manifest and prints the run plan for quick checks.
 
 ## Local / offline usage
 
@@ -57,6 +79,7 @@ existing `Omni group` fixture:
 ```bash
 npm run omnibench:v2:prepare -- benchmark/omnibench-v2/manifest.local.example.json
 npm run omnibench:v2:plan -- benchmark/omnibench-v2/manifest.local.example.json
+npm run omnibench:v2:generate -- benchmark/omnibench-v2/manifest.local.example.json
 ```
 
 `git clone` works with local paths, so no network is required. Replace `repo`

package/benchmark/omnibench-v2/run.mjs

@@ -2,20 +2,24 @@
 /**
  * OmniBench v2 runner / prompt generator.
  *
- * Generates per-repo per-arm per-run DSH prompts. Actual execution must happen
- * inside DSH Desktop; this script makes that reproducible.
+ * Generates per-repo per-arm per-run DSH prompts. With `--exec` and a
+ * configured `agentCommand` (or `--agent-command`), it also executes the run
+ * locally, runs baseline/verify commands, and writes machine-readable results.
  *
  * Usage:
- *   node benchmark/omnibench-v2/run.mjs <manifest.json>
+ *   node benchmark/omnibench-v2/run.mjs <manifest.json>            # plan + prompts
+ *   node benchmark/omnibench-v2/run.mjs <manifest.json> --exec     # execute agentCommand runs
+ *   node benchmark/omnibench-v2/run.mjs <manifest.json> --exec --agent-command "node agent.mjs"
  */
 
 import fs from 'node:fs'
 import path from 'node:path'
-import { fileURLToPath } from 'node:url'
+import { execSync } from 'node:child_process'
+import { fileURLToPath, pathToFileURL } from 'node:url'
 
 const here = path.dirname(fileURLToPath(import.meta.url))
 
-function buildPrompt(manifest, arm) {
+export function buildPrompt(manifest, arm) {
   const criteria = (manifest.acceptance || []).map((c) => `- ${c}`).join('\n')
   const task = `Task:\n${manifest.task}\n\nAcceptance criteria:\n${criteria}`
   if (arm === 'raw') {
@@ -24,10 +28,54 @@ function buildPrompt(manifest, arm) {
   return `${task}\n\nYou are using the Omni control plane. Follow engineering methodology, verify with real evidence, and do not claim completion without checks. When done, reply exactly "BENCHMARK: PASS" if you verified all criteria, otherwise "BENCHMARK: FAIL".`
 }
 
-function main() {
-  const manifestPath = process.argv[2] || path.join(here, 'manifest.local.example.json')
+export function readManifests(manifestPath) {
   const manifests = JSON.parse(fs.readFileSync(manifestPath, 'utf8'))
-  const list = Array.isArray(manifests) ? manifests : [manifests]
+  return Array.isArray(manifests) ? manifests : [manifests]
+}
+
+export function validateManifests(list) {
+  const errors = []
+  for (const m of list) {
+    if (!m.id) errors.push('missing id')
+    if (!m.repo) errors.push(`${m.id || '?'}: missing repo`)
+    if (!m.commit) errors.push(`${m.id || '?'}: missing commit`)
+    if (!m.task) errors.push(`${m.id || '?'}: missing task`)
+    if (!Array.isArray(m.acceptance) || !m.acceptance.length) errors.push(`${m.id || '?'}: missing acceptance`)
+    if (!m.runs || m.runs < 3) errors.push(`${m.id || '?'}: runs should be >= 3`)
+  }
+  return errors
+}
+
+export function ensureRepo(m) {
+  const workDir = path.resolve(process.env.OMNIBENCH_REPOS || path.join(here, 'repos'))
+  const dir = path.join(workDir, m.id)
+  fs.mkdirSync(workDir, { recursive: true })
+  if (!fs.existsSync(path.join(dir, '.git'))) {
+    console.log(`cloning ${m.repo} -> ${dir}`)
+    execSync(`git clone "${m.repo}" "${dir}"`, { stdio: 'inherit' })
+  }
+  console.log(`checkout ${m.id} @ ${m.commit}`)
+  execSync(`git -C "${dir}" checkout ${m.commit}`, { stdio: 'inherit' })
+  return dir
+}
+
+export function runCommand(command, cwd, label) {
+  if (!command) return { skipped: true }
+  const start = Date.now()
+  try {
+    const output = execSync(command, { cwd, encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] })
+    return { skipped: false, exitCode: 0, output, durationMs: Date.now() - start }
+  } catch (error) {
+    return {
+      skipped: false,
+      exitCode: error.status ?? 1,
+      output: String(error.stdout || '') + String(error.stderr || ''),
+      durationMs: Date.now() - start,
+    }
+  }
+}
+
+export function generatePrompts(list) {
   const outDir = path.join(here, 'prompts')
   fs.mkdirSync(outDir, { recursive: true })
   let count = 0
@@ -40,8 +88,115 @@ function main() {
       }
     }
   }
-  console.log(`Generated ${count} DSH prompts in ${outDir}`)
-  console.log('Run each prompt inside a DSH Desktop session with the Omni Router preset, then collect results.')
+  return count
+}
+
+export function writeResults(results, resultsDir) {
+  fs.mkdirSync(resultsDir, { recursive: true })
+  const file = path.join(resultsDir, `omnibench-v2-${Date.now()}.json`)
+  fs.writeFileSync(file, JSON.stringify(results, null, 2), 'utf8')
+  return file
+}
+
+function main() {
+  const manifestPath = process.argv[2] || path.join(here, 'manifest.local.example.json')
+  const execMode = process.argv.includes('--exec')
+  const agentCommandOverride = (() => {
+    const idx = process.argv.indexOf('--agent-command')
+    return idx !== -1 ? process.argv[idx + 1] : null
+  })()
+  const resultsDir = path.resolve(process.env.OMNIBENCH_RESULTS || path.join(here, 'results'))
+
+  const list = readManifests(manifestPath)
+  const errors = validateManifests(list)
+  if (errors.length) {
+    console.error('Invalid manifest:')
+    for (const e of errors) console.error(`- ${e}`)
+    process.exit(1)
+  }
+  const promptCount = generatePrompts(list)
+
+  let totalRuns = 0
+  for (const m of list) totalRuns += (m.runs || 1) * 2
+  console.log(`Generated ${promptCount} DSH prompts in ${path.join(here, 'prompts')}`)
+  console.log(`OmniBench v2 run plan (${totalRuns} agent runs)`)
+
+  if (!execMode) {
+    for (const m of list) {
+      console.log(`- ${m.id}: ${m.repo} @ ${m.commit} (${m.language}/${m.framework}) runs=${m.runs} -> raw ${m.runs} + omni ${m.runs}`)
+    }
+    console.log('Pass --exec and configure agentCommand to execute runs locally.')
+    return
+  }
+
+  const results = []
+  for (const m of list) {
+    const repoDir = ensureRepo(m)
+    for (const arm of ['raw', 'omni']) {
+      for (let i = 1; i <= (m.runs || 3); i++) {
+        const agentCommand = agentCommandOverride || m.agentCommand
+        const promptFile = path.join(here, 'prompts', `${m.id}-${arm}-${i}.txt`)
+        const prompt = fs.readFileSync(promptFile, 'utf8')
+        console.log(`\n=== ${m.id} ${arm} run ${i} ===`)
+        if (m.setupCommand) {
+          const setup = runCommand(m.setupCommand, repoDir, 'setup')
+          console.log(`setup exit=${setup.exitCode ?? 'skip'}`)
+        }
+        const baseline = m.baselineCommand ? runCommand(m.baselineCommand, repoDir, 'baseline') : null
+        if (baseline) console.log(`baseline exit=${baseline.exitCode ?? 'skip'}`)
+
+        let agent = null
+        if (agentCommand) {
+          const command = `${agentCommand} ${JSON.stringify(prompt)}`
+          agent = runCommand(command, repoDir, 'agent')
+          console.log(`agent exit=${agent.exitCode ?? 'skip'} duration=${agent.durationMs ?? 0}ms`)
+        } else {
+          console.log(`agentCommand not configured; run prompt manually in DSH Desktop:\n  ${promptFile}`)
+        }
+
+        const verify = m.verifyCommand ? runCommand(m.verifyCommand, repoDir, 'verify') : null
+        if (verify) console.log(`verify exit=${verify.exitCode ?? 'skip'}`)
+
+        let success = null
+        if (verify && verify.skipped !== true) {
+          success = verify.exitCode === 0
+        } else if (agent && agent.skipped !== true) {
+          success = agent.exitCode === 0 && /BENCHMARK:\s*PASS/i.test(agent.output || '')
+        }
+
+        results.push({
+          id: m.id,
+          arm,
+          run: i,
+          repo: m.repo,
+          commit: m.commit,
+          task: m.task,
+          success,
+          agentExitCode: agent?.exitCode ?? null,
+          verifyExitCode: verify?.exitCode ?? null,
+          durationMs: (agent?.durationMs || 0) + (verify?.durationMs || 0),
+          baselineExitCode: baseline?.exitCode ?? null,
+          agentOutput: agent?.output?.slice(0, 4000) || '',
+          verifyOutput: verify?.output?.slice(0, 4000) || '',
+          metrics: {
+            tokens: 0,
+            cost: 0,
+            toolCalls: 0,
+          },
+          promptFile,
+          ranAt: new Date().toISOString(),
+        })
+      }
+    }
+  }
+
+  const file = writeResults(results, resultsDir)
+  const executed = results.filter((r) => r.success !== null).length
+  const passed = results.filter((r) => r.success === true).length
+  console.log(`\nWrote ${results.length} result record(s) to ${file}`)
+  console.log(`Executed ${executed}/${results.length} runs; passed ${passed}.`)
 }
 
-main()
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
+  main()
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dsh-omni-router",
-  "version": "2.0.0",
+  "version": "2.2.0",
   "description": "Omni - a reliability and orchestration control plane for DeepSeek Harness, designed to turn fast models into dependable long-horizon software-engineering agents.",
   "type": "module",
   "main": "./lib/index.js",
@@ -41,7 +41,7 @@
     }
   },
   "scripts": {
-    "test": "node test/omni-router.test.mjs && node test/agent-chain.test.mjs && node test/skill-suggest.test.mjs && node test/methodology.test.mjs && node test/compat.test.mjs && node test/project-brain.test.mjs && node test/mission-planner.test.mjs && node test/agent-runtime.test.mjs && node test/judge.test.mjs && node test/memory.test.mjs && node test/memory-v3.test.mjs && node test/memory-engine.test.mjs && node test/mission-resume.test.mjs && node test/engineering-benchmark.test.mjs && node test/benchmark-runner.test.mjs && node test/benchmark-results.test.mjs && node test/continuous-eval.test.mjs && node --no-warnings test/project-brain-v2.test.mjs && node --no-warnings test/project-brain-v3.test.mjs && node test/visual-qa.test.mjs && node test/evidence.test.mjs && node test/evidence-store.test.mjs && node test/evidence-engine.test.mjs && node test/task-decision.test.mjs && node test/task-compiler.test.mjs && node test/capability-brain.test.mjs && node test/capability-manifest.test.mjs && node test/capability-sandbox.test.mjs && node test/planner-dag.test.mjs && node test/dynamic-context.test.mjs && node test/mission-dag.test.mjs && node test/failure-taxonomy.test.mjs && node test/context-expansion.test.mjs && node test/hybrid-retrieval.test.mjs && node test/dependency-graph.test.mjs && node test/ast-provider.test.mjs && node test/experience-router.test.mjs && node test/benchmark-tasks.test.mjs && node test/bundle-installer.test.mjs",
+    "test": "node test/omni-router.test.mjs && node test/agent-chain.test.mjs && node test/skill-suggest.test.mjs && node test/methodology.test.mjs && node test/compat.test.mjs && node test/project-brain.test.mjs && node test/mission-planner.test.mjs && node test/agent-runtime.test.mjs && node test/judge.test.mjs && node test/memory.test.mjs && node test/memory-v3.test.mjs && node test/memory-engine.test.mjs && node test/mission-resume.test.mjs && node test/engineering-benchmark.test.mjs && node test/benchmark-runner.test.mjs && node test/benchmark-results.test.mjs && node test/continuous-eval.test.mjs && node --no-warnings test/project-brain-v2.test.mjs && node --no-warnings test/project-brain-v3.test.mjs && node test/visual-qa.test.mjs && node test/evidence.test.mjs && node test/evidence-store.test.mjs && node test/evidence-engine.test.mjs && node test/task-decision.test.mjs && node test/task-compiler.test.mjs && node test/capability-brain.test.mjs && node test/capability-manifest.test.mjs && node test/capability-sandbox.test.mjs && node test/capability-auditor.test.mjs && node test/capability-quality.test.mjs && node test/capability-solver.test.mjs && node test/capability-provisioner.test.mjs && node test/capability-performance.test.mjs && node test/planner-dag.test.mjs && node test/dynamic-context.test.mjs && node test/mission-dag.test.mjs && node test/failure-taxonomy.test.mjs && node test/context-expansion.test.mjs && node test/hybrid-retrieval.test.mjs && node test/dependency-graph.test.mjs && node test/ast-provider.test.mjs && node test/experience-router.test.mjs && node test/benchmark-tasks.test.mjs && node test/bundle-installer.test.mjs && node test/omnibench-runner.test.mjs",
     "benchmark": "node benchmark/run.mjs",
     "benchmark:engineering": "node benchmark/engineering-run.mjs",
     "benchmark:compare": "node benchmark/compare.mjs",
@@ -50,6 +50,7 @@
     "omnibench:v2:plan": "node benchmark/omnibench-v2/runner-stub.mjs",
     "omnibench:v2:prepare": "node benchmark/omnibench-v2/prepare.mjs",
     "omnibench:v2:generate": "node benchmark/omnibench-v2/run.mjs",
+    "omnibench:v2:run": "node benchmark/omnibench-v2/run.mjs --exec",
     "analyze": "node benchmark/analyze.mjs",
     "llm-eval": "node benchmark/llm-eval.mjs",
     "install:preset": "node scripts/install-preset.mjs"

package/src/agent-runtime.mjs

@@ -165,6 +165,7 @@ export async function runDagLoop(dag, {
   maxRepairs = 0,
   maxSameActionRetries = 0,
   budget = {},
+  onProgress = null,
 } = {}) {
   const b = {
     maxSteps: maxSteps ?? budget.steps ?? 50,
@@ -231,6 +232,18 @@ export async function runDagLoop(dag, {
       actions.push({ taskId: task.id, observation })
     }
     step += 1
+    if (typeof onProgress === 'function') {
+      try {
+        await onProgress({
+          dag: current,
+          step,
+          actions,
+          metrics: { step, replanCount, repairCount, sameActionCount, tokenUsage, cost, toolCalls },
+        })
+      } catch {
+        // Progress callbacks are best-effort; they must never abort the mission.
+      }
+    }
   }
 
   const done = isMissionDagComplete(current)

package/src/capability-auditor.mjs

@@ -0,0 +1,90 @@
+/**
+ * Capability Auditor.
+ *
+ * Answers "what does the current Harness environment lack?" for both a
+ * baseline (first-run / after DSH update) and a specific task. It is the
+ * front door of the Capability Auto-Provisioning loop.
+ */
+
+export const OMNI_CODING_BASELINE = [
+  'repository.read',
+  'repository.search',
+  'source.write',
+  'shell',
+  'test.run',
+  'debugging',
+  'verification',
+  'code.review',
+  'git.workflow',
+]
+
+const SEVERITY = {
+  'repository.read': 'high',
+  'repository.search': 'high',
+  'source.write': 'high',
+  'shell': 'high',
+  'test.run': 'high',
+  'debugging': 'high',
+  'verification': 'high',
+  'code.review': 'medium',
+  'git.workflow': 'medium',
+  'browser.navigation': 'medium',
+  'browser.interaction': 'medium',
+  'browser.screenshot': 'medium',
+  'frontend.validation': 'medium',
+  'github.remote': 'medium',
+  'gitlab.remote': 'medium',
+  'db.inspection': 'low',
+  'container.run': 'low',
+  'security.review': 'medium',
+  'performance.analysis': 'low',
+}
+
+export function severityForCapability(capability) {
+  return SEVERITY[capability] || 'low'
+}
+
+export function auditCapabilities(brain, required = []) {
+  const availableSet = new Set()
+  for (const provider of brain?.capabilities || []) {
+    for (const cap of provider.capabilities || []) availableSet.add(cap)
+  }
+  const requiredList = [...new Set(required.map((c) => String(c).trim()).filter(Boolean))]
+  const available = requiredList.filter((c) => availableSet.has(c))
+  const missing = requiredList.filter((c) => !availableSet.has(c))
+  return {
+    required: requiredList,
+    available,
+    missing,
+    coverage: requiredList.length ? Math.round((available.length / requiredList.length) * 1000) / 1000 : 1,
+    gaps: missing.map((capability) => ({
+      capability,
+      severity: severityForCapability(capability),
+    })),
+  }
+}
+
+export function baselineAudit(brain, baseline = OMNI_CODING_BASELINE) {
+  return auditCapabilities(brain, baseline)
+}
+
+export function taskTimeAudit(brain, requirements = []) {
+  return auditCapabilities(brain, requirements)
+}
+
+export function formatCapabilityAudit(audit = {}) {
+  const lines = [
+    `Capability audit: ${audit.available?.length || 0}/${audit.required?.length || 0} available (coverage ${audit.coverage ?? 0})`,
+  ]
+  if (audit.required?.length) lines.push(`Required: ${audit.required.join(', ')}`)
+  if (audit.available?.length) lines.push(`Available: ${audit.available.join(', ')}`)
+  if (audit.missing?.length) {
+    lines.push(`Missing (${audit.missing.length}):`)
+    for (const gap of audit.gaps || []) {
+      lines.push(`- ${gap.capability} [${gap.severity}]`)
+    }
+  } else {
+    lines.push('No missing capabilities.')
+  }
+  return lines.join('\n')
+}

package/src/capability-performance.mjs

@@ -0,0 +1,103 @@
+/**
+ * Capability Performance Learning.
+ *
+ * Tracks whether a provisioned plugin/skill actually improves task success,
+ * reduces false completion, or only adds token/catalog overhead. This feeds
+ * demotion/removal decisions and benchmark plugin-contribution analysis.
+ */
+
+export function createPerformanceRegistry(initial = {}) {
+  return {
+    providers: { ...(initial.providers || {}) },
+  }
+}
+
+export function recordProvisionOutcome(registry, providerId, outcome = {}) {
+  const previous = registry.providers[providerId] || {
+    runs: 0,
+    successBefore: null,
+    successAfter: null,
+    falseCompletionBefore: null,
+    falseCompletionAfter: null,
+    tokensBefore: null,
+    tokensAfter: null,
+    toolErrorsBefore: null,
+    toolErrorsAfter: null,
+  }
+  const next = {
+    ...previous,
+    runs: (previous.runs || 0) + 1,
+    successBefore: outcome.successBefore ?? previous.successBefore,
+    successAfter: outcome.successAfter ?? previous.successAfter,
+    falseCompletionBefore: outcome.falseCompletionBefore ?? previous.falseCompletionBefore,
+    falseCompletionAfter: outcome.falseCompletionAfter ?? previous.falseCompletionAfter,
+    tokensBefore: outcome.tokensBefore ?? previous.tokensBefore,
+    tokensAfter: outcome.tokensAfter ?? previous.tokensAfter,
+    toolErrorsBefore: outcome.toolErrorsBefore ?? previous.toolErrorsBefore,
+    toolErrorsAfter: outcome.toolErrorsAfter ?? previous.toolErrorsAfter,
+    lastUpdated: new Date().toISOString(),
+  }
+  return {
+    ...registry,
+    providers: {
+      ...registry.providers,
+      [providerId]: next,
+    },
+  }
+}
+
+export function evaluateProviderValue(record = {}) {
+  const hasBefore = record.successBefore !== null && record.successBefore !== undefined
+  const hasAfter = record.successAfter !== null && record.successAfter !== undefined
+  if (!hasBefore || !hasAfter) return { value: 0, label: 'unknown' }
+
+  const successDelta = record.successAfter - record.successBefore
+  const falseCompletionDelta = (record.falseCompletionBefore ?? 0) - (record.falseCompletionAfter ?? 0)
+  const tokenRatio = record.tokensBefore ? Math.max(0, ((record.tokensAfter ?? 0) - record.tokensBefore) / record.tokensBefore) : 0
+  const toolErrorDelta = (record.toolErrorsAfter ?? 0) - (record.toolErrorsBefore ?? 0)
+
+  const value = (
+    successDelta * 0.5 +
+    falseCompletionDelta * 0.3 -
+    tokenRatio * 0.1 -
+    Math.max(0, toolErrorDelta) * 0.1
+  )
+  const rounded = Math.round(value * 1000) / 1000
+  let label = 'neutral'
+  if (rounded >= 0.05) label = 'high'
+  else if (rounded > 0) label = 'medium'
+  else if (rounded < 0) label = 'negative'
+  return { value: rounded, label, successDelta, falseCompletionDelta, tokenRatio, toolErrorDelta }
+}
+
+export function recommendDemotion(registry, providerId, options = {}) {
+  const record = registry.providers[providerId]
+  if (!record) return { providerId, recommendation: 'unknown', reason: 'no performance record' }
+  const value = evaluateProviderValue(record)
+  const usageDays = Number(options.usageDays ?? 0)
+  const uniqueCapabilities = Number(options.uniqueCapabilities ?? 0)
+  const coveredBy = options.coveredBy || []
+
+  if (value.label === 'negative' && usageDays >= 30 && uniqueCapabilities === 0) {
+    return { providerId, recommendation: 'remove', reason: 'negative value, no unique capability, low usage', value }
+  }
+  if (value.label === 'negative') {
+    return { providerId, recommendation: 'demote', reason: 'negative measured value', value }
+  }
+  if (usageDays >= 90 && uniqueCapabilities === 0 && coveredBy.length > 0) {
+    return { providerId, recommendation: 'remove', reason: 'redundant and unused for 90+ days', value }
+  }
+  if (value.label === 'high' || value.label === 'medium') {
+    return { providerId, recommendation: 'keep', reason: 'positive measured value', value }
+  }
+  return { providerId, recommendation: 'watch', reason: 'insufficient signal', value }
+}
+
+export function formatPerformanceRegistry(registry = {}) {
+  const lines = []
+  for (const [id, record] of Object.entries(registry.providers || {})) {
+    const value = evaluateProviderValue(record)
+    lines.push(`- ${id}: ${value.label} (${value.value}) runs=${record.runs || 0}`)
+  }
+  return lines.length ? lines.join('\n') : '(empty performance registry)'
+}