@minhpnq1807/contextos 0.6.0 → 0.6.2

package/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## 0.6.2
+
+- **Live agent leaderboard:** Added `ctx leaderboard --agents codex,gemini` and `npm run leaderboard:agents` to run the hallucination benchmark through installed Codex/Gemini CLIs with timeouts and skip/error reporting for missing or unauthenticated agents.
+- **Live GIF capture:** Added `npm run demo:capture` to regenerate the three launch GIFs from real local `ctx` command output across ContextOS and skill-routing fixture repos.
+- **Community skills sync:** Added `scripts/sync-community-skills.mjs`, `npm run sync:community-skills`, and a scheduled/manual GitHub Action that opens PRs from `khovan123/contextOS-skills` back into `community-skills/`.
+
+## 0.6.1
+
+- **Hallucination Leaderboard:** Added `ctx leaderboard --hallucination` and `npm run leaderboard:hallucination` to compare raw prompt-only skill guesses against ContextOS evidence-routed skill selection across 20 fixture tasks.
+
 ## 0.6.0
 
 - **Launch demo framing:** Added Agent Hallucination Benchmark messaging, same-prompt/same-model/different-context copy, and `docs/launch-demos.md` with three short demo scripts: hallucination benchmark, AGENTS.md lost-in-the-middle, and repo-aware skills.

package/README.md

@@ -28,7 +28,7 @@ Published package: [`@minhpnq1807/contextos`](https://www.npmjs.com/package/@min
 
 ## Demo
 
-![ContextOS demo: same prompt, different repo, correct skills](docs/demo/contextos-demo.gif)
+![ContextOS demo: same prompt, different repo, correct skills](docs/demo/same-prompt-different-context.gif)
 
 Same prompt. Same model. Different context.
 
@@ -42,6 +42,19 @@ ctx skills doctor -- "fix deployed"
 | `vercel.json`, `next`, GitHub workflow | `vercel-deployment`, `github-actions-ci-cd`, `env-secret-management` |
 | ContextOS repo with no app deploy evidence | no deployment skill selected |
 
+More 10-second demos:
+
+| Demo | GIF |
+| --- | --- |
+| AGENTS.md Lost In The Middle | [docs/demo/agents-lost-middle.gif](docs/demo/agents-lost-middle.gif) |
+| ContextOS Ready Gold | [docs/demo/contextos-ready.gif](docs/demo/contextos-ready.gif) |
+
+Regenerate the GIFs from real local `ctx` command output:
+
+```bash
+npm run demo:capture
+```
+
 ## Agent Hallucination Benchmark
 
 Generic agents often guess deployment tooling from the prompt alone:
@@ -80,6 +93,19 @@ Skill Router internal fixture benchmark:
 
 This is an internal fixture benchmark, not an external real-world benchmark. It is designed to prove the router behavior across controlled Expo/EAS, Next/Vercel, Docker, Railway/Render, Firebase, auth, database, testing, mobile, and adversarial negative-gate cases.
 
+Hallucination leaderboard:
+
+```bash
+ctx leaderboard --hallucination
+```
+
+Current local result across 20 fixture tasks and 12 repo contexts:
+
+| System | Correct Skill |
+| --- | ---: |
+| Raw Agent | 10.0% |
+| ContextOS + Codex | 80.0% |
+
 Example hook context injected before the agent works:
 
 ```text
@@ -246,6 +272,8 @@ The score checks project `AGENTS.md` rules, project skill packs under `.codex/sk
 | `ctx stats` | Show workspace-level usage and effectiveness metrics. |
 | `ctx benchmark -- "task"` | Compare raw AGENTS.md ordering vs ContextOS scheduling. |
 | `ctx benchmark --skills` | Run the Skill Router eval benchmark. |
+| `ctx leaderboard --hallucination` | Compare raw prompt-only guesses vs ContextOS routing. |
+| `ctx leaderboard --agents codex,gemini` | Run the live CLI leaderboard when Codex/Gemini credentials are available. |
 | `ctx sync --rules` | Sync AGENTS/Ruler/MCP config across agents. |
 | `ctx sync --skills` | Sync skills across agents through skillshare. |
 | `ctx sync --workflows` | Sync workflow markdown across Claude/Codex/Antigravity. |
@@ -570,6 +598,8 @@ This warning comes from a transitive dependency in the local embedding/WASM stac
 | `ctx stats` | Shows aggregate runtime metrics for the current workspace. | You want to know whether ContextOS is active and useful over time. | Prints sectioned tables for prompt/report counts, injection rate, efficiency, rule outcomes, hook events, last prompt, and last report. |
 | `ctx benchmark -- "task"` | Compares baseline AGENTS.md ordering with ContextOS task-aware scheduling. | You want a before/after signal for lost-in-the-middle risk. | Prints tables for parsed/actionable/filtered rules, baseline middle-risk, scheduled high/mid rules, recency reminder status, and top scored rules. |
 | `ctx benchmark --skills` | Runs the Skill Router eval benchmark. | You want evidence for skill routing accuracy and negative gates. | Prints top-1 accuracy, top-3 recall, false positive rate, confidence calibration, and negative gate accuracy across `eval/skill-routing` fixtures. |
+| `ctx leaderboard --hallucination` | Compares raw prompt-only skill guesses with ContextOS evidence routing. | You want launch evidence for the hallucination problem. | Runs 20 fixture tasks across 10+ repo contexts and prints Raw Agent vs ContextOS correctness plus sample failures. |
+| `ctx leaderboard --agents codex,gemini` | Runs the same benchmark shape through installed agent CLIs. | You want real agent output instead of the deterministic raw baseline. | Calls `codex exec` in read-only mode and the local Gemini CLI with timeouts; missing or unauthenticated CLIs are reported as skipped/errors instead of blocking. |
 | `ctx sync --rules` | Syncs project rules and MCP servers through Ruler. | You want Codex, Claude Code, and Antigravity to share one project rule/MCP source of truth. | Ensures `.ruler/ruler.toml`, injects `ctx-mcp`, imports existing MCP servers from Codex and project `.mcp.json`, runs `ruler apply --agents codex,claude,antigravity`, mirrors MCP servers to Antigravity MCP configs, and verifies generated config. |
 | `ctx sync --rules --agents <list>` | Syncs only selected agents through Ruler. | You want to update one or two agents without touching the others. | Accepts comma-separated values such as `codex`, `claude`, `agy`, `antigravity`, or `codex,claude,agy`; `agy` is normalized to Ruler's `antigravity`. |
 | `ctx sync --rules --dry-run` | Previews Ruler sync without writing files or running apply. | You want to inspect behavior before changing project config. | Prints the same flow with dry-run status. |

package/bin/ctx.js

@@ -20,6 +20,8 @@ import { defaultDataRoot, workspaceDataDir, workspaceMarkerPath } from "../plugi
 import { installMcpTelemetryProxies } from "../plugins/ctx/lib/mcp-proxy-install.js";
 import { benchmarkWorkspace, formatBenchmark } from "../plugins/ctx/lib/benchmark.js";
 import { formatSkillRoutingBenchmark, runSkillRoutingEval } from "../eval/skill-routing/run-eval.js";
+import { formatHallucinationLeaderboard, runHallucinationLeaderboard } from "../eval/hallucination/run-leaderboard.js";
+import { formatAgentLeaderboard, runAgentLeaderboard } from "../eval/hallucination/run-agent-leaderboard.js";
 import { copyDir, copyPackageRoot, syncPackageRoot } from "../plugins/ctx/lib/package-install.js";
 import { installClaudeHooks } from "../plugins/ctx/lib/claude-hooks.js";
 import { installClaudeMcp } from "../plugins/ctx/lib/claude-mcp.js";
@@ -197,6 +199,8 @@ Usage:
   ctx stats                                         Show workspace statistics
   ctx benchmark -- "task"                           Benchmark workspace for a task
   ctx benchmark --skills                            Run skill routing eval benchmark
+  ctx leaderboard --hallucination                   Compare raw agent guesses vs ContextOS routing
+  ctx leaderboard --agents codex,gemini             Run live CLI leaderboard for installed agents
   ctx sync --rules                                  Sync AGENTS.md rules to all agents
   ctx sync --rules --agents <names>                 Sync rules to specific agents only
   ctx sync --rules --dry-run                        Preview rule sync without writing
@@ -1034,6 +1038,23 @@ try {
     if (!task.trim()) throw new Error('Usage: ctx benchmark -- "task"');
     console.log(formatBenchmark(benchmarkWorkspace({ cwd: process.cwd(), task })));
     }
+  } else if (command === "leaderboard") {
+    if (args.includes("--hallucination")) {
+      console.log(formatHallucinationLeaderboard(await runHallucinationLeaderboard({ rootDir })));
+    } else if (args.includes("--agents")) {
+      const index = args.indexOf("--agents");
+      const agents = String(args[index + 1] || "").split(",").map((agent) => agent.trim()).filter(Boolean);
+      const limitIndex = args.indexOf("--limit");
+      const timeoutIndex = args.indexOf("--timeout-ms");
+      console.log(formatAgentLeaderboard(runAgentLeaderboard({
+        rootDir,
+        agents: agents.length ? agents : undefined,
+        caseLimit: limitIndex >= 0 ? Number(args[limitIndex + 1]) : undefined,
+        timeoutMs: timeoutIndex >= 0 ? Number(args[timeoutIndex + 1]) : undefined
+      })));
+    } else {
+      throw new Error("Usage: ctx leaderboard --hallucination OR ctx leaderboard --agents codex,gemini");
+    }
   } else if (command === "skills") {
     if (args[1] === "doctor") {
       const marker = args.indexOf("--");

package/docs/demo/agents-lost-middle.txt

@@ -0,0 +1,25 @@
+$ ctx benchmark -- "fix failing test"
+ContextOS benchmark
+
+Summary
+-------
+Metric                Value
+--------------------  -------------------------
+Task                  fix failing test
+Rules parsed          7
+Actionable rules      5
+Filtered rules        2
+Relevant rules        1
+Baseline middle-risk  1/1 relevant rules (100%)
+ContextOS scheduled   1 high, 0 mid
+Recency reminder      enabled
+
+Top Rules
+---------
+Score  Rule                                                                                      Reasons
+-----  ----------------------------------------------------------------------------------------  ----------
+0.50   IMPORTANT: This project has a knowledge graph. ALWAYS use code-review-graph MCP tools...  imperative
+0.00   AGENTS.md
+0.00   Centralised AI agent instructions. Add coding guidelines, style guides, and project c...
+0.00   Ruler concatenates all .md files in this directory (and subdirectories), starting wit...
+0.00   Use `query_graph` pattern="tests_for" to check coverage.

package/docs/demo/capture-live-demos.mjs

@@ -0,0 +1,76 @@
+#!/usr/bin/env node
+import { execFileSync } from "node:child_process";
+import fs from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { benchmarkWorkspace, formatBenchmark } from "../../plugins/ctx/lib/benchmark.js";
+import { formatContextOSReady, inspectContextOSReady } from "../../plugins/ctx/lib/certification.js";
+import { formatHallucinationLeaderboard, runHallucinationLeaderboard } from "../../eval/hallucination/run-leaderboard.js";
+import { runSkillRoutingEval } from "../../eval/skill-routing/run-eval.js";
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const repoRoot = path.resolve(__dirname, "..", "..");
+const render = path.join(__dirname, "render-terminal-gif.mjs");
+const leaderboard = await runHallucinationLeaderboard({ rootDir: repoRoot });
+const skillEval = await runSkillRoutingEval({ rootDir: repoRoot });
+
+const demos = [
+  {
+    log: "same-prompt-different-context.txt",
+    gif: "same-prompt-different-context.gif",
+    steps: [
+      ["ctx leaderboard --hallucination", formatHallucinationLeaderboard(leaderboard)],
+      ["ctx skills doctor -- \"fix deployed\"  # Expo fixture", routeSummary("expo-eas", "fix deployed")],
+      ["ctx skills doctor -- \"fix deployed\"  # Next/Vercel fixture", routeSummary("next-vercel", "fix deployed")]
+    ]
+  },
+  {
+    log: "agents-lost-middle.txt",
+    gif: "agents-lost-middle.gif",
+    steps: [
+      ["ctx benchmark -- \"fix failing test\"", formatBenchmark(benchmarkWorkspace({ cwd: repoRoot, task: "fix failing test" }))]
+    ]
+  },
+  {
+    log: "contextos-ready.txt",
+    gif: "contextos-ready.gif",
+    steps: [
+      ["ctx doctor", formatContextOSReady(inspectContextOSReady({ cwd: repoRoot }))]
+    ]
+  }
+];
+
+for (const demo of demos) {
+  const logPath = path.join(__dirname, demo.log);
+  const gifPath = path.join(__dirname, demo.gif);
+  const chunks = [];
+  for (const [label, output] of demo.steps) {
+    chunks.push(`$ ${label}`);
+    chunks.push(cleanOutput(output));
+    chunks.push("");
+  }
+  fs.writeFileSync(logPath, chunks.join("\n").trimEnd() + "\n", "utf8");
+  execFileSync(process.execPath, [render, logPath, gifPath], { cwd: repoRoot, stdio: "inherit" });
+}
+
+function cleanOutput(output) {
+  return String(output || "")
+    .split(/\r?\n/)
+    .map((line) => line.trimEnd())
+    .join("\n")
+    .trim();
+}
+
+function routeSummary(fixture, prompt) {
+  const row = skillEval.rows.find((item) => item.fixture === fixture && item.prompt === prompt);
+  if (!row) return "No route found.";
+  return [
+    "ContextOS skill doctor",
+    `fixture: ${fixture}`,
+    `prompt: ${prompt}`,
+    `selected: ${row.selectedIds.join(", ") || "(none)"}`,
+    `expected: ${row.expected.join(", ") || "(none)"}`,
+    `rejected: ${row.forbidden.join(", ") || "(none)"}`
+  ].join("\n");
+}

package/docs/demo/contextos-ready.txt

@@ -0,0 +1,14 @@
+$ ctx doctor
+Repository Score
+
+Rules: 100
+Skills: 100
+Workflows: 100
+
+Overall:
+ContextOS Ready Gold
+
+Evidence:
+- Rules: 1 AGENTS.md source(s), 5 actionable rule(s)
+- Skills: 3 skill(s), 3 metadata file(s)
+- Workflows: 2 workflow(s), 2 with agent chain(s)

package/docs/demo/render-terminal-gif.mjs

@@ -35,7 +35,7 @@ for (let count = 1; count <= displayLines.length; count += frameStep) {
 frames.push(writeFrame({ tmpDir, index: frames.length, lines: displayLines }));
 
 fs.mkdirSync(path.dirname(outputPath), { recursive: true });
-execFileSync("convert", ["-delay", "12", "-loop", "0", ...frames, outputPath], { stdio: "inherit" });
+execFileSync("convert", ["-limit", "time", "120", "-delay", "12", "-loop", "0", ...frames, outputPath], { stdio: "inherit" });
 console.log(`Wrote ${outputPath}`);
 
 function writeFrame({ tmpDir, index, lines }) {

package/docs/demo/same-prompt-different-context.txt

@@ -0,0 +1,51 @@
+$ ctx leaderboard --hallucination
+Hallucination Leaderboard
+Repos: 12
+Tasks: 20
+
+System              Correct Skill
+------------------  -------------
+Raw Agent           10.0%
+ContextOS + Codex   80.0%
+
+Sample failures:
+- expo-eas: "fix deployed"
+  expected: eas, mobile-deployment, github-actions-ci-cd
+  raw: eas, env-secret-management, railway-render-deployment ✗
+  contextos: eas, github-actions-ci-cd, mobile-deployment ✓
+- next-vercel: "fix deployed"
+  expected: vercel-deployment, github-actions-ci-cd, env-secret-management
+  raw: eas, env-secret-management, railway-render-deployment ✗
+  contextos: vercel-deployment, github-actions-ci-cd, env-secret-management ✓
+- docker-node: "docker image build failed"
+  expected: docker, build-log-debugging
+  raw: docker, build-log-debugging, github-actions-ci-cd ✗
+  contextos: build-log-debugging, docker ✓
+- railway-render: "Railway deploy health check failed"
+  expected: railway-render-deployment, build-log-debugging
+  raw: railway-render-deployment, build-log-debugging, firebase-hosting ✗
+  contextos: build-log-debugging, railway-render-deployment ✓
+- firebase-hosting: "deploy firebase hosting"
+  expected: firebase-hosting
+  raw: firebase-hosting, flutter-firebase, railway-render-deployment ✗
+  contextos: firebase-hosting ✓
+- nest-prisma: "optimize slow prisma queries"
+  expected: prisma, nestjs-module
+  raw: prisma, nestjs-module, android-signing ✗
+  contextos: nestjs-module, prisma ✓
+
+$ ctx skills doctor -- "fix deployed"  # Expo fixture
+ContextOS skill doctor
+fixture: expo-eas
+prompt: fix deployed
+selected: eas, github-actions-ci-cd, mobile-deployment
+expected: eas, mobile-deployment, github-actions-ci-cd
+rejected: vercel-deployment
+
+$ ctx skills doctor -- "fix deployed"  # Next/Vercel fixture
+ContextOS skill doctor
+fixture: next-vercel
+prompt: fix deployed
+selected: vercel-deployment, github-actions-ci-cd, env-secret-management
+expected: vercel-deployment, github-actions-ci-cd, env-secret-management
+rejected: eas

package/docs/launch-demos.md

@@ -4,6 +4,8 @@ These are demo scripts for explaining ContextOS quickly. They are intentionally
 
 ## 1. Agent Hallucination Benchmark
 
+GIF: [`docs/demo/same-prompt-different-context.gif`](demo/same-prompt-different-context.gif)
+
 Prompt:
 
 ```text
@@ -39,6 +41,8 @@ Same prompt. Same model. Different context.
 
 ## 2. AGENTS.md Lost In The Middle
 
+GIF: [`docs/demo/agents-lost-middle.gif`](demo/agents-lost-middle.gif)
+
 Setup:
 
 ```text
@@ -71,6 +75,8 @@ Important repo rules should not depend on where they appear in a long file.
 
 ## 3. Repo-Aware Skills
 
+GIF: [`docs/demo/same-prompt-different-context.gif`](demo/same-prompt-different-context.gif)
+
 Prompt:
 
 ```text
@@ -103,3 +109,19 @@ Message:
 ```text
 Context is not extra text. It changes the correct answer.
 ```
+
+## 4. ContextOS Ready
+
+GIF: [`docs/demo/contextos-ready.gif`](demo/contextos-ready.gif)
+
+Command:
+
+```bash
+ctx doctor
+```
+
+Message:
+
+```text
+Repos now have a target: AGENTS.md + skills + workflows + evidence.
+```

package/eval/hallucination/run-agent-leaderboard.js

@@ -0,0 +1,233 @@
+#!/usr/bin/env node
+import { execFileSync, spawnSync } from "node:child_process";
+import fs from "node:fs";
+import os from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { parseEvalYaml } from "../skill-routing/run-eval.js";
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const repoRoot = path.resolve(__dirname, "..", "..");
+const skillEvalRoot = path.resolve(__dirname, "..", "skill-routing");
+const DEFAULT_AGENTS = ["codex", "gemini"];
+const DEFAULT_CASE_LIMIT = 5;
+const DEFAULT_TIMEOUT_MS = 120000;
+
+export function runAgentLeaderboard({
+  agents = DEFAULT_AGENTS,
+  casesPath = path.join(skillEvalRoot, "cases.yaml"),
+  caseLimit = DEFAULT_CASE_LIMIT,
+  timeoutMs = DEFAULT_TIMEOUT_MS,
+  rootDir = repoRoot
+} = {}) {
+  const config = parseEvalYaml(fs.readFileSync(casesPath, "utf8"));
+  const cases = config.cases
+    .filter((row) => row.expected?.length)
+    .slice(0, caseLimit);
+  const skillIds = config.skills.map((skill) => skill.id);
+  const systems = [];
+
+  for (const agent of agents) {
+    const binary = findBinary(agent);
+    if (!binary) {
+      systems.push({ name: agent, status: "skipped", reason: "binary not found", rows: [], correctRate: 0 });
+      continue;
+    }
+    const rows = cases.map((testCase) => runAgentCase({
+      agent,
+      binary,
+      testCase,
+      skillIds,
+      timeoutMs,
+      rootDir
+    }));
+    const completed = rows.filter((row) => row.status === "ok");
+    const correct = completed.filter((row) => row.correct).length;
+    systems.push({
+      name: agent,
+      status: completed.length ? "ok" : "skipped",
+      reason: completed.length ? "" : firstReason(rows),
+      rows,
+      correctRate: completed.length ? correct / completed.length : 0
+    });
+  }
+
+  return {
+    mode: "live-agent-cli",
+    caseCount: cases.length,
+    systems
+  };
+}
+
+export function formatAgentLeaderboard(result) {
+  const lines = [
+    "Live Agent Leaderboard",
+    `Mode: ${result.mode}`,
+    `Tasks: ${result.caseCount}`,
+    "",
+    "System    Status    Correct Skill",
+    "--------  --------  -------------"
+  ];
+  for (const system of result.systems) {
+    const score = system.status === "ok" ? percent(system.correctRate) : system.reason;
+    lines.push(`${system.name.padEnd(8)}  ${system.status.padEnd(8)}  ${score}`);
+  }
+  lines.push("", "Cases:");
+  for (const system of result.systems) {
+    lines.push(`- ${system.name}`);
+    for (const row of system.rows.slice(0, 5)) {
+      lines.push(`  - ${row.fixture}: "${row.prompt}"`);
+      lines.push(`    selected: ${row.selectedIds.join(", ") || "(none)"} ${row.correct ? "✓" : "✗"}`);
+      if (row.status !== "ok") lines.push(`    status: ${row.status}; ${row.reason}`);
+    }
+  }
+  return lines.join("\n");
+}
+
+function runAgentCase({ agent, binary, testCase, skillIds, timeoutMs, rootDir }) {
+  const cwd = testCase.fixture === "contextos"
+    ? rootDir
+    : path.join(skillEvalRoot, "fixtures", testCase.fixture);
+  const prompt = buildPrompt({ task: testCase.prompt, skillIds });
+  const startedAt = Date.now();
+  const result = spawnSync(binary, agentArgs({ agent, cwd, prompt }), {
+    cwd,
+    encoding: "utf8",
+    timeout: timeoutMs,
+    maxBuffer: 1024 * 1024 * 4,
+    env: {
+      ...process.env,
+      NO_COLOR: "1",
+      CI: process.env.CI || "1"
+    }
+  });
+  const output = `${result.stdout || ""}\n${result.stderr || ""}`.trim();
+  const selectedIds = parseSkillIds(output, skillIds).slice(0, 3);
+  const correct = isCorrect({
+    selectedIds,
+    expected: testCase.expected || [],
+    allowed: testCase.allowed || [],
+    forbidden: testCase.forbidden || []
+  });
+  return {
+    status: result.error ? "error" : result.status === 0 ? "ok" : "error",
+    reason: result.error?.message || (result.status === 0 ? "" : `exit ${result.status}`),
+    prompt: testCase.prompt,
+    fixture: testCase.fixture,
+    expected: testCase.expected || [],
+    selectedIds,
+    correct,
+    durationMs: Date.now() - startedAt
+  };
+}
+
+function agentArgs({ agent, cwd, prompt }) {
+  if (agent === "codex") {
+    return [
+      "exec",
+      "--cd", cwd,
+      "--sandbox", "read-only",
+      "--ask-for-approval", "never",
+      prompt
+    ];
+  }
+  if (agent === "gemini") {
+    const template = process.env.CONTEXTOS_GEMINI_CMD;
+    if (template) return expandTemplate(template, { cwd, prompt });
+    return ["-p", prompt];
+  }
+  return [prompt];
+}
+
+function buildPrompt({ task, skillIds }) {
+  return [
+    "You are evaluating a repository for a coding-agent skill router benchmark.",
+    "Do not edit files. Do not run commands.",
+    `Task: ${task}`,
+    `Allowed skill IDs: ${skillIds.join(", ")}`,
+    "Return only the top skill IDs as comma-separated text. No explanations."
+  ].join("\n");
+}
+
+function parseSkillIds(output, skillIds) {
+  const normalized = new Map(skillIds.map((id) => [normalize(id), id]));
+  const found = [];
+  for (const token of String(output || "").split(/[^A-Za-z0-9_.@-]+/)) {
+    const id = normalized.get(normalize(token));
+    if (id && !found.includes(id)) found.push(id);
+  }
+  return found;
+}
+
+function isCorrect({ selectedIds, expected, allowed, forbidden }) {
+  const selected = new Set(selectedIds);
+  const accepted = new Set([...expected, ...allowed]);
+  return expected.every((skill) => selected.has(skill))
+    && forbidden.every((skill) => !selected.has(skill))
+    && selectedIds.every((skill) => accepted.has(skill));
+}
+
+function findBinary(name) {
+  const safeName = shellQuote(name);
+  const candidates = [
+    path.join(os.homedir(), ".local", "bin", safeName),
+    path.join(os.homedir(), ".npm-global", "bin", safeName),
+    path.join(os.homedir(), ".nvm", "current", "bin", safeName),
+    `/mnt/c/Users/admin/AppData/Roaming/npm/${safeName}`,
+    `/mnt/c/Users/admin/AppData/Roaming/npm/${safeName}.cmd`
+  ];
+  for (const candidate of candidates) {
+    if (fs.existsSync(candidate)) return candidate;
+  }
+  for (const command of [
+    `command -v ${safeName}`,
+    `source ~/.profile >/dev/null 2>&1 || true; source ~/.bashrc >/dev/null 2>&1 || true; command -v ${safeName}`
+  ]) {
+    try {
+      const found = execFileSync("bash", ["-lc", command], { encoding: "utf8" }).trim();
+      if (found) return found;
+    } catch {
+      // continue
+    }
+  }
+  return "";
+}
+
+function firstReason(rows) {
+  return rows.find((row) => row.reason)?.reason || "no completed cases";
+}
+
+function normalize(value) {
+  return String(value || "").toLowerCase().replace(/[^a-z0-9]+/g, "");
+}
+
+function shellQuote(value) {
+  return String(value).replace(/[^A-Za-z0-9_./-]/g, "");
+}
+
+function expandTemplate(template, vars) {
+  const file = path.join(os.tmpdir(), `contextos-agent-prompt-${process.pid}-${Date.now()}.txt`);
+  fs.writeFileSync(file, vars.prompt, "utf8");
+  return String(template)
+    .replaceAll("{cwd}", vars.cwd)
+    .replaceAll("{prompt_file}", file)
+    .split(/\s+/)
+    .filter(Boolean);
+}
+
+function percent(value) {
+  return `${(value * 100).toFixed(1)}%`;
+}
+
+if (import.meta.url === `file://${process.argv[1]}`) {
+  const agentsArg = process.argv.find((arg) => arg.startsWith("--agents="));
+  const limitArg = process.argv.find((arg) => arg.startsWith("--limit="));
+  const timeoutArg = process.argv.find((arg) => arg.startsWith("--timeout-ms="));
+  const result = runAgentLeaderboard({
+    agents: agentsArg ? agentsArg.slice("--agents=".length).split(",").map((item) => item.trim()).filter(Boolean) : DEFAULT_AGENTS,
+    caseLimit: limitArg ? Number(limitArg.slice("--limit=".length)) : DEFAULT_CASE_LIMIT,
+    timeoutMs: timeoutArg ? Number(timeoutArg.slice("--timeout-ms=".length)) : DEFAULT_TIMEOUT_MS
+  });
+  console.log(formatAgentLeaderboard(result));
+}

package/eval/hallucination/run-leaderboard.js

@@ -0,0 +1,183 @@
+#!/usr/bin/env node
+import fs from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { parseEvalYaml, runSkillRoutingEval } from "../skill-routing/run-eval.js";
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const evalRoot = path.resolve(__dirname, "..", "skill-routing");
+const DEFAULT_CASE_LIMIT = 20;
+
+export async function runHallucinationLeaderboard({
+  rootDir = path.resolve(__dirname, "..", ".."),
+  casesPath = path.join(evalRoot, "cases.yaml"),
+  caseLimit = DEFAULT_CASE_LIMIT
+} = {}) {
+  const config = parseEvalYaml(fs.readFileSync(casesPath, "utf8"));
+  const selectedCases = selectLeaderboardCases(config.cases, caseLimit);
+  const wanted = new Set(selectedCases.map(caseId));
+  const contextos = await runSkillRoutingEval({ rootDir, casesPath, topK: 3, threshold: 0.5 });
+  const contextRows = contextos.rows
+    .filter((row) => wanted.has(caseId(row)))
+    .map((row) => evaluateRow({
+      prompt: row.prompt,
+      fixture: row.fixture,
+      expected: row.expected,
+      allowed: row.allowed,
+      forbidden: row.forbidden,
+      selectedIds: row.selectedIds
+    }));
+  const rawRows = selectedCases.map((testCase) => evaluateRow({
+    prompt: testCase.prompt,
+    fixture: testCase.fixture,
+    expected: testCase.expected || [],
+    allowed: testCase.allowed || [],
+    forbidden: testCase.forbidden || [],
+    selectedIds: rawAgentSkills({ prompt: testCase.prompt, skills: config.skills, topK: 3 })
+  }));
+
+  return {
+    caseCount: selectedCases.length,
+    repoCount: new Set(selectedCases.map((row) => row.fixture)).size,
+    systems: [
+      summarizeSystem("Raw Agent", rawRows),
+      summarizeSystem("ContextOS + Codex", contextRows)
+    ],
+    rows: selectedCases.map((testCase) => ({
+      prompt: testCase.prompt,
+      fixture: testCase.fixture,
+      expected: testCase.expected || [],
+      raw: rawRows.find((row) => row.prompt === testCase.prompt && row.fixture === testCase.fixture),
+      contextos: contextRows.find((row) => row.prompt === testCase.prompt && row.fixture === testCase.fixture)
+    }))
+  };
+}
+
+export function formatHallucinationLeaderboard(result) {
+  const lines = [
+    "Hallucination Leaderboard",
+    `Repos: ${result.repoCount}`,
+    `Tasks: ${result.caseCount}`,
+    "",
+    "System              Correct Skill",
+    "------------------  -------------"
+  ];
+  for (const system of result.systems) {
+    lines.push(`${system.name.padEnd(18)}  ${percent(system.correctRate)}`);
+  }
+  lines.push("", "Sample failures:");
+  const failures = result.rows
+    .filter((row) => !row.raw.correct || !row.contextos.correct)
+    .slice(0, 6);
+  if (!failures.length) {
+    lines.push("- none");
+  } else {
+    for (const row of failures) {
+      lines.push(`- ${row.fixture}: "${row.prompt}"`);
+      lines.push(`  expected: ${row.expected.join(", ") || "(none)"}`);
+      lines.push(`  raw: ${row.raw.selectedIds.join(", ") || "(none)"} ${row.raw.correct ? "✓" : "✗"}`);
+      lines.push(`  contextos: ${row.contextos.selectedIds.join(", ") || "(none)"} ${row.contextos.correct ? "✓" : "✗"}`);
+    }
+  }
+  return lines.join("\n");
+}
+
+function selectLeaderboardCases(cases, limit) {
+  const wantedFixtures = [
+    "expo-eas",
+    "next-vercel",
+    "docker-node",
+    "railway-render",
+    "firebase-hosting",
+    "nest-prisma",
+    "express-mongo-jwt",
+    "oauth-google",
+    "redis-cache",
+    "contextos",
+    "frontend-only-next",
+    "static-docs"
+  ];
+  const selected = [];
+  for (const fixture of wantedFixtures) {
+    const match = cases.find((row) => row.fixture === fixture && !selected.some((item) => caseId(item) === caseId(row)));
+    if (match) selected.push(match);
+    if (selected.length >= limit) return selected;
+  }
+  for (const row of cases) {
+    if (!selected.some((item) => caseId(item) === caseId(row))) selected.push(row);
+    if (selected.length >= limit) break;
+  }
+  return selected;
+}
+
+function rawAgentSkills({ prompt, skills, topK }) {
+  return skills
+    .map((skill) => ({ id: skill.id, score: rawPromptScore(prompt, skill) }))
+    .sort((a, b) => b.score - a.score || a.id.localeCompare(b.id))
+    .slice(0, topK)
+    .map((skill) => skill.id);
+}
+
+function rawPromptScore(prompt, skill) {
+  const promptTokens = new Set(tokenize(prompt));
+  const triggerTokens = tokenize([
+    skill.id,
+    skill.description,
+    ...(skill.positive_triggers?.prompts || [])
+  ].join(" "));
+  let score = 0;
+  for (const token of triggerTokens) {
+    if (promptTokens.has(token)) score += token.length > 5 ? 2 : 1;
+  }
+  return score;
+}
+
+function evaluateRow({ prompt, fixture, expected, allowed, forbidden, selectedIds }) {
+  const selected = new Set(selectedIds);
+  const accepted = new Set([...expected, ...allowed]);
+  const hasExpected = expected.length
+    ? expected.every((skill) => selected.has(skill))
+    : selectedIds.length === 0;
+  const hasForbidden = forbidden.some((skill) => selected.has(skill));
+  const hasUnexpected = selectedIds.some((skill) => !accepted.has(skill));
+  return {
+    prompt,
+    fixture,
+    expected,
+    allowed,
+    forbidden,
+    selectedIds,
+    correct: hasExpected && !hasForbidden && !hasUnexpected
+  };
+}
+
+function summarizeSystem(name, rows) {
+  const correct = rows.filter((row) => row.correct).length;
+  return {
+    name,
+    correct,
+    total: rows.length,
+    correctRate: rows.length ? correct / rows.length : 0
+  };
+}
+
+function tokenize(value) {
+  return String(value || "")
+    .toLowerCase()
+    .split(/[^a-z0-9@.-]+/)
+    .filter((token) => token.length > 2);
+}
+
+function caseId(row) {
+  return `${row.fixture}\0${row.prompt}`;
+}
+
+function percent(value) {
+  return `${(value * 100).toFixed(1)}%`;
+}
+
+if (import.meta.url === `file://${process.argv[1]}`) {
+  const result = await runHallucinationLeaderboard();
+  console.log(formatHallucinationLeaderboard(result));
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@minhpnq1807/contextos",
-  "version": "0.6.0",
+  "version": "0.6.2",
   "description": "Task-aware AGENTS.md context injection and compliance reporting for Codex, Claude Code, and Antigravity.",
   "type": "module",
   "bin": {
@@ -10,6 +10,7 @@
   "files": [
     "bin/",
     "plugins/",
+    "scripts/",
     ".codex/skills/",
     ".codex/workflows/",
     ".agents/",
@@ -27,6 +28,10 @@
     "build": "node bin/ctx.js --version",
     "validate:plugin": "node test/validate-plugin.js",
     "benchmark:skills": "node bin/ctx.js benchmark --skills",
+    "leaderboard:hallucination": "node eval/hallucination/run-leaderboard.js",
+    "leaderboard:agents": "node eval/hallucination/run-agent-leaderboard.js --agents=codex,gemini",
+    "demo:capture": "node docs/demo/capture-live-demos.mjs",
+    "sync:community-skills": "node scripts/sync-community-skills.mjs",
     "test:mcp": "node test/mcp-protocol-smoke.js"
   },
   "engines": {

package/plugins/ctx/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "ctx",
-  "version": "0.6.0",
+  "version": "0.6.2",
   "description": "Inject task-relevant AGENTS.md rules into Codex through plugin hooks.",
   "author": {
     "name": "ContextOS"

package/scripts/sync-community-skills.mjs

@@ -0,0 +1,40 @@
+#!/usr/bin/env node
+import fs from "node:fs";
+import path from "node:path";
+
+const [, , sourceArg = "external-skills", targetArg = "community-skills"] = process.argv;
+const source = path.resolve(sourceArg);
+const target = path.resolve(targetArg);
+const skip = new Set([".git", ".github", "scripts"]);
+
+if (!fs.existsSync(source)) {
+  console.error(`Missing source skills directory: ${source}`);
+  process.exit(1);
+}
+
+fs.mkdirSync(target, { recursive: true });
+
+for (const entry of fs.readdirSync(target, { withFileTypes: true })) {
+  if (entry.isDirectory()) fs.rmSync(path.join(target, entry.name), { recursive: true, force: true });
+  else if (entry.isFile() && entry.name !== "README.md") fs.rmSync(path.join(target, entry.name), { force: true });
+}
+
+for (const entry of fs.readdirSync(source, { withFileTypes: true })) {
+  if (skip.has(entry.name)) continue;
+  const from = path.join(source, entry.name);
+  const to = path.join(target, entry.name);
+  if (entry.isDirectory()) copyDir(from, to);
+  else if (entry.isFile() && entry.name === "README.md") fs.copyFileSync(from, to);
+}
+
+console.log(`Synced ContextOS skills from ${source} to ${target}`);
+
+function copyDir(from, to) {
+  fs.mkdirSync(to, { recursive: true });
+  for (const entry of fs.readdirSync(from, { withFileTypes: true })) {
+    const sourcePath = path.join(from, entry.name);
+    const targetPath = path.join(to, entry.name);
+    if (entry.isDirectory()) copyDir(sourcePath, targetPath);
+    else if (entry.isFile()) fs.copyFileSync(sourcePath, targetPath);
+  }
+}