@minhpnq1807/contextos 0.6.0 → 0.6.1

package/CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.6.1
+
+- **Hallucination Leaderboard:** Added `ctx leaderboard --hallucination` and `npm run leaderboard:hallucination` to compare raw prompt-only skill guesses against ContextOS evidence-routed skill selection across 20 fixture tasks.
+
 ## 0.6.0
 
 - **Launch demo framing:** Added Agent Hallucination Benchmark messaging, same-prompt/same-model/different-context copy, and `docs/launch-demos.md` with three short demo scripts: hallucination benchmark, AGENTS.md lost-in-the-middle, and repo-aware skills.

package/README.md

@@ -28,7 +28,7 @@ Published package: [`@minhpnq1807/contextos`](https://www.npmjs.com/package/@min
 
 ## Demo
 
-![ContextOS demo: same prompt, different repo, correct skills](docs/demo/contextos-demo.gif)
+![ContextOS demo: same prompt, different repo, correct skills](docs/demo/same-prompt-different-context.gif)
 
 Same prompt. Same model. Different context.
 
@@ -42,6 +42,13 @@ ctx skills doctor -- "fix deployed"
 | `vercel.json`, `next`, GitHub workflow | `vercel-deployment`, `github-actions-ci-cd`, `env-secret-management` |
 | ContextOS repo with no app deploy evidence | no deployment skill selected |
 
+More 10-second demos:
+
+| Demo | GIF |
+| --- | --- |
+| AGENTS.md Lost In The Middle | [docs/demo/agents-lost-middle.gif](docs/demo/agents-lost-middle.gif) |
+| ContextOS Ready Gold | [docs/demo/contextos-ready.gif](docs/demo/contextos-ready.gif) |
+
 ## Agent Hallucination Benchmark
 
 Generic agents often guess deployment tooling from the prompt alone:
@@ -80,6 +87,19 @@ Skill Router internal fixture benchmark:
 
 This is an internal fixture benchmark, not an external real-world benchmark. It is designed to prove the router behavior across controlled Expo/EAS, Next/Vercel, Docker, Railway/Render, Firebase, auth, database, testing, mobile, and adversarial negative-gate cases.
 
+Hallucination leaderboard:
+
+```bash
+ctx leaderboard --hallucination
+```
+
+Current local result across 20 fixture tasks and 12 repo contexts:
+
+| System | Correct Skill |
+| --- | ---: |
+| Raw Agent | 10.0% |
+| ContextOS + Codex | 80.0% |
+
 Example hook context injected before the agent works:
 
 ```text
@@ -246,6 +266,7 @@ The score checks project `AGENTS.md` rules, project skill packs under `.codex/sk
 | `ctx stats` | Show workspace-level usage and effectiveness metrics. |
 | `ctx benchmark -- "task"` | Compare raw AGENTS.md ordering vs ContextOS scheduling. |
 | `ctx benchmark --skills` | Run the Skill Router eval benchmark. |
+| `ctx leaderboard --hallucination` | Compare raw prompt-only guesses vs ContextOS routing. |
 | `ctx sync --rules` | Sync AGENTS/Ruler/MCP config across agents. |
 | `ctx sync --skills` | Sync skills across agents through skillshare. |
 | `ctx sync --workflows` | Sync workflow markdown across Claude/Codex/Antigravity. |
@@ -570,6 +591,7 @@ This warning comes from a transitive dependency in the local embedding/WASM stac
 | `ctx stats` | Shows aggregate runtime metrics for the current workspace. | You want to know whether ContextOS is active and useful over time. | Prints sectioned tables for prompt/report counts, injection rate, efficiency, rule outcomes, hook events, last prompt, and last report. |
 | `ctx benchmark -- "task"` | Compares baseline AGENTS.md ordering with ContextOS task-aware scheduling. | You want a before/after signal for lost-in-the-middle risk. | Prints tables for parsed/actionable/filtered rules, baseline middle-risk, scheduled high/mid rules, recency reminder status, and top scored rules. |
 | `ctx benchmark --skills` | Runs the Skill Router eval benchmark. | You want evidence for skill routing accuracy and negative gates. | Prints top-1 accuracy, top-3 recall, false positive rate, confidence calibration, and negative gate accuracy across `eval/skill-routing` fixtures. |
+| `ctx leaderboard --hallucination` | Compares raw prompt-only skill guesses with ContextOS evidence routing. | You want launch evidence for the hallucination problem. | Runs 20 fixture tasks across 10+ repo contexts and prints Raw Agent vs ContextOS correctness plus sample failures. |
 | `ctx sync --rules` | Syncs project rules and MCP servers through Ruler. | You want Codex, Claude Code, and Antigravity to share one project rule/MCP source of truth. | Ensures `.ruler/ruler.toml`, injects `ctx-mcp`, imports existing MCP servers from Codex and project `.mcp.json`, runs `ruler apply --agents codex,claude,antigravity`, mirrors MCP servers to Antigravity MCP configs, and verifies generated config. |
 | `ctx sync --rules --agents <list>` | Syncs only selected agents through Ruler. | You want to update one or two agents without touching the others. | Accepts comma-separated values such as `codex`, `claude`, `agy`, `antigravity`, or `codex,claude,agy`; `agy` is normalized to Ruler's `antigravity`. |
 | `ctx sync --rules --dry-run` | Previews Ruler sync without writing files or running apply. | You want to inspect behavior before changing project config. | Prints the same flow with dry-run status. |

package/bin/ctx.js

@@ -20,6 +20,7 @@ import { defaultDataRoot, workspaceDataDir, workspaceMarkerPath } from "../plugi
 import { installMcpTelemetryProxies } from "../plugins/ctx/lib/mcp-proxy-install.js";
 import { benchmarkWorkspace, formatBenchmark } from "../plugins/ctx/lib/benchmark.js";
 import { formatSkillRoutingBenchmark, runSkillRoutingEval } from "../eval/skill-routing/run-eval.js";
+import { formatHallucinationLeaderboard, runHallucinationLeaderboard } from "../eval/hallucination/run-leaderboard.js";
 import { copyDir, copyPackageRoot, syncPackageRoot } from "../plugins/ctx/lib/package-install.js";
 import { installClaudeHooks } from "../plugins/ctx/lib/claude-hooks.js";
 import { installClaudeMcp } from "../plugins/ctx/lib/claude-mcp.js";
@@ -197,6 +198,7 @@ Usage:
   ctx stats                                         Show workspace statistics
   ctx benchmark -- "task"                           Benchmark workspace for a task
   ctx benchmark --skills                            Run skill routing eval benchmark
+  ctx leaderboard --hallucination                   Compare raw agent guesses vs ContextOS routing
   ctx sync --rules                                  Sync AGENTS.md rules to all agents
   ctx sync --rules --agents <names>                 Sync rules to specific agents only
   ctx sync --rules --dry-run                        Preview rule sync without writing
@@ -1034,6 +1036,12 @@ try {
     if (!task.trim()) throw new Error('Usage: ctx benchmark -- "task"');
     console.log(formatBenchmark(benchmarkWorkspace({ cwd: process.cwd(), task })));
     }
+  } else if (command === "leaderboard") {
+    if (args.includes("--hallucination")) {
+      console.log(formatHallucinationLeaderboard(await runHallucinationLeaderboard({ rootDir })));
+    } else {
+      throw new Error("Usage: ctx leaderboard --hallucination");
+    }
   } else if (command === "skills") {
     if (args[1] === "doctor") {
       const marker = args.indexOf("--");

package/docs/demo/agents-lost-middle.txt

@@ -0,0 +1,28 @@
+$ cat AGENTS.md
+1. General style
+2. Formatting
+3. Test names
+...
+37. IMPORTANT: Always use code-review-graph before grep.
+...
+52. Release notes
+
+$ codex "fix failing test"
+Raw agent starts with grep
+Rule followed: no
+
+$ ctx debug -- "fix failing test"
+ContextOS debug
+Critical ContextOS rules:
+- IMPORTANT: Always use code-review-graph before grep.
+
+Suggested files to check:
+- test/score-context.test.js
+- plugins/ctx/lib/score-context.js
+
+$ codex + ContextOS
+Rule followed: yes
+Evidence: graph checked before file reads
+
+AGENTS.md did not change.
+The rule moved from buried context into runtime context.

package/docs/demo/contextos-ready.txt

@@ -0,0 +1,20 @@
+$ ctx doctor
+Repository Score
+
+Rules: 100
+Skills: 100
+Workflows: 100
+
+Overall:
+ContextOS Ready Gold
+
+Evidence:
+- Rules: 1 AGENTS.md source(s), 5 actionable rule(s)
+- Skills: 3 skill(s), 3 metadata file(s)
+- Workflows: 2 workflow(s), 2 with agent chain(s)
+
+$ badge
+[ContextOS Ready Gold]
+
+Repos now have a target:
+AGENTS.md + skills + workflows + evidence.

package/docs/demo/same-prompt-different-context.txt

@@ -0,0 +1,26 @@
+$ ctx leaderboard --hallucination
+Hallucination Leaderboard
+Repos: 12
+Tasks: 20
+
+System              Correct Skill
+------------------  -------------
+Raw Agent           10.0%
+ContextOS + Codex   80.0%
+
+$ ctx skills doctor -- "fix deployed"  # Expo repo
+ContextOS skill doctor
+1. eas high confidence
+   evidence: eas.json, app.json, expo dependency
+2. mobile-deployment high confidence
+3. github-actions-ci-cd medium confidence
+
+$ ctx skills doctor -- "fix deployed"  # Next.js repo
+ContextOS skill doctor
+1. vercel-deployment high confidence
+   evidence: vercel.json, next dependency
+2. github-actions-ci-cd high confidence
+3. env-secret-management medium confidence
+
+Same prompt. Same model. Different repo evidence.
+ContextOS routes the right skill before the agent edits code.

package/docs/launch-demos.md

@@ -4,6 +4,8 @@ These are demo scripts for explaining ContextOS quickly. They are intentionally
 
 ## 1. Agent Hallucination Benchmark
 
+GIF: [`docs/demo/same-prompt-different-context.gif`](demo/same-prompt-different-context.gif)
+
 Prompt:
 
 ```text
@@ -39,6 +41,8 @@ Same prompt. Same model. Different context.
 
 ## 2. AGENTS.md Lost In The Middle
 
+GIF: [`docs/demo/agents-lost-middle.gif`](demo/agents-lost-middle.gif)
+
 Setup:
 
 ```text
@@ -71,6 +75,8 @@ Important repo rules should not depend on where they appear in a long file.
 
 ## 3. Repo-Aware Skills
 
+GIF: [`docs/demo/same-prompt-different-context.gif`](demo/same-prompt-different-context.gif)
+
 Prompt:
 
 ```text
@@ -103,3 +109,19 @@ Message:
 ```text
 Context is not extra text. It changes the correct answer.
 ```
+
+## 4. ContextOS Ready
+
+GIF: [`docs/demo/contextos-ready.gif`](demo/contextos-ready.gif)
+
+Command:
+
+```bash
+ctx doctor
+```
+
+Message:
+
+```text
+Repos now have a target: AGENTS.md + skills + workflows + evidence.
+```

package/eval/hallucination/run-leaderboard.js

@@ -0,0 +1,183 @@
+#!/usr/bin/env node
+import fs from "node:fs";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+
+import { parseEvalYaml, runSkillRoutingEval } from "../skill-routing/run-eval.js";
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const evalRoot = path.resolve(__dirname, "..", "skill-routing");
+const DEFAULT_CASE_LIMIT = 20;
+
+export async function runHallucinationLeaderboard({
+  rootDir = path.resolve(__dirname, "..", ".."),
+  casesPath = path.join(evalRoot, "cases.yaml"),
+  caseLimit = DEFAULT_CASE_LIMIT
+} = {}) {
+  const config = parseEvalYaml(fs.readFileSync(casesPath, "utf8"));
+  const selectedCases = selectLeaderboardCases(config.cases, caseLimit);
+  const wanted = new Set(selectedCases.map(caseId));
+  const contextos = await runSkillRoutingEval({ rootDir, casesPath, topK: 3, threshold: 0.5 });
+  const contextRows = contextos.rows
+    .filter((row) => wanted.has(caseId(row)))
+    .map((row) => evaluateRow({
+      prompt: row.prompt,
+      fixture: row.fixture,
+      expected: row.expected,
+      allowed: row.allowed,
+      forbidden: row.forbidden,
+      selectedIds: row.selectedIds
+    }));
+  const rawRows = selectedCases.map((testCase) => evaluateRow({
+    prompt: testCase.prompt,
+    fixture: testCase.fixture,
+    expected: testCase.expected || [],
+    allowed: testCase.allowed || [],
+    forbidden: testCase.forbidden || [],
+    selectedIds: rawAgentSkills({ prompt: testCase.prompt, skills: config.skills, topK: 3 })
+  }));
+
+  return {
+    caseCount: selectedCases.length,
+    repoCount: new Set(selectedCases.map((row) => row.fixture)).size,
+    systems: [
+      summarizeSystem("Raw Agent", rawRows),
+      summarizeSystem("ContextOS + Codex", contextRows)
+    ],
+    rows: selectedCases.map((testCase) => ({
+      prompt: testCase.prompt,
+      fixture: testCase.fixture,
+      expected: testCase.expected || [],
+      raw: rawRows.find((row) => row.prompt === testCase.prompt && row.fixture === testCase.fixture),
+      contextos: contextRows.find((row) => row.prompt === testCase.prompt && row.fixture === testCase.fixture)
+    }))
+  };
+}
+
+export function formatHallucinationLeaderboard(result) {
+  const lines = [
+    "Hallucination Leaderboard",
+    `Repos: ${result.repoCount}`,
+    `Tasks: ${result.caseCount}`,
+    "",
+    "System              Correct Skill",
+    "------------------  -------------"
+  ];
+  for (const system of result.systems) {
+    lines.push(`${system.name.padEnd(18)}  ${percent(system.correctRate)}`);
+  }
+  lines.push("", "Sample failures:");
+  const failures = result.rows
+    .filter((row) => !row.raw.correct || !row.contextos.correct)
+    .slice(0, 6);
+  if (!failures.length) {
+    lines.push("- none");
+  } else {
+    for (const row of failures) {
+      lines.push(`- ${row.fixture}: "${row.prompt}"`);
+      lines.push(`  expected: ${row.expected.join(", ") || "(none)"}`);
+      lines.push(`  raw: ${row.raw.selectedIds.join(", ") || "(none)"} ${row.raw.correct ? "✓" : "✗"}`);
+      lines.push(`  contextos: ${row.contextos.selectedIds.join(", ") || "(none)"} ${row.contextos.correct ? "✓" : "✗"}`);
+    }
+  }
+  return lines.join("\n");
+}
+
+function selectLeaderboardCases(cases, limit) {
+  const wantedFixtures = [
+    "expo-eas",
+    "next-vercel",
+    "docker-node",
+    "railway-render",
+    "firebase-hosting",
+    "nest-prisma",
+    "express-mongo-jwt",
+    "oauth-google",
+    "redis-cache",
+    "contextos",
+    "frontend-only-next",
+    "static-docs"
+  ];
+  const selected = [];
+  for (const fixture of wantedFixtures) {
+    const match = cases.find((row) => row.fixture === fixture && !selected.some((item) => caseId(item) === caseId(row)));
+    if (match) selected.push(match);
+    if (selected.length >= limit) return selected;
+  }
+  for (const row of cases) {
+    if (!selected.some((item) => caseId(item) === caseId(row))) selected.push(row);
+    if (selected.length >= limit) break;
+  }
+  return selected;
+}
+
+function rawAgentSkills({ prompt, skills, topK }) {
+  return skills
+    .map((skill) => ({ id: skill.id, score: rawPromptScore(prompt, skill) }))
+    .sort((a, b) => b.score - a.score || a.id.localeCompare(b.id))
+    .slice(0, topK)
+    .map((skill) => skill.id);
+}
+
+function rawPromptScore(prompt, skill) {
+  const promptTokens = new Set(tokenize(prompt));
+  const triggerTokens = tokenize([
+    skill.id,
+    skill.description,
+    ...(skill.positive_triggers?.prompts || [])
+  ].join(" "));
+  let score = 0;
+  for (const token of triggerTokens) {
+    if (promptTokens.has(token)) score += token.length > 5 ? 2 : 1;
+  }
+  return score;
+}
+
+function evaluateRow({ prompt, fixture, expected, allowed, forbidden, selectedIds }) {
+  const selected = new Set(selectedIds);
+  const accepted = new Set([...expected, ...allowed]);
+  const hasExpected = expected.length
+    ? expected.every((skill) => selected.has(skill))
+    : selectedIds.length === 0;
+  const hasForbidden = forbidden.some((skill) => selected.has(skill));
+  const hasUnexpected = selectedIds.some((skill) => !accepted.has(skill));
+  return {
+    prompt,
+    fixture,
+    expected,
+    allowed,
+    forbidden,
+    selectedIds,
+    correct: hasExpected && !hasForbidden && !hasUnexpected
+  };
+}
+
+function summarizeSystem(name, rows) {
+  const correct = rows.filter((row) => row.correct).length;
+  return {
+    name,
+    correct,
+    total: rows.length,
+    correctRate: rows.length ? correct / rows.length : 0
+  };
+}
+
+function tokenize(value) {
+  return String(value || "")
+    .toLowerCase()
+    .split(/[^a-z0-9@.-]+/)
+    .filter((token) => token.length > 2);
+}
+
+function caseId(row) {
+  return `${row.fixture}\0${row.prompt}`;
+}
+
+function percent(value) {
+  return `${(value * 100).toFixed(1)}%`;
+}
+
+if (import.meta.url === `file://${process.argv[1]}`) {
+  const result = await runHallucinationLeaderboard();
+  console.log(formatHallucinationLeaderboard(result));
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@minhpnq1807/contextos",
-  "version": "0.6.0",
+  "version": "0.6.1",
   "description": "Task-aware AGENTS.md context injection and compliance reporting for Codex, Claude Code, and Antigravity.",
   "type": "module",
   "bin": {
@@ -27,6 +27,7 @@
     "build": "node bin/ctx.js --version",
     "validate:plugin": "node test/validate-plugin.js",
     "benchmark:skills": "node bin/ctx.js benchmark --skills",
+    "leaderboard:hallucination": "node eval/hallucination/run-leaderboard.js",
     "test:mcp": "node test/mcp-protocol-smoke.js"
   },
   "engines": {

package/plugins/ctx/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "ctx",
-  "version": "0.6.0",
+  "version": "0.6.1",
   "description": "Inject task-relevant AGENTS.md rules into Codex through plugin hooks.",
   "author": {
     "name": "ContextOS"