@inetafrica/open-claudia 2.7.3 → 2.7.4

package/CHANGELOG.md

@@ -1,6 +1,9 @@
 # Changelog
 
-## v2.7.3
+## v2.7.4
+- **The dream now audits tool usage — raw-op detector + usage KPI (Phase 3/item 9 + enforcement 5c).** The guard log knew about *blocked* commands (denies) and *authorised* one-offs (`keyring exec` bypasses), but operational work done raw that the gate didn't cover — inline heredoc scripts fed to an interpreter, raw DB clients, mutating `kubectl` — left no trace, so the tool-usage KPI had a blind spot exactly where the "repeated heredocs slip through" failure lives. New deterministic detector in `core/tool-guard.js` (`matchRawOp`/`noteRawOp`): high-precision patterns (`heredoc-interpreter`, `db-client`, `kubectl-mutate`), wired into the runner's Bash-stream chokepoint so every command is observed — **observation only, never blocking**; the authorised channels and anything the deny-gate already owns are skipped so one command never yields two guard events. Reads (`kubectl get/logs`, `rollout status`, `cat <<EOF` file writes, mentioning a client in grep) stay unflagged.
+- **Doc/code drift flag completes the dream tool-pass.** Alongside the existing hygiene flags (dangling pack links, cold never-run tools, near-duplicate pairs), the pass now flags tools whose source was edited after TOOL.md was last touched, and verbs that telemetry proves are in real use but the docs never mention (`tools.docDrift()`; `help`/`(default)` ignored). Flag only — the fix is a `tool note`. Zero false flags on the live 12-tool registry at ship time.
+- **Nightly KPI in the dream report, computed from stamps — never prose.** The dream's tool pass now appends `📈 Tool usage since last dream`: % of operational actions via `tool run` (target >90%, denominator = tool runs + escape-hatch bypasses + raw-op sightings; gate denies reported separately since blocked ≠ completed), bypass count **with reasons**, raw-ops by pattern, and failure-at-use counts naming the failing tools. Sources are `state.json` run-history stamps (new `tools.runsSince(sinceTs)`) + `tool-guard.jsonl` (new `readGuardEvents(sinceTs)`) — per the plan, the KPI is never parsed out of conversation prose. Escape-hatch bypasses surface in the dream report only (no immediate chat pings — pings would train us both to ignore them). Test coverage: raw-op matrix incl. no-double-count with denies, event grouping/windowing, `runsSince` window math.
 - **Surfaced tools now carry their condition (tiered surfacing, Phase 2/item 6).** The per-turn "Tools that may help here" block rendered one identity line per tool — name, description, keyring needs — so the agent knew a tool *existed* but not what state it was in, and had to spend a `tool show` (or just run it blind) to find out it was broken, never proven, or failing since Tuesday. Each surfaced tool is now a **2-line headline**: line 1 is identity (name — description, risk tier, keyring needs, why it surfaced), line 2 is condition — the TOOL.md **State** headline (first line, truncated), the open **known-issue count** (⚠ n), and **last-run health** from telemetry (`last run OK/FAILED (exit n) <date>, n runs`, or `never run`; legacy flat tools without exit telemetry degrade to `last used`). This completes the tiering ladder: 2-line headline always → TOOL.md via `tool show` → source only via `--source` — you read the manual, not the disassembly, and now the cover tells you if the manual has errata. New `tools.toolCondition()` with unit coverage in `test-tools.js` (headline extraction, `(none yet)` filtering, health formats, clean degrade) and end-to-end render assertions in `test-recall-discoverer.js`.
 
 ## v2.7.2

package/core/dream.js

@@ -1192,6 +1192,42 @@ async function runDream({ trigger = "manual" } = {}) {
       if (dupPairs.size) {
         bits.push(`👯 ${dupPairs.size} near-duplicate tool pair(s): ${[...dupPairs].join(", ")} — consider merging into one tool with subcommands.`);
       }
+      // Doc/code drift: source edited after TOOL.md, or verbs in real use that
+      // the docs never mention. Flag only — the fix is a tool note.
+      const drifted = allTools
+        .map((t) => { const d = toolsLib.docDrift(t); return d ? `${t.name} (${d})` : null; })
+        .filter(Boolean);
+      if (drifted.length) {
+        bits.push(`📝 ${drifted.length} tool(s) with doc/code drift: ${drifted.join("; ")} — refresh the docs with tool note.`);
+      }
+      // 5c usage KPI: what fraction of operational actions went through the
+      // authorised channel, computed from telemetry stamps (state.json run
+      // history) + the guard log — never prose parsing. Denominator counts
+      // COMPLETED actions: tool runs + logged escape-hatch bypasses + raw-op
+      // sightings; gate denies are reported separately (blocked ≠ completed).
+      try {
+        const guardLib = require("./tool-guard");
+        const sinceTs = dreamState.lastRun || new Date(Date.now() - 7 * 86400000).toISOString();
+        const g = guardLib.readGuardEvents(sinceTs);
+        const r = toolsLib.runsSince(sinceTs);
+        const ops = r.runs + g.bypasses.length + g.rawOps.length;
+        if (ops) {
+          const pct = Math.round((100 * r.runs) / ops);
+          const kpiBits = [`${pct}% of operational actions via tool run (${r.runs}/${ops}, target >90%)`];
+          if (g.bypasses.length) {
+            const reasons = g.bypasses.map((b) => b.reason).filter(Boolean).slice(0, 3).join("; ");
+            kpiBits.push(`${g.bypasses.length} escape-hatch bypass(es)${reasons ? ` — ${reasons}` : ""}`);
+          }
+          if (g.rawOps.length) {
+            const byPattern = {};
+            for (const evt of g.rawOps) byPattern[evt.pattern || "?"] = (byPattern[evt.pattern || "?"] || 0) + 1;
+            kpiBits.push(`${g.rawOps.length} raw op(s) outside tools: ${Object.entries(byPattern).map(([k, v]) => `${k} ×${v}`).join(", ")}`);
+          }
+          if (g.denies.length) kpiBits.push(`${g.denies.length} denied by the gate`);
+          if (r.failures) kpiBits.push(`${r.failures} failed run(s) at use: ${r.failedTools.join(", ")}`);
+          bits.push(`📈 Tool usage since last dream: ${kpiBits.join(" · ")}.`);
+        }
+      } catch (e) { /* KPI is best-effort */ }
       toolNote = bits.join("\n");
     } catch (e) { /* tool graph is best-effort (old node) */ }
 

package/core/runner.js

@@ -958,6 +958,11 @@ async function runClaude(prompt, cwd, replyToMsgId, opts = {}) {
   const noteRecallFromShell = (command) => {
     try {
       const cmd = String(command || "");
+      // 5c raw-op audit: every Bash command in the stream passes here, so
+      // operational work done raw — heredoc scripts, DB clients, mutating
+      // kubectl — gets stamped to the guard log for the dream's KPI pass.
+      // Observation only; blocking is the deny-gate hook's job, not this.
+      try { require("./tool-guard").noteRawOp(cmd); } catch (e) { /* best-effort */ }
       if (!cmd.includes("open-claudia")) return;
       let m;
       const packRe = /\bpack\s+show\s+["']?([a-z0-9][\w.-]*)/gi;

package/core/tool-guard.js

@@ -107,6 +107,76 @@ function logGuardEvent(evt) {
   } catch (e) { /* best effort */ }
 }
 
+// ── 5c raw-op audit (observation, never blocking) ──────────────────────────
+// The deny-gate above catches acting commands aimed at KNOWN prod surfaces;
+// this detector catches the SHAPE of operational work done raw that should
+// have been a tool — inline heredoc scripts fed to an interpreter, raw DB
+// clients, mutating kubectl. Same high-precision-over-recall doctrine as the
+// gate: reads and probes (kubectl get/logs, grep, ping) stay unflagged, so
+// the dream's KPI measures real operational bypassing, not diagnosis noise.
+const RAW_OP_PATTERNS = [
+  {
+    // An interpreter or DB shell fed an inline heredoc script — the exact
+    // "re-derive the script instead of reusing a tool" smell. cat/tee heredocs
+    // (file writes) deliberately don't match.
+    id: "heredoc-interpreter",
+    re: /(?:^|[|;&(`'"]|\$\()\s*(?:sudo\s+)?(?:node|python3?|ruby|perl|php|deno|bun|mongo|mongosh|psql|mysql|redis-cli)\b[^\n]*<</,
+  },
+  {
+    id: "db-client",
+    re: /(?:^|[|;&(`'"]|\$\()\s*(?:sudo\s+)?(?:mongo|mongosh|psql|mysql|redis-cli)\b/,
+  },
+  {
+    // Cluster mutations raw from the shell (GitOps or the prod-k8s tool are
+    // the channels). kubectl get/describe/logs/top/events/port-forward and
+    // rollout status/history are reads — always free per the hard rules.
+    id: "kubectl-mutate",
+    re: /\bkubectl\s+(?:exec|apply|delete|scale|patch|edit|create|replace|cordon|uncordon|drain|taint|label|annotate|set)\b|\bkubectl\s+rollout\s+(?:restart|undo|pause|resume)\b/,
+  },
+];
+
+// Pure: null, or the matched raw-op pattern. The authorised channels and
+// anything the deny-gate owns (already logged as a deny by the hook) are
+// skipped so one command never produces two guard events.
+function matchRawOp(command) {
+  const cmd = String(command || "");
+  if (!cmd.trim()) return null;
+  if (ALLOW_RE.test(cmd)) return null;
+  if (matchRule(cmd)) return null;
+  for (const p of RAW_OP_PATTERNS) if (p.re.test(cmd)) return p;
+  return null;
+}
+
+// Stamp a sighting to the guard log. Called from the runner's Bash-stream
+// chokepoint on every command; must never throw into the stream parser.
+function noteRawOp(command) {
+  try {
+    const p = matchRawOp(command);
+    if (!p) return null;
+    logGuardEvent({ kind: "raw-op", pattern: p.id, command: String(command).slice(0, 300) });
+    return p;
+  } catch (e) { return null; }
+}
+
+// KPI data source (5c): guard events since a timestamp, grouped by kind.
+// Malformed lines are skipped — the log is append-only best-effort.
+function readGuardEvents(sinceTs) {
+  const out = { denies: [], bypasses: [], rawOps: [] };
+  let raw = "";
+  try { raw = fs.readFileSync(GUARD_LOG, "utf-8"); } catch (e) { return out; }
+  const since = sinceTs ? Date.parse(sinceTs) : NaN;
+  for (const line of raw.split("\n")) {
+    if (!line.trim()) continue;
+    let evt;
+    try { evt = JSON.parse(line); } catch (e) { continue; }
+    if (!Number.isNaN(since) && (!evt.ts || Date.parse(evt.ts) < since)) continue;
+    if (evt.kind === "deny") out.denies.push(evt);
+    else if (evt.kind === "bypass") out.bypasses.push(evt);
+    else if (evt.kind === "raw-op") out.rawOps.push(evt);
+  }
+  return out;
+}
+
 // Hook entry, pure enough to test: takes the PreToolUse stdin payload (object
 // or JSON string), returns { exit, stderr }. exit 2 = deny (stderr is fed back
 // to the model); exit 0 = allow. Anything unexpected → allow (fail-open).
@@ -160,4 +230,8 @@ function ensureHookSettings() {
   return SETTINGS_FILE;
 }
 
-module.exports = { RULES, GUARD_LOG, SETTINGS_FILE, matchRule, denyMessage, logGuardEvent, runHook, hookSettings, ensureHookSettings };
+module.exports = {
+  RULES, RAW_OP_PATTERNS, GUARD_LOG, SETTINGS_FILE,
+  matchRule, denyMessage, logGuardEvent, runHook, hookSettings, ensureHookSettings,
+  matchRawOp, noteRawOp, readGuardEvents,
+};

package/core/tools.js

@@ -375,6 +375,44 @@ function findTool(name) {
   return readTool(needle);
 }
 
+// Window telemetry for the 5c usage KPI: `tool run` executions since a
+// timestamp, counted from state.json history entries — the CLI's authoritative
+// per-run stamps, never prose parsing. History is capped per tool, so a tool
+// that ran more than HISTORY_CAP times in the window undercounts slightly;
+// acceptable for a nightly ratio.
+function runsSince(sinceTs) {
+  const since = sinceTs ? Date.parse(sinceTs) : NaN;
+  const out = { runs: 0, failures: 0, byTool: {}, failedTools: [] };
+  for (const t of listTools()) {
+    const hist = Array.isArray(t.history) ? t.history : [];
+    const inWindow = hist.filter((h) => h && h.at && (Number.isNaN(since) || Date.parse(h.at) >= since));
+    if (!inWindow.length) continue;
+    const fails = inWindow.filter((h) => h.exit !== 0).length;
+    out.runs += inWindow.length;
+    out.failures += fails;
+    out.byTool[t.name] = { runs: inWindow.length, failures: fails };
+    if (fails) out.failedTools.push(`${t.name} ×${fails}`);
+  }
+  return out;
+}
+
+// Doc/code drift check for the dream's tool pass: the source was edited after
+// TOOL.md was last touched (docs describe an older tool), or a verb telemetry
+// proves is in real use never appears in the docs. Returns a short reason or
+// null. The 60s slack absorbs creation-time write ordering (source + doc
+// skeleton land moments apart).
+function docDrift(t) {
+  if (!t || t.layout !== "dir" || !t.docFile) return null;
+  try {
+    if (fs.statSync(t.file).mtimeMs > fs.statSync(t.docFile).mtimeMs + 60000) return "source newer than TOOL.md";
+  } catch (e) { /* doc or source unreadable → fall through to verb check */ }
+  const doc = readToolDoc(t.name) || "";
+  const verbs = Object.keys(t.verbs || {}).filter((v) => v && v !== "(default)" && v !== "help" && !v.startsWith("-"));
+  const undocd = verbs.filter((v) => !doc.includes(v)).slice(0, 3);
+  if (undocd.length) return `verb${undocd.length === 1 ? "" : "s"} ${undocd.join(", ")} used but undocumented`;
+  return null;
+}
+
 // Build the header comment block for a script that doesn't already declare one.
 // Uses the comment prefix that matches the script's shebang/extension.
 function buildHeader(prefix, { name, description, pack, requires, usage, risk }) {
@@ -712,7 +750,7 @@ function toolNameFromPath(filePath) {
 
 module.exports = {
   TOOLS_DIR, MARKER, USAGE_FILE, RISK_TIERS,
-  parseHeader, readTool, readToolDoc, toolCondition, listTools, findTool, addTool, removeTool, updateToolHeader,
+  parseHeader, readTool, readToolDoc, toolCondition, listTools, findTool, runsSince, docDrift, addTool, removeTool, updateToolHeader,
   runEnv, missingRequires, toolNameFromPath, sanitizeName, ensureDir, normalizeRisk,
   toolPaths, readState, writeState, recordRunResult, runGate,
   matchTools, recordRun, recordCreate, readUsage,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@inetafrica/open-claudia",
-  "version": "2.7.3",
+  "version": "2.7.4",
   "description": "Your always-on AI coding assistant — Claude Code, Cursor Agent, and OpenAI Codex via Telegram or Kazee Chat",
   "main": "bot.js",
   "bin": {

package/test-tool-guard.js

@@ -68,6 +68,43 @@ assert.ok(lines.some((l) => l.kind === "deny" && l.rule === "inet-central-api"),
 assert.ok(lines.some((l) => l.kind === "bypass" && l.reason === "no tool yet"), "keyring exec bypass is logged");
 assert.ok(lines.every((l) => l.ts), "every guard event is timestamped");
 
+// ── raw-op audit (5c): operational work done raw is observed, never blocked ──
+assert.strictEqual(guard.matchRawOp("node <<'EOF'\nfetch('https://api.example.com/x')\nEOF").id, "heredoc-interpreter",
+  "inline node heredoc is the canonical re-derived-script smell");
+assert.strictEqual(guard.matchRawOp("python3 <<PY\nprint(1)\nPY").id, "heredoc-interpreter");
+assert.strictEqual(guard.matchRawOp('echo start && mongosh <<JS\ndb.x.find()\nJS').id, "heredoc-interpreter",
+  "heredoc after && still detected");
+assert.strictEqual(guard.matchRawOp('mongosh "mongodb://db.internal/app" --eval "db.users.count()"').id, "db-client");
+assert.strictEqual(guard.matchRawOp("psql -h db.internal -c 'select 1'").id, "db-client");
+assert.strictEqual(guard.matchRawOp("kubectl delete pod api-0 -n prod").id, "kubectl-mutate");
+assert.strictEqual(guard.matchRawOp("kubectl apply -f deploy.yaml").id, "kubectl-mutate");
+assert.strictEqual(guard.matchRawOp("kubectl rollout restart deploy/api").id, "kubectl-mutate");
+assert.strictEqual(guard.matchRawOp("kubectl exec api-0 -- cat /etc/config").id, "kubectl-mutate",
+  "raw kubectl exec is an op vector even for reads inside the pod");
+assert.strictEqual(guard.matchRawOp("cat <<'EOF' > notes.md\nhello\nEOF"), null, "heredoc into a file write is not an op");
+assert.strictEqual(guard.matchRawOp("kubectl get pods -A"), null, "read-only kubectl stays unflagged");
+assert.strictEqual(guard.matchRawOp("kubectl rollout status deploy/api"), null, "rollout status is a read");
+assert.strictEqual(guard.matchRawOp("kubectl logs api-7f9 -n prod"), null, "kubectl logs is a read");
+assert.strictEqual(guard.matchRawOp("node script.js"), null, "running a checked-in script is not the heredoc smell");
+assert.strictEqual(guard.matchRawOp("grep -rn mongosh docs/"), null, "mentioning a client binary mid-line is not acting");
+assert.strictEqual(guard.matchRawOp("open-claudia tool run prod-k8s exec api-0 'cat /etc/config'"), null,
+  "authorised channel never flagged");
+assert.strictEqual(guard.matchRawOp('open-claudia keyring exec --reason "x" -- mongosh mongodb://db/x'), null,
+  "escape hatch logs its own bypass, not a raw-op");
+assert.strictEqual(guard.matchRawOp("kubectl exec mongo-0 -- mongosh"), null,
+  "deny-rule matches belong to the gate — one command never yields two guard events");
+assert.strictEqual(guard.noteRawOp("kubectl delete pod api-1 -n prod").id, "kubectl-mutate", "noteRawOp returns the pattern");
+assert.strictEqual(guard.noteRawOp("ls -la"), null, "benign commands log nothing");
+
+// ── readGuardEvents: since-filtered grouping (the KPI's data source) ──
+const evts = guard.readGuardEvents("2000-01-01T00:00:00Z");
+assert.ok(evts.denies.length >= 1, "denies grouped");
+assert.ok(evts.bypasses.length >= 1, "bypasses grouped");
+assert.ok(evts.rawOps.some((e) => e.pattern === "kubectl-mutate"), "raw-ops grouped with their pattern");
+assert.strictEqual(guard.readGuardEvents(new Date(Date.now() + 86400000).toISOString()).rawOps.length, 0,
+  "future since → empty window");
+assert.ok(guard.readGuardEvents(null).denies.length >= 1, "null since → whole log");
+
 // ── hook settings: a valid claude --settings payload, idempotent writer ──
 const sPath = guard.ensureHookSettings();
 assert.strictEqual(sPath, process.env.DENY_GATE_SETTINGS, "settings path is env-overridable (test isolation)");

package/test-tools.js

@@ -334,6 +334,34 @@ tools.writeState("condtool", { createdAt: "2026-07-01T00:00:00Z", lastUsed: "202
 assert.ok(/last run OK 2026-07-03, 4 runs/.test(tools.toolCondition(tools.findTool("condtool"))), "healthy last run reads OK");
 assert.strictEqual(tools.toolCondition(null), "", "no tool → empty digest");
 
+// ── runsSince: windowed run counts from state.json history (5c KPI source) ──
+tools.writeState("condtool", {
+  createdAt: "2026-07-01T00:00:00Z", lastUsed: "2099-01-02T00:00:00Z", runCount: 7, lastExit: 1,
+  history: [
+    { at: "2099-01-02T00:00:00Z", args: "list --json", exit: 1, ms: 40 },
+    { at: "2099-01-01T00:00:00Z", args: "list", exit: 0, ms: 35 },
+    { at: "2000-01-01T00:00:00Z", args: "list", exit: 0, ms: 30 },
+  ],
+});
+const win = tools.runsSince("2098-12-31T00:00:00Z");
+assert.strictEqual(win.byTool.condtool.runs, 2, "only history entries inside the window count");
+assert.strictEqual(win.byTool.condtool.failures, 1, "non-zero exits count as failures at use");
+assert.ok(win.failedTools.includes("condtool ×1"), "failing tools are named for the dream report");
+assert.strictEqual(tools.runsSince("2099-06-01T00:00:00Z").runs, 0, "empty window → zero runs");
+
+// ── docDrift: docs stale vs source/telemetry (dream tool-pass flag) ──
+assert.strictEqual(tools.docDrift(tools.findTool("condtool")), null, "fresh docs → no drift");
+tools.writeState("condtool", {
+  createdAt: "2026-07-01T00:00:00Z", lastUsed: "2099-01-02T00:00:00Z", runCount: 7, lastExit: 0,
+  verbs: { list: { runs: 3 }, stages: { runs: 1 }, help: { runs: 1 }, "(default)": { runs: 1 } },
+});
+assert.strictEqual(tools.docDrift(tools.findTool("condtool")), "verb stages used but undocumented",
+  "a verb in real use that TOOL.md never mentions is drift (help/(default) ignored)");
+const future = new Date(Date.now() + 3600000);
+fs.utimesSync(tools.findTool("condtool").file, future, future);
+assert.strictEqual(tools.docDrift(tools.findTool("condtool")), "source newer than TOOL.md", "source edited after docs is drift");
+assert.strictEqual(tools.docDrift(null), null, "no tool → no drift");
+
 // ── remove ──
 assert.ok(tools.removeTool("hello"), "removeTool returns the removed tool");
 assert.strictEqual(tools.findTool("hello"), null, "removed tool is gone");