@inetafrica/open-claudia 2.7.2 → 2.7.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/core/dream.js +36 -0
- package/core/recall/discoverer.js +12 -3
- package/core/runner.js +5 -0
- package/core/tool-guard.js +75 -1
- package/core/tools.js +73 -1
- package/package.json +1 -1
- package/test-recall-discoverer.js +16 -0
- package/test-tool-guard.js +37 -0
- package/test-tools.js +53 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## v2.7.4
|
|
4
|
+
- **The dream now audits tool usage — raw-op detector + usage KPI (Phase 3/item 9 + enforcement 5c).** The guard log knew about *blocked* commands (denies) and *authorised* one-offs (`keyring exec` bypasses), but operational work done raw that the gate didn't cover — inline heredoc scripts fed to an interpreter, raw DB clients, mutating `kubectl` — left no trace, so the tool-usage KPI had a blind spot exactly where the "repeated heredocs slip through" failure lives. New deterministic detector in `core/tool-guard.js` (`matchRawOp`/`noteRawOp`): high-precision patterns (`heredoc-interpreter`, `db-client`, `kubectl-mutate`), wired into the runner's Bash-stream chokepoint so every command is observed — **observation only, never blocking**; the authorised channels and anything the deny-gate already owns are skipped so one command never yields two guard events. Reads (`kubectl get/logs`, `rollout status`, `cat <<EOF` file writes, mentioning a client in grep) stay unflagged.
|
|
5
|
+
- **Doc/code drift flag completes the dream tool-pass.** Alongside the existing hygiene flags (dangling pack links, cold never-run tools, near-duplicate pairs), the pass now flags tools whose source was edited after TOOL.md was last touched, and verbs that telemetry proves are in real use but the docs never mention (`tools.docDrift()`; `help`/`(default)` ignored). Flag only — the fix is a `tool note`. Zero false flags on the live 12-tool registry at ship time.
|
|
6
|
+
- **Nightly KPI in the dream report, computed from stamps — never prose.** The dream's tool pass now appends `📈 Tool usage since last dream`: % of operational actions via `tool run` (target >90%, denominator = tool runs + escape-hatch bypasses + raw-op sightings; gate denies reported separately since blocked ≠ completed), bypass count **with reasons**, raw-ops by pattern, and failure-at-use counts naming the failing tools. Sources are `state.json` run-history stamps (new `tools.runsSince(sinceTs)`) + `tool-guard.jsonl` (new `readGuardEvents(sinceTs)`) — per the plan, the KPI is never parsed out of conversation prose. Escape-hatch bypasses surface in the dream report only (no immediate chat pings — pings would train us both to ignore them). Test coverage: raw-op matrix incl. no-double-count with denies, event grouping/windowing, `runsSince` window math.
|
|
7
|
+
- **Surfaced tools now carry their condition (tiered surfacing, Phase 2/item 6).** The per-turn "Tools that may help here" block rendered one identity line per tool — name, description, keyring needs — so the agent knew a tool *existed* but not what state it was in, and had to spend a `tool show` (or just run it blind) to find out it was broken, never proven, or failing since Tuesday. Each surfaced tool is now a **2-line headline**: line 1 is identity (name — description, risk tier, keyring needs, why it surfaced), line 2 is condition — the TOOL.md **State** headline (first line, truncated), the open **known-issue count** (⚠ n), and **last-run health** from telemetry (`last run OK/FAILED (exit n) <date>, n runs`, or `never run`; legacy flat tools without exit telemetry degrade to `last used`). This completes the tiering ladder: 2-line headline always → TOOL.md via `tool show` → source only via `--source` — you read the manual, not the disassembly, and now the cover tells you if the manual has errata. New `tools.toolCondition()` with unit coverage in `test-tools.js` (headline extraction, `(none yet)` filtering, health formats, clean degrade) and end-to-end render assertions in `test-recall-discoverer.js`.
|
|
8
|
+
|
|
3
9
|
## v2.7.2
|
|
4
10
|
- **Credential confinement — keyring creds now live ONLY inside tool runs (enforcement 5b).** Until now `botSubprocessEnv()` merged the whole operational keyring into *every* agent subprocess, so the agent's own Bash (and any sub-agent) could `curl` a prod API with `$inet_central_user` directly — the tool layer was optional. That merge is gone: agent shells and sub-agents run cred-free, and `tools.runEnv(tool)` injects **only the keys a tool declares via `--requires`**, read fresh from the keyring at run time (least privilege; `process.env` still wins on conflict). Audited all 11 live tools before shipping: every keyring key read is already declared, so nothing breaks. The system prompt's Preauth bullet now tells the truth, and `keyring list` no longer claims env-wide availability.
|
|
5
11
|
- **Tool-first deny-gate on the agent's shell (enforcement 5b).** A PreToolUse hook (wired via `--settings` into the main runner *and* sub-agent spawns; new `core/tool-guard.js` + `open-claudia deny-gate-hook`) blocks raw operational commands against known prod surfaces and redirects to the covering tool by name — or to the exact `tool scaffold` command when nothing covers the surface yet. v1 rules are deliberately high-precision: an **acting** binary (curl/wget/ssh/mongosh/…) aimed at `central.inet.africa`, `ticketcentral`, `10.200.0.0/16`, `kubectl exec … mongo(sh)`, or `ssh codersafrica@` — while `grep`/`cat`/`ping` mentioning the same surfaces, read-only `kubectl`, and plain `git push` stay free (reading is always allowed; acting goes through tools). Quotes count as command separators so `bash -c "curl …"` can't dodge the gate. Every failure mode in the gate fails OPEN — a broken guard must never brick the shell. This gate is the honest enforcement for surfaces where confinement is toothless (kubectl/ssh use ambient creds, not keyring keys); the accepted residual dodge is writing the command into a script file first.
|
package/core/dream.js
CHANGED
|
@@ -1192,6 +1192,42 @@ async function runDream({ trigger = "manual" } = {}) {
|
|
|
1192
1192
|
if (dupPairs.size) {
|
|
1193
1193
|
bits.push(`👯 ${dupPairs.size} near-duplicate tool pair(s): ${[...dupPairs].join(", ")} — consider merging into one tool with subcommands.`);
|
|
1194
1194
|
}
|
|
1195
|
+
// Doc/code drift: source edited after TOOL.md, or verbs in real use that
|
|
1196
|
+
// the docs never mention. Flag only — the fix is a tool note.
|
|
1197
|
+
const drifted = allTools
|
|
1198
|
+
.map((t) => { const d = toolsLib.docDrift(t); return d ? `${t.name} (${d})` : null; })
|
|
1199
|
+
.filter(Boolean);
|
|
1200
|
+
if (drifted.length) {
|
|
1201
|
+
bits.push(`📝 ${drifted.length} tool(s) with doc/code drift: ${drifted.join("; ")} — refresh the docs with tool note.`);
|
|
1202
|
+
}
|
|
1203
|
+
// 5c usage KPI: what fraction of operational actions went through the
|
|
1204
|
+
// authorised channel, computed from telemetry stamps (state.json run
|
|
1205
|
+
// history) + the guard log — never prose parsing. Denominator counts
|
|
1206
|
+
// COMPLETED actions: tool runs + logged escape-hatch bypasses + raw-op
|
|
1207
|
+
// sightings; gate denies are reported separately (blocked ≠ completed).
|
|
1208
|
+
try {
|
|
1209
|
+
const guardLib = require("./tool-guard");
|
|
1210
|
+
const sinceTs = dreamState.lastRun || new Date(Date.now() - 7 * 86400000).toISOString();
|
|
1211
|
+
const g = guardLib.readGuardEvents(sinceTs);
|
|
1212
|
+
const r = toolsLib.runsSince(sinceTs);
|
|
1213
|
+
const ops = r.runs + g.bypasses.length + g.rawOps.length;
|
|
1214
|
+
if (ops) {
|
|
1215
|
+
const pct = Math.round((100 * r.runs) / ops);
|
|
1216
|
+
const kpiBits = [`${pct}% of operational actions via tool run (${r.runs}/${ops}, target >90%)`];
|
|
1217
|
+
if (g.bypasses.length) {
|
|
1218
|
+
const reasons = g.bypasses.map((b) => b.reason).filter(Boolean).slice(0, 3).join("; ");
|
|
1219
|
+
kpiBits.push(`${g.bypasses.length} escape-hatch bypass(es)${reasons ? ` — ${reasons}` : ""}`);
|
|
1220
|
+
}
|
|
1221
|
+
if (g.rawOps.length) {
|
|
1222
|
+
const byPattern = {};
|
|
1223
|
+
for (const evt of g.rawOps) byPattern[evt.pattern || "?"] = (byPattern[evt.pattern || "?"] || 0) + 1;
|
|
1224
|
+
kpiBits.push(`${g.rawOps.length} raw op(s) outside tools: ${Object.entries(byPattern).map(([k, v]) => `${k} ×${v}`).join(", ")}`);
|
|
1225
|
+
}
|
|
1226
|
+
if (g.denies.length) kpiBits.push(`${g.denies.length} denied by the gate`);
|
|
1227
|
+
if (r.failures) kpiBits.push(`${r.failures} failed run(s) at use: ${r.failedTools.join(", ")}`);
|
|
1228
|
+
bits.push(`📈 Tool usage since last dream: ${kpiBits.join(" · ")}.`);
|
|
1229
|
+
}
|
|
1230
|
+
} catch (e) { /* KPI is best-effort */ }
|
|
1195
1231
|
toolNote = bits.join("\n");
|
|
1196
1232
|
} catch (e) { /* tool graph is best-effort (old node) */ }
|
|
1197
1233
|
|
|
@@ -515,12 +515,21 @@ async function run(ctx) {
|
|
|
515
515
|
}
|
|
516
516
|
let toolBlock = "";
|
|
517
517
|
if (toolMatches.length) {
|
|
518
|
+
// Tiered injection (Phase 2): a 2-line headline per tool — identity line
|
|
519
|
+
// (name, description, risk tier, keyring needs, why surfaced) plus a
|
|
520
|
+
// condition line (TOOL.md State headline, open issues, last-run health) so
|
|
521
|
+
// the agent knows the tool's shape AND state without loading the manual.
|
|
518
522
|
const lines = toolMatches.map((m) => {
|
|
519
523
|
const t = toolsLib.findTool(m.name);
|
|
520
|
-
const
|
|
521
|
-
|
|
524
|
+
const meta = [];
|
|
525
|
+
if (t && t.risk) meta.push(t.risk);
|
|
526
|
+
if (t && t.requires && t.requires.length) meta.push(`keyring: ${t.requires.join(", ")}`);
|
|
527
|
+
const head = `- ${m.name} — ${m.description || "(no description)"}${meta.length ? ` (${meta.join("; ")})` : ""}${m.why ? ` — ${m.why}` : ""}`;
|
|
528
|
+
let cond = "";
|
|
529
|
+
try { cond = toolsLib.toolCondition(t); } catch (e) {}
|
|
530
|
+
return cond ? `${head}\n ↳ ${cond}` : head;
|
|
522
531
|
});
|
|
523
|
-
toolBlock = `\n\n## Tools that may help here\nSurfaced for this turn —
|
|
532
|
+
toolBlock = `\n\n## Tools that may help here\nSurfaced for this turn — read the manual with \`open-claudia tool show <name>\`, then run with \`open-claudia tool run <name> [args]\`:\n${lines.join("\n")}\n`;
|
|
524
533
|
}
|
|
525
534
|
|
|
526
535
|
// 5a tool-scout: replace the silence of an empty/partial tool match with an
|
package/core/runner.js
CHANGED
|
@@ -958,6 +958,11 @@ async function runClaude(prompt, cwd, replyToMsgId, opts = {}) {
|
|
|
958
958
|
const noteRecallFromShell = (command) => {
|
|
959
959
|
try {
|
|
960
960
|
const cmd = String(command || "");
|
|
961
|
+
// 5c raw-op audit: every Bash command in the stream passes here, so
|
|
962
|
+
// operational work done raw — heredoc scripts, DB clients, mutating
|
|
963
|
+
// kubectl — gets stamped to the guard log for the dream's KPI pass.
|
|
964
|
+
// Observation only; blocking is the deny-gate hook's job, not this.
|
|
965
|
+
try { require("./tool-guard").noteRawOp(cmd); } catch (e) { /* best-effort */ }
|
|
961
966
|
if (!cmd.includes("open-claudia")) return;
|
|
962
967
|
let m;
|
|
963
968
|
const packRe = /\bpack\s+show\s+["']?([a-z0-9][\w.-]*)/gi;
|
package/core/tool-guard.js
CHANGED
|
@@ -107,6 +107,76 @@ function logGuardEvent(evt) {
|
|
|
107
107
|
} catch (e) { /* best effort */ }
|
|
108
108
|
}
|
|
109
109
|
|
|
110
|
+
// ── 5c raw-op audit (observation, never blocking) ──────────────────────────
|
|
111
|
+
// The deny-gate above catches acting commands aimed at KNOWN prod surfaces;
|
|
112
|
+
// this detector catches the SHAPE of operational work done raw that should
|
|
113
|
+
// have been a tool — inline heredoc scripts fed to an interpreter, raw DB
|
|
114
|
+
// clients, mutating kubectl. Same high-precision-over-recall doctrine as the
|
|
115
|
+
// gate: reads and probes (kubectl get/logs, grep, ping) stay unflagged, so
|
|
116
|
+
// the dream's KPI measures real operational bypassing, not diagnosis noise.
|
|
117
|
+
const RAW_OP_PATTERNS = [
|
|
118
|
+
{
|
|
119
|
+
// An interpreter or DB shell fed an inline heredoc script — the exact
|
|
120
|
+
// "re-derive the script instead of reusing a tool" smell. cat/tee heredocs
|
|
121
|
+
// (file writes) deliberately don't match.
|
|
122
|
+
id: "heredoc-interpreter",
|
|
123
|
+
re: /(?:^|[|;&(`'"]|\$\()\s*(?:sudo\s+)?(?:node|python3?|ruby|perl|php|deno|bun|mongo|mongosh|psql|mysql|redis-cli)\b[^\n]*<</,
|
|
124
|
+
},
|
|
125
|
+
{
|
|
126
|
+
id: "db-client",
|
|
127
|
+
re: /(?:^|[|;&(`'"]|\$\()\s*(?:sudo\s+)?(?:mongo|mongosh|psql|mysql|redis-cli)\b/,
|
|
128
|
+
},
|
|
129
|
+
{
|
|
130
|
+
// Cluster mutations raw from the shell (GitOps or the prod-k8s tool are
|
|
131
|
+
// the channels). kubectl get/describe/logs/top/events/port-forward and
|
|
132
|
+
// rollout status/history are reads — always free per the hard rules.
|
|
133
|
+
id: "kubectl-mutate",
|
|
134
|
+
re: /\bkubectl\s+(?:exec|apply|delete|scale|patch|edit|create|replace|cordon|uncordon|drain|taint|label|annotate|set)\b|\bkubectl\s+rollout\s+(?:restart|undo|pause|resume)\b/,
|
|
135
|
+
},
|
|
136
|
+
];
|
|
137
|
+
|
|
138
|
+
// Pure: null, or the matched raw-op pattern. The authorised channels and
|
|
139
|
+
// anything the deny-gate owns (already logged as a deny by the hook) are
|
|
140
|
+
// skipped so one command never produces two guard events.
|
|
141
|
+
function matchRawOp(command) {
|
|
142
|
+
const cmd = String(command || "");
|
|
143
|
+
if (!cmd.trim()) return null;
|
|
144
|
+
if (ALLOW_RE.test(cmd)) return null;
|
|
145
|
+
if (matchRule(cmd)) return null;
|
|
146
|
+
for (const p of RAW_OP_PATTERNS) if (p.re.test(cmd)) return p;
|
|
147
|
+
return null;
|
|
148
|
+
}
|
|
149
|
+
|
|
150
|
+
// Stamp a sighting to the guard log. Called from the runner's Bash-stream
|
|
151
|
+
// chokepoint on every command; must never throw into the stream parser.
|
|
152
|
+
function noteRawOp(command) {
|
|
153
|
+
try {
|
|
154
|
+
const p = matchRawOp(command);
|
|
155
|
+
if (!p) return null;
|
|
156
|
+
logGuardEvent({ kind: "raw-op", pattern: p.id, command: String(command).slice(0, 300) });
|
|
157
|
+
return p;
|
|
158
|
+
} catch (e) { return null; }
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
// KPI data source (5c): guard events since a timestamp, grouped by kind.
|
|
162
|
+
// Malformed lines are skipped — the log is append-only best-effort.
|
|
163
|
+
function readGuardEvents(sinceTs) {
|
|
164
|
+
const out = { denies: [], bypasses: [], rawOps: [] };
|
|
165
|
+
let raw = "";
|
|
166
|
+
try { raw = fs.readFileSync(GUARD_LOG, "utf-8"); } catch (e) { return out; }
|
|
167
|
+
const since = sinceTs ? Date.parse(sinceTs) : NaN;
|
|
168
|
+
for (const line of raw.split("\n")) {
|
|
169
|
+
if (!line.trim()) continue;
|
|
170
|
+
let evt;
|
|
171
|
+
try { evt = JSON.parse(line); } catch (e) { continue; }
|
|
172
|
+
if (!Number.isNaN(since) && (!evt.ts || Date.parse(evt.ts) < since)) continue;
|
|
173
|
+
if (evt.kind === "deny") out.denies.push(evt);
|
|
174
|
+
else if (evt.kind === "bypass") out.bypasses.push(evt);
|
|
175
|
+
else if (evt.kind === "raw-op") out.rawOps.push(evt);
|
|
176
|
+
}
|
|
177
|
+
return out;
|
|
178
|
+
}
|
|
179
|
+
|
|
110
180
|
// Hook entry, pure enough to test: takes the PreToolUse stdin payload (object
|
|
111
181
|
// or JSON string), returns { exit, stderr }. exit 2 = deny (stderr is fed back
|
|
112
182
|
// to the model); exit 0 = allow. Anything unexpected → allow (fail-open).
|
|
@@ -160,4 +230,8 @@ function ensureHookSettings() {
|
|
|
160
230
|
return SETTINGS_FILE;
|
|
161
231
|
}
|
|
162
232
|
|
|
163
|
-
module.exports = {
|
|
233
|
+
module.exports = {
|
|
234
|
+
RULES, RAW_OP_PATTERNS, GUARD_LOG, SETTINGS_FILE,
|
|
235
|
+
matchRule, denyMessage, logGuardEvent, runHook, hookSettings, ensureHookSettings,
|
|
236
|
+
matchRawOp, noteRawOp, readGuardEvents,
|
|
237
|
+
};
|
package/core/tools.js
CHANGED
|
@@ -316,6 +316,40 @@ function readToolDoc(name) {
|
|
|
316
316
|
try { return fs.readFileSync(p.doc, "utf-8"); } catch (e) { return ""; }
|
|
317
317
|
}
|
|
318
318
|
|
|
319
|
+
function sectionOf(doc, heading) {
|
|
320
|
+
const m = String(doc || "").match(new RegExp(`##\\s*${heading}[^\\n]*\\n([\\s\\S]*?)(?=\\n##\\s|$)`, "i"));
|
|
321
|
+
return m ? m[1] : "";
|
|
322
|
+
}
|
|
323
|
+
|
|
324
|
+
// One-line condition digest for the per-turn surfacing block (tiered injection:
|
|
325
|
+
// headline always → TOOL.md via `tool show` → source only via --source). Rides
|
|
326
|
+
// under the tool's headline so the agent knows its condition BEFORE running it:
|
|
327
|
+
// TOOL.md's State headline, open known-issue count, last-run health from
|
|
328
|
+
// telemetry. "" when there is nothing worth saying.
|
|
329
|
+
function toolCondition(t) {
|
|
330
|
+
if (!t || !t.name) return "";
|
|
331
|
+
const parts = [];
|
|
332
|
+
const doc = readToolDoc(t.name);
|
|
333
|
+
if (doc) {
|
|
334
|
+
const headline = sectionOf(doc, "State").split("\n")
|
|
335
|
+
.map((l) => l.trim())
|
|
336
|
+
.find((l) => l && !/^\(none/i.test(l) && !l.startsWith("#"));
|
|
337
|
+
if (headline) parts.push(headline.length > 120 ? `${headline.slice(0, 117)}…` : headline);
|
|
338
|
+
const issues = sectionOf(doc, "Known issues").split("\n").filter((l) => /^\s*-\s*\S/.test(l)).length;
|
|
339
|
+
if (issues) parts.push(`⚠ ${issues} known issue${issues === 1 ? "" : "s"}`);
|
|
340
|
+
}
|
|
341
|
+
if (!t.runCount) parts.push("never run");
|
|
342
|
+
else {
|
|
343
|
+
const when = (t.lastUsed || "").slice(0, 10);
|
|
344
|
+
// Legacy flat tools have no lastExit — usage count is all we can say.
|
|
345
|
+
const health = typeof t.lastExit === "number"
|
|
346
|
+
? (t.lastExit === 0 ? "last run OK" : `last run FAILED (exit ${t.lastExit})`)
|
|
347
|
+
: "last used";
|
|
348
|
+
parts.push(`${health}${when ? ` ${when}` : ""}, ${t.runCount} run${t.runCount === 1 ? "" : "s"}`);
|
|
349
|
+
}
|
|
350
|
+
return parts.join(" · ");
|
|
351
|
+
}
|
|
352
|
+
|
|
319
353
|
function listTools() {
|
|
320
354
|
let entries;
|
|
321
355
|
try { entries = fs.readdirSync(TOOLS_DIR); } catch (e) { return []; }
|
|
@@ -341,6 +375,44 @@ function findTool(name) {
|
|
|
341
375
|
return readTool(needle);
|
|
342
376
|
}
|
|
343
377
|
|
|
378
|
+
// Window telemetry for the 5c usage KPI: `tool run` executions since a
|
|
379
|
+
// timestamp, counted from state.json history entries — the CLI's authoritative
|
|
380
|
+
// per-run stamps, never prose parsing. History is capped per tool, so a tool
|
|
381
|
+
// that ran more than HISTORY_CAP times in the window undercounts slightly;
|
|
382
|
+
// acceptable for a nightly ratio.
|
|
383
|
+
function runsSince(sinceTs) {
|
|
384
|
+
const since = sinceTs ? Date.parse(sinceTs) : NaN;
|
|
385
|
+
const out = { runs: 0, failures: 0, byTool: {}, failedTools: [] };
|
|
386
|
+
for (const t of listTools()) {
|
|
387
|
+
const hist = Array.isArray(t.history) ? t.history : [];
|
|
388
|
+
const inWindow = hist.filter((h) => h && h.at && (Number.isNaN(since) || Date.parse(h.at) >= since));
|
|
389
|
+
if (!inWindow.length) continue;
|
|
390
|
+
const fails = inWindow.filter((h) => h.exit !== 0).length;
|
|
391
|
+
out.runs += inWindow.length;
|
|
392
|
+
out.failures += fails;
|
|
393
|
+
out.byTool[t.name] = { runs: inWindow.length, failures: fails };
|
|
394
|
+
if (fails) out.failedTools.push(`${t.name} ×${fails}`);
|
|
395
|
+
}
|
|
396
|
+
return out;
|
|
397
|
+
}
|
|
398
|
+
|
|
399
|
+
// Doc/code drift check for the dream's tool pass: the source was edited after
|
|
400
|
+
// TOOL.md was last touched (docs describe an older tool), or a verb telemetry
|
|
401
|
+
// proves is in real use never appears in the docs. Returns a short reason or
|
|
402
|
+
// null. The 60s slack absorbs creation-time write ordering (source + doc
|
|
403
|
+
// skeleton land moments apart).
|
|
404
|
+
function docDrift(t) {
|
|
405
|
+
if (!t || t.layout !== "dir" || !t.docFile) return null;
|
|
406
|
+
try {
|
|
407
|
+
if (fs.statSync(t.file).mtimeMs > fs.statSync(t.docFile).mtimeMs + 60000) return "source newer than TOOL.md";
|
|
408
|
+
} catch (e) { /* doc or source unreadable → fall through to verb check */ }
|
|
409
|
+
const doc = readToolDoc(t.name) || "";
|
|
410
|
+
const verbs = Object.keys(t.verbs || {}).filter((v) => v && v !== "(default)" && v !== "help" && !v.startsWith("-"));
|
|
411
|
+
const undocd = verbs.filter((v) => !doc.includes(v)).slice(0, 3);
|
|
412
|
+
if (undocd.length) return `verb${undocd.length === 1 ? "" : "s"} ${undocd.join(", ")} used but undocumented`;
|
|
413
|
+
return null;
|
|
414
|
+
}
|
|
415
|
+
|
|
344
416
|
// Build the header comment block for a script that doesn't already declare one.
|
|
345
417
|
// Uses the comment prefix that matches the script's shebang/extension.
|
|
346
418
|
function buildHeader(prefix, { name, description, pack, requires, usage, risk }) {
|
|
@@ -678,7 +750,7 @@ function toolNameFromPath(filePath) {
|
|
|
678
750
|
|
|
679
751
|
module.exports = {
|
|
680
752
|
TOOLS_DIR, MARKER, USAGE_FILE, RISK_TIERS,
|
|
681
|
-
parseHeader, readTool, readToolDoc, listTools, findTool, addTool, removeTool, updateToolHeader,
|
|
753
|
+
parseHeader, readTool, readToolDoc, toolCondition, listTools, findTool, runsSince, docDrift, addTool, removeTool, updateToolHeader,
|
|
682
754
|
runEnv, missingRequires, toolNameFromPath, sanitizeName, ensureDir, normalizeRisk,
|
|
683
755
|
toolPaths, readState, writeState, recordRunResult, runGate,
|
|
684
756
|
matchTools, recordRun, recordCreate, readUsage,
|
package/package.json
CHANGED
|
@@ -26,6 +26,16 @@ toolsLib.addTool((() => {
|
|
|
26
26
|
fs.writeFileSync(s, "#!/usr/bin/env bash\necho audit\n");
|
|
27
27
|
return s;
|
|
28
28
|
})(), { name: "mikrotik-audit", description: "audit the mikrotik router fleet inventory" });
|
|
29
|
+
// Give the fixture a condition (Phase 2 tiered surfacing): TOOL.md State
|
|
30
|
+
// headline + a known issue + failed-run telemetry, so the 2-line headline can
|
|
31
|
+
// be asserted end-to-end through the renderer.
|
|
32
|
+
fs.writeFileSync(path.join(process.env.TOOLS_DIR, "mikrotik-audit", "TOOL.md"), [
|
|
33
|
+
"# mikrotik-audit", "", "## Interface", "audit the mikrotik router fleet inventory", "",
|
|
34
|
+
"## Known issues", "- [2026-07-01] R2 times out on list", "",
|
|
35
|
+
"## State", "Covers 12 routers; R2 flaky.", "",
|
|
36
|
+
"## Journal", "- [2026-07-01] Created.", "",
|
|
37
|
+
].join("\n"));
|
|
38
|
+
toolsLib.writeState("mikrotik-audit", { createdAt: "2026-07-01T00:00:00Z", lastUsed: "2026-07-01T10:00:00.000Z", runCount: 3, lastExit: 1 });
|
|
29
39
|
|
|
30
40
|
// --- tiered pre-gate ---
|
|
31
41
|
assert.strictEqual(disc.needsRecall("", 0), false, "empty → no recall");
|
|
@@ -114,6 +124,12 @@ const helpers = { packsLib, entitiesLib, mergeMatches, buildPackBlock, buildEnti
|
|
|
114
124
|
assert.ok(outTool.toolMatches.some((m) => m.name === "mikrotik-audit"), "matching tool is kept (seeds-tier user seed)");
|
|
115
125
|
assert.ok(/mikrotik-audit/.test(outTool.toolBlock), "kept tool renders into the toolBlock");
|
|
116
126
|
assert.ok(/Tools that may help here/.test(outTool.toolBlock), "toolBlock carries its heading");
|
|
127
|
+
// Phase 2 tiered surfacing: headline line 1 carries identity (risk tier),
|
|
128
|
+
// line 2 carries condition — State headline, issue count, last-run health.
|
|
129
|
+
assert.ok(/\(write\)/.test(outTool.toolBlock), "identity line carries the risk tier");
|
|
130
|
+
assert.ok(/↳ Covers 12 routers; R2 flaky\./.test(outTool.toolBlock), "condition line leads with the TOOL.md State headline");
|
|
131
|
+
assert.ok(/⚠ 1 known issue/.test(outTool.toolBlock), "condition line counts open known issues");
|
|
132
|
+
assert.ok(/last run FAILED \(exit 1\) 2026-07-01, 3 runs/.test(outTool.toolBlock), "condition line reports last-run health");
|
|
117
133
|
|
|
118
134
|
// full-tier fail-open: with no judge, an unjudged TOOL is a guess — dropped.
|
|
119
135
|
// (Measured before the fix: a failed walk kept 15 unjudged seeds including
|
package/test-tool-guard.js
CHANGED
|
@@ -68,6 +68,43 @@ assert.ok(lines.some((l) => l.kind === "deny" && l.rule === "inet-central-api"),
|
|
|
68
68
|
assert.ok(lines.some((l) => l.kind === "bypass" && l.reason === "no tool yet"), "keyring exec bypass is logged");
|
|
69
69
|
assert.ok(lines.every((l) => l.ts), "every guard event is timestamped");
|
|
70
70
|
|
|
71
|
+
// ── raw-op audit (5c): operational work done raw is observed, never blocked ──
|
|
72
|
+
assert.strictEqual(guard.matchRawOp("node <<'EOF'\nfetch('https://api.example.com/x')\nEOF").id, "heredoc-interpreter",
|
|
73
|
+
"inline node heredoc is the canonical re-derived-script smell");
|
|
74
|
+
assert.strictEqual(guard.matchRawOp("python3 <<PY\nprint(1)\nPY").id, "heredoc-interpreter");
|
|
75
|
+
assert.strictEqual(guard.matchRawOp('echo start && mongosh <<JS\ndb.x.find()\nJS').id, "heredoc-interpreter",
|
|
76
|
+
"heredoc after && still detected");
|
|
77
|
+
assert.strictEqual(guard.matchRawOp('mongosh "mongodb://db.internal/app" --eval "db.users.count()"').id, "db-client");
|
|
78
|
+
assert.strictEqual(guard.matchRawOp("psql -h db.internal -c 'select 1'").id, "db-client");
|
|
79
|
+
assert.strictEqual(guard.matchRawOp("kubectl delete pod api-0 -n prod").id, "kubectl-mutate");
|
|
80
|
+
assert.strictEqual(guard.matchRawOp("kubectl apply -f deploy.yaml").id, "kubectl-mutate");
|
|
81
|
+
assert.strictEqual(guard.matchRawOp("kubectl rollout restart deploy/api").id, "kubectl-mutate");
|
|
82
|
+
assert.strictEqual(guard.matchRawOp("kubectl exec api-0 -- cat /etc/config").id, "kubectl-mutate",
|
|
83
|
+
"raw kubectl exec is an op vector even for reads inside the pod");
|
|
84
|
+
assert.strictEqual(guard.matchRawOp("cat <<'EOF' > notes.md\nhello\nEOF"), null, "heredoc into a file write is not an op");
|
|
85
|
+
assert.strictEqual(guard.matchRawOp("kubectl get pods -A"), null, "read-only kubectl stays unflagged");
|
|
86
|
+
assert.strictEqual(guard.matchRawOp("kubectl rollout status deploy/api"), null, "rollout status is a read");
|
|
87
|
+
assert.strictEqual(guard.matchRawOp("kubectl logs api-7f9 -n prod"), null, "kubectl logs is a read");
|
|
88
|
+
assert.strictEqual(guard.matchRawOp("node script.js"), null, "running a checked-in script is not the heredoc smell");
|
|
89
|
+
assert.strictEqual(guard.matchRawOp("grep -rn mongosh docs/"), null, "mentioning a client binary mid-line is not acting");
|
|
90
|
+
assert.strictEqual(guard.matchRawOp("open-claudia tool run prod-k8s exec api-0 'cat /etc/config'"), null,
|
|
91
|
+
"authorised channel never flagged");
|
|
92
|
+
assert.strictEqual(guard.matchRawOp('open-claudia keyring exec --reason "x" -- mongosh mongodb://db/x'), null,
|
|
93
|
+
"escape hatch logs its own bypass, not a raw-op");
|
|
94
|
+
assert.strictEqual(guard.matchRawOp("kubectl exec mongo-0 -- mongosh"), null,
|
|
95
|
+
"deny-rule matches belong to the gate — one command never yields two guard events");
|
|
96
|
+
assert.strictEqual(guard.noteRawOp("kubectl delete pod api-1 -n prod").id, "kubectl-mutate", "noteRawOp returns the pattern");
|
|
97
|
+
assert.strictEqual(guard.noteRawOp("ls -la"), null, "benign commands log nothing");
|
|
98
|
+
|
|
99
|
+
// ── readGuardEvents: since-filtered grouping (the KPI's data source) ──
|
|
100
|
+
const evts = guard.readGuardEvents("2000-01-01T00:00:00Z");
|
|
101
|
+
assert.ok(evts.denies.length >= 1, "denies grouped");
|
|
102
|
+
assert.ok(evts.bypasses.length >= 1, "bypasses grouped");
|
|
103
|
+
assert.ok(evts.rawOps.some((e) => e.pattern === "kubectl-mutate"), "raw-ops grouped with their pattern");
|
|
104
|
+
assert.strictEqual(guard.readGuardEvents(new Date(Date.now() + 86400000).toISOString()).rawOps.length, 0,
|
|
105
|
+
"future since → empty window");
|
|
106
|
+
assert.ok(guard.readGuardEvents(null).denies.length >= 1, "null since → whole log");
|
|
107
|
+
|
|
71
108
|
// ── hook settings: a valid claude --settings payload, idempotent writer ──
|
|
72
109
|
const sPath = guard.ensureHookSettings();
|
|
73
110
|
assert.strictEqual(sPath, process.env.DENY_GATE_SETTINGS, "settings path is env-overridable (test isolation)");
|
package/test-tools.js
CHANGED
|
@@ -309,6 +309,59 @@ try {
|
|
|
309
309
|
assert.ok(fs.readFileSync(path.join(process.env.TOOLS_DIR, ".gitignore"), "utf-8").includes("state.json"), "state.json is gitignored");
|
|
310
310
|
} catch (e) { console.log(" (git not available — history assertions skipped)"); }
|
|
311
311
|
|
|
312
|
+
// ── toolCondition: the surfacing digest (Phase 2 tiered injection) — TOOL.md
|
|
313
|
+
// State headline + open-issue count + last-run health; degrades cleanly ──
|
|
314
|
+
const condSrc = path.join(tmp, "condtool.sh");
|
|
315
|
+
fs.writeFileSync(condSrc, "#!/usr/bin/env bash\necho ok\n");
|
|
316
|
+
tools.addTool(condSrc, { name: "condtool", description: "probes widgets" });
|
|
317
|
+
const freshCond = tools.toolCondition(tools.findTool("condtool"));
|
|
318
|
+
assert.ok(/Scaffolded .* — not yet proven/.test(freshCond), "fresh tool leads with the skeleton State line");
|
|
319
|
+
assert.ok(/never run/.test(freshCond), "zero runs read as never run");
|
|
320
|
+
assert.ok(!/known issue/.test(freshCond), "\"(none yet)\" placeholder is not an issue");
|
|
321
|
+
fs.writeFileSync(path.join(process.env.TOOLS_DIR, "condtool", "TOOL.md"), [
|
|
322
|
+
"# condtool", "", "## Interface", "probes widgets", "",
|
|
323
|
+
"## Known issues", "- [2026-07-01] list verb times out on slow links", "- [2026-07-02] json flag ignored", "",
|
|
324
|
+
"## State", "Covers 12 widget hosts; watch host-7.", "More detail that must not surface.", "",
|
|
325
|
+
"## Journal", "- [2026-07-01] Created.", "",
|
|
326
|
+
].join("\n"));
|
|
327
|
+
tools.writeState("condtool", { createdAt: "2026-07-01T00:00:00Z", lastUsed: "2026-07-02T10:00:00.000Z", runCount: 3, lastExit: 1 });
|
|
328
|
+
const cond = tools.toolCondition(tools.findTool("condtool"));
|
|
329
|
+
assert.ok(cond.startsWith("Covers 12 widget hosts; watch host-7."), "State headline (first line only) leads the digest");
|
|
330
|
+
assert.ok(!/must not surface/.test(cond), "only the State headline surfaces, not the whole section");
|
|
331
|
+
assert.ok(/⚠ 2 known issues/.test(cond), "open known-issue count rides along");
|
|
332
|
+
assert.ok(/last run FAILED \(exit 1\) 2026-07-02, 3 runs/.test(cond), "failed last run carries exit, date, and run count");
|
|
333
|
+
tools.writeState("condtool", { createdAt: "2026-07-01T00:00:00Z", lastUsed: "2026-07-03T09:00:00.000Z", runCount: 4, lastExit: 0 });
|
|
334
|
+
assert.ok(/last run OK 2026-07-03, 4 runs/.test(tools.toolCondition(tools.findTool("condtool"))), "healthy last run reads OK");
|
|
335
|
+
assert.strictEqual(tools.toolCondition(null), "", "no tool → empty digest");
|
|
336
|
+
|
|
337
|
+
// ── runsSince: windowed run counts from state.json history (5c KPI source) ──
|
|
338
|
+
tools.writeState("condtool", {
|
|
339
|
+
createdAt: "2026-07-01T00:00:00Z", lastUsed: "2099-01-02T00:00:00Z", runCount: 7, lastExit: 1,
|
|
340
|
+
history: [
|
|
341
|
+
{ at: "2099-01-02T00:00:00Z", args: "list --json", exit: 1, ms: 40 },
|
|
342
|
+
{ at: "2099-01-01T00:00:00Z", args: "list", exit: 0, ms: 35 },
|
|
343
|
+
{ at: "2000-01-01T00:00:00Z", args: "list", exit: 0, ms: 30 },
|
|
344
|
+
],
|
|
345
|
+
});
|
|
346
|
+
const win = tools.runsSince("2098-12-31T00:00:00Z");
|
|
347
|
+
assert.strictEqual(win.byTool.condtool.runs, 2, "only history entries inside the window count");
|
|
348
|
+
assert.strictEqual(win.byTool.condtool.failures, 1, "non-zero exits count as failures at use");
|
|
349
|
+
assert.ok(win.failedTools.includes("condtool ×1"), "failing tools are named for the dream report");
|
|
350
|
+
assert.strictEqual(tools.runsSince("2099-06-01T00:00:00Z").runs, 0, "empty window → zero runs");
|
|
351
|
+
|
|
352
|
+
// ── docDrift: docs stale vs source/telemetry (dream tool-pass flag) ──
|
|
353
|
+
assert.strictEqual(tools.docDrift(tools.findTool("condtool")), null, "fresh docs → no drift");
|
|
354
|
+
tools.writeState("condtool", {
|
|
355
|
+
createdAt: "2026-07-01T00:00:00Z", lastUsed: "2099-01-02T00:00:00Z", runCount: 7, lastExit: 0,
|
|
356
|
+
verbs: { list: { runs: 3 }, stages: { runs: 1 }, help: { runs: 1 }, "(default)": { runs: 1 } },
|
|
357
|
+
});
|
|
358
|
+
assert.strictEqual(tools.docDrift(tools.findTool("condtool")), "verb stages used but undocumented",
|
|
359
|
+
"a verb in real use that TOOL.md never mentions is drift (help/(default) ignored)");
|
|
360
|
+
const future = new Date(Date.now() + 3600000);
|
|
361
|
+
fs.utimesSync(tools.findTool("condtool").file, future, future);
|
|
362
|
+
assert.strictEqual(tools.docDrift(tools.findTool("condtool")), "source newer than TOOL.md", "source edited after docs is drift");
|
|
363
|
+
assert.strictEqual(tools.docDrift(null), null, "no tool → no drift");
|
|
364
|
+
|
|
312
365
|
// ── remove ──
|
|
313
366
|
assert.ok(tools.removeTool("hello"), "removeTool returns the removed tool");
|
|
314
367
|
assert.strictEqual(tools.findTool("hello"), null, "removed tool is gone");
|