vigiles 12.0.0 → 12.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. package/README.md +22 -11
  2. package/action.yml +73 -0
  3. package/dist/audit-report.d.ts +11 -0
  4. package/dist/audit-report.js +1 -0
  5. package/dist/audit-report.template.html +36 -26
  6. package/dist/claude-code.d.ts +2 -0
  7. package/dist/claude-code.js +9 -1
  8. package/dist/cli-commands.d.ts +1 -1
  9. package/dist/cli-commands.js +0 -1
  10. package/dist/cli.js +96 -135
  11. package/dist/core/rule-meta.js +8 -0
  12. package/dist/core/skill-description-budget.d.ts +42 -0
  13. package/dist/core/skill-description-budget.js +47 -0
  14. package/dist/core/types.d.ts +9 -0
  15. package/dist/core/validate.js +3 -0
  16. package/dist/doc-command-coverage.d.ts +20 -0
  17. package/dist/doc-command-coverage.js +60 -0
  18. package/dist/eval-cost.d.ts +75 -0
  19. package/dist/eval-cost.js +134 -0
  20. package/dist/eval.d.ts +4 -0
  21. package/dist/eval.js +46 -6
  22. package/dist/observe.d.ts +109 -0
  23. package/dist/observe.js +164 -0
  24. package/dist/research-index.d.ts +31 -0
  25. package/dist/research-index.js +48 -0
  26. package/dist/scaffold-test.js +3 -2
  27. package/dist/scan-behavioral.d.ts +42 -0
  28. package/dist/scan-behavioral.js +67 -0
  29. package/dist/scan.d.ts +3 -23
  30. package/dist/scan.js +18 -69
  31. package/dist/setup-plan.d.ts +1 -1
  32. package/dist/setup-plan.js +1 -0
  33. package/package.json +1 -1
  34. package/skills/adopt-spec/SKILL.md +10 -1
  35. package/skills/debug-my-harness/SKILL.md +56 -0
  36. package/skills/edit-spec/SKILL.md +1 -0
  37. package/skills/strengthen/SKILL.md +4 -0
  38. package/skills/test-harness/SKILL.md +17 -0
  39. package/dist/core/hook-spec.d.ts +0 -74
  40. package/dist/core/hook-spec.js +0 -130
@@ -0,0 +1,48 @@
1
+ "use strict";
2
+ /**
3
+ * Research-index completeness — the deterministic FLOOR keeping the `research/`
4
+ * corpus and its index (`research/CLAUDE.md.spec.ts`) in sync. The spec's
5
+ * `keyFiles` map is the AGENT-FACING index of every research doc; the compiler
6
+ * already verifies the OTHER direction (every indexed path EXISTS, else
7
+ * `vigiles compile` fails), so the only open gap is a doc that was ADDED but
8
+ * never indexed. This check closes it: every `research/*.md` (except the
9
+ * human-facing `README.md`) must appear in the index, or the dogfood test fails.
10
+ *
11
+ * Pure — the caller supplies the doc filenames and the index content (the spec
12
+ * source, where an entry is AUTHORED), so it runs over the real `research/` dir
13
+ * in a test or over any file set. Bidirectional sync = compiler (index ⊆ docs)
14
+ * + this check (docs ⊆ index).
15
+ */
16
+ Object.defineProperty(exports, "__esModule", { value: true });
17
+ exports.INDEX_EXEMPT = void 0;
18
+ exports.unindexedResearchDocs = unindexedResearchDocs;
19
+ exports.deadIndexEntries = deadIndexEntries;
20
+ /** Docs that are the index itself / human front-door, not indexed entries. */
21
+ exports.INDEX_EXEMPT = ["README.md"];
22
+ /**
23
+ * Research doc basenames (e.g. `roadmap.md`) NOT referenced anywhere in
24
+ * `indexContent`. A doc counts as indexed if its repo-relative path
25
+ * (`research/<name>.md`) appears in the index — the exact form the spec's
26
+ * `keyFiles` keys use. Exempt docs (the README) are never flagged.
27
+ */
28
+ function unindexedResearchDocs(docFilenames, indexContent, exempt = exports.INDEX_EXEMPT) {
29
+ return docFilenames.filter((name) => !exempt.includes(name) && !indexContent.includes(`research/${name}`));
30
+ }
31
+ /**
32
+ * Index entries pointing at a `research/<name>.md` that no longer exists on
33
+ * disk. The compiler catches this at compile time (a missing `keyFiles` path is
34
+ * a compile error), so this is a belt-and-suspenders reader for a test that
35
+ * wants to assert it directly without invoking the compiler.
36
+ */
37
+ function deadIndexEntries(docFilenames, indexContent) {
38
+ const present = new Set(docFilenames);
39
+ const refs = indexContent.matchAll(/research\/([\w.-]+\.md)/g);
40
+ const dead = new Set();
41
+ for (const m of refs) {
42
+ const name = m[1];
43
+ if (!present.has(name))
44
+ dead.add(name);
45
+ }
46
+ return [...dead];
47
+ }
48
+ //# sourceMappingURL=research-index.js.map
@@ -3,7 +3,8 @@ Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.scaffoldTest = scaffoldTest;
4
4
  exports.formatScaffolds = formatScaffolds;
5
5
  /**
6
- * `vigiles scaffold-test` — the deterministic test-gen engine (B1 v0).
6
+ * The deterministic test-gen engine (B1 v0) — skill-internal (the `test-harness`
7
+ * skill drives it; there is no standalone CLI verb).
7
8
  *
8
9
  * Free-form in, a RUNNABLE starter test out. Given an existing hand-written
9
10
  * skill / subagent / hook, emit a scaffolded `*.harness.mjs` / `*.eval.mjs` at the
@@ -35,7 +36,7 @@ function header(title, run) {
35
36
  "/**",
36
37
  ` * ${title}`,
37
38
  " *",
38
- " * Generated by `vigiles scaffold-test` — a STARTER, not a finished test. Fill in",
39
+ " * Generated by vigiles (the test-harness skill) — a STARTER, not a finished test. Fill in",
39
40
  " * the TODOs (they're where a human/model must supply judgement), then run:",
40
41
  ` * ${run}`,
41
42
  " */",
@@ -135,6 +135,48 @@ export declare function measurePluginSelectionWith(dir: string, promptSet: Trigg
135
135
  export declare function measurePluginSelection(dir: string, promptSet: TriggerPromptSet, opts?: SelectionOptions): Promise<SelectionReport>;
136
136
  /** Format the selection-collision matrix as a scan-report section. */
137
137
  export declare function formatSelectionReport(r: SelectionReport): string;
138
+ /** Options for {@link measureSelectionMatrix}: {@link SelectionOptions} plus an
139
+ * optional explicit prompt set (auto-derived from descriptions when omitted). */
140
+ export interface SelectionMatrixOptions extends SelectionOptions {
141
+ /**
142
+ * Per-skill recall prompts. Omit for ZERO-SETUP — prompts are auto-derived
143
+ * from each skill's description (the same generator the audit trigger tier
144
+ * uses). Supply your own for a curated collision benchmark; only the `prompts`
145
+ * array per skill is read (any `irrelevant` bank is ignored here).
146
+ */
147
+ readonly prompts?: TriggerPromptSet;
148
+ }
149
+ /**
150
+ * Measure a plugin's skill-SELECTION collision matrix — "when I ask for skill i's
151
+ * job, does ONLY skill i fire?" The first-class, assertable form of the cross-skill
152
+ * collision measurement (pair with {@link assertNoCollision}). The matrix diagonal
153
+ * is recall; off-diagonal mass is collision — skill j hijacking skill i's prompt,
154
+ * the failure that breaks a multi-skill plugin and that per-skill trigger-rate
155
+ * (each skill in ISOLATION) structurally can't see.
156
+ *
157
+ * ZERO-SETUP: with no `prompts`, they're derived from each model-invocable skill's
158
+ * description. Claude Code only (Codex has no skill-selection event to read); needs
159
+ * the `claude` CLI + model auth, else `available: false`. Thin promotion of
160
+ * {@link measurePluginSelection}. See research/plugin-selection-collision.md.
161
+ */
162
+ export declare function measureSelectionMatrix(dir: string, opts?: SelectionMatrixOptions): Promise<SelectionReport>;
163
+ /**
164
+ * Injectable core of {@link measureSelectionMatrix} (for tests): auto-derive the
165
+ * prompts (unless supplied) and drive the matrix via a fake/real probe.
166
+ */
167
+ export declare function measureSelectionMatrixWith(dir: string, probe: HarnessProbe, opts?: SelectionMatrixOptions): Promise<SelectionReport>;
168
+ /**
169
+ * Assert a plugin's skills don't hijack each other — the gate over a
170
+ * {@link SelectionReport} from {@link measureSelectionMatrix}. `maxOffDiagonal`
171
+ * caps EACH skill's collision rate (fraction of its own prompts on which a SIBLING
172
+ * fired); `maxPluginCollision` caps the plugin-wide rate. With neither set it
173
+ * demands ZERO collision. THROWS (never a silent green) when nothing was measured
174
+ * — an unavailable harness or a zero-run report is a gap, not a pass.
175
+ */
176
+ export declare function assertNoCollision(report: SelectionReport, opts?: {
177
+ maxOffDiagonal?: number;
178
+ maxPluginCollision?: number;
179
+ }): void;
138
180
  /** Does a skill description assert a hard constraint (→ an adversarial-gate candidate)? */
139
181
  export declare function isGateDescription(description: string): boolean;
140
182
  /** A skill considered for gate detection — name + its (model-visible) description. */
@@ -22,6 +22,9 @@ exports.buildSelectionReport = buildSelectionReport;
22
22
  exports.measurePluginSelectionWith = measurePluginSelectionWith;
23
23
  exports.measurePluginSelection = measurePluginSelection;
24
24
  exports.formatSelectionReport = formatSelectionReport;
25
+ exports.measureSelectionMatrix = measureSelectionMatrix;
26
+ exports.measureSelectionMatrixWith = measureSelectionMatrixWith;
27
+ exports.assertNoCollision = assertNoCollision;
25
28
  exports.isGateDescription = isGateDescription;
26
29
  exports.detectGateSkills = detectGateSkills;
27
30
  exports.gateRubric = gateRubric;
@@ -33,6 +36,7 @@ const node_os_1 = require("node:os");
33
36
  const node_path_1 = require("node:path");
34
37
  const node_child_process_1 = require("node:child_process");
35
38
  const scan_js_1 = require("./scan.js");
39
+ const audit_prompts_js_1 = require("./audit-prompts.js");
36
40
  const judge_js_1 = require("./judge.js");
37
41
  const eval_js_1 = require("./eval.js");
38
42
  const harness_assert_js_1 = require("./harness-assert.js");
@@ -397,6 +401,69 @@ function formatSelectionReport(r) {
397
401
  }
398
402
  return lines.join("\n");
399
403
  }
404
+ /**
405
+ * Measure a plugin's skill-SELECTION collision matrix — "when I ask for skill i's
406
+ * job, does ONLY skill i fire?" The first-class, assertable form of the cross-skill
407
+ * collision measurement (pair with {@link assertNoCollision}). The matrix diagonal
408
+ * is recall; off-diagonal mass is collision — skill j hijacking skill i's prompt,
409
+ * the failure that breaks a multi-skill plugin and that per-skill trigger-rate
410
+ * (each skill in ISOLATION) structurally can't see.
411
+ *
412
+ * ZERO-SETUP: with no `prompts`, they're derived from each model-invocable skill's
413
+ * description. Claude Code only (Codex has no skill-selection event to read); needs
414
+ * the `claude` CLI + model auth, else `available: false`. Thin promotion of
415
+ * {@link measurePluginSelection}. See research/plugin-selection-collision.md.
416
+ */
417
+ async function measureSelectionMatrix(dir, opts = {}) {
418
+ return measurePluginSelection(dir, resolveSelectionPrompts(dir, opts), opts);
419
+ }
420
+ /**
421
+ * Injectable core of {@link measureSelectionMatrix} (for tests): auto-derive the
422
+ * prompts (unless supplied) and drive the matrix via a fake/real probe.
423
+ */
424
+ async function measureSelectionMatrixWith(dir, probe, opts = {}) {
425
+ return measurePluginSelectionWith(dir, resolveSelectionPrompts(dir, opts), probe, opts);
426
+ }
427
+ /** The prompts for a selection run: explicit if given, else auto-derived from the
428
+ * model-invocable skills' descriptions (the zero-setup path). */
429
+ function resolveSelectionPrompts(dir, opts) {
430
+ return (opts.prompts ??
431
+ (0, audit_prompts_js_1.autoTriggerPrompts)((0, scan_js_1.scanPlugin)(dir)
432
+ .skills.filter((s) => !s.userInvoked && s.hasDescription)
433
+ .map((s) => ({ name: s.name, description: s.description ?? "" }))));
434
+ }
435
+ /**
436
+ * Assert a plugin's skills don't hijack each other — the gate over a
437
+ * {@link SelectionReport} from {@link measureSelectionMatrix}. `maxOffDiagonal`
438
+ * caps EACH skill's collision rate (fraction of its own prompts on which a SIBLING
439
+ * fired); `maxPluginCollision` caps the plugin-wide rate. With neither set it
440
+ * demands ZERO collision. THROWS (never a silent green) when nothing was measured
441
+ * — an unavailable harness or a zero-run report is a gap, not a pass.
442
+ */
443
+ function assertNoCollision(report, opts = {}) {
444
+ if (!report.available)
445
+ throw new Error(`selection matrix unavailable — ${report.note ?? "n/a"}`);
446
+ if (report.n === 0)
447
+ throw new Error(`selection matrix measured nothing — ${report.note ?? "no runs"}`);
448
+ // Enforce the per-skill ceiling when asked, OR by default (name = NoCollision);
449
+ // if only the plugin-wide cap is given, don't also silently demand zero per-skill.
450
+ const maxOff = opts.maxOffDiagonal ??
451
+ (opts.maxPluginCollision === undefined ? 0 : undefined);
452
+ if (maxOff !== undefined) {
453
+ const worst = report.perSkill
454
+ .filter((s) => s.n > 0)
455
+ .reduce((w, s) => (w && w.collisionRate >= s.collisionRate ? w : s), undefined);
456
+ if (worst && worst.collisionRate > maxOff) {
457
+ const top = worst.collidesWith[0];
458
+ const tail = top ? ` (top collider: ${top.skill} ${pct(top.rate)})` : "";
459
+ throw new Error(`expected each skill's collision rate ≤ ${String(maxOff)}, but ${worst.skill} = ${worst.collisionRate.toFixed(2)}${tail}`);
460
+ }
461
+ }
462
+ if (opts.maxPluginCollision !== undefined &&
463
+ report.collisionRate > opts.maxPluginCollision) {
464
+ throw new Error(`expected plugin collision rate ≤ ${String(opts.maxPluginCollision)}, got ${report.collisionRate.toFixed(2)}`);
465
+ }
466
+ }
400
467
  // ─── Enforcement-gate detection (for the adversarial-gate eval) ───────────────
401
468
  //
402
469
  // A skill whose description states a HARD CONSTRAINT ("always write tests first",
package/dist/scan.d.ts CHANGED
@@ -17,6 +17,7 @@ import { type ToolIssue } from "./core/tool-contract.js";
17
17
  import { type HookEventIssue } from "./core/hook-events.js";
18
18
  import { type McpIssue } from "./core/mcp-config.js";
19
19
  import { type DescriptionOverlap } from "./core/description-overlap.js";
20
+ import { type DescriptionBudgetIssue } from "./core/skill-description-budget.js";
20
21
  import { type McpToolIssue } from "./core/mcp-tool.js";
21
22
  import { type McpContractToolError } from "./core/mcp.js";
22
23
  import { type McpHookIssue } from "./core/mcp-hook.js";
@@ -28,8 +29,6 @@ import { type DelegationTrifectaFinding } from "./core/delegation-trifecta.js";
28
29
  import { type HookBlockFinding } from "./core/hook-block-ineffective.js";
29
30
  import { type HookMatcherFinding } from "./core/hook-matcher.js";
30
31
  import { type PurityLevel, type EffectSurface } from "./core/effects.js";
31
- /** A named writing system. The label `unexpectedScript` reports + the config's expectation parse into this. */
32
- export type Script = "Latin" | "Cyrillic" | "Han" | "Japanese" | "Korean" | "Arabic" | "Hebrew" | "Greek" | "Devanagari" | "Thai";
33
32
  export interface ScanSkill {
34
33
  readonly name: string;
35
34
  readonly path: string;
@@ -41,15 +40,6 @@ export interface ScanSkill {
41
40
  */
42
41
  readonly description?: string;
43
42
  readonly userInvoked: boolean;
44
- /**
45
- * The description's dominant script when it DIFFERS from the expected one
46
- * (default `"Latin"`), else null. The model's skill-selection context is
47
- * English-centric, so a description in another script carries a cross-language
48
- * trigger risk — it may under-fire on English prompts. A RISK flag, not a
49
- * defect (a language-matched audience is fine); measure the real gap with the
50
- * `audit` trigger tier / `measureTriggerRate`.
51
- */
52
- readonly descriptionScript: Script | null;
53
43
  /**
54
44
  * SKILL.md body references to a bundled file (`scripts/`/`references/`/`assets/`
55
45
  * or a relative markdown link with an extension) that don't resolve on disk
@@ -221,6 +211,8 @@ export interface ScanReport {
221
211
  readonly mcpHookIssues: readonly McpHookIssue[];
222
212
  /** Pairs of model-invocable skills whose descriptions are near-identical (precision collision). */
223
213
  readonly descriptionOverlaps: readonly DescriptionOverlap[];
214
+ /** Model-invocable skills whose description is so long the trigger signal is buried. */
215
+ readonly descriptionBudgetIssues: readonly DescriptionBudgetIssue[];
224
216
  /**
225
217
  * Lethal-trifecta findings across subagents + model-invocable skills — a unit
226
218
  * holding all three legs (read-private + ingest-untrusted + exfiltrate). Each
@@ -299,18 +291,6 @@ export interface SurfaceClassifier {
299
291
  readonly isAgent: (f: string) => boolean;
300
292
  readonly isCommand: (f: string) => boolean;
301
293
  }
302
- /**
303
- * The description's dominant alphabetic script when it DIFFERS from `expected`
304
- * (default `"Latin"`) — the cross-language trigger-risk signal. The model's
305
- * skill-selection context is English-centric, so a description written mostly in
306
- * another script may under-fire on English prompts. `expected` is a configurable
307
- * default, not a value judgement: a Russian-targeted pack sets it to `"Cyrillic"`
308
- * so its Cyrillic descriptions pass and an English one is flagged instead.
309
- * Returns null when the dominant script IS the expected one (or there's no
310
- * alphabetic content). Shared by `scan` and the future lint rule (one detector,
311
- * no drift). The ≥20% guard avoids a near-empty string tripping on one letter.
312
- */
313
- export declare function unexpectedScript(text: string, expected?: Script): Script | null;
314
294
  /**
315
295
  * A compiled `vigiles/hook` artifact runs through the `hook-runtime run-program`
316
296
  * runtime entrypoint; any other hook command is hand-written (a shell script or
package/dist/scan.js CHANGED
@@ -13,7 +13,6 @@
13
13
  * stack on top later; this core stays pure so it runs anywhere in CI for free.
14
14
  */
15
15
  Object.defineProperty(exports, "__esModule", { value: true });
16
- exports.unexpectedScript = unexpectedScript;
17
16
  exports.isManagedHookCommand = isManagedHookCommand;
18
17
  exports.preferCompiledHooksMessage = preferCompiledHooksMessage;
19
18
  exports.scanPlugin = scanPlugin;
@@ -35,6 +34,7 @@ const hook_normalize_js_1 = require("./core/hook-normalize.js");
35
34
  const linters_js_1 = require("./core/linters.js");
36
35
  const frontmatter_read_js_1 = require("./core/frontmatter-read.js");
37
36
  const description_overlap_js_1 = require("./core/description-overlap.js");
37
+ const skill_description_budget_js_1 = require("./core/skill-description-budget.js");
38
38
  const mcp_tool_js_1 = require("./core/mcp-tool.js");
39
39
  const mcp_js_1 = require("./core/mcp.js");
40
40
  const mcp_hook_js_1 = require("./core/mcp-hook.js");
@@ -111,58 +111,6 @@ function skillName(path) {
111
111
  .split("/")
112
112
  .pop() ?? path);
113
113
  }
114
- // [Unicode \p{Script=…} property value (Node native, no dependency), our Script
115
- // label]. Japanese kana fold to "Japanese". Latin is the DEFAULT expectation (the
116
- // selector is English-centric), but it's just a default — a language-matched pack
117
- // can declare a different expectation, and then the OTHER script is the mismatch.
118
- const SCRIPTS = [
119
- ["Latin", "Latin"],
120
- ["Cyrillic", "Cyrillic"],
121
- ["Han", "Han"],
122
- ["Hiragana", "Japanese"],
123
- ["Katakana", "Japanese"],
124
- ["Hangul", "Korean"],
125
- ["Arabic", "Arabic"],
126
- ["Hebrew", "Hebrew"],
127
- ["Greek", "Greek"],
128
- ["Devanagari", "Devanagari"],
129
- ["Thai", "Thai"],
130
- ];
131
- /** Letter counts per named script label (Japanese kana folded together). */
132
- function scriptCounts(text) {
133
- const counts = new Map();
134
- for (const [script, label] of SCRIPTS) {
135
- const n = (text.match(new RegExp(`\\p{Script=${script}}`, "gu")) ?? [])
136
- .length;
137
- if (n > 0)
138
- counts.set(label, (counts.get(label) ?? 0) + n);
139
- }
140
- return counts;
141
- }
142
- /**
143
- * The description's dominant alphabetic script when it DIFFERS from `expected`
144
- * (default `"Latin"`) — the cross-language trigger-risk signal. The model's
145
- * skill-selection context is English-centric, so a description written mostly in
146
- * another script may under-fire on English prompts. `expected` is a configurable
147
- * default, not a value judgement: a Russian-targeted pack sets it to `"Cyrillic"`
148
- * so its Cyrillic descriptions pass and an English one is flagged instead.
149
- * Returns null when the dominant script IS the expected one (or there's no
150
- * alphabetic content). Shared by `scan` and the future lint rule (one detector,
151
- * no drift). The ≥20% guard avoids a near-empty string tripping on one letter.
152
- */
153
- function unexpectedScript(text, expected = "Latin") {
154
- const counts = scriptCounts(text);
155
- let total = 0;
156
- let dominant = null;
157
- for (const [label, count] of counts) {
158
- total += count;
159
- if (!dominant || count > dominant.count)
160
- dominant = { label, count };
161
- }
162
- if (!dominant || dominant.label === expected)
163
- return null;
164
- return dominant.count / total >= 0.2 ? dominant.label : null;
165
- }
166
114
  /**
167
115
  * The first prose paragraph of a SKILL.md body (after the frontmatter and any
168
116
  * leading `#` headings) — Claude Code's FALLBACK skill description when the
@@ -236,7 +184,6 @@ function scanSkills(files, cls, ctx) {
236
184
  hasDescription: Boolean(effectiveDesc && effectiveDesc.length >= 20),
237
185
  description: effectiveDesc?.trim(),
238
186
  userInvoked,
239
- descriptionScript: effectiveDesc ? unexpectedScript(effectiveDesc) : null,
240
187
  resourceIssues,
241
188
  trifecta,
242
189
  // A SKILL.md opening with `name:`/`description:` but no `---` fence loads
@@ -254,7 +201,7 @@ function scanSkills(files, cls, ctx) {
254
201
  * logic as `scanSkills` (frontmatter `description` ← first body paragraph), then
255
202
  * the NCD precision-proxy. See description-overlap.ts.
256
203
  */
257
- function descriptionOverlapsFor(files, cls) {
204
+ function modelInvocableSkillSurfaces(files, cls) {
258
205
  const surfaces = [];
259
206
  for (const [path, md] of Object.entries(files)) {
260
207
  if (!cls.isSkill(path))
@@ -267,7 +214,18 @@ function descriptionOverlapsFor(files, cls) {
267
214
  continue;
268
215
  surfaces.push({ name: fm.name ?? skillName(path), description });
269
216
  }
270
- return (0, description_overlap_js_1.findDescriptionOverlaps)(surfaces);
217
+ return surfaces;
218
+ }
219
+ function descriptionOverlapsFor(files, cls) {
220
+ return (0, description_overlap_js_1.findDescriptionOverlaps)(modelInvocableSkillSurfaces(files, cls));
221
+ }
222
+ /**
223
+ * Model-invocable skills whose description is so long the trigger signal is
224
+ * buried (heuristic proxy; degrades recall + precision). Same surfaces as the
225
+ * overlap check. See skill-description-budget.ts.
226
+ */
227
+ function descriptionBudgetFor(files, cls) {
228
+ return (0, skill_description_budget_js_1.findDescriptionBudgetIssues)(modelInvocableSkillSurfaces(files, cls));
271
229
  }
272
230
  function scanAgents(files, dialect, declaredServers, cls) {
273
231
  const out = [];
@@ -797,6 +755,7 @@ function scanPlugin(dir, layout, dialect = dialect_js_1.claudeCodeDialect) {
797
755
  mcpIssues: (0, mcp_config_js_1.verifyMcpServers)(mcpServers),
798
756
  mcpHookIssues: (0, mcp_hook_js_1.verifyMcpHookTargets)(loaded.settings.hooks, declaredServers, dialect),
799
757
  descriptionOverlaps: descriptionOverlapsFor(loaded.files, cls),
758
+ descriptionBudgetIssues: descriptionBudgetFor(loaded.files, cls),
800
759
  trifectaFindings,
801
760
  skillResourceIssues: skillResourceFindings,
802
761
  skillFenceIssues: skillFenceFindings,
@@ -923,7 +882,7 @@ function section(title, lines, count = lines.length) {
923
882
  return [];
924
883
  return [`${title} (${String(count)}):`, ...lines, ""];
925
884
  }
926
- /** One skill's report line: ✓/⚠ + name + notes (no-trigger, user-invoked, language risk). */
885
+ /** One skill's report line: ✓/⚠ + name + notes (no-trigger, user-invoked). */
927
886
  function skillLine(s) {
928
887
  if (!s.hasDescription) {
929
888
  return ` ⚠ ${s.name} (no usable description — no frontmatter description and no body text — can't trigger)`;
@@ -931,11 +890,7 @@ function skillLine(s) {
931
890
  const notes = [];
932
891
  if (s.userInvoked)
933
892
  notes.push("user-invoked");
934
- if (s.descriptionScript) {
935
- notes.push(`description in ${s.descriptionScript} — cross-language trigger risk`);
936
- }
937
- const mark = s.descriptionScript ? "⚠" : "✓";
938
- return ` ${mark} ${s.name}${notes.length ? ` (${notes.join("; ")})` : ""}`;
893
+ return ` ✓ ${s.name}${notes.length ? ` (${notes.join("; ")})` : ""}`;
939
894
  }
940
895
  /** One agent's report block: ✗ (broken contract) / ⚠ (inherits all) / ✓ + issues + purity. */
941
896
  function agentLines(a) {
@@ -1002,6 +957,7 @@ function formatScanReport(r) {
1002
957
  out.push(...section("MCP config", r.mcpIssues.map((i) => ` ✗ ${i.message}`)));
1003
958
  out.push(...section("MCP hook targets", r.mcpHookIssues.map((i) => ` ✗ ${i.message}`)));
1004
959
  out.push(...section("Description overlap (precision risk)", r.descriptionOverlaps.map((o) => ` ⚠ ${o.message}`)));
960
+ out.push(...section("Description budget (trigger-signal risk)", r.descriptionBudgetIssues.map((o) => ` ⚠ ${o.message}`)));
1005
961
  out.push(...section("Lethal trifecta (prompt-injection exfil risk)", r.trifectaFindings.map((t) => ` ${t.finding.severity === "hard" ? "✗" : "⚠"} ${t.kind} ${t.name} (${t.path}): ${t.finding.message}`)));
1006
962
  out.push(...section("Skill bundled resources", r.skillResourceIssues.map((s) => ` ✗ ${s.name}: ${s.finding.ref} (line ${String(s.finding.line)}) — bundled resource not found`)));
1007
963
  out.push(...section("Invisible skills (missing frontmatter fence)", r.skillFenceIssues.map((s) => ` ✗ ${s.name} (${s.path}): opens with \`${s.finding.key}:\` but no \`---\` fence — loads as body, never fires`)));
@@ -1028,13 +984,6 @@ function formatScanReport(r) {
1028
984
  if (warnings.length > 0) {
1029
985
  out.push("Warnings:", ...warnings.map((w) => ` - ${w}`), "");
1030
986
  }
1031
- // Cross-language trigger risk is a RISK, not a structural defect (a
1032
- // language-matched audience is fine), so it's reported separately from the
1033
- // verdict — it points at the behavioral column, it doesn't fail the scan.
1034
- const mismatched = r.skills.filter((s) => s.descriptionScript);
1035
- if (mismatched.length > 0) {
1036
- out.push(`⚠ ${String(mismatched.length)} skill(s) have descriptions in an unexpected script (cross-language trigger risk) — measure with \`vigiles measure\``, "");
1037
- }
1038
987
  // Skill-metadata is a RECOMMENDATION, not a structural defect (the skill loads
1039
988
  // via fallbacks) — reported as a soft note, never counted in the verdict.
1040
989
  if (r.skillMetaIssues.length > 0) {
@@ -95,7 +95,7 @@ export declare const WORKFLOW_RULES: readonly ["require-instructions-spec", "unt
95
95
  * keep their own default severities. Named for the group taxonomy
96
96
  * (research/install-enforcement-dx.md).
97
97
  */
98
- export declare const NUDGE_RULES: readonly ["frontmatter-valid", "skill-frontmatter", "prefer-compiled-hooks", "unmarked-refs", "lethal-trifecta", "skill-resource-resolves", "skill-missing-fence", "plugin-dir-layout", "delegation-trifecta", "hook-block-ineffective", "hook-matcher"];
98
+ export declare const NUDGE_RULES: readonly ["skill-description-budget", "frontmatter-valid", "skill-frontmatter", "prefer-compiled-hooks", "unmarked-refs", "lethal-trifecta", "skill-resource-resolves", "skill-missing-fence", "plugin-dir-layout", "delegation-trifecta", "hook-block-ineffective", "hook-matcher"];
99
99
  export declare function mergeProjectConfig(existing: Record<string, unknown>, opts: {
100
100
  harness: string | string[];
101
101
  strict: boolean;
@@ -121,6 +121,7 @@ exports.WORKFLOW_RULES = [
121
121
  * (research/install-enforcement-dx.md).
122
122
  */
123
123
  exports.NUDGE_RULES = [
124
+ "skill-description-budget",
124
125
  "frontmatter-valid",
125
126
  "skill-frontmatter",
126
127
  "prefer-compiled-hooks",
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "vigiles",
3
- "version": "12.0.0",
3
+ "version": "12.2.0",
4
4
  "description": "Lint & test the harness your AI agent runs on — verify the references in your CLAUDE.md / AGENTS.md and test that your hooks and skills actually work.",
5
5
  "keywords": [
6
6
  "claude-code",
@@ -7,7 +7,16 @@ argument-hint: <path to CLAUDE.md, defaults to CLAUDE.md>
7
7
 
8
8
  Start a typed `CLAUDE.md.spec.ts` from an existing hand-written CLAUDE.md (or AGENTS.md). This is the non-destructive adoption path — you keep your existing instruction file as the starting point and get type safety going forward.
9
9
 
10
- > **Faithful by default, and reversible.** Adoption is non-destructive: the goal is a spec that compiles back to the user's existing file as closely as possible — preserve every rule, command, key file, and prose section. Don't upgrade `guidance()` to `enforce()` here; that's a separate, opt-in step (the `strengthen` skill). And it's never a one-way door — `vigiles eject <file>` hands the file back as plain hand-owned markdown anytime. For the lightest touch with no spec at all, inline `<!-- vigiles:enforce ... -->` comments are verified by `vigiles lint` with the same engine.
10
+ ## Adoption rules
11
+
12
+ Adoption is the **safe, faithful on-ramp — never an upgrade in disguise.** These are non-negotiable:
13
+
14
+ - **Faithful.** Preserve every rule, command, key file, and prose section as-is. Invent nothing — the spec must compile back to ~the user's existing file.
15
+ - **Non-destructive.** Never edit the original `CLAUDE.md` / `AGENTS.md`. Only write the new `.spec.ts`. Never auto-`compile` over the file — switching it to spec-managed is a separate, explicit step the user runs with a diff to review.
16
+ - **Don't escalate enforcement.** Keep `guidance()` as `guidance()`. Upgrading to `enforce()` has a cost (config/plugins, possible false positives) and is a separate opt-in step — the `strengthen` skill. Adoption is **not** turning on strict / `workflow` gating.
17
+ - **Reversible.** `vigiles eject <file>` hands the file back as plain hand-owned markdown anytime — it's never a one-way door. Tell the user this.
18
+ - **Ask before writing.** Present the generated spec and a conversion summary first; write only on the user's yes.
19
+ - **A lighter touch exists.** For no spec at all, inline `<!-- vigiles:enforce ... -->` comments are verified by `vigiles lint` with the same engine.
11
20
 
12
21
  ## Instructions
13
22
 
@@ -0,0 +1,56 @@
1
+ ---
2
+ name: debug-my-harness
3
+ description: Diagnose why an agent harness misbehaved by reading the local flight-recorder ledger (.vigiles/runs.jsonl) — which skills fired or got hijacked, which hooks blocked or wrongly allowed, which subagent tool-contract violations happened, and how a skill's trigger rate moved. Use when asked why a skill stopped firing, why a hook didn't block, why the wrong skill ran, or to debug/investigate what the harness actually did. NOT for writing new rules (use strengthen) or editing the spec (use edit-spec).
4
+ ---
5
+
6
+ Diagnose harness misbehavior from the **flight recorder** — the local, append-only ledger
7
+ at `.vigiles/runs.jsonl` that vigiles writes as your harness runs. It records what actually
8
+ happened, so you debug from evidence instead of guessing.
9
+
10
+ ## What's in the ledger
11
+
12
+ One JSON record per line, each with a `kind`:
13
+
14
+ - `hook` — a compiled-hook gate decision: `{event, decision: allow|deny|ask, mode: enforce|observe, rule, cmd, reason}`.
15
+ - `agent` — a subagent tool-contract decision: `{name, tool, allowed, reason}` (a `false` = the agent went outside its lane).
16
+ - `skill` — a skill activation: `{name, fired}`.
17
+ - `eval` — a measured metric: `{name, metric, value}` (e.g. trigger-rate recall/precision).
18
+ - `capability-diff` — a blast-radius change: `{pr, added, removed, widened}`.
19
+
20
+ ## Instructions
21
+
22
+ ### Step 1: Read the ledger
23
+
24
+ Read `.vigiles/runs.jsonl` (JSONL — one record per line; tolerate a torn last line). If it's
25
+ absent or empty, say so — there's nothing recorded yet; suggest running the harness (or
26
+ `vigiles audit`) first. Do NOT fabricate records.
27
+
28
+ ### Step 2: Answer the specific question, evidence-first
29
+
30
+ Match the user's question to the ledger:
31
+
32
+ - **"Why did skill X stop firing / why does the wrong one run?"** — count `skill` fires by
33
+ name over time. If X's fire-rate dropped, look for a sibling that fired on the same kinds
34
+ of prompts (a **selection collision**) and check their descriptions for overlap. Recommend
35
+ differentiating or merging the descriptions.
36
+ - **"Why didn't my hook block that?"** — find `hook` records for the event. A `decision:
37
+ allow` on something that should be denied, or `mode: observe` (shadow, never blocks), or
38
+ the absence of any record, tells you which. Recommend flipping `observe`→`enforce` or
39
+ fixing the gate logic.
40
+ - **"Did a subagent misbehave?"** — list `agent` records with `allowed: false`: the agent
41
+ reached for a tool outside its declared contract. Point at the contract to tighten or widen.
42
+ - **"Is it getting worse?"** — compare `eval` metric values (recall/precision) across runs;
43
+ a downward trend is drift (often after a harness/model upgrade).
44
+
45
+ ### Step 3: Recommend a fix, tied to the evidence
46
+
47
+ Prefer **promoting an ignored-but-decidable rule from prose to a deterministic gate**: a
48
+ repeated `agent` violation or a rule the agent keeps breaking → a compiled hook or a tighter
49
+ tool-contract (the `strengthen` skill can help). A description collision → differentiate the
50
+ skill descriptions. Always cite the specific records you based the diagnosis on.
51
+
52
+ ### Step 4: Offer the next step
53
+
54
+ If the fix is a spec change, hand off to `edit-spec`. If it's promoting guidance to a linter
55
+ rule, hand off to `strengthen`. If a behavioral claim needs measuring (does the skill fire
56
+ now?), hand off to `test-harness` (`measureTriggerRate`).
@@ -136,6 +136,7 @@ If the vigiles plugin is installed (`/plugin marketplace add zernie/vigiles` the
136
136
 
137
137
  ## Important
138
138
 
139
+ - **Do what's asked; don't silently escalate enforcement.** Add the rule the user asked for. If making it an `enforce()` would need a linter-config edit or a plugin install (a cost), or if it could fail a clean CI, **say so and let the user choose** — don't change linter config or flip on strict / `workflow` gating on your own. A pure win (a rule that's already enabled) you can just apply. The `strengthen` skill owns `guidance()` → `enforce()` upgrades.
139
140
  - **Never edit CLAUDE.md or AGENTS.md directly** — they have a vigiles hash comment and are build artifacts
140
141
  - **The spec is TypeScript** — you get type checking, autocomplete, and verified references
141
142
  - **`enforce()` rules are verified** — the compiler checks the rule exists AND is enabled in your linter config
@@ -5,6 +5,10 @@ description: Upgrade a vigiles spec's guidance() rules to enforce() — scan the
5
5
 
6
6
  Scan spec files for `guidance()` rules and suggest `enforce()` replacements backed by real linter rules.
7
7
 
8
+ ## Principle: auto the free wins, nudge for the costs
9
+
10
+ The dividing line is **cost, not strictness**. A `guidance()` → `enforce()` swap where the linter rule **already exists and is enabled** is a pure win — free, reversible, no false-positive risk — so apply it (Tier 1 below). Anything that **costs** something — editing linter config, installing a plugin, or a change that could fail a clean CI — is the user's call: **present it with the tradeoff spelled out and let them choose** (Tiers 2–4). Never silently edit config, install a dependency, or escalate the repo into strict gating. This is the `init` enforcement model (structural = on by default, workflow/strict = opt-in) applied at edit time.
11
+
8
12
  ## Instructions
9
13
 
10
14
  ### Step 0: Choose Mode
@@ -190,6 +190,23 @@ job that asserts the capability is present, run **`vigiles test --no-skip`** so
190
190
  skipped tier fails — a green-with-skips is untested surface. Keep unit +
191
191
  deterministic tests in CI (free); run evals locally or on a schedule with auth.
192
192
 
193
+ ### After a real-model run: TELL THE USER WHAT IT SPENT
194
+
195
+ Whenever you run a real-model eval (`runEval` / `measureArms` / `measureTriggerRate`
196
+ / `measure`), **surface the spend to the user in your reply** — don't let a paid run
197
+ be silent. `runEval` prints a cost block to stderr and every report carries `usage`
198
+ (`report.arms[*].usage`: `totalCostUsd` + token counts). Relay, in plain words:
199
+
200
+ - **tokens spent** and the **API-equivalent `$`** (`total_cost_usd` — what it _would_
201
+ cost at metered API rates);
202
+ - **how it was billed** — "on your Claude subscription (**$0 metered**)" if you're
203
+ logged in, or a **⚠ warning** if `ANTHROPIC_API_KEY` is set (that run was billed
204
+ **per token** — tell them to unset it and `claude login` to run free).
205
+
206
+ We do **not** show "% of your subscription" — Anthropic doesn't expose a plan's
207
+ quota, so any percentage would be invented. Tokens + API-equivalent `$` + the
208
+ billed-to line is the honest, complete picture. Keep the user's cost visible, always.
209
+
193
210
  ## Step 5 — Lock the eval so CI stays honest (you do this automatically)
194
211
 
195
212
  Real-model evals run on the user's subscription — locally, never in CI. So **as