skill-autoroute 0.9.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -260,7 +260,67 @@ npx skill-autoroute words no-ai-slop remove polish
260
260
  - Each word or phrase matches as a whole word, case-insensitive. It is plain text, not a regex.
261
261
  - A match on your own words puts that skill first among the hints.
262
262
  - `regenerate` and `install` keep your words. Up to 20 per skill, 60 characters each.
263
+ - You can also edit a skill's rules in `routes.json` by hand. `regenerate` and `install` keep rules you edited (they no longer match the hash stored with the generated rules) and tell you which skills they kept.
263
264
  - A word that also fires on ordinary replies (for example `fix` in "please fix this") is allowed, with a warning.
265
+ - Prefer specific phrases ("tests first", "execute the plan") over single common words. In one real setup, a prompt that
266
+ matched no skill (many non-English prompts, and plain phrasings like "write tests first") got a hint once the
267
+ phrase was added as an own word.
268
+
269
+ ---
270
+
271
+ ## Measure accuracy
272
+
273
+ The eval needs a clone of this repo (the `test/` folder is not in the npm package). It runs the hook's matcher on
274
+ labeled prompts. It makes no model call and no network call.
275
+
276
+ ```bash
277
+ npm run eval # synthetic corpus and fixture table
278
+ npm run eval -- --table ~/.config/skill-autoroute/routes.json --corpus my-cases.json # your own table and cases
279
+ npm run eval -- --json out.json # also write the numbers to a file
280
+ ```
281
+
282
+ A corpus is a JSON array of `{ "id", "kind": "prompt", "tool": "claude", "text", "expect": ["skill-name", ...] }`.
283
+ List the acceptable skills in `expect`; an empty list marks a prompt that should get no hint. It reports:
284
+
285
+ - **P@1**: how often the first hint is an expected skill.
286
+ - **P@3**: the share of the (up to 3) hints that are expected.
287
+ - **recall@3**: the share of expected skills that appear in the hints.
288
+ - **no-match false positives**: the share of empty-`expect` prompts that still got a hint.
289
+
290
+ ### Results so far
291
+
292
+ Measured on 2026-11-10 with the eval above. Same matcher, same prompts, only the named change differs.
293
+
294
+ | What changed | Cases | P@1 before | P@1 after | recall@3 before | recall@3 after | No-match false positives |
295
+ |---|---|---|---|---|---|---|
296
+ | Score ranking, synthetic corpus | 56 | 68.9% | 88.9% | 97.0% | 97.8% | 36.4% (unchanged) |
297
+ | Score ranking, real skills (hand-labeled) | 35 | 69.0% | 69.0% | 63.2% | 62.4% | 0% (unchanged) |
298
+ | Own words added, real skills (hand-labeled) | 35 | 69.0% | 93.1% | 63.2% | 76.4% | 0% (unchanged) |
299
+ | Own words added, second set written beforehand | 20 | 46.2% | 92.3% | 38.5% | 69.2% | 0% (unchanged) |
300
+
301
+ On 1,136 real prompts from one developer's history (321 hinted skill-and-prompt pairs, graded blind by Sonnet), hint
302
+ precision was 46.4% with the original generated rules. Rewriting nine skills' rules so the task word and object word sit
303
+ close together (what the generator now asks for) gave about 55%, and kept about 80% of the useful hints. Own words and
304
+ score ranking changed almost nothing on that history. This is a different method from the table above, so compare the
305
+ rows only with each other.
306
+
307
+ Score ranking costs about 0.3 ms per prompt (6.7 ms to 7.0 ms on a 227-skill table). Own words cost nothing extra.
308
+
309
+ Read it this way:
310
+
311
+ - Ordering did not help on real skills. The gain came from prompts that matched no skill at all. Adding specific
312
+ phrases (English and Indonesian) as own words fixed most of them.
313
+ - The real-skill rows come from one person's setup (227 skills) and labels. Those prompt sets are not published.
314
+ The second set was written before the own words, but the phrases were chosen knowing it, so part of that gain is
315
+ fitting. Expect a smaller gain on your own skills, and measure it.
316
+
317
+ What the numbers mean and do not mean:
318
+
319
+ - The bundled corpus is synthetic. It shows direction, not how your skills will behave. Write 20 to 30 prompts of your
320
+ own, with your own skill names, before you trust a change.
321
+ - Score ranking (`SKILL_AUTOROUTE_RANK=score`) raised P@1 on the synthetic corpus and made no difference on a
322
+ hand-labeled set of real skills. The bigger gain there came from adding own words for prompts that matched nothing.
323
+ - Labels are judgment calls. With about 35 prompts one case is worth about 3 points, so small differences are noise.
264
324
 
265
325
  ---
266
326
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "skill-autoroute",
3
- "version": "0.9.0",
3
+ "version": "0.10.0",
4
4
  "description": "Per-prompt skill router hook for Claude Code and Codex",
5
5
  "keywords": [
6
6
  "claude-code",
package/src/flow.js CHANGED
@@ -1,6 +1,7 @@
1
1
  // Command logic. bin/cli.js only parses argv and supplies `ui`; tests pass a non-interactive ui.
2
2
  import { statSync, readFileSync, rmSync, existsSync } from 'node:fs';
3
3
  import { spawnSync } from 'node:child_process';
4
+ import { createHash } from 'node:crypto';
4
5
  import { relative, sep } from 'node:path';
5
6
  import * as claude from './tools/claude.js';
6
7
  import * as codex from './tools/codex.js';
@@ -196,10 +197,13 @@ const entry = (r, gen, scope) => {
196
197
  scan: { verdict: r.scan.verdict, partial: r.scan.partial, findings: tableFindings(scope, r.skill, r.scan.findings) }, duplicateOf: r.dup?.of ?? null,
197
198
  };
198
199
  };
200
+ // Every entry stores a hash of the rules as generated. A rule list that no longer matches its hash was edited by the user.
201
+ export const rulesHash = (patterns) => createHash('sha256').update(JSON.stringify(patterns ?? [])).digest('hex').slice(0, 16);
202
+ const edited = (w) => Boolean(w?.rulesHash) && rulesHash(w.patterns) !== w.rulesHash;
199
203
  const tableOf = (ctx, skills, roots, log) => ({
200
204
  version: 1, generatedAt: new Date().toISOString(),
201
205
  generator: { ai: ctx.ai ? ctx.aiName : 'none', model: ctx.ai ? pickModel(ctx.aiName, 'triggers', ctx.overrides).model ?? null : null, overheadPerCall: ctx.overhead, cacheReadPerCall: ctx.cacheOverhead, rules: RULES_VERSION },
202
- roots, ...(log ? { log: true } : {}), skills,
206
+ roots, ...(log ? { log: true } : {}), skills: skills.map((s) => (s.rulesHash ? s : { ...s, rulesHash: rulesHash(s.patterns) })),
203
207
  });
204
208
 
205
209
  // Every tool config must parse before anything is written (throws naming the bad file).
@@ -295,6 +299,7 @@ export async function install(o, ui) {
295
299
  const gen = await generate(rows.filter((r) => r.enabled).map((r) => r.skill), ctx, ui);
296
300
  const table = tableOf(ctx, rows.map((r) => entry(r, gen, scope)), roots.map(keyOf(scope)), o.log);
297
301
  keepWords(table, readTable(routesPath(scope)));
302
+ keepEdited(table, readTable(routesPath(scope)), ui);
298
303
  while (!ui.yes) {
299
304
  const p = await ui.text('Preview: type a prompt to see its hints (empty to finish)');
300
305
  if (!p) break;
@@ -339,18 +344,20 @@ export async function regenerate(o, ui) {
339
344
  if (off) ui.log(`${n} turned off: ${show(label(r.scan))}; run \`npx skill-autoroute set ${n} on\` to override`);
340
345
  return { ...r, enabled: Boolean(w.enabled) && !off, dup: w.duplicateOf ? { of: w.duplicateOf } : undefined };
341
346
  });
342
- const gen = await generate([...rows.filter((r) => r.enabled).map((r) => r.skill), ...bare], ctx, ui);
347
+ const keep = (s) => !edited(was(s)); // rules the user edited are not rewritten
348
+ const gen = await generate([...rows.filter((r) => r.enabled && keep(r.skill)).map((r) => r.skill), ...bare.filter(keep)], ctx, ui);
343
349
  const byPath = new Map(rows.map((r) => [r.skill.skillPath, entry(r, gen, scope)]));
344
350
  for (const s of bare) {
345
351
  const g = gen.bySkill.get(s.name);
346
352
  // a failed AI call must not swap working older rules for keyword rules
347
353
  if (was(s).patterns?.length && (!g || gen.fellBack.includes(s.name))) continue;
348
- byPath.set(s.skillPath, { ...was(s), priority: g?.priority ?? 50, patterns: g?.rules ?? [] });
354
+ byPath.set(s.skillPath, { ...was(s), rulesHash: undefined, priority: g?.priority ?? 50, patterns: g?.rules ?? [] });
349
355
  }
350
356
  // Scope comes from this discovery, not the old entry: enabling a plugin in another project changes no skill file.
351
357
  const scoped = ({ projects, ...e }, s) => (s.projects ? { ...e, projects: s.projects } : e);
352
358
  const table = tableOf(ctx, skills.map((s) => scoped(byPath.get(s.skillPath) ?? was(s), s)), roots.map(key), old.log);
353
359
  keepWords(table, old);
360
+ keepEdited(table, old, ui);
354
361
  // unchanged rules still come from the old generator; after a full rebuild the new model wrote them all
355
362
  if (!o.ai && old.generator) table.generator = { ...old.generator, rules: RULES_VERSION, ...(outdated && ctx.ai ? { model: table.generator.model } : {}) };
356
363
  printRows(rows, ui);
@@ -366,6 +373,19 @@ function keepWords(table, old) {
366
373
  for (const s of table.skills) if (words.has(s.path)) s.words = words.get(s.path);
367
374
  }
368
375
 
376
+ // Rules the user edited in routes.json (they no longer match the stored hash) survive a rebuild, with their priority.
377
+ export function keepEdited(table, old, ui) {
378
+ const was = new Map((old?.skills ?? []).map((s) => [s.path, s]));
379
+ const names = [];
380
+ for (const s of table.skills) {
381
+ const w = was.get(s.path);
382
+ if (!edited(w)) continue;
383
+ Object.assign(s, { patterns: w.patterns, priority: w.priority, rulesHash: w.rulesHash });
384
+ names.push(show(s.name));
385
+ }
386
+ if (names.length) ui?.log(`kept your edited rules for ${names.length} skill${names.length === 1 ? '' : 's'}: ${names.slice(0, 5).join(', ')}${names.length > 5 ? ', ...' : ''}`);
387
+ }
388
+
369
389
  const MAX_OWN_WORDS = 20;
370
390
  const MAX_OWN_WORD = 60;
371
391
  export function setWords(o, ui, name, action, ...words) {
package/src/triggers.js CHANGED
@@ -25,7 +25,7 @@ const isGeneric = (rule) => genericMatch(rule) !== undefined;
25
25
  const CONTROL = /[\u0000-\u001f]/;
26
26
  const MAX_RULES = 6;
27
27
  // Bump when TRIGGER_PROMPT or rule limits change: regenerate then rebuilds every skill's rules.
28
- export const RULES_VERSION = 4;
28
+ export const RULES_VERSION = 5;
29
29
  const showCtl = (p) => p.replace(/[\u0000-\u001f]/g, (c) => `\\u${c.charCodeAt(0).toString(16).padStart(4, '0')}`);
30
30
 
31
31
  const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
@@ -77,11 +77,11 @@ For each skill return: name (copied exactly), priority (integer 1 to 100, lower
77
77
  Lessons from rules that went wrong:
78
78
  - Every pattern is a JSON string, so write a regex backslash doubled: "\\\\b(goroutine|channels)\\\\b" in JSON means the regex \\b(goroutine|channels)\\b. A single backslash before b is a backspace character and is rejected. Use word boundaries, not bare words.
79
79
  - Avoid plain-English collisions. "go ahead" is not the Go language, "authority" is not auth, "react" the verb is not React, "rust" is not rust on a pipe. Prefer terms only this domain uses (goroutine, oauth, useEffect, borrow checker).
80
- - Never use .* or .+ between words. For "A then later B" write two patterns in one rule, such as ["\\\\bA\\\\b", "\\\\bB\\\\b"], so both must match.
80
+ - Never use .* or .+. To tie a task word to an object word, write ONE pattern with a short bounded gap, task word first: "\\\\b(add|create|drop)\\\\b.{0,20}\\\\b(columns?|tables?)\\\\b" (gap of 10 to 30 characters). Two separate patterns in one rule match words anywhere in the prompt, so they fire when the two words only happen to appear in the same long message. Use that form only when order and distance do not matter.
81
81
  - No nested quantifiers like (a+)+. Keep each pattern under 200 characters. JavaScript syntax, no lookbehind.
82
82
  - Do not write a pattern that matches almost every prompt. A skill with no clear trigger gets a narrow rule or priority 90 to 100.
83
- - Rules that only match the skill's own name or jargon never fire: users describe the task, not the skill. Give each skill at least one rule built from the everyday words a user types for that task, with common synonyms and word forms in one group. A planning skill: ["\\\\b(write|draft|make)\\\\b", "\\\\b(plan|roadmap|spec)s?\\\\b"]. An API skill: "\\\\b(endpoint|api|route|webhook)s?\\\\b". A brainstorming skill: "\\\\b(brainstorm\\\\w*|ideas?|ideate)\\\\b". Pair a common word with a task word in one rule when the word alone is too broad.
84
- - Most prompts need no skill, and a wrong hint wastes the assistant's context. Fire only when the prompt asks for this skill's task. Build rules as a task word plus an object word in one rule, such as ["\\\\b(write|add|create)\\\\b", "\\\\bmigrations?\\\\b"]. Use a single pattern only for terms that name the task on their own (oauth, goroutine, database migration), never for one everyday word.
83
+ - Rules that only match the skill's own name or jargon never fire: users describe the task, not the skill. Give each skill at least one rule built from the everyday words a user types for that task, with common synonyms and word forms in one group. A planning skill: "\\\\b(write|draft|make)\\\\b.{0,15}\\\\b(plan|roadmap|spec)s?\\\\b". An API skill: "\\\\b(endpoint|api|route|webhook)s?\\\\b". A brainstorming skill: "\\\\b(brainstorm\\\\w*|ideas?|ideate)\\\\b". Pair a common word with a task word in one rule when the word alone is too broad.
84
+ - Most prompts need no skill, and a wrong hint wastes the assistant's context. Fire only when the prompt asks for this skill's task. Build rules as a task word plus an object word within a short gap, such as "\\\\b(write|add|create)\\\\b.{0,15}\\\\bmigrations?\\\". Use a single pattern only for terms that name the task on their own (oauth, goroutine, database migration), never for one everyday word.
85
85
  - Words people use all the time when chatting with a coding assistant never fire a rule alone: agent, assistant, chat, spec, plan, fix, build, workflow, skill, verify, check, test, run, doc, file, prompt, issue, change.
86
86
  - Object word lists must name the things people point at, not only the skill's own terms: files and text (README, docs, text, copy, changelog), data (table, column, field, index), UI (page, landing page, layout, screen, component). Verb lists must include plain verbs (make, add, change, update, write, improve) and outcome words (responsive, mobile, faster, cleaner). For example "polish the README text", "add a column to the users table" and "make this landing page responsive" should each fire the skill for that task.
87
87
  - At most 3 skills are shown per prompt, lowest priority number first. Give skills for one stack, file type or tool 10 to 30, and broad process skills (planning, debugging, review, testing) 40 to 70, so the specific skill wins a tie.