skill-autoroute 0.9.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +60 -0
- package/package.json +1 -1
- package/src/flow.js +23 -3
- package/src/triggers.js +4 -4
package/README.md
CHANGED
|
@@ -260,7 +260,67 @@ npx skill-autoroute words no-ai-slop remove polish
|
|
|
260
260
|
- Each word or phrase matches as a whole word, case-insensitive. It is plain text, not a regex.
|
|
261
261
|
- A match on your own words puts that skill first among the hints.
|
|
262
262
|
- `regenerate` and `install` keep your words. Up to 20 per skill, 60 characters each.
|
|
263
|
+
- You can also edit a skill's rules in `routes.json` by hand. `regenerate` and `install` keep rules you edited (they no longer match the hash stored with the generated rules) and tell you which skills they kept.
|
|
263
264
|
- A word that also fires on ordinary replies (for example `fix` in "please fix this") is allowed, with a warning.
|
|
265
|
+
- Prefer specific phrases ("tests first", "execute the plan") over single common words. In one real setup, a prompt that
|
|
266
|
+
matched no skill (many non-English prompts, and plain phrasings like "write tests first") got a hint once the
|
|
267
|
+
phrase was added as an own word.
|
|
268
|
+
|
|
269
|
+
---
|
|
270
|
+
|
|
271
|
+
## Measure accuracy
|
|
272
|
+
|
|
273
|
+
The eval needs a clone of this repo (the `test/` folder is not in the npm package). It runs the hook's matcher on
|
|
274
|
+
labeled prompts. It makes no model call and no network call.
|
|
275
|
+
|
|
276
|
+
```bash
|
|
277
|
+
npm run eval # synthetic corpus and fixture table
|
|
278
|
+
npm run eval -- --table ~/.config/skill-autoroute/routes.json --corpus my-cases.json # your own table and cases
|
|
279
|
+
npm run eval -- --json out.json # also write the numbers to a file
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
A corpus is a JSON array of `{ "id", "kind": "prompt", "tool": "claude", "text", "expect": ["skill-name", ...] }`.
|
|
283
|
+
List the acceptable skills in `expect`; an empty list marks a prompt that should get no hint. It reports:
|
|
284
|
+
|
|
285
|
+
- **P@1**: how often the first hint is an expected skill.
|
|
286
|
+
- **P@3**: the share of the (up to 3) hints that are expected.
|
|
287
|
+
- **recall@3**: the share of expected skills that appear in the hints.
|
|
288
|
+
- **no-match false positives**: the share of empty-`expect` prompts that still got a hint.
|
|
289
|
+
|
|
290
|
+
### Results so far
|
|
291
|
+
|
|
292
|
+
Measured on 2026-11-10 with the eval above. Same matcher, same prompts, only the named change differs.
|
|
293
|
+
|
|
294
|
+
| What changed | Cases | P@1 before | P@1 after | recall@3 before | recall@3 after | No-match false positives |
|
|
295
|
+
|---|---|---|---|---|---|---|
|
|
296
|
+
| Score ranking, synthetic corpus | 56 | 68.9% | 88.9% | 97.0% | 97.8% | 36.4% (unchanged) |
|
|
297
|
+
| Score ranking, real skills (hand-labeled) | 35 | 69.0% | 69.0% | 63.2% | 62.4% | 0% (unchanged) |
|
|
298
|
+
| Own words added, real skills (hand-labeled) | 35 | 69.0% | 93.1% | 63.2% | 76.4% | 0% (unchanged) |
|
|
299
|
+
| Own words added, second set written beforehand | 20 | 46.2% | 92.3% | 38.5% | 69.2% | 0% (unchanged) |
|
|
300
|
+
|
|
301
|
+
On 1,136 real prompts from one developer's history (321 hinted skill-and-prompt pairs, graded blind by Sonnet), hint
|
|
302
|
+
precision was 46.4% with the original generated rules. Rewriting nine skills' rules so the task word and object word sit
|
|
303
|
+
close together (what the generator now asks for) gave about 55%, and kept about 80% of the useful hints. Own words and
|
|
304
|
+
score ranking changed almost nothing on that history. This is a different method from the table above, so compare the
|
|
305
|
+
rows only with each other.
|
|
306
|
+
|
|
307
|
+
Score ranking costs about 0.3 ms per prompt (6.7 ms to 7.0 ms on a 227-skill table). Own words cost nothing extra.
|
|
308
|
+
|
|
309
|
+
Read it this way:
|
|
310
|
+
|
|
311
|
+
- Ordering did not help on real skills. The gain came from prompts that matched no skill at all. Adding specific
|
|
312
|
+
phrases (English and Indonesian) as own words fixed most of them.
|
|
313
|
+
- The real-skill rows come from one person's setup (227 skills) and labels. Those prompt sets are not published.
|
|
314
|
+
The second set was written before the own words, but the phrases were chosen knowing it, so part of that gain is
|
|
315
|
+
fitting. Expect a smaller gain on your own skills, and measure it.
|
|
316
|
+
|
|
317
|
+
What the numbers mean and do not mean:
|
|
318
|
+
|
|
319
|
+
- The bundled corpus is synthetic. It shows direction, not how your skills will behave. Write 20 to 30 prompts of your
|
|
320
|
+
own, with your own skill names, before you trust a change.
|
|
321
|
+
- Score ranking (`SKILL_AUTOROUTE_RANK=score`) raised P@1 on the synthetic corpus and made no difference on a
|
|
322
|
+
hand-labeled set of real skills. The bigger gain there came from adding own words for prompts that matched nothing.
|
|
323
|
+
- Labels are judgment calls. With about 35 prompts one case is worth about 3 points, so small differences are noise.
|
|
264
324
|
|
|
265
325
|
---
|
|
266
326
|
|
package/package.json
CHANGED
package/src/flow.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
// Command logic. bin/cli.js only parses argv and supplies `ui`; tests pass a non-interactive ui.
|
|
2
2
|
import { statSync, readFileSync, rmSync, existsSync } from 'node:fs';
|
|
3
3
|
import { spawnSync } from 'node:child_process';
|
|
4
|
+
import { createHash } from 'node:crypto';
|
|
4
5
|
import { relative, sep } from 'node:path';
|
|
5
6
|
import * as claude from './tools/claude.js';
|
|
6
7
|
import * as codex from './tools/codex.js';
|
|
@@ -196,10 +197,13 @@ const entry = (r, gen, scope) => {
|
|
|
196
197
|
scan: { verdict: r.scan.verdict, partial: r.scan.partial, findings: tableFindings(scope, r.skill, r.scan.findings) }, duplicateOf: r.dup?.of ?? null,
|
|
197
198
|
};
|
|
198
199
|
};
|
|
200
|
+
// Every entry stores a hash of the rules as generated. A rule list that no longer matches its hash was edited by the user.
|
|
201
|
+
export const rulesHash = (patterns) => createHash('sha256').update(JSON.stringify(patterns ?? [])).digest('hex').slice(0, 16);
|
|
202
|
+
const edited = (w) => Boolean(w?.rulesHash) && rulesHash(w.patterns) !== w.rulesHash;
|
|
199
203
|
const tableOf = (ctx, skills, roots, log) => ({
|
|
200
204
|
version: 1, generatedAt: new Date().toISOString(),
|
|
201
205
|
generator: { ai: ctx.ai ? ctx.aiName : 'none', model: ctx.ai ? pickModel(ctx.aiName, 'triggers', ctx.overrides).model ?? null : null, overheadPerCall: ctx.overhead, cacheReadPerCall: ctx.cacheOverhead, rules: RULES_VERSION },
|
|
202
|
-
roots, ...(log ? { log: true } : {}), skills,
|
|
206
|
+
roots, ...(log ? { log: true } : {}), skills: skills.map((s) => (s.rulesHash ? s : { ...s, rulesHash: rulesHash(s.patterns) })),
|
|
203
207
|
});
|
|
204
208
|
|
|
205
209
|
// Every tool config must parse before anything is written (throws naming the bad file).
|
|
@@ -295,6 +299,7 @@ export async function install(o, ui) {
|
|
|
295
299
|
const gen = await generate(rows.filter((r) => r.enabled).map((r) => r.skill), ctx, ui);
|
|
296
300
|
const table = tableOf(ctx, rows.map((r) => entry(r, gen, scope)), roots.map(keyOf(scope)), o.log);
|
|
297
301
|
keepWords(table, readTable(routesPath(scope)));
|
|
302
|
+
keepEdited(table, readTable(routesPath(scope)), ui);
|
|
298
303
|
while (!ui.yes) {
|
|
299
304
|
const p = await ui.text('Preview: type a prompt to see its hints (empty to finish)');
|
|
300
305
|
if (!p) break;
|
|
@@ -339,18 +344,20 @@ export async function regenerate(o, ui) {
|
|
|
339
344
|
if (off) ui.log(`${n} turned off: ${show(label(r.scan))}; run \`npx skill-autoroute set ${n} on\` to override`);
|
|
340
345
|
return { ...r, enabled: Boolean(w.enabled) && !off, dup: w.duplicateOf ? { of: w.duplicateOf } : undefined };
|
|
341
346
|
});
|
|
342
|
-
const
|
|
347
|
+
const keep = (s) => !edited(was(s)); // rules the user edited are not rewritten
|
|
348
|
+
const gen = await generate([...rows.filter((r) => r.enabled && keep(r.skill)).map((r) => r.skill), ...bare.filter(keep)], ctx, ui);
|
|
343
349
|
const byPath = new Map(rows.map((r) => [r.skill.skillPath, entry(r, gen, scope)]));
|
|
344
350
|
for (const s of bare) {
|
|
345
351
|
const g = gen.bySkill.get(s.name);
|
|
346
352
|
// a failed AI call must not swap working older rules for keyword rules
|
|
347
353
|
if (was(s).patterns?.length && (!g || gen.fellBack.includes(s.name))) continue;
|
|
348
|
-
byPath.set(s.skillPath, { ...was(s), priority: g?.priority ?? 50, patterns: g?.rules ?? [] });
|
|
354
|
+
byPath.set(s.skillPath, { ...was(s), rulesHash: undefined, priority: g?.priority ?? 50, patterns: g?.rules ?? [] });
|
|
349
355
|
}
|
|
350
356
|
// Scope comes from this discovery, not the old entry: enabling a plugin in another project changes no skill file.
|
|
351
357
|
const scoped = ({ projects, ...e }, s) => (s.projects ? { ...e, projects: s.projects } : e);
|
|
352
358
|
const table = tableOf(ctx, skills.map((s) => scoped(byPath.get(s.skillPath) ?? was(s), s)), roots.map(key), old.log);
|
|
353
359
|
keepWords(table, old);
|
|
360
|
+
keepEdited(table, old, ui);
|
|
354
361
|
// unchanged rules still come from the old generator; after a full rebuild the new model wrote them all
|
|
355
362
|
if (!o.ai && old.generator) table.generator = { ...old.generator, rules: RULES_VERSION, ...(outdated && ctx.ai ? { model: table.generator.model } : {}) };
|
|
356
363
|
printRows(rows, ui);
|
|
@@ -366,6 +373,19 @@ function keepWords(table, old) {
|
|
|
366
373
|
for (const s of table.skills) if (words.has(s.path)) s.words = words.get(s.path);
|
|
367
374
|
}
|
|
368
375
|
|
|
376
|
+
// Rules the user edited in routes.json (they no longer match the stored hash) survive a rebuild, with their priority.
|
|
377
|
+
export function keepEdited(table, old, ui) {
|
|
378
|
+
const was = new Map((old?.skills ?? []).map((s) => [s.path, s]));
|
|
379
|
+
const names = [];
|
|
380
|
+
for (const s of table.skills) {
|
|
381
|
+
const w = was.get(s.path);
|
|
382
|
+
if (!edited(w)) continue;
|
|
383
|
+
Object.assign(s, { patterns: w.patterns, priority: w.priority, rulesHash: w.rulesHash });
|
|
384
|
+
names.push(show(s.name));
|
|
385
|
+
}
|
|
386
|
+
if (names.length) ui?.log(`kept your edited rules for ${names.length} skill${names.length === 1 ? '' : 's'}: ${names.slice(0, 5).join(', ')}${names.length > 5 ? ', ...' : ''}`);
|
|
387
|
+
}
|
|
388
|
+
|
|
369
389
|
const MAX_OWN_WORDS = 20;
|
|
370
390
|
const MAX_OWN_WORD = 60;
|
|
371
391
|
export function setWords(o, ui, name, action, ...words) {
|
package/src/triggers.js
CHANGED
|
@@ -25,7 +25,7 @@ const isGeneric = (rule) => genericMatch(rule) !== undefined;
|
|
|
25
25
|
const CONTROL = /[\u0000-\u001f]/;
|
|
26
26
|
const MAX_RULES = 6;
|
|
27
27
|
// Bump when TRIGGER_PROMPT or rule limits change: regenerate then rebuilds every skill's rules.
|
|
28
|
-
export const RULES_VERSION =
|
|
28
|
+
export const RULES_VERSION = 5;
|
|
29
29
|
const showCtl = (p) => p.replace(/[\u0000-\u001f]/g, (c) => `\\u${c.charCodeAt(0).toString(16).padStart(4, '0')}`);
|
|
30
30
|
|
|
31
31
|
const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
|
@@ -77,11 +77,11 @@ For each skill return: name (copied exactly), priority (integer 1 to 100, lower
|
|
|
77
77
|
Lessons from rules that went wrong:
|
|
78
78
|
- Every pattern is a JSON string, so write a regex backslash doubled: "\\\\b(goroutine|channels)\\\\b" in JSON means the regex \\b(goroutine|channels)\\b. A single backslash before b is a backspace character and is rejected. Use word boundaries, not bare words.
|
|
79
79
|
- Avoid plain-English collisions. "go ahead" is not the Go language, "authority" is not auth, "react" the verb is not React, "rust" is not rust on a pipe. Prefer terms only this domain uses (goroutine, oauth, useEffect, borrow checker).
|
|
80
|
-
- Never use .* or
|
|
80
|
+
- Never use .* or .+. To tie a task word to an object word, write ONE pattern with a short bounded gap, task word first: "\\\\b(add|create|drop)\\\\b.{0,20}\\\\b(columns?|tables?)\\\\b" (gap of 10 to 30 characters). Two separate patterns in one rule match words anywhere in the prompt, so they fire when the two words only happen to appear in the same long message. Use that form only when order and distance do not matter.
|
|
81
81
|
- No nested quantifiers like (a+)+. Keep each pattern under 200 characters. JavaScript syntax, no lookbehind.
|
|
82
82
|
- Do not write a pattern that matches almost every prompt. A skill with no clear trigger gets a narrow rule or priority 90 to 100.
|
|
83
|
-
- Rules that only match the skill's own name or jargon never fire: users describe the task, not the skill. Give each skill at least one rule built from the everyday words a user types for that task, with common synonyms and word forms in one group. A planning skill:
|
|
84
|
-
- Most prompts need no skill, and a wrong hint wastes the assistant's context. Fire only when the prompt asks for this skill's task. Build rules as a task word plus an object word
|
|
83
|
+
- Rules that only match the skill's own name or jargon never fire: users describe the task, not the skill. Give each skill at least one rule built from the everyday words a user types for that task, with common synonyms and word forms in one group. A planning skill: "\\\\b(write|draft|make)\\\\b.{0,15}\\\\b(plan|roadmap|spec)s?\\\\b". An API skill: "\\\\b(endpoint|api|route|webhook)s?\\\\b". A brainstorming skill: "\\\\b(brainstorm\\\\w*|ideas?|ideate)\\\\b". Pair a common word with a task word in one rule when the word alone is too broad.
|
|
84
|
+
- Most prompts need no skill, and a wrong hint wastes the assistant's context. Fire only when the prompt asks for this skill's task. Build rules as a task word plus an object word within a short gap, such as "\\\\b(write|add|create)\\\\b.{0,15}\\\\bmigrations?\\\". Use a single pattern only for terms that name the task on their own (oauth, goroutine, database migration), never for one everyday word.
|
|
85
85
|
- Words people use all the time when chatting with a coding assistant never fire a rule alone: agent, assistant, chat, spec, plan, fix, build, workflow, skill, verify, check, test, run, doc, file, prompt, issue, change.
|
|
86
86
|
- Object word lists must name the things people point at, not only the skill's own terms: files and text (README, docs, text, copy, changelog), data (table, column, field, index), UI (page, landing page, layout, screen, component). Verb lists must include plain verbs (make, add, change, update, write, improve) and outcome words (responsive, mobile, faster, cleaner). For example "polish the README text", "add a column to the users table" and "make this landing page responsive" should each fire the skill for that task.
|
|
87
87
|
- At most 3 skills are shown per prompt, lowest priority number first. Give skills for one stack, file type or tool 10 to 30, and broad process skills (planning, debugging, review, testing) 40 to 70, so the specific skill wins a tie.
|