skill-autoroute 0.9.0 → 0.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +91 -11
- package/package.json +1 -1
- package/src/flow.js +23 -3
- package/src/triggers.js +4 -4
package/README.md
CHANGED
|
@@ -217,21 +217,37 @@ codex-cli 0.164.0. Your numbers will differ.
|
|
|
217
217
|
## Hint quality
|
|
218
218
|
|
|
219
219
|
A wrong hint costs tokens and teaches the agent to ignore hints, so the trigger rules aim for precise hints, not many
|
|
220
|
-
hints.
|
|
221
|
-
|
|
222
|
-
a known answer, 5 of which must get no hint.
|
|
220
|
+
hints. This table shows how often a hint was useful. It was measured on 1,616 real prompts from one developer's Claude
|
|
221
|
+
Code history (148 skills); Sonnet graded 90 of them blind.
|
|
223
222
|
|
|
223
|
+
| Trigger rules | Hints that were useful | Hint tokens per prompt |
|
|
224
|
+
| ---------------------------------------------------- | ---------------------- | ---------------------- |
|
|
225
|
+
| 0.2.0 | 36% | 12.9 |
|
|
226
|
+
| Broad single-word rules | 21% | 24.1 |
|
|
227
|
+
| **0.3.0 and later: task word plus object word** | **59%** | **10.8** |
|
|
224
228
|
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
| **0.3.0 and later: task word plus object word (sonnet)** | **59.1%** | **59.1%** | **26 / 18** | **10.8** | **12/13** |
|
|
229
|
+
Broader rules doubled the hints, but most of the new ones were wrong. Since 0.10.0 the rule generator also asks for the
|
|
230
|
+
task word and the object word to sit close together. On 1,136 prompts from one real history, useful hints rose from
|
|
231
|
+
46% to about 55%, and about 80% of the earlier useful hints were kept (a different sample, so compare rows within one
|
|
232
|
+
line only).
|
|
230
233
|
|
|
234
|
+
<details>
|
|
235
|
+
<summary>Full numbers for the 2026-10-28 measurement</summary>
|
|
236
|
+
|
|
237
|
+
| Trigger rules | Precision | Relative recall | Useful / wrong hints | Hint tokens per prompt | Fixed check |
|
|
238
|
+
| -------------------------------------------- | --------- | --------------- | -------------------- | ---------------------- | ----------- |
|
|
239
|
+
| 0.2.0 (haiku) | 35.5% | 50.0% | 22 / 40 | 12.9 | 10/13 |
|
|
240
|
+
| Broad single-word rules (haiku) | 21.0% | 59.1% | 26 / 98 | 24.1 | 9/13 |
|
|
241
|
+
| 0.3.0 and later (sonnet) | 59.1% | 59.1% | 26 / 18 | 10.8 | 12/13 |
|
|
231
242
|
|
|
232
|
-
Relative recall is the share of all useful hints that any version found.
|
|
233
|
-
|
|
234
|
-
tested and made hints worse (F1 63.2 down to 55.7), so it is not used.
|
|
243
|
+
Relative recall is the share of all useful hints that any version found. The fixed check has 13 prompts with a known
|
|
244
|
+
answer, 5 of which must get no hint. Guessing the kind of work from the prompt (question, change, risky) to gate or add
|
|
245
|
+
hints was also tested and made hints worse (F1 63.2 down to 55.7), so it is not used.
|
|
246
|
+
|
|
247
|
+
</details>
|
|
248
|
+
|
|
249
|
+
**Speed:** the hook starts in about 50 ms and the matching itself takes under 10 ms on a 227-skill table. A prompt with
|
|
250
|
+
no matching skill adds no tokens.
|
|
235
251
|
|
|
236
252
|
Examples with 0.3.0 and later (0.4.0 added [your own trigger words](#your-own-trigger-words); the rules are the same):
|
|
237
253
|
|
|
@@ -260,7 +276,71 @@ npx skill-autoroute words no-ai-slop remove polish
|
|
|
260
276
|
- Each word or phrase matches as a whole word, case-insensitive. It is plain text, not a regex.
|
|
261
277
|
- A match on your own words puts that skill first among the hints.
|
|
262
278
|
- `regenerate` and `install` keep your words. Up to 20 per skill, 60 characters each.
|
|
279
|
+
- You can also edit a skill's rules in `routes.json` by hand. `regenerate` and `install` keep rules you edited (they no longer match the hash stored with the generated rules) and tell you which skills they kept.
|
|
263
280
|
- A word that also fires on ordinary replies (for example `fix` in "please fix this") is allowed, with a warning.
|
|
281
|
+
- Prefer specific phrases ("tests first", "execute the plan") over single common words. In one real setup, a prompt that
|
|
282
|
+
matched no skill (many non-English prompts, and plain phrasings like "write tests first") got a hint once the
|
|
283
|
+
phrase was added as an own word.
|
|
284
|
+
|
|
285
|
+
---
|
|
286
|
+
|
|
287
|
+
## Measure accuracy
|
|
288
|
+
|
|
289
|
+
The eval needs a clone of this repo (the `test/` folder is not in the npm package). It runs the hook's matcher on
|
|
290
|
+
labeled prompts. It makes no model call and no network call.
|
|
291
|
+
|
|
292
|
+
```bash
|
|
293
|
+
npm run eval # synthetic corpus and fixture table
|
|
294
|
+
npm run eval -- --table ~/.config/skill-autoroute/routes.json --corpus my-cases.json # your own table and cases
|
|
295
|
+
npm run eval -- --json out.json # also write the numbers to a file
|
|
296
|
+
```
|
|
297
|
+
|
|
298
|
+
A corpus is a JSON array of `{ "id", "kind": "prompt", "tool": "claude", "text", "expect": ["skill-name", ...] }`.
|
|
299
|
+
List the acceptable skills in `expect`; an empty list marks a prompt that should get no hint. It reports:
|
|
300
|
+
|
|
301
|
+
- **P@1**: how often the first hint is an expected skill.
|
|
302
|
+
- **P@3**: the share of the (up to 3) hints that are expected.
|
|
303
|
+
- **recall@3**: the share of expected skills that appear in the hints.
|
|
304
|
+
- **no-match false positives**: the share of empty-`expect` prompts that still got a hint.
|
|
305
|
+
|
|
306
|
+
### Results so far
|
|
307
|
+
|
|
308
|
+
Measured on 2026-11-10 on one setup (227 skills). Higher is better.
|
|
309
|
+
|
|
310
|
+
| Change | Test | Before | After |
|
|
311
|
+
| --------------------------------------------------- | ------------------------------------ | ------ | ----- |
|
|
312
|
+
| Rules with close-together words (what 0.10.0 does) | Hints that were useful, 1,136 real prompts | 46% | about 55% |
|
|
313
|
+
| Your own trigger words for missed phrases | Right skill listed first, 35 hand-written prompts | 69% | 93% |
|
|
314
|
+
| Score ranking (opt-in, `SKILL_AUTOROUTE_RANK=score`) | Right skill listed first, 35 hand-written prompts | 69% | 69% (no change) |
|
|
315
|
+
|
|
316
|
+
In short: fixing rules and adding your own words helps. Reordering the hints did not help on real skills. It helped
|
|
317
|
+
only on the synthetic corpus, which is why it is off by default.
|
|
318
|
+
|
|
319
|
+
<details>
|
|
320
|
+
<summary>All measurements</summary>
|
|
321
|
+
|
|
322
|
+
| What changed | Cases | P@1 before | P@1 after | recall@3 before | recall@3 after | No-match false positives |
|
|
323
|
+
|---|---|---|---|---|---|---|
|
|
324
|
+
| Score ranking, synthetic corpus | 56 | 68.9% | 88.9% | 97.0% | 97.8% | 36.4% (unchanged) |
|
|
325
|
+
| Score ranking, real skills (hand-labeled) | 35 | 69.0% | 69.0% | 63.2% | 62.4% | 0% (unchanged) |
|
|
326
|
+
| Own words added, real skills (hand-labeled) | 35 | 69.0% | 93.1% | 63.2% | 76.4% | 0% (unchanged) |
|
|
327
|
+
| Own words added, second set written beforehand | 20 | 46.2% | 92.3% | 38.5% | 69.2% | 0% (unchanged) |
|
|
328
|
+
|
|
329
|
+
Score ranking costs about 0.3 ms per prompt (6.7 ms to 7.0 ms on a 227-skill table). Own words cost nothing extra.
|
|
330
|
+
On 1,136 real prompts (321 hinted skill-and-prompt pairs, graded blind by Sonnet), hint precision was 46.4% with the
|
|
331
|
+
original generated rules and about 55% with nine skills' rules rewritten, keeping about 80% of the useful hints. Own
|
|
332
|
+
words and score ranking changed almost nothing on that history.
|
|
333
|
+
|
|
334
|
+
</details>
|
|
335
|
+
|
|
336
|
+
What these numbers do not show:
|
|
337
|
+
|
|
338
|
+
- They come from one person's setup and labels, and the prompt sets are not published. The second set was written
|
|
339
|
+
before the own words, but the phrases were chosen knowing it, so part of that gain is fitting. Expect a smaller gain
|
|
340
|
+
on your own skills, and measure it.
|
|
341
|
+
- The bundled corpus is synthetic. It shows direction, not how your skills will behave. Write 20 to 30 prompts of your
|
|
342
|
+
own, with your own skill names, before you trust a change.
|
|
343
|
+
- Labels are judgment calls. With about 35 prompts one case is worth about 3 points, so small differences are noise.
|
|
264
344
|
|
|
265
345
|
---
|
|
266
346
|
|
package/package.json
CHANGED
package/src/flow.js
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
// Command logic. bin/cli.js only parses argv and supplies `ui`; tests pass a non-interactive ui.
|
|
2
2
|
import { statSync, readFileSync, rmSync, existsSync } from 'node:fs';
|
|
3
3
|
import { spawnSync } from 'node:child_process';
|
|
4
|
+
import { createHash } from 'node:crypto';
|
|
4
5
|
import { relative, sep } from 'node:path';
|
|
5
6
|
import * as claude from './tools/claude.js';
|
|
6
7
|
import * as codex from './tools/codex.js';
|
|
@@ -196,10 +197,13 @@ const entry = (r, gen, scope) => {
|
|
|
196
197
|
scan: { verdict: r.scan.verdict, partial: r.scan.partial, findings: tableFindings(scope, r.skill, r.scan.findings) }, duplicateOf: r.dup?.of ?? null,
|
|
197
198
|
};
|
|
198
199
|
};
|
|
200
|
+
// Every entry stores a hash of the rules as generated. A rule list that no longer matches its hash was edited by the user.
|
|
201
|
+
export const rulesHash = (patterns) => createHash('sha256').update(JSON.stringify(patterns ?? [])).digest('hex').slice(0, 16);
|
|
202
|
+
const edited = (w) => Boolean(w?.rulesHash) && rulesHash(w.patterns) !== w.rulesHash;
|
|
199
203
|
const tableOf = (ctx, skills, roots, log) => ({
|
|
200
204
|
version: 1, generatedAt: new Date().toISOString(),
|
|
201
205
|
generator: { ai: ctx.ai ? ctx.aiName : 'none', model: ctx.ai ? pickModel(ctx.aiName, 'triggers', ctx.overrides).model ?? null : null, overheadPerCall: ctx.overhead, cacheReadPerCall: ctx.cacheOverhead, rules: RULES_VERSION },
|
|
202
|
-
roots, ...(log ? { log: true } : {}), skills,
|
|
206
|
+
roots, ...(log ? { log: true } : {}), skills: skills.map((s) => (s.rulesHash ? s : { ...s, rulesHash: rulesHash(s.patterns) })),
|
|
203
207
|
});
|
|
204
208
|
|
|
205
209
|
// Every tool config must parse before anything is written (throws naming the bad file).
|
|
@@ -295,6 +299,7 @@ export async function install(o, ui) {
|
|
|
295
299
|
const gen = await generate(rows.filter((r) => r.enabled).map((r) => r.skill), ctx, ui);
|
|
296
300
|
const table = tableOf(ctx, rows.map((r) => entry(r, gen, scope)), roots.map(keyOf(scope)), o.log);
|
|
297
301
|
keepWords(table, readTable(routesPath(scope)));
|
|
302
|
+
keepEdited(table, readTable(routesPath(scope)), ui);
|
|
298
303
|
while (!ui.yes) {
|
|
299
304
|
const p = await ui.text('Preview: type a prompt to see its hints (empty to finish)');
|
|
300
305
|
if (!p) break;
|
|
@@ -339,18 +344,20 @@ export async function regenerate(o, ui) {
|
|
|
339
344
|
if (off) ui.log(`${n} turned off: ${show(label(r.scan))}; run \`npx skill-autoroute set ${n} on\` to override`);
|
|
340
345
|
return { ...r, enabled: Boolean(w.enabled) && !off, dup: w.duplicateOf ? { of: w.duplicateOf } : undefined };
|
|
341
346
|
});
|
|
342
|
-
const
|
|
347
|
+
const keep = (s) => !edited(was(s)); // rules the user edited are not rewritten
|
|
348
|
+
const gen = await generate([...rows.filter((r) => r.enabled && keep(r.skill)).map((r) => r.skill), ...bare.filter(keep)], ctx, ui);
|
|
343
349
|
const byPath = new Map(rows.map((r) => [r.skill.skillPath, entry(r, gen, scope)]));
|
|
344
350
|
for (const s of bare) {
|
|
345
351
|
const g = gen.bySkill.get(s.name);
|
|
346
352
|
// a failed AI call must not swap working older rules for keyword rules
|
|
347
353
|
if (was(s).patterns?.length && (!g || gen.fellBack.includes(s.name))) continue;
|
|
348
|
-
byPath.set(s.skillPath, { ...was(s), priority: g?.priority ?? 50, patterns: g?.rules ?? [] });
|
|
354
|
+
byPath.set(s.skillPath, { ...was(s), rulesHash: undefined, priority: g?.priority ?? 50, patterns: g?.rules ?? [] });
|
|
349
355
|
}
|
|
350
356
|
// Scope comes from this discovery, not the old entry: enabling a plugin in another project changes no skill file.
|
|
351
357
|
const scoped = ({ projects, ...e }, s) => (s.projects ? { ...e, projects: s.projects } : e);
|
|
352
358
|
const table = tableOf(ctx, skills.map((s) => scoped(byPath.get(s.skillPath) ?? was(s), s)), roots.map(key), old.log);
|
|
353
359
|
keepWords(table, old);
|
|
360
|
+
keepEdited(table, old, ui);
|
|
354
361
|
// unchanged rules still come from the old generator; after a full rebuild the new model wrote them all
|
|
355
362
|
if (!o.ai && old.generator) table.generator = { ...old.generator, rules: RULES_VERSION, ...(outdated && ctx.ai ? { model: table.generator.model } : {}) };
|
|
356
363
|
printRows(rows, ui);
|
|
@@ -366,6 +373,19 @@ function keepWords(table, old) {
|
|
|
366
373
|
for (const s of table.skills) if (words.has(s.path)) s.words = words.get(s.path);
|
|
367
374
|
}
|
|
368
375
|
|
|
376
|
+
// Rules the user edited in routes.json (they no longer match the stored hash) survive a rebuild, with their priority.
|
|
377
|
+
export function keepEdited(table, old, ui) {
|
|
378
|
+
const was = new Map((old?.skills ?? []).map((s) => [s.path, s]));
|
|
379
|
+
const names = [];
|
|
380
|
+
for (const s of table.skills) {
|
|
381
|
+
const w = was.get(s.path);
|
|
382
|
+
if (!edited(w)) continue;
|
|
383
|
+
Object.assign(s, { patterns: w.patterns, priority: w.priority, rulesHash: w.rulesHash });
|
|
384
|
+
names.push(show(s.name));
|
|
385
|
+
}
|
|
386
|
+
if (names.length) ui?.log(`kept your edited rules for ${names.length} skill${names.length === 1 ? '' : 's'}: ${names.slice(0, 5).join(', ')}${names.length > 5 ? ', ...' : ''}`);
|
|
387
|
+
}
|
|
388
|
+
|
|
369
389
|
const MAX_OWN_WORDS = 20;
|
|
370
390
|
const MAX_OWN_WORD = 60;
|
|
371
391
|
export function setWords(o, ui, name, action, ...words) {
|
package/src/triggers.js
CHANGED
|
@@ -25,7 +25,7 @@ const isGeneric = (rule) => genericMatch(rule) !== undefined;
|
|
|
25
25
|
const CONTROL = /[\u0000-\u001f]/;
|
|
26
26
|
const MAX_RULES = 6;
|
|
27
27
|
// Bump when TRIGGER_PROMPT or rule limits change: regenerate then rebuilds every skill's rules.
|
|
28
|
-
export const RULES_VERSION =
|
|
28
|
+
export const RULES_VERSION = 5;
|
|
29
29
|
const showCtl = (p) => p.replace(/[\u0000-\u001f]/g, (c) => `\\u${c.charCodeAt(0).toString(16).padStart(4, '0')}`);
|
|
30
30
|
|
|
31
31
|
const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
|
@@ -77,11 +77,11 @@ For each skill return: name (copied exactly), priority (integer 1 to 100, lower
|
|
|
77
77
|
Lessons from rules that went wrong:
|
|
78
78
|
- Every pattern is a JSON string, so write a regex backslash doubled: "\\\\b(goroutine|channels)\\\\b" in JSON means the regex \\b(goroutine|channels)\\b. A single backslash before b is a backspace character and is rejected. Use word boundaries, not bare words.
|
|
79
79
|
- Avoid plain-English collisions. "go ahead" is not the Go language, "authority" is not auth, "react" the verb is not React, "rust" is not rust on a pipe. Prefer terms only this domain uses (goroutine, oauth, useEffect, borrow checker).
|
|
80
|
-
- Never use .* or
|
|
80
|
+
- Never use .* or .+. To tie a task word to an object word, write ONE pattern with a short bounded gap, task word first: "\\\\b(add|create|drop)\\\\b.{0,20}\\\\b(columns?|tables?)\\\\b" (gap of 10 to 30 characters). Two separate patterns in one rule match words anywhere in the prompt, so they fire when the two words only happen to appear in the same long message. Use that form only when order and distance do not matter.
|
|
81
81
|
- No nested quantifiers like (a+)+. Keep each pattern under 200 characters. JavaScript syntax, no lookbehind.
|
|
82
82
|
- Do not write a pattern that matches almost every prompt. A skill with no clear trigger gets a narrow rule or priority 90 to 100.
|
|
83
|
-
- Rules that only match the skill's own name or jargon never fire: users describe the task, not the skill. Give each skill at least one rule built from the everyday words a user types for that task, with common synonyms and word forms in one group. A planning skill:
|
|
84
|
-
- Most prompts need no skill, and a wrong hint wastes the assistant's context. Fire only when the prompt asks for this skill's task. Build rules as a task word plus an object word
|
|
83
|
+
- Rules that only match the skill's own name or jargon never fire: users describe the task, not the skill. Give each skill at least one rule built from the everyday words a user types for that task, with common synonyms and word forms in one group. A planning skill: "\\\\b(write|draft|make)\\\\b.{0,15}\\\\b(plan|roadmap|spec)s?\\\\b". An API skill: "\\\\b(endpoint|api|route|webhook)s?\\\\b". A brainstorming skill: "\\\\b(brainstorm\\\\w*|ideas?|ideate)\\\\b". Pair a common word with a task word in one rule when the word alone is too broad.
|
|
84
|
+
- Most prompts need no skill, and a wrong hint wastes the assistant's context. Fire only when the prompt asks for this skill's task. Build rules as a task word plus an object word within a short gap, such as "\\\\b(write|add|create)\\\\b.{0,15}\\\\bmigrations?\\\". Use a single pattern only for terms that name the task on their own (oauth, goroutine, database migration), never for one everyday word.
|
|
85
85
|
- Words people use all the time when chatting with a coding assistant never fire a rule alone: agent, assistant, chat, spec, plan, fix, build, workflow, skill, verify, check, test, run, doc, file, prompt, issue, change.
|
|
86
86
|
- Object word lists must name the things people point at, not only the skill's own terms: files and text (README, docs, text, copy, changelog), data (table, column, field, index), UI (page, landing page, layout, screen, component). Verb lists must include plain verbs (make, add, change, update, write, improve) and outcome words (responsive, mobile, faster, cleaner). For example "polish the README text", "add a column to the users table" and "make this landing page responsive" should each fire the skill for that task.
|
|
87
87
|
- At most 3 skills are shown per prompt, lowest priority number first. Give skills for one stack, file type or tool 10 to 30, and broad process skills (planning, debugging, review, testing) 40 to 70, so the specific skill wins a tie.
|