skill-autoroute 0.10.0 → 0.10.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +48 -28
  2. package/package.json +1 -1
package/README.md CHANGED
@@ -217,21 +217,37 @@ codex-cli 0.164.0. Your numbers will differ.
217
217
  ## Hint quality
218
218
 
219
219
  A wrong hint costs tokens and teaches the agent to ignore hints, so the trigger rules aim for precise hints, not many
220
- hints. Measured on 2026-10-28 on 1,616 real prompts from one developer's Claude Code history, with 148 installed skills.
221
- Sonnet graded 90 sampled prompts blind (was each hinted skill useful for that prompt?). A fixed check has 13 prompts with
222
- a known answer, 5 of which must get no hint.
220
+ hints. This table shows how often a hint was useful. It was measured on 1,616 real prompts from one developer's Claude
221
+ Code history (148 skills); Sonnet graded 90 of them blind.
223
222
 
223
+ | Trigger rules | Hints that were useful | Hint tokens per prompt |
224
+ | ---------------------------------------------------- | ---------------------- | ---------------------- |
225
+ | 0.2.0 | 36% | 12.9 |
226
+ | Broad single-word rules | 21% | 24.1 |
227
+ | **0.3.0 and later: task word plus object word** | **59%** | **10.8** |
224
228
 
225
- | Trigger rules | Precision | Relative recall | Useful / wrong hints | Hint tokens per prompt | Fixed check |
226
- | ------------------------------------------ | --------- | --------------- | -------------------- | ---------------------- | ----------- |
227
- | 0.2.0 (haiku) | 35.5% | 50.0% | 22 / 40 | 12.9 | 10/13 |
228
- | Broad single-word rules (haiku) | 21.0% | 59.1% | 26 / 98 | 24.1 | 9/13 |
229
- | **0.3.0 and later: task word plus object word (sonnet)** | **59.1%** | **59.1%** | **26 / 18** | **10.8** | **12/13** |
229
+ Broader rules doubled the hints, but most of the new ones were wrong. Since 0.10.0 the rule generator also asks for the
230
+ task word and the object word to sit close together. On 1,136 prompts from one real history, useful hints rose from
231
+ 46% to about 55%, and about 80% of the earlier useful hints were kept (a different sample, so compare rows within one
232
+ line only).
230
233
 
234
+ <details>
235
+ <summary>Full numbers for the 2026-10-28 measurement</summary>
236
+
237
+ | Trigger rules | Precision | Relative recall | Useful / wrong hints | Hint tokens per prompt | Fixed check |
238
+ | -------------------------------------------- | --------- | --------------- | -------------------- | ---------------------- | ----------- |
239
+ | 0.2.0 (haiku) | 35.5% | 50.0% | 22 / 40 | 12.9 | 10/13 |
240
+ | Broad single-word rules (haiku) | 21.0% | 59.1% | 26 / 98 | 24.1 | 9/13 |
241
+ | 0.3.0 and later (sonnet) | 59.1% | 59.1% | 26 / 18 | 10.8 | 12/13 |
242
+
243
+ Relative recall is the share of all useful hints that any version found. The fixed check has 13 prompts with a known
244
+ answer, 5 of which must get no hint. Guessing the kind of work from the prompt (question, change, risky) to gate or add
245
+ hints was also tested and made hints worse (F1 63.2 down to 55.7), so it is not used.
231
246
 
232
- Relative recall is the share of all useful hints that any version found. Broader rules doubled the hints but most of the
233
- new ones were wrong. Guessing the kind of work from the prompt (question, change, risky) to gate or add hints was also
234
- tested and made hints worse (F1 63.2 down to 55.7), so it is not used.
247
+ </details>
248
+
249
+ **Speed:** the hook starts in about 50 ms and the matching itself takes under 10 ms on a 227-skill table. A prompt with
250
+ no matching skill adds no tokens.
235
251
 
236
252
  Examples with 0.3.0 and later (0.4.0 added [your own trigger words](#your-own-trigger-words); the rules are the same):
237
253
 
@@ -289,7 +305,19 @@ List the acceptable skills in `expect`; an empty list marks a prompt that should
289
305
 
290
306
  ### Results so far
291
307
 
292
- Measured on 2026-11-10 with the eval above. Same matcher, same prompts, only the named change differs.
308
+ Measured on 2026-11-10 on one setup (227 skills). Higher is better.
309
+
310
+ | Change | Test | Before | After |
311
+ | --------------------------------------------------- | ------------------------------------ | ------ | ----- |
312
+ | Rules with close-together words (what 0.10.0 does) | Hints that were useful, 1,136 real prompts | 46% | about 55% |
313
+ | Your own trigger words for missed phrases | Right skill listed first, 35 hand-written prompts | 69% | 93% |
314
+ | Score ranking (opt-in, `SKILL_AUTOROUTE_RANK=score`) | Right skill listed first, 35 hand-written prompts | 69% | 69% (no change) |
315
+
316
+ In short: fixing rules and adding your own words helps. Reordering the hints did not help on real skills. It helped
317
+ only on the synthetic corpus, which is why it is off by default.
318
+
319
+ <details>
320
+ <summary>All measurements</summary>
293
321
 
294
322
  | What changed | Cases | P@1 before | P@1 after | recall@3 before | recall@3 after | No-match false positives |
295
323
  |---|---|---|---|---|---|---|
@@ -298,28 +326,20 @@ Measured on 2026-11-10 with the eval above. Same matcher, same prompts, only the
298
326
  | Own words added, real skills (hand-labeled) | 35 | 69.0% | 93.1% | 63.2% | 76.4% | 0% (unchanged) |
299
327
  | Own words added, second set written beforehand | 20 | 46.2% | 92.3% | 38.5% | 69.2% | 0% (unchanged) |
300
328
 
301
- On 1,136 real prompts from one developer's history (321 hinted skill-and-prompt pairs, graded blind by Sonnet), hint
302
- precision was 46.4% with the original generated rules. Rewriting nine skills' rules so the task word and object word sit
303
- close together (what the generator now asks for) gave about 55%, and kept about 80% of the useful hints. Own words and
304
- score ranking changed almost nothing on that history. This is a different method from the table above, so compare the
305
- rows only with each other.
306
-
307
329
  Score ranking costs about 0.3 ms per prompt (6.7 ms to 7.0 ms on a 227-skill table). Own words cost nothing extra.
330
+ On 1,136 real prompts (321 hinted skill-and-prompt pairs, graded blind by Sonnet), hint precision was 46.4% with the
331
+ original generated rules and about 55% with nine skills' rules rewritten, keeping about 80% of the useful hints. Own
332
+ words and score ranking changed almost nothing on that history.
308
333
 
309
- Read it this way:
310
-
311
- - Ordering did not help on real skills. The gain came from prompts that matched no skill at all. Adding specific
312
- phrases (English and Indonesian) as own words fixed most of them.
313
- - The real-skill rows come from one person's setup (227 skills) and labels. Those prompt sets are not published.
314
- The second set was written before the own words, but the phrases were chosen knowing it, so part of that gain is
315
- fitting. Expect a smaller gain on your own skills, and measure it.
334
+ </details>
316
335
 
317
- What the numbers mean and do not mean:
336
+ What these numbers do not show:
318
337
 
338
+ - They come from one person's setup and labels, and the prompt sets are not published. The second set was written
339
+ before the own words, but the phrases were chosen knowing it, so part of that gain is fitting. Expect a smaller gain
340
+ on your own skills, and measure it.
319
341
  - The bundled corpus is synthetic. It shows direction, not how your skills will behave. Write 20 to 30 prompts of your
320
342
  own, with your own skill names, before you trust a change.
321
- - Score ranking (`SKILL_AUTOROUTE_RANK=score`) raised P@1 on the synthetic corpus and made no difference on a
322
- hand-labeled set of real skills. The bigger gain there came from adding own words for prompts that matched nothing.
323
343
  - Labels are judgment calls. With about 35 prompts one case is worth about 3 points, so small differences are noise.
324
344
 
325
345
  ---
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "skill-autoroute",
3
- "version": "0.10.0",
3
+ "version": "0.10.1",
4
4
  "description": "Per-prompt skill router hook for Claude Code and Codex",
5
5
  "keywords": [
6
6
  "claude-code",