skill-autoroute 0.10.0 → 0.10.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +48 -28
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -217,21 +217,37 @@ codex-cli 0.164.0. Your numbers will differ.
|
|
|
217
217
|
## Hint quality
|
|
218
218
|
|
|
219
219
|
A wrong hint costs tokens and teaches the agent to ignore hints, so the trigger rules aim for precise hints, not many
|
|
220
|
-
hints.
|
|
221
|
-
|
|
222
|
-
a known answer, 5 of which must get no hint.
|
|
220
|
+
hints. This table shows how often a hint was useful. It was measured on 1,616 real prompts from one developer's Claude
|
|
221
|
+
Code history (148 skills); Sonnet graded 90 of them blind.
|
|
223
222
|
|
|
223
|
+
| Trigger rules | Hints that were useful | Hint tokens per prompt |
|
|
224
|
+
| ---------------------------------------------------- | ---------------------- | ---------------------- |
|
|
225
|
+
| 0.2.0 | 36% | 12.9 |
|
|
226
|
+
| Broad single-word rules | 21% | 24.1 |
|
|
227
|
+
| **0.3.0 and later: task word plus object word** | **59%** | **10.8** |
|
|
224
228
|
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
| **0.3.0 and later: task word plus object word (sonnet)** | **59.1%** | **59.1%** | **26 / 18** | **10.8** | **12/13** |
|
|
229
|
+
Broader rules doubled the hints, but most of the new ones were wrong. Since 0.10.0 the rule generator also asks for the
|
|
230
|
+
task word and the object word to sit close together. On 1,136 prompts from one real history, useful hints rose from
|
|
231
|
+
46% to about 55%, and about 80% of the earlier useful hints were kept (a different sample, so compare rows within one
|
|
232
|
+
line only).
|
|
230
233
|
|
|
234
|
+
<details>
|
|
235
|
+
<summary>Full numbers for the 2026-10-28 measurement</summary>
|
|
236
|
+
|
|
237
|
+
| Trigger rules | Precision | Relative recall | Useful / wrong hints | Hint tokens per prompt | Fixed check |
|
|
238
|
+
| -------------------------------------------- | --------- | --------------- | -------------------- | ---------------------- | ----------- |
|
|
239
|
+
| 0.2.0 (haiku) | 35.5% | 50.0% | 22 / 40 | 12.9 | 10/13 |
|
|
240
|
+
| Broad single-word rules (haiku) | 21.0% | 59.1% | 26 / 98 | 24.1 | 9/13 |
|
|
241
|
+
| 0.3.0 and later (sonnet) | 59.1% | 59.1% | 26 / 18 | 10.8 | 12/13 |
|
|
242
|
+
|
|
243
|
+
Relative recall is the share of all useful hints that any version found. The fixed check has 13 prompts with a known
|
|
244
|
+
answer, 5 of which must get no hint. Guessing the kind of work from the prompt (question, change, risky) to gate or add
|
|
245
|
+
hints was also tested and made hints worse (F1 63.2 down to 55.7), so it is not used.
|
|
231
246
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
247
|
+
</details>
|
|
248
|
+
|
|
249
|
+
**Speed:** the hook starts in about 50 ms and the matching itself takes under 10 ms on a 227-skill table. A prompt with
|
|
250
|
+
no matching skill adds no tokens.
|
|
235
251
|
|
|
236
252
|
Examples with 0.3.0 and later (0.4.0 added [your own trigger words](#your-own-trigger-words); the rules are the same):
|
|
237
253
|
|
|
@@ -289,7 +305,19 @@ List the acceptable skills in `expect`; an empty list marks a prompt that should
|
|
|
289
305
|
|
|
290
306
|
### Results so far
|
|
291
307
|
|
|
292
|
-
Measured on 2026-11-10
|
|
308
|
+
Measured on 2026-11-10 on one setup (227 skills). Higher is better.
|
|
309
|
+
|
|
310
|
+
| Change | Test | Before | After |
|
|
311
|
+
| --------------------------------------------------- | ------------------------------------ | ------ | ----- |
|
|
312
|
+
| Rules with close-together words (what 0.10.0 does) | Hints that were useful, 1,136 real prompts | 46% | about 55% |
|
|
313
|
+
| Your own trigger words for missed phrases | Right skill listed first, 35 hand-written prompts | 69% | 93% |
|
|
314
|
+
| Score ranking (opt-in, `SKILL_AUTOROUTE_RANK=score`) | Right skill listed first, 35 hand-written prompts | 69% | 69% (no change) |
|
|
315
|
+
|
|
316
|
+
In short: fixing rules and adding your own words helps. Reordering the hints did not help on real skills. It helped
|
|
317
|
+
only on the synthetic corpus, which is why it is off by default.
|
|
318
|
+
|
|
319
|
+
<details>
|
|
320
|
+
<summary>All measurements</summary>
|
|
293
321
|
|
|
294
322
|
| What changed | Cases | P@1 before | P@1 after | recall@3 before | recall@3 after | No-match false positives |
|
|
295
323
|
|---|---|---|---|---|---|---|
|
|
@@ -298,28 +326,20 @@ Measured on 2026-11-10 with the eval above. Same matcher, same prompts, only the
|
|
|
298
326
|
| Own words added, real skills (hand-labeled) | 35 | 69.0% | 93.1% | 63.2% | 76.4% | 0% (unchanged) |
|
|
299
327
|
| Own words added, second set written beforehand | 20 | 46.2% | 92.3% | 38.5% | 69.2% | 0% (unchanged) |
|
|
300
328
|
|
|
301
|
-
On 1,136 real prompts from one developer's history (321 hinted skill-and-prompt pairs, graded blind by Sonnet), hint
|
|
302
|
-
precision was 46.4% with the original generated rules. Rewriting nine skills' rules so the task word and object word sit
|
|
303
|
-
close together (what the generator now asks for) gave about 55%, and kept about 80% of the useful hints. Own words and
|
|
304
|
-
score ranking changed almost nothing on that history. This is a different method from the table above, so compare the
|
|
305
|
-
rows only with each other.
|
|
306
|
-
|
|
307
329
|
Score ranking costs about 0.3 ms per prompt (6.7 ms to 7.0 ms on a 227-skill table). Own words cost nothing extra.
|
|
330
|
+
On 1,136 real prompts (321 hinted skill-and-prompt pairs, graded blind by Sonnet), hint precision was 46.4% with the
|
|
331
|
+
original generated rules and about 55% with nine skills' rules rewritten, keeping about 80% of the useful hints. Own
|
|
332
|
+
words and score ranking changed almost nothing on that history.
|
|
308
333
|
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
- Ordering did not help on real skills. The gain came from prompts that matched no skill at all. Adding specific
|
|
312
|
-
phrases (English and Indonesian) as own words fixed most of them.
|
|
313
|
-
- The real-skill rows come from one person's setup (227 skills) and labels. Those prompt sets are not published.
|
|
314
|
-
The second set was written before the own words, but the phrases were chosen knowing it, so part of that gain is
|
|
315
|
-
fitting. Expect a smaller gain on your own skills, and measure it.
|
|
334
|
+
</details>
|
|
316
335
|
|
|
317
|
-
What
|
|
336
|
+
What these numbers do not show:
|
|
318
337
|
|
|
338
|
+
- They come from one person's setup and labels, and the prompt sets are not published. The second set was written
|
|
339
|
+
before the own words, but the phrases were chosen knowing it, so part of that gain is fitting. Expect a smaller gain
|
|
340
|
+
on your own skills, and measure it.
|
|
319
341
|
- The bundled corpus is synthetic. It shows direction, not how your skills will behave. Write 20 to 30 prompts of your
|
|
320
342
|
own, with your own skill names, before you trust a change.
|
|
321
|
-
- Score ranking (`SKILL_AUTOROUTE_RANK=score`) raised P@1 on the synthetic corpus and made no difference on a
|
|
322
|
-
hand-labeled set of real skills. The bigger gain there came from adding own words for prompts that matched nothing.
|
|
323
343
|
- Labels are judgment calls. With about 35 prompts one case is worth about 3 points, so small differences are noise.
|
|
324
344
|
|
|
325
345
|
---
|