@ssheleg/make-skill 0.25.2 → 0.26.0

package/CHANGELOG.md

@@ -1,5 +1,82 @@
+## v0.26.0 — where a rule lives decides whether it exists
+
+`authoring.md` treated progressive disclosure as a budget question. It is also a
+**delivery** question, and that half was measured.
+
+**Soft channels failed completely.** A README, an `AGENTS.md` beside the code, comments
+inside a dependency directory, a warning field in an API response — none of them steered
+anything. Two failures inside that deserve their own names, because each looks like it
+should work: agents **rarely open files inside dependency directories**, so a rule written
+there is a rule nobody reads; and agents **parse the data out of an API response while
+ignoring the warning field in the same payload** — the bytes arrived, the guidance did not.
+
+**Hard channels worked**: the skill file itself, an error message, CLI help, an install
+prompt. So a rule that matters belongs in `SKILL.md` or a reference the body links, never
+in a neighbouring document that is merely nearby — **proximity is not loading**. And an
+error message is a steering surface rather than an apology: it arrives exactly when the
+agent is wrong, in a channel it is already reading.
+
+The numbers, all on hard channels: restructuring for progressive disclosure bought about
+**10%** better performance *at lower token cost*; promoting a skill from CLI login converted
+at **30–35%** to an install; moving templates and examples inside the skill rather than into
+the system prompt cut time-to-first-token by **18.1%**.
+
+**And the trigger corpus's negative half now carries the number that defends it.** House
+style already mandates *"~20 queries, half near-miss negatives"* — with no evidence that
+omitting them costs anything, which is exactly the shape of a rule an author trims under
+deadline. Moving to skill-based routing dropped triggering by about **20%** in targeted
+evals before negative examples and edge-case coverage were added back; with them the same
+skill reached **73% → 85%** routing accuracy. A corpus that is all positives measures
+whether the skill fires and never whether it stays quiet — and staying quiet is half of what
+routing is.
+
 # Changelog
 
+## v0.25.3 — the runner we said did not exist, and the date a claim about someone else now carries
+
+- **B-122 closed: `references/authoring.md` told authors for four weeks that
+  there was no built-in eval runner, and prescribed a home-rolled suite on that
+  basis.** `claude plugin eval` had shipped. The section now names it, its case
+  format (`evals/<case>/case.yaml`, or `prompt.md` beside `graders/*.md`), and
+  the fact that `--ablation with-without` performs steps 3 and 5 of the very
+  procedure the paragraph prescribes by hand — plus the detail that it publishes
+  its HTML report to claude.ai unless `--no-publish` is passed, which is the
+  wrong default for an unreleased skill. **The replacement is a measurement, not
+  a correction of one absence into another**: on 2026-08-31, Claude Code 2.1.236,
+  every path (`eval`, `eval init`, `eval init --bare`, `eval <target>`) prints
+  ``plugin eval` is currently in early access`, writes nothing and exits 1, with
+  no settings key, environment variable or feature flag turning it on. That is
+  why `test/evals/` remains what runs here, and the paragraph says which day it
+  should be migrated. Two derived statements went with it: `distribution.md`'s
+  layout comment called the suite *"run by a human"*, and `retrofit.md`'s item 8
+  asked only that evaluations *exist* — the state MSK-01 had just spent a whole
+  run closing.
+- **The class, not the sentence: `THIRD_PARTY_CLAIMS` in `test/validate.py`.**
+  A load-bearing claim about someone else's tool is registered beside the command
+  that re-checks it, and the guard fails unless the claim, that command and an
+  ISO date still sit within eight lines of each other — and fails equally when a
+  registered claim has been deleted from the doctrine but left in the registry,
+  so the registry cannot outlive what it describes. The skill's own rule *"No
+  time-sensitive statements"* sits 100 lines above the sentence that broke it and
+  nothing enforced it; this is the enforcement.
+- **A generic detector was built first, measured, and rejected with the number.**
+  Over the shipped doctrine a pattern for absence-claims flagged **10 lines, of
+  which roughly half are legitimate** — the fallback rows of
+  `host-capabilities.md` ("on any other host there are no subagents") are made of
+  that sentence shape on purpose. A guard with a ~50% false-positive rate is the
+  over-defense that gets switched off, so the explicit registry replaced it; the
+  reasoning is recorded in the code beside the table rather than lost.
+- **The three plants are in CI, and the group count still computes.** Added as
+  cases to the existing *claims, budgets and runnable commands* step rather than
+  as a new step, so `grep -c 'name: Negative self-test'` stays at **9** and
+  `CONTRIBUTING.md`'s *"9 negative self-test groups"* — a claim this validator
+  compares against the workflow — remains true. Each case asserts its own
+  expected message. The first local run of the guard **passed its plants for the
+  wrong reason**: the claim wraps a line break and the check was line-anchored,
+  so all three plants tripped the registry-rot branch instead of their own. The
+  matcher is whitespace-tolerant now, and every plant in the block asserts it
+  changed something before the validator is asked anything.
+
 ## v0.25.2 — the evals have been run, and the discovery check runs somewhere
 
 - **MSK-01 closed: the evaluation suite was executed against two models, and the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/make-skill",
-  "version": "0.25.2",
+  "version": "0.26.0",
   "description": "Create, retrofit, audit, and ship agent skills & Claude Code plugins the proven ssheleg way \u2014 conformance to the Agent Skills open standard AND Anthropic's platform rules (front-matter limits, disclosure budgets, per-surface runtime limits, the Skills API, evals) plus the Claude Code plugin reference (manifest schemas, component layout, claude plugin validate --strict), marketplace repo layout, version sync, validator + CI, multi-channel distribution (plugin, vercel skills CLI, npx, Cursor), npm gotchas, the review checklist for third-party skills, and MCP / A2A rules for protocol-connected skills. This package is the installer CLI.",
   "keywords": [
     "skill",

package/plugins/make-skill/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "make-skill",
   "displayName": "Make Skill",
   "description": "Create, retrofit, audit, and ship agent skills & Claude Code plugins the proven ssheleg way: conformance to the Agent Skills open standard, Anthropic's platform rules (surfaces, Skills API, evals) and the Claude Code plugin reference, marketplace repo layout, version sync, validator + CI, multi-channel distribution (plugin, vercel skills CLI, npx, Cursor), npm gotchas, end-to-end first publish, the review checklist for third-party skills, plus MCP / A2A references for protocol-connected skills.",
-  "version": "0.25.2",
+  "version": "0.26.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/make-skill/skills/make-skill/SKILL.md

@@ -5,7 +5,7 @@ license: MIT
 compatibility: Authoring works on any agent. The bundled scripts/ need python3. Publishing steps need git, gh, node and npm; the plugin gates need the claude CLI. Not usable on the Claude API surface, which has no network and no runtime package install.
 metadata:
   author: ssheleg
-  version: "0.25.2"
+  version: "0.26.0"
   homepage: https://github.com/ssheleg/make-skill
 ---
 

package/plugins/make-skill/skills/make-skill/references/authoring.md

@@ -14,6 +14,7 @@ and [agentskills.io](https://agentskills.io/skill-creation/best-practices).
 
 - Naming — what to call the skill
 - Description — the entire triggering budget
+- Where a rule lives decides whether it exists
 - Trigger eval loop — when firing is wrong
 - Degrees of freedom — how prescriptive to be
 - Body patterns worth copying
@@ -65,11 +66,50 @@ each of which changes whether the skill fires:
 - Agents skip skills for tasks they can already do in one step. Descriptions earn
   their keep on specialized or multi-step work.
 
+## Where a rule lives decides whether it exists
+
+**If your guidance was not in the loaded context, it did not happen.** That is not a
+figure of speech about attention — it is the measured difference between two kinds of
+channel.
+
+| | Channel | Result |
+|---|---|---|
+| **Soft** | a README, an `AGENTS.md` beside the code, comments inside a dependency directory, a warning field in an API response | **failed completely** |
+| **Hard** | the skill file itself, an error message, CLI help, an install prompt | worked |
+
+Two failures inside that are worth naming separately, because each looks like it should
+work. Agents **rarely open files inside dependency directories** — a rule written there is
+a rule nobody reads. And agents **parse the data out of an API response while ignoring the
+warning field in the same payload**: the bytes arrived, the guidance did not.
+
+The consequences for a skill author:
+
+- **A rule that matters belongs in `SKILL.md` or in a reference the body links**, not in a
+  neighbouring document that happens to be nearby in the repository. Proximity is not
+  loading.
+- **An error message is a steering surface**, and often the best one — it arrives exactly
+  when the agent is wrong, in the channel it is already reading. Error-based steering
+  reliably corrected requests where a soft warning did not.
+- **Restructuring for progressive disclosure is not only a budget move**: doing it bought
+  about **10% better performance at lower token cost**, because what remained in the body
+  was what had to be read every time.
+
+Two figures for the other end of the funnel, both about hard channels: promoting a skill
+from CLI login converted at **30–35%** to an install, and moving templates and examples
+*inside* the skill rather than into the system prompt cut time-to-first-token by **18.1%**.
+
 ## Trigger eval loop — when firing is wrong
 
 1. Write ~20 realistic queries: 8–10 `should_trigger: true`, 8–10 `false`. The
    valuable negatives are **near-misses** that share keywords but need something
    else.
+
+   **The negatives are the half an author deletes first, and the one with a number
+   behind it.** Moving to skill-based routing dropped triggering by about **20%** in
+   targeted evals before negative examples and edge-case coverage were added back; with
+   them the same skill reached **73% → 85%** routing accuracy. A trigger corpus that is
+   all positives measures whether the skill fires, never whether it stays quiet — and
+   staying quiet is half of what routing is.
 2. Run each 3× against the agent with the skill installed → trigger rate;
    pass threshold 0.5.
 3. Split 60% train / 40% validation, fixed across iterations. Tune only on train
@@ -242,8 +282,24 @@ skill documents imagined problems.
 4. **Write the minimum** that fixes the gaps.
 5. **Iterate** — re-run, compare to baseline, refine.
 
-Evaluation record — there is no built-in runner, so keep them in the repo as
-data and run them yourself. House layout: `test/evals/triggers.json` (~20
+**The upstream runner exists — check whether it runs for you before building
+around it.** `claude plugin eval <target>` reads `evals/<case>/case.yaml`, or a
+`prompt.md` beside `graders/*.md`, and `--ablation with-without` runs the
+no-plugin arm and reports the delta — steps 3 and 5 above, performed for you.
+It defaults to publishing its HTML report to claude.ai; `--no-publish` keeps it
+local, which is the right default for an unreleased skill.
+
+**Measured 2026-08-31 on Claude Code 2.1.236: every path — `eval`, `eval init`,
+`eval init --bare`, `eval <target>` — prints ``plugin eval` is currently in
+early access`, writes nothing and exits 1.** No settings key, environment
+variable or feature flag on that machine turned it on. That is why the layout
+below exists and is what actually runs here; the day `claude plugin eval --help`
+is followed by a run rather than that line, this suite is the one to migrate.
+**Re-check before trusting either half of this paragraph** — an absence is the
+most perishable thing a document can assert, and nothing in this repository
+changes on the day it stops being true.
+
+House layout: `test/evals/triggers.json` (~20
 queries, half near-miss negatives) and `test/evals/scenarios.json` (≥3, the
 shape below), with any input under `test/evals/fixtures/` — **never named
 `SKILL.md`**, which would ship as a real skill:

package/plugins/make-skill/skills/make-skill/references/distribution.md

@@ -65,7 +65,7 @@ the shape from `ssheleg/super-ux`:
 ├── cursor/rules/*.mdc                  # if agent-rules make sense for Cursor
 ├── bin/<name>.js + package.json        # npx installer (zero-dep Node)
 ├── test/validate.py                    # consistency validator (stdlib only)
-├── test/evals/                         # triggers.json + scenarios.json (data, run by a human)
+├── test/evals/                         # triggers.json + scenarios.json (driven here; `claude plugin eval` is gated, authoring.md)
 ├── .github/workflows/validate.yml      # validator on push+PR (+ release.yml, off by default)
 ├── install.sh                          # POSIX fallback
 ├── README.md (English-first), CHANGELOG.md, LICENSE (MIT)

package/plugins/make-skill/skills/make-skill/references/retrofit.md

@@ -98,11 +98,13 @@ Report the table before changing anything, then fix.
 7. **Validator**: present, green, and able to fail — run the negative test. CI
    present, last run `success`, with `claude plugin validate --strict` as its own
    job so an upstream outage cannot mask a house failure.
-8. **Evaluations** exist in `test/evals/` (`references/authoring.md`): ≥3
+8. **Evaluations** exist and have been executed (`references/authoring.md`): ≥3
    behavioral scenarios, a trigger set whose negatives are near-misses, both
    classes on both sides of the train/validation split, coexistence checked
    against the skills already installed, and a re-run on every model the skill
-   claims support for.
+   claims support for. Try `claude plugin eval` first and record what it did —
+   authored-but-never-executed is the state this item exists to catch, and the
+   house layout in `test/evals/` is the fallback for when that runner is gated.
 9. **README**: badges (npm/CI/license), install + update matrix, English-first
    prose, and the bundled `references/` listed so a reader sees what ships.
 10. **Distribution live-checks** (`references/distribution.md`): `npx --yes