@ssheleg/make-skill 0.25.3 → 0.27.0

package/CHANGELOG.md

@@ -1,3 +1,71 @@
+## v0.27.0 — the standard-keeper asked WHEN nine times and WHAT never
+
+`B-139`. `audit_skill.py` carried nine `DESC_*` rules — length, headroom, person, Russian
+triggers, XML, target, the `Use when` opener — and **not one asked whether the
+description says what the skill DOES.** That is the half of Anthropic's guidance `B-76`
+quoted directly: *a description that never says what the skill does passes*. Measured
+2026-09-03 across the family: **28 of 28 skills pass `DESC_USEWHEN`, 0 gaps** — the WHEN
+half is universal and the WHAT half was unchecked.
+
+`DESC_WHAT` measures the description with its mechanical parts removed — the `Use when`
+opener, the trigger list, the `Not for` clause and the opt-out sentence — and refuses
+what is left below 60 characters.
+
+**Built on the PARSED description, which is the whole reason the previous attempt was
+refused.** That prototype read raw front matter, reported a **0-character** WHAT half for
+six skills and missed the opening clause of twenty, because several descriptions are YAML
+block scalars (`>-`) a raw-text regex reads straight past. `parse_frontmatter` already
+resolves them.
+
+**The floor is stated with its margin.** Measured across the shipped family the smallest
+honest WHAT half is **149** characters (`ux-audit`) and the largest **949**
+(`seo-aeo-audit`), so 60 clears every real description by more than double while still
+catching `Use when the user asks. Triggers - "делай" / "do it".`, whose WHAT half is
+**13**. A case asserts that margin, so raising the floor without re-measuring fails.
+
+**This rule finds no gap today and that is said out loud.** A standard is for the
+description not yet written, and a rule that fires on nothing now is worth only what its
+plants prove — so it was watched refusing a WHEN-only description and watched going
+silent when disabled.
+
+**One case was reworded after being caught claiming somebody else's work.** It asserted
+the WHAT half is read from the parsed value rather than raw text — and the plant for that
+property is caught by an existing case guarding `parse_frontmatter`'s folding, one layer
+up. It now claims only what it holds: both spellings of one description reach the same
+verdict, neither passes vacuously, and the parsed value carries no newline.
+
+## v0.26.0 — where a rule lives decides whether it exists
+
+`authoring.md` treated progressive disclosure as a budget question. It is also a
+**delivery** question, and that half was measured.
+
+**Soft channels failed completely.** A README, an `AGENTS.md` beside the code, comments
+inside a dependency directory, a warning field in an API response — none of them steered
+anything. Two failures inside that deserve their own names, because each looks like it
+should work: agents **rarely open files inside dependency directories**, so a rule written
+there is a rule nobody reads; and agents **parse the data out of an API response while
+ignoring the warning field in the same payload** — the bytes arrived, the guidance did not.
+
+**Hard channels worked**: the skill file itself, an error message, CLI help, an install
+prompt. So a rule that matters belongs in `SKILL.md` or a reference the body links, never
+in a neighbouring document that is merely nearby — **proximity is not loading**. And an
+error message is a steering surface rather than an apology: it arrives exactly when the
+agent is wrong, in a channel it is already reading.
+
+The numbers, all on hard channels: restructuring for progressive disclosure bought about
+**10%** better performance *at lower token cost*; promoting a skill from CLI login converted
+at **30–35%** to an install; moving templates and examples inside the skill rather than into
+the system prompt cut time-to-first-token by **18.1%**.
+
+**And the trigger corpus's negative half now carries the number that defends it.** House
+style already mandates *"~20 queries, half near-miss negatives"* — with no evidence that
+omitting them costs anything, which is exactly the shape of a rule an author trims under
+deadline. Moving to skill-based routing dropped triggering by about **20%** in targeted
+evals before negative examples and edge-case coverage were added back; with them the same
+skill reached **73% → 85%** routing accuracy. A corpus that is all positives measures
+whether the skill fires and never whether it stays quiet — and staying quiet is half of what
+routing is.
+
 # Changelog
 
 ## v0.25.3 — the runner we said did not exist, and the date a claim about someone else now carries

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/make-skill",
-  "version": "0.25.3",
+  "version": "0.27.0",
   "description": "Create, retrofit, audit, and ship agent skills & Claude Code plugins the proven ssheleg way \u2014 conformance to the Agent Skills open standard AND Anthropic's platform rules (front-matter limits, disclosure budgets, per-surface runtime limits, the Skills API, evals) plus the Claude Code plugin reference (manifest schemas, component layout, claude plugin validate --strict), marketplace repo layout, version sync, validator + CI, multi-channel distribution (plugin, vercel skills CLI, npx, Cursor), npm gotchas, the review checklist for third-party skills, and MCP / A2A rules for protocol-connected skills. This package is the installer CLI.",
   "keywords": [
     "skill",

package/plugins/make-skill/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "make-skill",
   "displayName": "Make Skill",
   "description": "Create, retrofit, audit, and ship agent skills & Claude Code plugins the proven ssheleg way: conformance to the Agent Skills open standard, Anthropic's platform rules (surfaces, Skills API, evals) and the Claude Code plugin reference, marketplace repo layout, version sync, validator + CI, multi-channel distribution (plugin, vercel skills CLI, npx, Cursor), npm gotchas, end-to-end first publish, the review checklist for third-party skills, plus MCP / A2A references for protocol-connected skills.",
-  "version": "0.25.3",
+  "version": "0.27.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/make-skill/skills/make-skill/SKILL.md

@@ -5,7 +5,7 @@ license: MIT
 compatibility: Authoring works on any agent. The bundled scripts/ need python3. Publishing steps need git, gh, node and npm; the plugin gates need the claude CLI. Not usable on the Claude API surface, which has no network and no runtime package install.
 metadata:
   author: ssheleg
-  version: "0.25.3"
+  version: "0.27.0"
   homepage: https://github.com/ssheleg/make-skill
 ---
 

package/plugins/make-skill/skills/make-skill/references/authoring.md

@@ -14,6 +14,7 @@ and [agentskills.io](https://agentskills.io/skill-creation/best-practices).
 
 - Naming — what to call the skill
 - Description — the entire triggering budget
+- Where a rule lives decides whether it exists
 - Trigger eval loop — when firing is wrong
 - Degrees of freedom — how prescriptive to be
 - Body patterns worth copying
@@ -65,11 +66,50 @@ each of which changes whether the skill fires:
 - Agents skip skills for tasks they can already do in one step. Descriptions earn
   their keep on specialized or multi-step work.
 
+## Where a rule lives decides whether it exists
+
+**If your guidance was not in the loaded context, it did not happen.** That is not a
+figure of speech about attention — it is the measured difference between two kinds of
+channel.
+
+| | Channel | Result |
+|---|---|---|
+| **Soft** | a README, an `AGENTS.md` beside the code, comments inside a dependency directory, a warning field in an API response | **failed completely** |
+| **Hard** | the skill file itself, an error message, CLI help, an install prompt | worked |
+
+Two failures inside that are worth naming separately, because each looks like it should
+work. Agents **rarely open files inside dependency directories** — a rule written there is
+a rule nobody reads. And agents **parse the data out of an API response while ignoring the
+warning field in the same payload**: the bytes arrived, the guidance did not.
+
+The consequences for a skill author:
+
+- **A rule that matters belongs in `SKILL.md` or in a reference the body links**, not in a
+  neighbouring document that happens to be nearby in the repository. Proximity is not
+  loading.
+- **An error message is a steering surface**, and often the best one — it arrives exactly
+  when the agent is wrong, in the channel it is already reading. Error-based steering
+  reliably corrected requests where a soft warning did not.
+- **Restructuring for progressive disclosure is not only a budget move**: doing it bought
+  about **10% better performance at lower token cost**, because what remained in the body
+  was what had to be read every time.
+
+Two figures for the other end of the funnel, both about hard channels: promoting a skill
+from CLI login converted at **30–35%** to an install, and moving templates and examples
+*inside* the skill rather than into the system prompt cut time-to-first-token by **18.1%**.
+
 ## Trigger eval loop — when firing is wrong
 
 1. Write ~20 realistic queries: 8–10 `should_trigger: true`, 8–10 `false`. The
    valuable negatives are **near-misses** that share keywords but need something
    else.
+
+   **The negatives are the half an author deletes first, and the one with a number
+   behind it.** Moving to skill-based routing dropped triggering by about **20%** in
+   targeted evals before negative examples and edge-case coverage were added back; with
+   them the same skill reached **73% → 85%** routing accuracy. A trigger corpus that is
+   all positives measures whether the skill fires, never whether it stays quiet — and
+   staying quiet is half of what routing is.
 2. Run each 3× against the agent with the skill installed → trigger rate;
    pass threshold 0.5.
 3. Split 60% train / 40% validation, fixed across iterations. Tune only on train

package/plugins/make-skill/skills/make-skill/scripts/audit_skill.py

@@ -38,6 +38,48 @@ DESC_MAX = 1024         # spec: description is 1-1024 characters
 # budget AND the field that must grow when a near-miss skill appears ("say what
 # it is NOT for"). A description at 98% of cap cannot absorb that sentence.
 DESC_TARGET = 970
+# B-139 — the nine `DESC_*` rules ask WHEN and never WHAT.
+#
+# Anthropic's guidance asks a description to say what the skill DOES as well as when to
+# use it, and `B-76` quoted the failure directly: *a description that never says what the
+# skill does passes*. Measured 2026-09-03 across the family: **28 of 28 skills pass
+# `DESC_USEWHEN`, 0 gaps** — the WHEN half is universal and the WHAT half was unchecked.
+#
+# The WHAT half is the description with its mechanical parts removed: the `Use when`
+# opener, the trigger list, the `Not for` clause and the opt-out sentence. What remains is
+# the prose that names the act. A description that is an opener plus a trigger list leaves
+# almost nothing, which is exactly the shape the rule refuses.
+#
+# The floor is 60 and it is deliberately permissive. Measured on the shipped family, the
+# smallest honest WHAT half is **149** characters (`ux-audit`) and the largest 949, so 60
+# clears every real description by more than double and still catches
+# `Use when the user asks. Triggers - "x" / "у".`, whose WHAT half is 13.
+#
+# **This rule finds no gap today, and that is stated rather than hidden.** A standard is
+# for the description not yet written; a rule that fires on nothing now is only worth
+# anything if it was watched firing on a plant, which `test/checker_parity_test.py` does.
+DESC_WHAT_MIN = 60
+
+# Built on the PARSED description, never the raw front matter. A prototype was built
+# against raw text and refused: it reported a 0-character WHAT half for six skills and
+# missed the opening clause of twenty, because several descriptions are YAML block
+# scalars (`>-`) that a raw-text regex reads straight past. `parse_frontmatter` already
+# resolves them, and the prototype did not use it.
+_WHAT_OPENER = re.compile(r"^use when\s+", re.I)
+_WHAT_STRIP = (
+    re.compile(r"\bTriggers?\s*[-–—:].*", re.S | re.I),
+    re.compile(r"\bNot for\b.*", re.S | re.I),
+    re.compile(r"\bsay\s+['\"«].*", re.S | re.I),
+)
+
+
+def what_half(description):
+    """The prose that names the act, with the mechanical parts removed."""
+    core = _WHAT_OPENER.sub("", " ".join(str(description or "").split()))
+    for rx in _WHAT_STRIP:
+        core = rx.sub("", core)
+    return core.strip(" .,;—-")
+
 COMPAT_MAX = 500        # spec: compatibility is 1-500 characters
 BODY_MAX_LINES = 500    # spec + Anthropic: keep the body under 500 lines
 BODY_MAX_TOKENS = 5000  # spec + Anthropic: level-2 budget
@@ -270,6 +312,17 @@ def _check_description(a, fm, lines, rel, house):
             a.gap("DESC_RU", "description carries no Russian trigger phrases (house rule)", rel, ln)
         else:
             a.ok("DESC_RU", "description carries Russian trigger phrases (house rule)", rel, ln)
+        what = what_half(desc)
+        if len(what) < DESC_WHAT_MIN:
+            a.gap("DESC_WHAT", "description says WHEN and never WHAT: %d chars remain "
+                  "after the opener, the trigger list and the refusal are removed, and "
+                  "the floor is %d (house rule). Anthropic's guidance asks for both "
+                  "halves, and a description that is an opener plus a trigger list "
+                  "selects the skill without telling the model what it will do"
+                  % (len(what), DESC_WHAT_MIN), rel, ln)
+        else:
+            a.ok("DESC_WHAT", "description states WHAT the skill does in %d chars "
+                 "beyond its triggers (house rule)" % len(what), rel, ln)
         if DESC_TARGET < len(desc) <= DESC_MAX:
             a.gap("DESC_HEADROOM", "description is %d chars — inside the %d cap but past the "
                   "%d working limit (house rule): leave room for the 'what this is NOT for' "