@ssheleg/make-skill 0.8.0 → 0.8.1

package/CHANGELOG.md

@@ -1,5 +1,55 @@
 # Changelog
 
+## v0.8.1 — 2026-08-03
+
+A file-by-file re-read of everything 0.8.0 touched, and of everything it didn't.
+Fourteen findings, all fixed. The two that would have misled a reader:
+
+### Fixed
+- **The token estimate was asserted, not measured.** The validator's comment
+  claimed dense markdown runs "closer to 3.7 chars/token, so this under-reports";
+  nothing had ever been counted. Tokenizing the bundle gives **3.78–4.47**
+  chars/token (3.9 for `SKILL.md`), so the divisor is now 3.9, measured — and
+  the recalibration immediately failed the body it was checking, which is what a
+  calibrated rule is for. The audit checklist moved to a new
+  **`references/retrofit.md`**, bringing the body to ~4.6k tokens with headroom.
+- **`claude plugin details` and a real tokenizer disagree by ~40%** — the CLI
+  reported ~7.2k on-invoke for a `SKILL.md` that tokenizes at ~5.0k, because its
+  estimator assumes ~2.8 chars/token. Measured across six installed skills and
+  recorded as a gotcha: budget against a tokenizer, expect the CLI to look
+  alarming, and never gut a body that is genuinely inside the budget.
+- **`SECURITY.md` described paths the installers do not write.** It claimed
+  `bin/make-skill.js` and `install.sh` copy into `~/.claude/commands/`; that
+  command file was deleted in 0.6.x and CI asserts its absence on every run. The
+  threat model now matches the code, and states what `test/evals/fixtures/` is —
+  including that the malicious sample is inert data, never installed.
+- **The trigger split could not measure what it claimed.** `train: q01-q12` /
+  `validation: q13-q20` put all ten positives in train and left validation with
+  nothing but negatives, so the held-out half could only detect false-firing.
+  Split is now explicit ids with both classes on both sides, and the validator
+  rejects a half that holds only one class.
+- **`displayName` was required by the canon and absent from the skeletons.**
+  Both manifest templates now seed it, and the validator enforces it in the
+  templates, in `plugin.json`, and in the marketplace entry — where it belongs
+  (the marketplace root takes neither `displayName` nor `homepage`).
+- **`/make-skill` is only half the truth.** As a plugin the command is
+  `/make-skill:make-skill`; the bare form is what a skills-directory install
+  gets. README and canon now state both, and the canon says to write both rather
+  than promise one.
+- `distribution.md`'s table of contents listed a "shadow rule" section that did
+  not exist as a heading — the exact failure a table of contents prevents. It is
+  a real section now.
+- Counts that had drifted: CONTRIBUTING said CI runs "four negative self-tests"
+  (six groups), the issue-template contact link said "the other four skills"
+  (five), the retrofit fixture header claimed eight planted defects (nine, now
+  scored one per line).
+- Dead `EVAL_DIR` variable removed; `test/evals/` added to the Cursor rule's
+  layout tree and to the templates README, both of which still described a repo
+  without it; the PR template now asks for the two `--strict` runs and for an
+  eval re-run when the description changes.
+- Three more negative self-tests (split classes, `displayName` ×2), keeping the
+  rule that no validator rule ships without one.
+
 ## v0.8.0 — 2026-08-03
 
 Read Anthropic's four Agent Skills pages end to end (overview, authoring best
@@ -63,9 +113,10 @@ six new validator rules, and one self-violation fixed.
 - **This skill was over its own token budget.** `SKILL.md` was ~5.2k estimated
   tokens against a stated `< 5000` — a rule the repo enforced on everyone but
   itself, because only the 500-line half was ever checked. The body budget is
-  now measured (chars/4, frontmatter excluded as level-1 metadata) and the body
-  was cut to ~4.6k by moving the repo layout, the two Claude Code gates, the
-  validator spec and the release checklist into `references/distribution.md`.
+  now measured (frontmatter excluded as level-1 metadata) and the body was cut
+  by moving the repo layout, the two Claude Code gates, the validator spec and
+  the release checklist into `references/distribution.md`. (The divisor used
+  here was a guess; v0.8.1 measured it and cut further.)
 - **Reference files longer than 100 lines now open with a `## Contents` list.**
   Agents preview long files with `head`; without a table of contents they act on
   the first hundred lines and never learn the rest exists. All eight references

package/README.md

@@ -31,11 +31,14 @@ make a skill that turns our incident runbooks into a triage workflow
 …or point it at something that already exists:
 
 ```
-/make-skill audit ./skills/my-skill against the spec
+/make-skill:make-skill audit ./skills/my-skill against the spec
 ```
 
-Inside a skill repo, `/make-skill` with no argument detects the situation and runs
-the audit rather than asking you what you meant.
+With no argument it detects the situation — a `SKILL.md` or `.claude-plugin/` in
+the current directory means it runs the audit rather than asking you what you
+meant. The command is `/make-skill:make-skill` when installed as a plugin and
+`/make-skill` when it sits in a skills directory; both reach the same skill, and
+asking in plain language works either way.
 
 ## What it does
 
@@ -122,6 +125,7 @@ agent opens only when the situation calls for them:
 | `authoring.md` | the craft — naming, third-person descriptions, degrees of freedom, workflows and feedback loops, script rules, evaluation-driven development |
 | `surfaces.md` | Claude Code vs the Claude API vs claude.ai — the Skills API (upload, versions, 8 per request), and the no-network / no-package-install limits that break scripts moved between surfaces |
 | `enterprise.md` | installing a skill you didn't write, and running a fleet — risk tiers, the review checklist, the five approval gates, lifecycle, recall limits, rollback |
+| `retrofit.md` | the audit procedure — the 13-item checklist, what counts as evidence for a PASS, and the short form for a personal skill |
 | `claude-code-plugin.md` | the [Claude Code layer](https://code.claude.com/docs/en/plugins-reference) — `plugin.json` / `marketplace.json` schemas, plugin sources, component locations, host-only front-matter, path variables, cache and symlink rules, the `claude plugin` CLI |
 | `distribution.md` | the repo layout, every install channel, exact CLI flags, npm publishing traps, the release checklist |
 | `mcp.md` | [MCP](https://modelcontextprotocol.io) — skill vs server, primitives and methods, transports, consent and untrusted-output rules |

package/SECURITY.md

@@ -16,12 +16,16 @@ unless you'd prefer otherwise.
 `make-skill` ships Markdown plus two small installers. Knowing exactly what runs
 where is most of the threat model:
 
-- **`bin/make-skill.js`** (run via `npx`) and **`install.sh`** copy files into
-  `~/.claude/skills/make-skill` and `~/.claude/commands/`. They create and
-  overwrite only those paths, never elsewhere in `$HOME`, and overwrite only with
-  `--force`. Both are zero-dependency: no network calls, no postinstall script.
+- **`bin/make-skill.js`** (run via `npx`) and **`install.sh`** copy the skill
+  directory into `~/.claude/skills/make-skill` and nothing else — no command file,
+  no other path in `$HOME` — and overwrite only with `--force`. Both are
+  zero-dependency: no network calls, no postinstall script. CI asserts the
+  absence of `~/.claude/commands/make-skill.md` on every run.
 - **`test/validate.py`** reads repository files and exits with a status. It writes
-  nothing.
+  nothing. `test/evals/` is inert data — text a human feeds to an agent — and
+  `test/evals/fixtures/untrusted-skill.md` is a deliberately malicious *sample*
+  used to score a review; it is never installed or executed, and its endpoints
+  are `example.com` placeholders.
 - The Claude Code plugin and the `skills` CLI channels are handled by those tools,
   not by code in this repo.
 

package/cursor/rules/make-skill.mdc

@@ -77,6 +77,7 @@ cursor/rules/*.mdc                         # self-contained, NO relative links
 templates/*.md                             # skeletons — NEVER name one SKILL.md
 bin/<name>.js + package.json               # zero-dep npx installer
 test/validate.py                           # stdlib-only consistency validator
+test/evals/                                # triggers.json + scenarios.json (data, run by a human)
 .github/workflows/validate.yml (+ release.yml, off by default)
 install.sh, README.md (English-first), CHANGELOG.md, LICENSE (MIT)
 CONTRIBUTING.md + SECURITY.md              # required once the repo is public
@@ -132,6 +133,10 @@ checker, not a house opinion:
 - **Cursor .mdc:** embed contracts inline (files get copied into foreign projects).
 - **e2e npx from a NON-repo cwd** (inside the repo it resolves the local package →
   false "command not found").
+- **Two token counters, ~40% apart:** `claude plugin details` assumes ~2.8
+  chars/token and shows a far bigger on-invoke cost than a real tokenizer
+  (3.8–4.5 chars/token). Budget the body against a tokenizer; expect the CLI to
+  look alarming for a file that is genuinely inside 5000 tokens.
 - **Installing a skill you did not write is installing software.** Read every
   file, run scripts sandboxed, grep for network calls and credentials, and check
   the instructions for "ignore previous rules", hidden actions, or data routed

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/make-skill",
-  "version": "0.8.0",
+  "version": "0.8.1",
   "description": "Create, retrofit, audit, and ship agent skills & Claude Code plugins the proven ssheleg way — conformance to the Agent Skills open standard AND Anthropic's platform rules (front-matter limits, disclosure budgets, per-surface runtime limits, the Skills API, evals) plus the Claude Code plugin reference (manifest schemas, component layout, claude plugin validate --strict), marketplace repo layout, version sync, validator + CI, multi-channel distribution (plugin, vercel skills CLI, npx, Cursor), npm gotchas, the review checklist for third-party skills, and MCP / A2A rules for protocol-connected skills. This package is the installer CLI.",
   "keywords": [
     "skill",

package/plugins/make-skill/.claude-plugin/plugin.json

@@ -3,7 +3,7 @@
   "name": "make-skill",
   "displayName": "Make Skill",
   "description": "Create, retrofit, audit, and ship agent skills & Claude Code plugins the proven ssheleg way: conformance to the Agent Skills open standard, Anthropic's platform rules (surfaces, Skills API, evals) and the Claude Code plugin reference, marketplace repo layout, version sync, validator + CI, multi-channel distribution (plugin, vercel skills CLI, npx, Cursor), npm gotchas, end-to-end first publish, the review checklist for third-party skills, plus MCP / A2A references for protocol-connected skills.",
-  "version": "0.8.0",
+  "version": "0.8.1",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"

package/plugins/make-skill/skills/make-skill/SKILL.md

@@ -19,6 +19,7 @@ Copy from a working repo (usually `~/DATA/<name>`): **`ssheleg/super-ux`**
 | `references/authoring.md` | writing or tuning a body/description — naming, third-person rule, degrees of freedom, script rules, eval loops |
 | `references/surfaces.md` | shipping anywhere but Claude Code — Skills API upload/versions/8-per-request, claude.ai zip, the no-network / no-install limits |
 | `references/enterprise.md` | installing someone else's skill, or governing a fleet — risk tiers, review checklist, approval gates, lifecycle, recall limits |
+| `references/retrofit.md` | auditing an existing skill/repo — the 13-item checklist, the evidence rules, the short form for personal skills |
 | `references/claude-code-plugin.md` | anything shipping as a **Claude Code plugin/marketplace** — manifest schemas, component layout, LSP/monitors, path variables, `validate` failures |
 | `references/distribution.md` | the repo layout, publishing, channels, releases |
 | `references/mcp.md` | the skill calls/wraps/documents an **MCP** server, or you're choosing skill vs server |
@@ -109,11 +110,14 @@ House additions on top of the spec:
   passes; a skill built without a baseline documents imagined problems. Adding a
   member to an existing family → test **coexistence**: a broad description
   steals the triggers of the skills already installed.
-- Ship a one-command entry point `/<name>`: idempotent — inspect state → repair
-  missing pieces → status report → exactly ONE suggested next action. Detect
-  mode, never ask. (Pattern: super-ux `/ux`.) **The skill IS that command** —
-  a `commands/<name>.md` beside `skills/<name>/` registers `/<name>` twice; add
-  a `commands/*.md` only under a DIFFERENT name, with a quoted `argument-hint`.
+- Ship a one-command entry point: idempotent — inspect state → repair missing
+  pieces → status report → exactly ONE suggested next action. Detect mode, never
+  ask. (Pattern: super-ux `/ux`.) **The skill IS that command** — a
+  `commands/<name>.md` beside `skills/<name>/` registers the same command twice;
+  add a `commands/*.md` only under a DIFFERENT name, with a quoted
+  `argument-hint`. The invocation differs by channel: `/<skill>` from a skills
+  directory, `/<plugin>:<skill>` once it is installed as a plugin — write both
+  in the README rather than promising one.
 - Never overwrite user data: seed only when absent; overwrite only behind
   `--force`.
 
@@ -145,6 +149,8 @@ writing the first file.** What holds regardless:
 - **Both `--strict` runs green, in CI, as their own job** — and they read
   MANIFESTS only, so front-matter rules live in your own `test/validate.py`,
   which needs a negative self-test: a validator that can't fail is decoration.
+  Ship `$schema` and `displayName` in `plugin.json` AND in the marketplace
+  ENTRY (the marketplace root takes neither `displayName` nor `homepage`).
 - A public repo owes a reader an English-first README, `CONTRIBUTING.md` with
   the offline commands that verify a change, and `SECURITY.md` naming a private
   channel and what the installers touch.
@@ -163,46 +169,19 @@ next tag publishing without a human.
 ## Retrofit (bring an existing skill/repo up to standard)
 
 Audit first, fix second, in the same session. Verdict per item: PASS / GAP with
-evidence (`file:line` or command output) — never "looks fine".
-
-1. **Spec floor** (`references/agent-skills-spec.md`): name charset + ≤64 +
-   ==dir + no reserved word or angle bracket; description ≤1024, "Use when…",
-   third person, EN and RU triggers; no front-matter key outside spec ∪ host
-   extensions; body <500 lines / <5000 tokens; every bundled file one level
-   deep, linked from the body with a load trigger, `## Contents` past 100 lines.
-2. **Anthropic floor** if it ships as a plugin
-   (`references/claude-code-plugin.md`): both `claude plugin validate <plugin
-   dir|.> --strict` runs exit 0; `$schema` present; components at the plugin
-   root, never inside `.claude-plugin/`; `claude plugin details <name>` token
-   cost worth paying.
-3. **Surface honesty** (`references/surfaces.md`): every runtime need (network,
-   packages, MCP server) declared in `compatibility`; no script assuming an
-   install the target surface forbids.
-4. One-job check; shared contracts INSIDE the skill dir — verify by installing
-   via the skills CLI and checking the files actually arrived.
-5. Entry-point command exists, idempotent (inspect → repair → status → one next
-   action).
-6. Layout matches the tree; manifests complete; version sync ×4 (×5 with
-   `metadata.version`).
-7. Validator present, green, and able to fail (run the negative test); CI
-   present, last run `success`.
-8. **Evals exist** in `test/evals/` (`references/authoring.md`): ≥3 scenarios
-   plus should-NOT-trigger queries, coexistence checked against the installed
-   set, re-run on every model the skill claims.
-9. README: badges (npm/CI/license), install + update matrix, English-first,
-   bundled `references/` listed so a reader sees what ships.
-10. Distribution live-checks (`references/distribution.md`): `npx --yes skills
-    add <repo> --list` lists ONLY real skills; `npx <name>` works from a
-    non-repo cwd (if npm published); `.mdc` rules valid, no relative links.
-11. Repo meta: homepage + description + topics on the forge; LICENSE; CHANGELOG
-    current; public repos also CONTRIBUTING.md + SECURITY.md.
-12. Gotcha compliance: the list below, plus the installer traps in
-    `references/distribution.md` if the repo ships a CLI or validator.
-13. MCP or A2A involved: see Protocol-connected skills below.
+evidence — a `file:line` or the output of the command you actually ran. "Looks
+fine" is not a verdict, and neither is a PASS on a check that was reasoned about
+instead of executed.
+
+**The 13-item checklist is in `references/retrofit.md` — open it before
+auditing.** It covers, in order: the spec floor, the Anthropic plugin floor,
+surface honesty, one-job, entry point, layout and version sync, validator and
+CI, evaluations, README, distribution live-checks, repo meta, gotcha compliance,
+protocol dependencies. For a PERSONAL skill only three of them apply (spec
+floor, one-job, entry point) — that file says which.
 
 **Then:** report the gap table, fix everything fixable now, bump a minor/patch
-version, run the release checklist. For a PERSONAL skill, retrofit = items 1, 4,
-5 only.
+version, run the release checklist.
 
 ## Promote (personal → distributable)
 
@@ -301,6 +280,12 @@ Non-negotiables for any such skill:
 - **Time-branching text rots.** "Before August, use the old API" is wrong the day
   it ships — superseded material goes under `## Old patterns`, and a dated
   provenance line ("*read from the spec on 2026-08-03*") ages well.
+- **Two token counters, ~40% apart.** `claude plugin details` reported ~7.2k
+  on-invoke for a SKILL.md that a real tokenizer puts at ~5.0k (measured across
+  six installed skills: the CLI assumes ~2.8 chars/token, cl100k gives 3.8–4.5).
+  Its estimator is the pessimistic one — budget against a tokenizer, then expect
+  the CLI to show a bigger number, and never "fix" a body that is already
+  inside 5000 real tokens because the CLI looked alarming.
 
 ## Release (every version)
 

package/plugins/make-skill/skills/make-skill/references/claude-code-plugin.md

@@ -22,8 +22,8 @@ must also run on Cursor, Codex, or the skills CLI.
 ## Contents
 
 - The gate: `claude plugin validate`
-- `plugin.json` fields
-- `marketplace.json` fields, reserved names, plugin entries
+- `plugin.json` — `.claude-plugin/plugin.json` (field table)
+- `marketplace.json` — `.claude-plugin/marketplace.json` (fields, reserved names, plugin entries)
 - Component locations
 - Skill frontmatter — host extensions
 - Agents, hooks, MCP inside a plugin

package/plugins/make-skill/skills/make-skill/references/distribution.md

@@ -19,6 +19,8 @@ API or claude.ai instead is `references/surfaces.md`.
 - Release checklist (every version)
 - Toggleable release automation
 
+## The shadow rule (one channel per agent)
+
 Hard rule across all channels: **one channel per agent.** Two installs of the same
 skill on one agent = two listings, and the stale copy shadows the fresh one.
 

package/plugins/make-skill/skills/make-skill/references/retrofit.md

@@ -0,0 +1,91 @@
+# Retrofit — the audit procedure
+
+**Load this when:** auditing an existing skill, plugin or repo against this
+standard — including the no-argument `/make-skill` path, where a `SKILL.md`,
+`.claude-plugin/` or `plugins/*/skills/*/` in the current directory means Retrofit.
+
+## Contents
+
+- How to report (the only acceptable verdict format)
+- The 13-item audit checklist
+- Personal skills — the short form
+- After the audit
+
+## How to report
+
+Audit first, fix second, in the same session. One verdict per item:
+
+| Verdict | Requires |
+|---|---|
+| **PASS** | the check was actually run — a command's output or a `file:line` |
+| **GAP** | what is wrong, the evidence, and the fix that closes it |
+
+"Looks fine" is not a verdict. Neither is a PASS on a check that was reasoned
+about rather than executed: `python3 test/validate.py`, `claude plugin validate
+… --strict`, `npx skills add <repo> --list` all produce output, and the output is
+the evidence. Report the gap table before changing anything, then fix.
+
+## The 13-item audit checklist
+
+1. **Spec floor** (`references/agent-skills-spec.md`): `name` charset + ≤64 +
+   equal to the directory + no `anthropic`/`claude` + no angle brackets;
+   `description` ≤1024, starts "Use when…", third person, English AND Russian
+   triggers, says what it does and when; no front-matter key outside spec ∪ host
+   extensions; body <500 lines and <5000 tokens; every `references/`,
+   `scripts/`, `assets/` file one level deep, linked from the body with a stated
+   load trigger, and a `## Contents` list past 100 lines. A skill can pass every
+   house rule and still be invalid upstream.
+2. **Anthropic floor** if it ships as a plugin
+   (`references/claude-code-plugin.md`): `claude plugin validate <plugin dir>
+   --strict` and `claude plugin validate . --strict` both exit 0; `$schema` and
+   `displayName` in the manifest and in the marketplace entry; components at the
+   plugin root, never inside `.claude-plugin/`; `claude plugin details <name>`
+   token cost worth paying every session.
+3. **Surface honesty** (`references/surfaces.md`): every runtime need — network,
+   packages, MCP server — declared in `compatibility`; no script assuming an
+   install the target surface forbids. A `pip install` in a workflow step is a
+   Claude Code skill claiming to be portable.
+4. **One job.** Two concerns → two skills plus a shared contract, and the
+   contract lives INSIDE each skill directory. Verify by installing through the
+   skills CLI and listing what actually arrived, not by reading the repo.
+5. **Entry point** exists and is idempotent: inspect state → repair what is
+   missing → status report → exactly ONE suggested next action, detected rather
+   than asked. Check the invocation for the channel it ships on (`/<skill>` from
+   a skills dir, `/<plugin>:<skill>` as a plugin).
+6. **Layout and manifests**: matches the tree in `references/distribution.md`;
+   version sync ×4 (×5 when `SKILL.md` carries `metadata.version`).
+7. **Validator**: present, green, and able to fail — run the negative test. CI
+   present, last run `success`, with `claude plugin validate --strict` as its own
+   job so an upstream outage cannot mask a house failure.
+8. **Evaluations** exist in `test/evals/` (`references/authoring.md`): ≥3
+   behavioral scenarios, a trigger set whose negatives are near-misses, both
+   classes on both sides of the train/validation split, coexistence checked
+   against the skills already installed, and a re-run on every model the skill
+   claims support for.
+9. **README**: badges (npm/CI/license), install + update matrix, English-first
+   prose, and the bundled `references/` listed so a reader sees what ships.
+10. **Distribution live-checks** (`references/distribution.md`): `npx --yes
+    skills add <repo> --list` lists ONLY real skills; `npx <name>` works from a
+    non-repo cwd if it is published; `.mdc` rules valid and free of relative
+    links.
+11. **Repo meta**: homepage, description and topics set on the forge; LICENSE
+    present and declared in the front matter and the marketplace entry;
+    CHANGELOG current; public repos also carry CONTRIBUTING.md and SECURITY.md.
+12. **Gotcha compliance**: the list in `SKILL.md`, plus the installer traps in
+    `references/distribution.md` when the repo ships a CLI or a validator.
+13. **Protocol dependencies** if it touches MCP or A2A (`references/mcp.md`,
+    `references/a2a.md`): dependency and protocol version in `compatibility`,
+    discovery instead of hardcoded tool names, the untrusted-output rule stated,
+    interactive auth handled as a human step rather than a retry loop.
+
+## Personal skills — the short form
+
+A skill in `~/.claude/skills/<name>/` has no repo, no manifests, no CI and no
+distribution, so items 1, 4 and 5 are the whole audit. Do not invent a release
+process for a file that loads next session.
+
+## After the audit
+
+Report the gap table, fix everything fixable now, bump a minor or patch version,
+and run the release checklist in `references/distribution.md`. A retrofit that
+ends in a list of things someone should do later is a report, not a retrofit.

package/templates/README.md

@@ -27,5 +27,8 @@ level deep, each with a stated load trigger in the body. A sibling directory
 reaches Claude Code plugins but arrives broken on every other agent.
 
 Distributable repos also get the full tree (manifests, `bin/`, `test/validate.py`,
-workflows, `install.sh`, README/CHANGELOG/LICENSE) — copy those from a reference
-impl (`ssheleg/task-pipeline` or `ssheleg/super-ux`) per the SKILL.md layout.
+`test/evals/`, workflows, `install.sh`, README/CHANGELOG/LICENSE) — copy those
+from a reference impl (`ssheleg/task-pipeline` or `ssheleg/super-ux`) per the
+layout in `references/distribution.md`. The evaluation suite is not optional
+scaffolding: the canon requires ≥3 behavioral scenarios and a trigger set with
+near-miss negatives before a skill ships.

package/templates/marketplace.template.json

@@ -9,6 +9,7 @@
   "plugins": [
     {
       "name": "<plugin-name>",
+      "displayName": "<Plugin Name>",
       "source": "./plugins/<plugin-name>",
       "description": "<what the plugin gives an agent>",
       "version": "0.1.0",

package/templates/plugin.template.json

@@ -1,6 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "<plugin-name>",
+  "displayName": "<Plugin Name>",
   "description": "<what the plugin gives an agent, in one sentence>",
   "version": "0.1.0",
   "author": {