@vegastack/vegafactory 0.19.0

package/skill/skillify/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: skillify
+description: Repo-local skill factory and auditor for the vegafactory monorepo. Use when asked to "skillify this", "make this a skill", "turn this workflow into a skill", or scaffold a new skill under skills/; when asked to "audit this skill", "is this skill complete", "check skill completeness", or score a skill against the repo contract; when deciding whether a workflow, script, or prompt should become a skill at all; when a skill description under- or over-triggers and needs tuning; or when a skill in this repo is missing tests, a README, a refresh contract, behavioral eval evidence, or repo wiring. Only for skills inside this repository, not for authoring skills in other projects or installing third-party skills.
+---
+
+# Skillify
+
+Turn a raw workflow into a properly-skilled unit of this monorepo, or audit an existing skill against the repo contract in CONTRIBUTING.md. skill-maintainer is the nearest neighbour: it owns the standards and the repo and release operations (what must be true); skillify owns the procedure (how a skill gets there) and cites those standards rather than restating them.
+
+## The contract checklist
+
+Score every item pass / fail / N/A; N/A without a one-line rationale is a fail, and a justified N/A counts as a pass in the `<passed>/8` score. "The evidence" below is the audit report or the issue's evidence comment. skill-maintainer lives at `skills/repo-tooling/skill-maintainer/`. Numbering is stable and additive, so other skills can cite "skillify item 6".
+
+1. **SKILL.md spec-compliant** — frontmatter, name grammar, description limits, and body syntax per skill-maintainer's standards (operating rules 2–6); body ≤1,200 words with detail routed to references, or a named exception in the evidence; `node packages/cli/scripts/validate-skill.mjs <skill-dir>` (the check `bun run validate:skill` runs over every skill) agrees, and it also verifies every relative link in the skill's prose resolves. Body words: the text below the frontmatter, `wc -w`.
+2. **Description triggers well** — states triggering conditions only (never the workflow), shaped per [authoring](references/authoring.md), the standard being skill-maintainer's rule 4; a query set of realistic positives and near-miss negatives at `tests/fixtures/trigger-queries.json`, `ambiguous_with` naming the competing skill. `scripts/trigger-check.mjs` (`validate:triggers` in `bun run check`) blocks when two fixtures claim one query without naming each other.
+3. **Sharp boundary** — the SKILL.md body names its nearest-neighbor skill and the one-sentence axis of difference (or states it has none). Two skills answering the same trigger get merged, not shipped.
+4. **References routed** — `references/` holds detail only some invocations need, behind a routing table; SKILL.md keeps the workflow, one excellent example, and the routes. N/A for a self-contained skill.
+5. **Scripts deterministic and tested** — bundled scripts follow the conventions in [authoring](references/authoring.md) (dependency-free Node, `--json`, documented exit codes, dry-run default, atomic writes); unit tests cover every deterministic branch. N/A when the skill ships no scripts — its quality bar is item 6.
+6. **Behavioral eval passed** — the cases in `evals/evals.json` run with-skill vs baseline per the [eval playbook](references/eval-playbook.md), at most 3 improve cycles; pass, or ship with a KNOWN_GAPS section in the skill's README.
+7. **Freshness honest** — volatile facts live in refresh-tracked files, never in SKILL.md; a skill with none states the evergreen waiver in `refresh/REFRESH.md`.
+8. **Wired and green** — README, `agents/openai.yaml`, packaging entry, root README row, and changeset in place; `bun run check` passes.
+
+**Verdict:** all pass → `properly skilled`. At most two misses, neither item 1 nor item 6 → `close — create: <missing items>`. Otherwise → `needs skillify — run skillify on <target>`.
+
+## Phase 0 — Should this be a skill?
+
+- Will it be invoked 2+ times?
+- Is there more than ~20 lines of logic or judgment?
+- Is there a trigger phrase a user would actually say?
+
+Any no means a script or doc, not a skill: say why and stop.
+
+Then search the existing skills: if one nearly covers the intent, merge into it — a neighbor splitting a trigger family makes both trigger worse.
+
+**One skill = one capability = one coherent trigger family.** When the target spans intents users would invoke separately ("run the build", "roll back the deploy", "notify the team"), propose the split and ask which to skillify first.
+
+## Phase 1 — Audit
+
+For an existing skill, score the checklist against the tree and stop with the verdict:
+
+```
+Skill: <name>            Path: skills/<name>/ | skills/<group>/<name>/
+Score: <passed>/8        Verdict: <verdict>
+Missing: <item>: <one-line evidence> ...
+```
+
+A new skill is 0/8 — proceed. When the request was only an audit, deliver the verdict and the shortest path to `properly skilled`; editing waits for the ask.
+
+## Phase 2 — Elicit requirements
+
+Interview, skipping questions the conversation already answers:
+
+- **Triggers:** what exact phrases should invoke this? What near-miss requests should NOT? Which existing skill is the nearest neighbor?
+- **Output:** what does done look like: files, report shape, side effects?
+- **Edge cases:** empty input, missing config, offline, partial state.
+- **Existing behavior:** code or prose to absorb? What did its author correct over time?
+
+Mark every volatile fact the skill will state — version pins, model names, numeric limits, dated claims; those go to refresh-tracked files (item 7), or the skill takes the evergreen waiver.
+
+## Phase 3 — Scaffold and write
+
+Scaffold (dry run by default; `--write` creates; `--json` for machines):
+
+```sh
+node <skill-dir>/scripts/scaffold-skill.mjs <name> --dir <repo-root> [--write]
+```
+
+The scaffolder validates the name, refuses existing directories and symlinks, stages then renames, and performs the repo wiring itself; its dry-run output lists every refusal and wiring action, so read that rather than this file. Then write, in order:
+
+1. *Description + trigger query set* — the description per [authoring](references/authoring.md); the query set with near-miss negatives in `tests/fixtures/trigger-queries.json`.
+2. *SKILL.md body* — workflow, one excellent example, the nearest-neighbour boundary line, routing table; limits per skill-maintainer's standards, style per [authoring](references/authoring.md).
+3. *References* — detail only some invocations need.
+4. *Scripts* — deterministic, repeated work only (criteria in [authoring](references/authoring.md)).
+
+## Phase 4 — Behavioral eval — the quality gate
+
+Cases live in `evals/evals.json` (agentskills.io format: 2–3 realistic prompts, one boundary case, assertions added after the first run). Run them per the [eval playbook](references/eval-playbook.md): on Claude Code, `claude plugin eval <skill-dir> --ablation with-without --runs 3 --max-cost-usd <n> --threshold <t> --json`, results under `<skill>/evals/results/<timestamp>/` (gitignored); on every harness, the with-skill vs baseline subagent procedure on the same cases, writing the same result files. Workflow skills add the sandbox drill as their end-to-end proof.
+
+At most 3 cycles: eval → apply the top improvements → re-eval; pass, or ship with a KNOWN_GAPS section. If the baseline matches the with-skill output, the skill is not earning its tokens — cut or narrow it.
+
+## Phase 5 — Lock in
+
+1. Unit tests and fixtures for the deterministic branches of any bundled scripts; a prose-only skill keeps just its trigger-query fixture.
+2. `refresh/sources.json` + `refresh/REFRESH.md` for the volatile facts marked in Phase 2, or the evergreen waiver.
+3. Finish `README.md`, `agents/openai.yaml`, and the scaffolder's wiring TODOs (README row, changeset per dev-implement's changelog rule).
+
+## Phase 6 — Verify
+
+```sh
+node packages/cli/scripts/validate-skill.mjs <skill-dir>
+bun test <skill-dir>
+node packages/cli/scripts/structure.mjs check
+bun run check
+```
+
+Re-score the checklist and report `<passed>/8` with the verdict; anything below `properly skilled` ships only with named gaps.
+
+## Routing
+
+| Need | Read |
+|---|---|
+| description engineering, writing style, token economy, scripts vs instructions, volatile facts | [authoring](references/authoring.md) |
+| a walkthrough of one skillification | [authoring — worked example](references/authoring.md) |
+| failure patterns to check a draft against | [authoring — anti-patterns](references/authoring.md) |
+| eval method, pass criteria, trigger-query doctrine, workflow drills, cycles, KNOWN_GAPS, model guidance | [eval playbook](references/eval-playbook.md) |
+| name, description, body limits; listing budgets; portability; release and rename mechanics | the `skill-maintainer` skill |
+| the scaffolded starting points | `assets/templates/` |
+| skillify's freshness stance | [refresh/REFRESH.md](refresh/REFRESH.md) |

package/skill/skillify/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "VegaStack Skillify"
+  short_description: "Create and audit skills for the vegafactory repo"
+  default_prompt: "Use $skillify to audit or scaffold a skill in skills/ against the full repo contract."

package/skill/skillify/assets/templates/README.md.template

@@ -0,0 +1,26 @@
+# {{name}}
+
+TODO: one-paragraph summary for humans and agents browsing the repo. The agent entry point is [SKILL.md](SKILL.md); everything else loads progressively from there.
+
+## Install
+
+```sh
+npx @vegastack/vegafactory skills add {{name}} --global
+```
+{{groupInstallBlock}}
+`--global` installs into your home directory, where the skill is available in every project; drop it for a project-local install.
+
+## What's in this skill
+
+| Path | Purpose |
+|---|---|
+| [SKILL.md](SKILL.md) | Agent entry point |
+| [agents/openai.yaml](agents/openai.yaml) | Codex interface metadata |
+| [refresh/REFRESH.md](refresh/REFRESH.md) | Freshness contract |
+| [refresh/sources.json](refresh/sources.json) | Source registry for volatile claims |
+| `tests/` | Bun tests and fixtures (never packaged) |
+| `evals/` | Behavioral evals in the agentskills.io format (never packaged) |
+
+## Behavior
+
+TODO: what the skill does when invoked, its output contract, and its guardrails.

package/skill/skillify/assets/templates/REFRESH.md.template

@@ -0,0 +1,16 @@
+# Refresh contract — {{name}}
+
+Instructions for the scheduled refresh agent (and any human running a manual refresh). This file plus `sources.json` is the complete freshness contract for this skill.
+
+## What this skill claims
+
+TODO: apply the volatile-facts rule (skillify references/authoring.md) before shipping:
+
+- **Durable rules**: versionless principles in this skill's prose. The refresh agent NEVER edits these; if a source change invalidates one, open an issue quoting the evidence.
+- **Volatile facts**: version pins, model names, numeric vendor limits, URLs, dated claims. Each lives in a refresh-tracked file named here, evidenced by a `sources.json` entry. A sentence leaning on one vendor mechanism may be marked `<!-- source: SOURCE-ID -->`.
+
+TODO: name the refresh-tracked files and the sections the refresh agent may edit.
+
+## Evergreen waiver
+
+TODO: if this skill states no volatile facts, replace everything above with one line — "Evergreen: this skill asserts no version pins, vendor mechanism names, numeric vendor limits, or dated facts. Revisit if a future edit introduces any." — and keep `sources: []`.

package/skill/skillify/assets/templates/SKILL.md.template

@@ -0,0 +1,27 @@
+---
+name: {{name}}
+description: TODO - state only the conditions that should load this skill. Third person, trigger phrases in quotes first, symptoms and situations next, one clause naming what this skill is NOT for. Never summarize the workflow. Max 1024 characters, no angle brackets.
+---
+
+# {{name}}
+
+TODO: one-paragraph overview - what this skill guarantees and the single trigger family it serves.
+
+## When to use
+
+- TODO: concrete situations and user phrasings
+- TODO: when NOT to use - the nearest adjacent intent and where it goes instead
+- Nearest neighbor: TODO - name the closest existing skill and the one-sentence axis of difference (or state that none exists)
+
+## Workflow
+
+1. TODO
+2. TODO
+
+## Example
+
+TODO: one excellent, realistic example (input and output). One great example beats five mediocre ones.
+
+## Routing
+
+TODO: if this skill has references/, add a routing table (Need | Read) here; otherwise delete this section and record item 4 as N-A in the checklist with a rationale.

package/skill/skillify/assets/templates/evals.json.template

@@ -0,0 +1,12 @@
+{
+  "skill_name": "{{name}}",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "TODO: a realistic prompt a user would type — file paths, backstory, the hardest documented case",
+      "expected_output": "TODO: what success looks like, one human-readable sentence",
+      "files": [],
+      "assertions": []
+    }
+  ]
+}

package/skill/skillify/assets/templates/openai.yaml.template

@@ -0,0 +1,4 @@
+interface:
+  display_name: "{{name}}"
+  short_description: "TODO: imperative, under 60 characters"
+  default_prompt: "Use ${{name}} to TODO."

package/skill/skillify/assets/templates/skill.test.ts.template

@@ -0,0 +1,27 @@
+import { describe, expect, test } from 'bun:test'
+import { readFileSync } from 'node:fs'
+import { join, resolve } from 'node:path'
+import { validateSkill } from '{{validatorPath}}'
+
+const skillRoot = resolve(import.meta.dir, '..')
+
+describe('{{name}} contract', () => {
+  test('SKILL.md passes repo validation', () => {
+    const result = validateSkill(skillRoot)
+    expect(result.message).toBe('Skill is valid!')
+    expect(result.ok).toBe(true)
+  })
+
+  test('trigger query fixture is a small hard set with near-miss negatives', () => {
+    const queries = JSON.parse(readFileSync(join(skillRoot, 'tests/fixtures/trigger-queries.json'), 'utf8'))
+    const positives = queries.filter((entry: { should_trigger: boolean }) => entry.should_trigger)
+    const negatives = queries.filter((entry: { should_trigger: boolean }) => !entry.should_trigger)
+    expect(positives.length).toBeGreaterThanOrEqual(5)
+    expect(negatives.length).toBeGreaterThanOrEqual(4)
+    for (const entry of queries) expect(typeof entry.query).toBe('string')
+  })
+
+  // TODO: if this skill ships scripts/, add unit tests for every deterministic
+  // branch. A prose-only skill needs nothing more here - its quality bar is the
+  // behavioral eval of skillify Phase 4, which runs BEFORE tests lock anything in.
+})

package/skill/skillify/assets/templates/sources.json.template

@@ -0,0 +1,6 @@
+{
+  "schemaVersion": 1,
+  "retrievalBaseline": "{{date}}",
+  "note": "TODO: add one entry per volatile external fact this skill states (see skillify references/authoring.md, volatile facts), or record the one-line evergreen waiver in REFRESH.md and keep sources empty.",
+  "sources": []
+}

package/skill/skillify/assets/templates/trigger-queries.json.template

@@ -0,0 +1 @@
+[]

package/skill/skillify/references/authoring.md

@@ -0,0 +1,103 @@
+# Authoring reference
+
+Rules for writing the parts of a skill that determine whether it triggers, how much it costs, and whether it stays true over time.
+
+## Description engineering
+
+The description is the primary triggering mechanism: it is always in context, and the agent decides from it alone whether to load the body.
+
+- **Triggering conditions only — never the workflow.** A description that summarizes the process becomes a shortcut: agents follow the summary and skip the body. State when to load the skill, not what the skill will do step by step.
+- **Calm conditionals.** Write the trigger as "Use when …" with the phrasings users type and one "Not for …" clause — the standard and its reason are skill-maintainer's operating rule 4; this file only shows the shape.
+- **Third person.** The text is injected into a system prompt ("Creates and audits...", "Use when..."), never "I can help you...".
+- **Front-load trigger words.** Harness skill lists truncate long descriptions; the first clause must carry the strongest triggers.
+- **Cover the ways users actually ask.** Exact phrases in quotes, symptoms, file types, adjacent phrasings, casual variants. Include the situations where the skill competes with a neighbor and should win.
+- **Name the boundary.** One clause on what the skill is NOT for prevents over-triggering ("Only for skills inside this repository...").
+- **No angle brackets** — the repo validator rejects them.
+- **No `#` after a space.** The description is an unquoted YAML scalar, and real harness parsers treat ` #` as a comment start — everything after it silently vanishes from the skill listing. Write "issue 12", never "issue #12"; the repo validator rejects it.
+
+Bad: `Helps with skills.` (no triggers, no boundary)
+Bad: `Audits a skill by scoring checklist items, then scaffolds, evals, and wires it.` (workflow summary — the body will be skipped)
+Good: `Use when asked to "skillify this", "make this a skill", audit a skill's completeness, or decide whether a workflow should become a skill. Only for skills inside this repository.`
+
+## Writing style
+
+Rules for the prose itself — they exist because agents pay attention (and tokens) for every sentence, and because rule corpora rot in predictable ways:
+
+- **Prompt the positive.** Steering by prohibition drags the forbidden behavior into context and invites a wall of NEVERs that only ever grows. State the rule you want followed, once, with the reason it matters. Reserve negation for genuinely dangerous acts.
+- **Explain why, not ALL-CAPS what.** Modern models have good theory of mind; a rule with its reason generalizes, a bare MUST gets pattern-matched and misapplied. Reaching for caps or bold on every rule is a sign the rule needs a reason, not emphasis.
+- **Hunt no-ops and sediment.** A no-op is an instruction the model already obeys by default — it pays load to say nothing. Sediment is case law: a clause added for one past failure that never gets removed. When a failure exposes a gap, rewrite the existing rule in place; do not append an exception. Prune by deleting whole sentences, not trimming words.
+- **Budget the body.** Target 50–150 lines for a SKILL.md; treat 300 as the ceiling (the spec's 500 is a far bound, not a budget). One great example beats five mediocre ones. Anything only some invocations need goes behind a routed reference.
+- **Name the nearest neighbor.** One sentence in the body stating which existing skill is closest and the axis of difference. It keeps trigger families sharp better than any linter, and forces the merge conversation before a near-duplicate ships.
+
+## Numeric limits
+
+The numbers — name and description limits, body ceiling, and the Claude Code, Codex, and Hermes listing budgets — live in one place: skill-maintainer's Hard limits table, mirrored from the marked sentences in its references/standards.md and refreshed through its registry. Cite them per skill-maintainer's standards; a second copy here drifted once and would again.
+
+## Trigger query sets
+
+Every skill ships `tests/fixtures/trigger-queries.json`: an array of `{"query": "...", "should_trigger": true|false}`, with an optional `"ambiguous_with": ["skill-name"]` on entries where a neighboring skill competes for the query.
+
+- About 10 queries total: 5–6 should-trigger, 4–5 should-NOT-trigger. A small set of hard queries beats a long set of easy ones.
+- Should-trigger: different phrasings of the same intent — formal, casual, typo-ridden; cases that never name the skill but clearly need it; at least one `ambiguous_with` case this skill should win against its nearest neighbor.
+- Should-NOT-trigger: **near-misses only.** Queries sharing keywords or domain with the skill but needing something else — adjacent intents, ambiguous phrasing a naive keyword match would catch. "Write a fibonacci function" as a negative for a PDF skill tests nothing.
+- Queries must be realistic: concrete detail, file paths, a little backstory — what a user would actually type, not abstract category labels.
+
+The set is both a design artifact (it forces the description to draw a real boundary) and eval input (see the [eval playbook](eval-playbook.md)).
+
+## Token economy
+
+Three loading levels; spend accordingly:
+
+1. **Metadata (name + description)** — in every conversation, always. ~100 tokens. Every word must earn its place.
+2. **SKILL.md body** — loaded on every trigger. Keep the workflow, one excellent example, and a routing table; push everything else down a level. One great example beats five mediocre ones.
+3. **references/ / scripts/ / assets/** — loaded or executed on demand; effectively unlimited, but give each reference a clear routing condition so agents read only what the task needs.
+
+Cross-reference other skills by name (`skill-maintainer`) instead of restating their content. Don't document a script's flags in prose beyond what routing needs — the script's own `--help`/usage error is the source of truth.
+
+## Scripts vs instructions
+
+**Deterministic and repeated → script. Judgment → instructions.**
+
+- If two invocations with the same input must produce the same bytes, that is a script.
+- Watch eval/test transcripts for agents independently rewriting the same helper — three subagents each writing their own `parse_frontmatter` is a strong signal the skill should bundle that script once.
+- Repo conventions for bundled scripts: dependency-free Node (`.mjs`), runnable from the skill dir with plain relative paths, `--json` for machine-readable output, documented exit codes (0 ok / 1 finding-or-refusal / 2 usage), atomic writes (stage then rename), symlink refusal, and an explicit `--write` gate for anything mutating — dry-run is the default.
+- Do NOT script judgment: descriptions, verdicts, review prose. A script that fakes judgment produces confident garbage.
+
+## Volatile facts and refresh
+
+One rule: **version pins, model names, numeric limits, URLs, and anything with a date live in refresh-tracked files — never inline in SKILL.md.** Everything else is plain prose that refresh automation never touches.
+
+A volatile fact gets a `refresh/sources.json` entry (URL, checksum, threshold) and lives in a section the REFRESH.md names as editable; marking an individual sentence `<!-- source: SOURCE-ID -->` remains available when a durable rule leans on one vendor-named mechanism. A skill with no volatile facts of its own declares an **evergreen waiver** in REFRESH.md — one line stating why nothing decays — and keeps `sources: []`. Facts whose source of truth already lives in another skill's registry are mirrored with a `<!-- mirrored -->` marker, not duplicated as a second registry entry.
+
+## Worked example: skillifying a "release-notes" workflow
+
+```
+Phase 0: yes — run at every release, ~80 lines of conventions, trigger "draft the release notes";
+  nearest neighbor is skill-maintainer (release wiring) — different axis, proceed
+Phase 1: 0/8 (new)
+Phase 2: triggers "draft/write the release notes", NOT "write a changelog entry for this PR";
+  output = CHANGELOG section + npm summary; volatile: none owned → evergreen waiver
+Phase 3: scaffold-skill.mjs release-notes --dir . --write (wiring done by the scaffolder);
+  description + 10 trigger queries incl. ambiguous_with skill-maintainer; body + references/format.md;
+  no scripts (judgment-heavy, item 5 N/A)
+Phase 4: eval cycle 1 — baseline subagent invents section headings, with-skill misses breaking-change
+  callouts → add callout contract to SKILL.md; cycle 2 — with-skill clearly better, baseline still
+  wrong shape → pass
+Phase 5: trigger fixture locked; evergreen waiver written; README row description filled in
+Phase 6: bun run check green; 8/8 → properly skilled
+```
+
+## Anti-patterns
+
+- Writing lock-in tests before the behavioral eval — locks in mediocrity.
+- A description that summarizes the workflow — agents follow the description and skip the body.
+- Skipping the baseline run because "the output looks fine" — that is not evidence.
+- Eval without a fix cycle — vanity metrics.
+- Obviously-irrelevant negatives in the trigger query set — near-misses or nothing.
+- Version pins or model names in SKILL.md — volatile facts live in refresh-tracked locations.
+- Multi-intent skills spanning unrelated triggers — split them.
+- Two skills answering the same trigger — merge or kill one.
+- Steering by prohibition — a wall of NEVERs drags the forbidden behavior into context and grows forever; state the positive rule once and explain why it matters.
+- Patching every observed agent failure with a new clause — that is sediment; rewrite the existing rule in place instead of appending case law.
+- Unit tests for prose — a test asserting a markdown file contains a phrase proves nothing the eval didn't.
+- Building eval tooling — the runner and the subagent procedure exist; write cases, not harnesses.

package/skill/skillify/references/eval-playbook.md

@@ -0,0 +1,112 @@
+# Behavioral eval playbook
+
+How to prove a skill improves agent behavior before tests cement it. The tooling is your own subagents (Claude Code's Task tool, Codex's `.codex/agents` spawns, Hermes `delegate_task`) and, once its early-access gate lifts and it reads this case format, `claude plugin eval` (status in Model guidance); write cases, never harnesses, viewers or graders of your own — one format is what lets results compare across skills and releases.
+
+## Why eval before tests
+
+Tests lock in behavior; if the behavior is mediocre, tests lock in mediocrity. The eval proves the quality bar first, then unit tests cement it. If you didn't watch an agent fail *without* the skill, you don't know the skill teaches anything.
+
+## The case file — evals/evals.json
+
+Every skill carries its cases at `<skill>/evals/evals.json` in the agentskills.io format — the format this playbook's procedure reads and the one `claude plugin eval` is expected to read once verified; never packaged (like `tests/`), and `structure.mjs check` warns when it is missing or malformed.
+
+```json
+{
+  "skill_name": "<the frontmatter name>",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "what the user would type — paths, backstory, the hardest documented case",
+      "expected_output": "one human-readable description of success",
+      "files": ["evals/files/input.md"],
+      "assertions": ["the plan comment has an Interfaces block per task"]
+    }
+  ]
+}
+```
+
+- `id` is an integer, unique in the file; `files` are paths relative to the skill root under `evals/files/`; `assertions` is an array, empty until the first run.
+- 2–3 cases per skill: realistic (file paths, backstory), varied in formality, one boundary case the skill should route elsewhere or refuse.
+- `expected_output` is a sentence a human reads; assertions are objective and countable ("every finding carries `path:line`"), never exact phrasing.
+- Write prompts first, add assertions after the first run — the outputs show what "good" looks like. Retire an assertion that passes in both arms; it measures the model, not the skill.
+- `tests/fixtures/trigger-queries.json` stays the description-level fixture (would the skill load?); `evals.json` is the body-level one (did it help once loaded?).
+
+## Running on Claude Code — claude plugin eval
+
+```sh
+claude plugin eval <skill-dir> --ablation with-without --runs 3 --max-cost-usd 5 --threshold 0.7 --json --no-publish
+```
+
+`--ablation with-without` runs the baseline arm; `--runs 3` makes a stddev meaningful; `--max-cost-usd` aborts with exit 2 rather than overrunning; `--threshold 0.7` because a with-skill arm failing a third of its assertions is not earning its tokens, while `1.0` turns every grader wobble into a red run; `--no-publish` keeps the report local. Results land at `<skill>/evals/results/<timestamp>/aggregate-result.json`, gitignored; the aggregate JSON of the gating run is pasted into the evidence comment.
+
+## Running anywhere — the subagent procedure
+
+The same cases, run by hand on any harness:
+
+1. **Launch both arms per case in the same turn.** One subagent told to read and follow the SKILL.md at its path; one baseline given the identical prompt with no mention of the skill (when improving a skill, the baseline is a snapshot of the old version). Each dispatch carries the skill path or none, the task, the case's `files`, and an output directory.
+2. **Write to `.vegastack/.tmp/<issue>-<slug>/<skill>-workspace/iteration-<n>/eval-<id>/{with_skill,without_skill}/outputs/`**, never beside the skill directory: discovery treats every directory in a group as a skill, so a sibling workspace breaks `bun run check` even when gitignored.
+3. **Compare against the skill's claimed value**, reading transcripts, not just final outputs, into the result files below.
+
+## Result files
+
+The agentskills.io vocabulary, so a hand run and a runner run read alike:
+
+- `timing.json` per arm — `{ "total_tokens", "duration_ms" }` from the task-completion notification, saved at once (it is gone with the next message).
+- `grading.json` per arm — `{ "assertion_results": [{ "text", "passed", "evidence" }], "summary": { "passed", "failed", "total", "pass_rate" } }`; a PASS needs quoted evidence from the output.
+- `benchmark.json` at the iteration root — `{ "run_summary": { "with_skill": { "pass_rate": { "mean", "stddev" }, "time_seconds": {…}, "tokens": {…} }, "without_skill": {…}, "delta": {…} } }`; `aggregate-result.json` is the runner's equivalent.
+
+## Pass criteria
+
+The with-skill run must be *materially* better on the dimensions the skill exists for:
+
+- Output has the contract shape the skill defines (sections, verdicts, file layout) where the baseline's does not.
+- The with-skill agent uses bundled scripts/templates instead of reinventing them; a helper both arms wrote independently should become a bundled script.
+- No contract violations the checklist would catch (frontmatter keys, name grammar, missing pieces).
+- The transcript shows the skill's guidance being *used*, not just loaded — sections nobody used are bloat to cut.
+- **Colleague test.** Hand the SKILL.md alone, with no conversation context, to a fresh subagent and ask what it would do first and where it would look for the answer to one realistic request; any confusion is a defect in the skill, not in the reader, because the model that runs it never attended the design conversation.
+
+**Failure that matters most:** the baseline matches the with-skill output. Then the skill is not earning its tokens — cut it, narrow it, or fold it into a reference.
+
+## Trigger-query eval (description level)
+
+The body eval assumes the skill got loaded. Separately, walk `tests/fixtures/trigger-queries.json` and judge, per query, whether an agent seeing only the skill list (name + description) would load this skill. Near-miss negatives are the valuable half — a description that survives them draws a real boundary. Fix failures by editing the *description* (triggers, boundary clause), never by stuffing workflow summary into it.
+
+**Family-level re-run:** whenever a skill is added, renamed, or removed, re-run the trigger eval across the FULL installed set: cross-skill collisions (two descriptions claiming one query) only appear at family level, and the `ambiguous_with` entries in every skill's fixtures are the cases to walk first.
+
+## Workflow skills — the sandbox drill
+
+A skill whose value lives in a multi-turn, external-state workflow (gh labels, comments, branches) — dev-implement, dev-review, dev-ship — gets only partial proof from single-prompt runs: its cases cover the prose half (evidence comment, review comment, ship report). Its end-to-end proof is a **sandbox drill** — a throwaway repo walked through the real flow with the real tools — run before release-level claims, not per edit. Single-prompt evals remain the per-change gate; the drill is the workflow-level one.
+
+## Cycle protocol (at most 3)
+
+```
+CYCLE 1: run evals → list concrete improvements → apply them to the skill files
+CYCLE 2: re-run the SAME prompts → compare before/after → apply what remains
+CYCLE 3: re-run → pass, or ship with KNOWN_GAPS
+```
+
+Hard stop after 3 cycles: endless polishing is worse than an honest gap list. Fix the pattern that caused the failure, never one prompt's wording. When a rule is not landing, prefer removing it to rewriting it and let the eval prove the need: a rule the with-skill run never used is bloat, and rules written for earlier models are often too prescriptive for the current one.
+
+## KNOWN_GAPS format
+
+Ships at the bottom of the skill's README.md when the eval did not fully pass:
+
+```markdown
+## KNOWN_GAPS
+
+- gap: baseline-quality section ordering under multi-file inputs
+  evidence: eval cycle 3, prompt "draft release notes for the 0.4 tag"
+  why unresolved: would require restructuring the routing table
+  next: revisit when references/format.md is split by artifact type
+```
+
+One entry per gap: `gap` / `evidence` (cycle + prompt) / `why unresolved` / `next`. The eval summary (prompts, cycles, verdict per prompt) goes in the PR body, not the repo.
+
+## Model guidance
+
+<!-- volatile: everything in this section decays; keep model facts out of SKILL.md and update here only. -->
+
+- Runner status, 03-09-2026, Claude Code 2.1.247: `claude plugin eval` prints "plugin eval is currently in early access" and exits 1 for `init --bare` and for a run; `--help` shows it reads `evals/**/case.yaml` or `prompt.md + graders/*.md`. Until the gate lifts and the invocation above is verified against a bare `skills/<group>/<name>/` directory, the subagent procedure is the Claude-side proof too; re-probe with `claude plugin eval init --bare probe` in a scratch directory and update this line's date.
+- Run both arms of every comparison on the **same model** — otherwise you measure the model gap, not the skill.
+- Use the model that will actually run the skill in production (typically the one powering the current session); a weaker grader cannot judge a stronger writer.
+- If a separate grading pass is used, grade blind: give the grader both outputs unlabeled and ask which better satisfies the skill's stated contract.

package/skill/skillify/refresh/REFRESH.md

@@ -0,0 +1,23 @@
+# Refresh contract — skillify
+
+Instructions for the scheduled refresh agent (and any human running a manual refresh). This file plus `sources.json` is the complete freshness contract for this skill.
+
+## What this skill claims
+
+- **Durable rules** (SKILL.md, most of `references/`): the phase model, checklist structure, eval-before-tests ordering, description doctrine, the volatile-facts rule. Versionless; the refresh agent NEVER edits these. If external evidence invalidates one, open an issue quoting the evidence — do not edit.
+- **Mirrored volatile claims**: skillify does not track its own external sources. Its one volatile spot is marked in place:
+  - the model guidance section in `references/eval-playbook.md` (marked `<!-- volatile -->`).
+
+  It mirrors facts whose sources of truth are the agent-skills standards sources tracked in the **skill-maintainer** registry (`skills/repo-tooling/skill-maintainer/refresh/sources.json`). `sources` here is therefore deliberately empty, and the refresh runner treats this registry as valid with nothing to fetch.
+
+## How to refresh
+
+1. There is no deterministic fetch pass for this skill — an automated run against this registry selects zero sources and exits 0. Do not add sources here that duplicate skill-maintainer's; one registry per fact.
+2. When skill-maintainer's standards sources drift (spec limits, harness listing budgets, model catalogs), that skill's refresh PR is the evidence. Update skillify's marked section to match **on a normal branch** (refresh branches are CI-restricted to `refresh/` metadata), referencing the skill-maintainer refresh PR as evidence.
+3. If a future edit gives skillify a directly-owned external claim (its own vendor URL, pin, or protocol fact), stop mirroring for that claim: add a real entry to `sources.json` and mark the sentence `<!-- source: SOURCE-ID -->`.
+
+## Never
+
+- Never edit the phase model, checklist items, or verdict grammar in a refresh.
+- Never hand-write checksum/version/timestamp values anywhere in this repo.
+- Never duplicate a skill-maintainer source entry into this registry.

package/skill/skillify/refresh/sources.json

@@ -0,0 +1,6 @@
+{
+  "schemaVersion": 1,
+  "retrievalBaseline": "2026-08-08T00:00:00Z",
+  "note": "Deliberately empty. Skillify's only time-decaying claim (the model guidance in references/eval-playbook.md) is mirrored from the standards sources tracked in the skill-maintainer registry and is refreshed through that skill's registry, not duplicated here; numeric limits are cited from skill-maintainer, never copied. See REFRESH.md.",
+  "sources": []
+}