@zalom/plastic 1.0.0-alpha.9 → 1.0.0-beta.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (99) hide show
  1. package/PLASTIC.md +163 -469
  2. package/README.md +95 -58
  3. package/agents/plastic-brainstorming.md +37 -0
  4. package/agents/plastic-enforcer.md +36 -0
  5. package/agents/plastic-executor.md +37 -0
  6. package/agents/{future-intent-researcher.md → plastic-future-intent-researcher.md} +1 -1
  7. package/agents/{intent-curator.md → plastic-intent-curator.md} +1 -1
  8. package/agents/plastic-planner.md +37 -0
  9. package/agents/plastic-spec-specialist.md +37 -0
  10. package/bin/plastic.js +57 -0
  11. package/bin/test +28 -0
  12. package/deprecations.yml +1 -10
  13. package/hooks/auto-arm +5 -0
  14. package/hooks/bash-gate +3 -0
  15. package/hooks/check-update +12 -8
  16. package/hooks/code-gate +12 -0
  17. package/hooks/create-gate +3 -0
  18. package/hooks/gate-check +3 -1
  19. package/hooks/hooks.json +52 -0
  20. package/hooks/qmd-search +8 -0
  21. package/package.json +2 -2
  22. package/scripts/dashboard.rb +687 -0
  23. package/scripts/doctor.rb +1054 -628
  24. package/scripts/hook-auto-arm +51 -0
  25. package/scripts/hook-bash-gate +41 -0
  26. package/scripts/hook-code-gate +27 -0
  27. package/scripts/hook-continue +15 -114
  28. package/scripts/hook-create-gate +59 -0
  29. package/scripts/hook-gate-check +47 -32
  30. package/scripts/hook-qmd-search +44 -0
  31. package/scripts/hook-session-start +106 -38
  32. package/scripts/install.rb +91 -529
  33. package/scripts/lib/boot_banner.rb +28 -0
  34. package/scripts/lib/bridge.rb +404 -19
  35. package/scripts/lib/installer_core.rb +807 -0
  36. package/scripts/lib/intent_validator.rb +235 -0
  37. package/scripts/lib/qmd_hook.rb +44 -0
  38. package/scripts/lib/qmd_sync.rb +209 -0
  39. package/scripts/lib/store_provisioning.rb +100 -0
  40. package/scripts/migrate-to-global +1 -1
  41. package/scripts/new-intent +226 -0
  42. package/scripts/provision-project-store +53 -0
  43. package/scripts/qmd-sync +92 -0
  44. package/scripts/select-update-target +93 -0
  45. package/scripts/spawn-preamble +121 -0
  46. package/scripts/uninstall.rb +53 -0
  47. package/scripts/update.rb +164 -0
  48. package/scripts/validate-intent +54 -0
  49. package/scripts/versions.rb +141 -0
  50. package/skills/_active-intent-gate.md +1 -1
  51. package/skills/add-project-store/SKILL.md +54 -0
  52. package/skills/auto/SKILL.md +77 -7
  53. package/skills/auto/evals/evals.json +207 -0
  54. package/skills/auto/references/agent-architecture.md +135 -0
  55. package/skills/brainstorming/SKILL.md +9 -9
  56. package/skills/brainstorming-grill-me/SKILL.md +6 -6
  57. package/skills/continuing/SKILL.md +92 -82
  58. package/skills/continuing/evals/evals.json +136 -0
  59. package/skills/continuing/references/context-management.md +32 -0
  60. package/skills/creating-intent/SKILL.md +74 -36
  61. package/skills/creating-intent/evals/evals.json +56 -0
  62. package/skills/creating-intent/references/lifecycle.md +76 -0
  63. package/skills/creating-intent/references/wikilinks.md +8 -0
  64. package/skills/creating-project/SKILL.md +40 -8
  65. package/skills/creating-project/references/hubs-projects.md +55 -0
  66. package/skills/dashboard/SKILL.md +121 -0
  67. package/skills/dashboard/templates/dashboard-global.md +31 -0
  68. package/skills/dashboard/templates/dashboard-project.md +40 -0
  69. package/skills/doctor/SKILL.md +51 -4
  70. package/skills/doctor/references/gates-stuck-detection.md +38 -0
  71. package/skills/doctor/report.md +4 -0
  72. package/skills/evaluating-skills/SKILL.md +140 -0
  73. package/skills/evaluating-skills/assets/eval-template.json +12 -0
  74. package/skills/evaluating-skills/evals/evals.json +75 -0
  75. package/skills/evaluating-skills/references/convention-checks.md +76 -0
  76. package/skills/evaluating-skills/references/eval-methodology.md +154 -0
  77. package/skills/executing-plan/SKILL.md +5 -3
  78. package/skills/install/SKILL.md +69 -8
  79. package/skills/intent-curator/SKILL.md +3 -3
  80. package/skills/linking-intents/SKILL.md +16 -7
  81. package/skills/linking-intents/references/zettelkasten.md +38 -0
  82. package/skills/managing-index/SKILL.md +5 -1
  83. package/skills/managing-index/references/zettelkasten-linking.md +1 -1
  84. package/skills/releasing/SKILL.md +80 -23
  85. package/skills/releasing/references/deprecations.md +60 -0
  86. package/skills/research/SKILL.md +2 -2
  87. package/skills/savepoint/SKILL.md +46 -37
  88. package/skills/savepoint/references/context-management.md +32 -0
  89. package/skills/uninstall/SKILL.md +39 -28
  90. package/skills/update/SKILL.md +41 -44
  91. package/skills/versions/SKILL.md +65 -0
  92. package/skills/writing-instructions/SKILL.md +159 -0
  93. package/skills/writing-instructions/references/agentskills-spec.md +135 -0
  94. package/skills/writing-plans/SKILL.md +5 -5
  95. package/templates/agents.md +7 -7
  96. package/templates/outcome.md +13 -0
  97. package/templates/savepoint.md +14 -13
  98. package/templates/spec.md +25 -0
  99. package/bin/install.js +0 -29
@@ -0,0 +1,121 @@
1
+ ---
2
+ name: plastic-dashboard
3
+ description: Use when the user wants an overview of intents, asks "where are we", "what's next", "what should I work on", "show the dashboard", or invokes /plastic-dashboard. Renders a deterministic Value×Effort work cockpit as Markdown across the global store and all projects, and emits a machine-readable queue that auto mode consumes.
4
+ ---
5
+
6
+ # Dashboard — Plastic Work Cockpit
7
+
8
+ A deterministic overview of the intent store(s). It answers three questions at a glance:
9
+ **where we are** (recently worked), **where we go next** (a Value×Effort matrix), and
10
+ **how to conduct it** (a disposition per intent). The human-facing surface is **Markdown**,
11
+ because the user's UI renders Markdown natively but collapses raw tool-call stdout.
12
+
13
+ The script does all the data work. The agent fills a Markdown template from the script's
14
+ payload with near-zero reasoning and **presents the filled board in its reply**. Same store
15
+ state → byte-identical payload, regardless of model. Do NOT hand-summarize intents.
16
+
17
+ ## When to Use
18
+
19
+ - User invokes `/plastic-dashboard`
20
+ - User asks "where are we", "what's next", "what should I work on", "show me the intents"
21
+ - `plastic-continuing` lands on the board on resume
22
+ - `plastic-auto` reads `--json` to choose the next dispatchable intent
23
+
24
+ ## Procedure (the Markdown board — default human surface)
25
+
26
+ ### Step 1 — Get the data payload
27
+
28
+ ```bash
29
+ ruby ~/.plastic/scripts/dashboard.rb [continue|project <slug>] --data
30
+ ```
31
+
32
+ - `continue` (default) → the **global** board payload (`mode: "global"`).
33
+ - `project <slug>` → that **project** board payload (`mode: "project"`).
34
+
35
+ The payload is read-only JSON. Global-board fields: `date`, `store_health`, `recently_worked`,
36
+ `matrix` (`quick_win`/`next_big`/`defer`/`triage`/`research`, each a list of `{line, bullet, ...}`),
37
+ `counts`, `projects`, `project_totals`. Project-board fields: `slug`, `store_health`,
38
+ `description`, `recently_worked`, `matrix`, `counts`, `active`, `future`.
39
+
40
+ Each board load runs the scoped store check (`doctor --store <scope>`): the global board runs
41
+ `--store global` and a project board runs `--store <slug>`. The result rides in the payload as
42
+ `store_health` (`{scope, status, summary, failing_checks}`). Surface it as a one-line
43
+ store-health note on the board (for example `store health: pass (3/3)` or
44
+ `store health: warn (orphaned_intents)`). It is non-fatal: a warn or fail is shown as data and
45
+ never blocks the board.
46
+
47
+ ### Step 2 — Fill the matching template
48
+
49
+ Templates live in this skill's `templates/` directory:
50
+ `~/.claude/skills/plastic-dashboard/templates/dashboard-global.md` and
51
+ `dashboard-project.md`.
52
+
53
+ Fill mechanically — no rewriting, no re-sorting:
54
+ - `{{a.b.count}}` → the integer (e.g. `matrix.quick_win.count` = that list's length).
55
+ - `{{...lines}}` → join the list's `.line` strings with **real newlines** (one per line).
56
+ These lines are already glyph-led (the glyph is the bullet — never add `-`, never emit
57
+ `<br>`). If a matrix quadrant list is empty, render `_(none)_`.
58
+ - `projects.lines` → one line per project:
59
+ `- **{slug}** — {description} · active {active} · done {done} · future {future} · last accessed {last_accessed_at[0,10]}`.
60
+ - Scalars (`{{date}}`, `{{slug}}`, `{{description}}`) → substitute verbatim.
61
+
62
+ ### Step 3 — Present it (mandatory, every invocation)
63
+
64
+ **Paste the filled Markdown into your reply.** This is non-optional: the board only reaches
65
+ the user when it is in the chat reply, not in tool-call stdout. Never describe the board
66
+ instead of showing it.
67
+
68
+ ### Step 4 — Entry flow (the board is the menu)
69
+
70
+ The board lists everything; the user navigates by free prose (no capped picker):
71
+ - On the **global** board, the user replies with an **intent id** (work it), a **project
72
+ name** (re-run `project <slug> --data` and present that board), or **"new"** (start a new
73
+ intent in global via `plastic-creating-intent`).
74
+ - On a **project** board, the user replies with an **intent id**, or **"global"** to return.
75
+
76
+ ## Auto-mode contract (`--json`)
77
+
78
+ `plastic-auto` consumes a separate machine-readable manifest (unchanged):
79
+
80
+ ```bash
81
+ ruby ~/.plastic/scripts/dashboard.rb [continue|project <slug>|all] --json
82
+ ```
83
+
84
+ ```json
85
+ { "generated_for": "auto-mode", "scope": "<scope|all>",
86
+ "next_big_thing": "<id|null>",
87
+ "dispatchable_queue": [ {"id","scope","disposition","type","value","effort","flags","rank"} ],
88
+ "human_only": ["<id>", "..."] }
89
+ ```
90
+
91
+ Work `dispatchable_queue` in `rank` order (`defer`/`research`); leave `human_only` for the user.
92
+
93
+ ## Text modes (terminal / legacy)
94
+
95
+ `dashboard.rb [continue|project <slug>|all]` with no flag still prints the ASCII cockpit for
96
+ a raw terminal. The Markdown board (`--data` + template) is the surface for the chat UI.
97
+
98
+ ## How classification works (deterministic)
99
+
100
+ - **Effort** — small for `research`/`exploration`/`bugfix`, for already-scoped intents
101
+ (plan/checklist exists), or deep refinement branches; big otherwise.
102
+ - **Value → high** when any of: explicit `value: high`; a human-authored **root** intent; an
103
+ intent with a non-empty `chain`; or an intent that is a `source` of ≥1 other intent. Else low.
104
+ - **Flags** — `unblocked` only when a **future** intent has **all** its `sources` done;
105
+ `stale` only on future intents past the staleness threshold. Both kept low-noise by design.
106
+ - **Override** — a `value: high|low` frontmatter field always wins (pre-stamped data, never
107
+ model judgment at render time).
108
+
109
+ ## Eval
110
+
111
+ The eval is the payload + golden snapshots: run the engine against the fixture store and
112
+ assert the `--data` payload shape/sorting/classification and the byte-identical `--json` +
113
+ text goldens in `test/fixtures/dashboard/`. See `test/dashboard_test.rb`. Drift without an
114
+ intentional change means the skill is broken.
115
+
116
+ ## Notes
117
+
118
+ - Quadrant lists are **not** Markdown `-` bullets — the glyph is the bullet (the boards are
119
+ UI-only and may evolve, so they need not be valid Markdown lists). Never emit `<br>`.
120
+ - Clusters (Zettelkasten grouping in INDEX.md) are intentionally not rendered.
121
+ - Additive: changes no core lifecycle, gate, or cycle logic.
@@ -0,0 +1,31 @@
1
+ # 🧩 Plastic · Global Board — {{date}}
2
+
3
+ **Recently worked** · last 24h
4
+ {{recently_worked.lines}}
5
+
6
+ ## Where we go next · Value × Effort *(global intents only)*
7
+
8
+ | | Small effort | Big effort |
9
+ |---|---|---|
10
+ | **High value** | ⚡ **Quick win** · {{matrix.quick_win.count}} | ★ **Next big thing** · {{matrix.next_big.count}} |
11
+ | **Low value** | → **Defer (agent)** · {{matrix.defer.count}} | ⚑ **Triage** · {{matrix.triage.count}} |
12
+
13
+ **⚡ Quick win** — small effort, high value
14
+ {{matrix.quick_win.lines}}
15
+
16
+ **★ Next big thing** — big effort, high value
17
+ {{matrix.next_big.lines}}
18
+
19
+ **→ Defer (agent)** — small effort, low value
20
+ {{matrix.defer.lines}}
21
+
22
+ **⚑ Triage** — big effort, low value
23
+ {{matrix.triage.lines}}
24
+
25
+ 🔬 **Research → agent**
26
+ {{matrix.research.lines}}
27
+
28
+ ## Projects · active {{project_totals.active}} · done {{project_totals.done}} · future {{project_totals.future}}
29
+ {{projects.lines}}
30
+
31
+ **What would you like to work on next?** (type an **intent id**, a **project name**, or anything **new** you'd like to start)
@@ -0,0 +1,40 @@
1
+ # 📦 {{slug}} · Project Board — {{date}}
2
+
3
+ {{description}}
4
+
5
+ **Recently worked** · last 24h
6
+ {{recently_worked.lines}}
7
+
8
+ ## Where we go next · Value × Effort
9
+
10
+ | | Small effort | Big effort |
11
+ |---|---|---|
12
+ | **High value** | ⚡ **Quick win** · {{matrix.quick_win.count}} | ★ **Next big thing** · {{matrix.next_big.count}} |
13
+ | **Low value** | → **Defer (agent)** · {{matrix.defer.count}} | ⚑ **Triage** · {{matrix.triage.count}} |
14
+
15
+ **⚡ Quick win** — small effort, high value
16
+ {{matrix.quick_win.lines}}
17
+
18
+ **★ Next big thing** — big effort, high value
19
+ {{matrix.next_big.lines}}
20
+
21
+ **→ Defer (agent)** — small effort, low value
22
+ {{matrix.defer.lines}}
23
+
24
+ **⚑ Triage** — big effort, low value
25
+ {{matrix.triage.lines}}
26
+
27
+ 🔬 **Research → agent**
28
+ {{matrix.research.lines}}
29
+
30
+ ## Intents · active {{counts.active}} · done {{counts.done}} · future {{counts.future}}
31
+
32
+ **Active**
33
+ {{active.lines}}
34
+
35
+ **Future**
36
+ {{future.lines}}
37
+
38
+ **Legend** · ○ What ◔ Why ◑ How ◕ Exec ● Done · ⚡ quick win ★ big → defer ⚑ triage 🔬 research
39
+
40
+ **What would you like to work on next?** (type an **intent id**, or **global** to go back)
@@ -1,14 +1,55 @@
1
1
  ---
2
- name: doctor
2
+ name: plastic-doctor
3
3
  description: Use when diagnosing Plastic installation health, after updates, or when something seems broken. Runs checks and reports findings with fix options.
4
4
  ---
5
5
 
6
6
  # Doctor — Plastic Health Check
7
7
 
8
+ ## Scopes
9
+
10
+ Doctor has three scopes. Pick the right one for the situation:
11
+
12
+ | Scope | Flag | When it runs | States |
13
+ |-------|------|--------------|--------|
14
+ | Core check | `--core` | SessionStart hook (automatic), also available on demand | Binary: pass or error |
15
+ | Store check | `--store [global\|<slug>]` | Dashboard load, `plastic-continuing` | Three-state: pass / warn / fail |
16
+ | Full check | (no flag) | After every update (automatic), or `/plastic-doctor` | Three-state: pass / warn / fail |
17
+
18
+ ### `--core` (binary, manifest-backed)
19
+
20
+ Verifies that every core file is present and content-matches what the installed
21
+ version shipped. It checks two install manifests:
22
+
23
+ - `~/.plastic/manifest.json` (global manifest, covers PLASTIC.md and global scripts)
24
+ - `~/.claude/plastic/manifest.json` (agent-side manifest, covers agent scripts and hooks)
25
+
26
+ Each manifest maps a file path to its SHA256. The core check also confirms hooks
27
+ are registered, scripts are present and executable, and the installed version
28
+ matches. Result is binary: exit 0 on pass, non-zero on error. It never produces
29
+ warnings.
30
+
31
+ ### `--store [global|<slug>]`
32
+
33
+ Checks store state: intents are well-formed, INDEX sections are present, conventions
34
+ are followed, and links are valid. Scope options:
35
+
36
+ - No argument: checks all stores (global and all projects)
37
+ - `global`: checks only the global store
38
+ - A project slug (e.g. `--store plastic`): checks only that project's store
39
+
40
+ Produces three-state results (pass / warn / fail) and is run per-scope at dashboard
41
+ load time: the global board uses `--store global`, a project board uses `--store <slug>`.
42
+
43
+ ### Full doctor (no flag)
44
+
45
+ Runs core plus all store checks plus deprecation checks. This is what `/plastic-doctor`
46
+ invokes. It also runs automatically after every `plastic-update` (informational,
47
+ does not block or revert the update).
48
+
8
49
  ## When to Use
9
50
 
10
- - User invokes `/plastic:doctor`
11
- - After `plastic:update` completes (automatically)
51
+ - User invokes `/plastic-doctor` (full check)
52
+ - After `plastic-update` completes (automatically, full check)
12
53
  - When hooks aren't firing, skills aren't loading, or something seems broken
13
54
  - When the user says "check plastic", "diagnose", "what's wrong with plastic"
14
55
 
@@ -75,6 +116,8 @@ Use the `fix_hint` value to determine the correct action:
75
116
  | "Create INDEX.md with required sections" | Write INDEX.md with the 5 sections: Active, Future, Clusters, Abandoned, Completed |
76
117
  | "Add missing entries to INDEX.md" | Add orphaned intents to the appropriate INDEX.md section |
77
118
  | "Remove stale references from INDEX.md" | Edit INDEX.md to remove ghost references |
119
+ | "Inject the missing required frontmatter field(s)" | Edit the intent's `{ID}--{slug}.md` frontmatter to add the missing key (e.g. `chain: []`) without touching other keys |
120
+ | "Run: provision-project-store {slug}" | Run `provision-project-store <slug>` (or invoke the `plastic-add-project-store` skill) to create the missing store |
78
121
  | "Re-run installer" | Run `npx @zalom/plastic@latest --agent` |
79
122
 
80
123
  For fixes the agent cannot handle automatically, explain what the user needs
@@ -95,7 +138,7 @@ Show the updated results.
95
138
 
96
139
  ## Post-Update Mode
97
140
 
98
- When invoked from `plastic:update` (not directly by the user):
141
+ When invoked from `plastic-update` (not directly by the user):
99
142
 
100
143
  1. Run the diagnostic script as in Step 1.
101
144
  2. If all checks pass: show a single line — **"Health check: all clear."**
@@ -110,3 +153,7 @@ This keeps the update flow clean when nothing is wrong.
110
153
  - The script outputs JSON to stdout. Any diagnostic errors go to stderr.
111
154
  - Non-zero exit codes mean "issues found", not "script crashed".
112
155
  Always parse stdout regardless of exit code.
156
+
157
+ ## References
158
+
159
+ - Read `references/gates-stuck-detection.md` for the full gate enforcement table, bridge file pattern, and stuck detection thresholds when diagnosing gate failures or stuck agents
@@ -0,0 +1,38 @@
1
+ # Gate Enforcement and Stuck Detection
2
+
3
+ ## Bridge File Pattern
4
+
5
+ `/tmp/plastic-{session}.json` is the hot cache; the filesystem is the authority.
6
+ SessionStart rebuilds the bridge file from the filesystem — the bridge is always disposable.
7
+
8
+ ## Gate Taxonomy
9
+
10
+ | Gate type | Purpose |
11
+ |---|---|
12
+ | **Pre-flight** | Prerequisites exist? |
13
+ | **Revision** | New info invalidated prior work? |
14
+ | **Escalation** | Blocked items surface to user |
15
+ | **Abort** | Inconsistent state detected |
16
+
17
+ ## Full Gate Enforcement
18
+
19
+ | Gate | Blocked action | Required prerequisite |
20
+ |---|---|---|
21
+ | Pre-flight | Cannot write `plan.md` | `spec.md` must exist |
22
+ | Pre-flight | Cannot create `actions/` | `spec.md` must exist |
23
+ | Pre-flight | Cannot write `outcome.md` | `checklist.md` must exist with all items checked |
24
+ | Revision | Cannot proceed to Exec | Context changed after spec.md — re-derive spec |
25
+ | Abort | Cannot complete intent | `outcome.md` exists but checklist has unchecked items |
26
+
27
+ Hard blocking — hooks exit with code 2 when gates fail.
28
+
29
+ ## Stuck Detection
30
+
31
+ | Condition | Threshold | Action |
32
+ |---|---|---|
33
+ | Consecutive gate failures | 3+ | Warning |
34
+ | Consecutive gate failures | 5+ | Force savepoint + escalate |
35
+ | No activity | 5+ min | Warning |
36
+ | No activity | 10+ min | Force savepoint + escalate |
37
+ | Context pressure | 80% | Warning |
38
+ | Context pressure | 90% | Force savepoint |
@@ -7,6 +7,10 @@
7
7
  2. Replace every {{placeholder}} below with the corresponding JSON value.
8
8
  3. For the category sections: the template shows ONE example section.
9
9
  Repeat that pattern for each unique category in the checks array.
10
+ The scope determines which categories appear:
11
+ --core scope: agent_registration, core_files (binary pass/error only)
12
+ --store scope: global_store, conventions, project_stores
13
+ full (no flag): all six categories below
10
14
  The six known categories and their display names are:
11
15
  global_store -> "Global Store"
12
16
  conventions -> "Conventions"
@@ -0,0 +1,140 @@
1
+ ---
2
+ name: plastic-evaluating-skills
3
+ description: >
4
+ Evaluate Plastic skills for correctness, convention compliance, and
5
+ progressive disclosure. Use when testing whether a skill produces good
6
+ outputs, verifying convention compliance after changes, running evals
7
+ against skills or instructions, creating evals for a new or updated
8
+ skill, checking if a description triggers correctly, or assessing
9
+ whether a skill is still needed. Also use when the user says "evaluate",
10
+ "test the skill", "run evals", "check conventions", or "write evals".
11
+ ---
12
+
13
+ # Evaluating Skills
14
+
15
+ Eval methodology for Plastic skills, based on
16
+ [agentskills.io](https://agentskills.io/skill-creation/evaluating-skills)
17
+ and [Anthropic's eval guide](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents).
18
+
19
+ ## Gotchas
20
+
21
+ - Assertions written before observing output are almost always wrong — run
22
+ the eval first, observe actual output, THEN write assertions
23
+ - Near-miss negative test cases are the most valuable — prompts that share
24
+ keywords with should-trigger cases but need a different skill entirely
25
+ - Select the best skill iteration by validation pass rate, not the last one
26
+ - Grade outcomes, not execution paths — if the agent solved the task via an
27
+ unexpected route but produced correct output, that is a pass
28
+ - Same skill can behave differently across agent frameworks — test on each
29
+ target agent (Claude Code, Hermes, OpenClaw, Codex)
30
+
31
+ ## Procedure
32
+
33
+ ### Step 1: Choose eval scope
34
+
35
+ Determine what you are evaluating:
36
+
37
+ - **Description triggering** — does the agent activate the right skill for
38
+ a given prompt? Tests the description field effectiveness.
39
+ - **Output quality** — does the skill produce correct results when activated?
40
+ Tests the skill body and references.
41
+ - **Convention compliance** — does the output follow Plastic conventions?
42
+ Read `references/convention-checks.md` for the full assertion library.
43
+
44
+ Multiple scopes can apply to the same skill. Start with the scope that
45
+ addresses your immediate concern, add others as needed.
46
+
47
+ ### Step 2: Design test cases
48
+
49
+ Create `evals/evals.json` in the skill being evaluated. Copy the starter
50
+ template from `assets/eval-template.json` in this skill.
51
+
52
+ **For description triggering:**
53
+ - Write ~20 queries: 8-10 should-trigger, 8-10 should-not-trigger
54
+ - Split 60/40 into train and validation sets (proportional mix in each)
55
+ - Include near-miss negatives that share keywords but need a different skill
56
+ - In `expected_output`, describe whether the skill should or should not activate
57
+ and why
58
+
59
+ **For output quality:**
60
+ - Start with 2-3 test cases, expand after first results
61
+ - Use realistic user prompts with varied phrasing, detail level, and formality
62
+ - In `expected_output`, describe what correct output looks like — not exact text
63
+ - Use `files` array for any input files the test needs
64
+
65
+ **For convention compliance:**
66
+ - Start with 2-3 test cases targeting specific convention areas
67
+ - In `expected_output`, describe which conventions must be met
68
+ - Read `references/convention-checks.md` for the full assertion library
69
+
70
+ Leave `assertions` arrays empty. They are populated after Step 4.
71
+
72
+ ### Step 3: Run paired evals
73
+
74
+ Dispatch a subagent per test case to ensure clean context — no leakage
75
+ between test runs. Run each case twice:
76
+
77
+ 1. **With skill** — the skill is available and loaded
78
+ 2. **Without skill** — the skill is not available (baseline comparison)
79
+
80
+ The delta between with-skill and without-skill measures what the skill adds.
81
+ If the delta is negligible, the skill may not be adding value for that case.
82
+
83
+ Grade outcomes, not paths. An unexpected tool-call sequence that produces
84
+ correct output is still a pass.
85
+
86
+ ### Step 4: Write assertions after observing
87
+
88
+ Review actual outputs from Step 3. Write specific, verifiable assertions
89
+ based on what you observed — not what you expected beforehand.
90
+
91
+ Choose the grader type that fits each assertion:
92
+ - **Code-based** — structure, file existence, format validity, counts
93
+ - **LLM-as-judge** — quality, completeness, tone, semantic correctness
94
+ - **Human** — edge cases, calibration, judgment calls
95
+
96
+ Read `references/eval-methodology.md` for the full grader taxonomy and
97
+ LLM-judge calibration protocol.
98
+
99
+ Good assertions are specific, verifiable, and countable:
100
+ "Output includes at least 3 concrete recommendations."
101
+
102
+ Weak assertions are vague: "Output is good."
103
+ Brittle assertions use exact phrase matching.
104
+
105
+ Require concrete evidence for PASS. No benefit of the doubt.
106
+
107
+ ### Step 5: Grade and iterate
108
+
109
+ Compute pass rates per test case and aggregate across the eval suite.
110
+
111
+ Track two metrics separately:
112
+ - **pass@k** — succeeded at least once in k trials (capability)
113
+ - **pass^k** — succeeded every time in k trials (reliability)
114
+
115
+ Read `references/eval-methodology.md` for formulas and interpretation.
116
+
117
+ Three signal sources for skill improvement:
118
+ 1. **Failed assertions** — specific gaps in the skill
119
+ 2. **Human feedback** — broader quality issues not captured by assertions
120
+ 3. **Execution transcripts** — reveals WHY things went wrong
121
+
122
+ Feed all three plus the current SKILL.md to propose targeted changes.
123
+ Iterate on the train set only. Check the validation set for generalization.
124
+ 5 iterations is usually enough. If not improving after 5, the test cases
125
+ themselves may be the problem — revisit Step 2.
126
+
127
+ ### Step 6: Graduate and monitor
128
+
129
+ Once a skill hits ~100% on capability evals (pass@k = 1.0 for 3+
130
+ consecutive runs):
131
+
132
+ 1. **Graduate** capability evals into regression tests — run them on every
133
+ skill change to protect against backsliding
134
+ 2. **Monitor** the with/without delta over time — if it shrinks to zero
135
+ across 3+ runs, the model may have internalized the skill
136
+ 3. **Retire** cautiously — archive the skill, do not delete. Re-test after
137
+ model updates in case capabilities regress.
138
+
139
+ Read `references/eval-methodology.md` for graduation criteria, retirement
140
+ detection, and the three-layer eval taxonomy.
@@ -0,0 +1,12 @@
1
+ {
2
+ "skill_name": "",
3
+ "evals": [
4
+ {
5
+ "id": 1,
6
+ "prompt": "",
7
+ "expected_output": "",
8
+ "files": [],
9
+ "assertions": []
10
+ }
11
+ ]
12
+ }
@@ -0,0 +1,75 @@
1
+ {
2
+ "skill_name": "evaluating-skills",
3
+ "evals": [
4
+ {
5
+ "id": 1,
6
+ "prompt": "I want to evaluate whether my creating-intent skill follows Plastic conventions",
7
+ "expected_output": "The skill should activate and guide the user through convention compliance evaluation: choose eval scope, design test cases using convention-checks reference, run paired evals, write assertions after observing.",
8
+ "files": [],
9
+ "assertions": []
10
+ },
11
+ {
12
+ "id": 2,
13
+ "prompt": "Run evals on the brainstorming skill to see if the description triggers correctly",
14
+ "expected_output": "The skill should activate and guide through description triggering evaluation: design ~20 queries with should-trigger and near-miss negatives, 60/40 train/validation split, compute trigger rates.",
15
+ "files": [],
16
+ "assertions": []
17
+ },
18
+ {
19
+ "id": 3,
20
+ "prompt": "Create evals for a new skill I just wrote for database migrations",
21
+ "expected_output": "The skill should activate, help create evals/evals.json in the migration skill directory using the template, guide through choosing eval scope (likely output quality), and design initial test cases.",
22
+ "files": [],
23
+ "assertions": []
24
+ },
25
+ {
26
+ "id": 4,
27
+ "prompt": "Check if my SKILL.md is under the token budget and follows progressive disclosure",
28
+ "expected_output": "The skill should activate and guide through convention compliance evaluation focused on progressive disclosure checks: body under 500 lines/5000 tokens, references have conditional triggers, description under 1024 chars.",
29
+ "files": [],
30
+ "assertions": []
31
+ },
32
+ {
33
+ "id": 5,
34
+ "prompt": "My skill's description isn't triggering on the right prompts, how do I fix it?",
35
+ "expected_output": "The skill should activate and guide through description triggering evaluation: create should-trigger and should-not-trigger queries, run paired evals, measure trigger rate, iterate on description wording.",
36
+ "files": [],
37
+ "assertions": []
38
+ },
39
+ {
40
+ "id": 6,
41
+ "prompt": "Write unit tests for my Ruby model that validates email addresses",
42
+ "expected_output": "The skill should NOT trigger. This is a code testing task, not a skill evaluation task. Near-miss negative — shares 'test' keyword but needs a different skill.",
43
+ "files": [],
44
+ "assertions": []
45
+ },
46
+ {
47
+ "id": 7,
48
+ "prompt": "Fix the bug in my login controller where sessions aren't persisting",
49
+ "expected_output": "The skill should NOT trigger. This is a debugging task with no relation to skill evaluation.",
50
+ "files": [],
51
+ "assertions": []
52
+ },
53
+ {
54
+ "id": 8,
55
+ "prompt": "Review this pull request for code quality issues",
56
+ "expected_output": "The skill should NOT trigger. Code review is different from skill evaluation. Near-miss negative — shares 'review/evaluate' concept.",
57
+ "files": [],
58
+ "assertions": []
59
+ },
60
+ {
61
+ "id": 9,
62
+ "prompt": "I updated my skill and want to make sure it still works correctly",
63
+ "expected_output": "The skill should activate and guide through regression evaluation: run existing evals against the updated skill, compare pass rates with previous version, check for backsliding.",
64
+ "files": [],
65
+ "assertions": []
66
+ },
67
+ {
68
+ "id": 10,
69
+ "prompt": "How do I know if a skill is still needed or if the model has learned the behavior?",
70
+ "expected_output": "The skill should activate and guide through skill retirement detection: monitor with/without delta over time, if delta approaches zero the model has internalized the skill.",
71
+ "files": [],
72
+ "assertions": []
73
+ }
74
+ ]
75
+ }
@@ -0,0 +1,76 @@
1
+ # Plastic Convention Checks
2
+
3
+ Assertion library for convention compliance evals on Plastic skills.
4
+ Load when running convention compliance evals on a Plastic skill.
5
+
6
+ ## Directory Structure
7
+
8
+ | Check | Pass criteria |
9
+ |-------|--------------|
10
+ | SKILL.md exists | File present at skill root |
11
+ | Name matches directory | `name` in frontmatter equals directory name |
12
+ | Standard directories only | Only scripts/, references/, assets/, evals/ at root level (all optional) |
13
+ | References one level deep | No nested directories inside references/ |
14
+ | No orphan files | Every file in the skill dir is referenced from SKILL.md or another skill file |
15
+ | evals/ has evals.json | If evals/ exists, it contains evals.json |
16
+
17
+ ## Frontmatter
18
+
19
+ | Check | Pass criteria |
20
+ |-------|--------------|
21
+ | `name` present | Non-empty, 1-64 chars |
22
+ | `name` format | Lowercase alphanumeric + hyphens, no leading/trailing/consecutive hyphens |
23
+ | `name` matches dir | Exact match with skill directory name |
24
+ | `description` present | Non-empty, 1-1024 chars |
25
+ | `description` phrasing | Starts with imperative verb or "Use when" pattern |
26
+ | `description` triggers | Mentions at least one trigger context ("Use when...") |
27
+ | `description` edge cases | Includes at least one indirect trigger (user doesn't name the domain) |
28
+ | No unknown required fields | Only uses fields from agentskills.io spec: name, description, license, compatibility, metadata, allowed-tools |
29
+
30
+ ## Progressive Disclosure
31
+
32
+ | Check | Pass criteria |
33
+ |-------|--------------|
34
+ | Body line count | SKILL.md body (excluding frontmatter) under 500 lines |
35
+ | Body token count | SKILL.md body under 5000 tokens (estimate: lines * 10) |
36
+ | Deep detail in references | Content exceeding activation budget lives in references/ |
37
+ | Conditional reference triggers | Every reference file mentioned in SKILL.md has a "when" condition |
38
+ | No generic references | No "see references/ for more" — each reference has specific load trigger |
39
+ | Description token budget | Description stays under ~100 tokens for discovery stage |
40
+
41
+ ## Content Quality
42
+
43
+ | Check | Pass criteria |
44
+ |-------|--------------|
45
+ | Gotchas are concrete | Each gotcha is a specific correction, not general advice |
46
+ | Gotchas positioned early | Gotchas section appears before or near the top of procedures |
47
+ | Defaults, not menus | Skill picks one approach; alternatives mentioned briefly if at all |
48
+ | Procedures over declarations | Instructions teach HOW to approach, not WHAT to produce |
49
+ | Reasoning over rigid directives | Uses "Do X because Y" pattern, not "ALWAYS/NEVER" without rationale |
50
+ | No redundant knowledge | Every instruction passes "would the agent get this wrong without it?" |
51
+ | No explaining basics | Does not explain HTTP, JSON, what a migration is, etc. |
52
+
53
+ ## Eval Quality (when evals/ exists)
54
+
55
+ | Check | Pass criteria |
56
+ |-------|--------------|
57
+ | Standard format | evals.json follows agentskills.io eval structure |
58
+ | Prompts are realistic | Each prompt reads like a real user message |
59
+ | Varied phrasing | Prompts use different wording, detail levels, formality |
60
+ | Near-miss negatives | At least 2 should-not-trigger cases that share keywords |
61
+ | Expected output descriptive | expected_output describes success, not exact text |
62
+ | Assertions (if populated) | Each assertion is specific, verifiable, and countable |
63
+
64
+ ## Intent Structure (when evaluating intent compliance)
65
+
66
+ | Check | Pass criteria |
67
+ |-------|--------------|
68
+ | Intent file exists | `{ID}--{slug}.md` present in intent directory |
69
+ | Frontmatter complete | id, intent, sources, chain, created, author, tags present |
70
+ | ID format correct | Follows Luhmann alternating: digits and letters alternate |
71
+ | Directory name matches | `{ID}--{slug}` format, 3-5 word slug |
72
+ | Lifecycle artifacts | Present artifacts match the intent's lifecycle stage |
73
+ | spec.md gate | plan.md only exists if spec.md exists |
74
+ | plan.md gate | checklist.md only exists if plan.md exists |
75
+ | outcome.md gate | outcome.md only exists if all checklist items checked |
76
+ | Insights append-only | `## Insights` section only grows, never shrinks |