@zalom/plastic 1.0.0-alpha.10 → 1.0.0-alpha.11

package/skills/evaluating-skills/references/eval-methodology.md

@@ -0,0 +1,154 @@
+# Evaluation Methodology
+
+Load when choosing a grader type, interpreting pass rate results, or deciding
+whether to graduate or retire evals.
+
+Sources: agentskills.io, Anthropic eval engineering, Tessl eval framework,
+Philipp Schmid skill testing guide.
+
+## Three-Tier Grader Taxonomy
+
+Layer graders like a Swiss cheese model — no single tier catches everything.
+Use the simplest grader that covers the assertion. Combine tiers for coverage.
+
+### Code-Based Graders
+
+Best for structural and mechanical checks:
+- File existence and correct path
+- JSON/YAML validity
+- Line count, character count, token budget
+- Regex pattern matching (frontmatter fields, required sections)
+- Exit codes from bundled scripts
+
+Use as the default tier. Fast, deterministic, reproducible.
+
+### LLM-as-Judge Graders
+
+Best for subjective quality and semantic checks:
+- "Does this description convey when to use the skill?"
+- "Are the gotchas concrete corrections, not general advice?"
+- "Does the output address the user's actual intent?"
+
+Calibration protocol:
+1. Write a natural-language rubric (not binary pass/fail)
+2. Run the judge on 10-15 cases where you already know the correct grade
+3. Compare judge grades to your grades — adjust rubric until >80% agreement
+4. Spot-check with human grading periodically (every 5th eval run)
+
+Rubric template:
+- PASS: [specific criteria with examples]
+- PARTIAL: [what partial credit looks like]
+- FAIL: [specific failure modes]
+
+### Human Graders
+
+Best for edge cases and final calibration:
+- Novel failure modes the other tiers miss
+- Calibration set for LLM judges
+- Final sign-off on graduating evals to regression
+
+Use sparingly — human grading doesn't scale. Reserve for calibration
+and cases where code + LLM judges disagree.
+
+## pass@k vs pass^k
+
+Two metrics that diverge dramatically. Always track both.
+
+### pass@k — Capability
+
+"Did it succeed at least once in k trials?"
+
+Formula: pass@k = 1 - (1 - p)^k where p = single-trial pass rate
+
+Measures: Can the skill/agent do this at all?
+
+Example at p=0.7, k=10: pass@k = 97.2%
+
+Use when: evaluating whether a skill enables a new capability,
+initial development, deciding whether to invest more iteration.
+
+### pass^k — Reliability
+
+"Did it succeed every time in k trials?"
+
+Formula: pass^k = p^k
+
+Measures: Will it always do this correctly?
+
+Example at p=0.7, k=10: pass^k = 2.8%
+
+Use when: evaluating production readiness, regression testing,
+deciding whether a skill is reliable enough to ship.
+
+### Interpreting the Gap
+
+| pass@k | pass^k | Interpretation |
+|--------|--------|----------------|
+| High | High | Reliable — ready for production |
+| High | Low | Capable but flaky — needs iteration on consistency |
+| Low | Low | Not yet capable — needs fundamental skill improvement |
+| Low | High | Impossible (pass^k <= pass@k always) |
+
+Run k=3 minimum for meaningful results. k=5 for production decisions.
+
+## Capability-to-Regression Graduation
+
+Track pass rates across iterations. When a capability eval consistently
+hits ~100% (pass@k=1.0 for 3+ consecutive runs):
+
+1. Graduate the eval from "capability" to "regression"
+2. Regression evals run on every skill change — they protect against backsliding
+3. If a regression eval starts failing, the recent change broke something
+4. Investigate the failing regression before iterating further
+
+Graduation is one-way. Once an eval is regression, it stays regression
+unless the underlying requirement changes.
+
+## Skill Retirement Detection
+
+Monitor the with/without skill delta over time:
+
+1. Run paired evals (with-skill vs without-skill) periodically
+2. Compute the delta in pass rates
+3. If delta shrinks to near-zero across 3+ consecutive runs:
+   - The model has likely internalized the skill's knowledge
+   - The skill may be ready for retirement
+4. Before retiring: run one final full eval suite to confirm
+5. Archive the skill (don't delete — may need to restore if model changes)
+
+Common cause of false retirement signals: model update changed capabilities.
+Re-test after model updates.
+
+## Tessl Three-Layer Eval Taxonomy
+
+Three layers of increasing realism. Each catches failures the others miss.
+
+### Layer 1: Skill Review (Structural Lint)
+
+Does the skill itself follow best practices?
+- SKILL.md structure (frontmatter, sections, length)
+- Description quality (imperative, user-intent, trigger keywords)
+- Reference organization (conditional triggers, one level deep)
+- Convention compliance (for Plastic skills, see convention-checks.md)
+
+Fast, cheap, runs without executing the skill.
+
+### Layer 2: Task Evals (Synthetic)
+
+Does the skill improve agent output on synthetic tasks?
+- Paired with/without comparison
+- Controlled prompts with known-good expected outputs
+- Measures the delta the skill adds
+
+This is the core eval loop (Steps 2-5 in the SKILL.md procedure).
+
+### Layer 3: Repo Evals (Real Codebase)
+
+Does the skill work correctly in a real project context?
+- Install the skill in a real repository
+- Run real tasks (not synthetic prompts)
+- Measure whether the agent uses the skill correctly in situ
+
+Most expensive, most realistic. Catches skills that pass synthetic tests
+but fail under real project complexity. Use for high-stakes skills
+or before shipping to users.

package/skills/linking-intents/SKILL.md

@@ -70,3 +70,7 @@ Add a wikilink in the `## Links` section of **both** intents (bidirectional).
 
 ### 4. Update INDEX.md Clusters
 If both intents share a topic, ensure they're in the same cluster.
+
+## References
+
+- Read `references/zettelkasten.md` for the three Zettelkasten structures (Folgezettel, directed graph, tags), ID encoding rules, and dual-mode (Obsidian + programmatic) design

package/skills/linking-intents/references/zettelkasten.md

@@ -0,0 +1,33 @@
+# Zettelkasten Structure
+
+Plastic implements three Zettelkasten structures:
+
+| Structure | Implementation | Purpose |
+|---|---|---|
+| Folgezettel (linked list) | `sources` + `chain` in frontmatter | Sequential provenance |
+| Directed graph (web of notes) | `## Links` with wikilinks | Obsidian navigation |
+| Tag-based taxonomy | `tags` in frontmatter | Topic grouping |
+
+INDEX.md is a structure note (hub), not a table of contents.
+
+## Folgezettel IDs
+
+IDs encode lineage using Luhmann's alternating convention:
+- Root intents: sequential numbers (`1`, `2`, `3`...)
+- Branches alternate letters and numbers: `1` → `1a` → `1a1` → `1a1a` → ...
+- Multiple branches from the same parent increment: `1a`, `1b`, `1c`
+- IDs are assigned at creation time and never change
+
+## Knowledge Graph
+
+`sources` + `chain` form the double-linked knowledge graph:
+- `sources` = what fed into this intent (parents, inspirations, prerequisites)
+- `chain` = what this intent produced (children, follow-ups, spin-offs)
+
+## Dual-Mode
+
+This store works in two modes without modification:
+- **Obsidian** (human, offline) — browse, link, write markdown
+- **Programmatic** (any agent) — read/write via filesystem operations
+
+No special tooling required for either mode.

package/skills/managing-index/SKILL.md

@@ -64,3 +64,7 @@ When 3+ intents share tags but aren't in a cluster, suggest a new cluster headin
 Intents with no links (empty `sources`, empty `chain`, no `## Links` entries, not in any cluster) should be flagged for curation.
 
 REQUIRED BACKGROUND: linking-intents (for understanding connection types and Zettelkasten theory)
+
+## References
+
+- Read `references/zettelkasten-linking.md` for the three structural layers (Folgezettel, directed graph, tags) and how they map to INDEX.md organization

package/skills/releasing/SKILL.md

@@ -193,3 +193,7 @@ git tag -a v0.1.0 <commit-sha> -m "v0.1.0 — [description]"
 ```
 
 Use `git log --oneline` to find the right commits (look for version bump commits or major feature merges).
+
+## References
+
+- Read `references/deprecations.md` for the full deprecation process, severity levels, deprecations.yml schema, and dismissal rules when adding or managing deprecations

package/skills/releasing/references/deprecations.md

@@ -0,0 +1,44 @@
+# Deprecation Process
+
+When removing a feature, changing a convention, or making a breaking change:
+
+1. Add entry to `deprecations.yml` in the Plastic source root
+2. Set severity: `info` (awareness), `warning` (action needed), `critical` (urgent)
+3. Provide clear migration steps
+4. Set `removal` version at least 2 minor versions ahead
+5. SessionStart hook displays active deprecations automatically
+6. Remove the feature AND the deprecation entry together
+
+## Severity Levels
+
+| Severity | When to use | Dismissable? |
+|----------|-------------|--------------|
+| `info` | Upcoming change | Yes |
+| `warning` | Action needed | Yes (re-shown at removal) |
+| `critical` | Urgent/security | Never |
+
+## deprecations.yml Schema
+
+```yaml
+deprecations:
+  - id: unique-slug
+    severity: info | warning | critical
+    summary: "One-line description"
+    migration_steps:
+      - "Step 1"
+      - "Step 2"
+    introduced: "0.9.0"
+    removal: "1.0.0"
+    link: "optional URL"
+```
+
+## Dismissal
+
+Users dismiss by adding the id to `deprecations_dismissed` in `config.yml`:
+
+```yaml
+deprecations_dismissed:
+  - some-deprecated-feature
+```
+
+Critical deprecations and final-version warnings ignore dismissal.

package/skills/savepoint/SKILL.md

@@ -55,3 +55,7 @@ git commit -m "chore: savepoint — [active intent name]"
 
 ### 5. Notify User
 Tell the user: "Context is getting large. I've saved progress to intent [ID] — [name]. Please run `/clear` and say `continue` to resume."
+
+## References
+
+- Read `references/context-management.md` for the full save/continue protocol when the save flow needs debugging or you need to understand the full resume sequence

package/skills/savepoint/references/context-management.md

@@ -0,0 +1,32 @@
+# Context Management (Start-Save-Continue)
+
+## Save Point
+Triggered by PreCompact hook or manually:
+1. Find active intent(s) from `~/.plastic/INDEX.md`
+2. Update active intent's `checklist.md` (check off completed items)
+3. Update active intent's `savepoint.md` (in-progress, next steps, blockers, discoveries)
+4. Add observations to `## Insights`
+5. Update INDEX.md
+6. Commit: `cd ~/.plastic && git add . && git commit -m "chore: savepoint — [intent name]"`
+7. Notify user to `/clear`
+
+## Continue
+Triggered by UserPromptSubmit hook when user says "continue". Priority order:
+
+**1. Active intents first (resume work):**
+1. Read INDEX.md → find active intent(s)
+2. Read active intent's `intent.md` → what and why
+3. Read active intent's `savepoint.md` → where we left off
+4. Read active intent's `checklist.md` → what's next
+5. Announce: intent name, current state, next step, blockers
+6. Resume
+
+**2. No active intents → offer future intents:**
+1. List all future intents from INDEX.md
+2. Present them as options
+3. When user picks one, move to Active in INDEX.md
+
+**3. Stale future intents (untouched 3+ days) → triage:**
+- **activate** — start working on it now
+- **abandon** — mark as abandoned
+- **defer to agent** — implement, research, or ideate

package/skills/writing-instructions/SKILL.md

@@ -0,0 +1,159 @@
+---
+name: writing-instructions
+description: >
+  Write or restructure agent instructions, conventions files, and SKILL.md
+  content using progressive disclosure and the agentskills.io specification.
+  Use when creating PLASTIC.md, rewriting convention docs, authoring new
+  skills, restructuring large instruction files to fit context budgets,
+  or when instructions feel too long, too vague, or agents aren't following
+  them. Also use when the user says "progressive disclosure", "restructure
+  instructions", or "the instructions are too big".
+---
+
+# Writing Agent Instructions
+
+Based on the [agentskills.io specification](https://agentskills.io).
+
+## Progressive Disclosure Architecture
+
+All agent instructions follow three loading stages. Design for this — it's the
+architecture, not a nice-to-have.
+
+| Stage | What loads | Budget | Design for |
+|-------|-----------|--------|------------|
+| **Discovery** | name + description | ~100 tokens | Trigger accuracy |
+| **Activation** | Full instruction body | <5000 tokens / <500 lines | Core procedures |
+| **Execution** | references/, scripts/, assets/ | As needed | Deep detail |
+
+**The description carries the entire burden of triggering.** If agents aren't
+activating your skill, the description is the problem.
+
+## Procedure
+
+### Step 1: Audit the content
+
+Before writing or restructuring, classify every piece of content:
+
+| Classification | Where it goes | Example |
+|---------------|---------------|---------|
+| **Trigger context** | description field | "Use when...", keywords |
+| **Core procedure** | SKILL.md body | Step-by-step workflows |
+| **Gotchas** | SKILL.md body (early) | Facts that defy assumptions |
+| **Deep reference** | references/ | API details, full schemas |
+| **Templates** | assets/ or inline | Output format examples |
+| **Executable logic** | scripts/ | Validation, data processing |
+
+Ask for each piece: "Would the agent get this wrong without this?"
+If no — cut it. If unsure — test it.
+
+### Step 2: Write the description
+
+The description must convey WHEN to use, not just WHAT it does.
+
+**Rules:**
+- Imperative phrasing: "Use this skill when..." not "This skill does..."
+- Focus on user intent, not implementation
+- Err on the side of being pushy — list contexts explicitly
+- Include cases where user doesn't name the domain directly
+- Under 1024 characters (hard limit)
+- Include specific trigger keywords
+
+**Template:**
+```yaml
+description: >
+  [One sentence: what it does]. Use when [primary trigger context],
+  [secondary trigger], or when [indirect trigger where user doesn't
+  name the domain]. Also use when [edge case trigger].
+```
+
+### Step 3: Write the body
+
+**Principles (in priority order):**
+
+1. **Procedures over declarations** — Teach HOW to approach a class of problems,
+   not WHAT to produce for a specific instance
+2. **Defaults, not menus** — Pick one approach, mention alternatives briefly.
+   "Use X. For Y cases, use Z instead." Never present options as equals.
+3. **Calibrate control to fragility** — Prescriptive for fragile/sequential tasks,
+   flexible when multiple approaches are valid. Most skills have a mix.
+4. **Reasoning over rigid directives** — "Do X because Y tends to cause Z" beats
+   "ALWAYS do X, NEVER do Y"
+5. **Add what the agent lacks, omit what it knows** — No explaining HTTP, PDFs,
+   or what a migration is. Jump to what's non-obvious.
+
+**Structure:**
+```markdown
+# Title
+
+[1-2 sentence purpose statement]
+
+## Gotchas
+- [Highest-value content first — facts that defy assumptions]
+- [Each gotcha is a concrete correction, not general advice]
+
+## Procedure
+### Step 1: ...
+### Step 2: ...
+
+## Patterns
+[Only if the skill covers multiple approaches to similar problems]
+```
+
+### Step 4: Extract to references/
+
+Anything over 500 lines or 5000 tokens goes to `references/`. But tell the
+agent WHEN to load each file — conditional references, not a generic "see
+references/ for details."
+
+**Good:** "Read `references/api-errors.md` if the API returns a non-200 status"
+**Bad:** "See references/ for more information"
+
+### Step 5: Validate
+
+Run through this checklist:
+
+- [ ] Description under 1024 chars
+- [ ] Body under 500 lines / 5000 tokens
+- [ ] Every instruction passes "would the agent get this wrong without it?"
+- [ ] Gotchas are concrete corrections, not general advice
+- [ ] Defaults chosen, not menus presented
+- [ ] Control calibrated: prescriptive where fragile, flexible where tolerant
+- [ ] References have conditional load triggers
+- [ ] No explaining what the agent already knows
+
+For structured evaluation beyond this checklist (paired evals, pass rate
+tracking, regression testing), use `plastic:evaluating-skills`.
+
+## Gotchas
+
+- Hook `additionalContext` truncates at **10,000 characters**. If instructions
+  must load via hooks (not skills), they must fit this budget.
+- `~/.claude/rules/*.md` files load fully with no truncation — use for
+  always-loaded conventions that don't fit in a skill.
+- CLAUDE.md supports `@path/to/file` imports (max depth 4) — another way
+  to load large instruction sets without truncation.
+- Agents only consult skills for tasks beyond what they can handle alone.
+  Simple one-step requests may not trigger even with a perfect description.
+- Content from real domain expertise (runbooks, incident reports, code review)
+  dramatically outperforms LLM-generated instructions without project context.
+- The most common cause of agents not following instructions: the instruction
+  was too vague, didn't apply to the current task, or presented too many
+  options without a clear default. Read execution traces to diagnose.
+
+## For Convention Files (PLASTIC.md, CLAUDE.md)
+
+Convention files aren't skills — they load differently. Apply progressive
+disclosure by splitting:
+
+| Layer | Mechanism | Budget | Content |
+|-------|-----------|--------|---------|
+| **Always-on** | `~/.claude/rules/` or hook | <10K chars | Core identity, gotchas, critical procedures |
+| **On-demand** | Skills (SKILL.md) | <5K tokens each | Specific workflows activated by task |
+| **Deep reference** | `references/` in skills | Unlimited | Full specs, schemas, examples |
+
+**The always-on layer should answer: "What does this agent need to know about
+every single task?"** Everything else activates on demand.
+
+## References
+
+- Read `references/agentskills-spec.md` for the complete agentskills.io specification including frontmatter fields, description optimization methodology, evaluation framework, and script design requirements

package/skills/writing-instructions/references/agentskills-spec.md

@@ -0,0 +1,135 @@
+# agentskills.io Full Reference
+
+Source: https://agentskills.io (all sections, verified June 2026)
+
+## Specification Details
+
+### Frontmatter Fields
+
+| Field | Required | Constraints |
+|-------|----------|-------------|
+| name | Yes | 1-64 chars. Lowercase alphanumeric + hyphens. No leading/trailing/consecutive hyphens. Must match directory name. |
+| description | Yes | 1-1024 chars. Non-empty. What + when. |
+| license | No | Short — name or filename reference |
+| compatibility | No | 1-500 chars. Environment requirements only when needed. |
+| metadata | No | String→string map. Use unique key names. |
+| allowed-tools | No | Space-separated. Experimental. |
+
+### Progressive Disclosure Token Budgets
+
+- Discovery: ~100 tokens per skill (name + description only)
+- Activation: <5000 tokens / <500 lines recommended for SKILL.md body
+- Execution: Unbounded — files in scripts/, references/, assets/ load as needed
+
+### File References
+
+- Use relative paths from skill root
+- Keep one level deep from SKILL.md
+- Agent resolves paths automatically
+
+## Description Optimization
+
+### Evaluation Methodology
+
+1. Create ~20 eval queries (8-10 should-trigger, 8-10 should-not)
+2. Split 60/40 train/validation (proportional mix in each)
+3. Run each query 3 times, compute trigger rate
+4. Pass threshold: 0.5
+5. Near-miss negatives are most valuable (share keywords, need different thing)
+6. Iterate on train set only, validate on held-out set
+7. Select best by validation pass rate, not last iteration
+8. 5 iterations usually sufficient
+
+### Description Anti-patterns
+
+- "Helps with PDFs" — too vague, no trigger context
+- "Process CSV files" — no when/why, no user-intent focus
+- Implementation details instead of user intent
+- Missing edge case triggers (user doesn't name the domain)
+
+## Instruction Best Practices
+
+### Gotchas — Highest-Value Content
+
+Concrete corrections, not general advice:
+
+```markdown
+## Gotchas
+- The `users` table uses soft deletes. Queries must include
+  `WHERE deleted_at IS NULL`.
+- User ID is `user_id` in DB, `uid` in auth, `accountId` in billing.
+  All three refer to the same value.
+- The `/health` endpoint returns 200 even if DB is down. Use `/ready`.
+```
+
+### Calibrating Control
+
+Prescriptive when:
+- Operations are fragile
+- Consistency matters
+- Specific sequence must be followed
+
+Flexible when:
+- Multiple approaches are valid
+- Task tolerates variation
+- Explaining WHY is more effective than rigid rules
+
+### Instruction Patterns
+
+1. **Validation loops**: Do work → validate → fix → repeat
+2. **Plan-validate-execute**: Create plan → validate vs source of truth → execute
+3. **Checklists**: Track progress, enforce dependencies, validation gates
+4. **Bundled scripts**: If agent reinvents same logic each run, bundle it
+5. **Templates**: Concrete output structures > prose descriptions
+
+## Script Design
+
+### Hard Requirements
+- No interactive prompts (hard requirement — agents hang indefinitely)
+- All input via flags, env vars, or stdin
+
+### Agent-Friendly Design
+- --help as primary interface documentation
+- Helpful error messages: what wrong + what expected + what to try
+- Structured output (JSON/CSV/TSV), data on stdout, diagnostics on stderr
+- Idempotent operations (agents may retry)
+- Dry-run for destructive operations
+- Meaningful exit codes documented in --help
+- Output size control: default to summaries, support --offset pagination
+- Agent harnesses truncate at 10-30K characters
+
+## Evaluation Framework
+
+### Test Case Structure
+```json
+{
+  "skill_name": "name",
+  "evals": [{
+    "id": 1,
+    "prompt": "realistic user message",
+    "expected_output": "what success looks like",
+    "files": ["evals/files/input.csv"],
+    "assertions": ["specific, verifiable checks"]
+  }]
+}
+```
+
+### Running Evals
+- With-skill vs without-skill (or previous version) comparison
+- Clean context per run (subagents or separate sessions)
+- Capture timing: total_tokens, duration_ms
+- Start with 2-3 test cases, expand after first results
+
+### Assertion Quality
+Good: Programmatically verifiable, specific, countable
+Weak: Vague ("the output is good")
+Brittle: Exact phrase matching
+
+Principle: Require concrete evidence for PASS. No benefit of the doubt.
+
+### Iteration Loop
+1. Run evals → grade assertions → aggregate benchmarks
+2. Identify failures (assertions, human feedback, execution transcripts)
+3. Feed all three + SKILL.md to LLM for proposed changes
+4. Apply changes → re-run → compare
+5. Stop when consistently empty feedback or no meaningful improvement