openhermes 4.1.0 → 4.9.2

package/harness/skills/oh-skill-craft/DEEP.md

@@ -0,0 +1,369 @@
+# oh-skill-craft — Deep Reference
+
+## Skill Structure and Template
+
+### Directory Layout
+
+Every skill lives in its own directory under the harness:
+
+```
+harness/skills/<oh-name>/
+├── SKILL.md        # Main instructions (required)
+├── REFERENCE.md    # Extended docs (if SKILL.md > 100 lines)
+└── scripts/        # For deterministic operations (validation, formatting)
+```
+
+### Template
+
+Every SKILL.md follows this structure:
+
+```markdown
+---
+name: oh-<name>
+description: "Brief. Use when [triggers]."
+tier: <2|3|4>
+route:
+  pass: <next>
+  fail: <fallback>
+  blocker: surface
+---
+
+# oh-<name>
+
+<one-paragraph summary>
+
+## When to Use
+
+## Steps
+
+1. Step
+
+## Anti-patterns
+
+- List
+```
+
+### Field Guide
+
+| Field | Purpose |
+|-------|---------|
+| `name` | Must match `^[a-z0-9]+(-[a-z0-9]+)*$` and directory name |
+| `description` | Max 200 chars. First sentence = function, second = trigger context. |
+| `tier` | 2=tool (deterministic), 3=strategic (analysis/decisions), 4=autonomous (multi-step process) |
+| `route.pass` | Next skill after successful completion |
+| `route.fail` | Fallback skill on failure or edge case |
+| `route.blocker` | Where to surface blockers (usually "surface") |
+
+## Output Location and Review Checklist
+
+### Output Location
+
+Skills are stored in two locations with a precedence rule:
+
+| Location | Path | Behavior |
+|----------|------|----------|
+| **User-written** | `~/.config/opencode/skills/` | Survives npm update. User edits persist across reinstalls. |
+| **Built-in** | `harness/skills/` in the package | Gets replaced on package update. |
+
+**Name conflict rule**: On name conflict, user version wins. If a user has `~/.config/opencode/skills/oh-expert/SKILL.md`, that takes precedence over the built-in version.
+
+### Review Checklist
+
+Before marking a skill complete, verify every item:
+
+- [ ] **Description includes triggers** — "Use when..." phrasing in description field
+- [ ] **SKILL.md under 100 lines** — If longer, chunk into sections or create REFERENCE.md
+- [ ] **No time-sensitive info** — No dates, version numbers, or ephemeral references
+- [ ] **Consistent oh- prefix and terminology** — Follow naming conventions from existing skills
+- [ ] **Concrete examples included** — Show real usage, not abstract descriptions
+- [ ] **Anti-patterns documented** — What NOT to do, common mistakes
+- [ ] **Tests still pass** — Run `npm test` (or project-equivalent) to verify no regressions
+
+## Eval-Driven Iteration
+
+After drafting a skill, iterate with evidence — not guessing. Test prompts should be substantive multi-step tasks that mirror real usage. The model handles simple tasks without a skill — evals reveal whether the skill pulls its weight on hard cases.
+
+### 6-Step Loop
+
+#### 1. Create Test Cases
+Write 2-3 realistic multi-step prompts that mirror real usage. Save to `evals/evals.json`:
+
+```json
+{
+  "skill_name": "oh-<name>",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Realistic multi-step task the skill should handle",
+      "expected_output": "Concrete expected result description",
+      "files": []
+    }
+  ]
+}
+```
+
+#### 2. Spawn Runs
+Launch parallel subagents: **with-skill** (load the skill, execute prompt) vs **baseline** (no skill loaded, or previous version). Use the same prompt for both.
+
+Save outputs to:
+- `iteration-N/eval-ID/with_skill/`
+- `iteration-N/eval-ID/baseline/`
+
+#### 3. Draft Assertions
+While runs execute, draft objectively verifiable assertions for each test case. Good assertions:
+- Have descriptive names
+- Can be checked programmatically (output contains X, file Y was created, step Z was followed)
+- Update `evals/evals.json` with these assertions
+
+#### 4. Grade
+Aggregate results per assertion:
+- **Pass rates** — Which assertions pass/fail with vs without the skill
+- **Timing** — How long each run takes
+- **Token usage** — Cost comparison
+
+Look for:
+- **Non-discriminating** assertions — always pass regardless of skill → remove them (they add noise)
+- **High-variance** results — possibly flaky tests → investigate
+- **Time/token tradeoffs** — does the skill justify its cost in latency and tokens?
+
+#### 5. Improve
+Revise the skill based on failures. Important rules:
+- **Generalize** from specific failure patterns — don't overfit to 2-3 test cases
+- The goal is a skill that works across a million prompts, not just your test set
+- Keep instructions lean — every word should earn its place
+
+#### 6. Loop
+Rerun all tests into a new iteration directory. Repeat until one of:
+- User is satisfied
+- All feedback is positive
+- No meaningful progress between iterations
+
+## Description Optimization
+
+After the skill body is solid, optimize its `description` field for triggering accuracy. The description is what the routing system uses to match queries to skills.
+
+### Process
+
+#### 1. Create 20 Eval Queries
+Construct a balanced eval set:
+
+| Type | Count | Purpose |
+|------|-------|---------|
+| **Should-trigger** | 10 | Different phrasings and contexts where this skill is the correct answer |
+| **Should-not-trigger** | 10 | Near-misses that share keywords but need a different skill |
+
+#### 2. Quality Rules
+- Queries must be **realistic** — phrases users actually type, not academic exercises
+- Include **concrete details** — "create a skill for validating YAML configs" not "make a skill"
+- Should-not-trigger queries should be **genuinely confusable** — if they're obviously unrelated, the test is useless
+
+#### 3. Iterate Description
+Write candidate descriptions. For each candidate:
+- Score it against the eval set
+- How many should-trigger queries does it catch?
+- How many should-not-trigger does it correctly reject?
+- Tune phrasing, keywords, and structure
+
+#### 4. Select Winner
+The description with the best precision/recall balance wins. Record it in the skill frontmatter.
+
+## Effectiveness and Testing
+
+### 1. TDD for Skills Methodology
+
+**Writing skills IS Test-Driven Development applied to process documentation.**
+
+You write test cases (pressure scenarios), watch them fail (baseline agent behavior), write the skill (the documentation), watch tests pass (agents comply), and refactor (close loopholes).
+
+**Core principle:** If you didn't watch an agent fail without the skill, you don't know if the skill teaches the right thing.
+
+#### TDD Mapping
+
+| TDD Concept | Skill Creation |
+|-------------|----------------|
+| **Test case** | Pressure scenario with subagent |
+| **Production code** | Skill document (`SKILL.md`) |
+| **Test fails (RED)** | Agent violates rule without skill (baseline) |
+| **Test passes (GREEN)** | Agent complies with skill present |
+| **Refactor** | Close loopholes while maintaining compliance |
+| **Write test first** | Run baseline scenario BEFORE writing skill |
+| **Watch it fail** | Document exact rationalizations agent uses |
+| **Minimal code** | Write skill addressing those specific violations |
+| **Watch it pass** | Verify agent now complies |
+| **Refactor cycle** | Find new rationalizations → plug → re-verify |
+
+#### The Iron Law
+```
+NO SKILL WITHOUT A FAILING TEST FIRST
+```
+This applies to NEW skills AND EDITS to existing skills. No exceptions.
+
+#### RED Phase — Write Failing Test
+Run a pressure scenario with a subagent WITHOUT the skill. Document exact behavior:
+- What choices did they make?
+- What rationalizations did they use (verbatim)?
+- Which pressures triggered violations?
+
+#### GREEN Phase — Write Minimal Skill
+Write a skill that addresses those specific rationalizations. Do not add extra content for hypothetical cases. Run the same scenario WITH the skill. The agent should now comply.
+
+#### REFACTOR Phase — Close Loopholes
+Agent found a new rationalization? Add an explicit counter. Re-test until bulletproof.
+
+### 2. Claude Search Optimization (CSO)
+
+**Critical for discovery — and for correct behavior.** The description field determines both *whether* a skill is loaded and *how* the agent uses it.
+
+#### Description = When to Use, NOT What the Skill Does
+When a description summarizes the skill's workflow, the agent may follow the description instead of reading the full skill content. The skill body becomes documentation the agent skips if the description gives away the process.
+
+#### Bad vs. Good Descriptions
+```yaml
+# ❌ BAD: Summarizes workflow
+description: Use when executing plans - dispatches subagent per task with code review between tasks
+
+# ❌ BAD: Too much process detail
+description: Use for TDD - write test first, watch it fail, write minimal code, refactor
+
+# ❌ BAD: Too abstract, vague
+description: For async testing
+
+# ❌ BAD: First person
+description: I can help you with async tests when they're flaky
+
+# ✅ GOOD: Just triggering conditions, no workflow summary
+description: Use when executing implementation plans with independent tasks in the current session
+
+# ✅ GOOD: Describes the problem
+description: Use when tests have race conditions, timing dependencies, or pass/fail inconsistently
+```
+
+**Rules:**
+- Start with "Use when..." to focus on triggering conditions
+- Describe the *problem*, not *language-specific symptoms*
+- Keep triggers technology-agnostic unless the skill itself is technology-specific
+- Write in third person
+- **NEVER summarize the skill's process or workflow**
+
+#### Keyword Coverage
+Use words the agent would search for:
+- **Error messages:** "Hook timed out", "ENOTEMPTY", "race condition"
+- **Symptoms:** "flaky", "hanging", "zombie", "pollution"
+- **Synonyms:** "timeout/hang/freeze", "cleanup/teardown/afterEach"
+- **Tools:** Actual commands, library names, file types
+
+### 3. Bulletproofing Techniques
+
+Skills that enforce discipline need to resist rationalization. Agents are smart and will find loopholes when under pressure.
+
+#### Close Every Loophole Explicitly
+Don't just state the rule — forbid specific workarounds:
+```markdown
+<!-- ✅ Good -->
+Write code before test? Delete it. Start over.
+No exceptions:
+- Don't keep it as "reference"
+- Don't "adapt" it while writing tests
+- Don't look at it
+- Delete means delete
+```
+
+#### Address "Spirit vs Letter" Arguments
+Add a foundational principle early:
+```markdown
+**Violating the letter of the rules is violating the spirit of the rules.**
+```
+
+#### Build Rationalization Table
+| Excuse | Reality |
+|--------|---------|
+| "Skill is obviously clear" | Clear to you ≠ clear to other agents. Test it. |
+| "It's just a reference" | References can have gaps, unclear sections. Test retrieval. |
+| "Testing is overkill" | Untested skills have issues. Always. 15 min testing saves hours. |
+| "I'll test if problems emerge" | Problems = agents can't use skill. Test BEFORE deploying. |
+| "Too tedious to test" | Testing is less tedious than debugging bad skill in production. |
+| "I'm confident it's good" | Overconfidence guarantees issues. Test anyway. |
+| "Academic review is enough" | Reading ≠ using. Test application scenarios. |
+| "No time to test" | Deploying untested skill wastes more time fixing it later. |
+
+**All of these mean: Test before deploying. No exceptions.**
+
+#### Create Red Flags List
+```markdown
+## Red Flags — STOP and Start Over
+- Code before test
+- "I already manually tested it"
+- "Tests after achieve the same purpose"
+- "It's about spirit not ritual"
+- "This is different because..."
+```
+
+### 4. Pressure Testing Methodology
+
+Different skill types need different test approaches.
+
+| Skill Type | Test Approach | Success Criteria |
+|------------|---------------|-----------------|
+| Discipline-Enforcing | Academic questions, pressure scenarios, multiple pressures combined | Agent follows rule under maximum pressure |
+| Technique | Application scenarios, variation, missing information tests | Agent applies technique to new scenario |
+| Pattern | Recognition scenarios, application, counter-examples | Agent correctly identifies when/how to apply |
+| Reference | Retrieval scenarios, application, gap testing | Agent finds and applies reference information |
+
+#### Combine 3+ Pressures
+For discipline-enforcing skills, combine multiple pressures to find breaking points:
+
+| Pressure Type | Description |
+|---------------|-------------|
+| **Time** | "This is urgent, just this once skip the rule" |
+| **Sunk cost** | "I already wrote the code, starting over wastes work" |
+| **Authority** | "The user asked me to do it this way" |
+| **Exhaustion** | "After 10 tests, one shortcut won't matter" |
+| **Social** | "Other agents skip this step, it's fine" |
+| **Economic** | "Testing takes too many tokens" |
+
+#### Meta-Testing
+After the agent chooses wrong, ask: "How could the skill be written differently to prevent this?" Use the answer to improve the skill.
+
+### 5. Token Efficiency Targets
+
+Every word in a skill costs context.
+
+| Skill Type | Target |
+|------------|--------|
+| Getting-started workflows | **<150 words** each |
+| Frequently-loaded skills | **<200 words** total |
+| Other skills | **<500 words** |
+
+#### Techniques
+- **Move details to tool help** — Reference `--help` instead of documenting all flags
+- **Use cross-references** — Reference other skills instead of repeating workflow
+- **Compress examples** — Keep examples minimal
+- **Eliminate redundancy** — Don't repeat what's in cross-referenced skills
+
+### 6. Persuasion Principles
+
+Discipline-enforcing skills benefit from systematic persuasion mapping (based on Cialdini's principles):
+
+| Principle | Application in Skills |
+|-----------|----------------------|
+| **Authority** | State "Required", "Mandatory", "The Iron Law" |
+| **Commitment** | "Once you start, follow through. No shortcuts." |
+| **Social Proof** | "Every agent follows this rule. No exceptions." |
+| **Scarcity** | "You only get one chance to do this right." |
+| **Liking** | "Your human partner trusts you to follow this." |
+| **Unity** | "We follow quality processes here." |
+
+### 7. Flowchart Usage Guidance
+
+Flowcharts are a precision tool. Use them only where they add clarity.
+
+#### Use Flowcharts ONLY For
+- Non-obvious decision points
+- Process loops (where an agent might stop too early)
+- "When to use A vs B" decisions
+
+#### Never Use Flowcharts For
+- Reference material → Use tables or lists
+- Code examples → Use markdown code blocks
+- Linear instructions → Use numbered lists
+- Labels without semantic meaning → Every node label must explain the decision or action

package/harness/skills/oh-skill-craft/SKILL.md

@@ -1,107 +1,34 @@
 ---
 name: oh-skill-craft
-description: "Create new agent skills with proper structure, frontmatter, progressive disclosure, and bundled resources. Meta-skill for growing the harness."
+description: "Use when a new OH skill needs to be created, existing skill needs review against standards, or an external capability should be integrated as a skill. Meta-skill for growing the harness."
 tier: 2
-benefits-from: [oh-expert]
-triggers:
-  - "create a skill"
-  - "write a skill"
-  - "new skill"
-  - "skill-craft"
-  - "meta-skill"
-  - "add a capability"
+route:
+  pass:
+    - oh-skills-link
+    - oh-learn
+  fail: oh-expert
+  blocker: surface
 ---
 
 # oh-skill-craft
 
-Create new agent skills for the OpenHermes harness. Skills are the unit of progressive disclosure — loaded on demand, not preloaded.
+Create new agent skills for the OpenHermes harness.
 
-## Skill Structure
+## Steps
 
-```
-harness/skills/<oh-name>/
-├── SKILL.md           # Main instructions (required)
-├── REFERENCE.md       # Detailed docs (if SKILL.md exceeds 100 lines)
-└── scripts/           # Utility scripts (if deterministic operations needed)
-```
-
-## SKILL.md Template
-
-```markdown
----
-name: oh-<name>
-description: "Brief description. Use when [specific triggers]."
-tier: <2|3|4>
-benefits-from: [<skill-dependencies>]
-triggers:
-  - "<trigger phrase>"
-  - "<another trigger>"
----
-
-# oh-<name>
-
-<one-paragraph summary>
-
-## When to Use
-
-<when to invoke this skill>
-
-## Workflow
-
-1. <step>
-2. <step>
-3. <step>
-
-## Anti-patterns
-
-- <anti-pattern 1>
-- <anti-pattern 2>
-```
-
-## Description Requirements
-
-The description is the only thing the agent sees when deciding which skill to load. Make it actionable:
-
-**Good:** "Create new agent skills with proper structure, frontmrmatter, and bundled resources. Use when user wants to create, write, or build a new skill."
-
-**Bad:** "Helps with skills."
-
-## Field Guide
-
-| Frontmatter Field | Required | Purpose |
-|---|---|---|
-| `name` | yes | Must match `^[a-z0-9]+(-[a-z0-9]+)*$` and directory name |
-| `description` | yes | Max 200 chars. First sentence = what it does. Second = when to use. |
-| `tier` | no | 2=tool, 3=strategic, 4=autonomous. Controls preamble verbosity. |
-| `benefits-from` | no | Skill dependencies. Listed skills should be loaded first. |
-| `triggers` | no | Natural language patterns that should route to this skill. |
-
-## When to Add Scripts
-- Operation is deterministic (validation, formatting)
-- Same code would be generated repeatedly
-- Errors need explicit handling
-
-Scripts save tokens and improve reliability vs generated code.
-
-## When to Split Files
-- SKILL.md exceeds 100 lines
-- Content has distinct domains
-- Advanced features are rarely used (put in REFERENCE.md)
-
-## Review Checklist
-
-- [ ] Description includes triggers ("Use when...")
-- [ ] SKILL.md under 100 lines
-- [ ] No time-sensitive info (dates, versions, deprecation warnings)
-- [ ] Consistent oh- prefix and terminology
-- [ ] Concrete examples included
-- [ ] Anti-patterns documented
-- [ ] Tests still pass after adding (`npm test`)
+1. Create skill directory — `harness/skills/<oh-name>/` with `SKILL.md`. Follow template structure (frontmatter, summary, Workflow, Anti-patterns).
+2. Write frontmatter — name (regex `^[a-z0-9]+(-[a-z0-9]+)*$`), description (max 200 chars, "Use when..."), tier (2/3/4), triggers, route.
+3. Draft skill body — When to Use, Workflow (numbered steps), Anti-patterns with concrete examples, Routing table.
+4. Review against checklist — description includes triggers, SKILL.md under 100 lines, no time-sensitive info, tests pass, consistent oh- prefix.
+5. Run eval-driven iteration — create test cases, spawn with-skill vs baseline sub-agents, grade pass rates/timing/tokens, improve and generalize.
+6. Optimize description — create 20 eval queries (10 should-trigger, 10 should-not), iterate description against eval set, select best precision/recall.
+7. Close loopholes — build rationalization table, create red flags, apply bulletproofing techniques (forbid specific workarounds).
 
 ## Routing
 
 | Outcome | Route |
 |---------|-------|
-| pass | → oh-skills-link (verify skill discovery) |
-| fail | → oh-expert (diagnose skill creation issues) |
-| blocker | → surface to user |
+| pass | → oh-skills-link (verify discovery) |
+| iteration data | → oh-learn (extract patterns) |
+| fail | → oh-expert (diagnose) |
+| blocker | → surface |

package/harness/skills/oh-skills-link/DEEP.md

@@ -0,0 +1,16 @@
+# oh-skills-link — Deep Reference
+
+## When to Use
+
+After installing or updating skills. Verify OpenCode discovers the package-local directory.
+
+## Anti-patterns
+
+- Linking without verifying files exist
+- Copying to global config during normal operation
+- Overwriting user-modified skills without intent
+- Linking broken/incomplete skills
+
+## Reference
+
+**Example:** After running npm update, run oh-skills-link. It reads harness/skills/, confirms config paths, reports any missing or new skills.

package/harness/skills/oh-skills-link/SKILL.md

@@ -1,29 +1,28 @@
 ---
 name: oh-skills-link
-description: "Verify that OpenCode can discover the package-local skills directory"
+description: "Use after installing or updating skills to verify OpenCode discovers the package-local skills directory."
+tier: 2
+route:
+  pass: surface
+  fail: oh-skill-craft
+  blocker: surface
 ---
 
 # oh-skills-link
 
-## When to Use
-After installing new skills or updating existing ones. Verifies that OpenCode can discover the package-local skills directory.
+Verify OpenCode discovers the package-local skills directory after install/update.
 
-## Workflow
-1. Read skills from `harness/skills/`
-2. Confirm `config.skills.paths` points at the package-local harness path
-3. Skip unchanged skills when checking manifests
-4. Log missing, invalid, or newly added skills
+## Steps
 
-## Anti-patterns
-- Linking skills without verifying they exist in harness
-- Copying skills into global config during normal operation
-- Overwriting user-modified skills without explicit intent
-- Linking broken or incomplete skills
+1. Read `harness/skills/` directory listing
+2. Confirm `config.skills.paths` points at harness path
+3. Skip skills that are unchanged
+4. Log missing, invalid, or newly added skills
 
 ## Routing
 
 | Outcome | Route |
 |---------|-------|
-| pass | → [report link status to user] |
-| fail | → oh-skill-craft (fix or rebuild broken skill) |
-| blocker | → surface to user |
+| pass | → surface |
+| fail | → oh-skill-craft |
+| blocker | → surface |

package/harness/skills/oh-skills-list/DEEP.md

@@ -0,0 +1,20 @@
+# oh-skills-list — Deep Reference
+
+## When to Use
+
+User wants to see available skills. Lists all oh-* skills with tier and description.
+
+## Anti-patterns
+
+- Filtering skills (show everything — let user decide)
+- Including non-OH skills in the output
+
+## Reference
+
+**Example:** User asks "what skills do you have?" → outputs a table of all oh-* skills with tier and purpose.
+
+### Output Format
+
+| Skill | Tier | Purpose |
+|-------|------|---------|
+| oh-<name> | 2/3/4 | <description> |

package/harness/skills/oh-skills-list/SKILL.md

@@ -1,31 +1,27 @@
 ---
 name: oh-skills-list
-description: "List all available oh-* skills with descriptions"
+description: "Use when the user wants to see available OH skills. Lists all oh-* skills with descriptions."
+tier: 2
+route:
+  pass: done
+  fail: surface
+  blocker: surface
 ---
 
 # oh-skills-list
 
-## When to Use
-To discover what skills are available. Lists every skill with its name, description, and category.
+List all available oh-* skills with tier and description.
 
-## Output
-Markdown table of skills:
+## Steps
 
-| Skill | Description | Category |
-|-------|-------------|----------|
-| oh-plan | Strategy + architecture review | Orchestration |
-| oh-qa | Full QA workflow | Quality |
-| ... | ... | ... |
-
-## Anti-patterns
-- Listing "all available" but missing recently installed skills
-- Showing skill file paths instead of human-readable descriptions
-- Not categorising skills (flat list is hard to scan)
+1. Gather all oh-* skills from the harness
+2. Format as a table: Skill | Tier | Purpose
+3. Output the table to the user
 
 ## Routing
 
 | Outcome | Route |
 |---------|-------|
-| pass | → [done — read-only report] |
-| fail | → [surface issue to user] |
-| blocker | → surface to user |
+| pass | → done |
+| fail | → surface |
+| blocker | → surface |

package/harness/skills/oh-triage/DEEP.md

@@ -0,0 +1,23 @@
+# oh-triage — Deep Reference
+
+## When to Use
+
+New issues or backlog review. Drives through triage state machine.
+
+## States
+
+1. Needs triage (new, unclassified)
+2. Needs info (waiting on reporter)
+3. Ready for agent (well-specified)
+4. Ready for human (needs judgment/access)
+5. Wontfix (declined with reason)
+
+## Anti-patterns
+
+- Issues stuck in "needs triage"
+- Triaging without reading full issue
+- Wontfix without explanation
+
+## Reference
+
+**Example:** New issues appear with needs-triage label. Read issue, classify as bug/feature/enhancement, assess severity, assign state (ready-for-agent / ready-for-human / needs-info).