@dzhechkov/skills-bto 1.0.0

package/templates/.claude/skills/bto/references/eval-patterns.md

@@ -0,0 +1,202 @@
+# Multi-Agent Evaluation Patterns
+
+## Overview
+
+Research-backed patterns for using multiple LLM agents to evaluate content. Each pattern trades off cost, reliability, and depth.
+
+---
+
+## Pattern 1: Judge Panel (Used by BTO)
+
+**Architecture:** N independent judges evaluate the same artifact in parallel.
+
+```
+Artifact ──┬──→ Judge 1 (Expert)     ──┐
+           ├──→ Judge 2 (Critic)      ──┼──→ Aggregator → Report
+           └──→ Judge 3 (Auditor)     ──┘
+```
+
+**Strengths:**
+- Multi-perspective coverage
+- Parallel execution (fast)
+- Disagreement detection reveals ambiguous quality
+
+**Weaknesses:**
+- Judges may converge to similar opinions (groupthink)
+- More expensive than single judge
+- Requires aggregation logic
+
+**When to use:** Standard evaluation of well-defined artifacts. BTO default.
+
+**Mitigation for groupthink:**
+- Give each judge a distinct role and perspective
+- Critic judge explicitly instructed to be strict (calibrated lower)
+- Different temperature settings if available
+
+---
+
+## Pattern 2: Adversarial Red/Blue
+
+**Architecture:** One agent creates, another attacks.
+
+```
+Artifact ──→ Blue Team (Defender)  ──→ Assessment
+         ──→ Red Team (Attacker)   ──→ Vulnerabilities
+                                   ──→ Reconciliation
+```
+
+**Strengths:**
+- Excellent at finding weaknesses
+- Simulates real-world usage patterns
+- Defender forced to justify every choice
+
+**Weaknesses:**
+- Can be overly negative
+- Expensive (multiple rounds)
+- Not suited for overall quality scoring
+
+**When to use:** Security-sensitive artifacts, high-stakes decisions, or when robustness is critical.
+
+---
+
+## Pattern 3: Consensus-Based (Multi-Round Debate)
+
+**Architecture:** Agents debate until convergence.
+
+```
+Round 1: Each judge evaluates independently
+Round 2: Judges see each other's evaluations, revise
+Round 3: Final convergence (or flag for human)
+```
+
+**Strengths:**
+- High-confidence final scores
+- Reduces individual judge bias
+- Self-correcting
+
+**Weaknesses:**
+- Very expensive (3x the calls)
+- Risk of conformity pressure
+- Slow (sequential rounds)
+
+**When to use:** Critical decisions where confidence matters more than speed.
+
+---
+
+## Pattern 4: Constitutional Evaluation
+
+**Architecture:** Evaluate against a set of explicit principles.
+
+```
+Principles ──→ Evaluator ──→ Per-principle score ──→ Report
+```
+
+**Principles example for Claude Code artifacts:**
+1. Instructions must be unambiguous
+2. Every section must serve a purpose
+3. Anti-patterns must be documented
+4. Examples must demonstrate the happy path
+5. Failure modes must be handled
+
+**Strengths:**
+- Deterministic criteria
+- Easy to explain scores
+- Reproducible
+
+**Weaknesses:**
+- Misses emergent quality issues
+- Principles may not cover everything
+- Rigid
+
+**When to use:** Compliance checking, standardized quality gates.
+
+---
+
+## Pattern 5: Hierarchical Evaluation
+
+**Architecture:** Fast cheap filter → detailed expensive evaluation.
+
+```
+Layer 0: Deterministic ──pass──→ Layer 1: Haiku ──pass──→ Layer 2: Sonnet Panel
+             │                        │
+             fail                     fail
+             ↓                        ↓
+         Quick Fix               Detailed Feedback
+```
+
+**Strengths:**
+- Cost-efficient (most artifacts filtered early)
+- Progressive detail
+- Fast feedback for obvious issues
+
+**Weaknesses:**
+- Layer 0 can miss semantic issues
+- Layer 1 may have different calibration than Layer 2
+
+**When to use:** Batch evaluation, CI/CD pipelines, optimization loops. BTO uses this pattern.
+
+---
+
+## Aggregation Strategies
+
+### 1. Weighted Average (BTO Default)
+```
+score = Σ(weight_i × judge_i_score) / Σ(weight_i)
+```
+- Simple, interpretable
+- Weights reflect judge importance
+- BTO weights: Expert 0.4, Critic 0.3, Auditor 0.3
+
+### 2. Majority Voting
+```
+verdict = mode(judge_verdicts)
+```
+- Good for pass/fail decisions
+- Requires odd number of judges
+- Ignores score magnitude
+
+### 3. Bayesian Aggregation
+```
+P(quality | scores) ∝ Π P(score_i | quality) × P(quality)
+```
+- Accounts for judge reliability
+- Requires calibration data
+- More complex to implement
+
+### 4. Min-Score (Conservative)
+```
+score = min(all_judge_scores)
+```
+- Most conservative
+- Good for safety-critical artifacts
+- May be overly pessimistic
+
+### 5. Trimmed Mean
+```
+score = mean(scores after removing highest and lowest)
+```
+- Outlier-resistant
+- Requires ≥5 judges
+- Good for large panels
+
+---
+
+## Anti-Conformity Measures
+
+Prevent judges from converging to meaningless agreement:
+
+1. **Role Differentiation** — Each judge has unique focus and scoring calibration
+2. **Critic Calibration** — Critic judge instructed to score ~5-6 average (strict)
+3. **Independent Evaluation** — No judge sees other judges' scores
+4. **Disagreement as Signal** — High disagreement flags important quality dimensions
+5. **Diverse Prompts** — Each judge gets a differently-framed evaluation prompt
+
+## Cost Optimization
+
+| Approach | Judges | Model | Est. Cost | Use When |
+|----------|--------|-------|-----------|----------|
+| Layer 0 only | 0 | none | Free | CI/CD pre-check |
+| Layer 1 | 1 | haiku | ~$0.001 | Quick iteration |
+| Layer 2 | 3 | sonnet | ~$0.01 | Thorough evaluation |
+| Full + Meta | 3+1 | sonnet+opus | ~$0.05 | Critical artifacts |
+| Consensus | 3×3 | sonnet | ~$0.03 | High-stakes decisions |

package/templates/.claude/skills/bto/references/judge-rubrics.md

@@ -0,0 +1,183 @@
+# Judge Rubrics — Evaluation Criteria by Artifact Type
+
+## Universal Dimensions (1-10 scale)
+
+All artifact types are evaluated on these 5 dimensions:
+
+| Dimension | What it Measures | 1-3 (Poor) | 4-6 (Adequate) | 7-8 (Good) | 9-10 (Excellent) |
+|-----------|-----------------|------------|----------------|------------|-------------------|
+| METHODOLOGY | Approach soundness | No clear structure | Basic structure present | Well-designed protocol | Rigorous, innovative approach |
+| DEPTH | Content thoroughness | Superficial | Covers basics | Thorough treatment | Comprehensive with nuances |
+| CORRECTNESS | Accuracy of claims/instructions | Major errors | Minor inaccuracies | Accurate | Precise and verifiable |
+| USABILITY | Ease of use | Confusing | Usable with effort | Clear and navigable | Intuitive, self-documenting |
+| ROBUSTNESS | Edge case handling | No coverage | Some mentioned | Well-covered | Exhaustive with fallbacks |
+
+---
+
+## Skill Rubric
+
+### METHODOLOGY
+- 9-10: Clear multi-step protocol, modular design, explicit decision points
+- 7-8: Good protocol structure, some modularity
+- 4-6: Basic steps listed, no clear decision flow
+- 1-3: No protocol, just description of what skill does
+
+### DEPTH
+- 9-10: Detailed modules, rich references, multiple examples, anti-patterns
+- 7-8: Good detail in SKILL.md, has references and examples
+- 4-6: Basic overview, minimal references
+- 1-3: Stub-level content, no supporting files
+
+### CORRECTNESS
+- 9-10: All instructions produce expected output, cross-references valid
+- 7-8: Instructions work, minor reference issues
+- 4-6: Some instructions ambiguous, broken references
+- 1-3: Instructions would produce wrong output
+
+### USABILITY
+- 9-10: Quick start section, clear navigation, progressive disclosure
+- 7-8: Well-organized, findable information
+- 4-6: Readable but poorly organized
+- 1-3: Wall of text, no structure
+
+### ROBUSTNESS
+- 9-10: Anti-patterns documented, failure modes handled, abort conditions defined
+- 7-8: Anti-patterns section present, some edge cases
+- 4-6: Mentions some risks
+- 1-3: No failure mode coverage
+
+---
+
+## Command Rubric
+
+### METHODOLOGY
+- 9-10: Clear step-by-step flow, proper checkpoints, skill loading pattern
+- 7-8: Good flow, checkpoint present
+- 4-6: Basic steps, missing checkpoint
+- 1-3: No clear execution flow
+
+### DEPTH
+- 9-10: Handles all parameter combinations, parallel agent usage
+- 7-8: Good parameter handling, some agent usage
+- 4-6: Basic parameter handling
+- 1-3: No parameter handling
+
+### CORRECTNESS
+- 9-10: $ARGUMENTS properly used, skill paths correct, agent configs valid
+- 7-8: Mostly correct references
+- 4-6: Some incorrect references
+- 1-3: Wrong skill paths or broken references
+
+### USABILITY
+- 9-10: Clear usage line, example invocations, helpful error messages
+- 7-8: Usage documented, some examples
+- 4-6: Basic usage, no examples
+- 1-3: Unclear how to invoke
+
+### ROBUSTNESS
+- 9-10: Input validation, fallbacks, timeout handling
+- 7-8: Some input validation
+- 4-6: Minimal validation
+- 1-3: No error handling
+
+---
+
+## Research Artifact Rubric
+
+### METHODOLOGY
+- 9-10: PARANOID mode, systematic research plan, structured sections
+- 7-8: Good research structure, mostly verified
+- 4-6: Basic structure, some unverified claims
+- 1-3: Unstructured, many unverified claims
+
+### DEPTH
+- 9-10: Multiple sources per claim, competitive analysis, quantitative data
+- 7-8: Good source coverage, some quantitative data
+- 4-6: Few sources, mostly qualitative
+- 1-3: No sources, opinion-based
+
+### CORRECTNESS
+- 9-10: All claims sourced, no [UNVERIFIED] tags, real products/companies
+- 7-8: Mostly sourced, few [UNVERIFIED]
+- 4-6: Some sourced, several [UNVERIFIED]
+- 1-3: Unsourced, potentially hallucinated
+
+### USABILITY
+- 9-10: Executive summary, clear sections, actionable takeaways
+- 7-8: Well-organized, some actionable content
+- 4-6: Readable but hard to act on
+- 1-3: Dense, unstructured
+
+### ROBUSTNESS
+- 9-10: Limitations documented, alternative views considered, confidence levels
+- 7-8: Some limitations noted
+- 4-6: Minimal caveats
+- 1-3: No limitations or alternative views
+
+---
+
+## Rule Rubric
+
+### METHODOLOGY
+- 9-10: Clear pattern/signal/fix structure, auto-detection instructions
+- 7-8: Good table structure, detection guidance
+- 4-6: List format, basic detection
+- 1-3: Unstructured prose
+
+### DEPTH
+- 9-10: 8+ patterns, each with specific signals and actionable fixes
+- 7-8: 5-7 patterns, good specificity
+- 4-6: 3-4 patterns, generic
+- 1-3: 1-2 patterns, vague
+
+### CORRECTNESS
+- 9-10: All patterns are real, signals are detectable, fixes work
+- 7-8: Mostly accurate patterns
+- 4-6: Some questionable patterns
+- 1-3: Patterns don't match reality
+
+### USABILITY
+- 9-10: Scannable table format, clear action items
+- 7-8: Table format, understandable
+- 4-6: List format, requires interpretation
+- 1-3: Hard to use as reference
+
+### ROBUSTNESS
+- 9-10: Covers common AND edge case patterns, priority ordering
+- 7-8: Good pattern coverage
+- 4-6: Only obvious patterns
+- 1-3: Misses critical patterns
+
+---
+
+## Agent Template Rubric
+
+### METHODOLOGY
+- 9-10: Clear isolation, model selection, integration protocol
+- 7-8: Good agent config, clear purpose
+- 4-6: Basic config, unclear integration
+- 1-3: No clear agent architecture
+
+### DEPTH
+- 9-10: Detailed prompt, context specification, output format
+- 7-8: Good prompt, some output guidance
+- 4-6: Basic prompt only
+- 1-3: Stub template
+
+### CORRECTNESS
+- 9-10: Model choice justified, isolation enforced, no conflict with other agents
+- 7-8: Reasonable model choice, basic isolation
+- 4-6: Questionable model choice
+- 1-3: Wrong model or no isolation
+
+### USABILITY
+- 9-10: Ready to use with Agent tool, clear parameters
+- 7-8: Mostly ready, minor adjustments needed
+- 4-6: Needs significant adaptation
+- 1-3: Not usable as-is
+
+### ROBUSTNESS
+- 9-10: Timeout handling, failure protocol, cost bounds
+- 7-8: Some failure handling
+- 4-6: No failure handling but stable
+- 1-3: Could cause issues (infinite loops, cost overrun)

package/templates/.claude/skills/bto/references/optimization-methods.md

@@ -0,0 +1,139 @@
+# Prompt Optimization Methods
+
+## Overview
+
+Survey of automated prompt optimization approaches. BTO uses an EvoPrompt-inspired evolutionary approach adapted for Claude Code artifacts.
+
+---
+
+## Method 1: APE (Automatic Prompt Engineer)
+
+**Paper:** Zhou et al., 2022 — "Large Language Models Are Human-Level Prompt Engineers"
+
+**Approach:**
+1. Generate candidate prompts from task description + examples
+2. Score each candidate on a held-out evaluation set
+3. Select the best-performing prompt
+
+**Strengths:** Simple, effective for single-prompt optimization
+**Weaknesses:** Requires evaluation dataset, single-round (no iteration)
+**BTO Relevance:** Inspiration for variant generation step
+
+---
+
+## Method 2: OPRO (Optimization by Prompting)
+
+**Paper:** Yang et al., 2023 — "Large Language Models as Optimizers"
+
+**Approach:**
+1. Maintain a trajectory of (prompt, score) pairs
+2. Ask LLM to generate better prompt given the trajectory
+3. Evaluate and add to trajectory
+4. Repeat
+
+**Key Insight:** LLMs can optimize when shown optimization trajectory.
+
+**Strengths:** Leverages LLM understanding of what makes prompts good
+**Weaknesses:** Can get stuck in local optima, expensive
+**BTO Relevance:** Trajectory concept used in crossover step
+
+---
+
+## Method 3: EvoPrompt (Evolutionary)
+
+**Paper:** Guo et al., 2023 — "Connecting Large Language Models with Evolutionary Algorithms"
+
+**Approach:**
+1. Initialize population of N prompts
+2. Apply genetic operators: mutation + crossover
+3. Evaluate fitness (task performance)
+4. Select top-K, repeat
+
+**Genetic Operators:**
+- **Mutation:** Rephrase, add/remove constraints, change structure
+- **Crossover:** Combine best parts of two prompts
+
+**Strengths:** Explores diverse solutions, avoids local optima
+**Weaknesses:** Expensive (many evaluations), slow convergence
+**BTO Relevance:** Primary inspiration for OPTIMIZE module
+
+### BTO Adaptation of EvoPrompt
+
+```
+EvoPrompt (Original)           BTO OPTIMIZE (Adapted)
+─────────────────              ─────────────────────
+Random init population    →    5 targeted mutations (strategy-driven)
+Generic mutation          →    Named strategies (Rephrase/Restructure/etc.)
+Random crossover          →    Strength-based crossover (section-level)
+Task accuracy metric      →    Multi-dimensional judge panel scores
+Many generations          →    3 rounds (cost-bounded)
+```
+
+---
+
+## Method 4: TextGrad (Textual Gradients)
+
+**Paper:** Yuksekgonul et al., 2024 — "TextGrad: Automatic Differentiation via Text"
+
+**Approach:**
+1. Evaluate prompt output quality
+2. Generate "textual gradient" — natural language feedback on what to improve
+3. Apply gradient as edit instructions
+4. Repeat
+
+**Strengths:** Targeted improvements, interpretable changes
+**Weaknesses:** Requires clear evaluation criteria, can overfit
+**BTO Relevance:** Judge feedback as textual gradients for mutation targeting
+
+---
+
+## Method 5: DSPy (Programmatic)
+
+**Framework:** Khattab et al., 2023 — "DSPy: Compiling Declarative Language Model Calls"
+
+**Approach:**
+1. Define modules with typed signatures
+2. Compile with optimizer (BootstrapFewShot, MIPRO, etc.)
+3. Optimizer tunes prompts + few-shot examples
+4. Evaluate on metric
+
+**Strengths:** Systematic, reproducible, handles multi-step pipelines
+**Weaknesses:** Requires code framework, Python-specific
+**BTO Relevance:** Modular skill design parallels DSPy module concept
+
+---
+
+## Comparison Matrix
+
+| Method | Iterations | Cost | Diversity | Interpretability | Best For |
+|--------|-----------|------|-----------|-----------------|----------|
+| APE | 1 | Low | Medium | High | Simple prompts |
+| OPRO | 5-20 | High | Low | High | Single metric |
+| EvoPrompt | 5-50 | Very High | High | Medium | Complex prompts |
+| TextGrad | 3-10 | Medium | Low | Very High | Targeted fixes |
+| DSPy | Auto | Variable | Medium | Low | Pipelines |
+| **BTO** | **3** | **Medium** | **High** | **High** | **Claude Code artifacts** |
+
+---
+
+## BTO Design Decisions
+
+### Why Evolutionary (not gradient)?
+- Claude Code artifacts are structured documents, not single prompts
+- Mutations at section level are more meaningful than token-level edits
+- Diversity prevents converging to one style
+
+### Why 3 rounds (not more)?
+- Diminishing returns after round 3
+- Cost budget: ~15 evaluations is practical
+- Layer 2 on final round ensures quality
+
+### Why strategy-driven mutations (not random)?
+- Named strategies map to specific weaknesses
+- More interpretable changelog
+- Faster convergence than random exploration
+
+### Why section-level crossover?
+- Skills have clear section boundaries
+- Strengths are often section-specific
+- Easier to preserve coherence than token-level mixing