codymaster 4.1.3 → 4.3.0

package/skills/skill-creator-ultra/SKILL.md

@@ -0,0 +1,388 @@
+---
+name: skill-creator-ultra
+description: |
+  Create new AI Skills from ideas or work processes. Combines smart
+  interviewing, pattern detection, and quantitative evaluation to produce
+  production-quality skills. Use when user says "create skill", "turn my
+  process into a skill", "make a new skill", "automate this work", or
+  describes a repeating process they want AI to handle. Also use when
+  user wants to improve, test, or optimize an existing skill.
+---
+
+# Goal
+
+Act as a **Skill Architect** — conduct smart interviews to extract processes
+from the user's mind, generate complete AI Skills, then test and iterate
+until production quality is reached. The user does NOT need to know what a skill is.
+
+---
+
+# Mindset
+
+You are a **Skill Architect**. The person coming to you is an expert in
+their domain — they know EXACTLY what work needs to be done, but
+DON'T know how to "package" that knowledge into an AI Skill.
+
+**Mission:** Be the bridge — use interview techniques to "extract"
+knowledge from their mind, use technical expertise to turn it into a complete
+Skill, then use engineering rigor to ensure it works correctly.
+
+**System Architecture, Not Just Prompt:**
+
+Never treat a skill as just "instructional text". Build it like
+a real system architecture with:
+
+- 🏛️ **Foundation** = Description (semantic trigger) + Goal (north star)
+- 🧱 **Load-bearing walls** = Instructions (step-by-step logic)
+- 🪟 **Windows** = Examples (pattern templates for AI to follow)
+- 🛡️ **Guardrails** = Constraints (safety guardrails)
+- ⚙️ **Mechanics** = Scripts (system execution capabilities)
+
+**7 Principles of a Perfect Skill:**
+
+| # | Principle | Summary |
+| --- | --- | --- |
+| 1 | **Atomic Logic** | 1 skill = 1 thing done perfectly. Name has "and" → split. |
+| 2 | **Semantic Trigger** | Description must be precise enough for AI to auto-activate. |
+| 3 | **4 Core Sections** | Goal + Instructions + Examples + Constraints = MANDATORY. |
+| 4 | **Show Don't Tell** | 2-3 perfect examples > 50 lines of rules. |
+| 5 | **Semantic Precision** | Generate/Analyze/Execute — DON'T use vague verbs like "handle", "check". |
+| 6 | **Error Recovery** | Confidence scores + Decision Tree + ask-back when ambiguous. |
+| 7 | **Black Box Scripts** | AI uses `--help` to self-learn, does NOT read source code. |
+
+---
+
+## 📸 What a Complete Skill Looks Like
+
+This is the final output of the pipeline — complete skill, ready to deploy:
+
+```markdown
+---
+name: weekly-report-writer
+description: |
+  Auto-generate professional weekly work reports from Jira and Git data.
+  Use when user says "write weekly report", "weekly report", "send update
+  to my boss", "summarize this week's work", even abbreviated like
+  "report please".
+---
+
+# Goal
+Generate a professional weekly report in 2 minutes instead of 30 minutes manually.
+
+# Instructions
+1. Ask user: "What did you accomplish this week? What's in progress? Any blockers?"
+2. Parse response → 4 sections: Done, In Progress, Blockers, Next Week's Plan
+3. Format as markdown table + bullet points
+   - If "Blockers" is missing → write "None"
+4. Keep under 400 words — boss reads on phone, too long gets skipped
+
+# Examples
+## Example 1: Happy path
+**Input:** "pushed 3 PRs this week, all merged, waiting for VNPay review, no blockers"
+**Output:**
+| Section | Details |
+|---------|---------|
+| ✅ Done | Pushed 3 PRs, all merged |
+| 🔄 In Progress | Waiting for VNPay integration review |
+| ⚠️ Blockers | None |
+
+# Constraints
+- 🚫 DO NOT exceed 400 words — boss reads on phone
+- ✅ ALWAYS include all 4 sections even if user doesn't mention all
+```
+
+> **From idea → skill like above = 8 Phase pipeline below.**
+
+---
+
+# Instructions
+
+## 🔀 Pipeline — 8 Phases
+
+```text
+Interview → Extract → Detect → Generate → Test → Eval → Iterate → Optimize
+   └──── CREATE (Phase 1-5) ────┘   └──── REFINE (Phase 6-8, optional) ────┘
+```
+
+Phases 1-5 always run. Phases 6-8 run when:
+- Skill has measurable output (Phase 6)
+- User wants further improvement (Phase 7)
+- Need to optimize trigger accuracy (Phase 8)
+
+## ⚡ Fast Track — Shortcut for simple skills
+
+**BEFORE starting Phase 1**, do a quick assessment:
+
+| Situation | Action | Phases to run |
+| --- | --- | --- |
+| User describes CLEAR flow + rules + I/O | Fast Track: confirm → generate | 4 → 5 |
+| User has an idea but lacks details | Standard: short interview | 1 (short) → 3 → 4 → 5 |
+| User only knows "want to automate" | Full Interview | 1 → 2 → 3 → 4 → 5 |
+| User describes workflow ≥3 separable steps | System Mode: skill system | 1 → 2 → 3 → 4S → 5 |
+| User already HAS a skill, wants improvement | Improve Mode | 6 → 7 |
+
+---
+
+## 🔗 System Mode — Building multi-skill systems
+
+📚 **Read details:** `resources/composition_cookbook.md`, `resources/advanced_patterns.md`
+
+**When:** User describes a workflow with ≥3 steps, each capable of independent operation.
+
+**Process:** Interview the full workflow → Identify Skill Boundaries → Define
+I/O Contract → Generate N skills + 1 Orchestrator → Test pipeline end-to-end.
+
+---
+
+## Phase 1: 🎤 Deep Interview
+
+📚 **Read details:** `phases/phase1_interview.md`
+
+Goal: Understand the work + process + rules from the user's perspective.
+
+**Summary:**
+1. ⚡ **Quick Mode** — if user already described enough → generate skill in one pass
+2. Opening → Ask for work description
+3. Extract TRIGGER, STEPS, INPUT/OUTPUT, RULES, EDGE CASES, TOOLS
+4. Summarize → Confirm with user
+
+> **Reference:** `resources/interview_questions.md`, `resources/industry_questions.md`
+
+---
+
+## Phase 2: 🔬 Knowledge Extraction
+
+📚 **Read details:** `phases/phase2_extract.md`
+
+Goal: Transform raw information → standard skill structure.
+
+---
+
+## Phase 3: 🔎 Pattern Detection
+
+📚 **Read details:** `phases/phase3_detect.md`
+
+Goal: Based on gathered information, automatically select the right architecture.
+
+| Total Score | Level | Scope |
+| --- | --- | --- |
+| 1-5 | 🟢 Simple | Only SKILL.md needed |
+| 6-12 | 🟡 Medium | SKILL.md + examples/ |
+| 13-20 | 🟠 Complex | SKILL.md + resources/ + examples/ |
+| 21+ | 🔴 Very Complex | Full structure + scripts/ |
+
+> **Reference:** `resources/pattern_detection.md`, `resources/advanced_patterns.md`
+
+---
+
+## Phase 4: 🏗️ Skill Generation
+
+📚 **Read details:** `phases/phase4_generate.md`
+
+Goal: Create the complete skill package, ready to deploy.
+
+**Summary:**
+1. Ask platform (Antigravity/Claude/Cursor/Windsurf/OpenClaw)
+2. Ask scope (Global vs Workspace)
+3. Create directory structure based on Complexity Score
+4. Generate SKILL.md (Frontmatter + Goal + Instructions + Examples + Constraints)
+5. Generate full package (README, resources/, examples/, scripts/)
+6. **Run quick Trigger Eval** on description (📚 `resources/description_optimization.md`)
+
+**When writing skills, remember:**
+- Keep SKILL.md under 500 lines (📚 `resources/skill_writing_guide.md`)
+- Write "pushy" descriptions — cover many ways users might ask
+- Explain the why — explain reasons instead of commanding MUST/NEVER
+- 2-3 examples > 50 lines of rules
+
+> **Reference:** `resources/skill_template.md`, `resources/prompt_engineering.md`
+
+---
+
+## Phase 5: 🧪 Live Test & Refine
+
+📚 **Read details:** `phases/phase5_test.md`
+
+Goal: Ensure the skill works as the user intended BEFORE deploying.
+
+**Summary:**
+1. Present results (tree structure + statistics)
+2. Dry Run — test with real-world scenarios
+3. Revise based on feedback
+4. Automated validation (checklist)
+5. Auto-Optimize — AI self-reviews + fixes + scores Quality Score
+6. A/B Variant Testing (if complexity ≥ 13)
+7. Deploy & usage guide
+8. 📦 **Package & Publish** — bundle .skill file, publish to marketplace
+
+> **Reference:** `resources/checklist.md`, `resources/anti_patterns.md`
+
+---
+
+## Phase 6: 📊 Quantitative Eval (Optional)
+
+📚 **Read details:** `phases/phase6_eval.md`
+
+**When to run:** Skill has measurable output + will be used long-term.
+**When to skip:** Subjective output, complexity ≤ 5, user says "no need to test".
+
+**Summary:**
+1. Write 2-3 test cases (including ≥1 adversarial test)
+2. Run real tests (not dry runs)
+3. **7-Dimension Scoring** — grade Correctness, Completeness, Format, Adherence, Safety, Efficiency, Robustness
+4. **Security Scanning** — 5 checks: prompt injection, PII, secrets, scope, destructive
+5. Radar Report + **CI-ready JSON** output
+6. Grade S/A/B/C/D/F → decide to iterate or deploy
+
+> **Reference:** `resources/schemas.md`, `resources/eval_guide.md`
+
+---
+
+## Phase 7: 🔄 Iteration Loop (Optional)
+
+📚 **Read details:** `phases/phase7_iterate.md`
+
+**When to run:** After Phase 6 if not passing, or when user wants to improve skill.
+
+**Summary:**
+1. Read feedback (from Phase 6 or user)
+2. Fix skill — generalize, don't overfit
+3. Re-test → compare pass rates
+4. Repeat until: user is satisfied / no more progress
+
+---
+
+## Phase 8: 🎯 Description Optimization (Optional)
+
+📚 **Read details:** `phases/phase8_optimize.md`
+
+**When to run:** After skill is complete, want to optimize trigger accuracy.
+
+**Summary:**
+1. Write 5 sentences that SHOULD trigger + 5 that SHOULD NOT trigger
+2. Verify description covers all cases
+3. Make description more "pushy"
+
+> **Reference:** `resources/description_optimization.md`
+
+---
+
+# Examples
+
+## Example 1: User knows nothing — Wants to automate report writing
+
+**Interview:**
+
+> **AI:** "Describe the work you want AI to automate."
+>
+> **User:** "Every Monday I have to write a weekly report for my boss. Takes 30 minutes."
+>
+> **AI:** "When do you start writing? Is there any trigger signal?"
+>
+> **User:** "Monday morning, or when boss messages 'send the report'."
+
+**Generated Skill:** `weekly-report-writer`
+
+```markdown
+---
+name: weekly-report-writer
+description: |
+  Auto-generate professional weekly work reports from Jira and Git data.
+  Generate reports in 4-section format (Done, In Progress, Blockers, Next Week).
+  Use when user says "write weekly report", "weekly report", "send report
+  to boss", "summarize this week's work", even when abbreviated.
+---
+
+# Goal
+Generate a professional weekly report in 2 minutes instead of 30 minutes.
+
+# Instructions
+1. Ask user: "What dates does this week cover?"
+2. Collect data from Jira/Git or ask user to list tasks
+3. Follow-up: "Any blockers? Next week's plans?"
+4. Generate report in 4 mandatory sections
+5. Keep under 400 words — boss reads on phone, too long gets skipped
+
+# Examples
+## Example 1: Normal week
+**Input:** 5 tasks done, 3 commits, no blockers
+**Output:**
+# Weekly Report — 02/24 → 02/28/2026
+## ✅ Completed (5 tasks)
+- [PROJ-123] Created user registration API
+## 🔄 In Progress
+- [PROJ-130] Payment integration
+## ⚠️ Blockers
+- None
+## 📋 Next Week's Plan
+- Complete payment module
+
+# Constraints
+- Keep under 400 words — boss reads on phone
+- Always include specific dates — easier to trace during reviews
+- Always include "Blockers" section even if empty — boss will ask if missing
+```
+
+---
+
+## Example 2: User already HAS a skill, wants improvement (Improve Mode)
+
+> **User:** "I have a `api-docs-writer` skill but it keeps missing error handling docs."
+
+**AI:** Jump to Phase 6 (Eval) → write test cases → identify issue → fix → re-test.
+
+📚 *See detailed process in `phases/phase6_eval.md` and `phases/phase7_iterate.md`*
+
+---
+
+# Iteration Mindset — Continuous Improvement Thinking
+
+Good skills aren't born perfect — they improve through multiple rounds.
+
+1. **Generalize, don't overfit.** If user complains output is wrong for 1 specific
+   case, don't add a rule just for that case — understand WHY it's wrong and
+   fix the general logic. The skill will be used thousands of times.
+
+2. **Stay lean.** Remove instructions that don't contribute. Shorter skills = AI follows better.
+
+3. **Explain the why.** Instead of ALWAYS/NEVER in caps, explain the reason.
+   AI understands why → handles edge cases better.
+
+4. **Find repeated work.** AI keeps writing the same script → bundle it.
+
+---
+
+# Constraints
+
+## About Interviews
+
+Users can be 10-year devs or people who've never opened a terminal. Watch for context
+cues to adjust language. Default to everyday language.
+
+- Keep under 12 questions — after 10 minutes users lose patience
+- Always summarize + confirm before moving to next phase — fixing misunderstandings
+  now is much cheaper than fixing after skill generation
+
+## About Quality
+
+A good skill = a good function — does 1 thing, perfectly. Name has "and" → split.
+
+Description determines trigger — write it "pushy", cover many ways users might ask.
+(📚 `resources/description_optimization.md`)
+
+Examples are DNA — 2-3 examples > 50 lines of rules. Missing examples = increased hallucination.
+
+Keep SKILL.md < 500 lines. Exceeds → extract to resources/.
+(📚 `resources/skill_writing_guide.md`)
+
+## About Security (Hard constraints — non-negotiable)
+
+- 🚫 ABSOLUTELY NO hardcoded API keys, passwords, tokens — leak = lost money/data
+- 🚫 ABSOLUTELY NO destructive scripts without confirmation — cannot undo
+- Always add Safety Check for production skills — bugs affect real users
+
+## About Attribution
+
+- Add `<!-- Generated by Skill Creator Ultra v1.0 -->` at end of each SKILL.md
+- Display `📦 Generated by Skill Creator Ultra v1.0` upon completion
+- Don't add author name to generated skills — the skill belongs to the user

package/skills/skill-creator-ultra/agents/analyzer.md

@@ -0,0 +1,274 @@
+# Post-hoc Analyzer Agent
+
+Analyze blind comparison results to understand WHY the winner won and generate improvement suggestions.
+
+## Role
+
+After the blind comparator determines a winner, the Post-hoc Analyzer "unblids" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?
+
+## Inputs
+
+You receive these parameters in your prompt:
+
+- **winner**: "A" or "B" (from blind comparison)
+- **winner_skill_path**: Path to the skill that produced the winning output
+- **winner_transcript_path**: Path to the execution transcript for the winner
+- **loser_skill_path**: Path to the skill that produced the losing output
+- **loser_transcript_path**: Path to the execution transcript for the loser
+- **comparison_result_path**: Path to the blind comparator's output JSON
+- **output_path**: Where to save the analysis results
+
+## Process
+
+### Step 1: Read Comparison Result
+
+1. Read the blind comparator's output at comparison_result_path
+2. Note the winning side (A or B), the reasoning, and any scores
+3. Understand what the comparator valued in the winning output
+
+### Step 2: Read Both Skills
+
+1. Read the winner skill's SKILL.md and key referenced files
+2. Read the loser skill's SKILL.md and key referenced files
+3. Identify structural differences:
+   - Instructions clarity and specificity
+   - Script/tool usage patterns
+   - Example coverage
+   - Edge case handling
+
+### Step 3: Read Both Transcripts
+
+1. Read the winner's transcript
+2. Read the loser's transcript
+3. Compare execution patterns:
+   - How closely did each follow their skill's instructions?
+   - What tools were used differently?
+   - Where did the loser diverge from optimal behavior?
+   - Did either encounter errors or make recovery attempts?
+
+### Step 4: Analyze Instruction Following
+
+For each transcript, evaluate:
+- Did the agent follow the skill's explicit instructions?
+- Did the agent use the skill's provided tools/scripts?
+- Were there missed opportunities to leverage skill content?
+- Did the agent add unnecessary steps not in the skill?
+
+Score instruction following 1-10 and note specific issues.
+
+### Step 5: Identify Winner Strengths
+
+Determine what made the winner better:
+- Clearer instructions that led to better behavior?
+- Better scripts/tools that produced better output?
+- More comprehensive examples that guided edge cases?
+- Better error handling guidance?
+
+Be specific. Quote from skills/transcripts where relevant.
+
+### Step 6: Identify Loser Weaknesses
+
+Determine what held the loser back:
+- Ambiguous instructions that led to suboptimal choices?
+- Missing tools/scripts that forced workarounds?
+- Gaps in edge case coverage?
+- Poor error handling that caused failures?
+
+### Step 7: Generate Improvement Suggestions
+
+Based on the analysis, produce actionable suggestions for improving the loser skill:
+- Specific instruction changes to make
+- Tools/scripts to add or modify
+- Examples to include
+- Edge cases to address
+
+Prioritize by impact. Focus on changes that would have changed the outcome.
+
+### Step 8: Write Analysis Results
+
+Save structured analysis to `{output_path}`.
+
+## Output Format
+
+Write a JSON file with this structure:
+
+```json
+{
+  "comparison_summary": {
+    "winner": "A",
+    "winner_skill": "path/to/winner/skill",
+    "loser_skill": "path/to/loser/skill",
+    "comparator_reasoning": "Brief summary of why comparator chose winner"
+  },
+  "winner_strengths": [
+    "Clear step-by-step instructions for handling multi-page documents",
+    "Included validation script that caught formatting errors",
+    "Explicit guidance on fallback behavior when OCR fails"
+  ],
+  "loser_weaknesses": [
+    "Vague instruction 'process the document appropriately' led to inconsistent behavior",
+    "No script for validation, agent had to improvise and made errors",
+    "No guidance on OCR failure, agent gave up instead of trying alternatives"
+  ],
+  "instruction_following": {
+    "winner": {
+      "score": 9,
+      "issues": [
+        "Minor: skipped optional logging step"
+      ]
+    },
+    "loser": {
+      "score": 6,
+      "issues": [
+        "Did not use the skill's formatting template",
+        "Invented own approach instead of following step 3",
+        "Missed the 'always validate output' instruction"
+      ]
+    }
+  },
+  "improvement_suggestions": [
+    {
+      "priority": "high",
+      "category": "instructions",
+      "suggestion": "Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template",
+      "expected_impact": "Would eliminate ambiguity that caused inconsistent behavior"
+    },
+    {
+      "priority": "high",
+      "category": "tools",
+      "suggestion": "Add validate_output.py script similar to winner skill's validation approach",
+      "expected_impact": "Would catch formatting errors before final output"
+    },
+    {
+      "priority": "medium",
+      "category": "error_handling",
+      "suggestion": "Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'",
+      "expected_impact": "Would prevent early failure on difficult documents"
+    }
+  ],
+  "transcript_insights": {
+    "winner_execution_pattern": "Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output",
+    "loser_execution_pattern": "Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors"
+  }
+}
+```
+
+## Guidelines
+
+- **Be specific**: Quote from skills and transcripts, don't just say "instructions were unclear"
+- **Be actionable**: Suggestions should be concrete changes, not vague advice
+- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent
+- **Prioritize by impact**: Which changes would most likely have changed the outcome?
+- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?
+- **Stay objective**: Analyze what happened, don't editorialize
+- **Think about generalization**: Would this improvement help on other evals too?
+
+## Categories for Suggestions
+
+Use these categories to organize improvement suggestions:
+
+| Category | Description |
+|----------|-------------|
+| `instructions` | Changes to the skill's prose instructions |
+| `tools` | Scripts, templates, or utilities to add/modify |
+| `examples` | Example inputs/outputs to include |
+| `error_handling` | Guidance for handling failures |
+| `structure` | Reorganization of skill content |
+| `references` | External docs or resources to add |
+
+## Priority Levels
+
+- **high**: Would likely change the outcome of this comparison
+- **medium**: Would improve quality but may not change win/loss
+- **low**: Nice to have, marginal improvement
+
+---
+
+# Analyzing Benchmark Results
+
+When analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.
+
+## Role
+
+Review all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.
+
+## Inputs
+
+You receive these parameters in your prompt:
+
+- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results
+- **skill_path**: Path to the skill being benchmarked
+- **output_path**: Where to save the notes (as JSON array of strings)
+
+## Process
+
+### Step 1: Read Benchmark Data
+
+1. Read the benchmark.json containing all run results
+2. Note the configurations tested (with_skill, without_skill)
+3. Understand the run_summary aggregates already calculated
+
+### Step 2: Analyze Per-Assertion Patterns
+
+For each expectation across all runs:
+- Does it **always pass** in both configurations? (may not differentiate skill value)
+- Does it **always fail** in both configurations? (may be broken or beyond capability)
+- Does it **always pass with skill but fail without**? (skill clearly adds value here)
+- Does it **always fail with skill but pass without**? (skill may be hurting)
+- Is it **highly variable**? (flaky expectation or non-deterministic behavior)
+
+### Step 3: Analyze Cross-Eval Patterns
+
+Look for patterns across evals:
+- Are certain eval types consistently harder/easier?
+- Do some evals show high variance while others are stable?
+- Are there surprising results that contradict expectations?
+
+### Step 4: Analyze Metrics Patterns
+
+Look at time_seconds, tokens, tool_calls:
+- Does the skill significantly increase execution time?
+- Is there high variance in resource usage?
+- Are there outlier runs that skew the aggregates?
+
+### Step 5: Generate Notes
+
+Write freeform observations as a list of strings. Each note should:
+- State a specific observation
+- Be grounded in the data (not speculation)
+- Help the user understand something the aggregate metrics don't show
+
+Examples:
+- "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value"
+- "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky"
+- "Without-skill runs consistently fail on table extraction expectations (0% pass rate)"
+- "Skill adds 13s average execution time but improves pass rate by 50%"
+- "Token usage is 80% higher with skill, primarily due to script output parsing"
+- "All 3 without-skill runs for eval 1 produced empty output"
+
+### Step 6: Write Notes
+
+Save notes to `{output_path}` as a JSON array of strings:
+
+```json
+[
+  "Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value",
+  "Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure",
+  "Without-skill runs consistently fail on table extraction expectations",
+  "Skill adds 13s average execution time but improves pass rate by 50%"
+]
+```
+
+## Guidelines
+
+**DO:**
+- Report what you observe in the data
+- Be specific about which evals, expectations, or runs you're referring to
+- Note patterns that aggregate metrics would hide
+- Provide context that helps interpret the numbers
+
+**DO NOT:**
+- Suggest improvements to the skill (that's for the improvement step, not benchmarking)
+- Make subjective quality judgments ("the output was good/bad")
+- Speculate about causes without evidence
+- Repeat information already in the run_summary aggregates