@elizaos/skills 2.0.0-alpha.3

package/skills/testing-handbook-skills/skills/testing-handbook-generator/discovery.md

@@ -0,0 +1,452 @@
+# Discovery Workflow
+
+Methodology for analyzing the Testing Handbook and identifying skill candidates.
+
+**Quick Navigation:**
+- [Phase 0: Locate Handbook](#phase-0-locate-handbook)
+- [Phase 1: Handbook Analysis](#phase-1-handbook-analysis)
+- [Phase 2: Plan Generation](#phase-2-plan-generation)
+- [Phase 3: Prepare Generation Context](#phase-3-prepare-generation-context)
+
+## Progress Tracking
+
+Use TodoWrite throughout discovery to track progress and give visibility to the user:
+
+```
+Discovery phase todos:
+- [ ] Locate handbook repository
+- [ ] Scan /fuzzing/ sections
+- [ ] Scan /static-analysis/ sections
+- [ ] Scan /crypto/ sections
+- [ ] Scan /web/ sections
+- [ ] Build candidate list with types
+- [ ] Resolve conflicts and edge cases
+- [ ] Present plan to user
+- [ ] Await user approval
+```
+
+Mark each todo as `in_progress` when starting and `completed` when done. This helps users understand where you are in the process.
+
+## Phase 0: Locate Handbook
+
+Before analysis, locate or obtain the Testing Handbook repository.
+
+### Step 1: Check Common Locations
+
+```bash
+# Check common locations (simple version)
+for dir in ./testing-handbook ../testing-handbook ~/testing-handbook; do
+  [ -d "$dir/content/docs" ] && handbook_path="$dir" && echo "✓ Found: $dir" && break
+done
+[ -z "$handbook_path" ] && echo "Handbook not found in common locations"
+```
+
+### Step 2: Ask User (if not found)
+
+If handbook not found in common locations:
+> "Where is the Testing Handbook repository located? (full path)"
+
+### Step 3: Clone as Last Resort (if user agrees)
+
+```bash
+git clone --depth=1 https://github.com/trailofbits/testing-handbook.git
+handbook_path="./testing-handbook"
+```
+
+### Set handbook_path Variable
+
+Once located, set `handbook_path` and use it for all subsequent paths:
+```bash
+handbook_path="/path/to/testing-handbook"  # Set to actual location
+```
+
+All paths below are relative to `{handbook_path}`.
+
+### Error Recovery
+
+| Failure | Detection | Recovery |
+|---------|-----------|----------|
+| Handbook not in common locations | Step 1 finds nothing | Ask user for path (Step 2) |
+| User doesn't know path | User says "I don't know" | Offer to clone (Step 3) |
+| Clone fails (network/permissions) | git clone returns error | Report error, ask user to clone manually and provide path |
+| `content/docs/` missing | Directory doesn't exist after locating | Invalid handbook - ask user to verify it's the correct repo |
+| Handbook is outdated | User mentions old content | Suggest `git pull` in handbook directory |
+
+## Phase 1: Handbook Analysis
+
+### 1.1 Scan Directory Structure
+
+Scan the handbook at:
+```
+{handbook_path}/content/docs/
+```
+
+Directory structure pattern:
+```
+docs/
+├── fuzzing/
+│   ├── _index.md              # Section overview
+│   ├── c-cpp/
+│   │   ├── _index.md          # Language subsection
+│   │   ├── 10-libfuzzer/      # Tool directory
+│   │   │   └── index.md
+│   │   ├── 11-aflpp/
+│   │   └── techniques/        # Shared techniques
+│   └── rust/
+├── static-analysis/
+│   ├── semgrep/
+│   │   ├── _index.md
+│   │   ├── 00-installation.md
+│   │   ├── 10-advanced.md
+│   │   ├── 20-ci.md
+│   │   └── 99-resources.md
+│   └── codeql/
+├── crypto/
+│   ├── wycheproof/
+│   └── constant_time_tool/
+└── web/
+    └── burp/
+```
+
+### 1.2 Parse Frontmatter
+
+Each markdown file has YAML frontmatter:
+
+```yaml
+---
+title: "Semgrep"
+weight: 2
+summary: "Fast static analysis for finding bugs..."
+bookCollapseSection: true
+draft: false                    # Check this!
+---
+```
+
+**Key fields:**
+| Field | Purpose |
+|-------|---------|
+| `title` | Skill name candidate |
+| `summary` | Skill description source |
+| `weight` | Ordering (lower = more important) |
+| `bookCollapseSection` | Indicates major section |
+| `draft` | If `true`, skip this section |
+
+### 1.3 Identify Skill Candidates
+
+**Decision Table:**
+
+For each directory found, apply the first matching rule:
+
+| Directory Pattern | Has | Skill Type | Action |
+|-------------------|-----|------------|--------|
+| `_index.md` with `bookCollapseSection: true` | - | Container | Scan children, don't create skill for container itself |
+| `**/static-analysis/[name]/` | Numbered files (00-, 10-) | Tool | Create tool skill |
+| `**/fuzzing/[lang]/[name]/` | `index.md` or numbered files | Fuzzer | Create fuzzer skill |
+| `**/fuzzing/techniques/[name]/` | Any `.md` files | Technique | Create technique skill |
+| `**/crypto/[name]/` | Any `.md` files | Domain | Create domain skill |
+| `**/web/[name]/` | Numbered files or `_index.md` | Tool | Create tool skill (but check exclusions) |
+| Named `techniques/` | Subdirectories | Container | Create one technique skill per subdirectory |
+| Any other | Only `_index.md` | Skip | Not enough content |
+
+**Hard Exclusions (GUI-only tools):**
+
+Some handbook sections describe tools that require graphical user interfaces and cannot be operated by Claude. Skip these unconditionally:
+
+| Section | Tool | Reason |
+|---------|------|--------|
+| `**/web/burp/` | Burp Suite | GUI-based HTTP proxy; requires visual interaction |
+
+These tools are excluded because Claude cannot:
+- Launch or interact with GUI applications
+- Click buttons, navigate menus, or view visual elements
+- Operate browser-based or desktop UI tools
+
+For web security testing, prefer CLI-based alternatives documented elsewhere (e.g., `curl`, `httpie`, custom scripts).
+
+**Classification priority** (when multiple patterns match):
+1. Most specific path wins (deeper = higher priority)
+2. Type preference: Tool > Fuzzer > Technique > Domain
+3. When in doubt, flag for user review in the plan
+
+**Examples:**
+
+| Path | Matches | Result |
+|------|---------|--------|
+| `/fuzzing/c-cpp/10-libfuzzer/` | Fuzzer pattern | `libfuzzer` (fuzzer) |
+| `/static-analysis/semgrep/` | Tool pattern | `semgrep` (tool) |
+| `/fuzzing/techniques/writing-harnesses/` | Technique pattern | `harness-writing` (technique) |
+| `/crypto/wycheproof/` | Domain pattern | `wycheproof` (domain) |
+
+### 1.4 Build Candidate List
+
+For each candidate, extract:
+
+```yaml
+- name: libfuzzer
+  type: fuzzer
+  source: /docs/fuzzing/c-cpp/10-libfuzzer/
+  summary: "libFuzzer is a coverage-guided fuzzer..."
+  weight: 1
+  has_resources: true           # Has 99-resources.md
+  related_sections:
+    - /docs/fuzzing/techniques/01-writing-harnesses/
+    - /docs/fuzzing/techniques/03-asan/
+```
+
+### 1.5 Candidate Prioritization
+
+When many candidates exist, prioritize by:
+
+| Priority | Criterion | Rationale |
+|----------|-----------|-----------|
+| 1 | Weight field (lower = higher) | Handbook author's intent |
+| 2 | Content depth (more numbered files) | More complete documentation |
+| 3 | Has resources file | External links add value |
+| 4 | Core section (fuzzing, static-analysis) | Fundamental topics |
+| 5 | Recently updated | More relevant content |
+
+**Conflict resolution:**
+- If multiple patterns match a section, use the **most specific** (deepest path wins)
+- If a section could be multiple types, prefer: Tool > Fuzzer > Technique > Domain
+- When in doubt, flag for user review in the plan
+
+## Phase 2: Plan Generation
+
+### 2.1 Plan Format
+
+Present to user in this format:
+
+```markdown
+# Skill Generation Plan
+
+## Summary
+- **Handbook analyzed:** {handbook_path}
+- **Total sections:** 25
+- **Skills to generate:** 8
+- **Sections to skip:** 2
+
+---
+
+## Skills to Generate
+
+| # | Skill Name | Source Section | Type | Related Sections |
+|---|------------|----------------|------|------------------|
+| 1 | libfuzzer | /fuzzing/c-cpp/10-libfuzzer/ | Fuzzer | techniques/asan, techniques/harness |
+| 2 | aflpp | /fuzzing/c-cpp/11-aflpp/ | Fuzzer | techniques/asan |
+| 3 | cargo-fuzz | /fuzzing/rust/10-cargo-fuzz/ | Fuzzer | - |
+| 4 | wycheproof | /crypto/wycheproof/ | Domain | - |
+| 5 | fuzz-harness-writing | /fuzzing/techniques/01-writing-harnesses/ | Technique | - |
+| 6 | coverage-analysis | /fuzzing/c-cpp/techniques/01-coverage/ | Technique | - |
+| 7 | address-sanitizer | /fuzzing/techniques/03-asan/ | Technique | - |
+
+---
+
+## Skipped Sections
+
+| Section | Reason |
+|---------|--------|
+| /docs/dynamic-analysis/ | `draft: true` in frontmatter |
+| /docs/fuzzing/3-python.md | Single file, insufficient content |
+| /docs/web/burp/ | GUI-only tool (excluded) |
+
+---
+
+## External Resources to Fetch
+
+| Section | Resource Count | Source File |
+|---------|---------------|-------------|
+| Fuzzing | 5 | /fuzzing/91-resources.md |
+| Semgrep | 3 | /static-analysis/semgrep/99-resources.md |
+| CodeQL | 4 | /static-analysis/codeql/99-resources.md |
+
+---
+
+## Actions
+
+- [ ] Confirm plan and proceed with generation
+- [ ] Modify: Remove skill #X from plan
+- [ ] Modify: Change skill #Y type
+- [ ] Cancel generation
+```
+
+Make the actions navigable and selectable if possible using built-in tool like TodoWrite. 
+
+### 2.2 User Interaction
+
+After presenting plan:
+
+1. Wait for user confirmation or modifications
+2. Apply any modifications to plan
+3. Proceed with generation only after explicit approval
+
+**Acceptable modifications:**
+- Remove skills from plan
+- Change skill type
+- Skip updates
+- Add custom related sections
+- Change skill names
+
+## Phase 3: Prepare Generation Context
+
+This phase prepares everything needed for generation agents. It combines content aggregation with agent handoff preparation.
+
+### 3.1 Collect Content Per Skill
+
+For each approved skill, collect content from:
+
+1. **Primary section:** Main `_index.md` or `index.md`
+2. **Numbered files:** `00-installation.md`, `10-advanced.md`, etc.
+3. **Related sections:** As specified in `related_sections`
+4. **Resources:** From `99-resources.md` (titles only)
+
+### 3.2 Content Processing Rules
+
+Content processing rules are defined in the agent prompt template.
+
+**Authoritative source:** [agent-prompt.md](agent-prompt.md#critical-rules)
+
+The agent prompt contains:
+- Hugo shortcode conversion table
+- Code block preservation rules
+- Image/video handling
+- YAML frontmatter requirements
+- Pre-write validation checklist
+- **Line count splitting rules** (when to split large skills)
+
+Do not duplicate these rules here. Generation agents receive them via the prompt template.
+
+### 3.3 External Resources
+
+Extract from `99-resources.md` or `91-resources.md`:
+
+```markdown
+## Resources
+
+- [Introduction to Semgrep](https://www.youtube.com/watch?v=...) - Trail of Bits Webinar
+- [Semgrep Documentation](https://semgrep.dev/docs/) - Official docs
+- [Custom Rules Guide](https://semgrep.dev/docs/writing-rules/) - Rule authoring
+```
+
+**For non-video resources (documentation, blogs, guides):**
+- Use WebFetch to retrieve content
+- Extract key insights, techniques, code examples
+- Summarize actionable information for the skill
+- Include attribution with URL
+
+**For video resources (YouTube, Vimeo, etc.):**
+- Extract title and URL only
+- Do NOT attempt to fetch video content
+- Include brief description if available in handbook
+
+**Video URL patterns to skip fetching:**
+- `youtube.com`, `youtu.be`
+- `vimeo.com`
+- `*.mp4`, `*.webm`, `*.mov` direct links
+- `twitch.tv`, `dailymotion.com`
+
+### 3.4 Edge Case Handling
+
+| Situation | Detection | Action |
+|-----------|-----------|--------|
+| Empty handbook section | Directory exists but no `.md` files | Skip, add to "Skipped Sections" with reason |
+| Draft content | `draft: true` in frontmatter | Skip entirely, do not include in plan |
+| Missing `_index.md` | Directory has content but no index | Use first numbered file for metadata |
+| Conflicting frontmatter | Different titles in `_index.md` vs content | Use `_index.md` values, note discrepancy |
+| Missing resources file | No `99-resources.md` or `91-resources.md` | Omit Resources section from generated skill |
+| Circular references | Section A references B, B references A | Include each once, note relationship |
+| Very large section | >20 markdown files | Flag for splitting (see agent-prompt.md) |
+| Incomplete section | `TODO` or placeholder text | Flag in plan for user decision |
+
+### 3.5 Build Per-Skill Package
+
+Each skill generation agent receives variables that map directly to the agent prompt template:
+
+```yaml
+# Agent prompt variables (see agent-prompt.md for full template)
+name: "libfuzzer"                              # {name} - skill name (lowercase)
+type: "fuzzer"                                 # {type} - tool|fuzzer|technique|domain
+pass: 1                                        # {pass} - 1=content only, 2=cross-refs only
+handbook_path: "/path/to/testing-handbook"     # {handbook_path} - absolute path
+section_path: "fuzzing/c-cpp/10-libfuzzer"     # {section_path} - relative to content/docs/
+output_dir: "/path/to/skills/plugins/testing-handbook-skills/skills"  # {output_dir}
+template_path: "skills/testing-handbook-generator/templates/fuzzer-skill.md"  # {template_path}
+related_sections: "fuzzing/techniques/01-writing-harnesses, fuzzing/techniques/03-asan"
+# ^ Use comma-separated list OR empty string "" if no related sections
+
+# Metadata (for orchestrator tracking, not passed to agent)
+metadata:
+  title: "libFuzzer"                           # From frontmatter
+  summary: "Coverage-guided fuzzer..."         # From frontmatter
+  has_resources: true                          # Has 99-resources.md
+  estimated_lines: 350                         # Approximate output size
+```
+
+**Variable mapping to agent prompt:**
+
+| Package Field | Agent Prompt Variable | Example |
+|---------------|----------------------|---------|
+| `name` | `{name}` | `libfuzzer` |
+| `type` | `{type}` | `fuzzer` |
+| `pass` | `{pass}` | `1` or `2` |
+| `handbook_path` | `{handbook_path}` | `/path/to/testing-handbook` |
+| `section_path` | `{section_path}` | `fuzzing/c-cpp/10-libfuzzer` |
+| `output_dir` | `{output_dir}` | `/path/to/skills/plugins/testing-handbook-skills/skills` |
+| `template_path` | `{template_path}` | `skills/testing-handbook-generator/templates/fuzzer-skill.md` |
+| `related_sections` | `{related_sections}` | Comma-separated list or empty string |
+
+**Building the package:**
+
+```bash
+# Construct output_dir from current location
+output_dir="$(cd .. && pwd)"  # Parent of testing-handbook-generator
+
+# Construct template_path from type
+template_path="skills/testing-handbook-generator/templates/${type}-skill.md"
+
+# Format related sections (empty string if none)
+if [ ${#related_sections[@]} -eq 0 ]; then
+  related_sections=""
+else
+  related_sections=$(IFS=', '; echo "${related_sections[*]}")
+fi
+```
+
+### 3.6 Pre-Generation Validation
+
+Before launching generation agents, verify:
+
+```
+For each skill candidate:
+├─ Primary content exists and is non-empty
+├─ Frontmatter has title and summary
+├─ At least one code block present (for tool/fuzzer types)
+├─ No unresolved Hugo shortcodes in source content
+├─ Related sections (if any) are accessible
+└─ Template file exists for skill type
+```
+
+**If validation fails:**
+- Log the specific failure
+- Move candidate to "Skipped Sections" with reason
+- Continue with remaining candidates
+
+### 3.7 Launch Generation
+
+**Trigger conditions:**
+- User has approved plan (explicit confirmation)
+- All source paths verified accessible
+- All per-skill packages prepared
+
+**Launch sequence:**
+1. Launch Pass 1 agents in parallel (content generation)
+2. Wait for all Pass 1 agents to complete
+3. Run Pass 2 (cross-reference population) - see [SKILL.md](SKILL.md#two-pass-generation-phase-3)
+4. Run validator on all generated skills
+
+### 3.8 Success Criteria
+
+Phase 3 complete when:
+- [ ] All skill packages prepared with variables
+- [ ] Pass 1 agents launched and completed
+- [ ] Pass 2 cross-references populated
+- [ ] All generated skills pass validation