@amsterdamdatalabs/enact-extensions 0.1.3 → 0.1.8

package/extensions/plugin-dev/skills/command-development/references/testing-strategies.md

@@ -1,723 +0,0 @@
-# Command Testing Strategies
-
-Comprehensive strategies for testing slash commands before deployment and distribution.
-
-## Overview
-
-Testing commands ensures they work correctly, handle edge cases, and provide good user experience. A systematic testing approach catches issues early and builds confidence in command reliability.
-
-## Testing Levels
-
-### Level 1: Syntax and Structure Validation
-
-**What to test:**
-
-- YAML frontmatter syntax
-- Markdown format
-- File location and naming
-
-**How to test:**
-
-```bash
-# Validate YAML frontmatter
-head -n 20 .claude/commands/my-command.md | grep -A 10 "^---"
-
-# Check for closing frontmatter marker
-head -n 20 .claude/commands/my-command.md | grep -c "^---" # Should be 2
-
-# Verify file has .md extension
-ls .claude/commands/*.md
-
-# Check file is in correct location
-test -f .claude/commands/my-command.md && echo "Found" || echo "Missing"
-```
-
-**Automated validation script:**
-
-```bash
-#!/bin/bash
-# validate-command.sh
-
-COMMAND_FILE="$1"
-
-if [ ! -f "$COMMAND_FILE" ]; then
-  echo "ERROR: File not found: $COMMAND_FILE"
-  exit 1
-fi
-
-# Check .md extension
-if [[ ! "$COMMAND_FILE" =~ \.md$ ]]; then
-  echo "ERROR: File must have .md extension"
-  exit 1
-fi
-
-# Validate YAML frontmatter if present
-if head -n 1 "$COMMAND_FILE" | grep -q "^---"; then
-  # Count frontmatter markers
-  MARKERS=$(head -n 50 "$COMMAND_FILE" | grep -c "^---")
-  if [ "$MARKERS" -ne 2 ]; then
-    echo "ERROR: Invalid YAML frontmatter (need exactly 2 '---' markers)"
-    exit 1
-  fi
-  echo "✓ YAML frontmatter syntax valid"
-fi
-
-# Check for empty file
-if [ ! -s "$COMMAND_FILE" ]; then
-  echo "ERROR: File is empty"
-  exit 1
-fi
-
-echo "✓ Command file structure valid"
-```
-
-### Level 2: Frontmatter Field Validation
-
-**What to test:**
-
-- Field types correct
-- Values in valid ranges
-- Required fields present (if any)
-
-**Validation script:**
-
-```bash
-#!/bin/bash
-# validate-frontmatter.sh
-
-COMMAND_FILE="$1"
-
-# Extract YAML frontmatter
-FRONTMATTER=$(sed -n '/^---$/,/^---$/p' "$COMMAND_FILE" | sed '1d;$d')
-
-if [ -z "$FRONTMATTER" ]; then
-  echo "No frontmatter to validate"
-  exit 0
-fi
-
-# Check 'model' field if present
-if echo "$FRONTMATTER" | grep -q "^model:"; then
-  MODEL=$(echo "$FRONTMATTER" | grep "^model:" | cut -d: -f2 | tr -d ' ')
-  if ! echo "sonnet opus haiku" | grep -qw "$MODEL"; then
-    echo "ERROR: Invalid model '$MODEL' (must be sonnet, opus, or haiku)"
-    exit 1
-  fi
-  echo "✓ Model field valid: $MODEL"
-fi
-
-# Check 'allowed-tools' field format
-if echo "$FRONTMATTER" | grep -q "^allowed-tools:"; then
-  echo "✓ allowed-tools field present"
-  # Could add more sophisticated validation here
-fi
-
-# Check 'description' length
-if echo "$FRONTMATTER" | grep -q "^description:"; then
-  DESC=$(echo "$FRONTMATTER" | grep "^description:" | cut -d: -f2-)
-  LENGTH=${#DESC}
-  if [ "$LENGTH" -gt 80 ]; then
-    echo "WARNING: Description length $LENGTH (recommend < 60 chars)"
-  else
-    echo "✓ Description length acceptable: $LENGTH chars"
-  fi
-fi
-
-echo "✓ Frontmatter fields valid"
-```
-
-### Level 3: Manual Command Invocation
-
-**What to test:**
-
-- Command appears in `/help`
-- Command executes without errors
-- Output is as expected
-
-**Test procedure:**
-
-```bash
-# 1. Start Claude Code
-claude --debug
-
-# 2. Check command appears in help
-> /help
-# Look for your command in the list
-
-# 3. Invoke command without arguments
-> /my-command
-# Check for reasonable error or behavior
-
-# 4. Invoke with valid arguments
-> /my-command arg1 arg2
-# Verify expected behavior
-
-# 5. Check debug logs
-tail -f ~/.claude/debug-logs/latest
-# Look for errors or warnings
-```
-
-### Level 4: Argument Testing
-
-**What to test:**
-
-- Positional arguments work ($1, $2, etc.)
-- $ARGUMENTS captures all arguments
-- Missing arguments handled gracefully
-- Invalid arguments detected
-
-**Test matrix:**
-
-| Test Case     | Command                  | Expected Result                              |
-| ------------- | ------------------------ | -------------------------------------------- |
-| No args       | `/cmd`                   | Graceful handling or useful message          |
-| One arg       | `/cmd arg1`              | $1 substituted correctly                     |
-| Two args      | `/cmd arg1 arg2`         | $1 and $2 substituted                        |
-| Extra args    | `/cmd a b c d`           | All captured or extras ignored appropriately |
-| Special chars | `/cmd "arg with spaces"` | Quotes handled correctly                     |
-| Empty arg     | `/cmd ""`                | Empty string handled                         |
-
-**Test script:**
-
-```bash
-#!/bin/bash
-# test-command-arguments.sh
-
-COMMAND="$1"
-
-echo "Testing argument handling for /$COMMAND"
-echo
-
-echo "Test 1: No arguments"
-echo "  Command: /$COMMAND"
-echo "  Expected: [describe expected behavior]"
-echo "  Manual test required"
-echo
-
-echo "Test 2: Single argument"
-echo "  Command: /$COMMAND test-value"
-echo "  Expected: 'test-value' appears in output"
-echo "  Manual test required"
-echo
-
-echo "Test 3: Multiple arguments"
-echo "  Command: /$COMMAND arg1 arg2 arg3"
-echo "  Expected: All arguments used appropriately"
-echo "  Manual test required"
-echo
-
-echo "Test 4: Special characters"
-echo "  Command: /$COMMAND \"value with spaces\""
-echo "  Expected: Entire phrase captured"
-echo "  Manual test required"
-```
-
-### Level 5: File Reference Testing
-
-**What to test:**
-
-- @ syntax loads file contents
-- Non-existent files handled
-- Large files handled appropriately
-- Multiple file references work
-
-**Test procedure:**
-
-```bash
-# Create test files
-echo "Test content" > /tmp/test-file.txt
-echo "Second file" > /tmp/test-file-2.txt
-
-# Test single file reference
-> /my-command /tmp/test-file.txt
-# Verify file content is read
-
-# Test non-existent file
-> /my-command /tmp/nonexistent.txt
-# Verify graceful error handling
-
-# Test multiple files
-> /my-command /tmp/test-file.txt /tmp/test-file-2.txt
-# Verify both files processed
-
-# Test large file
-dd if=/dev/zero of=/tmp/large-file.bin bs=1M count=100
-> /my-command /tmp/large-file.bin
-# Verify reasonable behavior (may truncate or warn)
-
-# Cleanup
-rm /tmp/test-file*.txt /tmp/large-file.bin
-```
-
-### Level 6: Bash Execution Testing
-
-**What to test:**
-
-- ` commands execute correctly
-- Command output included in prompt
-- Command failures handled
-- Security: only allowed commands run
-
-**Test procedure:**
-
-```bash
-# Create test command with bash execution
-cat > .claude/commands/test-bash.md << 'EOF'
----
-description: Test bash execution
-allowed-tools: Bash(echo:*), Bash(date:*)
----
-
-Current date: `date`
-Test output: `echo "Hello from bash"`
-
-Analysis of output above...
-EOF
-
-# Test in Claude Code
-> /test-bash
-# Verify:
-# 1. Date appears correctly
-# 2. Echo output appears
-# 3. No errors in debug logs
-
-# Test with disallowed command (should fail or be blocked)
-cat > .claude/commands/test-forbidden.md << 'EOF'
----
-description: Test forbidden command
-allowed-tools: Bash(echo:*)
----
-
-Trying forbidden: `ls -la /`
-EOF
-
-> /test-forbidden
-# Verify: Permission denied or appropriate error
-```
-
-### Level 7: Integration Testing
-
-**What to test:**
-
-- Commands work with other plugin components
-- Commands interact correctly with each other
-- State management works across invocations
-- Workflow commands execute in sequence
-
-**Test scenarios:**
-
-#### Scenario 1: Command + Hook Integration
-
-```bash
-# Setup: Command that triggers a hook
-# Test: Invoke command, verify hook executes
-
-# Command: .claude/commands/risky-operation.md
-# Hook: PreToolUse that validates the operation
-
-> /risky-operation
-# Verify: Hook executes and validates before command completes
-```
-
-#### Scenario 2: Command Sequence
-
-```bash
-# Setup: Multi-command workflow
-> /workflow-init
-# Verify: State file created
-
-> /workflow-step2
-# Verify: State file read, step 2 executes
-
-> /workflow-complete
-# Verify: State file cleaned up
-```
-
-#### Scenario 3: Command + MCP Integration
-
-```bash
-# Setup: Command uses MCP tools
-# Test: Verify MCP server accessible
-
-> /mcp-command
-# Verify:
-# 1. MCP server starts (if stdio)
-# 2. Tool calls succeed
-# 3. Results included in output
-```
-
-## Automated Testing Approaches
-
-### Command Test Suite
-
-Create a test suite script:
-
-```bash
-#!/bin/bash
-# test-commands.sh - Command test suite
-
-TEST_DIR=".claude/commands"
-FAILED_TESTS=0
-
-echo "Command Test Suite"
-echo "=================="
-echo
-
-for cmd_file in "$TEST_DIR"/*.md; do
-  cmd_name=$(basename "$cmd_file" .md)
-  echo "Testing: $cmd_name"
-
-  # Validate structure
-  if ./validate-command.sh "$cmd_file"; then
-    echo "  ✓ Structure valid"
-  else
-    echo "  ✗ Structure invalid"
-    ((FAILED_TESTS++))
-  fi
-
-  # Validate frontmatter
-  if ./validate-frontmatter.sh "$cmd_file"; then
-    echo "  ✓ Frontmatter valid"
-  else
-    echo "  ✗ Frontmatter invalid"
-    ((FAILED_TESTS++))
-  fi
-
-  echo
-done
-
-echo "=================="
-echo "Tests complete"
-echo "Failed: $FAILED_TESTS"
-
-exit $FAILED_TESTS
-```
-
-### Pre-Commit Hook
-
-Validate commands before committing:
-
-```bash
-#!/bin/bash
-# .git/hooks/pre-commit
-
-echo "Validating commands..."
-
-COMMANDS_CHANGED=$(git diff --cached --name-only | grep "\.claude/commands/.*\.md")
-
-if [ -z "$COMMANDS_CHANGED" ]; then
-  echo "No commands changed"
-  exit 0
-fi
-
-for cmd in $COMMANDS_CHANGED; do
-  echo "Checking: $cmd"
-
-  if ! ./scripts/validate-command.sh "$cmd"; then
-    echo "ERROR: Command validation failed: $cmd"
-    exit 1
-  fi
-done
-
-echo "✓ All commands valid"
-```
-
-### Continuous Testing
-
-Test commands in CI/CD:
-
-```yaml
-# .github/workflows/test-commands.yml
-name: Test Commands
-
-on: [push, pull_request]
-
-jobs:
-  test:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-
-      - name: Validate command structure
-        run: |
-          for cmd in .claude/commands/*.md; do
-            echo "Testing: $cmd"
-            ./scripts/validate-command.sh "$cmd"
-          done
-
-      - name: Validate frontmatter
-        run: |
-          for cmd in .claude/commands/*.md; do
-            ./scripts/validate-frontmatter.sh "$cmd"
-          done
-
-      - name: Check for TODOs
-        run: |
-          if grep -r "TODO" .claude/commands/; then
-            echo "ERROR: TODOs found in commands"
-            exit 1
-          fi
-```
-
-## Edge Case Testing
-
-### Test Edge Cases
-
-**Empty arguments:**
-
-```bash
-> /cmd ""
-> /cmd '' ''
-```
-
-**Special characters:**
-
-```bash
-> /cmd "arg with spaces"
-> /cmd arg-with-dashes
-> /cmd arg_with_underscores
-> /cmd arg/with/slashes
-> /cmd 'arg with "quotes"'
-```
-
-**Long arguments:**
-
-```bash
-> /cmd $(python -c "print('a' * 10000)")
-```
-
-**Unusual file paths:**
-
-```bash
-> /cmd ./file
-> /cmd ../file
-> /cmd ~/file
-> /cmd "/path with spaces/file"
-```
-
-**Bash command edge cases:**
-
-```markdown
-# Commands that might fail
-
-`exit 1`
-`false`
-`command-that-does-not-exist`
-
-# Commands with special output
-
-`echo ""`
-`cat /dev/null`
-`yes | head -n 1000000`
-```
-
-## Performance Testing
-
-### Response Time Testing
-
-```bash
-#!/bin/bash
-# test-command-performance.sh
-
-COMMAND="$1"
-
-echo "Testing performance of /$COMMAND"
-echo
-
-for i in {1..5}; do
-  echo "Run $i:"
-  START=$(date +%s%N)
-
-  # Invoke command (manual step - record time)
-  echo "  Invoke: /$COMMAND"
-  echo "  Start time: $START"
-  echo "  (Record end time manually)"
-  echo
-done
-
-echo "Analyze results:"
-echo "  - Average response time"
-echo "  - Variance"
-echo "  - Acceptable threshold: < 3 seconds for fast commands"
-```
-
-### Resource Usage Testing
-
-```bash
-# Monitor Claude Code during command execution
-# In terminal 1:
-claude --debug
-
-# In terminal 2:
-watch -n 1 'ps aux | grep claude'
-
-# Execute command and observe:
-# - Memory usage
-# - CPU usage
-# - Process count
-```
-
-## User Experience Testing
-
-### Usability Checklist
-
-- [ ] Command name is intuitive
-- [ ] Description is clear in `/help`
-- [ ] Arguments are well-documented
-- [ ] Error messages are helpful
-- [ ] Output is formatted readably
-- [ ] Long-running commands show progress
-- [ ] Results are actionable
-- [ ] Edge cases have good UX
-
-### User Acceptance Testing
-
-Recruit testers:
-
-```markdown
-# Testing Guide for Beta Testers
-
-## Command: /my-new-command
-
-### Test Scenarios
-
-1. **Basic usage:**
-   - Run: `/my-new-command`
-   - Expected: [describe]
-   - Rate clarity: 1-5
-
-2. **With arguments:**
-   - Run: `/my-new-command arg1 arg2`
-   - Expected: [describe]
-   - Rate usefulness: 1-5
-
-3. **Error case:**
-   - Run: `/my-new-command invalid-input`
-   - Expected: Helpful error message
-   - Rate error message: 1-5
-
-### Feedback Questions
-
-1. Was the command easy to understand?
-2. Did the output meet your expectations?
-3. What would you change?
-4. Would you use this command regularly?
-```
-
-## Testing Checklist
-
-Before releasing a command:
-
-### Structure
-
-- [ ] File in correct location
-- [ ] Correct .md extension
-- [ ] Valid YAML frontmatter (if present)
-- [ ] Markdown syntax correct
-
-### Functionality
-
-- [ ] Command appears in `/help`
-- [ ] Description is clear
-- [ ] Command executes without errors
-- [ ] Arguments work as expected
-- [ ] File references work
-- [ ] Bash execution works (if used)
-
-### Edge Cases
-
-- [ ] Missing arguments handled
-- [ ] Invalid arguments detected
-- [ ] Non-existent files handled
-- [ ] Special characters work
-- [ ] Long inputs handled
-
-### Integration
-
-- [ ] Works with other commands
-- [ ] Works with hooks (if applicable)
-- [ ] Works with MCP (if applicable)
-- [ ] State management works
-
-### Quality
-
-- [ ] Performance acceptable
-- [ ] No security issues
-- [ ] Error messages helpful
-- [ ] Output formatted well
-- [ ] Documentation complete
-
-### Distribution
-
-- [ ] Tested by others
-- [ ] Feedback incorporated
-- [ ] README updated
-- [ ] Examples provided
-
-## Debugging Failed Tests
-
-### Common Issues and Solutions
-
-#### Issue: Command not appearing in /help
-
-```bash
-# Check file location
-ls -la .claude/commands/my-command.md
-
-# Check permissions
-chmod 644 .claude/commands/my-command.md
-
-# Check syntax
-head -n 20 .claude/commands/my-command.md
-
-# Restart Claude Code
-claude --debug
-```
-
-#### Issue: Arguments not substituting
-
-```bash
-# Verify syntax
-grep '\$1' .claude/commands/my-command.md
-grep '\$ARGUMENTS' .claude/commands/my-command.md
-
-# Test with simple command first
-echo "Test: \$1 and \$2" > .claude/commands/test-args.md
-```
-
-#### Issue: Bash commands not executing
-
-```bash
-# Check allowed-tools
-grep "allowed-tools" .claude/commands/my-command.md
-
-# Verify command has bash execution syntax (! followed by backtick)
-# Note: Command files use actual ! syntax, not [BANG] placeholder
-grep -E '!\x60' .claude/commands/my-command.md
-
-# Test command manually
-date
-echo "test"
-```
-
-#### Issue: File references not working
-
-```bash
-# Check @ syntax
-grep '@' .claude/commands/my-command.md
-
-# Verify file exists
-ls -la /path/to/referenced/file
-
-# Check permissions
-chmod 644 /path/to/referenced/file
-```
-
-## Best Practices
-
-1. **Test early, test often**: Validate as you develop
-2. **Automate validation**: Use scripts for repeatable checks
-3. **Test edge cases**: Don't just test the happy path
-4. **Get feedback**: Have others test before wide release
-5. **Document tests**: Keep test scenarios for regression testing
-6. **Monitor in production**: Watch for issues after release
-7. **Iterate**: Improve based on real usage data