pipecraft 0.0.0-releaseit

package/tests/TEST_STRUCTURE.md

@@ -0,0 +1,256 @@
+# Pipecraft Test Structure
+
+This document outlines the complete test structure for the Pipecraft project.
+
+## 📁 Directory Structure
+
+```
+tests/
+├── unit/                    # Unit tests for individual components
+│   ├── config.test.ts       # Configuration parsing tests
+│   ├── idempotency.test.ts  # Idempotency logic tests
+│   ├── tag-template.test.ts # Tag template tests
+│   └── versioning.test.ts   # Versioning logic tests
+├── integration/             # Integration tests for component interactions
+│   └── cli.test.ts         # CLI command integration tests
+├── e2e/                     # End-to-end tests for complete workflows
+│   └── workflow-generation.test.ts # Complete workflow generation tests
+├── github-local/            # Local GitHub Actions testing with Act
+│   ├── README.md           # Local testing documentation
+│   └── run-local-tests.sh   # Local test runner (to be created)
+├── github-live/             # Live GitHub Actions testing
+│   └── README.md           # Live testing documentation
+├── fixtures/                # Test data and configurations
+│   ├── basic-config.json   # Basic test configuration
+│   └── invalid-config.json # Invalid test configuration
+├── debugging/               # GitHub Actions debugging tools
+│   ├── debug-workflows.sh  # Workflow debugging script
+│   ├── debug-utils.ts      # Debugging utilities
+│   ├── debug-workflow.test.ts # Debugging test suite
+│   ├── iterative-debug.ts  # Iterative debugging system
+│   └── run-debug-tests.sh  # Debug test runner
+├── README.md               # Main test documentation
+├── setup.ts               # Test setup configuration
+└── TEST_STRUCTURE.md      # This file
+```
+
+## 🧪 Test Categories
+
+### 1. Unit Tests (`tests/unit/`)
+- **Purpose**: Test individual functions, classes, and modules in isolation
+- **Scope**: Single components with mocked dependencies
+- **Speed**: Fast (< 1 second per test)
+- **Examples**: Config parsing, versioning logic, template rendering
+- **Files**: `config.test.ts`, `idempotency.test.ts`, `tag-template.test.ts`, `versioning.test.ts`
+
+### 2. Integration Tests (`tests/integration/`)
+- **Purpose**: Test interactions between multiple components
+- **Scope**: Component integration with real dependencies
+- **Speed**: Medium (1-10 seconds per test)
+- **Examples**: CLI commands, workflow generation, configuration validation
+- **Files**: `cli.test.ts`
+
+### 3. End-to-End Tests (`tests/e2e/`)
+- **Purpose**: Test complete user workflows from start to finish
+- **Scope**: Full application flow with real file system
+- **Speed**: Slow (10+ seconds per test)
+- **Examples**: Complete workflow generation, version management, deployment
+- **Files**: `workflow-generation.test.ts`
+
+### 4. GitHub Local Testing (`tests/github-local/`)
+- **Purpose**: Test GitHub Actions workflows locally using Act
+- **Scope**: Local simulation of GitHub Actions environment
+- **Speed**: Medium (5-30 seconds per test)
+- **Examples**: Workflow execution, job dependencies, environment variables
+- **Tools**: Act (Nectos Act)
+- **Files**: `README.md`, `run-local-tests.sh` (to be created)
+
+### 5. GitHub Live Testing (`tests/github-live/`)
+- **Purpose**: Test workflows on actual GitHub Actions
+- **Scope**: Real GitHub environment with live workflows
+- **Speed**: Slow (1-10 minutes per test)
+- **Examples**: Live workflow execution, performance testing, debugging
+- **Tools**: GitHub CLI, GitHub Actions API
+- **Files**: `README.md`, `run-live-tests.sh` (to be created)
+
+### 6. Debugging Tools (`tests/debugging/`)
+- **Purpose**: Debug and analyze GitHub Actions failures
+- **Scope**: Workflow analysis, log parsing, failure diagnosis
+- **Speed**: Variable (depends on analysis depth)
+- **Examples**: Workflow debugging, failure analysis, iterative fixing
+- **Files**: `debug-workflows.sh`, `debug-utils.ts`, `debug-workflow.test.ts`, `iterative-debug.ts`, `run-debug-tests.sh`
+
+## 🚀 Running Tests
+
+### All Tests
+```bash
+# Run all test suites
+pnpm test
+
+# Run with coverage
+pnpm test:coverage
+
+# Run in watch mode
+pnpm test:watch
+```
+
+### Specific Test Types
+```bash
+# Unit tests only
+pnpm test tests/unit/
+
+# Integration tests only
+pnpm test tests/integration/
+
+# E2E tests only
+pnpm test tests/e2e/
+
+# GitHub local tests only
+./tests/github-local/run-local-tests.sh
+
+# GitHub live tests only
+./tests/github-live/run-live-tests.sh
+
+# Debugging tools
+./tests/debugging/debug-workflows.sh -w "My Pipeline"
+```
+
+## 📋 Test Requirements
+
+### Prerequisites
+- Node.js (v18+)
+- pnpm
+- Git
+- GitHub CLI (for live testing)
+- Act (for local GitHub Actions testing)
+- Docker (for Act)
+
+### Environment Variables
+```bash
+# For GitHub live testing
+export GITHUB_TOKEN="your_github_token"
+export GITHUB_REPOSITORY="owner/repo"
+
+# For debugging
+export GITHUB_TOKEN="your_github_token"
+export GITHUB_REPOSITORY_OWNER="owner"
+export GITHUB_REPOSITORY="repo"
+```
+
+## 🔧 Test Configuration
+
+### Vitest Configuration
+Tests use Vitest with the following configuration:
+- TypeScript support
+- ES modules
+- Coverage reporting
+- Parallel execution
+- Watch mode support
+
+### Test Data
+- **Fixtures**: `tests/fixtures/` - Static test data and configurations
+- **Mocks**: Inline mocks for external dependencies
+- **Temporary files**: Created and cleaned up automatically
+
+## 📊 Test Coverage
+
+We aim for:
+- **Unit tests**: 90%+ coverage
+- **Integration tests**: 80%+ coverage
+- **E2E tests**: Critical paths covered
+- **GitHub local tests**: All workflow types tested
+- **Live tests**: Production-like scenarios
+
+## 🐛 Debugging Tests
+
+### Common Issues
+1. **Permission errors**: Ensure test files are executable
+2. **Environment issues**: Check required environment variables
+3. **Dependency issues**: Run `pnpm install` to update dependencies
+4. **GitHub API limits**: Use appropriate rate limiting for live tests
+
+### Debug Commands
+```bash
+# Debug specific test
+pnpm test tests/unit/config.test.ts --reporter=verbose
+
+# Debug with Node inspector
+node --inspect-brk node_modules/.bin/vitest tests/unit/config.test.ts
+
+# Debug GitHub local tests
+./tests/github-local/run-local-tests.sh --verbose --debug
+
+# Debug GitHub live tests
+./tests/github-live/run-live-tests.sh --debug --workflow "My Pipeline"
+```
+
+## 📝 Writing Tests
+
+### Unit Test Example
+```typescript
+import { describe, it, expect } from 'vitest'
+import { parseConfig } from '../src/utils/config'
+
+describe('Config Parser', () => {
+  it('should parse valid configuration', () => {
+    const config = parseConfig('{"version": "1.0.0"}')
+    expect(config.version).toBe('1.0.0')
+  })
+})
+```
+
+### Integration Test Example
+```typescript
+import { describe, it, expect } from 'vitest'
+import { runCLI } from './helpers/cli'
+
+describe('CLI Integration', () => {
+  it('should generate workflows', async () => {
+    const result = await runCLI(['generate'])
+    expect(result.exitCode).toBe(0)
+  })
+})
+```
+
+### E2E Test Example
+```typescript
+import { describe, it, expect } from 'vitest'
+import { generateWorkflows } from './helpers/workflow'
+
+describe('E2E Workflow Generation', () => {
+  it('should generate complete workflow suite', async () => {
+    const result = await generateWorkflows()
+    expect(result.success).toBe(true)
+    expect(result.workflows).toHaveLength(7)
+  })
+})
+```
+
+## 🔄 Continuous Integration
+
+### GitHub Actions
+Tests run automatically on:
+- Pull requests
+- Pushes to main branch
+- Scheduled runs (nightly)
+
+### Test Matrix
+- Node.js versions: 18, 20, 22
+- Operating systems: Ubuntu, macOS, Windows
+- Test types: Unit, Integration, E2E
+
+## 📚 Additional Resources
+
+- [Vitest Documentation](https://vitest.dev/)
+- [Act Documentation](https://github.com/nektos/act)
+- [GitHub Actions Documentation](https://docs.github.com/en/actions)
+- [Testing Best Practices](./testing-best-practices.md)
+
+## 🤝 Contributing
+
+When adding new tests:
+1. Follow the existing structure and naming conventions
+2. Add appropriate test data to fixtures
+3. Update this document if adding new test categories
+4. Ensure tests are fast and reliable
+5. Add documentation for complex test scenarios

package/tests/act/run-act-tests.sh

@@ -0,0 +1,345 @@
+#!/bin/bash
+
+# Act Testing Suite Runner
+# Tests GitHub Actions workflows locally using Act
+
+set -euo pipefail
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+PURPLE='\033[0;35m'
+CYAN='\033[0;36m'
+NC='\033[0m' # No Color
+
+# Configuration
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+TESTS_DIR="$SCRIPT_DIR"
+FIXTURES_DIR="$TESTS_DIR/fixtures"
+OUTPUT_DIR="$TESTS_DIR/output"
+VERBOSE=false
+DEBUG=false
+WORKFLOW=""
+CLEANUP=true
+
+# Function to print colored output
+print_status() {
+    local status=$1
+    local message=$2
+    
+    case $status in
+        "SUCCESS") echo -e "${GREEN}✓${NC} $message" ;;
+        "FAILURE") echo -e "${RED}✗${NC} $message" ;;
+        "WARNING") echo -e "${YELLOW}⚠${NC} $message" ;;
+        "INFO") echo -e "${BLUE}ℹ${NC} $message" ;;
+        "DEBUG") echo -e "${PURPLE}🐛${NC} $message" ;;
+        *) echo -e "${CYAN}?${NC} $message" ;;
+    esac
+}
+
+# Function to show usage
+show_usage() {
+    cat << EOF
+Usage: $0 [OPTIONS]
+
+Act Testing Suite Runner
+
+OPTIONS:
+    -w, --workflow FILE      Test specific workflow file
+    -v, --verbose            Verbose output
+    -d, --debug              Debug mode with extra logging
+    -c, --no-cleanup         Don't clean up test artifacts
+    -h, --help               Show this help message
+
+EXAMPLES:
+    $0                                    # Run all workflow tests
+    $0 -w pipeline.yml                   # Test specific workflow
+    $0 -v -d                             # Verbose debug mode
+    $0 -w job.changes.yml --no-cleanup   # Test changes workflow, keep artifacts
+
+EOF
+}
+
+# Function to check prerequisites
+check_prerequisites() {
+    print_status "INFO" "Checking prerequisites..."
+    
+    local missing_deps=()
+    
+    # Check for required commands
+    if ! command -v act &> /dev/null; then
+        missing_deps+=("act")
+    fi
+    
+    if ! command -v docker &> /dev/null; then
+        missing_deps+=("docker")
+    fi
+    
+    if ! command -v jq &> /dev/null; then
+        missing_deps+=("jq")
+    fi
+    
+    if [ ${#missing_deps[@]} -gt 0 ]; then
+        print_status "FAILURE" "Missing dependencies: ${missing_deps[*]}"
+        echo -e "${YELLOW}Please install missing dependencies:${NC}"
+        for dep in "${missing_deps[@]}"; do
+            case $dep in
+                "act") echo "  - Act: curl https://raw.githubusercontent.com/nektos/act/master/install.sh | sudo bash" ;;
+                "docker") echo "  - Docker: https://docs.docker.com/get-docker/" ;;
+                "jq") echo "  - jq: brew install jq (macOS) or apt-get install jq (Ubuntu)" ;;
+            esac
+        done
+        exit 1
+    fi
+    
+    # Check if Docker is running
+    if ! docker info &> /dev/null; then
+        print_status "FAILURE" "Docker is not running"
+        echo -e "${YELLOW}Please start Docker and try again${NC}"
+        exit 1
+    fi
+    
+    print_status "SUCCESS" "All prerequisites satisfied"
+}
+
+# Function to setup test environment
+setup_test_environment() {
+    print_status "INFO" "Setting up test environment..."
+    
+    # Create output directory
+    mkdir -p "$OUTPUT_DIR"
+    
+    # Create test configuration
+    cat > "$PROJECT_ROOT/.actrc" << EOF
+# Act configuration for testing
+-P ubuntu-latest=catthehacker/ubuntu:act-latest
+-P ubuntu-20.04=catthehacker/ubuntu:act-20.04
+-P ubuntu-18.04=catthehacker/ubuntu:act-18.04
+EOF
+    
+    # Create test environment file
+    cat > "$PROJECT_ROOT/.env" << EOF
+# Test environment variables
+GITHUB_ACTIONS=true
+GITHUB_WORKFLOW=test-workflow
+GITHUB_RUN_ID=123456789
+GITHUB_RUN_NUMBER=1
+GITHUB_ACTOR=test-user
+GITHUB_REPOSITORY=test-org/test-repo
+GITHUB_EVENT_NAME=push
+GITHUB_SHA=abc123def456
+GITHUB_REF=refs/heads/main
+GITHUB_HEAD_REF=""
+GITHUB_BASE_REF=""
+GITHUB_WORKSPACE=$PROJECT_ROOT
+GITHUB_TOKEN=test-token
+EOF
+    
+    print_status "SUCCESS" "Test environment setup complete"
+}
+
+# Function to test a specific workflow
+test_workflow() {
+    local workflow_file=$1
+    local workflow_name=$(basename "$workflow_file" .yml)
+    
+    print_status "INFO" "Testing workflow: $workflow_name"
+    
+    # Check if workflow file exists
+    if [ ! -f "$workflow_file" ]; then
+        print_status "FAILURE" "Workflow file not found: $workflow_file"
+        return 1
+    fi
+    
+    # Test workflow syntax
+    if [ "$VERBOSE" = true ]; then
+        print_status "DEBUG" "Validating workflow syntax..."
+    fi
+    
+    # Run Act test
+    local act_args=("-W" "$workflow_file" "--dry-run")
+    
+    if [ "$VERBOSE" = true ]; then
+        act_args+=("--verbose")
+    fi
+    
+    if [ "$DEBUG" = true ]; then
+        act_args+=("--debug")
+    fi
+    
+    # Add environment file
+    act_args+=("--env-file" "$PROJECT_ROOT/.env")
+    
+    # Run the test
+    if act "${act_args[@]}" > "$OUTPUT_DIR/${workflow_name}.log" 2>&1; then
+        print_status "SUCCESS" "Workflow $workflow_name passed"
+        return 0
+    else
+        print_status "FAILURE" "Workflow $workflow_name failed"
+        if [ "$VERBOSE" = true ]; then
+            echo "Error details:"
+            cat "$OUTPUT_DIR/${workflow_name}.log"
+        fi
+        return 1
+    fi
+}
+
+# Function to test all workflows
+test_all_workflows() {
+    print_status "INFO" "Testing all workflows..."
+    
+    local workflows_dir="$PROJECT_ROOT/.github/workflows"
+    local failed_tests=0
+    local total_tests=0
+    
+    # Find all workflow files
+    local workflow_files=()
+    if [ -d "$workflows_dir" ]; then
+        while IFS= read -r -d '' file; do
+            workflow_files+=("$file")
+        done < <(find "$workflows_dir" -name "*.yml" -print0)
+    fi
+    
+    if [ ${#workflow_files[@]} -eq 0 ]; then
+        print_status "WARNING" "No workflow files found in $workflows_dir"
+        return 0
+    fi
+    
+    # Test each workflow
+    for workflow_file in "${workflow_files[@]}"; do
+        total_tests=$((total_tests + 1))
+        if test_workflow "$workflow_file"; then
+            print_status "SUCCESS" "✅ $(basename "$workflow_file")"
+        else
+            print_status "FAILURE" "❌ $(basename "$workflow_file")"
+            failed_tests=$((failed_tests + 1))
+        fi
+    done
+    
+    # Summary
+    echo ""
+    print_status "INFO" "Test Summary:"
+    echo "  Total workflows: $total_tests"
+    echo "  Passed: $((total_tests - failed_tests))"
+    echo "  Failed: $failed_tests"
+    
+    if [ $failed_tests -gt 0 ]; then
+        print_status "FAILURE" "Some workflows failed testing"
+        return 1
+    else
+        print_status "SUCCESS" "All workflows passed testing"
+        return 0
+    fi
+}
+
+# Function to cleanup test artifacts
+cleanup_test_artifacts() {
+    if [ "$CLEANUP" = true ]; then
+        print_status "INFO" "Cleaning up test artifacts..."
+        
+        # Remove test files
+        rm -f "$PROJECT_ROOT/.actrc"
+        rm -f "$PROJECT_ROOT/.env"
+        
+        # Clean up output directory
+        if [ -d "$OUTPUT_DIR" ]; then
+            rm -rf "$OUTPUT_DIR"
+        fi
+        
+        print_status "SUCCESS" "Cleanup complete"
+    fi
+}
+
+# Function to show test results
+show_test_results() {
+    if [ -d "$OUTPUT_DIR" ]; then
+        echo ""
+        print_status "INFO" "Test Results:"
+        echo "  Output directory: $OUTPUT_DIR"
+        
+        if [ "$VERBOSE" = true ]; then
+            echo "  Log files:"
+            find "$OUTPUT_DIR" -name "*.log" -exec basename {} \; | while read -r log_file; do
+                echo "    - $log_file"
+            done
+        fi
+    fi
+}
+
+# Parse command line arguments
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        -w|--workflow)
+            WORKFLOW="$2"
+            shift 2
+            ;;
+        -v|--verbose)
+            VERBOSE=true
+            shift
+            ;;
+        -d|--debug)
+            DEBUG=true
+            shift
+            ;;
+        -c|--no-cleanup)
+            CLEANUP=false
+            shift
+            ;;
+        -h|--help)
+            show_usage
+            exit 0
+            ;;
+        *)
+            echo -e "${RED}Unknown option: $1${NC}" >&2
+            show_usage
+            exit 1
+            ;;
+    esac
+done
+
+# Main execution
+main() {
+    echo -e "${BLUE}Act Testing Suite${NC}"
+    echo -e "${CYAN}================${NC}"
+    echo ""
+    
+    # Check prerequisites
+    check_prerequisites
+    
+    # Setup test environment
+    setup_test_environment
+    
+    # Run tests
+    if [ -n "$WORKFLOW" ]; then
+        # Test specific workflow
+        if test_workflow "$WORKFLOW"; then
+            print_status "SUCCESS" "Workflow test completed successfully"
+        else
+            print_status "FAILURE" "Workflow test failed"
+            exit 1
+        fi
+    else
+        # Test all workflows
+        if test_all_workflows; then
+            print_status "SUCCESS" "All workflow tests completed successfully"
+        else
+            print_status "FAILURE" "Some workflow tests failed"
+            exit 1
+        fi
+    fi
+    
+    # Show results
+    show_test_results
+    
+    # Cleanup
+    cleanup_test_artifacts
+    
+    echo ""
+    print_status "SUCCESS" "Act testing complete!"
+}
+
+# Run main function
+main "$@"