tasker-engine 1.0.0

data/docs/QUICK_START.md

@@ -0,0 +1,270 @@
+# Quick Start Guide: Your First Tasker Workflow
+
+## Goal: Working Workflow in 5 Minutes
+
+This guide will get you from zero to a working Tasker application with complete workflows in **5 minutes**. We'll use our automated demo application builder to create a full-featured Rails app with real-world workflow examples, then explore how to customize and extend them.
+
+**🚀 New in Tasker 2.5.0**: This guide leverages our enterprise-grade demo application builder with:
+- **Automated Setup**: One-command installation with complete Rails application
+- **Real-World Examples**: E-commerce, inventory, and customer management workflows
+- **Performance Optimization**: Dynamic concurrency with configurable execution settings
+- **Full Observability**: OpenTelemetry tracing and Prometheus metrics integration
+- **Production Ready**: Complete with Redis, Sidekiq, and comprehensive documentation
+
+*Why this approach?* Instead of manual setup, we'll use proven patterns from our demo builder that create production-ready applications instantly.
+
+## Prerequisites (1 minute)
+
+Ensure you have:
+- **Ruby 3.2+** with bundler
+- **PostgreSQL** running locally
+- **Basic terminal access**
+- **Optional**: Redis for caching (will be configured automatically)
+
+## Installation & Setup (2 minutes)
+
+### Option 1: Automated Demo Application (Recommended)
+
+Create a complete Tasker application with real-world workflows instantly:
+
+```bash
+# Interactive setup with full observability stack
+curl -fsSL https://raw.githubusercontent.com/tasker-systems/tasker/main/scripts/install-tasker-app.sh | bash
+
+# Or specify your preferences (traditional Rails setup)
+curl -fsSL https://raw.githubusercontent.com/tasker-systems/tasker/main/scripts/install-tasker-app.sh | bash -s -- \
+  --app-name my-tasker-demo \
+  --tasks ecommerce,inventory,customer \
+  --observability \
+  --non-interactive
+
+# NEW: Docker-based development environment (v2.6.0)
+curl -fsSL https://raw.githubusercontent.com/tasker-systems/tasker/main/scripts/install-tasker-app.sh | bash -s -- \
+  --app-name my-tasker-demo \
+  --docker \
+  --with-observability
+```
+
+This creates a complete Rails application with:
+- ✅ **Tasker gem** installed and configured
+- ✅ **All 21 migrations** executed with database views/functions
+- ✅ **3 complete workflows** (e-commerce, inventory, customer management)
+- ✅ **Redis & Sidekiq** configured for background processing
+- ✅ **OpenTelemetry** tracing and **Prometheus** metrics
+- ✅ **Performance configuration** with execution tuning examples
+
+**Skip to "Exploring Your Workflows"** below to start using your new application!
+
+### Option 2: Manual Installation (Existing Rails App)
+
+If you have an existing Rails application:
+
+```bash
+# Add to Gemfile
+echo 'gem "tasker", git: "https://github.com/tasker-systems/tasker.git", tag: "v2.5.0"' >> Gemfile
+
+# Install and setup
+bundle install
+bundle exec rails tasker:install:migrations
+bundle exec rails tasker:install:database_objects  # Critical step!
+bundle exec rails db:migrate
+bundle exec rails tasker:setup
+
+# Mount the engine
+echo 'mount Tasker::Engine, at: "/tasker"' >> config/routes.rb
+```
+
+## Exploring Your Workflows (2 minutes)
+
+If you used the automated demo application builder, you now have a complete Rails application with three working workflows. Let's explore them!
+
+### Start Your Application
+
+#### Traditional Rails Setup
+```bash
+cd your-app-name  # Use the name you chose during installation
+
+# Start the services
+bundle exec redis-server &          # Background Redis
+bundle exec sidekiq &               # Background Sidekiq
+bundle exec rails server            # Rails application
+```
+
+#### Docker Setup (if you used --docker)
+```bash
+cd your-app-name  # Use the name you chose during installation
+
+# Start all services with Docker
+./bin/docker-dev up-full            # Includes observability stack
+# OR
+./bin/docker-dev up                 # Core services only
+
+# View application logs
+./bin/docker-dev logs
+```
+
+### Explore the Demo Workflows
+
+Your application includes three complete, production-ready workflows:
+
+#### 1. **E-commerce Order Processing**
+- **File**: `app/tasks/ecommerce/order_processing_handler.rb`
+- **Steps**: Validate order → Process payment → Update inventory → Send confirmation
+- **Features**: Retry logic, error handling, real API integration with DummyJSON
+
+#### 2. **Inventory Management**
+- **File**: `app/tasks/inventory/stock_management_handler.rb`
+- **Steps**: Check stock levels → Update quantities → Generate reports → Send alerts
+- **Features**: Conditional logic, parallel processing, data aggregation
+
+#### 3. **Customer Management**
+- **File**: `app/tasks/customer/profile_management_handler.rb`
+- **Steps**: Validate customer → Update profile → Sync external systems → Send notifications
+- **Features**: External API calls, data transformation, notification patterns
+
+### Test the Workflows
+
+Access your application's interfaces:
+
+```bash
+# Visit these URLs in your browser:
+open http://localhost:3000/tasker/graphql     # GraphQL API interface
+open http://localhost:3000/tasker/api-docs    # REST API documentation
+open http://localhost:3000/tasker/metrics     # Prometheus metrics endpoint
+```
+
+#### Create and Execute a Task via GraphQL
+
+1. **Open GraphQL Interface**: Navigate to `http://localhost:3000/tasker/graphql`
+
+2. **Create an E-commerce Order Task**:
+```graphql
+mutation {
+  createTask(input: {
+    taskName: "ecommerce_order_processing"
+    context: {
+      order_id: 123
+      customer_id: 456
+      items: [
+        { product_id: 1, quantity: 2, price: 29.99 }
+        { product_id: 2, quantity: 1, price: 49.99 }
+      ]
+      payment_method: "credit_card"
+    }
+  }) {
+    task {
+      taskId
+      currentState
+      workflowSteps {
+        name
+        currentState
+      }
+    }
+  }
+}
+```
+
+3. **Monitor Task Progress**:
+```graphql
+query {
+  task(taskId: "your-task-id-here") {
+    taskId
+    currentState
+    workflowSteps {
+      name
+      currentState
+      results
+      attempts
+    }
+  }
+}
+```
+
+## Performance Configuration (1 minute)
+
+Your demo application includes comprehensive execution configuration examples. Explore the performance tuning options:
+
+### View Current Configuration
+
+```ruby
+# In Rails console
+rails console
+
+# Check current execution settings
+config = Tasker.configuration.execution
+puts "Min concurrent steps: #{config.min_concurrent_steps}"
+puts "Max concurrent steps: #{config.max_concurrent_steps_limit}"
+puts "Concurrency cache duration: #{config.concurrency_cache_duration} seconds"
+```
+
+### Environment-Specific Tuning
+
+Your application includes configuration examples in:
+- **`config/initializers/tasker.rb`**: Main configuration with execution settings
+- **`config/execution_tuning_examples.rb`**: 7 environment-specific examples:
+  - Development: Conservative settings (2-6 concurrent steps)
+  - Production: High-performance (5-25 concurrent steps)
+  - High-Performance: Maximum throughput (10-50 concurrent steps)
+  - API-Heavy: Optimized for external APIs (3-8 concurrent steps)
+  - Testing: Minimal concurrency for reliability (1-3 concurrent steps)
+
+### Monitor Performance
+
+```bash
+# View metrics in your browser
+open http://localhost:3000/tasker/metrics
+
+# Check system health
+curl http://localhost:3000/tasker/health/status | jq
+```
+
+## Creating Custom Workflows
+
+Want to create your own workflow? Use our proven patterns:
+
+### 1. Generate New Workflow Structure
+
+```bash
+# Generate a new task handler
+rails generate tasker:task_handler WelcomeHandler --module_namespace WelcomeUser
+
+# This creates:
+# - app/tasks/welcome_user/welcome_handler.rb (task handler class)
+# - config/tasker/tasks/welcome_user/welcome_handler.yaml (workflow configuration)
+# - spec/tasks/welcome_user/welcome_handler_spec.rb (test file)
+```
+
+### 2. Configure Your Workflow
+
+Edit `config/tasker/tasks/welcome_user/welcome_handler.yaml`:
+
+```yaml
+---
+name: welcome_user
+module_namespace: WelcomeUser
+task_handler_class: WelcomeHandler
+
+schema:
+  type: object
+  required:
+    - user_id
+  properties:
+    user_id:
+      type: integer
+
+step_templates:
+  - name: validate_user
+    description: Ensure user exists and is valid for welcome email
+    handler_class: WelcomeUser::StepHandler::ValidateUserHandler
+
+  - name: generate_content
+    description: Generate personalized welcome email content
+    depends_on_step: validate_user
+    handler_class: WelcomeUser::StepHandler::GenerateContentHandler
+
+  - name: send_email
+    description: Send the welcome email to the user
+    depends_on_step: generate_content
+    handler_class: WelcomeUser::StepHandler::SendEmailHandler
+    default_retryable: true
+    default_retry_limit: 3

data/docs/REGISTRY_SYSTEMS.md

@@ -0,0 +1,373 @@
+# Registry System Architecture
+
+## Overview
+
+Tasker features enterprise-grade registry systems that provide thread-safe operations, comprehensive validation, structured logging, and event integration. All registry systems have been modernized to use `Concurrent::Hash` storage and unified patterns.
+
+## Registry Systems
+
+### HandlerFactory Registry
+
+The core registry for task handler management with namespace and version support.
+
+**Features**:
+- **Thread-Safe Operations**: `Concurrent::Hash` storage eliminates race conditions
+- **3-Level Registry**: `namespace_name → handler_name → version → handler_class`
+- **Interface Validation**: Fail-fast validation with detailed error messages
+- **Conflict Resolution**: `replace: true` parameter for graceful updates
+- **Structured Logging**: Every operation logged with correlation IDs
+
+**Usage**:
+```ruby
+# Thread-safe registration
+Tasker::HandlerFactory.instance.register(
+  'payment_processor',
+  PaymentHandler,
+  namespace_name: 'payments',
+  version: '2.1.0',
+  replace: true  # Handles conflicts gracefully
+)
+
+# Thread-safe retrieval
+handler = Tasker::HandlerFactory.instance.get(
+  'payment_processor',
+  namespace_name: 'payments',
+  version: '2.1.0'
+)
+
+# List handlers in namespace
+handlers = Tasker::HandlerFactory.instance.list_handlers(namespace: 'payments')
+
+# Registry statistics
+stats = Tasker::HandlerFactory.instance.stats
+# => {
+#   total_handlers: 45,
+#   namespaces: ["payments", "inventory", "notifications"],
+#   versions: ["1.0.0", "1.1.0", "2.0.0"],
+#   thread_safe: true,
+#   last_registration: "2024-01-15T10:30:45Z"
+# }
+```
+
+### PluginRegistry System
+
+Format-based plugin discovery with auto-discovery capabilities for telemetry exporters.
+
+**Features**:
+- **Format-Based Discovery**: Register plugins by export format (`:json`, `:csv`, `:prometheus`)
+- **Auto-Discovery**: Automatic plugin detection and registration
+- **Thread-Safe Operations**: Mutex-synchronized operations
+- **Interface Validation**: Method arity checking for plugin interfaces
+- **Event Integration**: Plugin registration triggers event system
+
+**Usage**:
+```ruby
+# Register custom exporter
+Tasker::Telemetry::PluginRegistry.register(
+  'custom_json_exporter',
+  CustomJsonExporter,
+  format: :json,
+  replace: true
+)
+
+# Find plugins by format
+json_exporters = Tasker::Telemetry::PluginRegistry.find_by(format: :json)
+
+# Auto-discovery
+Tasker::Telemetry::PluginRegistry.auto_discover_plugins
+
+# Registry statistics
+stats = Tasker::Telemetry::PluginRegistry.stats
+# => {
+#   total_plugins: 12,
+#   formats: [:json, :csv, :prometheus],
+#   auto_discovery_enabled: true,
+#   thread_safe: true
+# }
+```
+
+### SubscriberRegistry System
+
+Centralized event subscriber management with comprehensive validation.
+
+**Features**:
+- **Centralized Management**: Single registry for all event subscribers
+- **Event Validation**: Validates subscriber methods match event names
+- **Thread-Safe Operations**: Concurrent access protection
+- **Health Monitoring**: Built-in health checks and statistics
+- **Auto-Registration**: Automatic registration during class loading
+
+**Usage**:
+```ruby
+# Register event subscriber
+Tasker::Registry::SubscriberRegistry.register(
+  'notification_subscriber',
+  NotificationSubscriber,
+  events: ['task.completed', 'task.failed']
+)
+
+# List subscribers for event
+subscribers = Tasker::Registry::SubscriberRegistry.find_by_event('task.completed')
+
+# Registry health check
+health = Tasker::Registry::SubscriberRegistry.health_check
+# => { status: "healthy", subscriber_count: 15, events_covered: 45 }
+```
+
+## Structured Logging
+
+Every registry operation includes comprehensive structured logging with correlation IDs:
+
+```json
+{
+  "timestamp": "2024-01-15T10:30:45Z",
+  "correlation_id": "tsk_abc123_def456",
+  "component": "handler_factory",
+  "message": "Registry item registered",
+  "environment": "production",
+  "tasker_version": "2.4.1",
+  "process_id": 12345,
+  "thread_id": "abc123",
+  "entity_type": "task_handler",
+  "entity_id": "payments/payment_processor/2.1.0",
+  "entity_class": "PaymentHandler",
+  "registry_name": "handler_factory",
+  "options": {
+    "namespace_name": "payments",
+    "version": "2.1.0",
+    "replace": true
+  },
+  "event_type": "registered"
+}
+```
+
+## Interface Validation
+
+All registries include fail-fast validation with detailed error messages:
+
+```ruby
+# Example validation error
+begin
+  Tasker::HandlerFactory.instance.register('invalid_handler', InvalidClass)
+rescue Tasker::Registry::ValidationError => e
+  puts e.message
+  # => "Handler validation failed: InvalidClass does not implement required method 'process'.
+  #     Required methods: [process, initialize_task!].
+  #     Available methods: [initialize, new].
+  #     Suggestion: Inherit from Tasker::TaskHandler::Base"
+end
+```
+
+## Event Integration
+
+Registry operations are fully integrated with Tasker's 56-event system:
+
+**Registry Events**:
+- `registry.handler_registered` - Handler registration events
+- `registry.plugin_registered` - Plugin registration events
+- `registry.subscriber_registered` - Subscriber registration events
+- `registry.validation_failed` - Validation failure events
+- `registry.conflict_resolved` - Conflict resolution events
+
+**Event Payloads**:
+```ruby
+# Handler registration event
+{
+  registry_type: 'handler_factory',
+  entity_id: 'payments/payment_processor/2.1.0',
+  entity_class: 'PaymentHandler',
+  namespace_name: 'payments',
+  version: '2.1.0',
+  options: { replace: true },
+  correlation_id: 'tsk_abc123'
+}
+```
+
+## Thread Safety
+
+All registry systems use thread-safe storage and operations:
+
+**Storage Types**:
+- **HandlerFactory**: `Concurrent::Hash` with nested concurrent structures
+- **PluginRegistry**: `Concurrent::Hash` with mutex synchronization
+- **SubscriberRegistry**: `Concurrent::Hash` with atomic operations
+
+**Thread Safety Guarantees**:
+- **Atomic Operations**: Registration and retrieval are atomic
+- **Race Condition Prevention**: No partial state during concurrent access
+- **Memory Consistency**: All threads see consistent registry state
+- **Deadlock Prevention**: Proper lock ordering and timeout handling
+
+## Health Monitoring
+
+Built-in health monitoring and statistics for all registries:
+
+```ruby
+# Individual registry health
+handler_health = Tasker::HandlerFactory.instance.health_check
+plugin_health = Tasker::Telemetry::PluginRegistry.health_check
+subscriber_health = Tasker::Registry::SubscriberRegistry.health_check
+
+# Comprehensive registry system health
+registry_health = Tasker::Registry.system_health
+# => {
+#   status: "healthy",
+#   registries: {
+#     handler_factory: { status: "healthy", count: 45 },
+#     plugin_registry: { status: "healthy", count: 12 },
+#     subscriber_registry: { status: "healthy", count: 15 }
+#   },
+#   thread_safe: true,
+#   total_entities: 72
+# }
+```
+
+## Performance Characteristics
+
+**Registry Operation Performance**:
+- **Registration**: O(1) average case with thread-safe operations
+- **Retrieval**: O(1) lookup with concurrent access
+- **Listing**: O(n) where n is entities in scope
+- **Statistics**: O(1) with cached computation
+
+**Memory Usage**:
+- **Efficient Storage**: Minimal memory overhead per registered entity
+- **Concurrent Structures**: Memory-safe concurrent access
+- **Garbage Collection**: Proper cleanup and memory management
+
+## Best Practices
+
+### Registration Patterns
+
+```ruby
+# ✅ Good: Use replace parameter for updates
+Tasker::HandlerFactory.instance.register(
+  'payment_processor',
+  PaymentHandler,
+  namespace_name: 'payments',
+  version: '2.1.0',
+  replace: true
+)
+
+# ❌ Avoid: Registration without conflict handling
+Tasker::HandlerFactory.instance.register(
+  'payment_processor',
+  PaymentHandler,
+  namespace_name: 'payments',
+  version: '2.1.0'
+  # Will raise error if already exists
+)
+```
+
+### Error Handling
+
+```ruby
+# ✅ Good: Handle validation errors gracefully
+begin
+  Tasker::HandlerFactory.instance.register(name, handler_class, options)
+rescue Tasker::Registry::ValidationError => e
+  logger.error "Handler registration failed: #{e.message}"
+  # Handle error appropriately
+rescue Tasker::Registry::ConflictError => e
+  logger.warn "Handler conflict: #{e.message}"
+  # Decide whether to use replace: true
+end
+```
+
+### Performance Optimization
+
+```ruby
+# ✅ Good: Batch operations when possible
+handlers = Tasker::HandlerFactory.instance.list_handlers(namespace: 'payments')
+handlers.each { |name, versions| process_handler(name, versions) }
+
+# ❌ Avoid: Individual lookups in loops
+payment_handlers.each do |name|
+  handler = Tasker::HandlerFactory.instance.get(name, namespace_name: 'payments')
+  process_handler(handler)
+end
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**Thread Safety Issues**:
+```ruby
+# Symptom: Inconsistent registry state
+# Solution: Ensure all access goes through registry APIs
+# ❌ Don't access internal storage directly
+# ✅ Use registry methods for all operations
+```
+
+**Validation Failures**:
+```ruby
+# Symptom: Registration fails with validation error
+# Solution: Ensure classes implement required interfaces
+class MyHandler < Tasker::TaskHandler::Base
+  # Implements required methods automatically
+end
+```
+
+**Memory Leaks**:
+```ruby
+# Symptom: Growing memory usage
+# Solution: Use proper cleanup in test environments
+after(:each) do
+  Tasker::HandlerFactory.instance.clear_test_handlers!
+end
+```
+
+### Debugging Registry State
+
+```ruby
+# Check registry contents
+puts Tasker::HandlerFactory.instance.stats.to_json
+
+# Verify thread safety
+puts Tasker::HandlerFactory.instance.health_check[:thread_safe]
+
+# Monitor registration events
+Tasker::Events.subscribe('registry.handler_registered') do |event|
+  puts "Handler registered: #{event[:entity_id]}"
+end
+```
+
+## Migration Guide
+
+### Upgrading from Legacy Registry
+
+If upgrading from older Tasker versions:
+
+1. **Update Registration Code**:
+```ruby
+# Old (pre-2.3.0)
+Tasker::HandlerFactory.register('handler', HandlerClass)
+
+# New (2.3.0+)
+Tasker::HandlerFactory.instance.register(
+  'handler',
+  HandlerClass,
+  namespace_name: 'default',
+  version: '1.0.0'
+)
+```
+
+2. **Handle Thread Safety**:
+```ruby
+# Old: Manual synchronization required
+# New: Thread safety built-in
+```
+
+3. **Update Error Handling**:
+```ruby
+# Old: Generic exceptions
+# New: Specific validation and conflict errors
+```
+
+## Related Documentation
+
+- [Developer Guide](DEVELOPER_GUIDE.md) - HandlerFactory usage patterns
+- [Event System](EVENT_SYSTEM.md) - Registry event integration
+- [Telemetry](TELEMETRY.md) - Plugin registry usage
+- [Health Monitoring](HEALTH.md) - Registry health endpoints