tasker-engine 1.0.0

data/docs/VISION.md

@@ -0,0 +1,584 @@
+# Tasker Strategic Roadmap: Distributed AI-Integrated Workflow Orchestration
+
+## Vision Statement
+
+**Transform Tasker from a Rails-native workflow engine into a distributed, AI-integrated orchestration platform that enables polyglot systems to coordinate complex workflows while maintaining the reliability, observability, and predictability that makes Tasker production-ready.**
+
+## Core Architectural Evolution
+
+### Current Foundation (Tasker 2.4.0)
+- ✅ **Single-system orchestration** with Rails-native patterns
+- ✅ **High-performance SQL functions** for step readiness calculation
+- ✅ **Thread-safe registry systems** with structured logging
+- ✅ **Comprehensive event system** with OpenTelemetry integration
+- ✅ **Production-proven reliability** with exponential backoff and retry logic
+
+### Target Architecture Vision
+- 🎯 **Distributed workflow coordination** across polyglot systems
+- 🎯 **Rust-based performance core** with language bindings
+- 🎯 **AI-integrated workflow design** and failure resolution
+- 🎯 **Structured result contracts** with type safety
+- 🎯 **Service bus event architecture** for cross-system coordination
+
+---
+
+## Phase 1: Foundation Strengthening (6-9 months)
+*"Making Tasker ready for distributed architecture"*
+
+### 1.1 Structured Result Contracts
+**Problem**: Free-form `step.results` creates integration brittleness in distributed systems.
+
+**Solution**: Self-describing step result schemas with validation.
+
+```ruby
+# Current (2.4.0)
+def process(task, sequence, step)
+  { user_id: 123, status: 'created' }  # Free-form results
+end
+
+# Target Architecture
+class CreateUserHandler < Tasker::StepHandler::Base
+  result_contract do
+    success_schema do
+      field :user_id, Integer, required: true
+      field :email, String, format: :email
+      field :created_at, DateTime
+    end
+
+    error_schema do
+      field :error_code, String, enum: ['validation_failed', 'duplicate_email']
+      field :error_message, String
+      field :retry_after, Integer, optional: true
+    end
+  end
+
+  def process(task, sequence, step)
+    # Results automatically validated against contract
+    create_user_result
+  end
+end
+```
+
+**Benefits**:
+- **Type safety** for dependent steps
+- **Integration testing** can validate contracts
+- **API documentation** auto-generated from schemas
+- **Foundation for Rust FFI** type mappings
+
+### 1.2 Event-Driven Step Architecture
+**Problem**: Current step execution is tightly coupled to single-system boundaries.
+
+**Solution**: Event-driven step execution with pluggable handlers.
+
+```ruby
+# Local Step Handler (current pattern)
+class LocalPaymentHandler < Tasker::StepHandler::Base
+  def process(task, sequence, step)
+    PaymentService.charge(task.context['amount'])
+  end
+end
+
+# Event-Driven Step Handler (distributed-ready)
+class DistributedPaymentHandler < Tasker::StepHandler::EventDriven
+  event_target 'payment_service_queue'
+  timeout 30.seconds
+
+  def create_event_payload(task, sequence, step)
+    {
+      event_type: 'payment.charge_requested',
+      order_id: task.context['order_id'],
+      amount: task.context['amount'],
+      reply_to: "tasker.#{task.id}.#{step.id}"
+    }
+  end
+
+  def validate_event_response(response)
+    # Validate against result contract
+    response.has_key?('transaction_id') && response['status'] == 'completed'
+  end
+end
+```
+
+### 1.3 Service Bus Integration Layer
+**Problem**: No coordination mechanism for cross-system workflows.
+
+**Solution**: Pluggable service bus abstraction with multiple backend support.
+
+```ruby
+# Configuration
+Tasker.configuration do |config|
+  config.service_bus do |bus|
+    bus.backend = :rabbitmq  # or :kafka, :redis, :nats
+    bus.connection_url = ENV['SERVICE_BUS_URL']
+    bus.exchange_prefix = 'tasker.workflows'
+  end
+end
+
+# Usage in step handlers
+class ServiceBusStepHandler < Tasker::StepHandler::Base
+  include Tasker::ServiceBus::Publisher
+
+  def process(task, sequence, step)
+    publish_event('inventory.check_stock', {
+      product_ids: task.context['products'],
+      reply_to: reply_channel(step)
+    })
+
+    # Return immediately - completion handled by event consumer
+    { status: 'dispatched', event_id: SecureRandom.uuid }
+  end
+end
+```
+
+**Key Features**:
+- **Backend agnostic** (RabbitMQ, Kafka, Redis, NATS)
+- **Automatic retry logic** using service bus retry mechanisms
+- **Dead letter handling** for failed cross-system calls
+- **Event correlation** for tracking distributed operations
+
+---
+
+## Phase 2: Distributed Coordination (9-15 months)
+*"Enabling true polyglot workflow orchestration"*
+
+### 2.1 Cross-System Workflow Definition
+**Problem**: Workflows currently assume single-system ownership of all steps.
+
+**Solution**: Distributed workflow manifests with system ownership mapping.
+
+```yaml
+# config/tasker/distributed_workflows/order_processing.yaml
+name: distributed_order_processing
+version: 2.0.0
+systems:
+  rails_app:
+    base_url: 'https://orders.company.com'
+    capabilities: ['order.validate', 'order.finalize']
+  python_inventory:
+    base_url: 'https://inventory.company.com'
+    capabilities: ['inventory.check', 'inventory.reserve']
+  node_payments:
+    base_url: 'https://payments.company.com'
+    capabilities: ['payment.process', 'payment.refund']
+
+workflow_steps:
+  - name: validate_order
+    owner: rails_app
+    handler: 'Orders::ValidateOrderHandler'
+
+  - name: check_inventory
+    owner: python_inventory
+    depends_on: validate_order
+    event_pattern: 'inventory.check_stock'
+    timeout: 30s
+
+  - name: process_payment
+    owner: node_payments
+    depends_on: check_inventory
+    event_pattern: 'payment.charge'
+    timeout: 45s
+
+  - name: finalize_order
+    owner: rails_app
+    depends_on: process_payment
+    handler: 'Orders::FinalizeOrderHandler'
+```
+
+### 2.2 System Registry & Discovery
+**Problem**: No mechanism for systems to discover each other's capabilities.
+
+**Solution**: Distributed system registry with capability advertisement.
+
+```ruby
+# System registration
+Tasker::DistributedRegistry.register_system(
+  name: 'python_inventory',
+  version: '1.2.0',
+  capabilities: [
+    {
+      name: 'inventory.check_stock',
+      input_schema: InventoryCheckSchema,
+      output_schema: InventoryResultSchema,
+      timeout: 30.seconds
+    }
+  ],
+  health_check_url: 'https://inventory.company.com/health'
+)
+
+# Workflow coordinator uses registry for routing
+coordinator = Tasker::DistributedCoordinator.new
+coordinator.route_step('check_inventory', to: 'python_inventory')
+```
+
+### 2.3 Rust Core Foundation
+**Problem**: Performance bottlenecks and memory safety concerns for high-scale distributed coordination.
+
+**Solution**: Extract core orchestration logic to Rust with FFI bindings.
+
+```rust
+// Core Rust types
+#[derive(Debug, Serialize, Deserialize)]
+pub struct WorkflowStep {
+    pub id: i64,
+    pub name: String,
+    pub status: StepStatus,
+    pub owner_system: String,
+    pub dependencies: Vec<i64>,
+    pub result_contract: Option<ResultContract>,
+}
+
+// High-performance dependency resolution
+pub fn calculate_ready_steps(
+    workflow: &Workflow,
+    completed_steps: &[i64]
+) -> Vec<WorkflowStep> {
+    // Zero-allocation dependency graph traversal
+}
+
+// Ruby FFI binding
+impl WorkflowEngine {
+    #[magnus::method]
+    pub fn get_ready_steps(&self, workflow_id: i64) -> Vec<StepData> {
+        // Bridge between Rust and Ruby types
+    }
+}
+```
+
+**Performance Targets**:
+- **Dependency calculation**: <100μs for 1000+ step workflows
+- **Memory usage**: <1MB per 10,000 active workflows
+- **Cross-system coordination**: <10ms overhead per distributed step
+
+---
+
+## Phase 3: AI Integration Foundation (12-18 months)
+*"Making workflows intelligent and adaptive"*
+
+### 3.1 AI-Assisted Failure Resolution
+**Problem**: Static retry logic cannot adapt to novel failure scenarios.
+
+**Solution**: AI agents that can diagnose failures and suggest resolutions.
+
+```ruby
+class AIAssistedStepHandler < Tasker::StepHandler::Base
+  include Tasker::AI::FailureAnalysis
+
+  def process(task, sequence, step)
+    payment_result = PaymentService.charge(task.context['amount'])
+  rescue PaymentGatewayError => e
+    # AI analysis of failure context
+    diagnosis = analyze_failure(
+      error: e,
+      step_context: step.context,
+      historical_failures: similar_failures(e.class, 1.week)
+    )
+
+    if diagnosis.has_workaround?
+      # Present options to user/operator
+      present_resolution_options(diagnosis)
+      mark_step_pending_manual_resolution(step, diagnosis)
+    else
+      # Standard retry logic
+      raise e
+    end
+  end
+end
+
+# AI Diagnosis Response
+{
+  failure_type: 'payment_gateway_timeout',
+  confidence: 0.87,
+  suggested_workarounds: [
+    {
+      type: 'retry_with_different_gateway',
+      confidence: 0.72,
+      implementation: 'Switch to backup payment processor'
+    },
+    {
+      type: 'manual_payment_processing',
+      confidence: 0.95,
+      implementation: 'Process payment manually and mark resolved'
+    }
+  ],
+  historical_context: '23 similar failures in past week, 89% resolved with gateway switch'
+}
+```
+
+### 3.2 Natural Language Workflow Generation
+**Problem**: Creating workflows requires deep technical knowledge of Tasker patterns.
+
+**Solution**: AI agents that generate workflow configurations from natural language descriptions.
+
+```ruby
+# Natural language input
+user_description = """
+I need a workflow that:
+1. Validates a new user registration with email verification
+2. Creates their profile in our user service
+3. Sets up their billing account in Stripe
+4. Sends them a welcome email with onboarding steps
+5. If anything fails, notify our support team
+
+The user service is a Python FastAPI app, billing is handled by our Node.js payments service.
+"""
+
+# AI-generated workflow
+workflow_generator = Tasker::AI::WorkflowGenerator.new
+generated_config = workflow_generator.generate_from_description(
+  description: user_description,
+  available_systems: ['rails_main', 'python_users', 'node_payments'],
+  user_context: current_user
+)
+
+# Output: Complete YAML configuration + Ruby step handlers
+# - Proper dependency chains
+# - Error handling patterns
+# - System ownership mapping
+# - Result contract definitions
+```
+
+### 3.3 Intelligent Workflow Optimization
+**Problem**: Workflows may have non-optimal execution patterns or bottlenecks.
+
+**Solution**: AI analysis of workflow execution patterns with optimization suggestions.
+
+```ruby
+# Continuous workflow analysis
+class WorkflowIntelligence
+  def analyze_workflow_performance(workflow_name, time_window: 30.days)
+    executions = WorkflowExecution.where(name: workflow_name)
+                                  .where('created_at > ?', time_window.ago)
+
+    analysis = {
+      bottlenecks: identify_bottleneck_steps(executions),
+      parallel_opportunities: find_parallelization_options(executions),
+      retry_patterns: analyze_retry_effectiveness(executions),
+      suggested_optimizations: generate_optimization_suggestions(executions)
+    }
+
+    # AI-generated recommendations
+    {
+      critical_path_optimization: "Steps 'validate_user' and 'check_email' can run in parallel",
+      retry_tuning: "Step 'external_api_call' succeeds 94% on 2nd retry, consider reducing retry_limit to 2",
+      timeout_adjustment: "Step 'slow_calculation' averages 45s, increase timeout from 30s to 60s",
+      dependency_optimization: "Remove unnecessary dependency between 'send_email' and 'update_analytics'"
+    }
+  end
+end
+```
+
+---
+
+## Phase 4: Advanced AI Integration (18-24 months)
+*"Autonomous workflow management and creation"*
+
+### 4.1 No-Code Workflow Builder with AI
+**Problem**: Non-technical users cannot create or modify workflows.
+
+**Solution**: Visual workflow builder with AI-powered step generation.
+
+```ruby
+# AI-powered step suggestions
+class AIWorkflowBuilder
+  def suggest_next_steps(current_workflow, business_context)
+    # Analyze partial workflow and suggest completions
+    suggestions = ai_model.complete_workflow(
+      partial_workflow: current_workflow,
+      business_domain: business_context[:domain],
+      available_integrations: system_registry.available_capabilities,
+      user_patterns: user_workflow_history
+    )
+
+    # Return composable, pre-built step templates
+    suggestions.map do |suggestion|
+      {
+        step_name: suggestion[:name],
+        description: suggestion[:description],
+        handler_template: generate_handler_code(suggestion),
+        estimated_execution_time: suggestion[:timing],
+        dependencies: suggestion[:suggested_dependencies],
+        risk_level: suggestion[:complexity_score]
+      }
+    end
+  end
+end
+
+# Visual builder integration
+class WorkflowCanvas
+  def add_ai_suggested_step(suggestion)
+    # Create step with pre-generated handler
+    # Connect to existing workflow graph
+    # Validate dependencies and contracts
+    # Generate test cases automatically
+  end
+end
+```
+
+### 4.2 MCP Server Integration
+**Problem**: Limited programmatic access to Tasker insights and management.
+
+**Solution**: Native MCP servers for workflow intelligence and management.
+
+```typescript
+// MCP Server for Tasker Workflow Management
+export const taskerMCPServer = {
+  name: "tasker-workflow-orchestration",
+  version: "1.0.0",
+
+  tools: [
+    {
+      name: "analyze_workflow_performance",
+      description: "Analyze workflow execution patterns and suggest optimizations",
+      inputSchema: {
+        type: "object",
+        properties: {
+          workflow_name: { type: "string" },
+          time_window: { type: "string", default: "30d" }
+        }
+      }
+    },
+
+    {
+      name: "create_workflow_from_description",
+      description: "Generate a complete workflow from natural language description",
+      inputSchema: {
+        type: "object",
+        properties: {
+          description: { type: "string" },
+          target_systems: { type: "array", items: { type: "string" } }
+        }
+      }
+    },
+
+    {
+      name: "diagnose_workflow_failure",
+      description: "AI-powered diagnosis of workflow failures with resolution suggestions",
+      inputSchema: {
+        type: "object",
+        properties: {
+          task_id: { type: "string" },
+          include_historical_context: { type: "boolean", default: true }
+        }
+      }
+    },
+
+    {
+      name: "optimize_workflow_dependencies",
+      description: "Suggest dependency optimizations for improved parallelization",
+      inputSchema: {
+        type: "object",
+        properties: {
+          workflow_name: { type: "string" }
+        }
+      }
+    }
+  ]
+};
+
+// Integration with AI assistants
+async function handleWorkflowAnalysis(request: MCPRequest) {
+  const analysis = await TaskerIntelligence.analyzeWorkflow(request.params);
+
+  return {
+    performance_metrics: analysis.metrics,
+    optimization_suggestions: analysis.suggestions,
+    ai_insights: analysis.ai_recommendations,
+    actionable_improvements: analysis.improvements.map(formatForImplementation)
+  };
+}
+```
+
+### 4.3 Autonomous Workflow Healing
+**Problem**: Workflows may degrade over time or encounter novel failure modes.
+
+**Solution**: Self-healing workflows that adapt to changing conditions.
+
+```ruby
+class AutonomousWorkflowManager
+  def monitor_and_heal(workflow_name)
+    health_metrics = continuously_monitor_workflow(workflow_name)
+
+    if health_metrics.degraded?
+      healing_actions = generate_healing_strategy(health_metrics)
+
+      healing_actions.each do |action|
+        case action[:type]
+        when :adjust_timeouts
+          auto_adjust_step_timeouts(action[:step_name], action[:new_timeout])
+
+        when :add_circuit_breaker
+          inject_circuit_breaker_pattern(action[:step_name], action[:thresholds])
+
+        when :modify_retry_strategy
+          update_retry_configuration(action[:step_name], action[:strategy])
+
+        when :suggest_architectural_change
+          notify_engineering_team(action[:recommendation])
+        end
+      end
+    end
+  end
+
+  private
+
+  def generate_healing_strategy(health_metrics)
+    # AI-powered analysis of degradation patterns
+    # Historical pattern matching
+    # Predictive failure prevention
+    # Safe autonomous adjustments vs. human-required changes
+  end
+end
+```
+
+---
+
+## Implementation Strategy & Considerations
+
+### Technical Architecture Principles
+
+1. **Backward Compatibility**: Each phase builds on previous work without breaking existing functionality
+2. **Incremental Adoption**: Features can be adopted gradually without requiring full migration
+3. **Production Safety**: All AI-assisted features include human oversight and rollback mechanisms
+4. **Performance First**: Rust core provides foundation for high-scale distributed coordination
+5. **Type Safety**: Result contracts and schema validation prevent distributed system brittleness
+
+### Risk Mitigation Strategies
+
+**AI Integration Risks**:
+- **Hallucination Protection**: All AI-generated configurations validated against known patterns
+- **Human Oversight**: Critical decisions require human approval
+- **Rollback Mechanisms**: Easy reversal of AI-suggested changes
+- **Confidence Scoring**: AI suggestions include confidence levels and uncertainty bounds
+
+**Distributed System Risks**:
+- **Circuit Breakers**: Automatic fallback when remote systems are unavailable
+- **Timeout Management**: Configurable timeouts with exponential backoff
+- **Service Discovery**: Health checks and capability validation
+- **Data Consistency**: Eventually consistent with conflict resolution strategies
+
+**Performance Risks**:
+- **Rust Migration**: Incremental extraction with performance benchmarking
+- **Memory Management**: Bounded queues and resource limits
+- **Monitoring**: Comprehensive observability for distributed operations
+- **Graceful Degradation**: System continues operating with reduced functionality
+
+### Success Metrics
+
+**Phase 1**: Structured result contracts adopted, event-driven steps working, service bus integration functional
+**Phase 2**: Multi-system workflows executing successfully, Rust core performance targets met
+**Phase 3**: AI failure analysis reducing manual intervention by 60%, natural language workflow generation in use
+**Phase 4**: No-code workflow creation by non-technical users, autonomous healing preventing 80% of degradation issues
+
+---
+
+## Long-term Impact Vision
+
+**For Developers**: Tasker becomes the de facto standard for polyglot workflow orchestration, with AI assistance making complex distributed systems accessible to broader engineering teams.
+
+**For Operations**: Autonomous healing and AI-powered diagnostics dramatically reduce operational overhead while maintaining the reliability and observability that make Tasker production-ready.
+
+**For Business**: Natural language workflow creation and no-code builders enable business stakeholders to directly participate in process automation without requiring deep technical expertise.
+
+**For the Industry**: Tasker establishes new patterns for AI-integrated distributed systems that maintain the rigor and predictability required for business-critical workflows while providing the adaptability needed for novel situations.
+
+This roadmap preserves Tasker's core strength—reliable, observable, predictable workflow orchestration—while evolving it into an intelligent, distributed platform that can coordinate complex processes across heterogeneous systems with AI-powered assistance and autonomous optimization.

data/docs/WHY.md

@@ -0,0 +1,21 @@
+# Why Tasker?
+
+That's a good question - Tasker is a pretty specialized kind of abstraction that many organizations may never really need. But as event-driven architectures become the norm, and as even smaller organizations find themselves interacting with a significant number of microservices, SaaS platforms, data stores, event queues, and the like, managing this complexity becomes a problem at scale.
+
+## Doesn't Sidekiq already exist? (or insert your favorite queuing broker)
+
+It does! I love [Sidekiq](https://sidekiq.org/) and Tasker is built on top of it. But this solves a little bit of a different problem.
+
+In event-driven architectures, it is not uncommon for the successful completion of any single "task" to actually be dependent on a significant number of "steps" - and these steps often rely on interacting with a number of different external and internal systems, whether an external API, a local datastore, or an in-house microservice. The success of a given task is therefore dependent on the successful completion of each step, and steps can likewise be dependent on other steps.
+
+The task itself may be enqueued for processing more than once, while steps are in a backoff or retry state. There are situations where a task and all of it steps may be able to be processed sequentially and successfully completed. In this case, the first time a task is enqueued, it is processed to completion, and will not be enqueued again. However, there are situations where a step's status is still in a valid state, but not complete, waiting on other steps, waiting on remote backoff requests, waiting on retrying from a remote failure, etc. When working with integrated services, APIs, etc that may fail, having retryability and resiliency around *each step* is crucial. If a step fails, it can be retried up to its retry limit, before we consider it in a final-error state. It is only a task which has one or more steps in a final-error (no retries left) that would mark a task as itself in error and no longer re-enquable. Any task that has still-viable steps that cannot be processed immediately, will simply be re-enqueued. The task and its steps retain the state of inputs, outputs, successes, and failures, so that implementing logic for different workflows do not have to repeat this logic over and over.
+
+## Consider an Example
+
+Consider a common scenario of receiving an e-commerce order in a multi-channel sales scenario, where fulfillment is managed on-site by an organization. Fulfillment systems have different data stores than the e-commerce solution, of course, but changes to an "order" in the abstract may have mutual effects on both the e-commerce representation of an order and the fulfillment order. When a change should be made to one, very frequently that change should, in some manner, propagate to both. Or, similarly, when an order is shipped, perhaps final taxes need to be calculated and reported to a tax SaaS platform, have the response data stored, and finally in total synced to a data warehouse for financial consistency. The purpose of Tasker is to make it more straightforward to enable event-driven architectures to handle multi-step tasks in a consistent and predictable way, with exposure to visibility in terms of results, status of steps, retryability, timeouts and backoffs, etc.
+
+## Technology Choices
+
+I originally developed this as a [standalone application](https://github.com/tasker-systems/tasker_rails), but it felt like this would be a really good opportunity to convert it to a Rails Engine. For my day-to-day professional life I've been working pretty deeply with microservices and domain driven design patterns, but my current role has me back in a Rails monolith - in some ways, it feels good to be home! However, if we were ever going to use something like this in my current role, we would want it to be an Engine so it could be built and maintained external to our existing architecture.
+
+For this Rails Engine, I'm not going to include a lot of the sample lower-level handlers that the standalone application has written in Rust. However, you can [checkout the writeup](https://github.com/tasker-systems/tasker_rails#technology-choices) in that app if you're interested!