ruby_reactor 0.1.0

data/documentation/async_reactors.md

@@ -0,0 +1,369 @@
+# Async Reactors
+
+RubyReactor supports two asynchronous execution models: **Full Reactor Async** and **Step-Level Async**. Both models use Sidekiq for background processing with non-blocking retry mechanisms.
+
+## Overview
+
+Async execution provides several benefits:
+
+- **Non-blocking**: Workers are freed during retry delays
+- **Scalable**: Better resource utilization with large worker pools
+- **Reliable**: Automatic retry with configurable backoff strategies
+- **Observable**: Full visibility into execution state and retry attempts
+
+## Full Reactor Async
+
+When a reactor is marked as `async true`, the entire execution happens in a Sidekiq worker, including input validation.
+
+### Configuration
+
+```ruby
+class OrderProcessingReactor < RubyReactor::Reactor
+  async true  # Enable full reactor async
+
+  step :validate_order do
+    run { validate_order_logic }
+  end
+
+  step :process_payment do
+    run { process_payment_logic }
+  end
+
+  step :send_confirmation do
+    run { send_confirmation_logic }
+  end
+end
+```
+
+### Execution Flow
+
+```ruby
+# Synchronous call returns immediately
+async_result = OrderProcessingReactor.run(order_id: 123)
+
+# Check status later
+case async_result.status
+when :pending
+  puts "Execution is queued"
+when :running
+  puts "Execution is in progress"
+when :success
+  puts "Execution completed successfully"
+  result = async_result.result
+when :failed
+  puts "Execution failed: #{async_result.error}"
+end
+```
+
+### Architecture
+
+```mermaid
+graph LR
+    A[Client] --> B[Reactor.run<br/>async: true]
+    B --> C[Validate Inputs<br/>Synchronously]
+    C --> D[Queue Sidekiq Job<br/>with Context]
+    D --> E[Sidekiq Worker]
+    E --> F[Deserialize Context]
+    F --> G[Execute All Steps<br/>Sequentially]
+    G --> H{Result?}
+    H -->|Success| I[Return Success]
+    H -->|Failure| J[Run Compensation<br/>in Worker]
+    J --> K[Return Failure]
+```
+
+```
+Client → Reactor.run() → Queue Job → Sidekiq Worker → Execute All Steps
+```
+
+## Step-Level Async
+
+Individual steps can be marked as `async: true`. Execution runs synchronously until the first async step, then hands off to a Sidekiq worker for all remaining execution.
+
+### Configuration
+
+```ruby
+class OrderProcessingReactor < RubyReactor::Reactor
+  step :validate_order do
+    # Runs synchronously
+    run { validate_order_logic }
+  end
+
+  step :process_payment, async: true do
+    # First async step - triggers handoff to worker
+    run { process_payment_logic }
+  end
+
+  step :update_inventory do
+    # Runs in worker after handoff
+    run { update_inventory_logic }
+  end
+
+  step :send_confirmation do
+    # Runs in same worker
+    run { send_confirmation_logic }
+  end
+end
+```
+
+### Execution Flow
+
+```ruby
+# Runs validate_order synchronously, then hands off
+async_result = OrderProcessingReactor.run(order_id: 123)
+
+# All remaining steps execute in a single worker
+# Compensation and rollback work within the worker context
+```
+
+### Architecture
+
+```mermaid
+graph LR
+    A[Client] --> B[Reactor.run]
+    B --> C[Execute Sync Steps<br/>Until First Async]
+    C --> D{First Async<br/>Step Found?}
+    D -->|No| E[Execute All Steps<br/>Synchronously]
+    D -->|Yes| F[Queue Sidekiq Job<br/>with Context]
+    F --> G[Sidekiq Worker]
+    G --> H[Execute Remaining Steps<br/>Sequentially]
+    H --> I{Result?}
+    I -->|Success| J[Return Success]
+    I -->|Failure| K[Run Compensation<br/>in Worker]
+    K --> L[Return Failure]
+```
+
+```
+Client → Reactor.run() → Sync Steps → Queue Job → Worker → Remaining Steps
+```
+
+## Async Steps
+
+Individual steps can be configured with `async true`, which changes the execution behavior at the point where the first async step is encountered.
+
+### Key Behavior
+
+When an async step is encountered during synchronous execution:
+
+1. **All previous steps have already executed synchronously** in the main thread
+2. **The Sidekiq job is queued** at the moment the async step would be executed
+3. **The async step itself and all subsequent steps execute** in the Sidekiq worker
+4. **The main thread returns immediately** with an async result handle
+
+### Important Distinction
+
+- **Before async step**: All execution is synchronous
+- **At async step**: Job is queued instead of executing the step
+- **After async step**: All remaining execution happens in the worker
+
+### Example
+
+```ruby
+class OrderProcessingReactor < RubyReactor::Reactor
+  step :validate_input do
+    # Executes synchronously in main thread
+    run { validate_order_input }
+  end
+
+  step :check_inventory do
+    # Executes synchronously in main thread
+    run { check_inventory_levels }
+  end
+
+  step :process_payment do
+    async true
+    # Job is queued here - this step executes in worker
+    run { process_payment_logic }
+  end
+
+  step :update_inventory do
+    # Executes in worker
+    run { update_inventory_records }
+  end
+
+  step :send_notification do
+    # Executes in worker
+    run { send_order_confirmation }
+  end
+end
+```
+
+### Execution Flow with Mermaid
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Reactor
+    participant Sidekiq
+    participant Worker
+
+    Client->>Reactor: run(order_data)
+    Reactor->>Reactor: Execute validate_input (sync)
+    Reactor->>Reactor: Execute check_inventory (sync)
+    Reactor->>Reactor: Reach process_payment (async: true)
+    Reactor->>Sidekiq: Queue job with context
+    Reactor->>Client: Return AsyncResult (pending)
+    Sidekiq->>Worker: Process job
+    Worker->>Worker: Execute process_payment
+    Worker->>Worker: Execute update_inventory
+    Worker->>Worker: Execute send_notification
+    Worker->>Sidekiq: Mark job complete
+```
+
+```mermaid
+graph TD
+    A[Client Calls Reactor.run] --> B[Execute Previous Steps Synchronously]
+    B --> C[Encounter First Async Step]
+    C --> D[Queue Sidekiq Job<br/>with Current Context]
+    D --> E[Return AsyncResult to Client<br/>Status: :pending]
+    D --> F[Sidekiq Worker Receives Job]
+    F --> G[Deserialize Context]
+    G --> H[Execute Async Step<br/>and All Subsequent Steps]
+    H --> I{Execution<br/>Successful?}
+    I -->|Yes| J[Mark AsyncResult<br/>Status: :success]
+    I -->|No| K[Run Compensation<br/>in Worker Context]
+    K --> L[Mark AsyncResult<br/>Status: :failed]
+```
+
+### Critical Points
+
+- **Synchronous Prefix Guarantee**: Steps before the first async step always complete synchronously
+- **Single Handoff Point**: Only one job is queued per reactor execution
+- **Worker Execution**: The async step and all following steps run in the same worker
+- **Context Preservation**: Execution state is serialized and passed to the worker
+- **Compensation Scope**: All compensation for failed async execution happens in the worker
+
+## Retry Configuration
+
+Both async models support sophisticated retry mechanisms with non-blocking job requeuing.
+
+### Step-Level Retry
+
+```ruby
+class PaymentProcessingReactor < RubyReactor::Reactor
+  async true
+
+  step :validate_payment do
+    retries max_attempts: 3, backoff: :exponential, base_delay: 1.second
+    run { validate_payment_logic }
+  end
+
+  step :charge_card, async: true do
+    retries max_attempts: 5, backoff: :linear, base_delay: 5.seconds
+    run { charge_card_logic }
+  end
+
+  step :update_records do
+    # No retry - critical step
+    run { update_records_logic }
+  end
+end
+```
+
+### Reactor-Level Defaults
+
+```ruby
+class PaymentProcessingReactor < RubyReactor::Reactor
+  async true
+
+  # Set defaults for all steps
+  retry_defaults max_attempts: 3, backoff: :exponential, base_delay: 2.seconds
+
+  step :validate_payment do
+    # Inherits reactor defaults (3 attempts, exponential backoff)
+    run { validate_payment_logic }
+  end
+
+  step :charge_card do
+    # Override defaults for this step
+    retries max_attempts: 5, backoff: :linear, base_delay: 10.seconds
+    run { charge_card_logic }
+  end
+end
+```
+
+## Retry Strategies
+
+### Backoff Algorithms
+
+- **`:exponential`**: Delay doubles with each attempt (1s, 2s, 4s, 8s...)
+- **`:linear`**: Delay increases linearly (5s, 10s, 15s, 20s...)
+- **`:fixed`**: Same delay for each attempt (5s, 5s, 5s, 5s...)
+
+## Error Handling and Compensation
+
+Async reactors support full compensation and rollback in the worker context:
+
+```ruby
+class OrderProcessingReactor < RubyReactor::Reactor
+  async true
+
+  step :process_payment do
+    run { process_payment_logic }
+
+    undo do |payment_id:, **|
+      # Runs in worker if execution fails later
+      PaymentService.refund(payment_id)
+    end
+
+    compensate do |payment_id:, **|
+      # Runs in worker if execution fails later
+      PaymentService.refund(payment_id)
+    end
+  end
+
+  step :update_inventory do
+    run { update_inventory_logic }
+
+    compensate do |order:, **|
+      # Runs in worker on failure
+      InventoryService.restore(order)
+    end
+  end
+end
+```
+
+## Monitoring and Observability
+
+### Job Visibility
+
+Retries are visible in Sidekiq web UI with:
+- Step name and attempt number
+- Retry delay and timing
+- Success/failure status
+- Execution context
+
+## Configuration
+
+### Sidekiq Worker Setup
+
+```ruby
+# config/sidekiq.rb
+require 'ruby_reactor/worker'
+
+RubyReactor.configure do |config|
+  config.sidekiq_queue = :default
+  config.sidekiq_retry_count = 3
+  config.logger = Logger.new('log/ruby_reactor.log')
+end
+```
+
+## Performance Considerations
+
+### Worker Pool Sizing
+
+- **Full Reactor Async**: Size pool based on total reactor throughput
+- **Step-Level Async**: Size pool based on async step frequency
+
+### Context Size Limits
+
+- Redis has job size limits (~512MB)
+- TODO: Large contexts are automatically compressed
+- Consider external storage for very large execution states
+
+### Monitoring Metrics
+
+Track these key metrics:
+- Retry attempt counts per step
+- Average retry delays
+- Success rates after retries
+- Worker utilization during peak loads

data/documentation/composition.md

@@ -0,0 +1,199 @@
+# Composition
+
+RubyReactor allows you to compose reactors within other reactors using the `compose` DSL. This enables you to build complex workflows by reusing existing reactors or defining sub-workflows inline.
+
+## Inline Composition
+
+You can define a composed reactor inline using a block. This is useful for grouping related steps or defining a sub-workflow that doesn't need to be reused elsewhere.
+
+```ruby
+class UpdateUserReactor < RubyReactor::Reactor
+  input :user_id
+  input :profile_data
+
+  step :validate_user do
+    argument :user_id, input(:user_id)
+    run { |args| ... }
+  end
+
+  # Define a sub-workflow inline
+  compose :update_profile do
+    # You can define inputs for the inline reactor
+    argument :user_id, input(:user_id)
+    argument :data, input(:profile_data)
+
+    # Configure async execution for the sub-workflow
+    async true
+    
+    # Configure retries for steps within the sub-workflow
+    retries max_attempts: 3
+
+    step :update_bio do
+      argument :user_id, input(:user_id)
+      argument :bio, input(:data, :bio)
+      run { |args| ... }
+    end
+
+    step :update_avatar do
+      argument :user_id, input(:user_id)
+      argument :avatar, input(:data, :avatar)
+      run { |args| ... }
+    end
+  end
+
+  step :notify_user do
+    wait_for :update_profile
+    run { |args| ... }
+  end
+end
+```
+
+## Class-based Composition
+
+You can also compose an existing reactor class. This is ideal for reusable workflows.
+
+```ruby
+class ProfileUpdateReactor < RubyReactor::Reactor
+  input :user_id
+  input :data
+  
+  step :update_bio do ... end
+  step :update_avatar do ... end
+end
+
+class MainReactor < RubyReactor::Reactor
+  input :user_id
+  input :profile_data
+
+  # Compose the existing reactor
+  compose :update_profile, ProfileUpdateReactor do
+    argument :user_id, input(:user_id)
+    argument :data, input(:profile_data)
+  end
+end
+```
+
+## Multiple Compose Declarations
+
+A single reactor can include multiple `compose` declarations, allowing you to orchestrate several sub-workflows. You can mix both class-based and inline compositions, and combine them with regular steps.
+
+```ruby
+class OrderProcessingReactor < RubyReactor::Reactor
+  input :order_id
+  input :customer_data
+  input :payment_info
+
+  step :validate_order do
+    argument :order_id, input(:order_id)
+    run { |args| ... }
+  end
+
+  # First compose: Class-based reactor
+  compose :update_customer_profile, CustomerProfileReactor do
+    argument :customer_data, input(:customer_data)
+  end
+
+  # Second compose: Inline reactor
+  compose :process_payment do
+    argument :order_id, input(:order_id)
+    argument :payment_info, input(:payment_info)
+    
+    async true  # This sub-workflow can run async
+    
+    step :authorize_payment do
+      run { |args| ... }
+    end
+    
+    step :capture_payment do
+      run { |args| ... }
+    end
+  end
+
+  # Third compose: Another inline reactor
+  compose :allocate_inventory do
+    argument :order_id, input(:order_id)
+    argument :order, input(:validate_order)
+    
+    step :check_availability do
+      run { |args| ... }
+    end
+    
+    step :reserve_items do
+      run { |args| ... }
+    end
+  end
+
+  step :send_confirmation do
+    # Wait for all compose steps to complete
+    wait_for :update_customer_profile, :process_payment, :allocate_inventory
+    
+    argument :customer_email, input(:customer_data)
+    argument :order_id, input(:order_id)
+    run { |args| ... }
+  end
+end
+```
+
+### Execution Flow
+
+When you have multiple `compose` declarations:
+
+1. **Execution Order**: Composed reactors execute in topological order based on their dependencies. Dependencies are determined automatically when you reference results from other steps (using `result(:step_name)`), or explicitly using `wait_for`. If no dependencies exist.
+
+2. **Access Compose Results**: A compose step returns the final result of the composed reactor. By default, this is a hash containing all of its step results, unless the composed reactor uses the `returns` DSL to specify a custom return value:
+
+```ruby
+step :final_step do
+  # Get the complete result hash from the composed reactor
+  argument :payment_result, result(:process_payment)
+  
+  run { |args| 
+    # args[:payment_result] contains the full hash: 
+    # { authorize_payment: ..., capture_payment: ... }
+    # 
+    # args[:payment_status] contains just the capture_payment result
+  }
+end
+```
+
+3. **Async Execution**: If a composed reactor is marked with `async true`, execution will pause at that compose step, serialize the entire reactor context, and queue a background job. The worker will resume execution from that compose step and continue sequentially through remaining steps. Only one worker executes the main reactor at a time.
+
+4. **Shared Context**: All composed reactors share access to the parent reactor's inputs and results of previous steps and can be configured with different retry strategies.
+
+### Async Compose Execution Flow
+
+When you mark a compose as async:
+
+```ruby
+compose :process_payment do
+  async true
+  # ... steps
+end
+```
+
+The execution flow is:
+
+1. Parent reactor executes steps up to `process_payment`
+2. Serializes entire context and queues a background job
+3. Returns `AsyncResult` to caller
+4. Worker picks up job and resumes from `process_payment`
+5. After `process_payment` completes, continues to next step sequentially
+6. If another async step is encountered, the process repeats
+
+This ensures proper ordering and state consistency across async boundaries.
+
+## Nested Async Retries
+
+One of the powerful features of composition in RubyReactor is the handling of asynchronous retries within nested reactors.
+
+When a step inside a composed reactor fails and is configured to retry asynchronously (e.g., via Sidekiq), RubyReactor ensures that the entire execution context is preserved.
+
+1.  **Context Serialization**: The entire reactor tree, including the state of the parent reactor and the composed reactor, is serialized.
+2.  **Resume on Retry**: When the retry job executes, it resumes execution exactly from the failed step within the composed reactor.
+3.  **State Preservation**: All intermediate results and inputs from the parent reactor are available, ensuring that the composed reactor has everything it needs to complete.
+
+This behavior works seamlessly whether you are using inline composition or class-based composition.
+
+## Inspection
+
+The execution state of composed reactors is stored in the parent context under `composed_contexts`. This allows for inspection of the full execution tree, which is useful for debugging and building monitoring tools.