job-workflow 0.1.3

data/guides/PARALLEL_PROCESSING.md

@@ -0,0 +1,302 @@
+# Parallel Processing
+
+JobWorkflow enables parallel processing of collection elements by specifying the `each:` option in a `task` definition. Based on the Fork-Join pattern, it provides efficient and safe parallel execution.
+
+## Collection Task Basics
+
+### Simple Parallel Processing
+
+```ruby
+class BatchProcessingJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  argument :user_ids, "Array[Integer]", default: []
+  
+  # Prepare user IDs
+  task :fetch_user_ids, output: { ids: "Array[Integer]" } do |ctx|
+    { ids: User.active.pluck(:id) }
+  end
+  
+  # Process each user in parallel
+  task :process_users,
+       each: ->(ctx) { ctx.arguments.user_ids },
+       depends_on: [:fetch_user_ids],
+       output: { user_id: "Integer", status: "Symbol" } do |ctx|
+    user_id = ctx.each_value
+    user = User.find(user_id)
+    {
+      user_id: user_id,
+      status: user.process!
+    }
+  end
+  
+  # Aggregate results
+  task :aggregate_results, depends_on: [:process_users] do |ctx|
+    results = ctx.output[:process_users]
+    puts "Processed #{results.size} users"
+    # => [{ user_id: 1, status: :ok }, { user_id: 2, status: :ok }, ...]
+  end
+end
+```
+
+## Controlling Concurrency
+
+### Synchronous Execution (Default)
+
+By default, map tasks execute synchronously (in-process):
+
+```ruby
+# Synchronous map task (default)
+# All iterations execute sequentially in the current job
+task :process_items,
+     each: ->(ctx) { ctx.arguments.items } do |ctx|
+  process_item(ctx.each_value)
+end
+```
+
+### Asynchronous Execution with Concurrency
+
+To execute map task iterations in separate sub-jobs with concurrency control, use the `enqueue:` option with a Hash containing `condition:` and `concurrency:`:
+
+```ruby
+# Simplest form: enable parallel execution with default settings
+task :process_items,
+     each: ->(ctx) { ctx.arguments.items },
+     enqueue: true do |ctx|
+  process_item(ctx.each_value)
+end
+
+# Process up to 10 items concurrently in sub-jobs
+task :process_items,
+     each: ->(ctx) { ctx.arguments.items },
+     enqueue: { condition: ->(_ctx) { true }, concurrency: 10 } do |ctx|
+  process_item(ctx.each_value)
+end
+
+# Simplified syntax when condition is implicitly true
+task :process_items,
+     each: ->(ctx) { ctx.arguments.items },
+     enqueue: { concurrency: 10 } do |ctx|
+  process_item(ctx.each_value)
+end
+
+# When enqueue is enabled:
+# - Each iteration is executed in a separate sub-job
+# - Sub-jobs are created via perform_all_later
+# - Concurrency limit controls how many sub-jobs run in parallel
+# - Parent job waits for all sub-jobs to complete before continuing
+# - Outputs from sub-jobs are automatically collected
+```
+
+### Understanding `enqueue` Option
+
+The `enqueue:` option determines how map task iterations are executed:
+
+- **`enqueue:` is nil/false (default)**: Iterations execute synchronously in the current job
+  - Simple and fast for small datasets
+  - Good for CPU-bound operations
+  - No network overhead
+
+- **`enqueue: true`**: Each iteration is enqueued as a separate sub-job with default settings
+  - Simplest way to enable parallel execution
+  - No concurrency limit (executes as fast as workers allow)
+  - Good for I/O-bound operations with many workers
+
+- **`enqueue: { condition: ->(_ctx) { true }, concurrency: 10 }`**: Each iteration is enqueued as a separate sub-job
+  - Enables true parallel execution across multiple workers
+  - Better for I/O-bound operations (API calls, database queries)
+  - Can accept dynamic condition: `enqueue: { condition: ->(ctx) { ctx.arguments.use_concurrency? } }`
+  - Supports `queue:` option for custom queue: `enqueue: { queue: "critical", concurrency: 5 }`
+
+**Note**: `enqueue:` works with both regular tasks and map tasks. For map tasks, it enables asynchronous sub-job execution. For regular tasks, it allows conditional enqueueing as a separate job. Legacy syntax (`enqueue: ->(_ctx) { true }` as a Proc) is still supported for backward compatibility.
+
+## Fork-Join Pattern
+
+### Context Isolation
+
+Each parallel task has access to the same Context instance. Arguments are immutable and outputs should be returned:
+
+```ruby
+argument :items, "Array[Hash]"
+argument :shared_config, "Hash"
+
+task :parallel_processing,
+     each: ->(ctx) { ctx.arguments.items },
+     output: { item_result: "String" } do |ctx|
+  # Access current element via ctx.each_value
+  item = ctx.each_value
+  
+  # Can read arguments (immutable)
+  config = ctx.arguments.shared_config
+  
+  # Return output for this iteration
+  { item_result: process(item, config) }
+end
+```
+
+### Accessing Current Element
+
+When using `each:` option, access the current element via `ctx.each_value`:
+
+```ruby
+task :process_items,
+     each: ->(ctx) { ctx.arguments.items } do |ctx|
+  item = ctx.each_value  # Get current element
+  process(item)
+end
+```
+
+**Important**: `ctx.each_value` can only be called within Map Tasks (tasks with `each:` option). Calling it in regular tasks will raise an error:
+
+```ruby
+task :regular_task do |ctx|
+  ctx.each_value  # ❌ Error: "each_value can be called only within each_values block"
+end
+
+task :map_task,
+     each: ->(ctx) { ctx.arguments.items } do |ctx|
+  ctx.each_value  # ✅ OK: Returns current element
+end
+```
+
+## Matrix Processing (Multi-Axis Parallelization)
+
+### Cartesian Product Execution
+
+For scenarios where you need to process all combinations of multiple dimensions (e.g., all regions × all data types), use nested `each:` options to create a Cartesian product pattern:
+
+```ruby
+class DataProcessingJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  argument :regions, "Array[String]"
+  argument :data_types, "Array[String]"
+  
+  # First axis: process each region
+  task :process_by_region,
+       each: ->(ctx) { ctx.arguments.regions },
+       output: { region: "String", results: "Array[Hash]" },
+       enqueue: { concurrency: 5 } do |ctx|
+    region = ctx.each_value
+    # This will create sub-tasks for each region
+    { region: region, results: [] }
+  end
+  
+  # Second axis: for each region result, process each data type
+  # This creates a nested loop: 3 regions × 3 data types = 9 combinations
+  task :process_matrix,
+       each: ->(ctx) {
+         # Create combinations from first task's output
+         regions_data = ctx.output[:process_by_region]
+         data_types = ctx.arguments.data_types
+         
+         # Generate all combinations
+         regions_data.flat_map do |region_result|
+           data_types.map do |data_type|
+             { region: region_result[:region], data_type: data_type }
+           end
+         end
+       },
+       depends_on: [:process_by_region],
+       output: { region: "String", data_type: "String", result: "Hash" },
+       enqueue: { concurrency: 10 } do |ctx|
+    item = ctx.each_value
+    region = item[:region]
+    data_type = item[:data_type]
+    
+    # Process specific region + data_type combination
+    processed = process_data(region, data_type)
+    
+    {
+      region: region,
+      data_type: data_type,
+      result: processed
+    }
+  end
+  
+  # Aggregate all results
+  task :aggregate_matrix_results, depends_on: [:process_matrix] do |ctx|
+    results = ctx.output[:process_matrix]
+    # results is an array of 9 hashes, one per combination
+    summary = results.group_by { |r| r[:region] }
+    { summary: summary }
+  end
+end
+
+# Execution example
+DataProcessingJob.perform_later(
+  regions: ["us-east-1", "us-west-1", "eu-west-1"],
+  data_types: ["user", "order", "product"]
+)
+# => 3 regions × 3 data types = 9 parallel iterations (with concurrency limits)
+```
+
+### Advanced Matrix with Filtering
+
+When certain combinations should be excluded (e.g., avoid processing "legacy" data in "us-west-1"), filter the combinations:
+
+```ruby
+argument :regions, "Array[String]"
+argument :data_types, "Array[String]"
+
+task :process_filtered_matrix,
+     each: ->(ctx) {
+       regions = ctx.arguments.regions
+       data_types = ctx.arguments.data_types
+       
+       # Create combinations with explicit filtering
+       combinations = regions.flat_map do |region|
+         data_types.map { |data_type| { region: region, data_type: data_type } }
+       end
+       
+       # Exclude specific combinations
+       combinations.reject do |combo|
+         (combo[:region] == "us-west-1" && combo[:data_type] == "legacy") ||
+         (combo[:region] == "eu-west-1" && combo[:data_type] == "beta")
+       end
+     },
+     output: { region: "String", data_type: "String", status: "Symbol" },
+     enqueue: { concurrency: 10 } do |ctx|
+  combo = ctx.each_value
+  region = combo[:region]
+  data_type = combo[:data_type]
+  
+  {
+    region: region,
+    data_type: data_type,
+    status: process(region, data_type)
+  }
+end
+```
+
+### Performance Considerations
+
+When implementing matrix processing:
+
+1. **Concurrency Control**: Set appropriate `concurrency:` limits to avoid overwhelming workers
+   - High concurrency (20+): Suitable for I/O-bound operations (API calls, database queries)
+   - Low concurrency (2-5): Better for CPU-bound operations or rate-limited APIs
+
+2. **Output Size**: Watch out for large output collections
+   - With N×M combinations, the output array will have N×M elements
+   - Consider using a storage adapter (see [TASK_OUTPUTS.md](TASK_OUTPUTS.md#storage-adapters)) for large datasets
+
+3. **Timeout Settings**: Increase timeout for complex matrix operations
+   ```ruby
+   task :process_matrix,
+        each: ->(_ctx) { combinations },
+        timeout: 300.seconds,  # 5 minutes per iteration
+        enqueue: { concurrency: 5 } do |ctx|
+     # ...
+   end
+   ```
+
+4. **Error Handling**: Consider retry strategies for flaky matrix operations
+   ```ruby
+   task :process_matrix,
+        each: ->(_ctx) { combinations },
+        retry: { count: 3, strategy: :exponential },
+        enqueue: { concurrency: 5 } do |ctx|
+     # ...
+   end
+   ```

data/guides/PRODUCTION_DEPLOYMENT.md

@@ -0,0 +1,110 @@
+# Production Deployment
+
+> ⚠️ **Early Stage (v0.1.3):** JobWorkflow is still in early development. While this section outlines potential deployment patterns, please thoroughly test in your specific environment and monitor for any issues before relying on JobWorkflow in critical production systems.
+
+This section covers suggested settings and patterns for running JobWorkflow in production-like environments.
+
+## SolidQueue Configuration
+
+### Basic Configuration
+
+```ruby
+# config/application.rb
+config.active_job.queue_adapter = :solid_queue
+
+# config/queue.yml
+production:
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+  workers:
+    - queues: default
+      threads: 5
+      processes: 3
+      polling_interval: 0.1
+```
+
+### Optimizing Worker Processes
+
+```ruby
+# config/queue.yml
+production:
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+  
+  workers:
+    # High priority queue (orchestrator)
+    - queues: orchestrator
+      threads: 3
+      processes: 2
+      polling_interval: 0.1
+    
+    # Normal priority queue (child jobs)
+    - queues: default
+      threads: 10
+      processes: 5
+      polling_interval: 0.5
+    
+    # Low priority queue (batch processing)
+    - queues: batch
+      threads: 5
+      processes: 2
+      polling_interval: 1
+```
+
+## Auto Scaling (AWS ECS)
+
+JobWorkflow provides a simple autoscaling helper that updates an AWS ECS service `desired_count` based on queue latency.
+
+### Prerequisites
+
+- Currently supports **AWS ECS only** via `JobWorkflow::AutoScaling::Adapter::AwsAdapter`.
+- The autoscaling job must run **inside an ECS task** (uses ECS metadata via `ECS_CONTAINER_METADATA_URI_V4`).
+- Latency is read via `JobWorkflow::Queue.latency` which uses the configured queue adapter.
+- Scheduling (how often you evaluate scaling) is **out of scope**: enqueue this job periodically from your app/ops tooling.
+
+### Usage
+
+Create a job for autoscaling and configure it via `include JobWorkflow::AutoScaling`.
+
+```ruby
+class MyAutoScalingJob < ApplicationJob
+  include JobWorkflow::AutoScaling
+
+  # Target queue name
+  target_queue_name "default"
+
+  # desired_count range
+  min_count 2
+  max_count 10
+
+  # Scale step (e.g. 2 => 2,4,6...)
+  step_count 2
+
+  # Max latency (seconds). Scaling reaches max_count around this value.
+  max_latency 1800
+end
+```
+
+### Scaling model
+
+- Queue latency is bucketed into $0..max_latency$ and scaled from `min_count` to `max_count` by `step_count`.
+- Latency is retrieved via `JobWorkflow::Queue.latency(queue_name)`, which delegates to the configured queue adapter.
+
+## SolidCache Configuration
+
+### Basic Configuration
+
+```ruby
+# config/environments/production.rb
+config.cache_store = :solid_cache_store, {
+  expires_in: 1.day,
+  namespace: "myapp_production",
+  error_handler: ->(method:, returning:, exception:) {
+    Rails.logger.error "[SolidCache] Error in #{method}: #{exception.message}"
+    # Send to your error tracking service
+    ErrorTracker.capture(exception)
+  }
+}
+```

data/guides/QUEUE_MANAGEMENT.md

@@ -0,0 +1,141 @@
+# Queue Management
+
+JobWorkflow provides a unified interface for managing job queues through `JobWorkflow::Queue`. This abstraction works with any supported queue adapter (SolidQueue, etc.) and provides operations for monitoring and controlling queue behavior.
+
+## Basic Queue Operations
+
+### Checking Queue Status
+
+```ruby
+# Get current latency (time oldest job has been waiting)
+latency = JobWorkflow::Queue.latency(:default)  # => 5.2 (seconds)
+
+# Get queue size (number of pending jobs)
+size = JobWorkflow::Queue.size(:default)  # => 42
+
+# Clear all jobs from a queue (use with caution!)
+JobWorkflow::Queue.clear(:batch_processing)
+```
+
+## Queue Pause/Resume
+
+You can pause and resume job processing at the queue level. This is useful for:
+- Maintenance windows
+- Emergency stops when downstream services are unavailable
+- Controlled deployment rollouts
+
+### Pausing a Queue
+
+```ruby
+# Pause a queue - new jobs will be enqueued but not processed
+JobWorkflow::Queue.pause(:default)
+
+# Check if a queue is paused
+JobWorkflow::Queue.paused?(:default)  # => true
+
+# List all paused queues
+JobWorkflow::Queue.paused_queues  # => [:default]
+```
+
+### Resuming a Queue
+
+```ruby
+# Resume processing
+JobWorkflow::Queue.resume(:default)
+
+JobWorkflow::Queue.paused?(:default)  # => false
+```
+
+### Instrumentation Events
+
+Queue pause/resume operations emit instrumentation events that you can subscribe to:
+
+```ruby
+# Events emitted:
+# - queue.pause.job_workflow   (when a queue is paused)
+# - queue.resume.job_workflow  (when a queue is resumed)
+
+# Example: Custom notification on pause
+ActiveSupport::Notifications.subscribe("queue.pause.job_workflow") do |event|
+  SlackNotifier.notify("Queue #{event.payload[:queue_name]} has been paused")
+end
+```
+
+## Finding Workflows by Queue
+
+You can discover which workflow classes are configured to use a specific queue:
+
+```ruby
+# Get all workflow classes that use the :default queue
+workflows = JobWorkflow::Queue.workflows(:default)
+# => [OrderProcessingJob, UserRegistrationJob, ...]
+
+# Useful for impact analysis before pausing a queue
+JobWorkflow::Queue.workflows(:batch).each do |workflow_class|
+  puts "#{workflow_class.name} uses the batch queue"
+end
+```
+
+## Production Considerations
+
+### Pause/Resume Best Practices
+
+1. **Always notify stakeholders** before pausing production queues
+2. **Monitor queue size** while paused to avoid backlog buildup
+3. **Use instrumentation** to track pause/resume events in your observability stack
+4. **Test resume behavior** - ensure workers pick up jobs promptly after resume
+
+### Queue Design Patterns
+
+```ruby
+# Separate queues for different reliability requirements
+class CriticalPaymentJob < ApplicationJob
+  include JobWorkflow::DSL
+  queue_as :payments  # High-priority, rarely paused
+  # ...
+end
+
+class BatchReportJob < ApplicationJob
+  include JobWorkflow::DSL
+  queue_as :batch  # Low-priority, can be paused during peak hours
+  # ...
+end
+
+# Maintenance script example
+class MaintenanceService
+  def self.pause_non_critical_queues
+    [:batch, :reports, :notifications].each do |queue|
+      JobWorkflow::Queue.pause(queue)
+      Rails.logger.info "Paused queue: #{queue}"
+    end
+  end
+  
+  def self.resume_all_queues
+    JobWorkflow::Queue.paused_queues.each do |queue|
+      JobWorkflow::Queue.resume(queue)
+      Rails.logger.info "Resumed queue: #{queue}"
+    end
+  end
+end
+```
+
+### Monitoring Paused Queues
+
+```ruby
+# Health check endpoint
+class HealthController < ApplicationController
+  def queues
+    critical_queues = [:default, :payments]
+    paused_critical = critical_queues & JobWorkflow::Queue.paused_queues
+    
+    if paused_critical.any?
+      render json: { 
+        status: "warning", 
+        paused_critical_queues: paused_critical 
+      }, status: :service_unavailable
+    else
+      render json: { status: "ok" }
+    end
+  end
+end
+```