job-workflow 0.1.3

data/guides/INSTRUMENTATION.md

@@ -0,0 +1,131 @@
+# Instrumentation
+
+JobWorkflow provides a comprehensive instrumentation system built on `ActiveSupport::Notifications`. This enables:
+
+- **Structured Logging**: Automatic JSON log output for all workflow events
+- **OpenTelemetry Integration**: Distributed tracing with span creation
+- **Custom Subscribers**: Build your own event handlers
+
+## Architecture
+
+JobWorkflow uses `ActiveSupport::Notifications` as the single event source, with subscribers handling the events:
+
+```
+┌─────────────────────┐
+│   JobWorkflow Core      │
+│  (Runner/Context)   │
+└─────────┬───────────┘
+          │ instrument("task.job_workflow", payload)
+          ▼
+┌─────────────────────────────────────┐
+│   ActiveSupport::Notifications      │
+│         (Event Bus)                 │
+└─────────┬──────────┬────────────────┘
+          │          │
+          ▼          ▼
+┌─────────────┐  ┌──────────────────┐
+│ LogSubscriber│  │ OpenTelemetry   │
+│ (built-in)  │  │   Subscriber    │
+└─────────────┘  └──────────────────┘
+```
+
+## Event Types
+
+JobWorkflow emits multiple events for each operation to support both tracing and logging:
+
+### Tracing Events (for OpenTelemetry spans)
+
+| Event Name | Description | Key Payload Fields |
+|------------|-------------|-------------------|
+| `workflow.job_workflow` | Workflow execution span | `job_name`, `job_id`, `duration_ms` |
+| `task.job_workflow` | Task execution span | `task_name`, `each_index`, `retry_count`, `duration_ms` |
+| `throttle.acquire.job_workflow` | Semaphore acquisition span | `concurrency_key`, `concurrency_limit`, `duration_ms` |
+| `dependent.wait.job_workflow` | Dependency wait span | `dependent_task_name`, `duration_ms` |
+
+### Logging Events (for structured logs)
+
+| Event Name | Description | Key Payload Fields |
+|------------|-------------|-------------------|
+| `workflow.start.job_workflow` | Workflow started | `job_name`, `job_id` |
+| `workflow.complete.job_workflow` | Workflow completed | `job_name`, `job_id` |
+| `task.start.job_workflow` | Task started | `task_name`, `each_index`, `retry_count` |
+| `task.complete.job_workflow` | Task completed | `task_name`, `each_index`, `retry_count` |
+| `task.error.job_workflow` | Task error (used by runner) | `task_name`, `error_class`, `error_message` |
+| `task.skip.job_workflow` | Task skipped | `task_name`, `reason` |
+| `task.enqueue.job_workflow` | Sub-jobs enqueued | `task_name`, `sub_job_count` |
+| `task.retry.job_workflow` | Task retry | `task_name`, `attempt`, `max_attempts`, `delay_seconds`, `error_class` |
+| `throttle.acquire.start.job_workflow` | Semaphore acquisition started | `concurrency_key`, `concurrency_limit` |
+| `throttle.acquire.complete.job_workflow` | Semaphore acquisition completed | `concurrency_key`, `concurrency_limit` |
+| `throttle.release.job_workflow` | Semaphore released | `concurrency_key`, `concurrency_limit` |
+| `dependent.wait.start.job_workflow` | Dependency wait started | `dependent_task_name` |
+| `dependent.wait.complete.job_workflow` | Dependency wait completed | `dependent_task_name` |
+
+## Custom Event Instrumentation
+
+Use `ctx.instrument` within tasks to create custom spans:
+
+```ruby
+class DataProcessingJob < ApplicationJob
+  include JobWorkflow::DSL
+
+  task :fetch_data do |ctx|
+    # Create a custom instrumented span for API calls
+    ctx.instrument("api_call", endpoint: "/users", method: "GET") do
+      HTTP.get("https://api.example.com/users")
+    end
+  end
+
+  task :process_items, each: -> (ctx) { ctx.args.items } do |ctx|
+    ctx.instrument("item_processing", item_id: ctx.each_value[:id]) do
+      process_item(ctx.each_value)
+    end
+  end
+end
+```
+
+Custom events are published as `<operation>.job_workflow` and include:
+- `job_id`, `job_name`, `task_name`, `each_index` (automatic)
+- Any custom fields you provide
+- `duration_ms` (automatic)
+
+## Subscribing to Events
+
+### Using ActiveSupport::Notifications
+
+```ruby
+# config/initializers/job_workflow_monitoring.rb
+
+# Subscribe to all JobWorkflow events
+ActiveSupport::Notifications.subscribe(/\.job_workflow$/) do |name, start, finish, id, payload|
+  duration = (finish - start) * 1000
+  Rails.logger.info("JobWorkflow event: #{name}, duration: #{duration}ms")
+end
+
+# Subscribe to specific events
+ActiveSupport::Notifications.subscribe("task.retry.job_workflow") do |*args|
+  event = ActiveSupport::Notifications::Event.new(*args)
+  Bugsnag.notify("Task retry: #{event.payload[:task_name]}")
+end
+```
+
+### Custom Metrics Collection
+
+```ruby
+# Send metrics to StatsD/Datadog
+ActiveSupport::Notifications.subscribe("task.job_workflow") do |*args|
+  event = ActiveSupport::Notifications::Event.new(*args)
+  StatsD.timing(
+    "job_workflow.task.duration",
+    event.duration,
+    tags: ["task:#{event.payload[:task_name]}", "job:#{event.payload[:job_name]}"]
+  )
+end
+
+ActiveSupport::Notifications.subscribe("task.retry.job_workflow") do |*args|
+  event = ActiveSupport::Notifications::Event.new(*args)
+  StatsD.increment(
+    "job_workflow.task.retry",
+    tags: ["task:#{event.payload[:task_name]}", "error:#{event.payload[:error_class]}"]
+  )
+end
+```

data/guides/LIFECYCLE_HOOKS.md

@@ -0,0 +1,415 @@
+# Lifecycle Hooks
+
+JobWorkflow provides lifecycle hooks to insert processing before and after task execution. Use `before`, `after`, `around`, and `on_error` hooks to implement cross-cutting concerns such as logging, validation, metrics collection, error notification, and external monitoring integration.
+
+## Hook Scope
+
+Hooks can be applied globally (to all tasks) or to specific tasks.
+
+### Global Hooks (No Task Names)
+
+When no task names are specified, the hook applies to all tasks:
+
+```ruby
+class GlobalLoggingJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  # This hook runs before EVERY task
+  before do |ctx|
+    Rails.logger.info("Starting task execution")
+  end
+  
+  # This hook runs after EVERY task
+  after do |ctx|
+    Rails.logger.info("Task execution completed")
+  end
+  
+  task :first_task do |ctx|
+    # before and after hooks run here
+  end
+  
+  task :second_task do |ctx|
+    # before and after hooks also run here
+  end
+end
+```
+
+### Task-Specific Hooks (Single Task)
+
+Specify a task name to apply the hook only to that task:
+
+```ruby
+before :validate_order do |ctx|
+  # Only runs before :validate_order task
+end
+```
+
+### Multiple Task Hooks (Variable-Length Arguments)
+
+Specify multiple task names to apply the same hook to several tasks:
+
+```ruby
+before :task_a, :task_b, :task_c do |ctx|
+  # Runs before each of :task_a, :task_b, and :task_c
+end
+
+around :fetch_users, :fetch_orders, :fetch_products do |ctx, task|
+  start_time = Time.current
+  task.call
+  Metrics.timing("api.duration", Time.current - start_time)
+end
+```
+
+## Hook Types
+
+### before Hook
+
+Execute processing before task execution.
+
+```ruby
+class ValidationWorkflowJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  argument :order_id, "Integer"
+  
+  # Run validation in before hook
+  before :charge_payment do |ctx|
+    order = Order.find(ctx.arguments.order_id)
+    
+    # Check inventory
+    raise "Out of stock" unless order.items_in_stock?
+    
+    # Verify credit card
+    raise "Invalid card" unless order.valid_credit_card?
+  end
+  
+  task :charge_payment, output: { payment_id: "String" } do |ctx|
+    # Executes after validation passes
+    order_id = ctx.arguments.order_id
+    { payment_id: PaymentGateway.charge(order_id) }
+  end
+end
+```
+
+### after Hook
+
+Execute processing after task execution.
+
+```ruby
+class NotificationWorkflowJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  argument :user_id, "Integer"
+  
+  task :perform_action, output: { action_result: "Hash" } do |ctx|
+    user_id = ctx.arguments.user_id
+    { action_result: SomeService.perform(user_id) }
+  end
+  
+  # Send notification in after hook
+  after :perform_action do |ctx|
+    user_id = ctx.arguments.user_id
+    action_result = ctx.output[:perform_action].first.action_result
+    
+    UserMailer.action_completed(
+      user_id,
+      action_result
+    ).deliver_later
+    
+    # Record analytics
+    Analytics.track('action_completed', {
+      user_id: user_id,
+      result: action_result
+    })
+  end
+end
+```
+
+### around Hook
+
+Execute processing that wraps task execution. **Important:** You must call `task.call` to execute the task.
+
+```ruby
+class MetricsWorkflowJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  # Measure execution time
+  around :expensive_task do |ctx, task|
+    start_time = Time.current
+    
+    Rails.logger.info("Starting expensive_task")
+    
+    # Execute task - THIS IS REQUIRED
+    task.call
+    
+    duration = Time.current - start_time
+    Rails.logger.info("expensive_task completed in #{duration}s")
+    
+    # Send metrics
+    Metrics.timing('task.duration', duration, tags: {
+      task: 'expensive_task'
+    })
+  end
+  
+  task :expensive_task, output: { result: "String" } do |ctx|
+    { result: heavy_computation }
+  end
+end
+```
+
+## Execution Order
+
+Hooks are executed in definition order. When multiple hooks apply to a task, they execute as follows:
+
+```ruby
+class OrderedHooksJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  before do |ctx|
+    puts "1. Global before"
+  end
+  
+  before :my_task do |ctx|
+    puts "2. Task-specific before"
+  end
+  
+  around do |ctx, task|
+    puts "3. Global around (before)"
+    task.call
+    puts "6. Global around (after)"
+  end
+  
+  around :my_task do |ctx, task|
+    puts "4. Task-specific around (before)"
+    task.call
+    puts "5. Task-specific around (after)"
+  end
+  
+  task :my_task do |ctx|
+    puts "--- Task execution ---"
+  end
+  
+  after :my_task do |ctx|
+    puts "7. Task-specific after"
+  end
+  
+  after do |ctx|
+    puts "8. Global after"
+  end
+end
+
+# Output:
+# 1. Global before
+# 2. Task-specific before
+# 3. Global around (before)
+# 4. Task-specific around (before)
+# --- Task execution ---
+# 5. Task-specific around (after)
+# 6. Global around (after)
+# 7. Task-specific after
+# 8. Global after
+```
+
+## around Hook: task.call is Required
+
+In around hooks, you **must** call `task.call` to execute the task. If you forget to call it, JobWorkflow raises `TaskCallable::NotCalledError`:
+
+```ruby
+# ❌ BAD: Missing task.call
+around :my_task do |ctx, task|
+  puts "Before task"
+  # Forgot task.call!
+  puts "After task"
+end
+# => Raises: JobWorkflow::TaskCallable::NotCalledError:
+#    around hook for 'my_task' did not call task.call
+
+# ✅ GOOD: Properly calling task.call
+around :my_task do |ctx, task|
+  puts "Before task"
+  task.call  # Required!
+  puts "After task"
+end
+```
+
+Additionally, `task.call` can only be called once. Calling it multiple times raises `TaskCallable::AlreadyCalledError`:
+
+```ruby
+# ❌ BAD: Calling task.call multiple times
+around :my_task do |ctx, task|
+  task.call
+  task.call  # => Raises: JobWorkflow::TaskCallable::AlreadyCalledError
+end
+```
+
+## on_error Hook
+
+Execute processing when a task raises an exception. This hook is ideal for error notification, external monitoring integration, and error tracking.
+
+**Important:** `on_error` hooks do not suppress exceptions - they are for notification purposes only. After all hooks execute, the exception is re-raised.
+
+```ruby
+class ErrorNotificationWorkflowJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  argument :user_id, "Integer"
+  
+  # Global error hook - called for any task error
+  on_error do |ctx, exception, task|
+    ErrorNotificationService.notify(
+      exception: exception,
+      context: {
+        workflow: self.class.name,
+        task: task.task_name,
+        arguments: ctx.arguments.to_h
+      }
+    )
+  end
+  
+  # Task-specific error hook
+  on_error :critical_payment do |ctx, exception, task|
+    # Critical tasks get special handling
+    CriticalErrorHandler.handle(
+      task: task.task_name,
+      exception: exception,
+      severity: :high
+    )
+  end
+  
+  task :fetch_user, output: { user: "Hash" } do |ctx|
+    user = User.find(ctx.arguments.user_id)
+    { user: user.attributes }
+  end
+  
+  task :critical_payment, output: { payment_id: "String" } do |ctx|
+    # If this fails, both global and task-specific hooks run
+    payment = PaymentGateway.charge(ctx.arguments.user_id)
+    { payment_id: payment.id }
+  end
+end
+```
+
+### on_error Hook Parameters
+
+The `on_error` hook receives three parameters:
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `ctx` | `Context` | The workflow context at the time of failure |
+| `exception` | `StandardError` | The exception that was raised |
+| `task` | `Task` | The task object that failed |
+
+### Hook Execution Order
+
+When a task fails, error hooks execute in definition order (global first, then task-specific):
+
+```ruby
+class MultipleErrorHooksJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  on_error do |ctx, error, task|
+    puts "1. Global error handler"
+  end
+  
+  on_error :my_task do |ctx, error, task|
+    puts "2. Task-specific error handler"
+  end
+  
+  task :my_task do |ctx|
+    raise "Something went wrong"
+  end
+end
+
+# When :my_task fails, output:
+# 1. Global error handler
+# 2. Task-specific error handler
+# => Then RuntimeError is re-raised
+```
+
+### Practical Use Cases
+
+**Error Tracking Service:**
+```ruby
+on_error do |ctx, exception, task|
+  ErrorTracker.capture(exception, metadata: {
+    workflow: self.class.name,
+    task: task.task_name,
+    job_id: ctx.job_id
+  })
+end
+```
+
+**Real-time Alert Notification:**
+```ruby
+on_error :critical_task do |ctx, exception, task|
+  AlertService.notify(
+    severity: :critical,
+    message: "Task #{task.task_name} failed: #{exception.message}",
+    metadata: { workflow: self.class.name }
+  )
+end
+```
+
+**Structured Error Logging:**
+```ruby
+on_error do |ctx, exception, task|
+  Rails.logger.error({
+    event: "task_failure",
+    task: task.task_name,
+    error_class: exception.class.name,
+    error_message: exception.message,
+    backtrace: exception.backtrace&.first(10)
+  }.to_json)
+end
+```
+
+## Error Handling
+
+| Hook Type | Behavior on Exception |
+|-----------|----------------------|
+| `before` | Task is skipped, exception is re-raised |
+| `after` | Exception is re-raised (task result is preserved) |
+| `around` | Exception is re-raised |
+| `on_error` | Executes on task failure, then exception is re-raised |
+
+```ruby
+class ErrorHandlingJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  # If before hook raises, task won't execute
+  before :process_order do |ctx|
+    order = Order.find(ctx.arguments.order_id)
+    raise "Out of stock" unless order.items_in_stock?
+  end
+  
+  task :process_order do |ctx|
+    # Only executes if validation passes
+    OrderProcessor.process(ctx.arguments.order_id)
+  end
+end
+```
+
+## Hooks with Map Tasks (each/concurrency)
+
+When using hooks with map tasks (`each` or `concurrency`), the hooks execute for **each iteration**:
+
+```ruby
+class BatchWithHooksJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  argument :user_ids, "Array[Integer]"
+  
+  # This hook runs for EACH user
+  before :fetch_users do |ctx|
+    Rails.logger.info("Fetching user: #{ctx.each_value}")
+  end
+  
+  task :fetch_users,
+       each: ->(ctx) { ctx.arguments.user_ids },
+       output: { user: "Hash" } do |ctx|
+    { user: UserAPI.fetch(ctx.each_value) }
+  end
+end
+
+# With user_ids: [1, 2, 3], the before hook runs 3 times
+```

data/guides/NAMESPACES.md

@@ -0,0 +1,75 @@
+# Namespaces
+
+Logically grouping tasks improves readability and maintainability of complex workflows. JobWorkflow provides namespace functionality.
+
+## Basic Namespaces
+
+### namespace DSL
+
+Group related tasks.
+
+```ruby
+class ECommerceOrderJob < ApplicationJob
+  include JobWorkflow::DSL
+  
+  argument :order, "Order"
+  
+  # Payment-related tasks
+  namespace :payment do
+    task :validate do |ctx|
+      order = ctx.arguments.order
+      PaymentValidator.validate(order)
+    end
+    
+    task :charge, depends_on: [:"payment:validate"], output: { payment_result: "Hash" } do |ctx|
+      order = ctx.arguments.order
+      { payment_result: PaymentProcessor.charge(order) }
+    end
+    
+    task :send_receipt, depends_on: [:"payment:charge"] do |ctx|
+      order = ctx.arguments.order
+      payment_result = ctx.output[:"payment:charge"].first.payment_result
+      ReceiptMailer.send(order, payment_result)
+    end
+  end
+  
+  # Inventory-related tasks
+  namespace :inventory do
+    task :check_availability do |ctx|
+      order = ctx.arguments.order
+      InventoryService.check(order.items)
+    end
+    
+    task :reserve, depends_on: [:"inventory:check_availability"], output: { reserved: "bool" } do |ctx|
+      order = ctx.arguments.order
+      { reserved: InventoryService.reserve(order.items) }
+    end
+  end
+  
+  # Shipping-related tasks
+  namespace :shipping do
+    task :calculate_cost, output: { shipping_cost: "Float" } do |ctx|
+      order = ctx.arguments.order
+      { shipping_cost: ShippingCalculator.calculate(order) }
+    end
+    
+    task :create_label, depends_on: [:"shipping:calculate_cost"], output: { shipping_label: "String" } do |ctx|
+      order = ctx.arguments.order
+      { shipping_label: ShippingService.create_label(order) }
+    end
+  end
+end
+```
+
+Tasks in namespaces are identified as `:namespace:task_name` at runtime:
+
+```ruby
+# Executed tasks:
+# - :payment:validate
+# - :payment:charge
+# - :payment:send_receipt
+# - :inventory:check_availability
+# - :inventory:reserve
+# - :shipping:calculate_cost
+# - :shipping:create_label
+```

data/guides/OPENTELEMETRY_INTEGRATION.md

@@ -0,0 +1,86 @@
+# OpenTelemetry Integration
+
+JobWorkflow provides optional OpenTelemetry integration for distributed tracing. When enabled, all workflow and task executions create OpenTelemetry spans.
+
+## Prerequisites
+
+Install the OpenTelemetry gems:
+
+```ruby
+# Gemfile
+gem 'opentelemetry-api'
+gem 'opentelemetry-sdk'
+gem 'opentelemetry-exporter-otlp'  # Or your preferred exporter
+```
+
+## Configuration
+
+```ruby
+# config/initializers/opentelemetry.rb
+require 'opentelemetry/sdk'
+require 'opentelemetry/exporter/otlp'
+
+OpenTelemetry::SDK.configure do |c|
+  c.service_name = 'my-application'
+  c.use_all  # Auto-instrument Rails, HTTP clients, etc.
+end
+
+# Enable JobWorkflow OpenTelemetry integration
+JobWorkflow::Instrumentation::OpenTelemetrySubscriber.subscribe!
+```
+
+## Span Attributes
+
+JobWorkflow spans include the following attributes:
+
+| Attribute | Description |
+|-----------|-------------|
+| `job_workflow.job.name` | Job class name |
+| `job_workflow.job.id` | Unique job identifier |
+| `job_workflow.task.name` | Task name |
+| `job_workflow.task.each_index` | Index in map task iteration |
+| `job_workflow.task.retry_count` | Current retry attempt |
+| `job_workflow.concurrency.key` | Throttle concurrency key |
+| `job_workflow.concurrency.limit` | Throttle concurrency limit |
+| `job_workflow.error.class` | Exception class (on error) |
+| `job_workflow.error.message` | Exception message (on error) |
+
+## Span Naming
+
+Spans are named based on the event type:
+
+- `DataProcessingJob workflow` - Workflow execution
+- `DataProcessingJob.fetch_data task` - Task execution
+- `DataProcessingJob.process_items task` - Map task execution
+- `JobWorkflow throttle.acquire` - Throttle acquisition
+- `JobWorkflow dependent.wait` - Dependency waiting
+
+## Viewing Traces
+
+Configure your preferred backend (Jaeger, Zipkin, Honeycomb, Datadog, etc.):
+
+```ruby
+# Example: OTLP exporter
+OpenTelemetry::SDK.configure do |c|
+  c.add_span_processor(
+    OpenTelemetry::SDK::Trace::Export::BatchSpanProcessor.new(
+      OpenTelemetry::Exporter::OTLP::Exporter.new(
+        endpoint: 'http://localhost:4318/v1/traces'
+      )
+    )
+  )
+end
+```
+
+## Disabling OpenTelemetry
+
+To disable OpenTelemetry integration:
+
+```ruby
+# Unsubscribe from all events
+JobWorkflow::Instrumentation::OpenTelemetrySubscriber.unsubscribe!
+```
+
+## Error Handling
+
+OpenTelemetry subscriber errors are handled gracefully and do not affect workflow execution. Errors are reported via `OpenTelemetry.handle_error` if available.