fractor 0.1.6 → 0.1.8

data/examples/workflow/dead_letter_queue/README.adoc

@@ -0,0 +1,374 @@
+= Dead Letter Queue Workflow Example
+:toc:
+:toclevels: 3
+
+== Purpose
+
+This example demonstrates how to use Dead Letter Queue (DLQ) functionality in Fractor workflows to capture and manage permanently failed work items. The DLQ provides a safety net for work that cannot be successfully processed even after retry attempts.
+
+== What is a Dead Letter Queue?
+
+A Dead Letter Queue is a holding area for work items that have failed permanently and cannot be processed successfully. Instead of losing failed work or letting it crash the system, the DLQ:
+
+* Captures failed work items with full context
+* Preserves error information and retry history
+* Enables manual inspection and retry
+* Provides persistence options for durability
+* Supports custom notification handlers
+
+== Use Cases
+
+* *Error Analysis*: Inspect failed work to identify patterns and root causes
+* *Manual Recovery*: Review and manually retry failed items after fixing issues
+* *Alerting*: Trigger notifications when work fails permanently
+* *Audit Trail*: Maintain a record of all failures with context
+* *Batch Retry*: Retry multiple failed items after deploying fixes
+
+== Features Demonstrated
+
+=== Basic Dead Letter Queue
+
+The simplest configuration captures failed work automatically:
+
+[source,ruby]
+----
+class BasicDLQWorkflow
+  include Fractor::Workflow
+
+  workflow_name "basic_dlq_workflow"
+
+  # Configure Dead Letter Queue
+  configure_dead_letter_queue max_size: 100
+
+  job :unreliable_task,
+      worker: UnreliableWorker,
+      input: ->(ctx) { ctx.workflow_input },
+      retry: {
+        max_attempts: 3,
+        backoff: :exponential,
+        initial_delay: 0.1,
+      }
+
+  end_job :unreliable_task
+end
+----
+
+When retries are exhausted, the work is automatically added to the DLQ with:
+
+* The failed work item
+* The error that caused the failure
+* Workflow context (inputs, completed/failed jobs)
+* Retry metadata (attempts, total time, all errors)
+
+=== Custom Notification Handlers
+
+Add callbacks to be notified when work is added to the DLQ:
+
+[source,ruby]
+----
+class DLQWithHandlersWorkflow
+  include Fractor::Workflow
+
+  workflow_name "dlq_with_handlers_workflow"
+
+  configure_dead_letter_queue(
+    max_size: 50,
+    on_add: lambda { |entry|
+      # Send alert, log to monitoring system, etc.
+      puts "⚠️  Work added to DLQ:"
+      puts "   Error: #{entry.error.class.name}: #{entry.error.message}"
+      puts "   Timestamp: #{entry.timestamp}"
+      puts "   Metadata: #{entry.metadata.inspect}"
+    }
+  )
+
+  job :risky_task,
+      worker: UnreliableWorker,
+      input: ->(ctx) { ctx.workflow_input },
+      retry: {
+        max_attempts: 2,
+        backoff: :linear,
+        initial_delay: 0.1,
+      }
+
+  end_job :risky_task
+end
+----
+
+=== File Persistence
+
+Persist DLQ entries to disk for durability across restarts:
+
+[source,ruby]
+----
+class DLQWithPersistenceWorkflow
+  include Fractor::Workflow
+
+  workflow_name "dlq_with_persistence_workflow"
+
+  configure_dead_letter_queue(
+    max_size: 200,
+    persister: Fractor::Workflow::DeadLetterQueue::FilePersister.new(
+      directory: "tmp/dlq"
+    )
+  )
+
+  job :persistent_task,
+      worker: UnreliableWorker,
+      input: ->(ctx) { ctx.workflow_input },
+      retry: {
+        max_attempts: 3,
+        backoff: :exponential,
+        initial_delay: 0.1,
+      }
+
+  end_job :persistent_task
+end
+----
+
+Each failed work item is saved as a JSON file with all context and metadata.
+
+== Querying the Dead Letter Queue
+
+The DLQ provides multiple methods to query and filter entries:
+
+=== Get All Entries
+
+[source,ruby]
+----
+dlq = workflow.dead_letter_queue
+all_entries = dlq.all
+puts "Total entries: #{all_entries.size}"
+----
+
+=== Filter by Error Class
+
+[source,ruby]
+----
+standard_errors = dlq.by_error_class(StandardError)
+timeout_errors = dlq.by_error_class(Timeout::Error)
+----
+
+=== Filter by Time Range
+
+[source,ruby]
+----
+# Get recent failures (last hour)
+recent = dlq.by_time_range(Time.now - 3600, Time.now)
+
+# Get yesterday's failures
+yesterday_start = Time.now - 86400
+yesterday_end = Time.now - 86400 + 86400
+yesterday_failures = dlq.by_time_range(yesterday_start, yesterday_end)
+----
+
+=== Custom Filtering
+
+[source,ruby]
+----
+# Find entries with specific context
+query_test_entries = dlq.filter do |entry|
+  entry.context[:message]&.include?("Query test")
+end
+
+# Find entries for specific job
+job_entries = dlq.filter do |entry|
+  entry.metadata[:job_name] == "unreliable_task"
+end
+----
+
+== Retrying Failed Work
+
+=== Retry Single Entry
+
+[source,ruby]
+----
+entry = dlq.all.first
+
+dlq.retry_entry(entry) do |work, error, context|
+  # Custom retry logic
+  # Return result or raise to fail again
+  MyWorker.perform(work)
+end
+----
+
+=== Retry All Failed Work
+
+[source,ruby]
+----
+dlq.retry_all do |work, error, context|
+  # Attempt to reprocess each failed item
+  begin
+    MyWorker.perform(work)
+  rescue StandardError => e
+    # Log but don't fail the batch retry
+    puts "Retry failed: #{e.message}"
+    nil
+  end
+end
+----
+
+== DLQ Statistics
+
+Get aggregate information about the DLQ:
+
+[source,ruby]
+----
+stats = dlq.stats
+
+puts "Total entries: #{stats[:total]}"
+puts "Oldest entry: #{stats[:oldest_timestamp]}"
+puts "Newest entry: #{stats[:newest_timestamp]}"
+puts "Error types: #{stats[:error_types].inspect}"
+puts "Jobs: #{stats[:jobs].inspect}"
+----
+
+== Entry Structure
+
+Each DLQ entry contains:
+
+=== Core Information
+
+* `work`: The failed Work object with original payload
+* `error`: The exception that caused the failure
+* `timestamp`: When the entry was added to DLQ
+* `context`: Workflow context (inputs, job states)
+* `metadata`: Additional information
+
+=== Metadata Fields
+
+When added by the workflow executor, entries include:
+
+[source,ruby]
+----
+{
+  job_name: "task_name",
+  worker_class: "WorkerClass",
+  correlation_id: "uuid",
+  workflow_name: "workflow_name",
+  retry_attempts: 3,
+  total_retry_time: 5.2,
+  all_errors: ["Error 1", "Error 2", "Error 3"]
+}
+----
+
+== Configuration Options
+
+[options="header"]
+|===
+| Option | Type | Default | Description
+| `max_size` | Integer | 1000 | Maximum DLQ entries to retain
+| `persister` | Object | nil | Optional persistence strategy
+| `on_add` | Proc | nil | Callback when entry is added
+|===
+
+== Best Practices
+
+=== Size Management
+
+Configure `max_size` based on your error rate and retention needs:
+
+[source,ruby]
+----
+# High-volume system
+configure_dead_letter_queue max_size: 10000
+
+# Low-volume system
+configure_dead_letter_queue max_size: 100
+----
+
+=== Monitoring
+
+Set up monitoring for DLQ growth:
+
+[source,ruby]
+----
+configure_dead_letter_queue(
+  on_add: lambda { |entry|
+    # Send to monitoring system
+    StatsD.increment("dlq.entries")
+    StatsD.gauge("dlq.size", workflow.dead_letter_queue.size)
+
+    # Alert if DLQ grows too large
+    if workflow.dead_letter_queue.size > 500
+      AlertService.send("DLQ size exceeds threshold")
+    end
+  }
+)
+----
+
+=== Regular Cleanup
+
+Implement regular DLQ review and cleanup:
+
+[source,ruby]
+----
+# Review old entries
+old_entries = dlq.by_time_range(Time.now - 7.days, Time.now)
+
+# Remove resolved entries
+old_entries.each do |entry|
+  if issue_resolved?(entry)
+    dlq.remove(entry)
+  end
+end
+----
+
+=== Persistence Strategy
+
+Choose persistence based on requirements:
+
+* *Memory-only*: Fast, suitable for development and low-stakes scenarios
+* *File-based*: Durable, good for single-server deployments
+* *Redis/Database*: Centralized, required for multi-server deployments
+
+== Integration with Retry Logic
+
+The DLQ works seamlessly with retry configuration:
+
+[source,ruby]
+----
+job :task,
+    worker: Worker,
+    retry: {
+      max_attempts: 3,        # Try 3 times
+      backoff: :exponential,  # With exponential backoff
+      initial_delay: 1.0,
+    }
+----
+
+Flow:
+
+1. Job fails → First retry attempt
+2. Retry fails → Second retry attempt
+3. Retry fails → Third retry attempt
+4. All retries exhausted → **Added to DLQ**
+
+== Running the Examples
+
+[source,bash]
+----
+# Run all DLQ examples
+ruby examples/workflow/dead_letter_queue/dead_letter_queue_workflow.rb
+
+# Example output:
+# ================================================================================
+# Dead Letter Queue Workflow Examples
+# ================================================================================
+#
+# --- Example 1: Basic Dead Letter Queue ---
+# Running workflow with failing work that exhausts retries...
+#
+# ✓ Workflow failed as expected: Job 'unreliable_task' failed: ...
+#
+# Dead Letter Queue Status:
+#   Entries: 1
+#   Stats: {:total=>1, :oldest_timestamp=>..., :error_types=>...}
+# ...
+----
+
+== See Also
+
+* link:../retry/README.adoc[Retry Mechanism]
+* link:../circuit_breaker/README.adoc[Circuit Breaker]
+* link:../../../docs/workflows.adoc[Workflow Documentation]

data/examples/workflow/dead_letter_queue/dead_letter_queue_workflow.rb

@@ -0,0 +1,217 @@
+# frozen_string_literal: true
+
+require_relative "../../../lib/fractor"
+
+# Work item for DLQ workflow
+class DLQWork < Fractor::Work
+  attr_reader :action, :message
+
+  def initialize(action:, message:)
+    @action = action
+    @message = message
+    super({ action:, message: })
+  end
+end
+
+# Worker that intentionally fails for specific inputs
+class UnreliableWorker < Fractor::Worker
+  input_type DLQWork
+  output_type Hash
+
+  def process(work)
+    input = work.input
+
+    # Simulate different failure scenarios
+    case input[:action]
+    when "fail_always"
+      raise StandardError, "Permanent failure: #{input[:message]}"
+    when "fail_random"
+      raise StandardError, "Random failure" if rand < 0.7
+      { status: "success", data: input[:message] }
+    when "timeout"
+      sleep 10 # Simulate timeout
+      { status: "success", data: input[:message] }
+    else
+      { status: "success", data: input[:message] }
+    end
+  end
+end
+
+# Worker that always succeeds
+class ReliableWorker < Fractor::Worker
+  input_type Hash
+  output_type Hash
+
+  def process(work)
+    input = work.input
+    { status: "processed", message: input[:message] }
+  end
+end
+
+# Example 1: Basic Dead Letter Queue with automatic capture
+class BasicDLQWorkflow < Fractor::Workflow
+  workflow "basic_dlq_workflow" do
+    start_with "unreliable_task"
+    configure_dead_letter_queue max_size: 1000
+
+    job "unreliable_task" do
+      runs_with UnreliableWorker
+      inputs_from_workflow
+
+      # Retry up to 3 times with exponential backoff
+      retry_on_error max_attempts: 3,
+                     backoff: :exponential,
+                     initial_delay: 0.1,
+                     max_delay: 1
+
+      # Add error handler for logging
+      on_error do |error, context|
+        puts "Error in unreliable_task: #{error.message}"
+      end
+
+      outputs_to_workflow
+      terminates_workflow
+    end
+  end
+end
+
+# Example 2: DLQ with custom error handlers
+class DLQWithHandlersWorkflow < Fractor::Workflow
+  workflow "dlq_with_handlers_workflow" do
+    start_with "risky_task"
+    configure_dead_letter_queue max_size: 1000
+
+    job "risky_task" do
+      runs_with UnreliableWorker
+      inputs_from_workflow
+
+      # Retry up to 2 times with linear backoff
+      retry_on_error max_attempts: 2,
+                     backoff: :linear,
+                     initial_delay: 0.1
+
+      # Add error handler for logging
+      on_error do |error, _context|
+        puts "\n⚠️  Work added to DLQ:"
+        puts "   Error: #{error.class.name}: #{error.message}"
+        puts "   Timestamp: #{Time.now}"
+      end
+
+      outputs_to_workflow
+      terminates_workflow
+    end
+  end
+end
+
+# Example 3: DLQ with file persistence
+class DLQWithPersistenceWorkflow < Fractor::Workflow
+  workflow "dlq_with_persistence_workflow" do
+    start_with "persistent_task"
+    configure_dead_letter_queue max_size: 1000
+
+    job "persistent_task" do
+      runs_with UnreliableWorker
+      inputs_from_workflow
+
+      # Retry up to 3 times with exponential backoff
+      retry_on_error max_attempts: 3,
+                     backoff: :exponential,
+                     initial_delay: 0.1
+
+      # Add error handler for persistence simulation
+      on_error do |error, _context|
+        # Simulate file persistence
+        require "fileutils"
+        FileUtils.mkdir_p("tmp/dlq")
+        entry = {
+          error: error.class.name,
+          message: error.message,
+          timestamp: Time.now.to_s,
+        }
+        File.write("tmp/dlq/entry_#{Time.now.to_i}.json", entry.to_json)
+        puts "DLQ entry persisted to tmp/dlq/"
+      end
+
+      outputs_to_workflow
+      terminates_workflow
+    end
+  end
+end
+
+# Demonstration runners
+if __FILE__ == $PROGRAM_NAME
+  require "json"
+
+  puts "=" * 80
+  puts "Dead Letter Queue Workflow Examples"
+  puts "=" * 80
+
+  # Example 1: Basic DLQ
+  puts "\n--- Example 1: Basic Dead Letter Queue ---"
+  puts "Running workflow with failing work that exhausts retries..."
+
+  workflow1 = BasicDLQWorkflow.new
+  work1 = DLQWork.new(action: "fail_always", message: "Test 1")
+  begin
+    result1 = workflow1.execute(work1)
+    puts "Workflow completed (should not reach here)"
+  rescue Fractor::Workflow::WorkflowExecutionError => e
+    puts "\n✓ Workflow failed as expected: #{e.message}"
+  end
+
+  # Example 2: DLQ with handlers
+  puts "\n\n--- Example 2: DLQ with Custom Handlers ---"
+  puts "Running workflow with custom notification handlers..."
+
+  workflow2 = DLQWithHandlersWorkflow.new
+  work2 = DLQWork.new(action: "fail_always", message: "Test 2")
+  begin
+    result2 = workflow2.execute(work2)
+  rescue Fractor::Workflow::WorkflowExecutionError => e
+    puts "\n✓ Workflow failed, handler was triggered above"
+  end
+
+  # Example 3: DLQ with persistence
+  puts "\n\n--- Example 3: DLQ with File Persistence ---"
+  puts "Running workflow with file-persisted DLQ..."
+
+  require "fileutils"
+  FileUtils.mkdir_p("tmp/dlq")
+
+  workflow3 = DLQWithPersistenceWorkflow.new
+  work3 = DLQWork.new(action: "fail_always", message: "Test 3")
+  begin
+    result3 = workflow3.execute(work3)
+  rescue Fractor::Workflow::WorkflowExecutionError => e
+    puts "\n✓ Workflow failed, entry persisted to disk"
+
+    # Check if file was created
+    if Dir.exist?("tmp/dlq")
+      files = Dir.glob("tmp/dlq/*.json")
+      puts "DLQ Size: #{files.size}"
+      if files.any?
+        puts "First file: #{files.first}"
+        content = JSON.parse(File.read(files.first))
+        puts "Content: #{content.inspect}"
+      end
+    end
+  end
+
+  # Example 4: Successful execution
+  puts "\n\n--- Example 4: Successful Execution ---"
+  puts "Running workflow with successful work..."
+
+  workflow4 = BasicDLQWorkflow.new
+  work4 = DLQWork.new(action: "success", message: "Success Test")
+  begin
+    result4 = workflow4.execute(work4)
+    puts "\n✓ Workflow completed successfully!"
+    puts "Result: #{result4.output.inspect}"
+  rescue Fractor::Workflow::WorkflowExecutionError => e
+    puts "Workflow failed: #{e.message}"
+  end
+
+  puts "\n" + "=" * 80
+  puts "Dead Letter Queue examples complete!"
+  puts "=" * 80
+end