fractor 0.1.4 → 0.1.7

data/docs/_tutorials/data-processing-pipeline.adoc

@@ -0,0 +1,740 @@
+---
+layout: default
+title: Building a Data Processing Pipeline
+nav_order: 5
+---
+
+== Building a Data Processing Pipeline
+
+=== Overview
+
+In this 30-minute intermediate tutorial, you'll build a complete data processing pipeline that extracts data from CSV files, transforms it, validates it, and loads it into a database. This real-world example demonstrates how to structure complex pipelines using Fractor.
+
+**What you'll learn:**
+
+* Breaking down complex tasks into pipeline stages
+* Creating specialized workers for each stage
+* Handling errors and validation
+* Monitoring pipeline progress
+* Best practices for production pipelines
+
+**Prerequisites:**
+
+* Completed link:getting-started[Getting Started] tutorial
+* Basic understanding of link:../guides/core-concepts[Core Concepts]
+* Familiarity with Ruby classes and CSV processing
+
+=== The Problem
+
+You need to process customer data from multiple CSV files:
+
+1. **Extract**: Read CSV files and parse records
+2. **Transform**: Clean and normalize data (trim whitespace, format dates, etc.)
+3. **Validate**: Check data quality (required fields, valid emails, etc.)
+4. **Load**: Insert valid records into a database
+
+The pipeline should:
+
+* Process files in parallel for speed
+* Handle validation errors gracefully
+* Track success/failure statistics
+* Be production-ready with proper error handling
+
+=== Step 1: Set Up the Project
+
+Create a new project directory:
+
+[source,sh]
+----
+mkdir customer_pipeline
+cd customer_pipeline
+----
+
+Create the directory structure:
+
+[source,sh]
+----
+mkdir -p lib data
+touch lib/pipeline.rb
+touch lib/models.rb
+touch lib/workers.rb
+----
+
+Install Fractor:
+
+[source,sh]
+----
+gem install fractor
+----
+
+=== Step 2: Define Data Models
+
+Create `lib/models.rb` to define our data structures:
+
+[source,ruby]
+----
+require 'date'
+
+# Raw data from CSV
+class RawCustomer
+  attr_reader :data
+
+  def initialize(row)
+    @data = row
+  end
+
+  def [](key)
+    @data[key]
+  end
+end
+
+# Transformed and validated customer
+class Customer
+  attr_accessor :id, :name, :email, :phone, :signup_date, :country
+
+  def initialize(attrs = {})
+    @id = attrs[:id]
+    @name = attrs[:name]
+    @email = attrs[:email]
+    @phone = attrs[:phone]
+    @signup_date = attrs[:signup_date]
+    @country = attrs[:country]
+  end
+
+  def to_h
+    {
+      id: @id,
+      name: @name,
+      email: @email,
+      phone: @phone,
+      signup_date: @signup_date,
+      country: @country
+    }
+  end
+end
+
+# Validation result
+class ValidationResult
+  attr_reader :customer, :errors
+
+  def initialize(customer:, errors: [])
+    @customer = customer
+    @errors = errors
+  end
+
+  def valid?
+    @errors.empty?
+  end
+end
+----
+
+=== Step 3: Create Pipeline Workers
+
+Create `lib/workers.rb` with specialized workers for each stage:
+
+[source,ruby]
+----
+require 'fractor'
+require 'csv'
+require_relative 'models'
+
+# Stage 1: Extract - Read and parse CSV files
+class ExtractWorker < Fractor::Worker
+  def process(work)
+    filepath = work.input[:filepath]
+
+    customers = []
+    CSV.foreach(filepath, headers: true) do |row|
+      customers << RawCustomer.new(row.to_h)
+    end
+
+    Fractor::WorkResult.new(
+      result: {
+        filepath: filepath,
+        customers: customers,
+        count: customers.size
+      },
+      work: work
+    )
+  rescue => e
+    Fractor::WorkResult.new(
+      error: e,
+      error_code: :extraction_failed,
+      error_context: { filepath: filepath },
+      work: work
+    )
+  end
+end
+
+# Stage 2: Transform - Clean and normalize data
+class TransformWorker < Fractor::Worker
+  def process(work)
+    raw_customer = work.input[:customer]
+
+    customer = Customer.new(
+      id: raw_customer['id']&.strip,
+      name: normalize_name(raw_customer['name']),
+      email: raw_customer['email']&.strip&.downcase,
+      phone: normalize_phone(raw_customer['phone']),
+      signup_date: parse_date(raw_customer['signup_date']),
+      country: raw_customer['country']&.strip&.upcase
+    )
+
+    Fractor::WorkResult.new(
+      result: { customer: customer },
+      work: work
+    )
+  rescue => e
+    Fractor::WorkResult.new(
+      error: e,
+      error_code: :transformation_failed,
+      error_context: { raw_data: raw_customer.data },
+      work: work
+    )
+  end
+
+  private
+
+  def normalize_name(name)
+    return nil unless name
+    name.strip.split.map(&:capitalize).join(' ')
+  end
+
+  def normalize_phone(phone)
+    return nil unless phone
+    phone.gsub(/[^0-9]/, '')
+  end
+
+  def parse_date(date_str)
+    return nil unless date_str
+    Date.parse(date_str)
+  rescue ArgumentError
+    nil
+  end
+end
+
+# Stage 3: Validate - Check data quality
+class ValidateWorker < Fractor::Worker
+  EMAIL_REGEX = /\A[\w+\-.]+@[a-z\d\-]+(\.[a-z\d\-]+)*\.[a-z]+\z/i
+
+  def process(work)
+    customer = work.input[:customer]
+    errors = []
+
+    # Required field validation
+    errors << "ID is required" if customer.id.nil? || customer.id.empty?
+    errors << "Name is required" if customer.name.nil? || customer.name.empty?
+    errors << "Email is required" if customer.email.nil? || customer.email.empty?
+
+    # Format validation
+    if customer.email && !customer.email.match?(EMAIL_REGEX)
+      errors << "Email format is invalid"
+    end
+
+    if customer.phone && customer.phone.length < 10
+      errors << "Phone number is too short"
+    end
+
+    # Business logic validation
+    if customer.signup_date && customer.signup_date > Date.today
+      errors << "Signup date cannot be in the future"
+    end
+
+    result = ValidationResult.new(
+      customer: customer,
+      errors: errors
+    )
+
+    Fractor::WorkResult.new(
+      result: { validation_result: result },
+      work: work
+    )
+  rescue => e
+    Fractor::WorkResult.new(
+      error: e,
+      error_code: :validation_failed,
+      work: work
+    )
+  end
+end
+
+# Stage 4: Load - Insert into database (simulated)
+class LoadWorker < Fractor::Worker
+  def process(work)
+    validation_result = work.input[:validation_result]
+
+    unless validation_result.valid?
+      return Fractor::WorkResult.new(
+        error: "Validation failed: #{validation_result.errors.join(', ')}",
+        error_code: :invalid_data,
+        error_context: {
+          customer_id: validation_result.customer.id,
+          errors: validation_result.errors
+        },
+        work: work
+      )
+    end
+
+    customer = validation_result.customer
+
+    # Simulate database insert
+    # In production: db.insert(:customers, customer.to_h)
+    insert_to_database(customer)
+
+    Fractor::WorkResult.new(
+      result: {
+        customer_id: customer.id,
+        inserted: true
+      },
+      work: work
+    )
+  rescue => e
+    Fractor::WorkResult.new(
+      error: e,
+      error_code: :load_failed,
+      error_context: { customer_id: customer&.id },
+      work: work
+    )
+  end
+
+  private
+
+  def insert_to_database(customer)
+    # Simulate database operation
+    sleep(0.01) # Simulate network latency
+    puts "  ✓ Loaded customer: #{customer.id} - #{customer.name}"
+  end
+end
+----
+
+=== Step 4: Build the Pipeline
+
+Create `lib/pipeline.rb` to orchestrate the pipeline:
+
+[source,ruby]
+----
+require 'fractor'
+require_relative 'workers'
+require_relative 'models'
+
+class CustomerPipeline
+  attr_reader :stats
+
+  def initialize(csv_files)
+    @csv_files = csv_files
+    @stats = {
+      files_processed: 0,
+      customers_extracted: 0,
+      customers_loaded: 0,
+      validation_errors: 0,
+      processing_errors: 0
+    }
+  end
+
+  def run
+    puts "Starting customer data pipeline..."
+    puts "Processing #{@csv_files.size} files\n\n"
+
+    # Stage 1: Extract customers from CSV files
+    extracted_results = extract_stage
+
+    # Stage 2: Transform extracted customers
+    transformed_results = transform_stage(extracted_results)
+
+    # Stage 3: Validate transformed customers
+    validated_results = validate_stage(transformed_results)
+
+    # Stage 4: Load valid customers
+    load_stage(validated_results)
+
+    print_statistics
+  end
+
+  private
+
+  def extract_stage
+    puts "Stage 1: Extracting data from CSV files..."
+
+    supervisor = Fractor::Supervisor.new(
+      worker_pools: [
+        { worker_class: ExtractWorker, num_workers: 4 }
+      ]
+    )
+
+    # Create work items for each file
+    work_items = @csv_files.map do |filepath|
+      Fractor::Work.new(filepath: filepath)
+    end
+
+    supervisor.add_work_items(work_items)
+    supervisor.run
+
+    # Process results
+    results = supervisor.results.results
+    errors = supervisor.results.errors
+
+    @stats[:files_processed] = results.size
+    @stats[:processing_errors] += errors.size
+
+    results.each do |result|
+      @stats[:customers_extracted] += result.result[:count]
+    end
+
+    errors.each do |error|
+      puts "  ✗ Failed to extract #{error.error_context[:filepath]}: #{error.error}"
+    end
+
+    puts "  → Extracted #{@stats[:customers_extracted]} customers from #{results.size} files\n\n"
+
+    results
+  end
+
+  def transform_stage(extract_results)
+    puts "Stage 2: Transforming customer data..."
+
+    supervisor = Fractor::Supervisor.new(
+      worker_pools: [
+        { worker_class: TransformWorker, num_workers: 8 }
+      ]
+    )
+
+    # Create work items for each customer
+    work_items = extract_results.flat_map do |result|
+      result.result[:customers].map do |customer|
+        Fractor::Work.new(customer: customer)
+      end
+    end
+
+    supervisor.add_work_items(work_items)
+    supervisor.run
+
+    results = supervisor.results.results
+    errors = supervisor.results.errors
+
+    @stats[:processing_errors] += errors.size
+
+    puts "  → Transformed #{results.size} customers\n\n"
+
+    results
+  end
+
+  def validate_stage(transform_results)
+    puts "Stage 3: Validating customer data..."
+
+    supervisor = Fractor::Supervisor.new(
+      worker_pools: [
+        { worker_class: ValidateWorker, num_workers: 8 }
+      ]
+    )
+
+    work_items = transform_results.map do |result|
+      Fractor::Work.new(customer: result.result[:customer])
+    end
+
+    supervisor.add_work_items(work_items)
+    supervisor.run
+
+    results = supervisor.results.results
+    errors = supervisor.results.errors
+
+    @stats[:processing_errors] += errors.size
+
+    # Count validation failures
+    results.each do |result|
+      unless result.result[:validation_result].valid?
+        @stats[:validation_errors] += 1
+      end
+    end
+
+    puts "  → Validated #{results.size} customers (#{@stats[:validation_errors]} invalid)\n\n"
+
+    results
+  end
+
+  def load_stage(validate_results)
+    puts "Stage 4: Loading valid customers..."
+
+    supervisor = Fractor::Supervisor.new(
+      worker_pools: [
+        { worker_class: LoadWorker, num_workers: 4 }
+      ]
+    )
+
+    work_items = validate_results.map do |result|
+      Fractor::Work.new(validation_result: result.result[:validation_result])
+    end
+
+    supervisor.add_work_items(work_items)
+    supervisor.run
+
+    results = supervisor.results.results
+    errors = supervisor.results.errors
+
+    @stats[:customers_loaded] = results.size
+    # Note: LoadWorker returns errors for invalid data, not processing errors
+
+    puts "\n  → Loaded #{results.size} customers\n\n"
+  end
+
+  def print_statistics
+    puts "=" * 60
+    puts "Pipeline Statistics"
+    puts "=" * 60
+    puts "Files processed:        #{@stats[:files_processed]}"
+    puts "Customers extracted:    #{@stats[:customers_extracted]}"
+    puts "Customers loaded:       #{@stats[:customers_loaded]}"
+    puts "Validation errors:      #{@stats[:validation_errors]}"
+    puts "Processing errors:      #{@stats[:processing_errors]}"
+    puts "Success rate:           #{success_rate}%"
+    puts "=" * 60
+  end
+
+  def success_rate
+    return 0 if @stats[:customers_extracted] == 0
+    ((@stats[:customers_loaded].to_f / @stats[:customers_extracted]) * 100).round(2)
+  end
+end
+----
+
+=== Step 5: Create Test Data
+
+Create sample CSV files in the `data/` directory:
+
+[source,ruby]
+----
+# Create data/sample_customers_1.csv
+require 'csv'
+
+CSV.open('data/sample_customers_1.csv', 'w') do |csv|
+  csv << ['id', 'name', 'email', 'phone', 'signup_date', 'country']
+  csv << ['1', ' john doe ', 'john@example.com', '555-1234', '2024-01-15', 'us']
+  csv << ['2', 'JANE SMITH', 'jane@example.com', '555-5678', '2024-02-20', 'ca']
+  csv << ['3', 'bob wilson', 'invalid-email', '555-9999', '2024-03-10', 'uk']
+  csv << ['4', '', 'empty@example.com', '555-4321', '2024-04-05', 'au']
+end
+
+CSV.open('data/sample_customers_2.csv', 'w') do |csv|
+  csv << ['id', 'name', 'email', 'phone', 'signup_date', 'country']
+  csv << ['5', 'alice brown', 'alice@example.com', '555-1111', '2024-05-12', 'nz']
+  csv << ['6', 'charlie davis', 'charlie@example.com', '123', '2024-06-18', 'ie']
+  csv << ['7', 'eve martinez', 'eve@example.com', '555-2222', '2025-12-31', 'es']
+end
+----
+
+=== Step 6: Run the Pipeline
+
+Create a runner script `run_pipeline.rb`:
+
+[source,ruby]
+----
+require_relative 'lib/pipeline'
+
+# Find all CSV files in data directory
+csv_files = Dir.glob('data/*.csv')
+
+if csv_files.empty?
+  puts "No CSV files found in data/ directory"
+  exit 1
+end
+
+# Run the pipeline
+pipeline = CustomerPipeline.new(csv_files)
+pipeline.run
+----
+
+Run it:
+
+[source,sh]
+----
+ruby run_pipeline.rb
+----
+
+Expected output:
+
+[source]
+----
+Starting customer data pipeline...
+Processing 2 files
+
+Stage 1: Extracting data from CSV files...
+  → Extracted 7 customers from 2 files
+
+Stage 2: Transforming customer data...
+  → Transformed 7 customers
+
+Stage 3: Validating customer data...
+  → Validated 7 customers (3 invalid)
+
+Stage 4: Loading valid customers...
+  ✓ Loaded customer: 1 - John Doe
+  ✓ Loaded customer: 2 - Jane Smith
+  ✓ Loaded customer: 5 - Alice Brown
+
+  → Loaded 3 customers
+
+============================================================
+Pipeline Statistics
+============================================================
+Files processed:        2
+Customers extracted:    7
+Customers loaded:       3
+Validation errors:      3
+Processing errors:      0
+Success rate:           42.86%
+============================================================
+----
+
+=== Step 7: Add Error Monitoring
+
+Enhance the pipeline with error reporting:
+
+[source,ruby]
+----
+require 'fractor'
+require_relative 'lib/pipeline'
+
+# Set up error reporter
+reporter = Fractor::ErrorReporter.new
+
+# Register error handlers
+reporter.on_error do |work_result, job_name|
+  if work_result.critical?
+    puts "CRITICAL: #{job_name} - #{work_result.error.message}"
+  end
+end
+
+# Modify pipeline to use reporter (add to each stage)
+# ... supervisor.results.errors.each { |e| reporter.record(e, job_name: "extract") }
+
+# Run pipeline
+csv_files = Dir.glob('data/*.csv')
+pipeline = CustomerPipeline.new(csv_files)
+pipeline.run
+
+# Print error report
+puts "\n"
+puts reporter.formatted_report
+----
+
+=== Best Practices Demonstrated
+
+==== 1. Separation of Concerns
+
+Each worker has a single responsibility:
+
+* **ExtractWorker**: Only reads CSV files
+* **TransformWorker**: Only normalizes data
+* **ValidateWorker**: Only validates data
+* **LoadWorker**: Only inserts to database
+
+==== 2. Error Handling
+
+Comprehensive error handling at each stage:
+
+[source,ruby]
+----
+rescue => e
+  Fractor::WorkResult.new(
+    error: e,
+    error_code: :stage_specific_code,
+    error_context: { relevant: 'context' },
+    work: work
+  )
+end
+----
+
+==== 3. Progress Tracking
+
+Statistics collection throughout the pipeline:
+
+[source,ruby]
+----
+@stats[:customers_extracted] += result.result[:count]
+----
+
+==== 4. Parallel Processing
+
+Different worker counts optimized for each stage:
+
+* Extract: 4 workers (I/O bound)
+* Transform: 8 workers (CPU bound)
+* Validate: 8 workers (CPU bound)
+* Load: 4 workers (I/O bound)
+
+=== Enhancements
+
+==== 1. Add Retry Logic
+
+Use workflows for automatic retry:
+
+[source,ruby]
+----
+class CustomerWorkflow < Fractor::Workflow
+  workflow "customer-pipeline" do
+    job "extract" do
+      runs_with ExtractWorker
+      retry_on_error max_attempts: 3, backoff: :exponential
+    end
+
+    # ... more jobs
+  end
+end
+----
+
+==== 2. Add Performance Monitoring
+
+[source,ruby]
+----
+monitor = Fractor::PerformanceMonitor.new(supervisor)
+monitor.start
+
+# ... run pipeline ...
+
+puts monitor.report
+monitor.stop
+----
+
+==== 3. Add Dead Letter Queue
+
+Capture permanently failed records:
+
+[source,ruby]
+----
+workflow "customer-pipeline" do
+  configure_dead_letter_queue max_size: 1000
+
+  # ... jobs ...
+end
+
+# After execution
+dlq = workflow.dead_letter_queue
+dlq.all.each do |entry|
+  puts "Failed: #{entry.error.message}"
+end
+----
+
+=== Summary
+
+You've built a production-ready data processing pipeline that:
+
+✓ Processes data in parallel stages
+✓ Handles errors gracefully with proper error codes
+✓ Tracks detailed statistics
+✓ Validates data quality
+✓ Can be monitored and extended
+
+**Key takeaways:**
+
+1. Break complex processes into stages
+2. Create focused, single-responsibility workers
+3. Use proper error handling with context
+4. Track statistics for monitoring
+5. Optimize worker counts per stage
+6. Consider workflows for more complex pipelines
+
+=== Next Steps
+
+* Try the link:long-running-services[Creating Long-Running Services] tutorial
+* Learn about link:../guides/workflows[Workflows] for more complex patterns
+* Explore link:../reference/error-reporting[Error Reporting] for production monitoring
+* Check out link:../reference/examples[Real-World Examples]