ruby_reactor 0.2.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 598c9e49f00183da45e7b370c394445e9efddfe5952e3aabd701bbf954aa9712
-  data.tar.gz: 949ed81bd787d7bee47284e7f7b5905af3de4ae4057a90a4d0afa707959f5cbb
+  metadata.gz: d3ddfd9b6e65d03e5c00d7980563da39b4132fc1d5f0dbe14e5b782d039c7f68
+  data.tar.gz: 785cc8ac8d426433a00f378eafb8f312b343950ecf496e2af0a6fe9e48115c29
 SHA512:
-  metadata.gz: ae8f8f8a2916254068757d79d352df0a0148d4693f92238b456cecc1cb90d9f6cc77087f2b486e3359f5f0042779aecc9112df1da9898fa8efba6c9ed425ef39
-  data.tar.gz: 14fcbf6f880396176503008239e6517415977b19193f3f0d8600226acb7cf53d54a18bb79a42012646a1746d3202a62b73479715b6d088f69c208ce1c143aa92
+  metadata.gz: 549a49deff5d63e129c0b4c043cbc3987dbbb09b054907a5acb8941af85fedb5f9706bad8d56948bbc7ea889eba19174254962ca987e7835b4cd0a7bf3f9e165
+  data.tar.gz: 319399c2854cccb505fc7e8f6c964eeeb527bad375ed513b1a2c2ea89b77445283f287bbff5a0892902a56a5a56b5a9aff105a0164006abf407473ad3780a58d

data/README.md

@@ -1,9 +1,20 @@
+[![Gem Version](https://badge.fury.io/rb/ruby_reactor.svg)](https://badge.fury.io/rb/ruby_reactor)
+[![Build Status](https://github.com/arturictus/ruby_reactor/actions/workflows/main.yml/badge.svg)](https://github.com/arturictus/ruby_reactor/actions)  <!-- if you have CI -->
+[![Ruby Style Guide](https://img.shields.io/badge/code_style-rubocop-brightgreen.svg)](https://github.com/rubocop/rubocop)
+[![Downloads](https://img.shields.io/gem/dt/ruby_reactor.svg)](https://rubygems.org/gems/ruby_reactor)
+
 # RubyReactor
 
 A dynamic, dependency-resolving saga orchestrator for Ruby. Ruby Reactor implements the Saga pattern with compensation-based error handling and DAG-based execution planning. It leverages **Sidekiq** for asynchronous execution and **Redis** for state persistence.
 
 ![Payment workflow reactor](documentation/images/payment_workflow.png)
 
+## Why Ruby Reactor?
+
+Building complex business transactions often results in spaghetti code or brittle "god classes." Ruby Reactor solves this by implementing the **Saga Pattern** in a lightweight, developer-friendly package. It lets you define workflows as clear, dependency-driven steps without the boilerplate of heavy enterprise frameworks.
+
+The key value is **Reliability**: if any part of your workflow fails, Ruby Reactor automatically triggers compensation logic to undo previous steps, ensuring your system never ends up in a corrupted half-state. Whether you're coordinating microservices or monolith modules, you get atomic-like consistency with background processing built-in.
+
 ## Features
 
 - **DAG-based Execution**: Steps are executed based on their dependencies, allowing for parallel execution of independent steps.
@@ -14,6 +25,50 @@ A dynamic, dependency-resolving saga orchestrator for Ruby. Ruby Reactor impleme
 - **Interrupts**: Pause and resume workflows to wait for external events (webhooks, user approvals).
 - **Input Validation**: Integrated with `dry-validation` for robust input checking.
 
+## Comparison
+
+| Feature                  | Ruby Reactor | dry-transaction | Trailblazer | Custom Sidekiq Jobs |
+|--------------------------|--------------|-----------------|-------------|---------------------|
+| DAG/Parallel execution   | Yes          | No              | Limited     | Manual              |
+| Auto compensation/undo   | Yes          | No              | Manual      | Manual              |
+| Interrupts (pause/resume)| Yes          | No              | No          | Manual              |
+| Built-in web dashboard   | Yes          | No              | No          | No                  |
+| Async with Sidekiq       | Yes          | No              | Limited     | Yes                 |
+
+## Real-World Use Cases
+
+- **E-commerce Checkout**: Orchestrate inventory reservation, payment authorization, and shipping label generation. If payment fails, automatically release inventory and cancel the shipping request.
+- **Data Import Pipelines**: Ingest optional massive CSVs using `map` steps to validate and upsert records in parallel. If data validation fails for a chunk, fail fast or collect errors while letting valid chunks succeed.
+- **Subscription Billing**: Coordinate Stripe charges, invoice email generation, and internal entitlement updates. Use interrupts to pause the workflow when 3rd-party APIs are required to continue the workflow or when specific customer approval is needed.
+
+## Table of Contents
+- [Features](#features)
+- [Comparison](#comparison)
+- [Real-World Use Cases](#real-world-use-cases)
+- [Installation](#installation)
+- [Configuration](#configuration)
+- [Quick Start](#quick-start)
+- [Web Dashboard](#web-dashboard)
+  - [Rails Installation](#rails-installation)
+- [Usage](#usage)
+  - [Basic Example: User Registration](#basic-example-user-registration)
+  - [Async Execution](#async-execution)
+    - [Full Reactor Async](#full-reactor-async)
+    - [Step-Level Async](#step-level-async)
+  - [Interrupts (Pause & Resume)](#interrupts-pause--resume)
+  - [Map & Parallel Execution](#map--parallel-execution)
+    - [Map with Dynamic Source (ActiveRecord)](#map-with-dynamic-source-activerecord)
+  - [Input Validation](#input-validation)
+  - [Complex Workflows with Dependencies](#complex-workflows-with-dependencies)
+  - [Error Handling and Compensation](#error-handling-and-compensation)
+  - [Using Pre-defined Schemas](#using-pre-defined-schemas)
+  - [Testing](#testing)
+- [Documentation](#documentation)
+- [Future improvements](#future-improvements)
+- [Development](#development)
+- [Contributing](#contributing)
+- [Code of Conduct](#code-of-conduct)
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -50,6 +105,21 @@ RubyReactor.configure do |config|
 end
 ```
 
+
+## Quick Start
+
+```ruby
+class HelloReactor < RubyReactor::Reactor
+  step :greet do
+    run { Success("Hello from Ruby Reactor!") }
+  end
+  returns :greet
+end
+
+result = HelloReactor.run
+puts result.value  # => "Hello from Ruby Reactor!"
+```
+
 ## Web Dashboard
 
 RubyReactor comes with a built-in web dashboard to inspect reactor executions, view logs, and retry failed steps.
@@ -281,6 +351,42 @@ class DataProcessingReactor < RubyReactor::Reactor
 end
 ```
 
+By using `async true` with `batch_size`, the system applies **Back Pressure** to efficiently manage resources. [Read more about Back Pressure & Resource Management](documentation/data_pipelines.md#back-pressure--resource-management).
+
+#### Map with Dynamic Source (ActiveRecord)
+
+You can use a block for `source` to dynamically fetch data, such as from ActiveRecord queries. The result is wrapped in a `ResultEnumerator` for easy access to successes and failures.
+
+```ruby
+map :archive_old_users do
+  argument :days, input(:days)
+
+  # Dynamic source using ActiveRecord
+  source do |args|
+    User.where("last_login_at < ?", args[:days].days.ago)
+  end
+  
+  argument :user, element(:archive_old_users)
+  async true, batch_size: 100
+
+  step :archive do
+    argument :user, input(:user)
+    run { |args| args[:user].archive! }
+  end
+  
+  returns :archive
+end
+
+step :summary do
+  argument :results, result(:archive_old_users)
+  run do |args|
+    puts "Archived: #{args[:results].successes.count}"
+    puts "Failed: #{args[:results].failures.count}"
+    Success()
+  end
+end
+```
+
 ### Input Validation
 
 RubyReactor integrates with dry-validation for input validation:
@@ -575,6 +681,29 @@ class SchemaValidatedReactor < RubyReactor::Reactor
 end
 ```
 
+### Testing
+
+RubyReactor provides testing utilities for RSpec. See the [Testing with RSpec](documentation/testing.md) guide for comprehensive documentation.
+
+```ruby
+RSpec.describe PaymentReactor do
+  it "processes payment successfully" do
+    subject = test_reactor(PaymentReactor, order_id: 123, amount: 99.99)
+    
+    expect(subject).to be_success
+    expect(subject).to have_run_step(:charge_card).after(:validate_order)
+    expect(subject.step_result(:charge_card)).to include(status: "completed")
+  end
+
+  it "handles payment failures with mocked steps" do
+    subject = test_reactor(PaymentReactor, order_id: 123, amount: 99.99)
+      .mock_step(:charge_card) { Failure("Card declined") }
+    
+    expect(subject).to be_failure
+    expect(subject.error).to include("Card declined")
+  end
+end
+```
 
 ## Documentation
 
@@ -601,6 +730,9 @@ Configure robust retry policies for your steps. This guide details the available
 ### [Interrupts](documentation/interrupts.md)
 Learn how to pause and resume reactors to handle long-running processes, manual approvals, and asynchronous callbacks. Patterns for correlation IDs, timeouts, and payload validation.
 
+### [Testing with RSpec](documentation/testing.md)
+Comprehensive guide to testing reactors with RubyReactor's testing utilities. Learn about the `TestSubject` class for reactor execution and introspection, step mocking for isolating dependencies, testing nested and composed reactors, and custom RSpec matchers like `be_success`, `have_run_step`, and `have_retried_step`.
+
 ### Examples
 - [Order Processing](documentation/examples/order_processing.md) - Complete order processing workflow example
 - [Payment Processing](documentation/examples/payment_processing.md) - Payment handling with compensation

data/Rakefile

@@ -30,8 +30,8 @@ namespace :server do
   end
 end
 
-# require "rubocop/rake_task"
+require "rubocop/rake_task"
 
-# RuboCop::RakeTask.new
+RuboCop::RakeTask.new
 
 task default: %i[spec rubocop]

data/documentation/data_pipelines.md

@@ -43,53 +43,35 @@ class UserTransformationReactor < RubyReactor::Reactor
 end
 ```
 
-## Async Execution
+## Dynamic Sources & ActiveRecord
 
-For long-running or resource-intensive tasks, you can offload processing to background jobs using Sidekiq.
-
-To enable async execution, simply add the `async true` directive to your map definition.
+The `map` step supports a dynamic `source` block, which is particularly useful when working with ActiveRecord or when the collection depends on input arguments. Instead of passing a static collection, you can define a block that returns an Enumerable or an `ActiveRecord::Relation`.
 
 ```ruby
-map :process_orders do
-  source input(:orders)
-  argument :order, element(:process_orders)
-  
-  # Enable async execution via Sidekiq
-  async true
-
-  step :charge_card do
-    argument :order, input(:order)
-    run { PaymentService.charge(args[:order]) }
+map :process_products do
+  argument :filter, input(:filter)
+
+  # Dynamic source block
+  source do |args|
+    # This block executes at runtime
+    threshold = args[:filter][:stock]
+    Product.where("stock >= ?", threshold)
   end
 
-  returns :charge_card
+  argument :product, element(:process_products)
+  async true, batch_size: 100
+
+  step :process do
+    # ...
+  end
+  
+  returns :process
 end
 ```
 
-### Execution Flow
-
-```mermaid
-sequenceDiagram
-    participant Reactor
-    participant Redis
-    participant Sidekiq
-    participant Worker
-
-    Reactor->>Redis: Store Context
-    Reactor->>Sidekiq: Enqueue MapElementWorkers
-    Note over Reactor: Returns AsyncResult immediately
-    
-    loop For each element
-        Sidekiq->>Worker: Process Element
-        Worker->>Redis: Update Element Result
-    end
-
-    Worker->>Sidekiq: Enqueue MapCollectorWorker (when done)
-    Sidekiq->>Worker: Run Collector
-    Worker->>Redis: Store Final Result
-```
+When an `ActiveRecord::Relation` is returned, RubyReactor efficiently batches the query using database-level `OFFSET` and `LIMIT` based on the configured `batch_size`, preventing memory bloat by not loading all records at once.
 
-## Batch Processing
+## Batch Processing Mechanism
 
 When processing large datasets asynchronously, you can control the parallelism using `batch_size`. This limits how many Sidekiq jobs are enqueued simultaneously, preventing system overload.
 
@@ -107,10 +89,42 @@ map :bulk_import do
 end
 ```
 
-**How it works:**
-1. The system initially enqueues `batch_size` jobs.
-2. As each job completes, it triggers the next job in the queue.
-3. This maintains a steady stream of processing without flooding the queue.
+### Back Pressure & Resource Management
+
+When `async true` is used with a `batch_size`, RubyReactor implements an intelligent **back pressure** mechanism. Instead of flooding Redis and Sidekiq with millions of jobs immediately (which is the standard behavior for many background job systems), the system processes data in controlled chunks.
+
+This approach provides critical benefits for stability and scalability:
+
+1.  **Memory Efficiency**: By using `ActiveRecord` batching (`LIMIT` / `OFFSET`), only the current batch of records is loaded into memory. This allows processing datasets larger than available RAM.
+2.  **Redis Protection**: Prevents "Queue Explosion". Only a small number of job arguments are stored in Redis at any time, preventing OOM errors in your Redis instance.
+3.  **Database Stability**: Database load is distributed over time rather than spiking all at once.
+
+**Visualizing the Flow:**
+
+```mermaid
+graph TD
+    Start[Start Map] -->|Batch Size: N| BatchManager
+    
+    subgraph "Back Pressure Loop"
+        BatchManager[Batch Manager] -->|Fetch N Items| DB[(Database)]
+        DB --> Records
+        Records -->|Enqueue N Jobs| Sidekiq
+        
+        Sidekiq --> W1[Worker 1]
+        Sidekiq --> W2[Worker 2]
+        
+        W1 -.->|Complete| Check{Batch Done?}
+        W2 -.->|Complete| Check
+        
+        Check -->|No| Wait[Wait]
+        Check -->|Yes| Next[Trigger Next Batch]
+        Next --> BatchManager
+    end
+    
+    BatchManager -->|No More Items| Finish[Aggregator]
+```
+
+This ensures that the system works at the speed of your workers, not the speed of the enqueueing process, maintaining a constant and manageable resource footprint regardless of dataset size.
 
 ## Error Handling
 
@@ -128,9 +142,9 @@ map :strict_processing do
 end
 ```
 
-### Collecting Partial Results
+### Collecting Results (Successes & Failures)
 
-If you want to process all elements regardless of failures, set `fail_fast false`. You can then use a `collect` block to handle successes and failures separately.
+If you want to process all elements regardless of failures, set `fail_fast false`. The map step returns a `ResultEnumerator` that allows you to easily separate successful executions from failures.
 
 ```ruby
 map :resilient_processing do
@@ -145,18 +159,39 @@ map :resilient_processing do
   end
 
   returns :risky_operation
+end
+
+step :analyze_results do
+  argument :results, result(:resilient_processing)
+  
+  run do |args|
+    col = args[:results]
+    
+    # Iterate over successful results
+    col.successes.each do |value|
+      # 'value' is the direct return value of the map element
+      puts "Success: #{value}"
+    end
+
+    # Iterate over failures
+    col.failures.each do |error|
+      # 'error' is the failure object/message itself
+      puts "Error: #{error}"
+    end
+
+    # Note: Iterating the collection directly yields wrapped objects
+    col.each do |result|
+      if result.is_a?(RubyReactor::Success)
+        puts "Wrapped Value: #{result.value}"
+      else
+        puts "Wrapped Error: #{result.error}"
+      end
+    end
 
-  # Aggregate results
-  collect do |results|
-    # results is an array of Result objects (Success or Failure)
-    successful = results.select(&:success?).map(&:value)
-    failed = results.select(&:failure?).map(&:error)
-
-    {
-      processed: successful,
-      errors: failed,
-      success_rate: successful.length.to_f / results.length
-    }
+    Success({
+      success_count: col.successes.count,
+      failure_count: col.failures.count
+    })
   end
 end
 ```
@@ -192,33 +227,4 @@ end
 - **Async Mode**: Retries are handled by requeuing the Sidekiq job with a delay. This is non-blocking and efficient.
 - **Sync Mode**: Retries happen immediately within the execution thread (blocking).
 
-## Visualization
 
-### Async Batch Execution
-
-```mermaid
-graph TD
-    Start[Start Map] --> Init[Initialize Batch]
-    Init --> Q1["Queue Initial Batch<br/>(Size N)"]
-    
-    subgraph Workers
-        W1[Worker 1]
-        W2[Worker 2]
-        W3[Worker ...]
-    end
-    
-    Q1 --> W1
-    Q1 --> W2
-    
-    W1 -->|Complete| Next1{More Items?}
-    W2 -->|Complete| Next2{More Items?}
-    
-    Next1 -->|Yes| Q2[Queue Next Item]
-    Next2 -->|Yes| Q2
-    
-    Q2 --> W3
-    
-    Next1 -->|No| Check{All Done?}
-    Check -->|Yes| Collect[Run Collector]
-    Collect --> Finish[Final Result]
-```