ruby_reactor 0.2.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 598c9e49f00183da45e7b370c394445e9efddfe5952e3aabd701bbf954aa9712
4
- data.tar.gz: 949ed81bd787d7bee47284e7f7b5905af3de4ae4057a90a4d0afa707959f5cbb
3
+ metadata.gz: 863c83e92cbfab470e39107c580ceabce444133973b723662a89e5bf0211c3ab
4
+ data.tar.gz: 1209b5587efc055bc787731694a6dc2b7eab8b3d5a6ff729728aa29ea601e2e0
5
5
  SHA512:
6
- metadata.gz: ae8f8f8a2916254068757d79d352df0a0148d4693f92238b456cecc1cb90d9f6cc77087f2b486e3359f5f0042779aecc9112df1da9898fa8efba6c9ed425ef39
7
- data.tar.gz: 14fcbf6f880396176503008239e6517415977b19193f3f0d8600226acb7cf53d54a18bb79a42012646a1746d3202a62b73479715b6d088f69c208ce1c143aa92
6
+ metadata.gz: 146d14cd32b12c95764e493f2de2d826dbe074ec4e4b44613b788d748d52fe38644fd86f1590f6bb0b1a0d1840c2200ab29a6d8e78db27f8f6abf099faf74756
7
+ data.tar.gz: 3d5b7c43182ee2d1c5c06ba90b7e2177a3dffbecdc8fb3e91db14f99bcd02807ebfa98c0e240412b790ec1d69609dc30995d9ee1aea9c8cb8ecd69d4d0fb2ac7
data/README.md CHANGED
@@ -1,5 +1,16 @@
1
+ [![Gem Version](https://badge.fury.io/rb/ruby_reactor.svg)](https://badge.fury.io/rb/ruby_reactor)
2
+ [![Build Status](https://github.com/arturictus/ruby_reactor/actions/workflows/main.yml/badge.svg)](https://github.com/arturictus/ruby_reactor/actions) <!-- if you have CI -->
3
+ [![Ruby Style Guide](https://img.shields.io/badge/code_style-rubocop-brightgreen.svg)](https://github.com/rubocop/rubocop)
4
+ [![Downloads](https://img.shields.io/gem/dt/ruby_reactor.svg)](https://rubygems.org/gems/ruby_reactor)
5
+
1
6
  # RubyReactor
2
7
 
8
+ ## Why Ruby Reactor?
9
+
10
+ Building complex business transactions often results in spaghetti code or brittle "god classes." Ruby Reactor solves this by implementing the **Saga Pattern** in a lightweight, developer-friendly package. It lets you define workflows as clear, dependency-driven steps without the boilerplate of heavy enterprise frameworks.
11
+
12
+ The key value is **Reliability**: if any part of your workflow fails, Ruby Reactor automatically triggers compensation logic to undo previous steps, ensuring your system never ends up in a corrupted half-state. Whether you're coordinating microservices or monolith modules, you get atomic-like consistency with background processing built-in.
13
+
3
14
  A dynamic, dependency-resolving saga orchestrator for Ruby. Ruby Reactor implements the Saga pattern with compensation-based error handling and DAG-based execution planning. It leverages **Sidekiq** for asynchronous execution and **Redis** for state persistence.
4
15
 
5
16
  ![Payment workflow reactor](documentation/images/payment_workflow.png)
@@ -14,6 +25,49 @@ A dynamic, dependency-resolving saga orchestrator for Ruby. Ruby Reactor impleme
14
25
  - **Interrupts**: Pause and resume workflows to wait for external events (webhooks, user approvals).
15
26
  - **Input Validation**: Integrated with `dry-validation` for robust input checking.
16
27
 
28
+ ## Comparison
29
+
30
+ | Feature | Ruby Reactor | dry-transaction | Trailblazer | Custom Sidekiq Jobs |
31
+ |--------------------------|--------------|-----------------|-------------|---------------------|
32
+ | DAG/Parallel execution | Yes | No | Limited | Manual |
33
+ | Auto compensation/undo | Yes | No | Manual | Manual |
34
+ | Interrupts (pause/resume)| Yes | No | No | Manual |
35
+ | Built-in web dashboard | Yes | No | No | No |
36
+ | Async with Sidekiq | Yes | No | Limited | Yes |
37
+
38
+ ## Real-World Use Cases
39
+
40
+ - **E-commerce Checkout**: Orchestrate inventory reservation, payment authorization, and shipping label generation. If payment fails, automatically release inventory and cancel the shipping request.
41
+ - **Data Import Pipelines**: Ingest optional massive CSVs using `map` steps to validate and upsert records in parallel. If data validation fails for a chunk, fail fast or collect errors while letting valid chunks succeed.
42
+ - **Subscription Billing**: Coordinate Stripe charges, invoice email generation, and internal entitlement updates. Use interrupts to pause the workflow when 3rd-party APIs are required to continue the workflow or when specific customer approval is needed.
43
+
44
+ ## Table of Contents
45
+ - [Features](#features)
46
+ - [Comparison](#comparison)
47
+ - [Real-World Use Cases](#real-world-use-cases)
48
+ - [Installation](#installation)
49
+ - [Configuration](#configuration)
50
+ - [Quick Start](#quick-start)
51
+ - [Web Dashboard](#web-dashboard)
52
+ - [Rails Installation](#rails-installation)
53
+ - [Usage](#usage)
54
+ - [Basic Example: User Registration](#basic-example-user-registration)
55
+ - [Async Execution](#async-execution)
56
+ - [Full Reactor Async](#full-reactor-async)
57
+ - [Step-Level Async](#step-level-async)
58
+ - [Interrupts (Pause & Resume)](#interrupts-pause--resume)
59
+ - [Map & Parallel Execution](#map--parallel-execution)
60
+ - [Map with Dynamic Source (ActiveRecord)](#map-with-dynamic-source-activerecord)
61
+ - [Input Validation](#input-validation)
62
+ - [Complex Workflows with Dependencies](#complex-workflows-with-dependencies)
63
+ - [Error Handling and Compensation](#error-handling-and-compensation)
64
+ - [Using Pre-defined Schemas](#using-pre-defined-schemas)
65
+ - [Documentation](#documentation)
66
+ - [Future improvements](#future-improvements)
67
+ - [Development](#development)
68
+ - [Contributing](#contributing)
69
+ - [Code of Conduct](#code-of-conduct)
70
+
17
71
  ## Installation
18
72
 
19
73
  Add this line to your application's Gemfile:
@@ -50,6 +104,21 @@ RubyReactor.configure do |config|
50
104
  end
51
105
  ```
52
106
 
107
+
108
+ ## Quick Start
109
+
110
+ ```ruby
111
+ class HelloReactor < RubyReactor::Reactor
112
+ step :greet do
113
+ run { Success("Hello from Ruby Reactor!") }
114
+ end
115
+ returns :greet
116
+ end
117
+
118
+ result = HelloReactor.run
119
+ puts result.value # => "Hello from Ruby Reactor!"
120
+ ```
121
+
53
122
  ## Web Dashboard
54
123
 
55
124
  RubyReactor comes with a built-in web dashboard to inspect reactor executions, view logs, and retry failed steps.
@@ -281,6 +350,42 @@ class DataProcessingReactor < RubyReactor::Reactor
281
350
  end
282
351
  ```
283
352
 
353
+ By using `async true` with `batch_size`, the system applies **Back Pressure** to efficiently manage resources. [Read more about Back Pressure & Resource Management](documentation/data_pipelines.md#back-pressure--resource-management).
354
+
355
+ #### Map with Dynamic Source (ActiveRecord)
356
+
357
+ You can use a block for `source` to dynamically fetch data, such as from ActiveRecord queries. The result is wrapped in a `ResultEnumerator` for easy access to successes and failures.
358
+
359
+ ```ruby
360
+ map :archive_old_users do
361
+ argument :days, input(:days)
362
+
363
+ # Dynamic source using ActiveRecord
364
+ source do |args|
365
+ User.where("last_login_at < ?", args[:days].days.ago)
366
+ end
367
+
368
+ argument :user, element(:archive_old_users)
369
+ async true, batch_size: 100
370
+
371
+ step :archive do
372
+ argument :user, input(:user)
373
+ run { |args| args[:user].archive! }
374
+ end
375
+
376
+ returns :archive
377
+ end
378
+
379
+ step :summary do
380
+ argument :results, result(:archive_old_users)
381
+ run do |args|
382
+ puts "Archived: #{args[:results].successes.count}"
383
+ puts "Failed: #{args[:results].failures.count}"
384
+ Success()
385
+ end
386
+ end
387
+ ```
388
+
284
389
  ### Input Validation
285
390
 
286
391
  RubyReactor integrates with dry-validation for input validation:
data/Rakefile CHANGED
@@ -30,8 +30,8 @@ namespace :server do
30
30
  end
31
31
  end
32
32
 
33
- # require "rubocop/rake_task"
33
+ require "rubocop/rake_task"
34
34
 
35
- # RuboCop::RakeTask.new
35
+ RuboCop::RakeTask.new
36
36
 
37
37
  task default: %i[spec rubocop]
@@ -43,53 +43,35 @@ class UserTransformationReactor < RubyReactor::Reactor
43
43
  end
44
44
  ```
45
45
 
46
- ## Async Execution
46
+ ## Dynamic Sources & ActiveRecord
47
47
 
48
- For long-running or resource-intensive tasks, you can offload processing to background jobs using Sidekiq.
49
-
50
- To enable async execution, simply add the `async true` directive to your map definition.
48
+ The `map` step supports a dynamic `source` block, which is particularly useful when working with ActiveRecord or when the collection depends on input arguments. Instead of passing a static collection, you can define a block that returns an Enumerable or an `ActiveRecord::Relation`.
51
49
 
52
50
  ```ruby
53
- map :process_orders do
54
- source input(:orders)
55
- argument :order, element(:process_orders)
56
-
57
- # Enable async execution via Sidekiq
58
- async true
59
-
60
- step :charge_card do
61
- argument :order, input(:order)
62
- run { PaymentService.charge(args[:order]) }
51
+ map :process_products do
52
+ argument :filter, input(:filter)
53
+
54
+ # Dynamic source block
55
+ source do |args|
56
+ # This block executes at runtime
57
+ threshold = args[:filter][:stock]
58
+ Product.where("stock >= ?", threshold)
63
59
  end
64
60
 
65
- returns :charge_card
61
+ argument :product, element(:process_products)
62
+ async true, batch_size: 100
63
+
64
+ step :process do
65
+ # ...
66
+ end
67
+
68
+ returns :process
66
69
  end
67
70
  ```
68
71
 
69
- ### Execution Flow
70
-
71
- ```mermaid
72
- sequenceDiagram
73
- participant Reactor
74
- participant Redis
75
- participant Sidekiq
76
- participant Worker
77
-
78
- Reactor->>Redis: Store Context
79
- Reactor->>Sidekiq: Enqueue MapElementWorkers
80
- Note over Reactor: Returns AsyncResult immediately
81
-
82
- loop For each element
83
- Sidekiq->>Worker: Process Element
84
- Worker->>Redis: Update Element Result
85
- end
86
-
87
- Worker->>Sidekiq: Enqueue MapCollectorWorker (when done)
88
- Sidekiq->>Worker: Run Collector
89
- Worker->>Redis: Store Final Result
90
- ```
72
+ When an `ActiveRecord::Relation` is returned, RubyReactor efficiently batches the query using database-level `OFFSET` and `LIMIT` based on the configured `batch_size`, preventing memory bloat by not loading all records at once.
91
73
 
92
- ## Batch Processing
74
+ ## Batch Processing Mechanism
93
75
 
94
76
  When processing large datasets asynchronously, you can control the parallelism using `batch_size`. This limits how many Sidekiq jobs are enqueued simultaneously, preventing system overload.
95
77
 
@@ -107,10 +89,42 @@ map :bulk_import do
107
89
  end
108
90
  ```
109
91
 
110
- **How it works:**
111
- 1. The system initially enqueues `batch_size` jobs.
112
- 2. As each job completes, it triggers the next job in the queue.
113
- 3. This maintains a steady stream of processing without flooding the queue.
92
+ ### Back Pressure & Resource Management
93
+
94
+ When `async true` is used with a `batch_size`, RubyReactor implements an intelligent **back pressure** mechanism. Instead of flooding Redis and Sidekiq with millions of jobs immediately (which is the standard behavior for many background job systems), the system processes data in controlled chunks.
95
+
96
+ This approach provides critical benefits for stability and scalability:
97
+
98
+ 1. **Memory Efficiency**: By using `ActiveRecord` batching (`LIMIT` / `OFFSET`), only the current batch of records is loaded into memory. This allows processing datasets larger than available RAM.
99
+ 2. **Redis Protection**: Prevents "Queue Explosion". Only a small number of job arguments are stored in Redis at any time, preventing OOM errors in your Redis instance.
100
+ 3. **Database Stability**: Database load is distributed over time rather than spiking all at once.
101
+
102
+ **Visualizing the Flow:**
103
+
104
+ ```mermaid
105
+ graph TD
106
+ Start[Start Map] -->|Batch Size: N| BatchManager
107
+
108
+ subgraph "Back Pressure Loop"
109
+ BatchManager[Batch Manager] -->|Fetch N Items| DB[(Database)]
110
+ DB --> Records
111
+ Records -->|Enqueue N Jobs| Sidekiq
112
+
113
+ Sidekiq --> W1[Worker 1]
114
+ Sidekiq --> W2[Worker 2]
115
+
116
+ W1 -.->|Complete| Check{Batch Done?}
117
+ W2 -.->|Complete| Check
118
+
119
+ Check -->|No| Wait[Wait]
120
+ Check -->|Yes| Next[Trigger Next Batch]
121
+ Next --> BatchManager
122
+ end
123
+
124
+ BatchManager -->|No More Items| Finish[Aggregator]
125
+ ```
126
+
127
+ This ensures that the system works at the speed of your workers, not the speed of the enqueueing process, maintaining a constant and manageable resource footprint regardless of dataset size.
114
128
 
115
129
  ## Error Handling
116
130
 
@@ -128,9 +142,9 @@ map :strict_processing do
128
142
  end
129
143
  ```
130
144
 
131
- ### Collecting Partial Results
145
+ ### Collecting Results (Successes & Failures)
132
146
 
133
- If you want to process all elements regardless of failures, set `fail_fast false`. You can then use a `collect` block to handle successes and failures separately.
147
+ If you want to process all elements regardless of failures, set `fail_fast false`. The map step returns a `ResultEnumerator` that allows you to easily separate successful executions from failures.
134
148
 
135
149
  ```ruby
136
150
  map :resilient_processing do
@@ -145,18 +159,39 @@ map :resilient_processing do
145
159
  end
146
160
 
147
161
  returns :risky_operation
162
+ end
163
+
164
+ step :analyze_results do
165
+ argument :results, result(:resilient_processing)
166
+
167
+ run do |args|
168
+ col = args[:results]
169
+
170
+ # Iterate over successful results
171
+ col.successes.each do |value|
172
+ # 'value' is the direct return value of the map element
173
+ puts "Success: #{value}"
174
+ end
175
+
176
+ # Iterate over failures
177
+ col.failures.each do |error|
178
+ # 'error' is the failure object/message itself
179
+ puts "Error: #{error}"
180
+ end
181
+
182
+ # Note: Iterating the collection directly yields wrapped objects
183
+ col.each do |result|
184
+ if result.is_a?(RubyReactor::Success)
185
+ puts "Wrapped Value: #{result.value}"
186
+ else
187
+ puts "Wrapped Error: #{result.error}"
188
+ end
189
+ end
148
190
 
149
- # Aggregate results
150
- collect do |results|
151
- # results is an array of Result objects (Success or Failure)
152
- successful = results.select(&:success?).map(&:value)
153
- failed = results.select(&:failure?).map(&:error)
154
-
155
- {
156
- processed: successful,
157
- errors: failed,
158
- success_rate: successful.length.to_f / results.length
159
- }
191
+ Success({
192
+ success_count: col.successes.count,
193
+ failure_count: col.failures.count
194
+ })
160
195
  end
161
196
  end
162
197
  ```
@@ -192,33 +227,4 @@ end
192
227
  - **Async Mode**: Retries are handled by requeuing the Sidekiq job with a delay. This is non-blocking and efficient.
193
228
  - **Sync Mode**: Retries happen immediately within the execution thread (blocking).
194
229
 
195
- ## Visualization
196
230
 
197
- ### Async Batch Execution
198
-
199
- ```mermaid
200
- graph TD
201
- Start[Start Map] --> Init[Initialize Batch]
202
- Init --> Q1["Queue Initial Batch<br/>(Size N)"]
203
-
204
- subgraph Workers
205
- W1[Worker 1]
206
- W2[Worker 2]
207
- W3[Worker ...]
208
- end
209
-
210
- Q1 --> W1
211
- Q1 --> W2
212
-
213
- W1 -->|Complete| Next1{More Items?}
214
- W2 -->|Complete| Next2{More Items?}
215
-
216
- Next1 -->|Yes| Q2[Queue Next Item]
217
- Next2 -->|Yes| Q2
218
-
219
- Q2 --> W3
220
-
221
- Next1 -->|No| Check{All Done?}
222
- Check -->|Yes| Collect[Run Collector]
223
- Collect --> Finish[Final Result]
224
- ```
@@ -19,7 +19,7 @@ module RubyReactor
19
19
  # rubocop:disable Metrics/ParameterLists
20
20
  def self.perform_map_element_async(map_id:, element_id:, index:, serialized_inputs:, reactor_class_info:,
21
21
  strict_ordering:, parent_context_id:, parent_reactor_class_name:, step_name:,
22
- batch_size: nil, serialized_context: nil)
22
+ batch_size: nil, serialized_context: nil, fail_fast: nil)
23
23
  RubyReactor::SidekiqWorkers::MapElementWorker.perform_async(
24
24
  {
25
25
  "map_id" => map_id,
@@ -32,7 +32,8 @@ module RubyReactor
32
32
  "parent_reactor_class_name" => parent_reactor_class_name,
33
33
  "step_name" => step_name,
34
34
  "batch_size" => batch_size,
35
- "serialized_context" => serialized_context
35
+ "serialized_context" => serialized_context,
36
+ "fail_fast" => fail_fast
36
37
  }
37
38
  )
38
39
  end
@@ -75,9 +76,8 @@ module RubyReactor
75
76
  end
76
77
  # rubocop:enable Metrics/ParameterLists
77
78
 
78
- # rubocop:disable Metrics/ParameterLists
79
79
  def self.perform_map_execution_async(map_id:, serialized_inputs:, reactor_class_info:, strict_ordering:,
80
- parent_context_id:, parent_reactor_class_name:, step_name:)
80
+ parent_context_id:, parent_reactor_class_name:, step_name:, fail_fast: nil)
81
81
  RubyReactor::SidekiqWorkers::MapExecutionWorker.perform_async(
82
82
  {
83
83
  "map_id" => map_id,
@@ -86,10 +86,10 @@ module RubyReactor
86
86
  "strict_ordering" => strict_ordering,
87
87
  "parent_context_id" => parent_context_id,
88
88
  "parent_reactor_class_name" => parent_reactor_class_name,
89
- "step_name" => step_name
89
+ "step_name" => step_name,
90
+ "fail_fast" => fail_fast
90
91
  }
91
92
  )
92
93
  end
93
- # rubocop:enable Metrics/ParameterLists
94
94
  end
95
95
  end
@@ -68,6 +68,14 @@ module RubyReactor
68
68
  { "_type" => "Template::Value", "value" => serialize_value(value.instance_variable_get(:@value)) }
69
69
  when RubyReactor::Template::Result
70
70
  { "_type" => "Template::Result", "step_name" => value.step_name.to_s, "path" => value.path }
71
+ when RubyReactor::Map::ResultEnumerator
72
+ {
73
+ "_type" => "Map::ResultEnumerator",
74
+ "map_id" => value.map_id,
75
+ "reactor_class_name" => value.reactor_class_name,
76
+ "strict_ordering" => value.strict_ordering,
77
+ "batch_size" => value.batch_size
78
+ }
71
79
  when Hash
72
80
  value.transform_keys(&:to_s).transform_values { |v| serialize_value(v) }
73
81
  when Array
@@ -115,6 +123,13 @@ module RubyReactor
115
123
  RubyReactor::Template::Value.new(deserialize_value(value["value"]))
116
124
  when "Template::Result"
117
125
  RubyReactor::Template::Result.new(value["step_name"], value["path"])
126
+ when "Map::ResultEnumerator"
127
+ RubyReactor::Map::ResultEnumerator.new(
128
+ value["map_id"],
129
+ value["reactor_class_name"],
130
+ strict_ordering: value["strict_ordering"],
131
+ batch_size: value["batch_size"]
132
+ )
118
133
  else
119
134
  value
120
135
  end
@@ -32,8 +32,12 @@ module RubyReactor
32
32
  @argument_mappings[mapped_input_name] = source
33
33
  end
34
34
 
35
- def source(enumerable)
36
- @source_enumerable = enumerable
35
+ def source(enumerable = nil, &block)
36
+ @source_enumerable = if block
37
+ RubyReactor::Template::DynamicSource.new(@argument_mappings, &block)
38
+ else
39
+ enumerable
40
+ end
37
41
  end
38
42
 
39
43
  def async(async = true, batch_size: nil)
@@ -117,6 +117,7 @@ module RubyReactor
117
117
 
118
118
  def store_failed_map_context(context)
119
119
  return unless context.map_metadata && context.map_metadata[:map_id]
120
+ return unless context.map_metadata[:fail_fast]
120
121
 
121
122
  storage = RubyReactor.configuration.storage_adapter
122
123
  storage.store_map_failed_context_id(
@@ -39,7 +39,17 @@ module RubyReactor
39
39
 
40
40
  # Serialize context and requeue the job
41
41
  # Use root context if available to ensure we serialize the full tree
42
- context_to_serialize = @context.root_context || @context
42
+ # BUT for map elements (which have map_metadata), we must serialize the element context itself
43
+
44
+ context_to_serialize = if @context.map_metadata
45
+ @context
46
+ else
47
+ @context.root_context || @context
48
+ end
49
+
50
+ puts "SERIALIZING CONTEXT: #{context_to_serialize.reactor_class.name}"
51
+ puts "INPUTS KEYS: #{context_to_serialize.inputs.keys}" if context_to_serialize.respond_to?(:inputs)
52
+
43
53
  reactor_class_name = context_to_serialize.reactor_class.name
44
54
 
45
55
  serialized_context = ContextSerializer.serialize(context_to_serialize)
@@ -7,56 +7,94 @@ module RubyReactor
7
7
 
8
8
  def self.perform(arguments)
9
9
  arguments = arguments.transform_keys(&:to_sym)
10
- parent_context_id = arguments[:parent_context_id]
11
10
  map_id = arguments[:map_id]
11
+ parent_context_id = arguments[:parent_context_id]
12
12
  parent_reactor_class_name = arguments[:parent_reactor_class_name]
13
13
  step_name = arguments[:step_name]
14
14
  strict_ordering = arguments[:strict_ordering]
15
+ # timeout = arguments[:timeout]
15
16
 
16
17
  storage = RubyReactor.configuration.storage_adapter
18
+ parent_context_data = storage.retrieve_context(parent_context_id, parent_reactor_class_name)
19
+ parent_context = RubyReactor::Context.deserialize_from_retry(parent_context_data)
20
+
21
+ # Check if all tasks are completed
22
+ metadata = storage.retrieve_map_metadata(map_id, parent_reactor_class_name)
23
+ total_count = metadata ? metadata["count"].to_i : 0
24
+
25
+ results_count = storage.count_map_results(map_id, parent_reactor_class_name)
26
+
27
+ # Not done yet, requeue or wait?
28
+ # Actually Collector currently assumes we only call it when we expect completion or check progress
29
+ # Since map_offset tracks dispatching progress and might exceed count due to batching reservation,
30
+ # we must strictly check against the total count of elements.
31
+ # Check for fail_fast failure FIRST
32
+ failed_context_id = storage.retrieve_map_failed_context_id(map_id, parent_reactor_class_name)
33
+ if failed_context_id
34
+ # Resolve the class of the mapped reactor to retrieve its context
35
+ reactor_class = resolve_reactor_class(metadata["reactor_class_info"])
17
36
 
18
- # Retrieve parent context
19
- parent_context = load_parent_context_from_storage(
20
- parent_context_id,
37
+ failed_context_data = storage.retrieve_context(failed_context_id, reactor_class.name)
38
+
39
+ if failed_context_data
40
+ failed_context = RubyReactor::Context.deserialize_from_retry(failed_context_data)
41
+
42
+ # Resume parent execution (which marks step as failed)
43
+ resume_parent_execution(parent_context, step_name, RubyReactor::Failure(failed_context.failure_reason),
44
+ storage)
45
+ return
46
+ end
47
+ end
48
+
49
+ return if results_count < total_count
50
+
51
+ # Retrieve results lazily
52
+ results = RubyReactor::Map::ResultEnumerator.new(
53
+ map_id,
21
54
  parent_reactor_class_name,
22
- storage
55
+ strict_ordering: strict_ordering
23
56
  )
24
57
 
25
- # Retrieve results
26
- serialized_results = storage.retrieve_map_results(map_id, parent_reactor_class_name,
27
- strict_ordering: strict_ordering)
58
+ # Apply collect block (or default collection)
59
+ step_config = parent_context.reactor_class.steps[step_name.to_sym]
28
60
 
29
- results = serialized_results.map do |r|
30
- if r.is_a?(Hash) && r.key?("_error")
31
- RubyReactor::Failure(r["_error"])
32
- else
33
- RubyReactor::Success(r)
61
+ begin
62
+ final_result = apply_collect_block(results, step_config)
63
+
64
+ if final_result.failure?
65
+ # Optionally log failure internally or just rely on context status update
34
66
  end
67
+ rescue StandardError => e
68
+ final_result = RubyReactor::Failure(e)
35
69
  end
36
70
 
37
- # Get step config to check for collect block and other options
38
- parent_class = Object.const_get(parent_reactor_class_name)
39
- step_config = parent_class.steps[step_name.to_sym]
71
+ # Resume parent execution
72
+ resume_parent_execution(parent_context, step_name, final_result, storage)
73
+ rescue StandardError => e
74
+ puts "COLLECTOR CRASH: #{e.message}"
75
+ puts e.backtrace
76
+ raise e
77
+ end
40
78
 
41
- collect_block = step_config.arguments[:collect_block][:source].value
79
+ def self.apply_collect_block(results, step_config)
80
+ collect_block = step_config.arguments[:collect_block][:source].value if step_config.arguments[:collect_block]
42
81
  # TODO: Check allow_partial_failure option
43
82
 
44
- final_result = if collect_block
45
- begin
46
- # Pass all results (Success and Failure) to collect block
47
- collected = collect_block.call(results)
48
- RubyReactor::Success(collected)
49
- rescue StandardError => e
50
- RubyReactor::Failure(e)
51
- end
52
- else
53
- # Default behavior: fail if any failure
54
- first_failure = results.find(&:failure?)
55
- first_failure || RubyReactor::Success(results.map(&:value))
56
- end
57
-
58
- # Resume execution
59
- resume_parent_execution(parent_context, step_name, final_result, storage)
83
+ if collect_block
84
+ begin
85
+ # Pass Enumerator to collect block
86
+ collected = collect_block.call(results)
87
+ RubyReactor::Success(collected)
88
+ rescue StandardError => e
89
+ puts "COLLECTOR INNER EXCEPTION: #{e.message}"
90
+ puts e.backtrace
91
+ RubyReactor::Failure(e)
92
+ end
93
+ else
94
+ # Default behavior: Return Success(Enumerator).
95
+ # Logic for checking failures is deferred to the consumer of the enumerator.
96
+ RubyReactor::Success(results)
97
+ end
60
98
  end
61
99
  end
62
100
  end