ruby_reactor 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +105 -0
- data/Rakefile +2 -2
- data/documentation/data_pipelines.md +90 -84
- data/lib/ruby_reactor/async_router.rb +6 -6
- data/lib/ruby_reactor/context_serializer.rb +15 -0
- data/lib/ruby_reactor/dsl/map_builder.rb +6 -2
- data/lib/ruby_reactor/executor/result_handler.rb +1 -0
- data/lib/ruby_reactor/executor/retry_manager.rb +11 -1
- data/lib/ruby_reactor/map/collector.rb +71 -33
- data/lib/ruby_reactor/map/dispatcher.rb +162 -0
- data/lib/ruby_reactor/map/element_executor.rb +59 -56
- data/lib/ruby_reactor/map/execution.rb +16 -3
- data/lib/ruby_reactor/map/helpers.rb +2 -2
- data/lib/ruby_reactor/map/result_enumerator.rb +105 -0
- data/lib/ruby_reactor/step/map_step.rb +48 -16
- data/lib/ruby_reactor/storage/redis_adapter.rb +59 -0
- data/lib/ruby_reactor/template/dynamic_source.rb +32 -0
- data/lib/ruby_reactor/version.rb +1 -1
- metadata +5 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 863c83e92cbfab470e39107c580ceabce444133973b723662a89e5bf0211c3ab
|
|
4
|
+
data.tar.gz: 1209b5587efc055bc787731694a6dc2b7eab8b3d5a6ff729728aa29ea601e2e0
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 146d14cd32b12c95764e493f2de2d826dbe074ec4e4b44613b788d748d52fe38644fd86f1590f6bb0b1a0d1840c2200ab29a6d8e78db27f8f6abf099faf74756
|
|
7
|
+
data.tar.gz: 3d5b7c43182ee2d1c5c06ba90b7e2177a3dffbecdc8fb3e91db14f99bcd02807ebfa98c0e240412b790ec1d69609dc30995d9ee1aea9c8cb8ecd69d4d0fb2ac7
|
data/README.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
|
+
[](https://badge.fury.io/rb/ruby_reactor)
|
|
2
|
+
[](https://github.com/arturictus/ruby_reactor/actions) <!-- if you have CI -->
|
|
3
|
+
[](https://github.com/rubocop/rubocop)
|
|
4
|
+
[](https://rubygems.org/gems/ruby_reactor)
|
|
5
|
+
|
|
1
6
|
# RubyReactor
|
|
2
7
|
|
|
8
|
+
## Why Ruby Reactor?
|
|
9
|
+
|
|
10
|
+
Building complex business transactions often results in spaghetti code or brittle "god classes." Ruby Reactor solves this by implementing the **Saga Pattern** in a lightweight, developer-friendly package. It lets you define workflows as clear, dependency-driven steps without the boilerplate of heavy enterprise frameworks.
|
|
11
|
+
|
|
12
|
+
The key value is **Reliability**: if any part of your workflow fails, Ruby Reactor automatically triggers compensation logic to undo previous steps, ensuring your system never ends up in a corrupted half-state. Whether you're coordinating microservices or monolith modules, you get atomic-like consistency with background processing built-in.
|
|
13
|
+
|
|
3
14
|
A dynamic, dependency-resolving saga orchestrator for Ruby. Ruby Reactor implements the Saga pattern with compensation-based error handling and DAG-based execution planning. It leverages **Sidekiq** for asynchronous execution and **Redis** for state persistence.
|
|
4
15
|
|
|
5
16
|

|
|
@@ -14,6 +25,49 @@ A dynamic, dependency-resolving saga orchestrator for Ruby. Ruby Reactor impleme
|
|
|
14
25
|
- **Interrupts**: Pause and resume workflows to wait for external events (webhooks, user approvals).
|
|
15
26
|
- **Input Validation**: Integrated with `dry-validation` for robust input checking.
|
|
16
27
|
|
|
28
|
+
## Comparison
|
|
29
|
+
|
|
30
|
+
| Feature | Ruby Reactor | dry-transaction | Trailblazer | Custom Sidekiq Jobs |
|
|
31
|
+
|--------------------------|--------------|-----------------|-------------|---------------------|
|
|
32
|
+
| DAG/Parallel execution | Yes | No | Limited | Manual |
|
|
33
|
+
| Auto compensation/undo | Yes | No | Manual | Manual |
|
|
34
|
+
| Interrupts (pause/resume)| Yes | No | No | Manual |
|
|
35
|
+
| Built-in web dashboard | Yes | No | No | No |
|
|
36
|
+
| Async with Sidekiq | Yes | No | Limited | Yes |
|
|
37
|
+
|
|
38
|
+
## Real-World Use Cases
|
|
39
|
+
|
|
40
|
+
- **E-commerce Checkout**: Orchestrate inventory reservation, payment authorization, and shipping label generation. If payment fails, automatically release inventory and cancel the shipping request.
|
|
41
|
+
- **Data Import Pipelines**: Ingest optional massive CSVs using `map` steps to validate and upsert records in parallel. If data validation fails for a chunk, fail fast or collect errors while letting valid chunks succeed.
|
|
42
|
+
- **Subscription Billing**: Coordinate Stripe charges, invoice email generation, and internal entitlement updates. Use interrupts to pause the workflow when 3rd-party APIs are required to continue the workflow or when specific customer approval is needed.
|
|
43
|
+
|
|
44
|
+
## Table of Contents
|
|
45
|
+
- [Features](#features)
|
|
46
|
+
- [Comparison](#comparison)
|
|
47
|
+
- [Real-World Use Cases](#real-world-use-cases)
|
|
48
|
+
- [Installation](#installation)
|
|
49
|
+
- [Configuration](#configuration)
|
|
50
|
+
- [Quick Start](#quick-start)
|
|
51
|
+
- [Web Dashboard](#web-dashboard)
|
|
52
|
+
- [Rails Installation](#rails-installation)
|
|
53
|
+
- [Usage](#usage)
|
|
54
|
+
- [Basic Example: User Registration](#basic-example-user-registration)
|
|
55
|
+
- [Async Execution](#async-execution)
|
|
56
|
+
- [Full Reactor Async](#full-reactor-async)
|
|
57
|
+
- [Step-Level Async](#step-level-async)
|
|
58
|
+
- [Interrupts (Pause & Resume)](#interrupts-pause--resume)
|
|
59
|
+
- [Map & Parallel Execution](#map--parallel-execution)
|
|
60
|
+
- [Map with Dynamic Source (ActiveRecord)](#map-with-dynamic-source-activerecord)
|
|
61
|
+
- [Input Validation](#input-validation)
|
|
62
|
+
- [Complex Workflows with Dependencies](#complex-workflows-with-dependencies)
|
|
63
|
+
- [Error Handling and Compensation](#error-handling-and-compensation)
|
|
64
|
+
- [Using Pre-defined Schemas](#using-pre-defined-schemas)
|
|
65
|
+
- [Documentation](#documentation)
|
|
66
|
+
- [Future improvements](#future-improvements)
|
|
67
|
+
- [Development](#development)
|
|
68
|
+
- [Contributing](#contributing)
|
|
69
|
+
- [Code of Conduct](#code-of-conduct)
|
|
70
|
+
|
|
17
71
|
## Installation
|
|
18
72
|
|
|
19
73
|
Add this line to your application's Gemfile:
|
|
@@ -50,6 +104,21 @@ RubyReactor.configure do |config|
|
|
|
50
104
|
end
|
|
51
105
|
```
|
|
52
106
|
|
|
107
|
+
|
|
108
|
+
## Quick Start
|
|
109
|
+
|
|
110
|
+
```ruby
|
|
111
|
+
class HelloReactor < RubyReactor::Reactor
|
|
112
|
+
step :greet do
|
|
113
|
+
run { Success("Hello from Ruby Reactor!") }
|
|
114
|
+
end
|
|
115
|
+
returns :greet
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
result = HelloReactor.run
|
|
119
|
+
puts result.value # => "Hello from Ruby Reactor!"
|
|
120
|
+
```
|
|
121
|
+
|
|
53
122
|
## Web Dashboard
|
|
54
123
|
|
|
55
124
|
RubyReactor comes with a built-in web dashboard to inspect reactor executions, view logs, and retry failed steps.
|
|
@@ -281,6 +350,42 @@ class DataProcessingReactor < RubyReactor::Reactor
|
|
|
281
350
|
end
|
|
282
351
|
```
|
|
283
352
|
|
|
353
|
+
By using `async true` with `batch_size`, the system applies **Back Pressure** to efficiently manage resources. [Read more about Back Pressure & Resource Management](documentation/data_pipelines.md#back-pressure--resource-management).
|
|
354
|
+
|
|
355
|
+
#### Map with Dynamic Source (ActiveRecord)
|
|
356
|
+
|
|
357
|
+
You can use a block for `source` to dynamically fetch data, such as from ActiveRecord queries. The result is wrapped in a `ResultEnumerator` for easy access to successes and failures.
|
|
358
|
+
|
|
359
|
+
```ruby
|
|
360
|
+
map :archive_old_users do
|
|
361
|
+
argument :days, input(:days)
|
|
362
|
+
|
|
363
|
+
# Dynamic source using ActiveRecord
|
|
364
|
+
source do |args|
|
|
365
|
+
User.where("last_login_at < ?", args[:days].days.ago)
|
|
366
|
+
end
|
|
367
|
+
|
|
368
|
+
argument :user, element(:archive_old_users)
|
|
369
|
+
async true, batch_size: 100
|
|
370
|
+
|
|
371
|
+
step :archive do
|
|
372
|
+
argument :user, input(:user)
|
|
373
|
+
run { |args| args[:user].archive! }
|
|
374
|
+
end
|
|
375
|
+
|
|
376
|
+
returns :archive
|
|
377
|
+
end
|
|
378
|
+
|
|
379
|
+
step :summary do
|
|
380
|
+
argument :results, result(:archive_old_users)
|
|
381
|
+
run do |args|
|
|
382
|
+
puts "Archived: #{args[:results].successes.count}"
|
|
383
|
+
puts "Failed: #{args[:results].failures.count}"
|
|
384
|
+
Success()
|
|
385
|
+
end
|
|
386
|
+
end
|
|
387
|
+
```
|
|
388
|
+
|
|
284
389
|
### Input Validation
|
|
285
390
|
|
|
286
391
|
RubyReactor integrates with dry-validation for input validation:
|
data/Rakefile
CHANGED
|
@@ -43,53 +43,35 @@ class UserTransformationReactor < RubyReactor::Reactor
|
|
|
43
43
|
end
|
|
44
44
|
```
|
|
45
45
|
|
|
46
|
-
##
|
|
46
|
+
## Dynamic Sources & ActiveRecord
|
|
47
47
|
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
To enable async execution, simply add the `async true` directive to your map definition.
|
|
48
|
+
The `map` step supports a dynamic `source` block, which is particularly useful when working with ActiveRecord or when the collection depends on input arguments. Instead of passing a static collection, you can define a block that returns an Enumerable or an `ActiveRecord::Relation`.
|
|
51
49
|
|
|
52
50
|
```ruby
|
|
53
|
-
map :
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
argument :order, input(:order)
|
|
62
|
-
run { PaymentService.charge(args[:order]) }
|
|
51
|
+
map :process_products do
|
|
52
|
+
argument :filter, input(:filter)
|
|
53
|
+
|
|
54
|
+
# Dynamic source block
|
|
55
|
+
source do |args|
|
|
56
|
+
# This block executes at runtime
|
|
57
|
+
threshold = args[:filter][:stock]
|
|
58
|
+
Product.where("stock >= ?", threshold)
|
|
63
59
|
end
|
|
64
60
|
|
|
65
|
-
|
|
61
|
+
argument :product, element(:process_products)
|
|
62
|
+
async true, batch_size: 100
|
|
63
|
+
|
|
64
|
+
step :process do
|
|
65
|
+
# ...
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
returns :process
|
|
66
69
|
end
|
|
67
70
|
```
|
|
68
71
|
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
```mermaid
|
|
72
|
-
sequenceDiagram
|
|
73
|
-
participant Reactor
|
|
74
|
-
participant Redis
|
|
75
|
-
participant Sidekiq
|
|
76
|
-
participant Worker
|
|
77
|
-
|
|
78
|
-
Reactor->>Redis: Store Context
|
|
79
|
-
Reactor->>Sidekiq: Enqueue MapElementWorkers
|
|
80
|
-
Note over Reactor: Returns AsyncResult immediately
|
|
81
|
-
|
|
82
|
-
loop For each element
|
|
83
|
-
Sidekiq->>Worker: Process Element
|
|
84
|
-
Worker->>Redis: Update Element Result
|
|
85
|
-
end
|
|
86
|
-
|
|
87
|
-
Worker->>Sidekiq: Enqueue MapCollectorWorker (when done)
|
|
88
|
-
Sidekiq->>Worker: Run Collector
|
|
89
|
-
Worker->>Redis: Store Final Result
|
|
90
|
-
```
|
|
72
|
+
When an `ActiveRecord::Relation` is returned, RubyReactor efficiently batches the query using database-level `OFFSET` and `LIMIT` based on the configured `batch_size`, preventing memory bloat by not loading all records at once.
|
|
91
73
|
|
|
92
|
-
## Batch Processing
|
|
74
|
+
## Batch Processing Mechanism
|
|
93
75
|
|
|
94
76
|
When processing large datasets asynchronously, you can control the parallelism using `batch_size`. This limits how many Sidekiq jobs are enqueued simultaneously, preventing system overload.
|
|
95
77
|
|
|
@@ -107,10 +89,42 @@ map :bulk_import do
|
|
|
107
89
|
end
|
|
108
90
|
```
|
|
109
91
|
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
92
|
+
### Back Pressure & Resource Management
|
|
93
|
+
|
|
94
|
+
When `async true` is used with a `batch_size`, RubyReactor implements an intelligent **back pressure** mechanism. Instead of flooding Redis and Sidekiq with millions of jobs immediately (which is the standard behavior for many background job systems), the system processes data in controlled chunks.
|
|
95
|
+
|
|
96
|
+
This approach provides critical benefits for stability and scalability:
|
|
97
|
+
|
|
98
|
+
1. **Memory Efficiency**: By using `ActiveRecord` batching (`LIMIT` / `OFFSET`), only the current batch of records is loaded into memory. This allows processing datasets larger than available RAM.
|
|
99
|
+
2. **Redis Protection**: Prevents "Queue Explosion". Only a small number of job arguments are stored in Redis at any time, preventing OOM errors in your Redis instance.
|
|
100
|
+
3. **Database Stability**: Database load is distributed over time rather than spiking all at once.
|
|
101
|
+
|
|
102
|
+
**Visualizing the Flow:**
|
|
103
|
+
|
|
104
|
+
```mermaid
|
|
105
|
+
graph TD
|
|
106
|
+
Start[Start Map] -->|Batch Size: N| BatchManager
|
|
107
|
+
|
|
108
|
+
subgraph "Back Pressure Loop"
|
|
109
|
+
BatchManager[Batch Manager] -->|Fetch N Items| DB[(Database)]
|
|
110
|
+
DB --> Records
|
|
111
|
+
Records -->|Enqueue N Jobs| Sidekiq
|
|
112
|
+
|
|
113
|
+
Sidekiq --> W1[Worker 1]
|
|
114
|
+
Sidekiq --> W2[Worker 2]
|
|
115
|
+
|
|
116
|
+
W1 -.->|Complete| Check{Batch Done?}
|
|
117
|
+
W2 -.->|Complete| Check
|
|
118
|
+
|
|
119
|
+
Check -->|No| Wait[Wait]
|
|
120
|
+
Check -->|Yes| Next[Trigger Next Batch]
|
|
121
|
+
Next --> BatchManager
|
|
122
|
+
end
|
|
123
|
+
|
|
124
|
+
BatchManager -->|No More Items| Finish[Aggregator]
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
This ensures that the system works at the speed of your workers, not the speed of the enqueueing process, maintaining a constant and manageable resource footprint regardless of dataset size.
|
|
114
128
|
|
|
115
129
|
## Error Handling
|
|
116
130
|
|
|
@@ -128,9 +142,9 @@ map :strict_processing do
|
|
|
128
142
|
end
|
|
129
143
|
```
|
|
130
144
|
|
|
131
|
-
### Collecting
|
|
145
|
+
### Collecting Results (Successes & Failures)
|
|
132
146
|
|
|
133
|
-
If you want to process all elements regardless of failures, set `fail_fast false`.
|
|
147
|
+
If you want to process all elements regardless of failures, set `fail_fast false`. The map step returns a `ResultEnumerator` that allows you to easily separate successful executions from failures.
|
|
134
148
|
|
|
135
149
|
```ruby
|
|
136
150
|
map :resilient_processing do
|
|
@@ -145,18 +159,39 @@ map :resilient_processing do
|
|
|
145
159
|
end
|
|
146
160
|
|
|
147
161
|
returns :risky_operation
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
step :analyze_results do
|
|
165
|
+
argument :results, result(:resilient_processing)
|
|
166
|
+
|
|
167
|
+
run do |args|
|
|
168
|
+
col = args[:results]
|
|
169
|
+
|
|
170
|
+
# Iterate over successful results
|
|
171
|
+
col.successes.each do |value|
|
|
172
|
+
# 'value' is the direct return value of the map element
|
|
173
|
+
puts "Success: #{value}"
|
|
174
|
+
end
|
|
175
|
+
|
|
176
|
+
# Iterate over failures
|
|
177
|
+
col.failures.each do |error|
|
|
178
|
+
# 'error' is the failure object/message itself
|
|
179
|
+
puts "Error: #{error}"
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
# Note: Iterating the collection directly yields wrapped objects
|
|
183
|
+
col.each do |result|
|
|
184
|
+
if result.is_a?(RubyReactor::Success)
|
|
185
|
+
puts "Wrapped Value: #{result.value}"
|
|
186
|
+
else
|
|
187
|
+
puts "Wrapped Error: #{result.error}"
|
|
188
|
+
end
|
|
189
|
+
end
|
|
148
190
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
failed = results.select(&:failure?).map(&:error)
|
|
154
|
-
|
|
155
|
-
{
|
|
156
|
-
processed: successful,
|
|
157
|
-
errors: failed,
|
|
158
|
-
success_rate: successful.length.to_f / results.length
|
|
159
|
-
}
|
|
191
|
+
Success({
|
|
192
|
+
success_count: col.successes.count,
|
|
193
|
+
failure_count: col.failures.count
|
|
194
|
+
})
|
|
160
195
|
end
|
|
161
196
|
end
|
|
162
197
|
```
|
|
@@ -192,33 +227,4 @@ end
|
|
|
192
227
|
- **Async Mode**: Retries are handled by requeuing the Sidekiq job with a delay. This is non-blocking and efficient.
|
|
193
228
|
- **Sync Mode**: Retries happen immediately within the execution thread (blocking).
|
|
194
229
|
|
|
195
|
-
## Visualization
|
|
196
230
|
|
|
197
|
-
### Async Batch Execution
|
|
198
|
-
|
|
199
|
-
```mermaid
|
|
200
|
-
graph TD
|
|
201
|
-
Start[Start Map] --> Init[Initialize Batch]
|
|
202
|
-
Init --> Q1["Queue Initial Batch<br/>(Size N)"]
|
|
203
|
-
|
|
204
|
-
subgraph Workers
|
|
205
|
-
W1[Worker 1]
|
|
206
|
-
W2[Worker 2]
|
|
207
|
-
W3[Worker ...]
|
|
208
|
-
end
|
|
209
|
-
|
|
210
|
-
Q1 --> W1
|
|
211
|
-
Q1 --> W2
|
|
212
|
-
|
|
213
|
-
W1 -->|Complete| Next1{More Items?}
|
|
214
|
-
W2 -->|Complete| Next2{More Items?}
|
|
215
|
-
|
|
216
|
-
Next1 -->|Yes| Q2[Queue Next Item]
|
|
217
|
-
Next2 -->|Yes| Q2
|
|
218
|
-
|
|
219
|
-
Q2 --> W3
|
|
220
|
-
|
|
221
|
-
Next1 -->|No| Check{All Done?}
|
|
222
|
-
Check -->|Yes| Collect[Run Collector]
|
|
223
|
-
Collect --> Finish[Final Result]
|
|
224
|
-
```
|
|
@@ -19,7 +19,7 @@ module RubyReactor
|
|
|
19
19
|
# rubocop:disable Metrics/ParameterLists
|
|
20
20
|
def self.perform_map_element_async(map_id:, element_id:, index:, serialized_inputs:, reactor_class_info:,
|
|
21
21
|
strict_ordering:, parent_context_id:, parent_reactor_class_name:, step_name:,
|
|
22
|
-
batch_size: nil, serialized_context: nil)
|
|
22
|
+
batch_size: nil, serialized_context: nil, fail_fast: nil)
|
|
23
23
|
RubyReactor::SidekiqWorkers::MapElementWorker.perform_async(
|
|
24
24
|
{
|
|
25
25
|
"map_id" => map_id,
|
|
@@ -32,7 +32,8 @@ module RubyReactor
|
|
|
32
32
|
"parent_reactor_class_name" => parent_reactor_class_name,
|
|
33
33
|
"step_name" => step_name,
|
|
34
34
|
"batch_size" => batch_size,
|
|
35
|
-
"serialized_context" => serialized_context
|
|
35
|
+
"serialized_context" => serialized_context,
|
|
36
|
+
"fail_fast" => fail_fast
|
|
36
37
|
}
|
|
37
38
|
)
|
|
38
39
|
end
|
|
@@ -75,9 +76,8 @@ module RubyReactor
|
|
|
75
76
|
end
|
|
76
77
|
# rubocop:enable Metrics/ParameterLists
|
|
77
78
|
|
|
78
|
-
# rubocop:disable Metrics/ParameterLists
|
|
79
79
|
def self.perform_map_execution_async(map_id:, serialized_inputs:, reactor_class_info:, strict_ordering:,
|
|
80
|
-
parent_context_id:, parent_reactor_class_name:, step_name:)
|
|
80
|
+
parent_context_id:, parent_reactor_class_name:, step_name:, fail_fast: nil)
|
|
81
81
|
RubyReactor::SidekiqWorkers::MapExecutionWorker.perform_async(
|
|
82
82
|
{
|
|
83
83
|
"map_id" => map_id,
|
|
@@ -86,10 +86,10 @@ module RubyReactor
|
|
|
86
86
|
"strict_ordering" => strict_ordering,
|
|
87
87
|
"parent_context_id" => parent_context_id,
|
|
88
88
|
"parent_reactor_class_name" => parent_reactor_class_name,
|
|
89
|
-
"step_name" => step_name
|
|
89
|
+
"step_name" => step_name,
|
|
90
|
+
"fail_fast" => fail_fast
|
|
90
91
|
}
|
|
91
92
|
)
|
|
92
93
|
end
|
|
93
|
-
# rubocop:enable Metrics/ParameterLists
|
|
94
94
|
end
|
|
95
95
|
end
|
|
@@ -68,6 +68,14 @@ module RubyReactor
|
|
|
68
68
|
{ "_type" => "Template::Value", "value" => serialize_value(value.instance_variable_get(:@value)) }
|
|
69
69
|
when RubyReactor::Template::Result
|
|
70
70
|
{ "_type" => "Template::Result", "step_name" => value.step_name.to_s, "path" => value.path }
|
|
71
|
+
when RubyReactor::Map::ResultEnumerator
|
|
72
|
+
{
|
|
73
|
+
"_type" => "Map::ResultEnumerator",
|
|
74
|
+
"map_id" => value.map_id,
|
|
75
|
+
"reactor_class_name" => value.reactor_class_name,
|
|
76
|
+
"strict_ordering" => value.strict_ordering,
|
|
77
|
+
"batch_size" => value.batch_size
|
|
78
|
+
}
|
|
71
79
|
when Hash
|
|
72
80
|
value.transform_keys(&:to_s).transform_values { |v| serialize_value(v) }
|
|
73
81
|
when Array
|
|
@@ -115,6 +123,13 @@ module RubyReactor
|
|
|
115
123
|
RubyReactor::Template::Value.new(deserialize_value(value["value"]))
|
|
116
124
|
when "Template::Result"
|
|
117
125
|
RubyReactor::Template::Result.new(value["step_name"], value["path"])
|
|
126
|
+
when "Map::ResultEnumerator"
|
|
127
|
+
RubyReactor::Map::ResultEnumerator.new(
|
|
128
|
+
value["map_id"],
|
|
129
|
+
value["reactor_class_name"],
|
|
130
|
+
strict_ordering: value["strict_ordering"],
|
|
131
|
+
batch_size: value["batch_size"]
|
|
132
|
+
)
|
|
118
133
|
else
|
|
119
134
|
value
|
|
120
135
|
end
|
|
@@ -32,8 +32,12 @@ module RubyReactor
|
|
|
32
32
|
@argument_mappings[mapped_input_name] = source
|
|
33
33
|
end
|
|
34
34
|
|
|
35
|
-
def source(enumerable)
|
|
36
|
-
@source_enumerable =
|
|
35
|
+
def source(enumerable = nil, &block)
|
|
36
|
+
@source_enumerable = if block
|
|
37
|
+
RubyReactor::Template::DynamicSource.new(@argument_mappings, &block)
|
|
38
|
+
else
|
|
39
|
+
enumerable
|
|
40
|
+
end
|
|
37
41
|
end
|
|
38
42
|
|
|
39
43
|
def async(async = true, batch_size: nil)
|
|
@@ -117,6 +117,7 @@ module RubyReactor
|
|
|
117
117
|
|
|
118
118
|
def store_failed_map_context(context)
|
|
119
119
|
return unless context.map_metadata && context.map_metadata[:map_id]
|
|
120
|
+
return unless context.map_metadata[:fail_fast]
|
|
120
121
|
|
|
121
122
|
storage = RubyReactor.configuration.storage_adapter
|
|
122
123
|
storage.store_map_failed_context_id(
|
|
@@ -39,7 +39,17 @@ module RubyReactor
|
|
|
39
39
|
|
|
40
40
|
# Serialize context and requeue the job
|
|
41
41
|
# Use root context if available to ensure we serialize the full tree
|
|
42
|
-
|
|
42
|
+
# BUT for map elements (which have map_metadata), we must serialize the element context itself
|
|
43
|
+
|
|
44
|
+
context_to_serialize = if @context.map_metadata
|
|
45
|
+
@context
|
|
46
|
+
else
|
|
47
|
+
@context.root_context || @context
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
puts "SERIALIZING CONTEXT: #{context_to_serialize.reactor_class.name}"
|
|
51
|
+
puts "INPUTS KEYS: #{context_to_serialize.inputs.keys}" if context_to_serialize.respond_to?(:inputs)
|
|
52
|
+
|
|
43
53
|
reactor_class_name = context_to_serialize.reactor_class.name
|
|
44
54
|
|
|
45
55
|
serialized_context = ContextSerializer.serialize(context_to_serialize)
|
|
@@ -7,56 +7,94 @@ module RubyReactor
|
|
|
7
7
|
|
|
8
8
|
def self.perform(arguments)
|
|
9
9
|
arguments = arguments.transform_keys(&:to_sym)
|
|
10
|
-
parent_context_id = arguments[:parent_context_id]
|
|
11
10
|
map_id = arguments[:map_id]
|
|
11
|
+
parent_context_id = arguments[:parent_context_id]
|
|
12
12
|
parent_reactor_class_name = arguments[:parent_reactor_class_name]
|
|
13
13
|
step_name = arguments[:step_name]
|
|
14
14
|
strict_ordering = arguments[:strict_ordering]
|
|
15
|
+
# timeout = arguments[:timeout]
|
|
15
16
|
|
|
16
17
|
storage = RubyReactor.configuration.storage_adapter
|
|
18
|
+
parent_context_data = storage.retrieve_context(parent_context_id, parent_reactor_class_name)
|
|
19
|
+
parent_context = RubyReactor::Context.deserialize_from_retry(parent_context_data)
|
|
20
|
+
|
|
21
|
+
# Check if all tasks are completed
|
|
22
|
+
metadata = storage.retrieve_map_metadata(map_id, parent_reactor_class_name)
|
|
23
|
+
total_count = metadata ? metadata["count"].to_i : 0
|
|
24
|
+
|
|
25
|
+
results_count = storage.count_map_results(map_id, parent_reactor_class_name)
|
|
26
|
+
|
|
27
|
+
# Not done yet, requeue or wait?
|
|
28
|
+
# Actually Collector currently assumes we only call it when we expect completion or check progress
|
|
29
|
+
# Since map_offset tracks dispatching progress and might exceed count due to batching reservation,
|
|
30
|
+
# we must strictly check against the total count of elements.
|
|
31
|
+
# Check for fail_fast failure FIRST
|
|
32
|
+
failed_context_id = storage.retrieve_map_failed_context_id(map_id, parent_reactor_class_name)
|
|
33
|
+
if failed_context_id
|
|
34
|
+
# Resolve the class of the mapped reactor to retrieve its context
|
|
35
|
+
reactor_class = resolve_reactor_class(metadata["reactor_class_info"])
|
|
17
36
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
37
|
+
failed_context_data = storage.retrieve_context(failed_context_id, reactor_class.name)
|
|
38
|
+
|
|
39
|
+
if failed_context_data
|
|
40
|
+
failed_context = RubyReactor::Context.deserialize_from_retry(failed_context_data)
|
|
41
|
+
|
|
42
|
+
# Resume parent execution (which marks step as failed)
|
|
43
|
+
resume_parent_execution(parent_context, step_name, RubyReactor::Failure(failed_context.failure_reason),
|
|
44
|
+
storage)
|
|
45
|
+
return
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
return if results_count < total_count
|
|
50
|
+
|
|
51
|
+
# Retrieve results lazily
|
|
52
|
+
results = RubyReactor::Map::ResultEnumerator.new(
|
|
53
|
+
map_id,
|
|
21
54
|
parent_reactor_class_name,
|
|
22
|
-
|
|
55
|
+
strict_ordering: strict_ordering
|
|
23
56
|
)
|
|
24
57
|
|
|
25
|
-
#
|
|
26
|
-
|
|
27
|
-
strict_ordering: strict_ordering)
|
|
58
|
+
# Apply collect block (or default collection)
|
|
59
|
+
step_config = parent_context.reactor_class.steps[step_name.to_sym]
|
|
28
60
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
61
|
+
begin
|
|
62
|
+
final_result = apply_collect_block(results, step_config)
|
|
63
|
+
|
|
64
|
+
if final_result.failure?
|
|
65
|
+
# Optionally log failure internally or just rely on context status update
|
|
34
66
|
end
|
|
67
|
+
rescue StandardError => e
|
|
68
|
+
final_result = RubyReactor::Failure(e)
|
|
35
69
|
end
|
|
36
70
|
|
|
37
|
-
#
|
|
38
|
-
|
|
39
|
-
|
|
71
|
+
# Resume parent execution
|
|
72
|
+
resume_parent_execution(parent_context, step_name, final_result, storage)
|
|
73
|
+
rescue StandardError => e
|
|
74
|
+
puts "COLLECTOR CRASH: #{e.message}"
|
|
75
|
+
puts e.backtrace
|
|
76
|
+
raise e
|
|
77
|
+
end
|
|
40
78
|
|
|
41
|
-
|
|
79
|
+
def self.apply_collect_block(results, step_config)
|
|
80
|
+
collect_block = step_config.arguments[:collect_block][:source].value if step_config.arguments[:collect_block]
|
|
42
81
|
# TODO: Check allow_partial_failure option
|
|
43
82
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
resume_parent_execution(parent_context, step_name, final_result, storage)
|
|
83
|
+
if collect_block
|
|
84
|
+
begin
|
|
85
|
+
# Pass Enumerator to collect block
|
|
86
|
+
collected = collect_block.call(results)
|
|
87
|
+
RubyReactor::Success(collected)
|
|
88
|
+
rescue StandardError => e
|
|
89
|
+
puts "COLLECTOR INNER EXCEPTION: #{e.message}"
|
|
90
|
+
puts e.backtrace
|
|
91
|
+
RubyReactor::Failure(e)
|
|
92
|
+
end
|
|
93
|
+
else
|
|
94
|
+
# Default behavior: Return Success(Enumerator).
|
|
95
|
+
# Logic for checking failures is deferred to the consumer of the enumerator.
|
|
96
|
+
RubyReactor::Success(results)
|
|
97
|
+
end
|
|
60
98
|
end
|
|
61
99
|
end
|
|
62
100
|
end
|