ruby_reactor 0.8.0 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/CHANGELOG.md +10 -3
- data/README.md +46 -32
- data/lib/ruby_reactor/adapters/active_job/router.rb +2 -2
- data/lib/ruby_reactor/adapters/sidekiq/router.rb +2 -2
- data/lib/ruby_reactor/dsl/map_builder.rb +31 -5
- data/lib/ruby_reactor/map/helpers.rb +9 -0
- data/lib/ruby_reactor/step/map_step.rb +9 -4
- data/lib/ruby_reactor/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 0d5c03b684aeac0b3d682581e9746e59219622f07752a5b5e30b5381582c09c9
|
|
4
|
+
data.tar.gz: bccbbdf3eec2187481fba0c5d724b1b2b61d96a32c93139dcf5fcd20e42ff491
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ab06166bc3462aa2cf8f4569015575feb7c37ae025022ae84b886dcc171040c2a89a88dbc69208c8b1ec84be6dd19c08c418915856a510506402e7ab72f26044
|
|
7
|
+
data.tar.gz: cfb0973443827d2d50572f834b1d9d2760464eab4290097b38c73465fcb7268f9e381d211b4a2fe1815f6bb410ac56ffa64c4f61ed17f0a765966392c50494a4
|
data/CHANGELOG.md
CHANGED
|
@@ -129,6 +129,13 @@
|
|
|
129
129
|
ordinary `nil` success, and `result(:step)` hands the reader the `Halt` — the same way it
|
|
130
130
|
already hands over a `Failure`.
|
|
131
131
|
|
|
132
|
+
## [0.8.1](https://github.com/arturictus/ruby_reactor/compare/v0.8.0...v0.8.1) (2026-09-22)
|
|
133
|
+
|
|
134
|
+
|
|
135
|
+
### Miscellaneous Chores
|
|
136
|
+
|
|
137
|
+
* Update documentation clarifying background and async concepts ([#55](https://github.com/arturictus/ruby_reactor/issues/55)) ([968e32b](https://github.com/arturictus/ruby_reactor/commit/968e32bbc695e45471d04b7630d8cd437ff41624))
|
|
138
|
+
|
|
132
139
|
## [0.8.0](https://github.com/arturictus/ruby_reactor/compare/v0.7.1...v0.8.0) (2026-09-21)
|
|
133
140
|
|
|
134
141
|
|
|
@@ -198,9 +205,9 @@
|
|
|
198
205
|
point (`:x` stays in the calling process); the two coincide in a linear chain
|
|
199
206
|
but pin different steps in a DAG.
|
|
200
207
|
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
208
|
+
Also affected: the map-internal element dispatch option was renamed from
|
|
209
|
+
`async true` to `fan_out`, so `map :items do async true, batch_size: 2 end`
|
|
210
|
+
now raises and should become `map :items do fan_out batch_size: 2 end`.
|
|
204
211
|
|
|
205
212
|
One behavior change falls out of "exactly one hand-off point per reactor":
|
|
206
213
|
resuming a reactor past its hand-off point now finishes in the resuming
|
data/README.md
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
|
|
6
6
|
# RubyReactor
|
|
7
7
|
|
|
8
|
-
A dynamic, dependency-resolving saga orchestrator for Ruby. Ruby Reactor implements the Saga pattern with compensation-based error handling and DAG-based execution planning. It leverages **Sidekiq or ActiveJob** for
|
|
8
|
+
A dynamic, dependency-resolving saga orchestrator for Ruby. Ruby Reactor implements the Saga pattern with compensation-based error handling and DAG-based execution planning. It leverages **Sidekiq or ActiveJob** for background execution and **Redis** for state persistence.
|
|
9
9
|
|
|
10
10
|

|
|
11
11
|
|
|
@@ -18,13 +18,14 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
|
|
|
18
18
|
## Features
|
|
19
19
|
|
|
20
20
|
- **DAG-based Execution**: Steps are executed based on their dependencies, allowing for parallel execution of independent steps.
|
|
21
|
-
- **
|
|
21
|
+
- **Background Execution**: Hand a whole reactor, or everything after a declared step, to a background job via Sidekiq or ActiveJob (so any ActiveJob-compatible queue — Resque, Solid Queue, GoodJob, etc. — works too).
|
|
22
|
+
- **Async Steps & Reactors**: `async_step` and `async_reactor` dispatch independent units of work while the reactor keeps running; steps that read their result wait for it.
|
|
22
23
|
- **Map & Parallel Execution**: Iterate over collections in parallel with the `map` step, distributing work across multiple workers.
|
|
23
24
|
- **Retries**: Configurable retry logic for failed steps, with exponential backoff.
|
|
24
25
|
- **Compensation**: Automatic rollback of completed steps when a failure occurs.
|
|
25
26
|
- **Interrupts**: Pause and resume workflows to wait for external events (webhooks, user approvals).
|
|
26
27
|
- **Input Validation**: Integrated with `dry-validation` for robust input checking.
|
|
27
|
-
- **Distributed Locks, Semaphores, Rate Limits, Periods & Ordered Locks**: Coordinate across processes with Redis-backed primitives — exclusive locks for at-most-one-runner, semaphores for capacity caps, fixed-window rate limits for external APIs (single or multi-window like "3/sec AND 100/min"), `with_period` to dedup reactors to once per calendar bucket, and `with_ordered_lock` for strict transaction ordering via a monotonically increasing nonce assigned at enqueue.
|
|
28
|
+
- **Distributed Locks, Semaphores, Rate Limits, Periods & Ordered Locks**: Coordinate across processes with Redis-backed primitives — exclusive locks for at-most-one-runner, semaphores for capacity caps, fixed-window rate limits for external APIs (single or multi-window like "3/sec AND 100/min"), `with_period` to dedup reactors to once per calendar bucket, and `with_ordered_lock` for strict transaction ordering via a monotonically increasing nonce assigned at enqueue. Background jobs snooze on contention with smart `retry_after` instead of consuming retry budget.
|
|
28
29
|
|
|
29
30
|
## Comparison
|
|
30
31
|
|
|
@@ -35,7 +36,7 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
|
|
|
35
36
|
| Interrupts (pause/resume)| Yes | No | No | Manual |
|
|
36
37
|
| Locks / sem / rate / per | Yes | No | No | Manual |
|
|
37
38
|
| Built-in web dashboard | Yes | No | No | No |
|
|
38
|
-
|
|
|
39
|
+
| Background (Sidekiq/AJ) | Yes | No | Limited | Yes |
|
|
39
40
|
| Durable crash recovery | Yes | No | No | Manual |
|
|
40
41
|
|
|
41
42
|
## Real-World Use Cases
|
|
@@ -57,9 +58,10 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
|
|
|
57
58
|
- [Rails Installation](#rails-installation)
|
|
58
59
|
- [Usage](#usage)
|
|
59
60
|
- [Basic Example: User Registration](#basic-example-user-registration)
|
|
60
|
-
- [
|
|
61
|
-
- [Full
|
|
61
|
+
- [Background Execution](#background-execution)
|
|
62
|
+
- [Full Background](#full-background)
|
|
62
63
|
- [Background Hand-off](#background-hand-off)
|
|
64
|
+
- [Async Steps & Reactors](#async-steps--reactors)
|
|
63
65
|
- [`async_step`](#async_step-one-step-dispatched-on-its-own)
|
|
64
66
|
- [`async_reactor`](#async_reactor-a-whole-nested-reactor-running-independently)
|
|
65
67
|
- [Durability & Recovery](#durability--recovery)
|
|
@@ -119,7 +121,7 @@ RubyReactor.configure do |config|
|
|
|
119
121
|
|
|
120
122
|
## === Background job backend (Sidekiq by default, or ActiveJob) ===
|
|
121
123
|
|
|
122
|
-
## Queue used by RubyReactor's
|
|
124
|
+
## Queue used by RubyReactor's background worker. Default: :default.
|
|
123
125
|
# config.queue_name = :default
|
|
124
126
|
|
|
125
127
|
## Retry count for infrastructure failures only (deserialization, Redis,
|
|
@@ -180,8 +182,8 @@ RubyReactor.configure do |config|
|
|
|
180
182
|
## Logger. Default: Logger.new($stdout).
|
|
181
183
|
# config.logger = Logger.new($stdout)
|
|
182
184
|
|
|
183
|
-
##
|
|
184
|
-
## built-in ActiveJob adapter (see "
|
|
185
|
+
## Background job router. Default: RubyReactor::Adapters::Sidekiq::Router. Swap in the
|
|
186
|
+
## built-in ActiveJob adapter (see "Background Execution" below), or point at any
|
|
185
187
|
## custom adapter — it only needs to respond to
|
|
186
188
|
## `perform_async(context_id, reactor_class_name, **)`.
|
|
187
189
|
# config.async_router = RubyReactor::Adapters::ActiveJob::Router
|
|
@@ -398,7 +400,7 @@ else
|
|
|
398
400
|
end
|
|
399
401
|
```
|
|
400
402
|
|
|
401
|
-
###
|
|
403
|
+
### Background Execution
|
|
402
404
|
|
|
403
405
|
Execute reactors in the background using Sidekiq or ActiveJob. The backend is
|
|
404
406
|
chosen via `config.async_router` (defaults to the Sidekiq adapter); to run on
|
|
@@ -410,13 +412,14 @@ RubyReactor.configure do |config|
|
|
|
410
412
|
end
|
|
411
413
|
```
|
|
412
414
|
|
|
413
|
-
That's the only switch — everything below (
|
|
414
|
-
|
|
415
|
+
That's the only switch — everything below (background execution, `async_step`
|
|
416
|
+
/ `async_reactor`, durability, retries, snoozing) works identically on either
|
|
417
|
+
backend.
|
|
415
418
|
|
|
416
|
-
#### Full
|
|
419
|
+
#### Full Background
|
|
417
420
|
|
|
418
421
|
```ruby
|
|
419
|
-
class
|
|
422
|
+
class BackgroundReactor < RubyReactor::Reactor
|
|
420
423
|
background all: true # Entire reactor runs in background
|
|
421
424
|
|
|
422
425
|
step :long_running_task do
|
|
@@ -425,7 +428,7 @@ class AsyncReactor < RubyReactor::Reactor
|
|
|
425
428
|
end
|
|
426
429
|
|
|
427
430
|
# Returns immediately with DispatchResult
|
|
428
|
-
result =
|
|
431
|
+
result = BackgroundReactor.run(params)
|
|
429
432
|
```
|
|
430
433
|
|
|
431
434
|
#### Background Hand-off
|
|
@@ -493,6 +496,13 @@ changes where code runs, not the saga contract.
|
|
|
493
496
|
> to `async_step`/`async_reactor`, whose names mean something else. It now raises
|
|
494
497
|
> at class-definition time. The exact replacement is `background all: true`.
|
|
495
498
|
|
|
499
|
+
### Async Steps & Reactors
|
|
500
|
+
|
|
501
|
+
`async_step` and `async_reactor` dispatch an **independent** unit of work to its
|
|
502
|
+
own job while the reactor keeps running. Unlike `background`, they do not move
|
|
503
|
+
the reactor itself — they are separate units with their own outcome and
|
|
504
|
+
compensation.
|
|
505
|
+
|
|
496
506
|
#### `async_step`: one step, dispatched on its own
|
|
497
507
|
|
|
498
508
|
Where `background` relocates the *rest* of a reactor, `async_step` dispatches one
|
|
@@ -569,12 +579,12 @@ parent back.
|
|
|
569
579
|
|
|
570
580
|
Lock ownership is never shared across the async boundary: a child declaring a key
|
|
571
581
|
the parent holds fails at dispatch with an explanatory error rather than
|
|
572
|
-
deadlocking. See [Async
|
|
582
|
+
deadlocking. See [Background & Async Execution](documentation/background_and_async.md) for the full
|
|
573
583
|
rules.
|
|
574
584
|
|
|
575
585
|
### Durability & Recovery
|
|
576
586
|
|
|
577
|
-
|
|
587
|
+
Background reactors are durable: state lives in Redis, not in the job payload. Before
|
|
578
588
|
any background job is enqueued the root context is persisted, and after every
|
|
579
589
|
completed step a checkpoint advances the stored blob — so a crash re-runs at most
|
|
580
590
|
one step, never the whole reactor. Each running reactor also holds a short
|
|
@@ -740,7 +750,7 @@ class ChargeReactor < RubyReactor::Reactor
|
|
|
740
750
|
input :account_id
|
|
741
751
|
|
|
742
752
|
# Respect upstream Stripe rate limits: 3/sec and 100/min.
|
|
743
|
-
#
|
|
753
|
+
# Background workers snooze for exactly retry_after seconds instead of
|
|
744
754
|
# consuming the backend's retry budget.
|
|
745
755
|
with_rate_limit(
|
|
746
756
|
limits: { second: 3, minute: 100 }
|
|
@@ -753,7 +763,7 @@ class ChargeReactor < RubyReactor::Reactor
|
|
|
753
763
|
end
|
|
754
764
|
|
|
755
765
|
class OrderedTransactionReactor < RubyReactor::Reactor
|
|
756
|
-
|
|
766
|
+
background all: true
|
|
757
767
|
input :account_id
|
|
758
768
|
input :transaction
|
|
759
769
|
|
|
@@ -794,12 +804,12 @@ class ChargeReactor < RubyReactor::Reactor
|
|
|
794
804
|
end
|
|
795
805
|
```
|
|
796
806
|
|
|
797
|
-
Referencing an unregistered name raises `RubyReactor::RateLimitRegistry::UnknownLimitError`. Named limits hit the same enforcement path as inline ones, so
|
|
807
|
+
Referencing an unregistered name raises `RubyReactor::RateLimitRegistry::UnknownLimitError`. Named limits hit the same enforcement path as inline ones, so background snooze behavior is identical.
|
|
798
808
|
|
|
799
809
|
On contention:
|
|
800
810
|
|
|
801
811
|
- **Inline** (`Reactor.run`) raises `RubyReactor::Lock::AcquisitionError` / `RubyReactor::Semaphore::AcquisitionError` / `RubyReactor::RateLimit::ExceededError` / `RubyReactor::OrderedLock::WaitError`.
|
|
802
|
-
- **
|
|
812
|
+
- **Background** (Sidekiq or ActiveJob) snoozes the job via `perform_in(delay, ...)`. For rate limits the delay uses the error's `retry_after_seconds` hint (precise wakeup — the bucket roll time is known exactly); for locks, semaphores, and ordered-lock waits it's `lock_snooze_base_delay + jitter` (a short re-poll, since a held lock or a live blocker nonce typically clears in milliseconds). Snoozes do not count against the backend's retry budget. After `lock_snooze_max_attempts` snoozes the context is marked failed (ordered-lock waits bypass the cap — see the ordered-lock docs).
|
|
803
813
|
|
|
804
814
|
On dedup hits (period gate already marked), the reactor returns a `RubyReactor::Halt` result instead — no steps run, no exception:
|
|
805
815
|
|
|
@@ -847,8 +857,8 @@ class DataProcessingReactor < RubyReactor::Reactor
|
|
|
847
857
|
source input(:items)
|
|
848
858
|
argument :item, element(:process_items)
|
|
849
859
|
|
|
850
|
-
#
|
|
851
|
-
|
|
860
|
+
# Run elements as background jobs, in batches
|
|
861
|
+
fan_out batch_size: 50
|
|
852
862
|
|
|
853
863
|
step :transform do
|
|
854
864
|
argument :item, input(:item)
|
|
@@ -860,9 +870,13 @@ class DataProcessingReactor < RubyReactor::Reactor
|
|
|
860
870
|
end
|
|
861
871
|
```
|
|
862
872
|
|
|
863
|
-
|
|
873
|
+
A `fan_out` map is a **hand-off point**: the reactor stops at the map (the caller gets a `DispatchResult`), every element runs as its own background job, and once all outcomes are collected the reactor resumes in a worker with the steps after the map. It fans out the same way when the reactor is already running in a worker (e.g. `background all: true`).
|
|
874
|
+
|
|
875
|
+
By using `fan_out` with `batch_size`, the system applies **Back Pressure** to efficiently manage resources. [Read more about Back Pressure & Resource Management](documentation/data_pipelines.md#back-pressure--resource-management).
|
|
876
|
+
|
|
877
|
+
`batch_size` is optional: with `fan_out` alone, RubyReactor fans out one worker per element (defaulting the batch size to the full source size) and aggregates the outcomes into a `ResultEnumerator` — convenient for small collections, but with no back pressure. See [`fan_out` Without `batch_size`](documentation/data_pipelines.md#fan_out-without-batch_size).
|
|
864
878
|
|
|
865
|
-
|
|
879
|
+
> **Breaking change:** `async true` inside a `map` block has been **removed** — it read like `async_step`/`async_reactor`, which dispatch independent units the reactor does not stop for. It now raises at class-definition time. The exact replacement is `fan_out` (`fan_out batch_size: N`).
|
|
866
880
|
|
|
867
881
|
#### Map with Dynamic Source (ActiveRecord)
|
|
868
882
|
|
|
@@ -878,7 +892,7 @@ map :archive_old_users do
|
|
|
878
892
|
end
|
|
879
893
|
|
|
880
894
|
argument :user, element(:archive_old_users)
|
|
881
|
-
|
|
895
|
+
fan_out batch_size: 100
|
|
882
896
|
|
|
883
897
|
step :archive do
|
|
884
898
|
argument :user, input(:user)
|
|
@@ -1426,8 +1440,8 @@ Learn about the fundamental building blocks of RubyReactor: Reactors, Steps, Con
|
|
|
1426
1440
|
### [DAG (Directed Acyclic Graph)](documentation/DAG.md)
|
|
1427
1441
|
Deep dive into how RubyReactor manages dependencies. This guide explains how the Directed Acyclic Graph is constructed to ensure steps execute in the correct topological order, enabling automatic parallelization of independent steps.
|
|
1428
1442
|
|
|
1429
|
-
### [Async
|
|
1430
|
-
Explore the ways to move work off the calling process:
|
|
1443
|
+
### [Background & Async Execution](documentation/background_and_async.md)
|
|
1444
|
+
Explore the ways to move work off the calling process: background execution (`background all: true` and the `background after:`/`before:` hand-off) and async units (`async_step`, `async_reactor`). Learn how RubyReactor leverages Sidekiq or ActiveJob for background processing, non-blocking execution, and scalable worker management.
|
|
1431
1445
|
|
|
1432
1446
|
### [Composition](documentation/composition.md)
|
|
1433
1447
|
Discover how to build complex, modular workflows by composing reactors within other reactors. This guide covers inline composition, class-based composition, and how to manage dependencies between composed workflows.
|
|
@@ -1436,7 +1450,7 @@ Discover how to build complex, modular workflows by composing reactors within ot
|
|
|
1436
1450
|
Master the `map` feature for processing collections. Learn about parallel execution, batch processing for large datasets, and error handling strategies like fail-fast vs. partial result collection.
|
|
1437
1451
|
|
|
1438
1452
|
### [Retry Configuration](documentation/retry_configuration.md)
|
|
1439
|
-
Configure robust retry policies for your steps. This guide details the available backoff strategies (exponential, linear, fixed), how to configure retries at the reactor or step level, and how
|
|
1453
|
+
Configure robust retry policies for your steps. This guide details the available backoff strategies (exponential, linear, fixed), how to configure retries at the reactor or step level, and how background retries work without blocking workers.
|
|
1440
1454
|
|
|
1441
1455
|
### [Interrupts](documentation/interrupts.md)
|
|
1442
1456
|
Learn how to pause and resume reactors to handle long-running processes, manual approvals, and asynchronous callbacks. Patterns for correlation IDs, timeouts, and payload validation.
|
|
@@ -1446,11 +1460,11 @@ Comprehensive guide to testing reactors with RubyReactor's testing utilities. Le
|
|
|
1446
1460
|
|
|
1447
1461
|
### [Locks, Semaphores, Rate Limits, Periods & Ordered Locks](documentation/locks_and_semaphores.md)
|
|
1448
1462
|
|
|
1449
|
-
Coordinate access to shared resources across processes with Redis-backed primitives: exclusive locks (`with_lock`), concurrency-limiting semaphores (`with_semaphore`), fixed-window rate limits with multi-window quotas (`with_rate_limit`), calendar-bucketed dedup (`with_period`, returning `Halt` results), and strict sequential ordering via a monotonically increasing nonce assigned at enqueue (`with_ordered_lock`). Covers re-entrancy across composed reactors, TTL auto-extend, inline-vs-
|
|
1463
|
+
Coordinate access to shared resources across processes with Redis-backed primitives: exclusive locks (`with_lock`), concurrency-limiting semaphores (`with_semaphore`), fixed-window rate limits with multi-window quotas (`with_rate_limit`), calendar-bucketed dedup (`with_period`, returning `Halt` results), and strict sequential ordering via a monotonically increasing nonce assigned at enqueue (`with_ordered_lock`). Covers re-entrancy across composed reactors, TTL auto-extend, inline-vs-background contention behavior, smart `retry_after` snoozes for rate limits, snooze tuning, the token-based semaphore safety model, once-per-day/month/year scheduling patterns, ordered-lock counter reset on drain, poison-pill timeouts, and deadlock-safe composition rules.
|
|
1450
1464
|
|
|
1451
1465
|
### [Middlewares & OpenTelemetry](documentation/middlewares.md)
|
|
1452
1466
|
|
|
1453
|
-
Hook into the execution lifecycle with observer middlewares. Covers the full set of lifecycle events (reactor, step, retry, compensation/undo,
|
|
1467
|
+
Hook into the execution lifecycle with observer middlewares. Covers the full set of lifecycle events (reactor, step, retry, compensation/undo, background hand-off, locks/semaphores), writing and registering custom middlewares (global and per-reactor), and the built-in `RubyReactor::OpenTelemetry` tracing middleware — span structure, input/argument redaction, distributed trace propagation across background/retry boundaries, and custom exporters.
|
|
1454
1468
|
|
|
1455
1469
|
### Examples
|
|
1456
1470
|
- [Order Processing](documentation/examples/order_processing.md) - Complete order processing workflow example
|
|
@@ -1470,7 +1484,7 @@ Hook into the execution lifecycle with observer middlewares. Covers the full set
|
|
|
1470
1484
|
- [ ] Multiple storage adapters
|
|
1471
1485
|
- [X] Redis
|
|
1472
1486
|
- [ ] ActiveRecord
|
|
1473
|
-
- [X] Multiple
|
|
1487
|
+
- [X] Multiple background job adapters
|
|
1474
1488
|
- [X] Sidekiq
|
|
1475
1489
|
- [X] ActiveJob
|
|
1476
1490
|
- [X] OpenTelemetry support
|
|
@@ -10,13 +10,13 @@ module RubyReactor
|
|
|
10
10
|
def self.perform_async(context_id, reactor_class_name = nil, intermediate_results: {})
|
|
11
11
|
job_id = RubyReactor::Adapters::ActiveJob::Worker.perform_async(context_id, reactor_class_name)
|
|
12
12
|
RubyReactor::DispatchResult.new(job_id: job_id, intermediate_results: intermediate_results,
|
|
13
|
-
|
|
13
|
+
execution_id: context_id)
|
|
14
14
|
end
|
|
15
15
|
|
|
16
16
|
def self.perform_in(delay, context_id, reactor_class_name = nil, intermediate_results: {})
|
|
17
17
|
job_id = RubyReactor::Adapters::ActiveJob::Worker.perform_in(delay, context_id, reactor_class_name)
|
|
18
18
|
RubyReactor::DispatchResult.new(job_id: job_id, intermediate_results: intermediate_results,
|
|
19
|
-
|
|
19
|
+
execution_id: context_id)
|
|
20
20
|
end
|
|
21
21
|
|
|
22
22
|
# One `async_step`'s work, dispatched as its own independent unit. The
|
|
@@ -10,13 +10,13 @@ module RubyReactor
|
|
|
10
10
|
def self.perform_async(context_id, reactor_class_name = nil, intermediate_results: {})
|
|
11
11
|
job_id = RubyReactor::Adapters::Sidekiq::Worker.perform_async(context_id, reactor_class_name)
|
|
12
12
|
RubyReactor::DispatchResult.new(job_id: job_id, intermediate_results: intermediate_results,
|
|
13
|
-
|
|
13
|
+
execution_id: context_id)
|
|
14
14
|
end
|
|
15
15
|
|
|
16
16
|
def self.perform_in(delay, context_id, reactor_class_name = nil, intermediate_results: {})
|
|
17
17
|
job_id = RubyReactor::Adapters::Sidekiq::Worker.perform_in(delay, context_id, reactor_class_name)
|
|
18
18
|
RubyReactor::DispatchResult.new(job_id: job_id, intermediate_results: intermediate_results,
|
|
19
|
-
|
|
19
|
+
execution_id: context_id)
|
|
20
20
|
end
|
|
21
21
|
|
|
22
22
|
# One `async_step`'s work, dispatched as its own independent unit. The
|
|
@@ -20,7 +20,7 @@ module RubyReactor
|
|
|
20
20
|
end
|
|
21
21
|
@reactor = reactor
|
|
22
22
|
@argument_mappings = {}
|
|
23
|
-
@
|
|
23
|
+
@fan_out = false
|
|
24
24
|
@strict_ordering = true
|
|
25
25
|
@batch_size = nil
|
|
26
26
|
@source_enumerable = nil
|
|
@@ -40,9 +40,28 @@ module RubyReactor
|
|
|
40
40
|
end
|
|
41
41
|
end
|
|
42
42
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
43
|
+
# Run every element as its own background job. `batch_size` caps how many
|
|
44
|
+
# element jobs are enqueued at a time (back pressure); without it the whole
|
|
45
|
+
# source fans out at once. A fan-out map is a hand-off point: the reactor
|
|
46
|
+
# stops at the map and resumes in a worker once the collector has every
|
|
47
|
+
# element's outcome.
|
|
48
|
+
def fan_out(enabled = true, batch_size: nil)
|
|
49
|
+
@fan_out = enabled
|
|
50
|
+
self.batch_size(batch_size) unless batch_size.nil?
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# `async` on a map named element fan-out with the word that now means
|
|
54
|
+
# `async_step` / `async_reactor` — independent units the reactor does not
|
|
55
|
+
# stop for. A fan-out map does stop the reactor, so it gets its own word.
|
|
56
|
+
def async(*)
|
|
57
|
+
raise RubyReactor::Error::DeprecatedDslError.new(
|
|
58
|
+
"`async` inside a `map` block has been removed: it read as `async_step`/`async_reactor`, which " \
|
|
59
|
+
"dispatch independent units, whereas a fan-out map hands the reactor off until every element " \
|
|
60
|
+
"finishes. Use `fan_out` instead (`fan_out batch_size: N` for back pressure) — note this also " \
|
|
61
|
+
"changes worker dispatch: each element now runs in its own background worker rather than being " \
|
|
62
|
+
"suppressed inline by `inline_async_execution` as the old `async true` was.",
|
|
63
|
+
step: @name
|
|
64
|
+
)
|
|
46
65
|
end
|
|
47
66
|
|
|
48
67
|
def strict_ordering(enabled = true)
|
|
@@ -50,6 +69,13 @@ module RubyReactor
|
|
|
50
69
|
end
|
|
51
70
|
|
|
52
71
|
def batch_size(size)
|
|
72
|
+
unless size.is_a?(Integer) && size.positive?
|
|
73
|
+
raise RubyReactor::Error::ValidationError.new(
|
|
74
|
+
"`batch_size` must be a positive Integer, got #{size.inspect}",
|
|
75
|
+
step: @name
|
|
76
|
+
)
|
|
77
|
+
end
|
|
78
|
+
|
|
53
79
|
@batch_size = size
|
|
54
80
|
end
|
|
55
81
|
|
|
@@ -98,7 +124,7 @@ module RubyReactor
|
|
|
98
124
|
batch_size: { source: RubyReactor::Template::Value.new(@batch_size) },
|
|
99
125
|
collect_block: { source: RubyReactor::Template::Value.new(@collect_block) },
|
|
100
126
|
fail_fast: { source: RubyReactor::Template::Value.new(@fail_fast) },
|
|
101
|
-
|
|
127
|
+
fan_out: { source: RubyReactor::Template::Value.new(@fan_out) }
|
|
102
128
|
},
|
|
103
129
|
run_block: nil,
|
|
104
130
|
compensate_block: nil,
|
|
@@ -107,6 +107,15 @@ module RubyReactor
|
|
|
107
107
|
)
|
|
108
108
|
|
|
109
109
|
parent_context.current_step = nil
|
|
110
|
+
|
|
111
|
+
# Mark the context as a worker before resuming: resume_execution runs
|
|
112
|
+
# the collector's own worker (this call), not the reactor's normal
|
|
113
|
+
# caller. Without this, StepExecutor#handoff_at? (gated on
|
|
114
|
+
# `inline_async_execution`) re-evaluates any later `background`
|
|
115
|
+
# before:/after: cut point and enqueues a second hand-off, or runs an
|
|
116
|
+
# `after:` target here in the collector instead of the original
|
|
117
|
+
# dispatching worker.
|
|
118
|
+
parent_context.inline_async_execution = true
|
|
110
119
|
executor.resume_execution
|
|
111
120
|
end
|
|
112
121
|
|
|
@@ -9,7 +9,7 @@ module RubyReactor
|
|
|
9
9
|
# Initialize map state in context if not present
|
|
10
10
|
context.map_operations ||= {}
|
|
11
11
|
|
|
12
|
-
if
|
|
12
|
+
if fan_out?
|
|
13
13
|
run_async(context.current_step)
|
|
14
14
|
else
|
|
15
15
|
run_inline
|
|
@@ -69,10 +69,15 @@ module RubyReactor
|
|
|
69
69
|
|
|
70
70
|
private
|
|
71
71
|
|
|
72
|
-
|
|
73
|
-
|
|
72
|
+
# Fans out anywhere except inside a map element: an element's result and
|
|
73
|
+
# the map's completion counter are tracked by its own ElementExecutor job,
|
|
74
|
+
# so a nested hand-off there would escape that tracking. A reactor worker
|
|
75
|
+
# (`background`, collector resume, `async_reactor` child) fans out as the
|
|
76
|
+
# caller would — the collector resumes the reactor in a worker either way.
|
|
77
|
+
def fan_out?
|
|
78
|
+
return false if context.map_metadata || context.root_context&.map_metadata
|
|
74
79
|
|
|
75
|
-
inputs[:
|
|
80
|
+
inputs[:fan_out]
|
|
76
81
|
end
|
|
77
82
|
|
|
78
83
|
def run_inline
|
data/lib/ruby_reactor/version.rb
CHANGED