ruby_reactor 0.5.1 → 0.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/CHANGELOG.md +14 -0
- data/README.md +179 -26
- data/lib/ruby_reactor/configuration.rb +66 -2
- data/lib/ruby_reactor/context_serializer.rb +9 -4
- data/lib/ruby_reactor/dsl/compose_builder.rb +20 -0
- data/lib/ruby_reactor/dsl/lockable.rb +41 -1
- data/lib/ruby_reactor/executor/ordered_lock_support.rb +307 -0
- data/lib/ruby_reactor/executor/retry_manager.rb +7 -2
- data/lib/ruby_reactor/executor/step_executor.rb +25 -5
- data/lib/ruby_reactor/executor.rb +166 -52
- data/lib/ruby_reactor/lock.rb +13 -0
- data/lib/ruby_reactor/map/collector.rb +41 -0
- data/lib/ruby_reactor/map/dispatcher.rb +42 -0
- data/lib/ruby_reactor/map/element_executor.rb +39 -0
- data/lib/ruby_reactor/map/helpers.rb +10 -3
- data/lib/ruby_reactor/map/sweeper.rb +110 -0
- data/lib/ruby_reactor/ordered_lock.rb +158 -0
- data/lib/ruby_reactor/reactor.rb +48 -5
- data/lib/ruby_reactor/rspec/helpers.rb +6 -0
- data/lib/ruby_reactor/rspec/matchers.rb +66 -0
- data/lib/ruby_reactor/rspec/sidekiq_helpers.rb +70 -0
- data/lib/ruby_reactor/rspec/storage_reset.rb +23 -0
- data/lib/ruby_reactor/rspec/test_subject.rb +14 -28
- data/lib/ruby_reactor/rspec.rb +37 -0
- data/lib/ruby_reactor/sidekiq_adapter.rb +9 -8
- data/lib/ruby_reactor/sidekiq_workers/sweeper_worker.rb +73 -0
- data/lib/ruby_reactor/sidekiq_workers/worker.rb +82 -36
- data/lib/ruby_reactor/step/map_step.rb +18 -2
- data/lib/ruby_reactor/storage/redis_adapter.rb +84 -60
- data/lib/ruby_reactor/storage/redis_locking.rb +8 -0
- data/lib/ruby_reactor/storage/redis_ordered_locking.rb +382 -0
- data/lib/ruby_reactor/sweeper.rb +58 -0
- data/lib/ruby_reactor/version.rb +1 -1
- data/lib/ruby_reactor.rb +43 -0
- metadata +9 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 778bc5305c6d1f20833819afd9ccd46f5a5b4c2c135d8e63344d6530f9a733f1
|
|
4
|
+
data.tar.gz: 32e769816eba846f419e3f31e8290b94e8ff04fe6ea71fef125bb128b3085b82
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 592db1d0ef94153a4ea028aa3cdeb59f7e4c73929ebec5afd5a9795a93d27a0cfb0c4bd3e135ccf1b14c0329123be6cf0905ad4a141018993832d21212754db3
|
|
7
|
+
data.tar.gz: 2fd90e47af8e26cf2d58468a1f0629fd1dc04890df0a6e1704e8d6cba5c65b1e050f10f56628c27c7c90b0545d081e33169e3bfd62f416affd761b62edeb24a0
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.5.3](https://github.com/arturictus/ruby_reactor/compare/v0.5.2...v0.5.3) (2026-06-17)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Features
|
|
7
|
+
|
|
8
|
+
* Durability & Recovery ([#39](https://github.com/arturictus/ruby_reactor/issues/39)) ([103e583](https://github.com/arturictus/ruby_reactor/commit/103e5835b413eec2302fa63f3e998d487cfd9eaf))
|
|
9
|
+
|
|
10
|
+
## [0.5.2](https://github.com/arturictus/ruby_reactor/compare/v0.5.1...v0.5.2) (2026-06-14)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### Features
|
|
14
|
+
|
|
15
|
+
* Nonce lock ([#26](https://github.com/arturictus/ruby_reactor/issues/26)) ([5925cac](https://github.com/arturictus/ruby_reactor/commit/5925cac7af93f59be6c0a8a98ab020f96080f60b))
|
|
16
|
+
|
|
3
17
|
## [0.5.1](https://github.com/arturictus/ruby_reactor/compare/v0.5.0...v0.5.1) (2026-06-14)
|
|
4
18
|
|
|
5
19
|
|
data/README.md
CHANGED
|
@@ -24,7 +24,7 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
|
|
|
24
24
|
- **Compensation**: Automatic rollback of completed steps when a failure occurs.
|
|
25
25
|
- **Interrupts**: Pause and resume workflows to wait for external events (webhooks, user approvals).
|
|
26
26
|
- **Input Validation**: Integrated with `dry-validation` for robust input checking.
|
|
27
|
-
- **Distributed Locks, Semaphores, Rate Limits &
|
|
27
|
+
- **Distributed Locks, Semaphores, Rate Limits, Periods & Ordered Locks**: Coordinate across processes with Redis-backed primitives — exclusive locks for at-most-one-runner, semaphores for capacity caps, fixed-window rate limits for external APIs (single or multi-window like "3/sec AND 100/min"), `with_period` to dedup reactors to once per calendar bucket, and `with_ordered_lock` for strict transaction ordering via a monotonically increasing nonce assigned at enqueue. Async jobs snooze on contention with smart `retry_after` instead of consuming retry budget.
|
|
28
28
|
|
|
29
29
|
## Comparison
|
|
30
30
|
|
|
@@ -36,6 +36,7 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
|
|
|
36
36
|
| Locks / sem / rate / per | Yes | No | No | Manual |
|
|
37
37
|
| Built-in web dashboard | Yes | No | No | No |
|
|
38
38
|
| Async with Sidekiq | Yes | No | Limited | Yes |
|
|
39
|
+
| Durable crash recovery | Yes | No | No | Manual |
|
|
39
40
|
|
|
40
41
|
## Real-World Use Cases
|
|
41
42
|
|
|
@@ -44,6 +45,7 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
|
|
|
44
45
|
- **Subscription Billing**: Coordinate Stripe charges, invoice email generation, and internal entitlement updates. Use interrupts to pause the workflow when 3rd-party APIs are required to continue the workflow or when specific customer approval is needed.
|
|
45
46
|
|
|
46
47
|
## Table of Contents
|
|
48
|
+
|
|
47
49
|
- [Features](#features)
|
|
48
50
|
- [Comparison](#comparison)
|
|
49
51
|
- [Real-World Use Cases](#real-world-use-cases)
|
|
@@ -57,8 +59,9 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
|
|
|
57
59
|
- [Async Execution](#async-execution)
|
|
58
60
|
- [Full Reactor Async](#full-reactor-async)
|
|
59
61
|
- [Step-Level Async](#step-level-async)
|
|
62
|
+
- [Durability & Recovery](#durability--recovery)
|
|
60
63
|
- [Interrupts (Pause & Resume)](#interrupts-pause--resume)
|
|
61
|
-
- [Locks &
|
|
64
|
+
- [Locks, Semaphores & Ordered Locks](#locks-semaphores--ordered-locks)
|
|
62
65
|
- [Map & Parallel Execution](#map--parallel-execution)
|
|
63
66
|
- [Map with Dynamic Source (ActiveRecord)](#map-with-dynamic-source-activerecord)
|
|
64
67
|
- [Input Validation](#input-validation)
|
|
@@ -90,35 +93,96 @@ Or install it yourself as:
|
|
|
90
93
|
|
|
91
94
|
## Configuration
|
|
92
95
|
|
|
93
|
-
|
|
96
|
+
Every setting is **optional** — RubyReactor ships with the defaults shown. Drop
|
|
97
|
+
this into an initializer (e.g. `config/initializers/ruby_reactor.rb`); pasted as-is
|
|
98
|
+
it changes nothing, so it doubles as a reference of every knob.
|
|
99
|
+
|
|
100
|
+
> **Reading the block:** lines starting with `##` are documentation. Lines starting
|
|
101
|
+
> with a single `#` (a `config.…` call) are real settings commented at their
|
|
102
|
+
> default — uncomment one to enable it.
|
|
94
103
|
|
|
95
104
|
```ruby
|
|
96
105
|
RubyReactor.configure do |config|
|
|
97
|
-
|
|
98
|
-
|
|
106
|
+
## === Storage (Redis) ===
|
|
107
|
+
|
|
108
|
+
## Storage adapter. Default: :redis (the only adapter shipped today).
|
|
109
|
+
# config.storage.adapter = :redis
|
|
110
|
+
|
|
111
|
+
## Redis URL. Default: "redis://localhost:6379/0".
|
|
99
112
|
config.storage.redis_url = ENV.fetch("REDIS_URL", "redis://localhost:6379/0")
|
|
100
|
-
config.storage.redis_options = { timeout: 1 }
|
|
101
113
|
|
|
102
|
-
|
|
103
|
-
config.
|
|
104
|
-
config.sidekiq_retry_count = 3
|
|
114
|
+
## Extra options passed to Redis.new. Default: {}.
|
|
115
|
+
# config.storage.redis_options = { timeout: 1 }
|
|
105
116
|
|
|
106
|
-
|
|
107
|
-
# cannot acquire a lock or semaphore, it re-enqueues itself with this delay
|
|
108
|
-
# (plus jitter) up to `lock_snooze_max_attempts` times before giving up.
|
|
109
|
-
config.lock_snooze_base_delay = 5
|
|
110
|
-
config.lock_snooze_jitter = 5
|
|
111
|
-
config.lock_snooze_max_attempts = 20
|
|
117
|
+
## === Sidekiq ===
|
|
112
118
|
|
|
113
|
-
|
|
114
|
-
#
|
|
115
|
-
|
|
119
|
+
## Sidekiq queue used by RubyReactor's async worker. Default: :default.
|
|
120
|
+
# config.sidekiq_queue = :default
|
|
121
|
+
|
|
122
|
+
## Sidekiq retry count for infrastructure failures only (deserialization,
|
|
123
|
+
## Redis, network). Step retries are managed separately. Default: 3.
|
|
124
|
+
# config.sidekiq_retry_count = 3
|
|
125
|
+
|
|
126
|
+
## === Contention snooze (locks / semaphores / rate limits / ordered locks) ===
|
|
127
|
+
|
|
128
|
+
## When a Sidekiq worker cannot acquire a primitive it re-enqueues itself with
|
|
129
|
+
## `lock_snooze_base_delay + rand(0..lock_snooze_jitter)` seconds (rate-limit
|
|
130
|
+
## uses a precise `retry_after_seconds` hint from the error; ordered-lock waits
|
|
131
|
+
## re-poll at the base delay so a successor catches its blocker finishing fast),
|
|
132
|
+
## up to `lock_snooze_max_attempts` times before marking the context :failed.
|
|
133
|
+
## Set max_attempts to :infinity to never give up.
|
|
134
|
+
# config.lock_snooze_base_delay = 5
|
|
135
|
+
# config.lock_snooze_jitter = 5
|
|
136
|
+
# config.lock_snooze_max_attempts = 20
|
|
137
|
+
|
|
138
|
+
## === Durability & crash recovery (see "Durability & Recovery" below) ===
|
|
139
|
+
|
|
140
|
+
## Retention TTL (seconds) for stored reactor/map state. Must exceed your
|
|
141
|
+
## worst-case snooze/retry window; re-stamped on every write. Default: 86_400.
|
|
142
|
+
# config.context_ttl = 86_400
|
|
143
|
+
|
|
144
|
+
## TTL (seconds) for the per-context liveness lock. A live worker auto-extends
|
|
145
|
+
## it; its absence is the sweeper's "worker died" signal. Must exceed the
|
|
146
|
+
## longest a single step can run without yielding the GIL. Default: 60.
|
|
147
|
+
# config.context_lock_ttl = 60
|
|
148
|
+
|
|
149
|
+
## Minimum seconds between per-step checkpoints within one run. 0 = checkpoint
|
|
150
|
+
## after every step (strongest guarantee). Raise to coalesce mid-run writes for
|
|
151
|
+
## long reactors — only safe when steps are idempotent. Default: 0.
|
|
152
|
+
# config.checkpoint_min_interval = 0
|
|
153
|
+
|
|
154
|
+
## Recovery sweeper (the chain is kicked once by `RubyReactor.start_sweeper!`).
|
|
155
|
+
# config.sweeper_enabled = true # run recovery by default
|
|
156
|
+
# config.sweeper_interval = 30 # seconds between sweeps = recovery-latency bound
|
|
157
|
+
# config.sweeper_limit = 1000 # max contexts/maps inspected per sweep
|
|
116
158
|
|
|
117
|
-
|
|
118
|
-
|
|
159
|
+
## === Misc ===
|
|
160
|
+
|
|
161
|
+
## Logger. Default: Logger.new($stdout).
|
|
162
|
+
# config.logger = Logger.new($stdout)
|
|
163
|
+
|
|
164
|
+
## Async router. Default: RubyReactor::SidekiqAdapter. Swap for a custom adapter
|
|
165
|
+
## if you don't use Sidekiq — it only needs to respond to
|
|
166
|
+
## `perform_async(context_id, reactor_class_name, **)`.
|
|
167
|
+
# config.async_router = MyCustomAdapter
|
|
168
|
+
|
|
169
|
+
## === Examples (no default — set these to use the feature) ===
|
|
170
|
+
|
|
171
|
+
## Named rate limits shared across reactors. Reference with `with_rate_limit(:stripe)`.
|
|
172
|
+
# config.rate_limits.register(:stripe, limits: { second: 3, minute: 100 })
|
|
173
|
+
|
|
174
|
+
## OpenTelemetry / custom middlewares. Default: [].
|
|
175
|
+
# config.middlewares = [RubyReactor::OpenTelemetry]
|
|
119
176
|
end
|
|
120
177
|
```
|
|
121
178
|
|
|
179
|
+
You can also leave out the `configure` block entirely — defaults work for local development against a Redis on `localhost:6379`.
|
|
180
|
+
|
|
181
|
+
> **Crash recovery needs a kick.** The `sweeper_*` settings above only configure
|
|
182
|
+
> the recovery sweeper — they do not start it. Call `RubyReactor.start_sweeper!`
|
|
183
|
+
> once at boot (ideally from a Sidekiq `on(:startup)` hook) or no crashed reactor
|
|
184
|
+
> will ever resume. See [Durability & Recovery](#durability--recovery).
|
|
185
|
+
|
|
122
186
|
|
|
123
187
|
## Quick Start
|
|
124
188
|
|
|
@@ -323,6 +387,73 @@ def create(params)
|
|
|
323
387
|
end
|
|
324
388
|
```
|
|
325
389
|
|
|
390
|
+
### Durability & Recovery
|
|
391
|
+
|
|
392
|
+
Async reactors are durable: state lives in Redis, not in the job payload. Before
|
|
393
|
+
any background job is enqueued the root context is persisted, and after every
|
|
394
|
+
completed step a checkpoint advances the stored blob — so a crash re-runs at most
|
|
395
|
+
one step, never the whole reactor. Each running reactor also holds a short
|
|
396
|
+
**liveness lock** that a live worker auto-extends; its absence is how a dead
|
|
397
|
+
worker is detected.
|
|
398
|
+
|
|
399
|
+
**Recovery is not automatic until you start the sweeper.** A crashed worker's
|
|
400
|
+
reactor only resumes when the recovery sweeper notices the lapsed liveness lock
|
|
401
|
+
and re-enqueues it. The sweeper is a self-rescheduling chain — **kick it once per
|
|
402
|
+
process boot:**
|
|
403
|
+
|
|
404
|
+
The recommended spot is a Sidekiq server startup hook, so only the worker
|
|
405
|
+
process runs recovery (not your web/console/client processes):
|
|
406
|
+
|
|
407
|
+
```ruby
|
|
408
|
+
# config/initializers/sidekiq.rb
|
|
409
|
+
Sidekiq.configure_server do |config|
|
|
410
|
+
config.on(:startup) { RubyReactor.start_sweeper! }
|
|
411
|
+
end
|
|
412
|
+
```
|
|
413
|
+
|
|
414
|
+
Anywhere that runs once at boot works too — e.g. a Rails initializer:
|
|
415
|
+
|
|
416
|
+
```ruby
|
|
417
|
+
# config/initializers/ruby_reactor.rb
|
|
418
|
+
RubyReactor.start_sweeper!
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
That's all that's required: `start_sweeper!` is idempotent (safe to call on every
|
|
422
|
+
boot — duplicate kicks collapse to one chain), runs both the top-level reactor
|
|
423
|
+
sweeper and the map sweeper every `config.sweeper_interval` seconds, and stops if
|
|
424
|
+
you set `config.sweeper_enabled = false`. The interval is your recovery-latency
|
|
425
|
+
bound.
|
|
426
|
+
|
|
427
|
+
> **Sidekiq Enterprise `super_fetch` compatibility:** the chain is safe under
|
|
428
|
+
> reliable fetch. `super_fetch` re-runs a job whose worker died mid-execution, so
|
|
429
|
+
> a tick that crashes *after* enqueuing its successor but *before* acking would,
|
|
430
|
+
> with naive single-flight, be recovered alongside that successor and fork the
|
|
431
|
+
> chain (doubling every interval). RubyReactor avoids this: it never relies on
|
|
432
|
+
> "one job in the chain" — each next tick is claimed by a per-time-window lock, so
|
|
433
|
+
> a `super_fetch`-recovered tick computes the same window, loses the claim, and
|
|
434
|
+
> collapses back to a single successor. The startup hook above is likewise
|
|
435
|
+
> idempotent across multiple `super_fetch` server processes.
|
|
436
|
+
|
|
437
|
+
**Prefer your own scheduler?** Set `config.sweeper_enabled = false` (which makes
|
|
438
|
+
`start_sweeper!` a no-op) and drive recovery from cron, a Kubernetes `CronJob`,
|
|
439
|
+
`sidekiq-cron`, `sidekiq-scheduler`, or Rails recurring tasks. Each tick is one
|
|
440
|
+
call:
|
|
441
|
+
|
|
442
|
+
```ruby
|
|
443
|
+
RubyReactor.sweep_once # => { reactors: <n re-enqueued>, maps: <n recovered> }
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
For example, a rake task a system cron / CronJob can invoke:
|
|
447
|
+
|
|
448
|
+
```ruby
|
|
449
|
+
# lib/tasks/ruby_reactor.rake
|
|
450
|
+
namespace :ruby_reactor do
|
|
451
|
+
task sweep: :environment do
|
|
452
|
+
RubyReactor.sweep_once
|
|
453
|
+
end
|
|
454
|
+
end
|
|
455
|
+
```
|
|
456
|
+
|
|
326
457
|
### Interrupts (Pause & Resume)
|
|
327
458
|
|
|
328
459
|
Pause execution to wait for external events like webhooks or user approvals.
|
|
@@ -359,7 +490,7 @@ ApprovalReactor.continue_by_correlation_id(
|
|
|
359
490
|
)
|
|
360
491
|
```
|
|
361
492
|
|
|
362
|
-
### Locks &
|
|
493
|
+
### Locks, Semaphores & Ordered Locks
|
|
363
494
|
|
|
364
495
|
Coordinate across processes with Redis-backed primitives:
|
|
365
496
|
|
|
@@ -367,6 +498,7 @@ Coordinate across processes with Redis-backed primitives:
|
|
|
367
498
|
- **`with_semaphore`** — cap total concurrent runners per key (capacity control).
|
|
368
499
|
- **`with_rate_limit`** — fixed-window rate limit, single or multi-window ("3/sec AND 100/min"). Inline per-reactor, or reference a named limit registered once in `RubyReactor.configure` and shared across reactors.
|
|
369
500
|
- **`with_period`** — run at most once per calendar bucket (dedup / once-per-day, once-per-month, etc).
|
|
501
|
+
- **`with_ordered_lock`** — strict transaction ordering via a monotonically increasing nonce assigned at enqueue. Workers can only proceed when their nonce equals `last_completed + 1`.
|
|
370
502
|
|
|
371
503
|
```ruby
|
|
372
504
|
class RefundOrderReactor < RubyReactor::Reactor
|
|
@@ -423,6 +555,27 @@ class ChargeReactor < RubyReactor::Reactor
|
|
|
423
555
|
run { |args| Stripe.charge(args[:account_id]) }
|
|
424
556
|
end
|
|
425
557
|
end
|
|
558
|
+
|
|
559
|
+
class OrderedTransactionReactor < RubyReactor::Reactor
|
|
560
|
+
async
|
|
561
|
+
input :account_id
|
|
562
|
+
input :transaction
|
|
563
|
+
|
|
564
|
+
# Strict order: a monotonically increasing nonce is assigned at enqueue
|
|
565
|
+
# time (inside `Reactor.run`). Workers only execute when their nonce
|
|
566
|
+
# equals last_completed + 1; otherwise they snooze. After the sequence
|
|
567
|
+
# fully drains the counter resets to 0.
|
|
568
|
+
with_ordered_lock(poison_pill_timeout: 300) { |inputs| "txs:#{inputs[:account_id]}" }
|
|
569
|
+
|
|
570
|
+
step :apply do
|
|
571
|
+
argument :transaction, input(:transaction)
|
|
572
|
+
run { |args| Ledger.apply(args[:transaction]) }
|
|
573
|
+
end
|
|
574
|
+
end
|
|
575
|
+
|
|
576
|
+
# Caller-side order is preserved; the worker pool may pick jobs in any order
|
|
577
|
+
# but the gate enforces sequential execution per key.
|
|
578
|
+
[tx1, tx2, tx3].each { |tx| OrderedTransactionReactor.run(account_id: 42, transaction: tx) }
|
|
426
579
|
```
|
|
427
580
|
|
|
428
581
|
**Named global limits.** When several reactors hit the same external service, register the limit once and reference it by name. The name is the shared key base, so every reactor throttles against one bucket:
|
|
@@ -449,8 +602,8 @@ Referencing an unregistered name raises `RubyReactor::RateLimitRegistry::Unknown
|
|
|
449
602
|
|
|
450
603
|
On contention:
|
|
451
604
|
|
|
452
|
-
- **Inline** (`Reactor.run`) raises `RubyReactor::Lock::AcquisitionError` / `RubyReactor::Semaphore::AcquisitionError` / `RubyReactor::RateLimit::ExceededError`.
|
|
453
|
-
- **Async** (Sidekiq) snoozes the job via `perform_in(delay, ...)`. For rate limits the delay
|
|
605
|
+
- **Inline** (`Reactor.run`) raises `RubyReactor::Lock::AcquisitionError` / `RubyReactor::Semaphore::AcquisitionError` / `RubyReactor::RateLimit::ExceededError` / `RubyReactor::OrderedLock::WaitError`.
|
|
606
|
+
- **Async** (Sidekiq) snoozes the job via `perform_in(delay, ...)`. For rate limits the delay uses the error's `retry_after_seconds` hint (precise wakeup — the bucket roll time is known exactly); for locks, semaphores, and ordered-lock waits it's `lock_snooze_base_delay + jitter` (a short re-poll, since a held lock or a live blocker nonce typically clears in milliseconds). Snoozes do not count against the Sidekiq retry budget. After `lock_snooze_max_attempts` snoozes the context is marked failed (ordered-lock waits bypass the cap — see the ordered-lock docs).
|
|
454
607
|
|
|
455
608
|
On dedup hits (period gate already marked), the reactor returns a `RubyReactor::Skipped` result instead — no steps run, no exception:
|
|
456
609
|
|
|
@@ -472,7 +625,7 @@ step :ensure_active do
|
|
|
472
625
|
end
|
|
473
626
|
```
|
|
474
627
|
|
|
475
|
-
See [Locks, Semaphores, Rate Limits &
|
|
628
|
+
See [Locks, Semaphores, Rate Limits, Periods & Ordered Locks](documentation/locks_and_semaphores.md) for re-entrancy, auto-extend, multi-window quotas, bucket semantics, owner identity, snooze tuning, ordered-lock assignment + poison-pill semantics, and operational notes.
|
|
476
629
|
|
|
477
630
|
### Map & Parallel Execution
|
|
478
631
|
|
|
@@ -986,9 +1139,9 @@ Learn how to pause and resume reactors to handle long-running processes, manual
|
|
|
986
1139
|
### [Testing with RSpec](documentation/testing.md)
|
|
987
1140
|
Comprehensive guide to testing reactors with RubyReactor's testing utilities. Learn about the `TestSubject` class for reactor execution and introspection, step mocking for isolating dependencies, testing nested and composed reactors, and custom RSpec matchers like `be_success`, `have_run_step`, and `have_retried_step`.
|
|
988
1141
|
|
|
989
|
-
### [Locks, Semaphores, Rate Limits &
|
|
1142
|
+
### [Locks, Semaphores, Rate Limits, Periods & Ordered Locks](documentation/locks_and_semaphores.md)
|
|
990
1143
|
|
|
991
|
-
Coordinate access to shared resources across processes with Redis-backed primitives: exclusive locks (`with_lock`), concurrency-limiting semaphores (`with_semaphore`), fixed-window rate limits with multi-window quotas (`with_rate_limit`),
|
|
1144
|
+
Coordinate access to shared resources across processes with Redis-backed primitives: exclusive locks (`with_lock`), concurrency-limiting semaphores (`with_semaphore`), fixed-window rate limits with multi-window quotas (`with_rate_limit`), calendar-bucketed dedup (`with_period`, returning `Skipped` results), and strict sequential ordering via a monotonically increasing nonce assigned at enqueue (`with_ordered_lock`). Covers re-entrancy across composed reactors, TTL auto-extend, inline-vs-async contention behavior, smart `retry_after` snoozes for rate limits, snooze tuning, the token-based semaphore safety model, once-per-day/month/year scheduling patterns, ordered-lock counter reset on drain, poison-pill timeouts, and deadlock-safe composition rules.
|
|
992
1145
|
|
|
993
1146
|
### [Middlewares & OpenTelemetry](documentation/middlewares.md)
|
|
994
1147
|
|
|
@@ -9,12 +9,76 @@ module RubyReactor
|
|
|
9
9
|
|
|
10
10
|
attr_writer :sidekiq_queue, :sidekiq_retry_count, :logger, :async_router,
|
|
11
11
|
:lock_snooze_base_delay, :lock_snooze_jitter, :lock_snooze_max_attempts,
|
|
12
|
-
:middlewares
|
|
12
|
+
:middlewares, :context_ttl, :context_lock_ttl, :checkpoint_min_interval,
|
|
13
|
+
:sweeper_enabled, :sweeper_interval, :sweeper_limit
|
|
13
14
|
|
|
14
15
|
def sidekiq_queue
|
|
15
16
|
@sidekiq_queue ||= :default
|
|
16
17
|
end
|
|
17
18
|
|
|
19
|
+
# Retention TTL (seconds) for a stored reactor context. Storage is
|
|
20
|
+
# load-bearing for resume, so this must comfortably exceed the worst-case
|
|
21
|
+
# snooze/retry window. Refreshed on every checkpoint write.
|
|
22
|
+
def context_ttl
|
|
23
|
+
@context_ttl ||= 86_400
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
# Minimum wall-clock seconds between two PER-STEP durable checkpoints within a
|
|
27
|
+
# single worker run. The save-per-step checkpoint (`on_step_complete`) bounds
|
|
28
|
+
# crash re-execution to one step, but re-serializes and re-writes the WHOLE
|
|
29
|
+
# root blob after every Success — O(steps × context_size) writes for a long,
|
|
30
|
+
# large reactor. This throttle coalesces the mid-run intermediate checkpoints:
|
|
31
|
+
# a checkpoint is written only if at least this many seconds have elapsed since
|
|
32
|
+
# the last one. The final terminal/handoff state is ALWAYS persisted (by the
|
|
33
|
+
# run's ensure-save and the pre-enqueue checkpoint), so throttling only affects
|
|
34
|
+
# mid-run granularity. Tradeoff: with interval > 0, a crash may re-run every
|
|
35
|
+
# step completed inside the last interval — safe only when those steps are
|
|
36
|
+
# idempotent or side-effect-free.
|
|
37
|
+
#
|
|
38
|
+
# Default 0 -> checkpoint after EVERY step (strongest guarantee, no coalescing).
|
|
39
|
+
def checkpoint_min_interval
|
|
40
|
+
@checkpoint_min_interval ||= 0
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# Whether the recovery sweepers run. The host kicks the self-rescheduling
|
|
44
|
+
# chain once (`RubyReactor.start_sweeper!`, e.g. from an initializer); each
|
|
45
|
+
# tick re-checks this flag, so flipping it to false stops the chain at the
|
|
46
|
+
# next tick. Default on: durability is inert without a running sweeper, so
|
|
47
|
+
# recovery must work out of the box.
|
|
48
|
+
def sweeper_enabled
|
|
49
|
+
@sweeper_enabled = true if @sweeper_enabled.nil?
|
|
50
|
+
@sweeper_enabled
|
|
51
|
+
end
|
|
52
|
+
|
|
53
|
+
# Seconds between sweeps. This is the upper bound on recovery latency for a
|
|
54
|
+
# dead worker — lower it for faster recovery, raise it to cut scan load.
|
|
55
|
+
def sweeper_interval
|
|
56
|
+
@sweeper_interval ||= 30
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# Max contexts/maps inspected per sweep (passed to each sweeper's run_once).
|
|
60
|
+
def sweeper_limit
|
|
61
|
+
@sweeper_limit ||= 1000
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
# TTL (seconds) for the per-context liveness lock (`async:<id>`). Short by
|
|
65
|
+
# design — it is a liveness signal, not retention. A live worker auto-extends
|
|
66
|
+
# it (every ttl/3 s, from a background thread); its absence is the sweeper's
|
|
67
|
+
# "worker died" signal.
|
|
68
|
+
#
|
|
69
|
+
# SAFETY CONSTRAINT: this MUST exceed the longest a single step can run
|
|
70
|
+
# WITHOUT letting the auto-extend thread make progress. Under MRI the
|
|
71
|
+
# extender shares the GIL, so a step that holds the GIL continuously for
|
|
72
|
+
# longer than this TTL (a long CPU-bound pure-Ruby loop, a C extension that
|
|
73
|
+
# never releases the GIL, or a stop-the-world GC pause) lets the lock lapse.
|
|
74
|
+
# A lapsed lock looks "dead" to the sweeper, which may re-enqueue a duplicate
|
|
75
|
+
# that runs CONCURRENTLY with the still-live original — a double-run. I/O-bound
|
|
76
|
+
# steps release the GIL and keep the lock fresh, so the default 60s suits
|
|
77
|
+
# typical workloads; raise it if you run long synchronous CPU-bound steps.
|
|
78
|
+
def context_lock_ttl
|
|
79
|
+
@context_lock_ttl ||= 60
|
|
80
|
+
end
|
|
81
|
+
|
|
18
82
|
def sidekiq_retry_count
|
|
19
83
|
@sidekiq_retry_count ||= 3
|
|
20
84
|
end
|
|
@@ -36,7 +100,7 @@ module RubyReactor
|
|
|
36
100
|
end
|
|
37
101
|
|
|
38
102
|
def logger
|
|
39
|
-
@logger ||= Logger.new($
|
|
103
|
+
@logger ||= Logger.new($stdout)
|
|
40
104
|
end
|
|
41
105
|
|
|
42
106
|
def async_router
|
|
@@ -19,13 +19,18 @@ module RubyReactor
|
|
|
19
19
|
|
|
20
20
|
def deserialize(serialized_data)
|
|
21
21
|
decompressed = decompress_if_needed(serialized_data)
|
|
22
|
-
|
|
22
|
+
deserialize_hash(JSON.parse(decompressed, symbolize_names: false))
|
|
23
|
+
rescue JSON::ParserError => e
|
|
24
|
+
raise RubyReactor::Error::DeserializationError, "Failed to parse serialized context: #{e.message}"
|
|
25
|
+
end
|
|
23
26
|
|
|
27
|
+
# Deserialize from an already-parsed Hash (e.g. what the storage adapter's
|
|
28
|
+
# `retrieve_context` returns). Lets the rehydrate-by-id worker path avoid a
|
|
29
|
+
# second JSON parse while still schema-validating. Schema validation lives
|
|
30
|
+
# here so both the string and Hash entry points enforce it.
|
|
31
|
+
def deserialize_hash(data)
|
|
24
32
|
validate_schema_version(data)
|
|
25
|
-
|
|
26
33
|
Context.deserialize_from_retry(data)
|
|
27
|
-
rescue JSON::ParserError => e
|
|
28
|
-
raise RubyReactor::Error::DeserializationError, "Failed to parse serialized context: #{e.message}"
|
|
29
34
|
end
|
|
30
35
|
|
|
31
36
|
# rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength
|
|
@@ -41,6 +41,7 @@ module RubyReactor
|
|
|
41
41
|
end
|
|
42
42
|
|
|
43
43
|
def build
|
|
44
|
+
warn_if_child_has_ordered_lock!
|
|
44
45
|
dependencies = extract_dependencies_from_mappings
|
|
45
46
|
|
|
46
47
|
step_config = {
|
|
@@ -78,6 +79,25 @@ module RubyReactor
|
|
|
78
79
|
|
|
79
80
|
private
|
|
80
81
|
|
|
82
|
+
# Composed children bypass `Reactor#run`, so `assign_ordered_lock_nonce!`
|
|
83
|
+
# never fires for them — their `with_ordered_lock` declaration is silently
|
|
84
|
+
# ignored. Surface this at class load so users don't expect ordering
|
|
85
|
+
# enforcement that isn't happening. Nested ordered-lock sequences must be
|
|
86
|
+
# invoked as top-level `Reactor.run` to participate.
|
|
87
|
+
def warn_if_child_has_ordered_lock!
|
|
88
|
+
return unless @composed_reactor_class
|
|
89
|
+
return unless @composed_reactor_class.respond_to?(:ordered_lock_config)
|
|
90
|
+
return unless @composed_reactor_class.ordered_lock_config
|
|
91
|
+
|
|
92
|
+
parent_name = @reactor&.name || "<anonymous>"
|
|
93
|
+
child_name = @composed_reactor_class.name || "<anonymous>"
|
|
94
|
+
RubyReactor.configuration.logger.warn(
|
|
95
|
+
"RubyReactor: `with_ordered_lock` on #{child_name} is ignored when " \
|
|
96
|
+
"composed by #{parent_name}##{@name}. Nested ordered-lock sequences " \
|
|
97
|
+
"are independent and must run via top-level `Reactor.run` to be enforced."
|
|
98
|
+
)
|
|
99
|
+
end
|
|
100
|
+
|
|
81
101
|
def ensure_composed_reactor_class!
|
|
82
102
|
raise ArgumentError, "No block provided for inline compose" unless @composed_reactor_class
|
|
83
103
|
end
|
|
@@ -8,7 +8,7 @@ module RubyReactor
|
|
|
8
8
|
end
|
|
9
9
|
|
|
10
10
|
module ClassMethods
|
|
11
|
-
attr_reader :lock_config, :semaphore_config, :period_config, :rate_limit_config
|
|
11
|
+
attr_reader :lock_config, :semaphore_config, :period_config, :rate_limit_config, :ordered_lock_config
|
|
12
12
|
|
|
13
13
|
# Propagate lock/semaphore/period/rate-limit config to subclasses;
|
|
14
14
|
# without this a subclass of a configured reactor would silently lose
|
|
@@ -19,6 +19,7 @@ module RubyReactor
|
|
|
19
19
|
subclass.instance_variable_set(:@semaphore_config, @semaphore_config) if @semaphore_config
|
|
20
20
|
subclass.instance_variable_set(:@period_config, @period_config) if @period_config
|
|
21
21
|
subclass.instance_variable_set(:@rate_limit_config, @rate_limit_config) if @rate_limit_config
|
|
22
|
+
subclass.instance_variable_set(:@ordered_lock_config, @ordered_lock_config) if @ordered_lock_config
|
|
22
23
|
end
|
|
23
24
|
|
|
24
25
|
# Configure locking for this reactor
|
|
@@ -74,6 +75,45 @@ module RubyReactor
|
|
|
74
75
|
}
|
|
75
76
|
end
|
|
76
77
|
|
|
78
|
+
# Configure strict-ordering nonce gating for this reactor. A
|
|
79
|
+
# monotonically increasing nonce is assigned at enqueue time; the
|
|
80
|
+
# worker can only proceed when its nonce equals `last_completed + 1`.
|
|
81
|
+
# Otherwise the worker raises {OrderedLock::WaitError} and the Sidekiq
|
|
82
|
+
# worker snoozes via `perform_in`.
|
|
83
|
+
#
|
|
84
|
+
# Counters reset to 0 once the sequence fully drains (last_completed
|
|
85
|
+
# catches up to next). Re-entrancy is NOT supported — a nested reactor
|
|
86
|
+
# with its own `with_ordered_lock` is an independent sequence.
|
|
87
|
+
#
|
|
88
|
+
# @param poison_pill_timeout [Integer] seconds since the blocker nonce
|
|
89
|
+
# was assigned before the gate auto-advances past it. Protects
|
|
90
|
+
# against permanent head-of-line blocking from a caller that INCRed
|
|
91
|
+
# the counter but crashed before enqueueing.
|
|
92
|
+
# @param ttl [Integer] TTL on the counter keys, refreshed on every
|
|
93
|
+
# assign. Only fully-drained sequences GC themselves.
|
|
94
|
+
# @param strict [Boolean] When true (default), if any nonce in the
|
|
95
|
+
# sequence terminates with a `Failure`, all subsequent nonces are
|
|
96
|
+
# short-circuited with `Skipped(reason: :ordered_lock_chain_failed)`
|
|
97
|
+
# instead of executing. This models "stop the line on the first
|
|
98
|
+
# problem" pipelines (e.g. ledger transactions). When false, the
|
|
99
|
+
# sequence keeps executing every nonce in order regardless of prior
|
|
100
|
+
# failures. The poison state is per-key and clears on full drain. The
|
|
101
|
+
# check only applies to a fresh `execute`; an already-started run
|
|
102
|
+
# that paused (InterruptResult/AsyncResult) completes on resume even
|
|
103
|
+
# if the chain failed in the meantime.
|
|
104
|
+
# @yield [inputs] Block that returns the ordered-lock key string.
|
|
105
|
+
def with_ordered_lock(poison_pill_timeout: OrderedLock::DEFAULT_POISON_PILL_TIMEOUT,
|
|
106
|
+
ttl: OrderedLock::DEFAULT_TTL,
|
|
107
|
+
strict: true,
|
|
108
|
+
&block)
|
|
109
|
+
@ordered_lock_config = {
|
|
110
|
+
poison_pill_timeout: poison_pill_timeout,
|
|
111
|
+
ttl: ttl,
|
|
112
|
+
strict: strict,
|
|
113
|
+
key_proc: block
|
|
114
|
+
}
|
|
115
|
+
end
|
|
116
|
+
|
|
77
117
|
# Configure rate limiting for this reactor (fixed-window counter).
|
|
78
118
|
# Pass either a single window via `limit:` + `period:`, or a hash of
|
|
79
119
|
# windows via `limits:` for layered API quotas.
|