ruby_reactor 0.5.1 → 0.5.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. checksums.yaml +4 -4
  2. data/.release-please-manifest.json +1 -1
  3. data/CHANGELOG.md +14 -0
  4. data/README.md +179 -26
  5. data/lib/ruby_reactor/configuration.rb +66 -2
  6. data/lib/ruby_reactor/context_serializer.rb +9 -4
  7. data/lib/ruby_reactor/dsl/compose_builder.rb +20 -0
  8. data/lib/ruby_reactor/dsl/lockable.rb +41 -1
  9. data/lib/ruby_reactor/executor/ordered_lock_support.rb +307 -0
  10. data/lib/ruby_reactor/executor/retry_manager.rb +7 -2
  11. data/lib/ruby_reactor/executor/step_executor.rb +25 -5
  12. data/lib/ruby_reactor/executor.rb +166 -52
  13. data/lib/ruby_reactor/lock.rb +13 -0
  14. data/lib/ruby_reactor/map/collector.rb +41 -0
  15. data/lib/ruby_reactor/map/dispatcher.rb +42 -0
  16. data/lib/ruby_reactor/map/element_executor.rb +39 -0
  17. data/lib/ruby_reactor/map/helpers.rb +10 -3
  18. data/lib/ruby_reactor/map/sweeper.rb +110 -0
  19. data/lib/ruby_reactor/ordered_lock.rb +158 -0
  20. data/lib/ruby_reactor/reactor.rb +48 -5
  21. data/lib/ruby_reactor/rspec/helpers.rb +6 -0
  22. data/lib/ruby_reactor/rspec/matchers.rb +66 -0
  23. data/lib/ruby_reactor/rspec/sidekiq_helpers.rb +70 -0
  24. data/lib/ruby_reactor/rspec/storage_reset.rb +23 -0
  25. data/lib/ruby_reactor/rspec/test_subject.rb +14 -28
  26. data/lib/ruby_reactor/rspec.rb +37 -0
  27. data/lib/ruby_reactor/sidekiq_adapter.rb +9 -8
  28. data/lib/ruby_reactor/sidekiq_workers/sweeper_worker.rb +73 -0
  29. data/lib/ruby_reactor/sidekiq_workers/worker.rb +82 -36
  30. data/lib/ruby_reactor/step/map_step.rb +18 -2
  31. data/lib/ruby_reactor/storage/redis_adapter.rb +84 -60
  32. data/lib/ruby_reactor/storage/redis_locking.rb +8 -0
  33. data/lib/ruby_reactor/storage/redis_ordered_locking.rb +382 -0
  34. data/lib/ruby_reactor/sweeper.rb +58 -0
  35. data/lib/ruby_reactor/version.rb +1 -1
  36. data/lib/ruby_reactor.rb +43 -0
  37. metadata +9 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 54ecb36ac72eedf48af0025dff29d83986449586683300ce3fe8fd874c1412d5
4
- data.tar.gz: 5e3565e3e238bad746d93982ca7b01560893a68755be18fbfce95df6f54ce5b3
3
+ metadata.gz: 778bc5305c6d1f20833819afd9ccd46f5a5b4c2c135d8e63344d6530f9a733f1
4
+ data.tar.gz: 32e769816eba846f419e3f31e8290b94e8ff04fe6ea71fef125bb128b3085b82
5
5
  SHA512:
6
- metadata.gz: 708acdb0c74582cea4c33210ea1bac055080c7dd19c69d166db4b2c22705de2935346d6b09d4fc6c9cbb1bda5ba235cade92a88f04fe791e364bb78bda256138
7
- data.tar.gz: e3f03e46d71babe276224571eaad3e794c7d8c695e2865333f5021e680feb21a12cd3b33527035e1349dc600d01faec87b573b691d7e50521c4f0d81610c8b8f
6
+ metadata.gz: 592db1d0ef94153a4ea028aa3cdeb59f7e4c73929ebec5afd5a9795a93d27a0cfb0c4bd3e135ccf1b14c0329123be6cf0905ad4a141018993832d21212754db3
7
+ data.tar.gz: 2fd90e47af8e26cf2d58468a1f0629fd1dc04890df0a6e1704e8d6cba5c65b1e050f10f56628c27c7c90b0545d081e33169e3bfd62f416affd761b62edeb24a0
@@ -1,3 +1,3 @@
1
1
  {
2
- ".": "0.5.1"
2
+ ".": "0.5.3"
3
3
  }
data/CHANGELOG.md CHANGED
@@ -1,5 +1,19 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.5.3](https://github.com/arturictus/ruby_reactor/compare/v0.5.2...v0.5.3) (2026-06-17)
4
+
5
+
6
+ ### Features
7
+
8
+ * Durability & Recovery ([#39](https://github.com/arturictus/ruby_reactor/issues/39)) ([103e583](https://github.com/arturictus/ruby_reactor/commit/103e5835b413eec2302fa63f3e998d487cfd9eaf))
9
+
10
+ ## [0.5.2](https://github.com/arturictus/ruby_reactor/compare/v0.5.1...v0.5.2) (2026-06-14)
11
+
12
+
13
+ ### Features
14
+
15
+ * Nonce lock ([#26](https://github.com/arturictus/ruby_reactor/issues/26)) ([5925cac](https://github.com/arturictus/ruby_reactor/commit/5925cac7af93f59be6c0a8a98ab020f96080f60b))
16
+
3
17
  ## [0.5.1](https://github.com/arturictus/ruby_reactor/compare/v0.5.0...v0.5.1) (2026-06-14)
4
18
 
5
19
 
data/README.md CHANGED
@@ -24,7 +24,7 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
24
24
  - **Compensation**: Automatic rollback of completed steps when a failure occurs.
25
25
  - **Interrupts**: Pause and resume workflows to wait for external events (webhooks, user approvals).
26
26
  - **Input Validation**: Integrated with `dry-validation` for robust input checking.
27
- - **Distributed Locks, Semaphores, Rate Limits & Periods**: Coordinate across processes with Redis-backed primitives — exclusive locks for at-most-one-runner, semaphores for capacity caps, fixed-window rate limits for external APIs (single or multi-window like "3/sec AND 100/min"), and `with_period` to dedup reactors to once per calendar bucket (once per day/month/year/etc). Async jobs snooze on contention with smart `retry_after` instead of consuming retry budget.
27
+ - **Distributed Locks, Semaphores, Rate Limits, Periods & Ordered Locks**: Coordinate across processes with Redis-backed primitives — exclusive locks for at-most-one-runner, semaphores for capacity caps, fixed-window rate limits for external APIs (single or multi-window like "3/sec AND 100/min"), `with_period` to dedup reactors to once per calendar bucket, and `with_ordered_lock` for strict transaction ordering via a monotonically increasing nonce assigned at enqueue. Async jobs snooze on contention with smart `retry_after` instead of consuming retry budget.
28
28
 
29
29
  ## Comparison
30
30
 
@@ -36,6 +36,7 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
36
36
  | Locks / sem / rate / per | Yes | No | No | Manual |
37
37
  | Built-in web dashboard | Yes | No | No | No |
38
38
  | Async with Sidekiq | Yes | No | Limited | Yes |
39
+ | Durable crash recovery | Yes | No | No | Manual |
39
40
 
40
41
  ## Real-World Use Cases
41
42
 
@@ -44,6 +45,7 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
44
45
  - **Subscription Billing**: Coordinate Stripe charges, invoice email generation, and internal entitlement updates. Use interrupts to pause the workflow when 3rd-party APIs are required to continue the workflow or when specific customer approval is needed.
45
46
 
46
47
  ## Table of Contents
48
+
47
49
  - [Features](#features)
48
50
  - [Comparison](#comparison)
49
51
  - [Real-World Use Cases](#real-world-use-cases)
@@ -57,8 +59,9 @@ The key value is **Reliability**: if any part of your workflow fails, Ruby React
57
59
  - [Async Execution](#async-execution)
58
60
  - [Full Reactor Async](#full-reactor-async)
59
61
  - [Step-Level Async](#step-level-async)
62
+ - [Durability & Recovery](#durability--recovery)
60
63
  - [Interrupts (Pause & Resume)](#interrupts-pause--resume)
61
- - [Locks & Semaphores](#locks--semaphores)
64
+ - [Locks, Semaphores & Ordered Locks](#locks-semaphores--ordered-locks)
62
65
  - [Map & Parallel Execution](#map--parallel-execution)
63
66
  - [Map with Dynamic Source (ActiveRecord)](#map-with-dynamic-source-activerecord)
64
67
  - [Input Validation](#input-validation)
@@ -90,35 +93,96 @@ Or install it yourself as:
90
93
 
91
94
  ## Configuration
92
95
 
93
- Configure RubyReactor with your Sidekiq and Redis settings:
96
+ Every setting is **optional** — RubyReactor ships with the defaults shown. Drop
97
+ this into an initializer (e.g. `config/initializers/ruby_reactor.rb`); pasted as-is
98
+ it changes nothing, so it doubles as a reference of every knob.
99
+
100
+ > **Reading the block:** lines starting with `##` are documentation. Lines starting
101
+ > with a single `#` (a `config.…` call) are real settings commented at their
102
+ > default — uncomment one to enable it.
94
103
 
95
104
  ```ruby
96
105
  RubyReactor.configure do |config|
97
- # Redis configuration for state persistence
98
- config.storage.adapter = :redis
106
+ ## === Storage (Redis) ===
107
+
108
+ ## Storage adapter. Default: :redis (the only adapter shipped today).
109
+ # config.storage.adapter = :redis
110
+
111
+ ## Redis URL. Default: "redis://localhost:6379/0".
99
112
  config.storage.redis_url = ENV.fetch("REDIS_URL", "redis://localhost:6379/0")
100
- config.storage.redis_options = { timeout: 1 }
101
113
 
102
- # Sidekiq configuration for async execution
103
- config.sidekiq_queue = :default
104
- config.sidekiq_retry_count = 3
114
+ ## Extra options passed to Redis.new. Default: {}.
115
+ # config.storage.redis_options = { timeout: 1 }
105
116
 
106
- # Lock contention snooze behavior for async reactors. When a Sidekiq worker
107
- # cannot acquire a lock or semaphore, it re-enqueues itself with this delay
108
- # (plus jitter) up to `lock_snooze_max_attempts` times before giving up.
109
- config.lock_snooze_base_delay = 5
110
- config.lock_snooze_jitter = 5
111
- config.lock_snooze_max_attempts = 20
117
+ ## === Sidekiq ===
112
118
 
113
- # Named rate limits shared across reactors. Reference them with
114
- # `with_rate_limit(:stripe)`. See Locks, Semaphores, Rate Limits & Periods.
115
- config.rate_limits.register(:stripe, limits: { second: 3, minute: 100 })
119
+ ## Sidekiq queue used by RubyReactor's async worker. Default: :default.
120
+ # config.sidekiq_queue = :default
121
+
122
+ ## Sidekiq retry count for infrastructure failures only (deserialization,
123
+ ## Redis, network). Step retries are managed separately. Default: 3.
124
+ # config.sidekiq_retry_count = 3
125
+
126
+ ## === Contention snooze (locks / semaphores / rate limits / ordered locks) ===
127
+
128
+ ## When a Sidekiq worker cannot acquire a primitive it re-enqueues itself with
129
+ ## `lock_snooze_base_delay + rand(0..lock_snooze_jitter)` seconds (rate-limit
130
+ ## uses a precise `retry_after_seconds` hint from the error; ordered-lock waits
131
+ ## re-poll at the base delay so a successor catches its blocker finishing fast),
132
+ ## up to `lock_snooze_max_attempts` times before marking the context :failed.
133
+ ## Set max_attempts to :infinity to never give up.
134
+ # config.lock_snooze_base_delay = 5
135
+ # config.lock_snooze_jitter = 5
136
+ # config.lock_snooze_max_attempts = 20
137
+
138
+ ## === Durability & crash recovery (see "Durability & Recovery" below) ===
139
+
140
+ ## Retention TTL (seconds) for stored reactor/map state. Must exceed your
141
+ ## worst-case snooze/retry window; re-stamped on every write. Default: 86_400.
142
+ # config.context_ttl = 86_400
143
+
144
+ ## TTL (seconds) for the per-context liveness lock. A live worker auto-extends
145
+ ## it; its absence is the sweeper's "worker died" signal. Must exceed the
146
+ ## longest a single step can run without yielding the GIL. Default: 60.
147
+ # config.context_lock_ttl = 60
148
+
149
+ ## Minimum seconds between per-step checkpoints within one run. 0 = checkpoint
150
+ ## after every step (strongest guarantee). Raise to coalesce mid-run writes for
151
+ ## long reactors — only safe when steps are idempotent. Default: 0.
152
+ # config.checkpoint_min_interval = 0
153
+
154
+ ## Recovery sweeper (the chain is kicked once by `RubyReactor.start_sweeper!`).
155
+ # config.sweeper_enabled = true # run recovery by default
156
+ # config.sweeper_interval = 30 # seconds between sweeps = recovery-latency bound
157
+ # config.sweeper_limit = 1000 # max contexts/maps inspected per sweep
116
158
 
117
- # Logger configuration
118
- config.logger = Logger.new($stdout)
159
+ ## === Misc ===
160
+
161
+ ## Logger. Default: Logger.new($stdout).
162
+ # config.logger = Logger.new($stdout)
163
+
164
+ ## Async router. Default: RubyReactor::SidekiqAdapter. Swap for a custom adapter
165
+ ## if you don't use Sidekiq — it only needs to respond to
166
+ ## `perform_async(context_id, reactor_class_name, **)`.
167
+ # config.async_router = MyCustomAdapter
168
+
169
+ ## === Examples (no default — set these to use the feature) ===
170
+
171
+ ## Named rate limits shared across reactors. Reference with `with_rate_limit(:stripe)`.
172
+ # config.rate_limits.register(:stripe, limits: { second: 3, minute: 100 })
173
+
174
+ ## OpenTelemetry / custom middlewares. Default: [].
175
+ # config.middlewares = [RubyReactor::OpenTelemetry]
119
176
  end
120
177
  ```
121
178
 
179
+ You can also leave out the `configure` block entirely — defaults work for local development against a Redis on `localhost:6379`.
180
+
181
+ > **Crash recovery needs a kick.** The `sweeper_*` settings above only configure
182
+ > the recovery sweeper — they do not start it. Call `RubyReactor.start_sweeper!`
183
+ > once at boot (ideally from a Sidekiq `on(:startup)` hook) or no crashed reactor
184
+ > will ever resume. See [Durability & Recovery](#durability--recovery).
185
+
122
186
 
123
187
  ## Quick Start
124
188
 
@@ -323,6 +387,73 @@ def create(params)
323
387
  end
324
388
  ```
325
389
 
390
+ ### Durability & Recovery
391
+
392
+ Async reactors are durable: state lives in Redis, not in the job payload. Before
393
+ any background job is enqueued the root context is persisted, and after every
394
+ completed step a checkpoint advances the stored blob — so a crash re-runs at most
395
+ one step, never the whole reactor. Each running reactor also holds a short
396
+ **liveness lock** that a live worker auto-extends; its absence is how a dead
397
+ worker is detected.
398
+
399
+ **Recovery is not automatic until you start the sweeper.** A crashed worker's
400
+ reactor only resumes when the recovery sweeper notices the lapsed liveness lock
401
+ and re-enqueues it. The sweeper is a self-rescheduling chain — **kick it once per
402
+ process boot:**
403
+
404
+ The recommended spot is a Sidekiq server startup hook, so only the worker
405
+ process runs recovery (not your web/console/client processes):
406
+
407
+ ```ruby
408
+ # config/initializers/sidekiq.rb
409
+ Sidekiq.configure_server do |config|
410
+ config.on(:startup) { RubyReactor.start_sweeper! }
411
+ end
412
+ ```
413
+
414
+ Anywhere that runs once at boot works too — e.g. a Rails initializer:
415
+
416
+ ```ruby
417
+ # config/initializers/ruby_reactor.rb
418
+ RubyReactor.start_sweeper!
419
+ ```
420
+
421
+ That's all that's required: `start_sweeper!` is idempotent (safe to call on every
422
+ boot — duplicate kicks collapse to one chain), runs both the top-level reactor
423
+ sweeper and the map sweeper every `config.sweeper_interval` seconds, and stops if
424
+ you set `config.sweeper_enabled = false`. The interval is your recovery-latency
425
+ bound.
426
+
427
+ > **Sidekiq Enterprise `super_fetch` compatibility:** the chain is safe under
428
+ > reliable fetch. `super_fetch` re-runs a job whose worker died mid-execution, so
429
+ > a tick that crashes *after* enqueuing its successor but *before* acking would,
430
+ > with naive single-flight, be recovered alongside that successor and fork the
431
+ > chain (doubling every interval). RubyReactor avoids this: it never relies on
432
+ > "one job in the chain" — each next tick is claimed by a per-time-window lock, so
433
+ > a `super_fetch`-recovered tick computes the same window, loses the claim, and
434
+ > collapses back to a single successor. The startup hook above is likewise
435
+ > idempotent across multiple `super_fetch` server processes.
436
+
437
+ **Prefer your own scheduler?** Set `config.sweeper_enabled = false` (which makes
438
+ `start_sweeper!` a no-op) and drive recovery from cron, a Kubernetes `CronJob`,
439
+ `sidekiq-cron`, `sidekiq-scheduler`, or Rails recurring tasks. Each tick is one
440
+ call:
441
+
442
+ ```ruby
443
+ RubyReactor.sweep_once # => { reactors: <n re-enqueued>, maps: <n recovered> }
444
+ ```
445
+
446
+ For example, a rake task a system cron / CronJob can invoke:
447
+
448
+ ```ruby
449
+ # lib/tasks/ruby_reactor.rake
450
+ namespace :ruby_reactor do
451
+ task sweep: :environment do
452
+ RubyReactor.sweep_once
453
+ end
454
+ end
455
+ ```
456
+
326
457
  ### Interrupts (Pause & Resume)
327
458
 
328
459
  Pause execution to wait for external events like webhooks or user approvals.
@@ -359,7 +490,7 @@ ApprovalReactor.continue_by_correlation_id(
359
490
  )
360
491
  ```
361
492
 
362
- ### Locks & Semaphores
493
+ ### Locks, Semaphores & Ordered Locks
363
494
 
364
495
  Coordinate across processes with Redis-backed primitives:
365
496
 
@@ -367,6 +498,7 @@ Coordinate across processes with Redis-backed primitives:
367
498
  - **`with_semaphore`** — cap total concurrent runners per key (capacity control).
368
499
  - **`with_rate_limit`** — fixed-window rate limit, single or multi-window ("3/sec AND 100/min"). Inline per-reactor, or reference a named limit registered once in `RubyReactor.configure` and shared across reactors.
369
500
  - **`with_period`** — run at most once per calendar bucket (dedup / once-per-day, once-per-month, etc).
501
+ - **`with_ordered_lock`** — strict transaction ordering via a monotonically increasing nonce assigned at enqueue. Workers can only proceed when their nonce equals `last_completed + 1`.
370
502
 
371
503
  ```ruby
372
504
  class RefundOrderReactor < RubyReactor::Reactor
@@ -423,6 +555,27 @@ class ChargeReactor < RubyReactor::Reactor
423
555
  run { |args| Stripe.charge(args[:account_id]) }
424
556
  end
425
557
  end
558
+
559
+ class OrderedTransactionReactor < RubyReactor::Reactor
560
+ async
561
+ input :account_id
562
+ input :transaction
563
+
564
+ # Strict order: a monotonically increasing nonce is assigned at enqueue
565
+ # time (inside `Reactor.run`). Workers only execute when their nonce
566
+ # equals last_completed + 1; otherwise they snooze. After the sequence
567
+ # fully drains the counter resets to 0.
568
+ with_ordered_lock(poison_pill_timeout: 300) { |inputs| "txs:#{inputs[:account_id]}" }
569
+
570
+ step :apply do
571
+ argument :transaction, input(:transaction)
572
+ run { |args| Ledger.apply(args[:transaction]) }
573
+ end
574
+ end
575
+
576
+ # Caller-side order is preserved; the worker pool may pick jobs in any order
577
+ # but the gate enforces sequential execution per key.
578
+ [tx1, tx2, tx3].each { |tx| OrderedTransactionReactor.run(account_id: 42, transaction: tx) }
426
579
  ```
427
580
 
428
581
  **Named global limits.** When several reactors hit the same external service, register the limit once and reference it by name. The name is the shared key base, so every reactor throttles against one bucket:
@@ -449,8 +602,8 @@ Referencing an unregistered name raises `RubyReactor::RateLimitRegistry::Unknown
449
602
 
450
603
  On contention:
451
604
 
452
- - **Inline** (`Reactor.run`) raises `RubyReactor::Lock::AcquisitionError` / `RubyReactor::Semaphore::AcquisitionError` / `RubyReactor::RateLimit::ExceededError`.
453
- - **Async** (Sidekiq) snoozes the job via `perform_in(delay, ...)`. For rate limits the delay is the error's `retry_after_seconds` (precise wakeup); for locks/semaphores it's `lock_snooze_base_delay + jitter`. Snoozes do not count against the Sidekiq retry budget. After `lock_snooze_max_attempts` snoozes the context is marked failed.
605
+ - **Inline** (`Reactor.run`) raises `RubyReactor::Lock::AcquisitionError` / `RubyReactor::Semaphore::AcquisitionError` / `RubyReactor::RateLimit::ExceededError` / `RubyReactor::OrderedLock::WaitError`.
606
+ - **Async** (Sidekiq) snoozes the job via `perform_in(delay, ...)`. For rate limits the delay uses the error's `retry_after_seconds` hint (precise wakeup — the bucket roll time is known exactly); for locks, semaphores, and ordered-lock waits it's `lock_snooze_base_delay + jitter` (a short re-poll, since a held lock or a live blocker nonce typically clears in milliseconds). Snoozes do not count against the Sidekiq retry budget. After `lock_snooze_max_attempts` snoozes the context is marked failed (ordered-lock waits bypass the cap — see the ordered-lock docs).
454
607
 
455
608
  On dedup hits (period gate already marked), the reactor returns a `RubyReactor::Skipped` result instead — no steps run, no exception:
456
609
 
@@ -472,7 +625,7 @@ step :ensure_active do
472
625
  end
473
626
  ```
474
627
 
475
- See [Locks, Semaphores, Rate Limits & Periods](documentation/locks_and_semaphores.md) for re-entrancy, auto-extend, multi-window quotas, bucket semantics, owner identity, snooze tuning, and operational notes.
628
+ See [Locks, Semaphores, Rate Limits, Periods & Ordered Locks](documentation/locks_and_semaphores.md) for re-entrancy, auto-extend, multi-window quotas, bucket semantics, owner identity, snooze tuning, ordered-lock assignment + poison-pill semantics, and operational notes.
476
629
 
477
630
  ### Map & Parallel Execution
478
631
 
@@ -986,9 +1139,9 @@ Learn how to pause and resume reactors to handle long-running processes, manual
986
1139
  ### [Testing with RSpec](documentation/testing.md)
987
1140
  Comprehensive guide to testing reactors with RubyReactor's testing utilities. Learn about the `TestSubject` class for reactor execution and introspection, step mocking for isolating dependencies, testing nested and composed reactors, and custom RSpec matchers like `be_success`, `have_run_step`, and `have_retried_step`.
988
1141
 
989
- ### [Locks, Semaphores, Rate Limits & Periods](documentation/locks_and_semaphores.md)
1142
+ ### [Locks, Semaphores, Rate Limits, Periods & Ordered Locks](documentation/locks_and_semaphores.md)
990
1143
 
991
- Coordinate access to shared resources across processes with Redis-backed primitives: exclusive locks (`with_lock`), concurrency-limiting semaphores (`with_semaphore`), fixed-window rate limits with multi-window quotas (`with_rate_limit`), and calendar-bucketed dedup (`with_period`, returning `Skipped` results). Covers re-entrancy across composed reactors, TTL auto-extend, inline-vs-async contention behavior, smart `retry_after` snoozes for rate limits, snooze tuning, the token-based semaphore safety model, and once-per-day/month/year scheduling patterns.
1144
+ Coordinate access to shared resources across processes with Redis-backed primitives: exclusive locks (`with_lock`), concurrency-limiting semaphores (`with_semaphore`), fixed-window rate limits with multi-window quotas (`with_rate_limit`), calendar-bucketed dedup (`with_period`, returning `Skipped` results), and strict sequential ordering via a monotonically increasing nonce assigned at enqueue (`with_ordered_lock`). Covers re-entrancy across composed reactors, TTL auto-extend, inline-vs-async contention behavior, smart `retry_after` snoozes for rate limits, snooze tuning, the token-based semaphore safety model, once-per-day/month/year scheduling patterns, ordered-lock counter reset on drain, poison-pill timeouts, and deadlock-safe composition rules.
992
1145
 
993
1146
  ### [Middlewares & OpenTelemetry](documentation/middlewares.md)
994
1147
 
@@ -9,12 +9,76 @@ module RubyReactor
9
9
 
10
10
  attr_writer :sidekiq_queue, :sidekiq_retry_count, :logger, :async_router,
11
11
  :lock_snooze_base_delay, :lock_snooze_jitter, :lock_snooze_max_attempts,
12
- :middlewares
12
+ :middlewares, :context_ttl, :context_lock_ttl, :checkpoint_min_interval,
13
+ :sweeper_enabled, :sweeper_interval, :sweeper_limit
13
14
 
14
15
  def sidekiq_queue
15
16
  @sidekiq_queue ||= :default
16
17
  end
17
18
 
19
+ # Retention TTL (seconds) for a stored reactor context. Storage is
20
+ # load-bearing for resume, so this must comfortably exceed the worst-case
21
+ # snooze/retry window. Refreshed on every checkpoint write.
22
+ def context_ttl
23
+ @context_ttl ||= 86_400
24
+ end
25
+
26
+ # Minimum wall-clock seconds between two PER-STEP durable checkpoints within a
27
+ # single worker run. The save-per-step checkpoint (`on_step_complete`) bounds
28
+ # crash re-execution to one step, but re-serializes and re-writes the WHOLE
29
+ # root blob after every Success — O(steps × context_size) writes for a long,
30
+ # large reactor. This throttle coalesces the mid-run intermediate checkpoints:
31
+ # a checkpoint is written only if at least this many seconds have elapsed since
32
+ # the last one. The final terminal/handoff state is ALWAYS persisted (by the
33
+ # run's ensure-save and the pre-enqueue checkpoint), so throttling only affects
34
+ # mid-run granularity. Tradeoff: with interval > 0, a crash may re-run every
35
+ # step completed inside the last interval — safe only when those steps are
36
+ # idempotent or side-effect-free.
37
+ #
38
+ # Default 0 -> checkpoint after EVERY step (strongest guarantee, no coalescing).
39
+ def checkpoint_min_interval
40
+ @checkpoint_min_interval ||= 0
41
+ end
42
+
43
+ # Whether the recovery sweepers run. The host kicks the self-rescheduling
44
+ # chain once (`RubyReactor.start_sweeper!`, e.g. from an initializer); each
45
+ # tick re-checks this flag, so flipping it to false stops the chain at the
46
+ # next tick. Default on: durability is inert without a running sweeper, so
47
+ # recovery must work out of the box.
48
+ def sweeper_enabled
49
+ @sweeper_enabled = true if @sweeper_enabled.nil?
50
+ @sweeper_enabled
51
+ end
52
+
53
+ # Seconds between sweeps. This is the upper bound on recovery latency for a
54
+ # dead worker — lower it for faster recovery, raise it to cut scan load.
55
+ def sweeper_interval
56
+ @sweeper_interval ||= 30
57
+ end
58
+
59
+ # Max contexts/maps inspected per sweep (passed to each sweeper's run_once).
60
+ def sweeper_limit
61
+ @sweeper_limit ||= 1000
62
+ end
63
+
64
+ # TTL (seconds) for the per-context liveness lock (`async:<id>`). Short by
65
+ # design — it is a liveness signal, not retention. A live worker auto-extends
66
+ # it (every ttl/3 s, from a background thread); its absence is the sweeper's
67
+ # "worker died" signal.
68
+ #
69
+ # SAFETY CONSTRAINT: this MUST exceed the longest a single step can run
70
+ # WITHOUT letting the auto-extend thread make progress. Under MRI the
71
+ # extender shares the GIL, so a step that holds the GIL continuously for
72
+ # longer than this TTL (a long CPU-bound pure-Ruby loop, a C extension that
73
+ # never releases the GIL, or a stop-the-world GC pause) lets the lock lapse.
74
+ # A lapsed lock looks "dead" to the sweeper, which may re-enqueue a duplicate
75
+ # that runs CONCURRENTLY with the still-live original — a double-run. I/O-bound
76
+ # steps release the GIL and keep the lock fresh, so the default 60s suits
77
+ # typical workloads; raise it if you run long synchronous CPU-bound steps.
78
+ def context_lock_ttl
79
+ @context_lock_ttl ||= 60
80
+ end
81
+
18
82
  def sidekiq_retry_count
19
83
  @sidekiq_retry_count ||= 3
20
84
  end
@@ -36,7 +100,7 @@ module RubyReactor
36
100
  end
37
101
 
38
102
  def logger
39
- @logger ||= Logger.new($stderr)
103
+ @logger ||= Logger.new($stdout)
40
104
  end
41
105
 
42
106
  def async_router
@@ -19,13 +19,18 @@ module RubyReactor
19
19
 
20
20
  def deserialize(serialized_data)
21
21
  decompressed = decompress_if_needed(serialized_data)
22
- data = JSON.parse(decompressed, symbolize_names: false)
22
+ deserialize_hash(JSON.parse(decompressed, symbolize_names: false))
23
+ rescue JSON::ParserError => e
24
+ raise RubyReactor::Error::DeserializationError, "Failed to parse serialized context: #{e.message}"
25
+ end
23
26
 
27
+ # Deserialize from an already-parsed Hash (e.g. what the storage adapter's
28
+ # `retrieve_context` returns). Lets the rehydrate-by-id worker path avoid a
29
+ # second JSON parse while still schema-validating. Schema validation lives
30
+ # here so both the string and Hash entry points enforce it.
31
+ def deserialize_hash(data)
24
32
  validate_schema_version(data)
25
-
26
33
  Context.deserialize_from_retry(data)
27
- rescue JSON::ParserError => e
28
- raise RubyReactor::Error::DeserializationError, "Failed to parse serialized context: #{e.message}"
29
34
  end
30
35
 
31
36
  # rubocop:disable Metrics/CyclomaticComplexity, Metrics/MethodLength
@@ -41,6 +41,7 @@ module RubyReactor
41
41
  end
42
42
 
43
43
  def build
44
+ warn_if_child_has_ordered_lock!
44
45
  dependencies = extract_dependencies_from_mappings
45
46
 
46
47
  step_config = {
@@ -78,6 +79,25 @@ module RubyReactor
78
79
 
79
80
  private
80
81
 
82
+ # Composed children bypass `Reactor#run`, so `assign_ordered_lock_nonce!`
83
+ # never fires for them — their `with_ordered_lock` declaration is silently
84
+ # ignored. Surface this at class load so users don't expect ordering
85
+ # enforcement that isn't happening. Nested ordered-lock sequences must be
86
+ # invoked as top-level `Reactor.run` to participate.
87
+ def warn_if_child_has_ordered_lock!
88
+ return unless @composed_reactor_class
89
+ return unless @composed_reactor_class.respond_to?(:ordered_lock_config)
90
+ return unless @composed_reactor_class.ordered_lock_config
91
+
92
+ parent_name = @reactor&.name || "<anonymous>"
93
+ child_name = @composed_reactor_class.name || "<anonymous>"
94
+ RubyReactor.configuration.logger.warn(
95
+ "RubyReactor: `with_ordered_lock` on #{child_name} is ignored when " \
96
+ "composed by #{parent_name}##{@name}. Nested ordered-lock sequences " \
97
+ "are independent and must run via top-level `Reactor.run` to be enforced."
98
+ )
99
+ end
100
+
81
101
  def ensure_composed_reactor_class!
82
102
  raise ArgumentError, "No block provided for inline compose" unless @composed_reactor_class
83
103
  end
@@ -8,7 +8,7 @@ module RubyReactor
8
8
  end
9
9
 
10
10
  module ClassMethods
11
- attr_reader :lock_config, :semaphore_config, :period_config, :rate_limit_config
11
+ attr_reader :lock_config, :semaphore_config, :period_config, :rate_limit_config, :ordered_lock_config
12
12
 
13
13
  # Propagate lock/semaphore/period/rate-limit config to subclasses;
14
14
  # without this a subclass of a configured reactor would silently lose
@@ -19,6 +19,7 @@ module RubyReactor
19
19
  subclass.instance_variable_set(:@semaphore_config, @semaphore_config) if @semaphore_config
20
20
  subclass.instance_variable_set(:@period_config, @period_config) if @period_config
21
21
  subclass.instance_variable_set(:@rate_limit_config, @rate_limit_config) if @rate_limit_config
22
+ subclass.instance_variable_set(:@ordered_lock_config, @ordered_lock_config) if @ordered_lock_config
22
23
  end
23
24
 
24
25
  # Configure locking for this reactor
@@ -74,6 +75,45 @@ module RubyReactor
74
75
  }
75
76
  end
76
77
 
78
+ # Configure strict-ordering nonce gating for this reactor. A
79
+ # monotonically increasing nonce is assigned at enqueue time; the
80
+ # worker can only proceed when its nonce equals `last_completed + 1`.
81
+ # Otherwise the worker raises {OrderedLock::WaitError} and the Sidekiq
82
+ # worker snoozes via `perform_in`.
83
+ #
84
+ # Counters reset to 0 once the sequence fully drains (last_completed
85
+ # catches up to next). Re-entrancy is NOT supported — a nested reactor
86
+ # with its own `with_ordered_lock` is an independent sequence.
87
+ #
88
+ # @param poison_pill_timeout [Integer] seconds since the blocker nonce
89
+ # was assigned before the gate auto-advances past it. Protects
90
+ # against permanent head-of-line blocking from a caller that INCRed
91
+ # the counter but crashed before enqueueing.
92
+ # @param ttl [Integer] TTL on the counter keys, refreshed on every
93
+ # assign. Only fully-drained sequences GC themselves.
94
+ # @param strict [Boolean] When true (default), if any nonce in the
95
+ # sequence terminates with a `Failure`, all subsequent nonces are
96
+ # short-circuited with `Skipped(reason: :ordered_lock_chain_failed)`
97
+ # instead of executing. This models "stop the line on the first
98
+ # problem" pipelines (e.g. ledger transactions). When false, the
99
+ # sequence keeps executing every nonce in order regardless of prior
100
+ # failures. The poison state is per-key and clears on full drain. The
101
+ # check only applies to a fresh `execute`; an already-started run
102
+ # that paused (InterruptResult/AsyncResult) completes on resume even
103
+ # if the chain failed in the meantime.
104
+ # @yield [inputs] Block that returns the ordered-lock key string.
105
+ def with_ordered_lock(poison_pill_timeout: OrderedLock::DEFAULT_POISON_PILL_TIMEOUT,
106
+ ttl: OrderedLock::DEFAULT_TTL,
107
+ strict: true,
108
+ &block)
109
+ @ordered_lock_config = {
110
+ poison_pill_timeout: poison_pill_timeout,
111
+ ttl: ttl,
112
+ strict: strict,
113
+ key_proc: block
114
+ }
115
+ end
116
+
77
117
  # Configure rate limiting for this reactor (fixed-window counter).
78
118
  # Pass either a single window via `limit:` + `period:`, or a hash of
79
119
  # windows via `limits:` for layered API quotas.