ruby_reactor 0.8.7 → 0.8.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/.specify/feature.json +1 -1
- data/CHANGELOG.md +59 -2
- data/CLAUDE.md +1 -1
- data/README.md +38 -3
- data/lib/ruby_reactor/dsl/map_builder.rb +20 -1
- data/lib/ruby_reactor/error/recorded_failure.rb +18 -0
- data/lib/ruby_reactor/executor/compensation_manager.rb +21 -1
- data/lib/ruby_reactor/executor.rb +62 -10
- data/lib/ruby_reactor/interrupt_claims.rb +81 -0
- data/lib/ruby_reactor/map/step_rollback.rb +121 -0
- data/lib/ruby_reactor/reactor.rb +210 -95
- data/lib/ruby_reactor/rspec/matchers.rb +49 -0
- data/lib/ruby_reactor/rspec/test_subject.rb +17 -6
- data/lib/ruby_reactor/step/map_step.rb +12 -0
- data/lib/ruby_reactor/storage/adapter.rb +17 -0
- data/lib/ruby_reactor/storage/redis_adapter.rb +42 -0
- data/lib/ruby_reactor/version.rb +1 -1
- data/lib/ruby_reactor/web/api.rb +17 -0
- data/lib/ruby_reactor/web/public/assets/{index-C6ejszCQ.js → index-DxjQlW7Q.js} +8 -8
- data/lib/ruby_reactor/web/public/index.html +1 -1
- data/lib/ruby_reactor/worker.rb +73 -6
- data/specs/010-rollback-follow-ups/checklists/requirements.md +47 -0
- data/specs/010-rollback-follow-ups/contracts/api-surface.md +123 -0
- data/specs/010-rollback-follow-ups/contracts/resume-protocol.md +132 -0
- data/specs/010-rollback-follow-ups/data-model.md +139 -0
- data/specs/010-rollback-follow-ups/plan.md +273 -0
- data/specs/010-rollback-follow-ups/pr-description.md +110 -0
- data/specs/010-rollback-follow-ups/quickstart.md +157 -0
- data/specs/010-rollback-follow-ups/research.md +440 -0
- data/specs/010-rollback-follow-ups/spec.md +489 -0
- data/specs/010-rollback-follow-ups/tasks.md +1022 -0
- data/specs/future_improvements.md +17 -2
- metadata +14 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1d262a9eed4af540e9573901be72e882f2c5e013550d51881dccef9205c6e873
|
|
4
|
+
data.tar.gz: 7a499dd2203c8059bccf9638b621974784593cabe54fddbb88e16036b7c634c5
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 578c3ceef421e1718c8ec48e746feae08e226e085c235b6594ac881ce0aedcf0fea8e684508c966584a6bc803bfa11a4fbf24a0666b065285f1161b8b12505dd
|
|
7
|
+
data.tar.gz: 6ee564153b8a3ce213fddf09e91df37be6b967b5a284c5cd0cdbbc4bca8b68ae4a5b3b064c314492f6f53b1bfde4ae29173cef2439886064d10c742c677cc128
|
data/.specify/feature.json
CHANGED
data/CHANGELOG.md
CHANGED
|
@@ -239,6 +239,25 @@ Every breaking or shape-changing item of the rollback work, with what to change.
|
|
|
239
239
|
9. **`rollback_failures` entries may carry `map_step:` / `element_index:`** and the reasons
|
|
240
240
|
`:context_unavailable` / `:element_in_flight` (additive).
|
|
241
241
|
|
|
242
|
+
### Migration notes: rollback and resume follow-ups
|
|
243
|
+
|
|
244
|
+
1. **A resume that meets the reactor's held `with_lock` or `with_semaphore` no longer raises.**
|
|
245
|
+
`continue` used to raise `Lock::AcquisitionError` / `Semaphore::AcquisitionError` and leave
|
|
246
|
+
the run paused. It now validates the payload, accepts the resume, and returns a
|
|
247
|
+
`RubyReactor::DispatchResult`; a worker waits for the lock and finishes the run. A `rescue` of
|
|
248
|
+
those errors around `continue` simply stops firing. An invalid payload is still answered at
|
|
249
|
+
once, as before.
|
|
250
|
+
2. **A second resume of the same interrupt raises `ValidationError`** ("Cannot resume: interrupt
|
|
251
|
+
:x was already resumed"), in every execution mode, including `Sidekiq::Testing.inline!`.
|
|
252
|
+
3. **A resume for another ready interrupt of a `running` run is accepted** (it used to raise "the
|
|
253
|
+
reactor is running") and returns a `DispatchResult`.
|
|
254
|
+
4. **Interrupt attempt counts restart** for runs paused across the upgrade: they move out of the
|
|
255
|
+
run's context into their own record, so such a run may take up to `max_attempts - 1` extra
|
|
256
|
+
invalid payloads.
|
|
257
|
+
5. **A synchronous `Reactor.run` holds the run's liveness lock** while it executes. A
|
|
258
|
+
`Reactor.undo(id)` of a run still executing in its caller's process now waits for it, then
|
|
259
|
+
raises `Lock::AcquisitionError`, as it does for a run executing in a worker.
|
|
260
|
+
|
|
242
261
|
### Features
|
|
243
262
|
|
|
244
263
|
* **An `interrupt` inside a composed child pauses the top-level run.** Before, the compose step
|
|
@@ -248,8 +267,8 @@ Every breaking or shape-changing item of the rollback work, with what to change.
|
|
|
248
267
|
then the interrupt (`continue(id:, payload:, step_name: [:approval, :wait_for_manager])`), by id
|
|
249
268
|
or by the child interrupt's correlation id. Payload validation, `max_attempts`,
|
|
250
269
|
`resume: :background` and the resume guards apply as for a top-level interrupt. A resume
|
|
251
|
-
contended on the child's own `with_lock`/`with_semaphore` fails and rolls back the run.
|
|
252
|
-
|
|
270
|
+
contended on the child's own `with_lock`/`with_semaphore` fails and rolls back the run. Contention
|
|
271
|
+
on the top-level reactor's lock hands the resume to a worker. `Reactor.undo`
|
|
253
272
|
of the paused run rolls back the child's completed steps, then the parent's. A composed child
|
|
254
273
|
refuses a `continue` of its own. New `Reactor#ready_interrupt_steps` lists the pending
|
|
255
274
|
interrupts (Symbols, and paths for nested ones). `be_paused_at`, `have_ready_interrupts` and
|
|
@@ -258,6 +277,19 @@ Every breaking or shape-changing item of the rollback work, with what to change.
|
|
|
258
277
|
`ValidationError`. An `interrupt` inside a `map` element now fails that element with "not
|
|
259
278
|
supported inside a map element" instead of `NoMethodError`.
|
|
260
279
|
See [Interrupts inside composed reactors](documentation/interrupts.md#interrupts-inside-composed-reactors).
|
|
280
|
+
* **`undo_all` on a map: one call rolls back every completed element.** A map declaring
|
|
281
|
+
`undo_all { |completed_results| ... }` is rolled back (an atomic map's failure, a later step's
|
|
282
|
+
failure, or `Reactor.undo`) by calling the block once with a lazy, index-ordered Enumerable of
|
|
283
|
+
the completed elements' results, instead of replaying each element's own undos. A fan-out map
|
|
284
|
+
reads its stored results and dispatches no element rollback jobs; nothing is held in memory all
|
|
285
|
+
at once. Failed elements still roll themselves back; a block that raises or returns a
|
|
286
|
+
`Failure` is reported as one rollback failure (`kind: :undo_all`) and the steps before the map
|
|
287
|
+
are still undone. New matcher: `have_run_undo_all(:map).with_elements(n)`.
|
|
288
|
+
* **A resume contended on the reactor's lock or semaphore is accepted, not lost** (see migration
|
|
289
|
+
note 1). New test helpers: `resume(..., process_jobs: false)` and the `be_resume_deferred`
|
|
290
|
+
matcher.
|
|
291
|
+
* **Several interrupts can be resumed at once** (see migration note 3): each accepted resume is
|
|
292
|
+
applied once.
|
|
261
293
|
* **`fan_out` without `batch_size` is back-pressured: at most 50 element jobs per throw**
|
|
262
294
|
(`RubyReactor::Map::DEFAULT_BATCH_SIZE`), forward and rollback. Before, every element was
|
|
263
295
|
enqueued at once. A map of more than 50 elements with no declared `batch_size` now runs in
|
|
@@ -359,6 +391,24 @@ Every breaking or shape-changing item of the rollback work, with what to change.
|
|
|
359
391
|
|
|
360
392
|
### Bug Fixes
|
|
361
393
|
|
|
394
|
+
* **The recovery sweep no longer re-runs a run still executing in its caller's process.** A
|
|
395
|
+
synchronous `Reactor.run` (or an inline `continue`) now holds the run's liveness lock, renewed
|
|
396
|
+
while it executes, so a sweep during a long step leaves it alone; a killed process is still
|
|
397
|
+
recovered once the lock lapses (`context_lock_ttl`).
|
|
398
|
+
* **A caller's final save no longer overwrites a worker's progress.** A synchronous run that hands
|
|
399
|
+
off at a fan-out map (or a `background` step) keeps its lock until its final save, and a Worker
|
|
400
|
+
now takes the run's lock (waiting up to 2s) before it reads the run, so it always resumes from
|
|
401
|
+
the caller's final state.
|
|
402
|
+
* **Two resumes of one interrupt at the same instant: exactly one is accepted**, even in inline
|
|
403
|
+
job-testing mode, where both used to run.
|
|
404
|
+
* **A resumed run waiting on a reactor lock or semaphore is no longer marked `failed` without
|
|
405
|
+
rolling back.** The snooze limit (`lock_snooze_max_attempts`) now applies only before a run is
|
|
406
|
+
admitted; an admitted run (a deferred or `resume: :background` resume) keeps waiting and logs
|
|
407
|
+
`ruby_reactor.resume.waiting` once.
|
|
408
|
+
* **Manual undo finishes a `compensate` an interruption cut off.** A run aborted while its failing
|
|
409
|
+
step's `compensate` ran now records that step; `Reactor.undo(id)` runs the `compensate` again,
|
|
410
|
+
with the same arguments and reason, before undoing the completed steps (also inside composed
|
|
411
|
+
children and inline map elements). The dashboard flags such a run.
|
|
362
412
|
* **A fan-out map inside a composed reactor no longer leaves the root running forever.** The map's
|
|
363
413
|
completion resumed the child as a run of its own, and nothing resumed the root. Now the root
|
|
364
414
|
resumes and finishes, and a failure of the map or of any later step rolls back through the root.
|
|
@@ -461,6 +511,13 @@ Every breaking or shape-changing item of the rollback work, with what to change.
|
|
|
461
511
|
* Docs: a park keeps each level's lock without a second `:lock_acquired` only while the gap stays
|
|
462
512
|
within the lock's `ttl`; a lapsed lock is acquired again.
|
|
463
513
|
|
|
514
|
+
## [0.8.8](https://github.com/arturictus/ruby_reactor/compare/v0.8.7...v0.8.8) (2026-10-10)
|
|
515
|
+
|
|
516
|
+
|
|
517
|
+
### Bug Fixes
|
|
518
|
+
|
|
519
|
+
* Rollback and Resume Follow-ups ([#72](https://github.com/arturictus/ruby_reactor/issues/72)) ([251fe9f](https://github.com/arturictus/ruby_reactor/commit/251fe9f6a9499738b6abcd6befd3a99e7cebec9f))
|
|
520
|
+
|
|
464
521
|
## [0.8.7](https://github.com/arturictus/ruby_reactor/compare/v0.8.6...v0.8.7) (2026-10-08)
|
|
465
522
|
|
|
466
523
|
|
data/CLAUDE.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
<!-- SPECKIT START -->
|
|
2
2
|
For additional context about technologies to be used, project structure,
|
|
3
3
|
shell commands, and other important information, read the current plan
|
|
4
|
-
at specs/010-
|
|
4
|
+
at specs/010-rollback-follow-ups/plan.md
|
|
5
5
|
<!-- SPECKIT END -->
|
data/README.md
CHANGED
|
@@ -595,7 +595,12 @@ any background job is enqueued the root context is persisted, and after every
|
|
|
595
595
|
completed step a checkpoint advances the stored blob — so a crash re-runs at most
|
|
596
596
|
one step, never the whole reactor. Each running reactor also holds a short
|
|
597
597
|
**liveness lock** that a live worker auto-extends; its absence is how a dead
|
|
598
|
-
worker is detected.
|
|
598
|
+
worker is detected. A run executing in the caller's process (a synchronous
|
|
599
|
+
`Reactor.run`, an inline `continue`) holds the same lock while it executes, so a
|
|
600
|
+
sweep never runs it a second time, however long a step takes; if that process is
|
|
601
|
+
killed, the lock lapses within `context_lock_ttl` and the run is recovered like a
|
|
602
|
+
worker's. A worker takes the run's lock before it reads the run, so it always
|
|
603
|
+
resumes from the last save of whoever held it.
|
|
599
604
|
|
|
600
605
|
**Recovery is not automatic until you start the sweeper.** A crashed worker's
|
|
601
606
|
reactor only resumes when the recovery sweeper notices the lapsed liveness lock
|
|
@@ -707,6 +712,12 @@ ApprovalReactor.continue_by_correlation_id(
|
|
|
707
712
|
An interrupt inside a `compose`d child pauses the top-level run; resume it there by the step path,
|
|
708
713
|
e.g. `step_name: [:approval, :wait_for_manager]`. See
|
|
709
714
|
[Interrupts inside composed reactors](documentation/interrupts.md#interrupts-inside-composed-reactors).
|
|
715
|
+
`continue` validates the payload in the calling process first: an invalid one is answered with its
|
|
716
|
+
validation failure and nothing is stored. A valid one claims its interrupt, so of two resumes of the
|
|
717
|
+
same interrupt exactly one is accepted (the other raises "already resumed"). When the run is busy,
|
|
718
|
+
another interrupt's resume is still running, or the reactor's `with_lock`/`with_semaphore` is held,
|
|
719
|
+
the accepted resume is handed to a worker and `continue` returns a `DispatchResult` instead of
|
|
720
|
+
raising, so a webhook's resume is never lost. See [Resuming Execution](documentation/interrupts.md#resuming-execution).
|
|
710
721
|
|
|
711
722
|
### Locks, Semaphores & Ordered Locks
|
|
712
723
|
|
|
@@ -718,6 +729,8 @@ Coordinate across processes with Redis-backed primitives:
|
|
|
718
729
|
- **`with_period`** — run at most once per calendar bucket (dedup / once-per-day, once-per-month, etc).
|
|
719
730
|
- **`with_ordered_lock`** — strict transaction ordering via a monotonically increasing nonce assigned at enqueue. Workers can only proceed when their nonce equals `last_completed + 1`.
|
|
720
731
|
|
|
732
|
+
An interrupt's resume that meets the reactor's held lock or semaphore is accepted and deferred to a worker, which waits for it (`continue` returns a `DispatchResult`); a run already admitted is never failed by the snooze limit for waiting.
|
|
733
|
+
|
|
721
734
|
```ruby
|
|
722
735
|
class RefundOrderReactor < RubyReactor::Reactor
|
|
723
736
|
input :order_id
|
|
@@ -903,8 +916,8 @@ A `fan_out` map is a **hand-off point**: the reactor stops at the map (the calle
|
|
|
903
916
|
|
|
904
917
|
By using `fan_out` with `batch_size`, the system applies **Back Pressure** to efficiently manage resources. [Read more about Back Pressure & Resource Management](documentation/data_pipelines.md#back-pressure--resource-management).
|
|
905
918
|
|
|
906
|
-
**Rollback.** A map rolls back like a composed reactor
|
|
907
|
-
|
|
919
|
+
**Rollback.** A map rolls back like a composed reactor: the `undo`s already declared on the element
|
|
920
|
+
reactor's steps are each element's rollback, unless the map declares `undo_all` (below). When the map fails
|
|
908
921
|
(an element fails in an `atomic` map, the default, or `collect` raises), and when a later step fails or the run is
|
|
909
922
|
undone manually, every element that completed is rolled back, newest-started first, and the run ends
|
|
910
923
|
with the same `Failure` in inline and fan-out mode alike. A map rolls back the way it ran: an inline
|
|
@@ -915,6 +928,28 @@ failing atomic fan-out map lets elements already in flight finish before it repo
|
|
|
915
928
|
element `undo`s idempotent: one cut off by a killed worker runs again. See
|
|
916
929
|
[Rollback](documentation/data_pipelines.md#rollback).
|
|
917
930
|
|
|
931
|
+
**One bulk undo.** When an API can roll back many elements in one call (a bulk refund), declare
|
|
932
|
+
`undo_all` on the map. Every rollback of the map then calls it **once** with the completed elements'
|
|
933
|
+
results, instead of replaying each element's undos:
|
|
934
|
+
|
|
935
|
+
```ruby
|
|
936
|
+
map :charges, ChargeReactor do
|
|
937
|
+
source input(:payments)
|
|
938
|
+
argument :payment, element(:charges)
|
|
939
|
+
fan_out batch_size: 100
|
|
940
|
+
|
|
941
|
+
# A lazy Enumerable of the completed elements' results, in index order.
|
|
942
|
+
undo_all do |charges|
|
|
943
|
+
Payments.bulk_refund(charges.map { |charge| charge[:id] }.to_a)
|
|
944
|
+
end
|
|
945
|
+
end
|
|
946
|
+
```
|
|
947
|
+
|
|
948
|
+
A fan-out map dispatches no element rollback jobs for it, and nothing holds every result at once.
|
|
949
|
+
Failed elements still roll themselves back; a block that raises or returns a `Failure` is reported as
|
|
950
|
+
one rollback failure (`kind: :undo_all`), and the steps before the map are still undone. See
|
|
951
|
+
[`undo_all`](documentation/data_pipelines.md#undo_all-one-bulk-rollback).
|
|
952
|
+
|
|
918
953
|
`batch_size` is optional: with `fan_out` alone, no throw enqueues more than 50 element jobs (`RubyReactor::Map::DEFAULT_BATCH_SIZE`), forward and rollback, and the outcomes are aggregated into a `ResultEnumerator`. Set `batch_size` to change the throw size. See [`fan_out` Without `batch_size`](documentation/data_pipelines.md#fan_out-without-batch_size).
|
|
919
954
|
|
|
920
955
|
> **Breaking change:** `async true` inside a `map` block has been **removed** — it read like `async_step`/`async_reactor`, which dispatch independent units the reactor does not stop for. It now raises at class-definition time. The exact replacement is `fan_out` (`fan_out batch_size: N`).
|
|
@@ -84,6 +84,19 @@ module RubyReactor
|
|
|
84
84
|
@collect_block = block
|
|
85
85
|
end
|
|
86
86
|
|
|
87
|
+
# One call that rolls back every completed element (010 US7), instead of
|
|
88
|
+
# replaying each element's own undos: a bulk refund, a batch delete. The
|
|
89
|
+
# block gets a lazy Enumerable of the completed elements' results, in
|
|
90
|
+
# index order, on every rollback of the map.
|
|
91
|
+
def undo_all(&block)
|
|
92
|
+
raise RubyReactor::Error::ValidationError.new("map :#{@name} undo_all needs a block", step: @name) unless block
|
|
93
|
+
if @undo_all_block
|
|
94
|
+
raise RubyReactor::Error::ValidationError.new("map :#{@name} declares undo_all twice", step: @name)
|
|
95
|
+
end
|
|
96
|
+
|
|
97
|
+
@undo_all_block = block
|
|
98
|
+
end
|
|
99
|
+
|
|
87
100
|
# On (the default), the map succeeds only if every element does: the
|
|
88
101
|
# first element failure fails it, no new element starts, and every
|
|
89
102
|
# completed element is rolled back. Off, it completes with every
|
|
@@ -144,7 +157,8 @@ module RubyReactor
|
|
|
144
157
|
batch_size: { source: RubyReactor::Template::Value.new(@batch_size) },
|
|
145
158
|
collect_block: { source: RubyReactor::Template::Value.new(@collect_block) },
|
|
146
159
|
atomic: { source: RubyReactor::Template::Value.new(@atomic) },
|
|
147
|
-
fan_out: { source: RubyReactor::Template::Value.new(@fan_out) }
|
|
160
|
+
fan_out: { source: RubyReactor::Template::Value.new(@fan_out) },
|
|
161
|
+
**undo_all_argument
|
|
148
162
|
},
|
|
149
163
|
run_block: nil,
|
|
150
164
|
compensate_block: nil,
|
|
@@ -155,6 +169,11 @@ module RubyReactor
|
|
|
155
169
|
}
|
|
156
170
|
end
|
|
157
171
|
|
|
172
|
+
# Read at rollback from this declaration (`MapStep#undo_all_block`).
|
|
173
|
+
def undo_all_argument
|
|
174
|
+
@undo_all_block ? { undo_all_block: { source: RubyReactor::Template::Value.new(@undo_all_block) } } : {}
|
|
175
|
+
end
|
|
176
|
+
|
|
158
177
|
def extract_dependencies_from_mappings
|
|
159
178
|
dependencies = []
|
|
160
179
|
@argument_mappings.each_value do |source|
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RubyReactor
|
|
4
|
+
module Error
|
|
5
|
+
# The reason a cut-off `compensate` receives when a manual undo runs it
|
|
6
|
+
# again (010 R-10): the run was aborted while it ran, and the original
|
|
7
|
+
# exception cannot be rebuilt from storage. It carries that exception's
|
|
8
|
+
# message, and its class name as `original_class`.
|
|
9
|
+
class RecordedFailure < Base
|
|
10
|
+
attr_reader :original_class
|
|
11
|
+
|
|
12
|
+
def initialize(message, original_class:)
|
|
13
|
+
super(message)
|
|
14
|
+
@original_class = original_class
|
|
15
|
+
end
|
|
16
|
+
end
|
|
17
|
+
end
|
|
18
|
+
end
|
|
@@ -51,8 +51,22 @@ module RubyReactor
|
|
|
51
51
|
@context.undo_stack << step_info
|
|
52
52
|
end
|
|
53
53
|
|
|
54
|
+
# A failing step whose `compensate` was cut off by an interruption, as
|
|
55
|
+
# an aborted run keeps it for manual undo (010 R-10, DM §4); nil once that
|
|
56
|
+
# `compensate` returned. The `arguments` key marks the record: 009's
|
|
57
|
+
# hand-off states carry none.
|
|
58
|
+
def pending_record
|
|
59
|
+
return unless @pending && !@pending[:compensated]
|
|
60
|
+
|
|
61
|
+
{ "trigger" => "failure", "step" => @pending[:step].to_s, "compensated" => false,
|
|
62
|
+
"arguments" => ContextSerializer.serialize_value(@pending[:arguments] || {}),
|
|
63
|
+
"error" => recorded_error(@pending[:error]),
|
|
64
|
+
"failures" => ContextSerializer.serialize_value(@rollback_failures) }
|
|
65
|
+
end
|
|
66
|
+
|
|
54
67
|
def handle_step_failure(step_config, error, arguments)
|
|
55
|
-
@pending = { step: step_config.name, error: error, compensated: false
|
|
68
|
+
@pending = { step: step_config.name, error: error, compensated: false,
|
|
69
|
+
arguments: step_config.rollback_arguments(arguments) }
|
|
56
70
|
# A step whose OWN coordination acquisition failed (contention, or a
|
|
57
71
|
# bad key proc) never ran its body — "no step compensates, the
|
|
58
72
|
# contended step's work has not been attempted" (US3-1/T018), which
|
|
@@ -121,6 +135,12 @@ module RubyReactor
|
|
|
121
135
|
|
|
122
136
|
private
|
|
123
137
|
|
|
138
|
+
def recorded_error(error)
|
|
139
|
+
return error if error.is_a?(String)
|
|
140
|
+
|
|
141
|
+
{ "class" => error.class.name, "message" => error.respond_to?(:message) ? error.message : error.to_s }
|
|
142
|
+
end
|
|
143
|
+
|
|
124
144
|
def middlewares
|
|
125
145
|
@context.middlewares || RubyReactor::MiddlewareRunner.new([])
|
|
126
146
|
end
|
|
@@ -21,6 +21,12 @@ module RubyReactor
|
|
|
21
21
|
attr_reader :reactor_class, :context, :dependency_graph, :compensation_manager, :retry_manager, :result_handler,
|
|
22
22
|
:step_executor, :result, :middlewares
|
|
23
23
|
|
|
24
|
+
# The owner of a run lock (`async:<id>`) the caller already holds: a Worker
|
|
25
|
+
# or `continue` that locked the run before loading it (010 R-02, R-03).
|
|
26
|
+
# `acquire_context_lock` then re-enters it (the lock counts by owner), and
|
|
27
|
+
# the outer holder releases last.
|
|
28
|
+
attr_writer :context_lock_owner
|
|
29
|
+
|
|
24
30
|
def initialize(reactor_class, inputs = {}, context = nil)
|
|
25
31
|
# Resume, map, compose and background workers build an Executor without
|
|
26
32
|
# going through Reactor#run; the inferred wiring must exist there too.
|
|
@@ -109,6 +115,12 @@ module RubyReactor
|
|
|
109
115
|
input_validator = InputValidator.new(@reactor_class, @context)
|
|
110
116
|
input_validator.validate!
|
|
111
117
|
|
|
118
|
+
# A run in the caller's process holds its liveness lock too (010 R-01),
|
|
119
|
+
# so the sweeper never mistakes it for a dead worker's run and a Worker
|
|
120
|
+
# resumed by its hand-off waits for its final save. In a worker the
|
|
121
|
+
# executor runs `resume_execution`, which takes it there.
|
|
122
|
+
acquire_context_lock unless @context.inline_async_execution
|
|
123
|
+
|
|
112
124
|
reset_held_lock_keys!
|
|
113
125
|
acquire_locks_with_telemetry
|
|
114
126
|
|
|
@@ -168,6 +180,10 @@ module RubyReactor
|
|
|
168
180
|
release_locks unless @parked
|
|
169
181
|
leave_ordered_lock_scope
|
|
170
182
|
save_context if persist_context? && !skip_context_persist?
|
|
183
|
+
# Released only after that save (J-3, 009 R-13): a Worker waiting on the
|
|
184
|
+
# lock then reads this run's final state.
|
|
185
|
+
@acquired_context_lock&.release
|
|
186
|
+
@acquired_context_lock = nil
|
|
171
187
|
|
|
172
188
|
emit_lifecycle_completion(completed)
|
|
173
189
|
end
|
|
@@ -221,6 +237,10 @@ module RubyReactor
|
|
|
221
237
|
# holds it — composed/nested children resume inline under the root worker
|
|
222
238
|
# and must not contend on the root's own key.
|
|
223
239
|
acquire_context_lock
|
|
240
|
+
# The only place a claimed resume payload enters a context (010 R-04,
|
|
241
|
+
# J-5): under the run's lock, before the reactor-level lock or
|
|
242
|
+
# semaphore, so a contended resume still saves the payload it applied.
|
|
243
|
+
InterruptClaims.apply!(@context) if (@context.root_context || @context).equal?(@context)
|
|
224
244
|
|
|
225
245
|
reset_held_lock_keys!
|
|
226
246
|
|
|
@@ -249,7 +269,6 @@ module RubyReactor
|
|
|
249
269
|
@context.admit!
|
|
250
270
|
prepare_for_resume
|
|
251
271
|
save_context
|
|
252
|
-
@past_gates = true
|
|
253
272
|
|
|
254
273
|
@result = if @context.current_step
|
|
255
274
|
execute_current_step_and_continue
|
|
@@ -313,6 +332,7 @@ module RubyReactor
|
|
|
313
332
|
def undo_all(&after_pop)
|
|
314
333
|
saved = @context.rollback && @context.rollback["failures"]
|
|
315
334
|
@compensation_manager.restore_rollback_failures(saved) if saved
|
|
335
|
+
compensate_pending!
|
|
316
336
|
@compensation_manager.rollback_completed_steps(&after_pop)
|
|
317
337
|
clear_saved_rollback_failures
|
|
318
338
|
rescue Error::RollbackHandedOff
|
|
@@ -390,12 +410,6 @@ module RubyReactor
|
|
|
390
410
|
RubyReactor.reactor_storage_name(@reactor_class))
|
|
391
411
|
end
|
|
392
412
|
|
|
393
|
-
# True once `resume_execution` is past its reactor-level gates (context
|
|
394
|
-
# lock, lock, semaphore, period): from there on it may have run steps.
|
|
395
|
-
def past_gates?
|
|
396
|
-
@past_gates == true
|
|
397
|
-
end
|
|
398
|
-
|
|
399
413
|
# Reached only by an interruption — a signal, an exit, out of memory, an
|
|
400
414
|
# enclosing timeout (008 R-08, R-16); every other exception was rescued as
|
|
401
415
|
# `Error::Rescuable` and rolled back. Running user rollback code now is
|
|
@@ -404,7 +418,13 @@ module RubyReactor
|
|
|
404
418
|
# `Reactor#undo`; the `ensure` persists it, best effort. A worker run stays
|
|
405
419
|
# `running` and its job is redelivered.
|
|
406
420
|
def mark_aborted
|
|
407
|
-
|
|
421
|
+
return if @context.inline_async_execution
|
|
422
|
+
|
|
423
|
+
@context.status = :aborted
|
|
424
|
+
# The failing step's `compensate`, if the interruption cut it off: a
|
|
425
|
+
# manual undo runs it again before the undo stack (010 R-10, J-9).
|
|
426
|
+
record = @compensation_manager.pending_record
|
|
427
|
+
@context.rollback = record if record
|
|
408
428
|
end
|
|
409
429
|
|
|
410
430
|
# A rollback run from a `rescue Error::Rescuable` body is outside the
|
|
@@ -663,6 +683,8 @@ module RubyReactor
|
|
|
663
683
|
# defeat the guard. Only the root executor acquires — a composed/nested child
|
|
664
684
|
# resumes inline under the root worker and shares the root's lock, so it must
|
|
665
685
|
# not try to re-acquire the same key with a different owner (self-deadlock).
|
|
686
|
+
# Held by a worker's resume and by a run in the caller's process alike (010
|
|
687
|
+
# R-01); an outer holder's owner, when set, is re-entered (R-03).
|
|
666
688
|
def acquire_context_lock
|
|
667
689
|
root = @context.root_context || @context
|
|
668
690
|
return unless root.equal?(@context) # only the root executor holds it
|
|
@@ -980,7 +1002,7 @@ module RubyReactor
|
|
|
980
1002
|
# gives (I-7).
|
|
981
1003
|
def finish_rollback(state)
|
|
982
1004
|
@compensation_manager.restore_rollback_failures(state["failures"])
|
|
983
|
-
return finish_undo if state["trigger"] == "undo"
|
|
1005
|
+
return finish_undo(state) if state["trigger"] == "undo"
|
|
984
1006
|
|
|
985
1007
|
failure = ContextSerializer.deserialize_value(state["failure"])
|
|
986
1008
|
step_config = state["step"] && @reactor_class.steps[state["step"].to_sym]
|
|
@@ -998,8 +1020,38 @@ module RubyReactor
|
|
|
998
1020
|
@result_handler.handle_execution_error(e)
|
|
999
1021
|
end
|
|
1000
1022
|
|
|
1001
|
-
|
|
1023
|
+
# A manual undo ends `cancelled`, or `failed` when it was the failure of
|
|
1024
|
+
# an interrupt whose payload attempts ran out (010 R-08).
|
|
1025
|
+
# An aborted run's failing step whose `compensate` an interruption cut off
|
|
1026
|
+
# (010 R-10, J-9): run it again, with the recorded arguments and reason,
|
|
1027
|
+
# before the undo stack. Only that record carries `arguments`; 009's
|
|
1028
|
+
# hand-off states, which name the same keys, are finished elsewhere.
|
|
1029
|
+
def compensate_pending!
|
|
1030
|
+
state = @context.rollback
|
|
1031
|
+
return unless state.is_a?(Hash) && state.key?("arguments") && state["compensated"] == false
|
|
1032
|
+
|
|
1033
|
+
step_config = state["step"] && @reactor_class.steps[state["step"].to_sym]
|
|
1034
|
+
return unless step_config
|
|
1035
|
+
|
|
1036
|
+
@compensation_manager.compensate(step_config, recorded_reason(state["error"]),
|
|
1037
|
+
ContextSerializer.deserialize_value(state["arguments"]))
|
|
1038
|
+
state["compensated"] = true
|
|
1039
|
+
end
|
|
1040
|
+
|
|
1041
|
+
def recorded_reason(error)
|
|
1042
|
+
return error unless error.is_a?(Hash)
|
|
1043
|
+
|
|
1044
|
+
Error::RecordedFailure.new(error["message"], original_class: error["class"])
|
|
1045
|
+
end
|
|
1046
|
+
|
|
1047
|
+
def finish_undo(state)
|
|
1002
1048
|
@compensation_manager.rollback_completed_steps
|
|
1049
|
+
if state["failure_reason"]
|
|
1050
|
+
@context.status = "failed"
|
|
1051
|
+
@context.failure_reason = state["failure_reason"]
|
|
1052
|
+
return nil
|
|
1053
|
+
end
|
|
1054
|
+
|
|
1003
1055
|
@context.cancelled = true
|
|
1004
1056
|
@context.cancellation_reason = "Undo triggered"
|
|
1005
1057
|
@context.status = "cancelled"
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RubyReactor
|
|
4
|
+
# A resume claims its interrupt before anything runs (010 R-04): the first
|
|
5
|
+
# `SET NX` wins, in every execution mode, and the claim holds the validated
|
|
6
|
+
# payload. Only the execution that owns the run's lock copies a claim into
|
|
7
|
+
# the context (J-5), at the start of `Executor#resume_execution`, so a
|
|
8
|
+
# resume never writes the run from outside it.
|
|
9
|
+
module InterruptClaims
|
|
10
|
+
module_function
|
|
11
|
+
|
|
12
|
+
# true when this call won the interrupt; false when a resume already did.
|
|
13
|
+
# `path` is the interrupt's step path (010 R-05); a nested one is keyed `a.b`.
|
|
14
|
+
def claim!(context, path, payload)
|
|
15
|
+
storage.claim_interrupt_resume(context.context_id, storage_name(context), key(Array(path)),
|
|
16
|
+
JSON.generate(ContextSerializer.serialize_value(payload)))
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
# true when a resume already claimed the interrupt, applied or not.
|
|
20
|
+
def claimed?(context, path)
|
|
21
|
+
storage.retrieve_interrupt_resumes(context.context_id, storage_name(context), [key(Array(path))]).any?
|
|
22
|
+
end
|
|
23
|
+
|
|
24
|
+
# `{ step => payload }` (a nested interrupt: `{ [child, step] => payload }`)
|
|
25
|
+
# for the claimed interrupts that have no result yet.
|
|
26
|
+
def unapplied(context)
|
|
27
|
+
paths = pending_interrupts(context)
|
|
28
|
+
return {} if paths.empty?
|
|
29
|
+
|
|
30
|
+
raw = storage.retrieve_interrupt_resumes(context.context_id, storage_name(context), paths.map { |p| key(p) })
|
|
31
|
+
paths.filter_map do |path|
|
|
32
|
+
if raw.key?(key(path))
|
|
33
|
+
[path.one? ? path.first : path,
|
|
34
|
+
ContextSerializer.deserialize_value(JSON.parse(raw[key(path)]))]
|
|
35
|
+
end
|
|
36
|
+
end.to_h
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
# Copies every unapplied claim into the context that paused at it (010 R-06).
|
|
40
|
+
# Returns their steps.
|
|
41
|
+
def apply!(context)
|
|
42
|
+
claims = unapplied(context)
|
|
43
|
+
claims.each do |step, payload|
|
|
44
|
+
path = Array(step)
|
|
45
|
+
path[0...-1].reduce(context) { |ctx, name| child(ctx, name) }.set_result(path.last, payload)
|
|
46
|
+
end
|
|
47
|
+
claims.keys
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
# Step paths of the interrupts without a result, here and in paused composed children.
|
|
51
|
+
def pending_interrupts(context, prefix = [])
|
|
52
|
+
steps = context.reactor_class.respond_to?(:steps) ? context.reactor_class.steps : {}
|
|
53
|
+
interrupts = steps.select do |name, config|
|
|
54
|
+
config.respond_to?(:interrupt?) && config.interrupt? && !context.has_result?(name)
|
|
55
|
+
end
|
|
56
|
+
own = interrupts.keys.map { |name| prefix + [name] }
|
|
57
|
+
nested = steps.keys.flat_map do |name|
|
|
58
|
+
(c = child(context, name)) ? pending_interrupts(c, prefix + [name]) : []
|
|
59
|
+
end
|
|
60
|
+
own + nested
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def child(context, name)
|
|
64
|
+
entry = context.composed_contexts[name] || context.composed_contexts[name.to_s]
|
|
65
|
+
c = entry.is_a?(Hash) && (entry[:context] || entry["context"])
|
|
66
|
+
c if c.is_a?(Context) && c.status.to_s == "paused"
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def key(path)
|
|
70
|
+
path.join(".")
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def storage_name(context)
|
|
74
|
+
RubyReactor.reactor_storage_name(context.reactor_class)
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def storage
|
|
78
|
+
RubyReactor.configuration.storage_adapter
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
end
|
|
@@ -115,6 +115,127 @@ module RubyReactor
|
|
|
115
115
|
missing.map { |index| rollback_entry(index, :context_unavailable, "context expired") }
|
|
116
116
|
end
|
|
117
117
|
|
|
118
|
+
# 010 R-13 (P-6): one call to the map's `undo_all` block, with the
|
|
119
|
+
# completed elements' results (index order, read lazily, never all at
|
|
120
|
+
# once), instead of replaying each element. A fan-out map reads its
|
|
121
|
+
# result slots and dispatches no rollback jobs; an inline map reads its
|
|
122
|
+
# element contexts one at a time, and an element an interruption left
|
|
123
|
+
# `aborted` still replays its own completed steps (it has no result).
|
|
124
|
+
def bulk_rollback(map_id, step_name, block)
|
|
125
|
+
failures, count, results =
|
|
126
|
+
dispatched_map_id(step_name) ? fan_out_completed(map_id) : inline_completed(map_id, step_name)
|
|
127
|
+
failures << call_undo_all(block, results, step_name, count) if count.positive?
|
|
128
|
+
finish_rollback(step_name, count, failures.compact)
|
|
129
|
+
end
|
|
130
|
+
|
|
131
|
+
def fan_out_completed(map_id)
|
|
132
|
+
meta = rollback_storage.retrieve_map_metadata(map_id, context.reactor_class.name)
|
|
133
|
+
return [[rollback_entry(nil, :context_unavailable, "map records expired")], 0, []] unless meta
|
|
134
|
+
|
|
135
|
+
total = meta["count"].to_i
|
|
136
|
+
failures = []
|
|
137
|
+
count = 0
|
|
138
|
+
each_result_slot(map_id, total) do |index, raw|
|
|
139
|
+
next failures << rollback_entry(index, :context_unavailable, "context expired") if raw.nil?
|
|
140
|
+
|
|
141
|
+
count += 1 if completed_slot?(raw)
|
|
142
|
+
end
|
|
143
|
+
results = Enumerator.new do |yielder|
|
|
144
|
+
each_result_slot(map_id, total) do |_index, raw|
|
|
145
|
+
yielder << ContextSerializer.deserialize_value(raw) if raw && completed_slot?(raw)
|
|
146
|
+
end
|
|
147
|
+
end
|
|
148
|
+
[failures, count, results.lazy]
|
|
149
|
+
end
|
|
150
|
+
|
|
151
|
+
# ponytail: 1,000 slots per HMGET bounds memory; raise it if round trips dominate.
|
|
152
|
+
def each_result_slot(map_id, total, &block)
|
|
153
|
+
(0...total).each_slice(RubyReactor::Map::ResultEnumerator::DEFAULT_BATCH_SIZE) do |indexes|
|
|
154
|
+
slots = rollback_storage.retrieve_map_result_slots(map_id, context.reactor_class.name, indexes)
|
|
155
|
+
indexes.zip(slots).each(&block)
|
|
156
|
+
end
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
def completed_slot?(raw)
|
|
160
|
+
!(raw.is_a?(Hash) && (raw.key?("_error") || raw.key?("_halt") || raw.key?("_skipped")))
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
# Pass 1 replays aborted elements and counts completed ones; the lazy
|
|
164
|
+
# pass 2 is what the block reads.
|
|
165
|
+
def inline_completed(map_id, step_name)
|
|
166
|
+
total = rollback_storage.count_map_element_context_ids(map_id, context.reactor_class.name)
|
|
167
|
+
failures = []
|
|
168
|
+
seen = Set.new
|
|
169
|
+
unnamed = 0
|
|
170
|
+
count = 0
|
|
171
|
+
each_element(map_id, total) do |id, element|
|
|
172
|
+
next unnamed += 1 unless element
|
|
173
|
+
|
|
174
|
+
seen << Utils::FetchIndifferent.call(element.map_metadata || {}, :index).to_i
|
|
175
|
+
case element.status.to_s
|
|
176
|
+
when "completed" then count += 1
|
|
177
|
+
when "aborted"
|
|
178
|
+
outcome = inline_element_rollback(map_id, id, step_name)
|
|
179
|
+
failures.concat(ContextSerializer.deserialize_value(outcome["failures"]))
|
|
180
|
+
end
|
|
181
|
+
end
|
|
182
|
+
unseen = ->(indexes) { indexes.reject { |index| seen.include?(index) } }
|
|
183
|
+
results = Enumerator.new do |yielder|
|
|
184
|
+
each_element(map_id, total) do |_id, element|
|
|
185
|
+
yielder << element_result(element) if element && element.status.to_s == "completed"
|
|
186
|
+
end
|
|
187
|
+
end
|
|
188
|
+
[unavailable_entries(step_name, unnamed, unseen) + failures, count, results.lazy]
|
|
189
|
+
end
|
|
190
|
+
|
|
191
|
+
# Element contexts in index order (head of the index first), one loaded
|
|
192
|
+
# at a time, `Map::ROLLBACK_CHUNK` ids per read.
|
|
193
|
+
def each_element(map_id, total)
|
|
194
|
+
storage_name = RubyReactor.reactor_storage_name(element_class)
|
|
195
|
+
(0...total).each_slice(RubyReactor::Map::ROLLBACK_CHUNK) do |heads|
|
|
196
|
+
ids = rollback_storage.retrieve_map_element_context_ids_from_tail(
|
|
197
|
+
map_id, context.reactor_class.name, total - 1 - heads.last, heads.size, total: total
|
|
198
|
+
).reverse
|
|
199
|
+
ids.each do |id|
|
|
200
|
+
data = rollback_storage.retrieve_context(id, storage_name)
|
|
201
|
+
yield id, data && Context.deserialize_from_retry(data)
|
|
202
|
+
end
|
|
203
|
+
end
|
|
204
|
+
end
|
|
205
|
+
|
|
206
|
+
# What the element returned, as the map collected it
|
|
207
|
+
# (`ResultHandler#final_result`): its `returns` step, else every result.
|
|
208
|
+
def element_result(element)
|
|
209
|
+
klass = element.reactor_class || element_class
|
|
210
|
+
klass.return_step ? element.get_result(klass.return_step) : element.intermediate_results
|
|
211
|
+
end
|
|
212
|
+
|
|
213
|
+
def call_undo_all(block, results, step_name, count)
|
|
214
|
+
log_rollback("undo_all.started", step_name, count: count)
|
|
215
|
+
outcome = begin
|
|
216
|
+
block.call(results)
|
|
217
|
+
rescue RubyReactor::Error::Rescuable => e
|
|
218
|
+
e
|
|
219
|
+
end
|
|
220
|
+
failure = undo_all_failure(step_name, outcome)
|
|
221
|
+
context.append_execution_trace({ type: :undo_all, step: step_name, count: count, timestamp: Time.now,
|
|
222
|
+
result: failure ? failure[:message] : :ok })
|
|
223
|
+
log_rollback("undo_all.completed", step_name, count: count, failed: failure ? 1 : 0)
|
|
224
|
+
failure
|
|
225
|
+
end
|
|
226
|
+
|
|
227
|
+
def undo_all_failure(step_name, outcome)
|
|
228
|
+
return unless outcome.is_a?(Exception) || outcome.is_a?(RubyReactor::Failure)
|
|
229
|
+
|
|
230
|
+
error = outcome.is_a?(RubyReactor::Failure) ? outcome.error : outcome
|
|
231
|
+
{ step: step_name.to_sym, kind: :undo_all, reason: outcome.is_a?(Exception) ? :raised : :returned_failure,
|
|
232
|
+
message: error.respond_to?(:message) ? error.message : error.to_s }
|
|
233
|
+
end
|
|
234
|
+
|
|
235
|
+
def rollback_storage
|
|
236
|
+
RubyReactor.configuration.storage_adapter
|
|
237
|
+
end
|
|
238
|
+
|
|
118
239
|
def finish_rollback(step_name, total, failures)
|
|
119
240
|
log_rollback("completed", step_name, total: total, failed: failures.size)
|
|
120
241
|
return RubyReactor.Success() if failures.empty?
|