ruby_reactor 0.8.4 → 0.8.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/.specify/feature.json +1 -1
- data/CHANGELOG.md +196 -0
- data/CLAUDE.md +1 -1
- data/README.md +47 -11
- data/lib/ruby_reactor/dsl/async_macros.rb +30 -1
- data/lib/ruby_reactor/dsl/async_reactor_builder.rb +12 -6
- data/lib/ruby_reactor/dsl/compose_builder.rb +12 -6
- data/lib/ruby_reactor/dsl/interrupt_builder.rb +1 -3
- data/lib/ruby_reactor/dsl/map_builder.rb +0 -2
- data/lib/ruby_reactor/dsl/step_builder.rb +91 -19
- data/lib/ruby_reactor/error/argument_resolution_error.rb +19 -0
- data/lib/ruby_reactor/error/rescuable.rb +28 -0
- data/lib/ruby_reactor/executor/compensation_manager.rb +30 -26
- data/lib/ruby_reactor/executor/result_handler.rb +18 -16
- data/lib/ruby_reactor/executor/step_coordination.rb +11 -8
- data/lib/ruby_reactor/executor/step_executor.rb +59 -49
- data/lib/ruby_reactor/executor.rb +38 -4
- data/lib/ruby_reactor/map/collector.rb +21 -11
- data/lib/ruby_reactor/map/dispatcher.rb +29 -3
- data/lib/ruby_reactor/map/element_executor.rb +9 -3
- data/lib/ruby_reactor/map/helpers.rb +32 -2
- data/lib/ruby_reactor/map/result_enumerator.rb +18 -12
- data/lib/ruby_reactor/reactor.rb +24 -0
- data/lib/ruby_reactor/rspec/matchers.rb +19 -3
- data/lib/ruby_reactor/step/compose_step.rb +7 -1
- data/lib/ruby_reactor/step/map_step.rb +109 -4
- data/lib/ruby_reactor/step.rb +7 -0
- data/lib/ruby_reactor/step_worker.rb +46 -22
- data/lib/ruby_reactor/storage/adapter.rb +4 -0
- data/lib/ruby_reactor/storage/redis_adapter.rb +9 -0
- data/lib/ruby_reactor/storage/redis_reactor_scan.rb +1 -1
- data/lib/ruby_reactor/version.rb +1 -1
- data/lib/ruby_reactor/web/api.rb +1 -1
- data/lib/ruby_reactor/web/public/assets/{index-CeZU-ESu.js → index-CQbgHtd0.js} +10 -10
- data/lib/ruby_reactor/web/public/index.html +1 -1
- data/lib/ruby_reactor/worker.rb +3 -1
- data/lib/ruby_reactor.rb +17 -6
- data/specs/007-execution-flow-analysis/analysis/README.md +147 -0
- data/specs/007-execution-flow-analysis/analysis/execution-order.md +359 -0
- data/specs/007-execution-flow-analysis/analysis/findings-and-options.md +502 -0
- data/specs/007-execution-flow-analysis/analysis/invariants.md +109 -0
- data/specs/007-execution-flow-analysis/checklists/requirements.md +39 -0
- data/specs/007-execution-flow-analysis/contracts/report-structure.md +71 -0
- data/specs/007-execution-flow-analysis/data-model.md +83 -0
- data/specs/007-execution-flow-analysis/evidence/harness.rb +229 -0
- data/specs/007-execution-flow-analysis/evidence/output.txt +333 -0
- data/specs/007-execution-flow-analysis/evidence/probes/01_plain.rb +122 -0
- data/specs/007-execution-flow-analysis/evidence/probes/02_compose.rb +182 -0
- data/specs/007-execution-flow-analysis/evidence/probes/03_map.rb +232 -0
- data/specs/007-execution-flow-analysis/evidence/probes/04_async.rb +132 -0
- data/specs/007-execution-flow-analysis/evidence/probes/05_background.rb +58 -0
- data/specs/007-execution-flow-analysis/evidence/probes/06_coordination.rb +158 -0
- data/specs/007-execution-flow-analysis/evidence/probes/07_interrupts_manual.rb +185 -0
- data/specs/007-execution-flow-analysis/evidence/run.rb +15 -0
- data/specs/007-execution-flow-analysis/plan.md +127 -0
- data/specs/007-execution-flow-analysis/quickstart.md +51 -0
- data/specs/007-execution-flow-analysis/research.md +202 -0
- data/specs/007-execution-flow-analysis/spec.md +270 -0
- data/specs/007-execution-flow-analysis/tasks.md +257 -0
- data/specs/008-rollback-reliability/checklists/requirements.md +43 -0
- data/specs/008-rollback-reliability/contracts/api-surface.md +126 -0
- data/specs/008-rollback-reliability/contracts/rollback-semantics.md +76 -0
- data/specs/008-rollback-reliability/data-model.md +139 -0
- data/specs/008-rollback-reliability/plan.md +233 -0
- data/specs/008-rollback-reliability/quickstart.md +105 -0
- data/specs/008-rollback-reliability/research.md +653 -0
- data/specs/008-rollback-reliability/spec.md +561 -0
- data/specs/008-rollback-reliability/tasks.md +1110 -0
- data/specs/future_improvements.md +48 -0
- metadata +35 -2
|
@@ -0,0 +1,502 @@
|
|
|
1
|
+
# Findings, Documentation Audit & Options
|
|
2
|
+
|
|
3
|
+
Severity (research D7): **High** = completed side effects silently left in place, work that can
|
|
4
|
+
run twice, or rollback of work that never ran. **Medium** = defensible but differs by
|
|
5
|
+
mode/nesting, or contradicts documentation. **Low** = clarity/observability; predictable, but the
|
|
6
|
+
DSL or telemetry gives no hint.
|
|
7
|
+
|
|
8
|
+
All options are **proposals pending later analysis**, not decisions.
|
|
9
|
+
|
|
10
|
+
- [1. Findings](#1-findings)
|
|
11
|
+
- [2. Documentation audit](#2-documentation-audit)
|
|
12
|
+
- [3. Options](#3-options) (includes [the map `compensate_all` / `compensate_each` evaluation](#o-01-evaluation-compensate_all-vs-compensate_each))
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## 1. Findings
|
|
17
|
+
|
|
18
|
+
### F-01 · High · Map elements that already succeeded are never rolled back
|
|
19
|
+
|
|
20
|
+
- **Resolved by 008**: map rollback replays every completed element's undo stack, highest index first, in inline and fan-out mode (008 R-02, R-03). See `spec/ruby_reactor/rollback/map_rollback_spec.rb`.
|
|
21
|
+
|
|
22
|
+
- **Scenarios**: [O: S-map-01] [O: S-map-03] [O: S-map-04] [O: S-map-04b] [O: S-map-06] [O: S-map-07] [O: S-map-08]
|
|
23
|
+
- **Reader expects**: a map is "a step that happens N times". When the map fails, or a later step
|
|
24
|
+
fails, the elements that completed are rolled back the way a `compose` child is (README: "automatic
|
|
25
|
+
rollback of completed steps").
|
|
26
|
+
- **Actual**: only the **failed** element rolls back its own steps. Succeeded elements keep
|
|
27
|
+
their effects in both cases:
|
|
28
|
+
1. **The map fails** (fail_fast): elements before the failure are left in place, including their
|
|
29
|
+
nested composes (S-map-08). `MapStep#compensate` is a stub
|
|
30
|
+
(`# TODO: Implement compensation for map steps` → `Success()`)
|
|
31
|
+
[R: lib/ruby_reactor/step/map_step.rb:29-31].
|
|
32
|
+
2. **A later step fails**: every element is left in place. Inline, the map step's `undo` is the
|
|
33
|
+
base `Skipped` [R: lib/ruby_reactor/step.rb:57], and the element contexts are discarded. Fan-out,
|
|
34
|
+
the collector never even pushes the map step on the undo stack
|
|
35
|
+
[R: lib/ruby_reactor/map/helpers.rb:97].
|
|
36
|
+
3. There is no DSL surface to fix it locally: `MapBuilder` builds `compensate_block: nil,
|
|
37
|
+
undo_block: nil` [R: lib/ruby_reactor/dsl/map_builder.rb:130-131]. `undo` blocks written on the
|
|
38
|
+
element reactor's steps run only when **that element** fails.
|
|
39
|
+
- **Rollback reporting**: `rollback_failures` is empty. Nothing was attempted, so nothing is reported.
|
|
40
|
+
- **Doc conflict**: README.md:16, :25, :1309, :1398; documentation/data_pipelines.md:167;
|
|
41
|
+
documentation/DAG.md:228-240 (see §2).
|
|
42
|
+
- **Related**: INV-19, INV-20 (both VIOLATED, no spec coverage).
|
|
43
|
+
|
|
44
|
+
### F-02 · High · `compose` with `retries` resumes a child whose earlier steps were already undone
|
|
45
|
+
|
|
46
|
+
- **Resolved by 008**: a compose retry after a failed attempt starts a fresh child (008 R-05). See `spec/ruby_reactor/rollback/compose_retry_spec.rb`.
|
|
47
|
+
|
|
48
|
+
- **Scenarios**: [O: S-compose-05] [O: S-compose-05b]
|
|
49
|
+
- **Reader expects**: `retries` on a compose retries the sub-saga. After a failed attempt that
|
|
50
|
+
rolled the child back, the next attempt starts the child again.
|
|
51
|
+
- **Actual**: the failed attempt undoes `c1` (its effect is gone). The retry reuses the same child
|
|
52
|
+
context and `resume_execution`s it. `c1`'s result is still in `intermediate_results`, so `c1` is
|
|
53
|
+
treated as completed and **not re-run**. `c2` then runs against the result of an undone step, and
|
|
54
|
+
the reactor returns `c1`'s stale value. If a later parent step fails, only `c2` is undone
|
|
55
|
+
(S-compose-05b). The child ends "successful" with its first step's side effect missing.
|
|
56
|
+
[R: lib/ruby_reactor/step/compose_step.rb:97] [R: lib/ruby_reactor/executor/graph_manager.rb (mark_completed_steps_from_context)]
|
|
57
|
+
- **Doc conflict**: documentation/composition.md:184 ("can be configured with different retry
|
|
58
|
+
strategies") is silent on this.
|
|
59
|
+
- **Related**: INV-13 (VIOLATED, no coverage).
|
|
60
|
+
|
|
61
|
+
### F-03 · High · Some failures skip rollback entirely
|
|
62
|
+
|
|
63
|
+
- **Resolved by 008**: argument, condition and unknown `StandardError`s roll back and carry `step_name`; a non-`StandardError` marks a caller-process run `aborted` for a manual undo (008 R-06–R-08). See `spec/ruby_reactor/rollback/failure_rollback_spec.rb`, `aborted_execution_spec.rb`.
|
|
64
|
+
|
|
65
|
+
- **Scenarios**: [O: S-plain-07] [O: S-edge-03]
|
|
66
|
+
- **Reader expects**: "if any part of your workflow fails … automatically triggers compensation"
|
|
67
|
+
(README.md:16).
|
|
68
|
+
- **Actual**: two failure classes roll back nothing:
|
|
69
|
+
1. A `StandardError` raised while **resolving a step's arguments** (a `transform:` lambda, a
|
|
70
|
+
dynamic source, a `result(:x, path)` that raises). Resolution runs outside the step's rescue
|
|
71
|
+
[R: lib/ruby_reactor/executor/step_executor.rb:83]. The error reaches
|
|
72
|
+
`ResultHandler#build_execution_failure`, whose non-`Error::Base` branch is
|
|
73
|
+
"Unknown errors - don't rollback" [R: lib/ruby_reactor/executor/result_handler.rb:68-70].
|
|
74
|
+
The result is `Failure("Execution failed: …")` with no step name and an empty `rollback_failures`.
|
|
75
|
+
2. A non-`StandardError` exception in a body (e.g. `Timeout::ExitException`-like, custom
|
|
76
|
+
`Exception` subclasses) propagates to the caller with no rollback. A worker would redeliver
|
|
77
|
+
(INV-32); an inline caller gets partial effects.
|
|
78
|
+
- **Related**: INV-06 (VIOLATED, no coverage).
|
|
79
|
+
|
|
80
|
+
### F-04 · High · An `async_step`'s own `compensate` / `undo` never run; the docs say they do
|
|
81
|
+
|
|
82
|
+
- **Resolved by 008**: an `async_step` unit compensates itself once, in its own job, after its final attempt; an inline `undo` is rejected, a class `undo` warned (008 R-09). See `spec/ruby_reactor/rollback/async_step_compensate_spec.rb`.
|
|
83
|
+
|
|
84
|
+
- **Scenarios**: [O: S-async-02] [O: S-async-07]
|
|
85
|
+
- **Reader expects** (documentation/background_and_async.md:291-292): "`compensate` / `undo` blocks
|
|
86
|
+
declared on an `async_step` still register; they run only if the failure is surfaced into the
|
|
87
|
+
parent's compensation path this way".
|
|
88
|
+
- **Actual**: when a reader surfaces the unit's failure, the **reader** is compensated and the
|
|
89
|
+
parent's steps are undone, but `compensate:u` never runs. `StepWorker` never invokes rollback
|
|
90
|
+
hooks [R: lib/ruby_reactor/step_worker.rb:240-393], and the parent never pushed the unit
|
|
91
|
+
[R: lib/ruby_reactor/executor/result_handler.rb:137]. The blocks are accepted by the DSL and are
|
|
92
|
+
dead code. The same holds when the unit exhausts its retries (S-async-07).
|
|
93
|
+
- **Coverage gap**: [T: spec/ruby_reactor/dsl/async_step_spec.rb:111] asserts only that `:setup`
|
|
94
|
+
is compensated, so it passes either way.
|
|
95
|
+
- **Related**: INV-24, INV-29.
|
|
96
|
+
|
|
97
|
+
### F-05 · Medium · Fan-out fail-fast leaves a scheduling-dependent set of elements in place
|
|
98
|
+
|
|
99
|
+
- **Resolved by 008**: a fail-fast fan-out map settles every index before applying its failure, and every completed element is rolled back, so nothing is left in place whatever the job order (008 R-04). See `spec/ruby_reactor/rollback/map_fan_out_settle_spec.rb`.
|
|
100
|
+
|
|
101
|
+
- **Scenarios**: [O: S-map-04] vs [O: S-map-04b]
|
|
102
|
+
- **Actual**: the fail-fast marker is checked only when an element job **starts**
|
|
103
|
+
[R: lib/ruby_reactor/map/element_executor.rb:61, :145]. Elements that started, or finished,
|
|
104
|
+
before the failing element are kept. Later ones skip. Same input, different job order →
|
|
105
|
+
different leftovers (elements 0, 1 vs element 3). Combined with F-01, the leftover set is
|
|
106
|
+
neither rolled back nor predictable.
|
|
107
|
+
- **Doc conflict**: documentation/data_pipelines.md:167 ("fails immediately").
|
|
108
|
+
- **Related**: INV-22 (CONDITIONAL).
|
|
109
|
+
|
|
110
|
+
### F-06 · Medium · A raising `where`/`guard` compensates a step whose body never ran
|
|
111
|
+
|
|
112
|
+
- **Resolved by 008**: a raising `where`/`guard` is a never-started `ConditionError`: not compensated, not retried (008 R-06).
|
|
113
|
+
|
|
114
|
+
- **Scenario**: [O: S-edge-04]
|
|
115
|
+
- **Actual**: conditions are evaluated inside the step's rescue
|
|
116
|
+
[R: lib/ruby_reactor/executor/step_executor.rb:342]. The resulting `Failure` goes through
|
|
117
|
+
normal failure handling, which compensates the step. Contention, key errors and argument
|
|
118
|
+
validation are all correctly treated as never-started (S-lock-03, S-edge-05). This path isn't.
|
|
119
|
+
- **Doc conflict**: documentation/locks_and_semaphores.md:778 states the never-started
|
|
120
|
+
principle (for contention only).
|
|
121
|
+
- **Related**: INV-07 (CONDITIONAL).
|
|
122
|
+
|
|
123
|
+
### F-07 · Medium · `Halt` means different things at different nesting levels
|
|
124
|
+
|
|
125
|
+
- **Scenarios**: [O: S-compose-08] vs [O: S-map-11]
|
|
126
|
+
- **Actual**: a map element's `Halt` halts the parent. A composed child's `Halt` is converted to
|
|
127
|
+
`Success(nil)` [R: lib/ruby_reactor/step/compose_step.rb:106-114], and the parent **continues**
|
|
128
|
+
with `nil` as the compose result.
|
|
129
|
+
- **Doc conflict**: documentation/core_concepts.md:277 and documentation/composition.md:195
|
|
130
|
+
describe Halt and compose without this case.
|
|
131
|
+
- **Related**: INV-09 (VIOLATED).
|
|
132
|
+
|
|
133
|
+
### F-08 · Medium · `Reactor.undo(id)` runs outside the reactor-level lock
|
|
134
|
+
|
|
135
|
+
- **Scenario**: [O: S-intr-04]
|
|
136
|
+
- **Actual**: `Reactor#undo` builds an executor and calls `undo_all` directly
|
|
137
|
+
[R: lib/ruby_reactor/reactor.rb:188-192]. It never acquires the reactor's `with_lock` /
|
|
138
|
+
`with_semaphore`. The undo succeeded while another owner held `rk`. Step-level locks *are*
|
|
139
|
+
re-taken, so only reactor-level exclusion is lost.
|
|
140
|
+
- **Doc conflict**: documentation/interrupts.md:150-157 (Cancellation & Undo) is silent on
|
|
141
|
+
locking.
|
|
142
|
+
- **Related**: INV-35 (CONDITIONAL).
|
|
143
|
+
|
|
144
|
+
### F-09 · Medium · Dispatched units run after their dispatcher rolled back, including units of a failed map element
|
|
145
|
+
|
|
146
|
+
- **Scenarios**: [O: S-async-03] [O: S-async-06] [O: S-async-08] [O: S-map-12]
|
|
147
|
+
- **Actual**: parent rollback neither cancels nor undoes an `async_step`/`async_reactor`
|
|
148
|
+
(documented independence). A queued unit then performs its side effect **after** the rollback.
|
|
149
|
+
For an `async_step` inside a fan-out map element the unit escapes even the element: the element
|
|
150
|
+
rolled back, the unit still ran (S-map-12). This contradicts `ElementExecutor`'s stated intent
|
|
151
|
+
that async work "must execute inline here" [R: lib/ruby_reactor/map/element_executor.rb:51-55],
|
|
152
|
+
because `async_step` dispatch is deliberately not gated on that flag
|
|
153
|
+
[R: lib/ruby_reactor/executor/async_step_dispatch.rb:20-24].
|
|
154
|
+
- **Doc conflict**: README.md:545-550 and documentation/background_and_async.md:285-289 explain
|
|
155
|
+
independence, but not that a unit can run *after* its dispatcher's rollback.
|
|
156
|
+
- **Related**: INV-27 (VIOLATED).
|
|
157
|
+
|
|
158
|
+
### F-10 · Medium · Rollback coverage of composite constructs is asymmetric and invisible in the DSL
|
|
159
|
+
|
|
160
|
+
- **Scenarios**: [O: S-compose-02] vs [O: S-map-03] vs [O: S-async-06]
|
|
161
|
+
- **Actual**: three constructs that all "run more steps" differ completely, and nothing in the
|
|
162
|
+
reactor definition shows it:
|
|
163
|
+
|
|
164
|
+
| Construct | Child/element/unit completed, later parent failure | Child/element/unit fails |
|
|
165
|
+
|---|---|---|
|
|
166
|
+
| `compose` | undone (full child replay) | child self-rolls back, parent rolls back |
|
|
167
|
+
| `map` | **left in place** | failed element self-rolls back; siblings left |
|
|
168
|
+
| `async_step` | left in place | own hooks never run; parent unaffected unless a reader opts in |
|
|
169
|
+
| `async_reactor` | left in place | child self-rolls back; parent unaffected unless a reader opts in |
|
|
170
|
+
|
|
171
|
+
- **Resolved by 008** (the table rebuilt from 008's rollback contract, RS §2): one rule for every
|
|
172
|
+
construct, and each construct's own `compensate`/`undo` decides what rollback means for it. The
|
|
173
|
+
coordinator asks the step whether a success is tracked for undo (`rollback_tracked?`).
|
|
174
|
+
|
|
175
|
+
| Construct | Child/element/unit completed, later parent failure | Child/element/unit fails |
|
|
176
|
+
|---|---|---|
|
|
177
|
+
| `compose` | undone (full child replay) | child self-rolls back, parent rolls back; a retry starts a fresh child |
|
|
178
|
+
| `map` | **undone** (every completed element, highest index first) | failed element self-rolls back; completed siblings undone, then the parent rolls back |
|
|
179
|
+
| `async_step` | left in place (independent by design, documented) | the unit compensates itself once, in its own job; parent unaffected unless a reader opts in |
|
|
180
|
+
| `async_reactor` | left in place (independent by design, documented) | child self-rolls back; parent unaffected unless a reader opts in |
|
|
181
|
+
|
|
182
|
+
`compose`, `map` and the async macros accept no `compensate`/`undo` declaration of their own, so
|
|
183
|
+
a reader cannot tell from the class body what will be rolled back. This is the user's
|
|
184
|
+
"does the DSL help clearly understand" question.
|
|
185
|
+
- **Related**: F-01, F-04, INV-16, INV-19, INV-20, INV-25.
|
|
186
|
+
|
|
187
|
+
### F-11 · Low · Step-lock gap between the forward release and the rollback re-acquire
|
|
188
|
+
|
|
189
|
+
- **Scenarios**: [O: S-lock-02] [O: S-lock-05]
|
|
190
|
+
- **Actual**: a step lock is held for the body only. Rollback re-takes it (documented) but another
|
|
191
|
+
execution can hold it in between and change the protected resource. The undo is serialized with
|
|
192
|
+
that execution, but it runs against a possibly changed resource. It can also be skipped (reported)
|
|
193
|
+
when the key stays busy past `rollback_wait`.
|
|
194
|
+
- **Doc**: documentation/locks_and_semaphores.md:852-866 documents the re-take, not the gap.
|
|
195
|
+
- **Related**: INV-35, INV-36.
|
|
196
|
+
|
|
197
|
+
### F-12 · Low · `async_step` retries are invisible to middleware
|
|
198
|
+
|
|
199
|
+
- **Scenario**: [O: S-async-07]. There is no `retry_attempt` event, unlike every other retry path
|
|
200
|
+
[R: lib/ruby_reactor/step_worker.rb:275-289]. **Related**: INV-29.
|
|
201
|
+
|
|
202
|
+
### F-13 · Low · Failures without step attribution after rollback
|
|
203
|
+
|
|
204
|
+
- **Resolved by 008**: argument, condition, unknown and compensation failures all carry `step_name` and `reactor_name` (008 R-06, R-07).
|
|
205
|
+
|
|
206
|
+
- **Scenarios**: [O: S-plain-03] (compensate fails → `CompensationError` → "Execution error: …",
|
|
207
|
+
no `step_name`), [O: S-plain-07] (argument resolution → no `step_name`).
|
|
208
|
+
[R: lib/ruby_reactor/executor/result_handler.rb:64-70]. Violates Constitution IV ("every failure
|
|
209
|
+
MUST carry … step name").
|
|
210
|
+
|
|
211
|
+
### F-14 · Low · Dead, contradictory map collection default
|
|
212
|
+
|
|
213
|
+
- `Map::Helpers#apply_collect_block` ("Default behavior: fail if any failure")
|
|
214
|
+
[R: lib/ruby_reactor/map/helpers.rb:36-52] is shadowed by `Collector.apply_collect_block`
|
|
215
|
+
("Default behavior: Return Success(Enumerator)") [R: lib/ruby_reactor/map/collector.rb:108-126].
|
|
216
|
+
Only the latter is called. A reader of `helpers.rb` gets the wrong semantics.
|
|
217
|
+
|
|
218
|
+
### F-15 · Low · Reactor-level `lock_released` names a different key than `lock_acquired`
|
|
219
|
+
|
|
220
|
+
- **Scenario**: [O: S-lock-01] (`lock_acquired:rk`, `lock_released:lock:rk`). Known and pinned by
|
|
221
|
+
[T: spec/ruby_reactor/step_coordination/attribution_spec.rb:41]. **Related**: INV-38.
|
|
222
|
+
|
|
223
|
+
### F-16 · Low · Interrupt docs say "cancelled"; exhausted payload validation marks the run `failed`
|
|
224
|
+
|
|
225
|
+
- **Scenario**: [O: S-intr-02] ends `failure(approval)`, status `failed`
|
|
226
|
+
[R: lib/ruby_reactor/reactor.rb:411-415]. documentation/interrupts.md:40, :62, :141 say the
|
|
227
|
+
reactor is "cancelled and compensated". The rollback itself is as documented.
|
|
228
|
+
|
|
229
|
+
---
|
|
230
|
+
|
|
231
|
+
## 2. Documentation audit
|
|
232
|
+
|
|
233
|
+
Claims about ordering or rollback in README.md and `./documentation`. **CONFIRMED** rows are
|
|
234
|
+
listed too, so the audit is complete. Per plan.md Complexity Tracking, none of these files is edited
|
|
235
|
+
by this initiative. Each fix ships with the remedy chosen for its finding.
|
|
236
|
+
|
|
237
|
+
| File:line | Claim (quoted) | Actual | Finding |
|
|
238
|
+
|---|---|---|---|
|
|
239
|
+
| README.md:16 | "if any part of your workflow fails, Ruby Reactor automatically triggers compensation logic to undo previous steps, ensuring your system never ends up in a corrupted half-state" | Not for map elements, async units, argument-resolution errors or non-`StandardError` exceptions | F-01, F-03, F-04, F-09 |
|
|
240
|
+
| README.md:25 | "**Compensation**: Automatic rollback of completed steps when a failure occurs." | Same exceptions | F-01, F-03 |
|
|
241
|
+
| README.md:35 | "Auto compensation/undo \| Yes" | Partial (see F-10 table) | F-10 |
|
|
242
|
+
| README.md:545-550 | "A reader that returns `Failure` triggers compensation normally" | Triggers the reader's compensation and the parent's undos only. The unit's hooks never run, and the unit may run after the rollback | F-04, F-09 |
|
|
243
|
+
| README.md:841 | Halt: "already-completed steps are NOT compensated" | CONFIRMED [O: S-plain-05]. Nested behavior unstated | F-07 |
|
|
244
|
+
| README.md:1309 | "When a step fails, RubyReactor automatically undoes completed steps in reverse order, compensate only runs in the failing step…" | CONFIRMED for steps and composes [O: S-plain-09, S-compose-03]. Not for map elements | F-01 |
|
|
245
|
+
| README.md:1398 | "A rollback that did not complete is never silent." | True for **attempted** rollbacks [O: S-plain-04, S-lock-05]. Rollbacks never attempted (map elements, F-03) leave `rollback_failures` empty | F-01, F-03 |
|
|
246
|
+
| documentation/background_and_async.md:165-167 | "Compensation is unchanged. A worker-side failure compensates exactly as a same-process failure does" | CONFIRMED [O: S-bg-01, S-bg-02] | — |
|
|
247
|
+
| documentation/background_and_async.md:279-283 | "A later step that reads the result and returns `Failure` triggers compensation normally — so no failure is ever unrecoverable" | Reader + parent only; the unit's effect stays | F-04 |
|
|
248
|
+
| documentation/background_and_async.md:285-289 | "the dispatch itself never enters the parent's undo stack … independent compensation flows" | CONFIRMED [O: S-async-03, S-async-06]. Unstated: the unit can run *after* the rollback, and an `async_step` has **no** compensation flow at all | F-04, F-09 |
|
|
249
|
+
| documentation/background_and_async.md:291-292 | "`compensate` / `undo` blocks declared on an `async_step` still register; they run only if the failure is surfaced into the parent's compensation path this way." | **Contradicted**: they never run [O: S-async-02] | F-04 |
|
|
250
|
+
| documentation/composition.md:184 | "can be configured with different retry strategies" | A compose-level retry resumes an already-rolled-back child | F-02 |
|
|
251
|
+
| documentation/composition.md:195 | "Compensation \| fully linked — a child failure rolls the parent back" | CONFIRMED, including earlier sibling composes [O: S-compose-01, S-compose-03]. Missing row: a child `Halt` does not stop the parent | F-07 |
|
|
252
|
+
| documentation/data_pipelines.md:167 | "the entire map operation fails immediately if any single element fails" | Completed elements are kept, not rolled back. In fan-out, "immediately" means not-yet-started elements skip themselves | F-01, F-05 |
|
|
253
|
+
| documentation/data_pipelines.md:179 | `fail_fast false` collects successes and failures | CONFIRMED [O: S-map-02, S-map-05]. Unstated: failed elements roll back individually | — |
|
|
254
|
+
| documentation/data_pipelines.md:233-235 | per-element `retries` | CONFIRMED. Unstated: inline sleeps, fan-out re-enqueues [O: S-map-09, S-map-10] | — |
|
|
255
|
+
| documentation/core_concepts.md:321-331 | "Compensation runs in reverse order of successful steps" | CONFIRMED for steps [O: S-plain-09]. Map elements and async units are excluded without saying so | F-01, F-10 |
|
|
256
|
+
| documentation/core_concepts.md:277 | Halt: "completed steps are **not** compensated" | CONFIRMED | F-07 (nesting) |
|
|
257
|
+
| documentation/DAG.md:228-240 | "Cascading Compensation … continue to Step 1 → Consistent State Achieved" | Not achieved for maps, async units or F-03 failures | F-01, F-03 |
|
|
258
|
+
| documentation/locks_and_semaphores.md:777-778 | "the contended step itself does not compensate — … its own work was never attempted" | CONFIRMED [O: S-lock-03, S-edge-01]. The same principle is broken for a raising `where`/`guard` | F-06 |
|
|
259
|
+
| documentation/locks_and_semaphores.md:852-866 | Step rollback re-takes lock then semaphore, waiting `rollback_wait` | CONFIRMED [O: S-lock-02, S-lock-04, S-lock-05] | F-11 (gap unstated) |
|
|
260
|
+
| documentation/getting_started.md:231-232 | compensate the failing step, then undo in reverse | CONFIRMED | — |
|
|
261
|
+
| documentation/interrupts.md:40, :62, :141 | exhausted payload validation → "cancelled and compensated" | Rolled back as stated, but the status is **failed** [O: S-intr-02] | F-16 |
|
|
262
|
+
| documentation/interrupts.md:155-157 | `undo` runs undo blocks in reverse, then cancels | CONFIRMED [O: S-intr-04]. Unstated: no reactor lock | F-08 |
|
|
263
|
+
|
|
264
|
+
---
|
|
265
|
+
|
|
266
|
+
## 3. Options
|
|
267
|
+
|
|
268
|
+
Every option addresses finding(s), lists at least one con, and is a **proposal** only.
|
|
269
|
+
Compatibility: SemVer impact per Constitution V.
|
|
270
|
+
|
|
271
|
+
### Options for F-01 (map rollback) — and the user's Q3
|
|
272
|
+
|
|
273
|
+
The user's question: should `map` get `compensate_all` / `compensate_each`? The first design point
|
|
274
|
+
is that **two different moments** need covering, and the library already has a word for each:
|
|
275
|
+
|
|
276
|
+
- **The map itself fails** (fail-fast, partial success): the elements that succeeded so far must be
|
|
277
|
+
cleaned up. In library terms that is the map's **compensate**.
|
|
278
|
+
- **The map completed, and a later step fails**: all elements must be cleaned up. That is the map's
|
|
279
|
+
**undo**.
|
|
280
|
+
|
|
281
|
+
`compensate_all` / `compensate_each` as named would cover only the first, unless their semantics
|
|
282
|
+
also cover undo. That naming mismatch is itself a DSL-clarity issue (F-10).
|
|
283
|
+
|
|
284
|
+
#### O-01-a · Implicit element-undo replay (make `map` behave like `compose`)
|
|
285
|
+
|
|
286
|
+
- **Sketch**: `MapStep#compensate` and a new `MapStep#undo` replay each **succeeded element's own
|
|
287
|
+
undo stack**, newest element first, exactly as `ComposeStep#undo` replays a child
|
|
288
|
+
[R: lib/ruby_reactor/step/compose_step.rb:31-47]. No new DSL: the `undo` blocks authors already
|
|
289
|
+
write on element steps start running.
|
|
290
|
+
- **Pros**: consistent with compose (closes the F-10 asymmetry). Rollback granularity is per
|
|
291
|
+
element step, so partial failures inside an undo are reported per step on `rollback_failures`.
|
|
292
|
+
- **Cons**: the element contexts must be **kept**. Inline maps discard them today, and keeping N
|
|
293
|
+
contexts inside the parent blob risks `ContextTooLargeError`. Fan-out needs a rollback fan-out
|
|
294
|
+
(N jobs, or a serial loop in the collector). It is a **behavior change** for every existing map
|
|
295
|
+
whose element steps declare `undo` (MAJOR). "Newest element first" is ill-defined under fan-out
|
|
296
|
+
concurrency.
|
|
297
|
+
- **Open questions**: keep element contexts by id (fan-out already does:
|
|
298
|
+
`store_map_element_context_id`) or embed? Serial or parallel element undo? Honour `fail_fast false`?
|
|
299
|
+
|
|
300
|
+
#### O-01-b · `compensate_each` / `undo_each` block on the map (user proposal, per element)
|
|
301
|
+
|
|
302
|
+
- **Sketch**:
|
|
303
|
+
|
|
304
|
+
```ruby
|
|
305
|
+
map :charges, ChargeElement do
|
|
306
|
+
source input(:orders)
|
|
307
|
+
argument :order, element(:charges)
|
|
308
|
+
undo_each { |element_result, element| Refund.call(element_result[:charge_id]) }
|
|
309
|
+
end
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
Called once per **succeeded** element, on map failure (compensate) and on a later failure (undo).
|
|
313
|
+
- **Pros**: visible in the DSL. Needs only element **results**, which fan-out already stores
|
|
314
|
+
(`store_map_result`) and inline has in memory, so no context retention. Per-element error
|
|
315
|
+
isolation: one failed call becomes one `rollback_failures` entry and the rest continue.
|
|
316
|
+
- **Cons**: duplicates the element reactor's own step `undo`s (two places for the same cleanup)
|
|
317
|
+
and is coarser than them. For the undo moment the map step must be pushed on the undo stack in
|
|
318
|
+
fan-out mode (it isn't today, [R: lib/ruby_reactor/map/helpers.rb:97]). A `collect` block that
|
|
319
|
+
transformed the results means the raw per-element results must be kept separately. Under fan-out
|
|
320
|
+
fail-fast, elements still in flight when it runs finish later and escape it (see F-05 options).
|
|
321
|
+
- **Open questions**: argument order/shape (`element` = source item, `result` = element output)?
|
|
322
|
+
Sequential or parallel calls in fan-out?
|
|
323
|
+
|
|
324
|
+
#### O-01-c · `compensate_all` / `undo_all` block on the map (user proposal, bulk)
|
|
325
|
+
|
|
326
|
+
- **Sketch**: `undo_all { |results, elements| Charge.where(id: results.map { _1[:charge_id] }).refund_all }`,
|
|
327
|
+
called once with every succeeded element's result.
|
|
328
|
+
- **Pros**: matches bulk cleanup (one `DELETE … WHERE id IN`), which is the user's
|
|
329
|
+
`all_elements.destroy` example. It is a single rollback call to reason about and report. It works
|
|
330
|
+
identically inline and fan-out (the collector already holds a `ResultEnumerator`).
|
|
331
|
+
- **Cons**: all-or-nothing error handling. A failure halfway through a bulk undo is one opaque
|
|
332
|
+
`rollback_failures` entry. The full result set must stay available until the parent finishes
|
|
333
|
+
(fan-out results live under the context TTL). The in-flight-element problem from O-01-b applies
|
|
334
|
+
unchanged.
|
|
335
|
+
- **Open questions**: pass an enumerator (lazy, fan-out friendly) or an array? Should
|
|
336
|
+
`fail_fast false` successes be included when a later step fails? (Probably yes.)
|
|
337
|
+
|
|
338
|
+
#### O-01-d · Keep behavior, make it explicit
|
|
339
|
+
|
|
340
|
+
- **Sketch**: document "map elements are never rolled back", recommend `fail_fast false` plus a
|
|
341
|
+
following step whose `undo` cleans up from the collected results, and remove the `# TODO` stub.
|
|
342
|
+
- **Pros**: no behavior change (PATCH). Honest.
|
|
343
|
+
- **Cons**: the gap stays. Every author re-implements the same pattern. It contradicts the README
|
|
344
|
+
reliability promise, which would also have to change.
|
|
345
|
+
|
|
346
|
+
#### O-01 evaluation: `compensate_all` vs `compensate_each`
|
|
347
|
+
|
|
348
|
+
| Dimension | `compensate_each` / `undo_each` (O-01-b) | `compensate_all` / `undo_all` (O-01-c) | Element-undo replay (O-01-a) |
|
|
349
|
+
|---|---|---|---|
|
|
350
|
+
| Map fails, `fail_fast` on | called for each element that succeeded before the failure. Inline = deterministic set; fan-out = scheduling-dependent set (F-05) | one call with that same set | replays those elements' step undos |
|
|
351
|
+
| Map succeeds with `fail_fast false`, later step fails | called for each **successful** element (failed ones already self-rolled back) | one call with the successful subset | replay successful elements |
|
|
352
|
+
| Map fails with `fail_fast false` | n/a (map does not fail, unless a `collect` block raises, which then needs the compensate moment) | same | same |
|
|
353
|
+
| Inline | results in memory: easy | easy | needs element contexts kept (currently discarded) |
|
|
354
|
+
| Fan-out | results stored per index: available. In-flight elements finish after the call and escape it | same | contexts stored by id: available. Needs a rollback fan-out |
|
|
355
|
+
| Element retries | an element that succeeded after retries is included. One that exhausted is excluded (already self-rolled back) | same | same |
|
|
356
|
+
| Later-step failure (undo moment) | requires pushing the map step on the undo stack in fan-out mode | same | same |
|
|
357
|
+
| Data the block needs | per element: source item + result | all results (+ items) | none (element steps' own undo args/results) |
|
|
358
|
+
| Failure isolation | per element | whole batch | per element step |
|
|
359
|
+
| DSL visibility | explicit on the map | explicit on the map | implicit (like compose) |
|
|
360
|
+
| Duplication with element step `undo`s | yes | yes | none |
|
|
361
|
+
| Compatibility | additive (MINOR) if opt-in | additive (MINOR) if opt-in | behavior change (MAJOR) |
|
|
362
|
+
|
|
363
|
+
**Reading of the evidence (not a decision)**: the gap is real (INV-19 and INV-20 VIOLATED, zero
|
|
364
|
+
coverage). O-01-b and O-01-c are complementary rather than alternatives: per-element vs bulk is
|
|
365
|
+
the author's cleanup granularity, and both need the same two plumbing pieces:
|
|
366
|
+
|
|
367
|
+
1. The map step on the undo stack in every mode.
|
|
368
|
+
2. Retained element results.
|
|
369
|
+
|
|
370
|
+
They also share one unsolved problem: fan-out elements still in flight at compensate time (F-05).
|
|
371
|
+
O-01-a is the most consistent with `compose`, but the most expensive and the only breaking one.
|
|
372
|
+
Whatever is chosen, name the hooks after the two moments (`compensate…` for map failure,
|
|
373
|
+
`undo…` for a later failure) or document that one block covers both.
|
|
374
|
+
|
|
375
|
+
### Options for F-02 (compose retry resumes an undone child)
|
|
376
|
+
|
|
377
|
+
- **O-02-a · Fresh child per attempt**: when the compose step is retried after a failed attempt,
|
|
378
|
+
discard the stored child context and start a new one.
|
|
379
|
+
*Pros*: "retry the sub-saga" semantics, a one-line change at
|
|
380
|
+
[R: lib/ruby_reactor/step/compose_step.rb:97]. *Cons*: child steps without `undo` (left in
|
|
381
|
+
place) run again, so their side effect happens twice. Park/resume of a child mid-attempt must
|
|
382
|
+
still resume, so the fresh-child rule must distinguish "retry after failure" from "redelivery
|
|
383
|
+
after park". *Compat*: behavior fix (MINOR/PATCH, arguably a bug fix).
|
|
384
|
+
- **O-02-b · Rollback clears completion marks**: `rollback_completed_steps` removes each undone
|
|
385
|
+
step's result from `intermediate_results`, so any later resume re-runs it.
|
|
386
|
+
*Pros*: fixes the class of bug (any resume after a rollback), not just compose.
|
|
387
|
+
*Cons*: changes context state that dashboards and `execution_trace` readers see. Undone results
|
|
388
|
+
disappear, so post-mortem inspection loses data unless moved elsewhere.
|
|
389
|
+
*Compat*: MINOR, observable.
|
|
390
|
+
- **O-02-c · Disallow `retries` on `compose`** (definition-time error, point to inner-step
|
|
391
|
+
retries). *Pros*: removes the ambiguity. *Cons*: breaking (MAJOR). Loses a legitimate use (retry
|
|
392
|
+
a whole sub-saga after a transient failure deep inside it).
|
|
393
|
+
|
|
394
|
+
### Options for F-03 (failures that skip rollback)
|
|
395
|
+
|
|
396
|
+
- **O-03-a · Resolve arguments inside the step's rescue, as a never-started failure**: an
|
|
397
|
+
argument/transform error becomes the step's `Failure` (with `step_name`), classified like
|
|
398
|
+
contention: no compensate, completed steps undone.
|
|
399
|
+
*Pros*: consistent with INV-07 and fixes F-13's attribution too. *Cons*: the "deferred
|
|
400
|
+
resolution" paths (`async_step`, `background before:`) need the same treatment in their own
|
|
401
|
+
process. Needs a new never-started error class.
|
|
402
|
+
- **O-03-b · Roll back on every `StandardError`**: remove the "Unknown errors - don't rollback"
|
|
403
|
+
branch [R: lib/ruby_reactor/executor/result_handler.rb:68-70].
|
|
404
|
+
*Pros*: tiny change, covers unknown future paths. *Cons*: the branch exists deliberately.
|
|
405
|
+
Rolling back after an internal executor bug runs user undo code on possibly inconsistent state.
|
|
406
|
+
Error attribution is still missing (F-13).
|
|
407
|
+
- **O-03-c · Document the non-`StandardError` boundary** (for the `Exception` half): running user
|
|
408
|
+
undo code on `SignalException`/`NoMemoryError` is unsafe. Document that an inline run offers no
|
|
409
|
+
rollback here, and that a worker run is redelivered (INV-32).
|
|
410
|
+
*Cons*: inline callers keep partial effects. It is only a documentation fix.
|
|
411
|
+
|
|
412
|
+
### Options for F-04 (`async_step` rollback hooks never run)
|
|
413
|
+
|
|
414
|
+
- **O-04-a · Make the docs and DSL honest**: state that unit hooks never run, and reject (or warn
|
|
415
|
+
on) `compensate`/`undo` inside `async_step` at definition time.
|
|
416
|
+
*Pros*: no runtime change, removes the dead-code trap. *Cons*: loses a natural place for cleanup.
|
|
417
|
+
Authors must put it in the reader's `compensate`, far from the step. Rejecting would be MAJOR.
|
|
418
|
+
Warning is MINOR.
|
|
419
|
+
- **O-04-b · Unit-local saga**: `StepWorker` calls the unit's own `compensate` when its body finally
|
|
420
|
+
fails (after retries), inside the unit's job. This mirrors how an `async_reactor` child rolls
|
|
421
|
+
itself back (INV-28).
|
|
422
|
+
*Pros*: symmetric with `async_reactor`, the author's block becomes live, no cross-process state.
|
|
423
|
+
*Cons*: runs whether or not anyone reads the result (a behavior change for existing units that
|
|
424
|
+
declare `compensate`). Still no `undo` moment, since nobody tells a *successful* unit to roll back.
|
|
425
|
+
- **O-04-c · Parent-driven unit compensation on surfaced failure** (what the docs promise): when a
|
|
426
|
+
reader returns `Failure`, the parent's rollback also calls the unit's `compensate` with the
|
|
427
|
+
unit's recorded arguments/error, re-taking the unit step's locks.
|
|
428
|
+
*Pros*: matches documented intent. *Cons*: the parent's executor must run a step config against
|
|
429
|
+
another execution's recorded arguments and coordination owner. It only covers **read** units, and
|
|
430
|
+
it happens in the reader's process, after an arbitrary delay.
|
|
431
|
+
|
|
432
|
+
### Options for F-05 (scheduling-dependent fan-out leftovers)
|
|
433
|
+
|
|
434
|
+
- **O-05-a · Collector waits for in-flight elements before resolving a fail-fast failure**, then
|
|
435
|
+
runs the map compensation (O-01-*) over every success.
|
|
436
|
+
*Pros*: deterministic leftover set (all successes), required anyway for O-01-b/c to be complete.
|
|
437
|
+
*Cons*: failure latency grows to the slowest in-flight element. Needs "dispatched" vs "not
|
|
438
|
+
started" accounting per index.
|
|
439
|
+
- **O-05-b · Late elements self-compensate**: an element that finishes after the fail-fast marker
|
|
440
|
+
is set rolls itself back instead of storing its success.
|
|
441
|
+
*Pros*: no waiting. *Cons*: the element must run its whole saga before undoing it (wasted side
|
|
442
|
+
effects), and it races with the collector.
|
|
443
|
+
|
|
444
|
+
### Options for F-06 (raising `where`/`guard` compensates)
|
|
445
|
+
|
|
446
|
+
- **O-06-a** Classify condition errors as never-started (wrap them in a never-started error class
|
|
447
|
+
before result handling). *Cons*: slightly more error-class surface.
|
|
448
|
+
- **O-06-b** Document that conditions must not raise. *Cons*: leaves an easy trap.
|
|
449
|
+
|
|
450
|
+
### Options for F-07 (nested `Halt`)
|
|
451
|
+
|
|
452
|
+
- **O-07-a** Propagate a composed child's `Halt` as the parent's `Halt` (map-consistent).
|
|
453
|
+
*Cons*: behavior change for anyone relying on a child halting "locally".
|
|
454
|
+
- **O-07-b** Make an element's `Halt` stop only that element (compose-consistent).
|
|
455
|
+
*Cons*: changes map semantics. The collector must represent "halted element" in results
|
|
456
|
+
(it already can: `_halt` [R: lib/ruby_reactor/map/element_executor.rb:166-171]).
|
|
457
|
+
- **O-07-c** Keep both, document them side by side. *Cons*: the asymmetry remains.
|
|
458
|
+
|
|
459
|
+
### Options for F-08 (`Reactor.undo` outside the reactor lock)
|
|
460
|
+
|
|
461
|
+
- **O-08-a** `Reactor#undo` acquires the reactor-level lock/semaphore (with the configured `wait`)
|
|
462
|
+
before `undo_all`. *Cons*: a manual undo can now fail on contention. Needs an error shape.
|
|
463
|
+
- **O-08-b** Document it. *Cons*: concurrent forward run + manual undo stays possible.
|
|
464
|
+
|
|
465
|
+
### Options for F-09 (units run after rollback)
|
|
466
|
+
|
|
467
|
+
- **O-09-a · Cancellation marker**: a failing dispatcher marks its not-yet-started units cancelled.
|
|
468
|
+
`StepWorker`/`Worker` checks the marker before running the body. *Cons*: units already running
|
|
469
|
+
are unaffected (still a race). Adds a storage write per rollback.
|
|
470
|
+
- **O-09-b · Gate `async_step` dispatch inside map elements** (run inline there, as
|
|
471
|
+
`ElementExecutor` intends [R: lib/ruby_reactor/map/element_executor.rb:51-55]). *Cons*:
|
|
472
|
+
reverses a deliberate decision [R: lib/ruby_reactor/executor/async_step_dispatch.rb:20-24] and
|
|
473
|
+
serializes the unit into the element.
|
|
474
|
+
- **O-09-c** Document the ordering explicitly. *Cons*: behavior unchanged.
|
|
475
|
+
|
|
476
|
+
### Options for F-10 (DSL visibility)
|
|
477
|
+
|
|
478
|
+
- **O-10-a · Uniform rollback declarations on composites**: every construct that runs "more
|
|
479
|
+
steps" states its rollback policy in the DSL (`map … undo_each/undo_all`, `compose … rollback:
|
|
480
|
+
:replay` default, `async_step … compensate` either live or rejected).
|
|
481
|
+
*Cons*: more DSL surface. Composes would gain a knob they may not need.
|
|
482
|
+
- **O-10-b · Rollback plan introspection**: `Reactor.rollback_plan` (and a dashboard view) lists,
|
|
483
|
+
per step, what a failure after it would roll back and what it would leave in place, derived from
|
|
484
|
+
the rules in [execution-order.md](execution-order.md#1-rollback-algorithm).
|
|
485
|
+
*Pros*: no semantic change, helps readers and reviewers. *Cons*: the plan must be kept in sync
|
|
486
|
+
with the executor. It is descriptive only.
|
|
487
|
+
|
|
488
|
+
### Options for Low findings
|
|
489
|
+
|
|
490
|
+
| Finding | Option | Con |
|
|
491
|
+
|---|---|---|
|
|
492
|
+
| F-11 lock gap | Document the gap next to `rollback_wait` and recommend idempotent, state-checking undos | documentation only |
|
|
493
|
+
| F-12 async retries invisible | Emit `retry_attempt` from `StepWorker`'s loop (middlewares are already built there) | more events on an existing hook (observability change) |
|
|
494
|
+
| F-13 missing attribution | Carry `step_name` onto `CompensationError`-derived and resolution failures (overlaps O-03-a) | failure shape grows |
|
|
495
|
+
| F-14 dead collect default | Delete `Map::Helpers#apply_collect_block` | none known. Confirm no external caller |
|
|
496
|
+
| F-15 lock key asymmetry | Emit the unprefixed key on release (the spec pins the current shape, so update it) | event payload change (observability MINOR) |
|
|
497
|
+
| F-16 interrupt status docs | Fix interrupts.md to say `failed` | documentation only |
|
|
498
|
+
|
|
499
|
+
---
|
|
500
|
+
|
|
501
|
+
*All options above are proposals pending later analysis. Nothing in this initiative changes
|
|
502
|
+
library behavior, tests or the demo application.*
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
# Invariants
|
|
2
|
+
|
|
3
|
+
Testable propositions about ordering and rollback. Each has a status, evidence and existing-spec
|
|
4
|
+
coverage. Status scale: **HOLDS** (evidence agrees, no counter-example found) · **VIOLATED**
|
|
5
|
+
(a reproducible counter-example exists, cited as `[O]`) · **CONDITIONAL** (holds only under the
|
|
6
|
+
listed conditions) · **UNDETERMINED** (evidence insufficient).
|
|
7
|
+
|
|
8
|
+
Every `[O: S-…]` is a block in [`../evidence/output.txt`](../evidence/output.txt).
|
|
9
|
+
Details and sequences are in [execution-order.md](execution-order.md).
|
|
10
|
+
|
|
11
|
+
**Updated for 008.** INV-06, 07, 13, 19, 20, 22 and 24 now **HOLD**, each covered by a spec under
|
|
12
|
+
`spec/ruby_reactor/rollback/` that fails on the baseline. Their original counter-examples are in
|
|
13
|
+
git history.
|
|
14
|
+
|
|
15
|
+
## A. Ordering & the rollback algorithm
|
|
16
|
+
|
|
17
|
+
| Id | Statement | Scope | Status | Evidence | Coverage |
|
|
18
|
+
|---|---|---|---|---|---|
|
|
19
|
+
| INV-01 | The failing step's compensate runs **before** any undo. | all reactor executions | HOLDS | [R: lib/ruby_reactor/executor/compensation_manager.rb:35-65] [O: S-plain-01, S-compose-01, S-map-01] | [T: spec/ruby_reactor/order_processing_reactor_spec.rb:91] |
|
|
20
|
+
| INV-02 | Completed steps are undone in reverse completion order, including across independent DAG branches. | one execution's undo stack | HOLDS | [R: …/compensation_manager.rb:69] [O: S-plain-09] | [T: spec/ruby_reactor/order_processing_reactor_spec.rb:91] (linear only; DAG branches: none) |
|
|
21
|
+
| INV-03 | After a step fails, no further step of the same execution runs. | inline and worker | HOLDS | [R: lib/ruby_reactor/executor/step_executor.rb:47] [O: S-plain-01, S-bg-01] | implicit in most failure specs; no dedicated example |
|
|
22
|
+
| INV-04 | A compensate or undo that fails (returns `Failure` or raises) does not stop the remaining undos, and it is listed on `Failure#rollback_failures`. | all | HOLDS | [O: S-plain-03, S-plain-04, S-compose-07] | [T: spec/ruby_reactor/compensation_failure_spec.rb:30] [T: spec/ruby_reactor/step_coordination/rollback_under_contention_spec.rb:51, :59, :67, :78] |
|
|
23
|
+
| INV-05 | `Halt` never rolls back. `Skipped` steps are never undone. | inline | HOLDS | [O: S-plain-05, S-plain-06] | [T: spec/ruby_reactor/halt_status_spec.rb:30] [T: spec/ruby_reactor/skipped_rollback_spec.rb:6] |
|
|
24
|
+
| INV-06 | **Every** failure that occurs after at least one step has completed rolls back the completed steps. | all | HOLDS (008) for every exception, `StandardError` or not (R-16). Only an interruption (signal, exit, out of memory, enclosing timeout) runs no rollback, by design; the caller-process run is stored `aborted` and `Reactor.undo(id)` rolls it back | [O: S-plain-07, S-edge-03, S-edge-03b] | [T: spec/ruby_reactor/rollback/failure_rollback_spec.rb] [T: spec/ruby_reactor/rollback/aborted_execution_spec.rb] |
|
|
25
|
+
| INV-07 | A step whose body never started is never compensated. | all | HOLDS (008) | Own-coordination contention [O: S-lock-03, S-edge-01], refused async dispatch, argument/type validation [O: S-edge-05], and (since 008) argument resolution errors [O: S-plain-07]. `where`/`guard` were removed (008 R-15) [O: S-edge-04] | [T: spec/ruby_reactor/step_coordination/contention_spec.rb:173] [T: spec/ruby_reactor/rollback/failure_rollback_spec.rb] |
|
|
26
|
+
| INV-08 | A step whose body ran and then failed (including an invalid output) is compensated. | same-process steps | HOLDS (exception: async units, INV-24) | [O: S-plain-01, S-plain-08] | [T: spec/ruby_reactor/validations_spec.rb:1139] |
|
|
27
|
+
| INV-09 | Halt semantics are the same at every nesting level: a `Halt` anywhere stops the top-level execution without rollback. | compose, map | **VIOLATED** | A map element's `Halt` halts the parent [O: S-map-11]. A composed child's `Halt` becomes a plain `Success(nil)` and the parent **continues** [O: S-compose-08]. [R: lib/ruby_reactor/step/compose_step.rb:106-114] | none |
|
|
28
|
+
|
|
29
|
+
## B. Retries
|
|
30
|
+
|
|
31
|
+
| Id | Statement | Scope | Status | Evidence | Coverage |
|
|
32
|
+
|---|---|---|---|---|---|
|
|
33
|
+
| INV-10 | Retries are exhausted before compensation. Compensation runs exactly once, after the last attempt. | same-process steps, inline and worker | HOLDS | [O: S-retry-01, S-retry-04, S-bg-03] | [T: spec/ruby_reactor/step_retries/execution_paths_spec.rb:81] [T: spec/map/map_retry_spec.rb:49] |
|
|
34
|
+
| INV-11 | A failed attempt followed by a successful one is never compensated. | same-process steps | HOLDS | [O: S-retry-03, S-bg-04, S-map-09, S-map-10] | [T: spec/ruby_reactor/retry_reexecution_spec.rb:13] [T: spec/ruby_reactor/retry_signals_spec.rb:14] |
|
|
35
|
+
| INV-12 | A `retry: false` failure is not retried. | all | HOLDS | [O: S-retry-02] | [T: spec/ruby_reactor/retry_signals_spec.rb:32] |
|
|
36
|
+
| INV-13 | A retried unit of work re-runs from a state in which its already-rolled-back work is **not** treated as done. | step `retries` (incl. a composed child's steps) | HOLDS (008) | Only steps retry: `retries` on a `compose`/`async_reactor` is rejected at definition time (008 R-14) [O: S-compose-05]. A child's own step retries inside the child and a later failure undoes each child step once [O: S-compose-05b]. A park/resume still resumes | [T: spec/ruby_reactor/rollback/compose_retry_spec.rb] |
|
|
37
|
+
| INV-14 | Retry delivery depends only on where the step runs. Inline: in-process `sleep`. Worker/background: re-enqueue. Fan-out element: element re-enqueue. `async_step`: loop inside its job. | all | HOLDS (descriptive) | [O: S-retry-01, S-bg-03, S-map-09, S-map-10, S-async-07] | [T: spec/async_retry_integration_spec.rb:75] |
|
|
38
|
+
|
|
39
|
+
## C. Composition
|
|
40
|
+
|
|
41
|
+
| Id | Statement | Scope | Status | Evidence | Coverage |
|
|
42
|
+
|---|---|---|---|---|---|
|
|
43
|
+
| INV-15 | A failing composed child rolls back its own completed steps **before** the parent starts its rollback. | compose, any depth | HOLDS | [O: S-compose-01, S-compose-04] | [T: spec/compose_spec.rb:197] |
|
|
44
|
+
| INV-16 | A completed composed child is fully undone (all its completed steps, reverse order) when a later parent step fails. This includes **earlier sibling composes** when a later compose fails. | compose | HOLDS | [O: S-compose-02, S-compose-03, S-compose-06] | [T: spec/compose_spec.rb:192] (single child; sibling case: none) |
|
|
45
|
+
| INV-17 | Nesting unwinds innermost-first. Nesting depth never changes the rule. | compose ⊂ compose, map ⊂ compose, compose ⊂ map element | HOLDS | [O: S-compose-04, S-map-07, S-map-08] | none |
|
|
46
|
+
| INV-18 | A composed child's rollback failures reach the parent's `rollback_failures`. | compose | HOLDS | [O: S-compose-07] | [T: spec/ruby_reactor/step_coordination/rollback_under_contention_spec.rb:78] |
|
|
47
|
+
|
|
48
|
+
## D. Map
|
|
49
|
+
|
|
50
|
+
| Id | Statement | Scope | Status | Evidence | Coverage |
|
|
51
|
+
|---|---|---|---|---|---|
|
|
52
|
+
| INV-19 | When a map fails, its elements that already **succeeded** are rolled back. | map, fail_fast | HOLDS (008) | `MapStep#compensate` replays every completed element's undo stack, highest index first [O: S-map-01, S-map-04, S-map-04b, S-map-07, S-map-08] | [T: spec/ruby_reactor/rollback/map_rollback_spec.rb] [T: spec/ruby_reactor/rollback/map_fan_out_settle_spec.rb] |
|
|
53
|
+
| INV-20 | When a step **after** a completed map fails, the map's elements are rolled back. | map | HOLDS (008) | The map step's `undo` is the same element replay; a fan-out map is pushed on the undo stack by the collector [O: S-map-03, S-map-06] | [T: spec/ruby_reactor/rollback/map_rollback_spec.rb] [T: spec/ruby_reactor/rollback/map_fan_out_settle_spec.rb] |
|
|
54
|
+
| INV-21 | A failed element rolls back its own completed steps (compensate failing step, undo the rest) regardless of mode. | map element | HOLDS | [O: S-map-01, S-map-02, S-map-04, S-map-05, S-map-08] | none dedicated |
|
|
55
|
+
| INV-22 | *Restated in 008.* After a fail-fast failure, the set of elements **left in place** is deterministic: it is empty. (Which elements run in fan-out mode still depends on job scheduling, inherently.) | map, fail_fast | HOLDS (008) | The collector applies the failure only once every index has settled, then rolls back every completed element [O: S-map-04, S-map-04b] | [T: spec/ruby_reactor/rollback/map_fan_out_settle_spec.rb] (100 shuffled job orders, SC-003) |
|
|
56
|
+
| INV-23 | `fail_fast false`: the map step succeeds even when elements fail. Failed elements are rolled back individually, successes kept, and the consumer must inspect the results. | map | HOLDS (by design) | [O: S-map-02, S-map-05] | [T: spec/map/map_fail_fast_spec.rb:77] [T: spec/map/fail_fast_spec.rb:172, :196] |
|
|
57
|
+
|
|
58
|
+
## E. Async units (`async_step`, `async_reactor`)
|
|
59
|
+
|
|
60
|
+
| Id | Statement | Scope | Status | Evidence | Coverage |
|
|
61
|
+
|---|---|---|---|---|---|
|
|
62
|
+
| INV-24 | *Restated in 008.* A unit's own `compensate` runs once, in its own job, after its final attempt fails, whether or not a reader surfaces it; an `undo` on `async_step` is rejected (inline) or warned (class). | async_step | HOLDS (008) | [O: S-async-01, S-async-02, S-async-07] | [T: spec/ruby_reactor/rollback/async_step_compensate_spec.rb] [T: spec/ruby_reactor/dsl/async_step_spec.rb:111] (tightened) |
|
|
63
|
+
| INV-25 | A unit never enters the parent's undo stack, so parent rollback never touches it. | async_step, async_reactor | HOLDS (by design, documented) | [R: lib/ruby_reactor/executor/result_handler.rb:137] [O: S-async-03, S-async-06] | [T: spec/ruby_reactor/dsl/async_reactor_spec.rb:26, :123] |
|
|
64
|
+
| INV-26 | A unit's failure fails the parent only if a reader returns `Failure`. | async_step, async_reactor | HOLDS (by design) | [O: S-async-01, S-async-02, S-async-04, S-async-05] | [T: spec/ruby_reactor/dsl/async_step_spec.rb:98] [T: spec/ruby_reactor/dsl/async_reactor_spec.rb:16, :77] |
|
|
65
|
+
| INV-27 | A unit dispatched by an execution that then fails and rolls back does not perform its side effect **after** that rollback. | async_step, async_reactor, async_step inside a map element | **VIOLATED** | Unit bodies run after the parent's undo [O: S-async-03, S-async-06, S-async-08, S-map-12]. In production the relative order is a race | none |
|
|
66
|
+
| INV-28 | An `async_reactor` child rolls back its own steps on its own failure, in its worker. | async_reactor | HOLDS | [O: S-async-04, S-async-05] | [T: spec/ruby_reactor/dsl/async_reactor_spec.rb:16] |
|
|
67
|
+
| INV-29 | An async unit's retries behave like a same-process step's (one compensation at exhaustion, retry middleware events). | async_step | **VIOLATED** (compensation half fixed in 008) | One compensation at exhaustion since 008 [O: S-async-07]. Retries still loop inside the job with no `retry_attempt` event [R: lib/ruby_reactor/step_worker.rb:275-289] | [T: spec/ruby_reactor/dsl/async_step_spec.rb:123] (attempt count only) [T: spec/ruby_reactor/rollback/async_step_compensate_spec.rb] |
|
|
68
|
+
| INV-30 | A reader that cannot get the unit's result in time fails and rolls back the parent. | async_step reader | HOLDS | [O: S-async-08] | [T: spec/ruby_reactor/async_waiter_spec.rb] (waiter timeout only) |
|
|
69
|
+
|
|
70
|
+
## F. background / worker parity
|
|
71
|
+
|
|
72
|
+
| Id | Statement | Scope | Status | Evidence | Coverage |
|
|
73
|
+
|---|---|---|---|---|---|
|
|
74
|
+
| INV-31 | A worker-side run produces the same order and rollback coverage as an inline run of the same shape, including steps that ran in the caller before a `background after:`/`before:` hand-off. | background, fan-out map, compose in worker | HOLDS | [O: S-bg-01, S-bg-02, S-compose-06, S-map-04] (vs S-map-01) | [T: spec/ruby_reactor/step_retries/execution_paths_spec.rb:81] |
|
|
75
|
+
| INV-32 | A worker crash re-drives from the last checkpoint. Completed steps are not re-run, and the in-flight step may run twice (at-least-once). | background | HOLDS | [O: S-edge-02] [R: lib/ruby_reactor/executor.rb:55] | [T: spec/ruby_reactor/checkpoint_spec.rb:85] |
|
|
76
|
+
|
|
77
|
+
## G. Locks & coordination
|
|
78
|
+
|
|
79
|
+
| Id | Statement | Scope | Status | Evidence | Coverage |
|
|
80
|
+
|---|---|---|---|---|---|
|
|
81
|
+
| INV-33 | A reactor-level lock/semaphore is held from before the first step until after the last undo. | reactor `with_lock`/`with_semaphore` | HOLDS | [O: S-lock-01] [R: lib/ruby_reactor/executor.rb:161] | [T: spec/ruby_reactor/telemetry_spec.rb:353] (events only) |
|
|
82
|
+
| INV-34 | A reactor-level lock is released while the execution is paused at an interrupt and re-acquired on `continue`. | interrupts | HOLDS | [O: S-intr-01] | none |
|
|
83
|
+
| INV-35 | Every rollback of a step is serialized with forward runs by the same locks the forward run held. | step locks, `Reactor.undo` | **CONDITIONAL** | Step-level lock/semaphore is re-taken around compensate/undo [O: S-lock-02, S-lock-04]. The **reactor-level** lock is **not** taken by `Reactor.undo(id)` [O: S-intr-04] [R: lib/ruby_reactor/reactor.rb:188] | [T: spec/ruby_reactor/step_coordination/rollback_spec.rb:31, :52]; Reactor.undo: none |
|
|
84
|
+
| INV-36 | A rollback that cannot re-take its step lock within `rollback_wait` is skipped **and reported**, never silently dropped. | step locks | HOLDS | [O: S-lock-05] | [T: spec/ruby_reactor/step_coordination/rollback_under_contention_spec.rb:32] |
|
|
85
|
+
| INV-37 | A composed child re-enters the parent's reactor lock without releasing the parent's hold. | compose + reactor lock | HOLDS | [O: S-lock-06] (count back to 1 in `b`) | [T: spec/ruby_reactor/step_coordination/reentrancy_spec.rb:53] |
|
|
86
|
+
| INV-38 | Lock middleware events name the same key on acquire and release. | reactor-level lock | **VIOLATED** (known, pinned) | `lock_acquired:rk` vs `lock_released:lock:rk` [O: S-lock-01] | [T: spec/ruby_reactor/step_coordination/attribution_spec.rb:41] (documents the asymmetry) |
|
|
87
|
+
|
|
88
|
+
## H. Interrupts & manual control
|
|
89
|
+
|
|
90
|
+
| Id | Statement | Scope | Status | Evidence | Coverage |
|
|
91
|
+
|---|---|---|---|---|---|
|
|
92
|
+
| INV-39 | Steps completed before a pause are undone when the run fails after `continue`. | interrupts | HOLDS | [O: S-intr-01] | [T: spec/ruby_reactor/interrupt_undo_spec.rb:77] (explicit undo) |
|
|
93
|
+
| INV-40 | Invalid interrupt payload past `max_attempts` rolls back completed steps. | interrupts | HOLDS | [O: S-intr-02] | [T: spec/integration/interrupt_max_attempts_spec.rb:9] |
|
|
94
|
+
| INV-41 | `Reactor.cancel` never rolls back. `Reactor.undo(id)` rolls back and then cancels. | manual | HOLDS | [O: S-intr-03, S-intr-04] | [T: spec/ruby_reactor/interrupt_undo_spec.rb:77, :94] |
|
|
95
|
+
|
|
96
|
+
## Coverage summary
|
|
97
|
+
|
|
98
|
+
| Status | Count | Ids |
|
|
99
|
+
|---|---|---|
|
|
100
|
+
| HOLDS | 36 | INV-01–08, 10–26, 28, 30–34, 36, 37, 39–41 |
|
|
101
|
+
| VIOLATED | 4 | INV-09, 27, 29, 38 |
|
|
102
|
+
| CONDITIONAL | 1 | INV-35 |
|
|
103
|
+
| UNDETERMINED | 0 | — |
|
|
104
|
+
|
|
105
|
+
(41 invariants in total. INV-38's violation is known and pinned by a spec.)
|
|
106
|
+
|
|
107
|
+
**No existing spec covers** (after 008): INV-09, INV-27, INV-34, the `Reactor.undo` half of
|
|
108
|
+
INV-35, the DAG half of INV-02, and the sibling-compose half of INV-16. INV-17 is now covered by
|
|
109
|
+
the nested map/compose examples in `spec/ruby_reactor/rollback/map_rollback_spec.rb`.
|