ruby_reactor 0.8.6 → 0.8.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/.specify/feature.json +1 -1
- data/CHANGELOG.md +81 -0
- data/CLAUDE.md +1 -1
- data/README.md +42 -3
- data/lib/ruby_reactor/dsl/map_builder.rb +20 -1
- data/lib/ruby_reactor/error/recorded_failure.rb +18 -0
- data/lib/ruby_reactor/executor/compensation_manager.rb +27 -1
- data/lib/ruby_reactor/executor/result_handler.rb +16 -0
- data/lib/ruby_reactor/executor/retry_manager.rb +2 -2
- data/lib/ruby_reactor/executor/step_executor.rb +20 -0
- data/lib/ruby_reactor/executor.rb +62 -10
- data/lib/ruby_reactor/interrupt_claims.rb +81 -0
- data/lib/ruby_reactor/map/step_rollback.rb +121 -0
- data/lib/ruby_reactor/reactor.rb +270 -105
- data/lib/ruby_reactor/rspec/matchers.rb +64 -16
- data/lib/ruby_reactor/rspec/test_subject.rb +25 -24
- data/lib/ruby_reactor/step/compose_step.rb +10 -0
- data/lib/ruby_reactor/step/map_step.rb +12 -0
- data/lib/ruby_reactor/storage/adapter.rb +17 -0
- data/lib/ruby_reactor/storage/redis_adapter.rb +42 -0
- data/lib/ruby_reactor/version.rb +1 -1
- data/lib/ruby_reactor/web/api.rb +17 -0
- data/lib/ruby_reactor/web/public/assets/{index-C6ejszCQ.js → index-DxjQlW7Q.js} +8 -8
- data/lib/ruby_reactor/web/public/index.html +1 -1
- data/lib/ruby_reactor/worker.rb +73 -6
- data/specs/010-interrupt-in-composed-child/checklists/requirements.md +43 -0
- data/specs/010-interrupt-in-composed-child/contracts/api-surface.md +75 -0
- data/specs/010-interrupt-in-composed-child/data-model.md +40 -0
- data/specs/010-interrupt-in-composed-child/plan.md +149 -0
- data/specs/010-interrupt-in-composed-child/quickstart.md +64 -0
- data/specs/010-interrupt-in-composed-child/research.md +207 -0
- data/specs/010-interrupt-in-composed-child/spec.md +267 -0
- data/specs/010-interrupt-in-composed-child/tasks.md +315 -0
- data/specs/010-rollback-follow-ups/checklists/requirements.md +47 -0
- data/specs/010-rollback-follow-ups/contracts/api-surface.md +123 -0
- data/specs/010-rollback-follow-ups/contracts/resume-protocol.md +132 -0
- data/specs/010-rollback-follow-ups/data-model.md +139 -0
- data/specs/010-rollback-follow-ups/plan.md +273 -0
- data/specs/010-rollback-follow-ups/pr-description.md +110 -0
- data/specs/010-rollback-follow-ups/quickstart.md +157 -0
- data/specs/010-rollback-follow-ups/research.md +440 -0
- data/specs/010-rollback-follow-ups/spec.md +489 -0
- data/specs/010-rollback-follow-ups/tasks.md +1022 -0
- data/specs/future_improvements.md +17 -14
- metadata +22 -2
|
@@ -0,0 +1,315 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: "Task list for 010 Interrupt Inside a Composed Child"
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# Tasks: Interrupt Inside a Composed Child
|
|
6
|
+
|
|
7
|
+
**Input**: Design documents from `specs/010-interrupt-in-composed-child/`
|
|
8
|
+
|
|
9
|
+
**Prerequisites**: plan.md, spec.md, research.md, data-model.md, contracts/api-surface.md,
|
|
10
|
+
quickstart.md
|
|
11
|
+
|
|
12
|
+
**Tests**: Required. Constitution III makes tests test-first on real Redis. Each story's tests are
|
|
13
|
+
written and seen failing before its implementation tasks.
|
|
14
|
+
|
|
15
|
+
**Organization**: Grouped by user story (spec.md). R-xx refers to research.md, FR-xxx to spec.md.
|
|
16
|
+
|
|
17
|
+
## Format: `[ID] [P?] [Story] Description`
|
|
18
|
+
|
|
19
|
+
- **[P]**: can run in parallel (different files, no dependency on an incomplete task)
|
|
20
|
+
- **[Story]**: US1–US4 from spec.md
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Phase 1: Setup (Shared Fixtures)
|
|
25
|
+
|
|
26
|
+
**Purpose**: The reactors every story's specs use.
|
|
27
|
+
|
|
28
|
+
- [X] T001 Create `spec/ruby_reactor/interrupt_in_compose_spec.rb`, with its fixtures in `spec/support/interrupt_in_compose_fixtures.rb` (`module InterruptInComposeFixtures`, shared with the helpers spec in T033). The fixtures are built on `RollbackRecorder::Reactor` (`spec/support/rollback_recorder.rb`).
|
|
29
|
+
- `Child` (tag `"child"`): `input :seed, optional: true`; `recording_step :c1`; `interrupt :approve` (`wait_for :c1`, `correlation_id { |ctx| "approve-#{ctx.inputs[:seed]}" }`); `recording_step :c2, after: :approve`. `c2` records the approve payload: `argument :decision, result(:approve)` through the `&extra` block.
|
|
30
|
+
- `Root`: `recording_step :r1`; `compose :fulfil, Child` (`argument :seed, result(:r1)`); `recording_step :r2, after: :fulfil`.
|
|
31
|
+
- `Middle`: like `Root`, composing `Child` at `:fulfil`. `Top`: composes `Middle` at `:order`, then `recording_step :t2`.
|
|
32
|
+
- `TwoInterruptsChild`: `c1`, `interrupt :approve`, `interrupt :sign` (`wait_for :approve`), `c2`; and `TwoInterruptsRoot` composing it.
|
|
33
|
+
- `RootWithOwnInterrupt`: `r1`, `interrupt :audit` (`wait_for :r1`) and `compose :fulfil, Child` both ready after `r1`, then `r2` after both.
|
|
34
|
+
- `BackgroundChild`: `Child` with `interrupt :approve, resume: :background`; `BackgroundRoot` composing it.
|
|
35
|
+
- `ValidatedChild`: `interrupt :approve` with `validate_payload { required(:ok).filled(:bool) }` and `max_attempts: 1`; `ValidatedRetryChild` the same with `max_attempts: 3`; a root composing each.
|
|
36
|
+
- `before { RollbackRecorder.reset! }`, and a `drain` helper as in `spec/map/map_compose_fan_out_spec.rb`.
|
|
37
|
+
- [X] T002 [P] Add to `MapComposeFanOutSpec` in `spec/map/map_compose_fan_out_spec.rb`: a child fixture `InterruptChild` (same as `Child`, plus `interrupt :approve` after `:m`, with `c2` waiting for `:approve`) and `root :InterruptRoot, InterruptChild`.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Phase 2: Foundational
|
|
42
|
+
|
|
43
|
+
None. US1's pass-through is the base the other stories build on, and it is small enough to live in
|
|
44
|
+
US1.
|
|
45
|
+
|
|
46
|
+
---
|
|
47
|
+
|
|
48
|
+
## Phase 3: User Story 1 - Pause the root at an interrupt inside a composed child (Priority: P1) 🎯 MVP
|
|
49
|
+
|
|
50
|
+
**Goal**: A root whose composed child reaches an `interrupt` is stored `paused` and returns an
|
|
51
|
+
`InterruptResult` naming the root (FR-001–FR-004).
|
|
52
|
+
|
|
53
|
+
**Independent Test**: `InterruptInComposeSpec::Root.run({})`:
|
|
54
|
+
|
|
55
|
+
- the result is `paused?`, with `execution_id` equal to the root's id;
|
|
56
|
+
- `Root.find(id).context.status == "paused"`;
|
|
57
|
+
- the log is exactly `["run:r1", "run:child.c1"]`, with no undo events.
|
|
58
|
+
|
|
59
|
+
### Tests for User Story 1 (write first, confirm failing)
|
|
60
|
+
|
|
61
|
+
- [X] T003 [US1] In `spec/ruby_reactor/interrupt_in_compose_spec.rb`, describe "pausing" (US1-AS1). Running `Root`:
|
|
62
|
+
- returns an `InterruptResult`, with `paused?` true and `execution_id == Root.find(result.execution_id).context.context_id`;
|
|
63
|
+
- leaves the stored status `"paused"`, with `current_step` `:fulfil`;
|
|
64
|
+
- leaves the log at `%w[run:r1 run:child.c1]`.
|
|
65
|
+
- [X] T004 [US1] In the same file, assert that the paused result's `correlation_id` is `"approve-r1-value"` (the seed is `r1`'s value), and that `Root.find_by_correlation_id("approve-r1-value").context.context_id` is the root's id (US1-AS2, FR-004).
|
|
66
|
+
- [X] T005 [US1] In the same file, assert that running `Top` stores `Top` as `"paused"`, returns `Top`'s id, and that the log stops after `run:child.c1` (US1-AS4, FR-002).
|
|
67
|
+
- [X] T006 [P] [US1] In `spec/map/map_compose_fan_out_spec.rb`, replace the `pending` example "resumes the root once the child's interrupt after the map is continued" with a real one. Under `for_each_async_backend`:
|
|
68
|
+
- `run(MapComposeFanOutSpec::InterruptRoot)` leaves the root `"paused"`;
|
|
69
|
+
- the root's `composed_contexts[:c][:context].current_step` is `:approve`;
|
|
70
|
+
- no `run:child.c2` was logged.
|
|
71
|
+
|
|
72
|
+
The resume half is added in T024 (US1-AS3, FR-015).
|
|
73
|
+
|
|
74
|
+
### Implementation for User Story 1
|
|
75
|
+
|
|
76
|
+
- [X] T007 [P] [US1] In `lib/ruby_reactor/step/compose_step.rb`, `handle_execution_result` returns `RubyReactor::InterruptResult.new(execution_id: context.context_id, correlation_id: result.correlation_id, intermediate_results: context.intermediate_results)` when `result.is_a?(RubyReactor::InterruptResult)`. Put it before the `success?` line, with a comment: each compose level re-stamps its own id, so the top-level result names the root (R-02).
|
|
77
|
+
- [X] T008 [P] [US1] In `lib/ruby_reactor/executor/retry_manager.rb`, `handle_retry_result` passes `RubyReactor::InterruptResult` through unchanged. Add it to the `when RetryQueuedResult, RubyReactor::DispatchResult` arm (R-01).
|
|
78
|
+
- [X] T009 [P] [US1] In `lib/ruby_reactor/executor/result_handler.rb`, give `handle_step_result` a `when RubyReactor::InterruptResult` arm, before `else`, calling a new private `handle_interrupted(step_config)`. That method sets `@context.current_step = step_config.name`. Comment: `with_step` cleared it, and it is the root's resume cursor (R-03). Nothing is recorded as a result and the graph node stays incomplete.
|
|
79
|
+
- [X] T010 [US1] Run `bundle exec rspec spec/ruby_reactor/interrupt_in_compose_spec.rb spec/map/map_compose_fan_out_spec.rb spec/compose_spec.rb spec/ruby_reactor/interrupt_spec.rb`. T003–T006 must pass and nothing else may regress.
|
|
80
|
+
|
|
81
|
+
**Checkpoint**: Roots pause at any depth, sync and after a fan-out map.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Phase 4: User Story 2 - Resume the root by naming the nested interrupt (Priority: P1)
|
|
86
|
+
|
|
87
|
+
**Goal**: `continue` and `continue_by_correlation_id` on the root accept an Array path and finish
|
|
88
|
+
the run. Wrong names and direct child resumes are refused (FR-005–FR-011).
|
|
89
|
+
|
|
90
|
+
**Independent Test**: Pause `Root`, then `Root.continue(id:, payload: { ok: true }, step_name:
|
|
91
|
+
[:fulfil, :approve])`. Expect:
|
|
92
|
+
|
|
93
|
+
- status `"completed"`;
|
|
94
|
+
- log `%w[run:r1 run:child.c1 run:child.c2 run:r2]`;
|
|
95
|
+
- `c2` received the payload.
|
|
96
|
+
|
|
97
|
+
### Tests for User Story 2 (write first, confirm failing)
|
|
98
|
+
|
|
99
|
+
- [X] T011 [US2] In `spec/ruby_reactor/interrupt_in_compose_spec.rb`, describe "resuming" (US2-AS1, FR-008/FR-009):
|
|
100
|
+
- continuing with `[:fulfil, :approve]` completes the root;
|
|
101
|
+
- each step logs `run:` exactly once across pause and resume;
|
|
102
|
+
- `c2`'s stored result reflects the payload;
|
|
103
|
+
- string paths (`["fulfil", "approve"]`) work the same.
|
|
104
|
+
- [X] T012 [US2] In the same file, assert that `Root.continue_by_correlation_id(correlation_id: "approve-r1-value", payload:, step_name: [:fulfil, :approve])` completes the root (US2-AS2).
|
|
105
|
+
- [X] T013 [US2] In the same file, assert each of these raises `RubyReactor::Error::ValidationError` whose message includes `[:fulfil, :approve]`, leaving the status `"paused"` and the log unchanged (US2-AS4, FR-007, FR-011):
|
|
106
|
+
- `step_name: :approve`;
|
|
107
|
+
- `step_name: [:fulfil, :nope]`;
|
|
108
|
+
- `step_name: [:r2, :approve]`;
|
|
109
|
+
- `step_name: :r2`.
|
|
110
|
+
- [X] T014 [US2] In the same file, assert that a direct child resume raises `ValidationError` matching `/composed child/` and runs nothing (FR-005). Cover both:
|
|
111
|
+
- `Child.continue(id: <root>.context.composed_contexts[:fulfil][:context].context_id, …)`;
|
|
112
|
+
- `Child.continue_by_correlation_id(correlation_id: "approve-r1-value", …)`.
|
|
113
|
+
- [X] T015 [US2] In the same file, under `for_each_async_backend`, assert that continuing `BackgroundRoot` (US2-AS3, FR-010):
|
|
114
|
+
- returns a `DispatchResult`;
|
|
115
|
+
- completes after `drain`;
|
|
116
|
+
- calls the async router's `perform_async` with the root's id, and never with the child's id (pattern as in `spec/map/map_compose_fan_out_spec.rb` "signals the root's Worker").
|
|
117
|
+
- [X] T016 [US2] In the same file, assert that continuing the `ValidatedRetryChild` root with `{ ok: "x" }` (FR-010):
|
|
118
|
+
- raises `InputValidationError` from the class method;
|
|
119
|
+
- keeps the run `"paused"`;
|
|
120
|
+
- stores `private_data[:interrupt_attempts]["fulfil.approve"] == 1`;
|
|
121
|
+
- lets a valid payload then complete it.
|
|
122
|
+
- [X] T017 [US2] In the same file, cover multi-interrupt flows:
|
|
123
|
+
- `TwoInterruptsRoot`: continue `[:fulfil, :approve]`, which re-pauses with `ready_interrupt_steps == [[:fulfil, :sign]]`; continue `[:fulfil, :sign]`, which completes (US2-AS6).
|
|
124
|
+
- `RootWithOwnInterrupt`: `ready_interrupt_steps` contains both `:audit` and `[:fulfil, :approve]`; continuing each in turn completes the run (edge case).
|
|
125
|
+
- [X] T018 [US2] In the same file, assert that `Top`'s `ready_interrupt_steps == [[:order, :fulfil, :approve]]`, and that continuing with that path completes `Top` with `run:t2` logged once (FR-006).
|
|
126
|
+
- [X] T019 [US2] In the same file, assert that a `continue` while the resume is executing is refused with the existing "not paused" message (edge case, 008 FR-032). Stub `c2`'s run, via an `&extra` hook, to call `Root.continue` again with the same path, and expect `ValidationError` matching `/running, not paused/`.
|
|
127
|
+
|
|
128
|
+
### Implementation for User Story 2
|
|
129
|
+
|
|
130
|
+
- [X] T020 [P] [US2] In `lib/ruby_reactor/step/compose_step.rb`, `prepare_child_context` sets `child_context.private_data[:composed] = true` when it builds a new child context (R-07).
|
|
131
|
+
- [X] T021 [US2] In `lib/ruby_reactor/reactor.rb`, add public `ready_interrupt_steps` (R-05):
|
|
132
|
+
- Return `[]` unless `@context.status.to_s == "paused"`.
|
|
133
|
+
- Otherwise return `pending_interrupts(self.class, @context, [])`, a private recursive helper. It builds a `GraphManager` (as `validate_continue_step!` does), takes the ready steps and collects:
|
|
134
|
+
- for each `interrupt?` step, `prefix.empty? ? name : prefix + [name]`;
|
|
135
|
+
- for each ready step whose `composed_contexts[name]` is `type: :composed` with a `paused` `:context`, the recursion into `(child.reactor_class, child, prefix + [name])`.
|
|
136
|
+
- Read hash keys indifferently (symbol or string), since stored `composed_contexts` round-trip through the serializer.
|
|
137
|
+
- [X] T022 [US2] In `lib/ruby_reactor/reactor.rb`, `#continue` (R-06, R-07):
|
|
138
|
+
- **Child guard**: right after the `current_step` guard, raise `Error::ValidationError, "Cannot resume: #{self.class.name} is a composed child; continue its root run"` when `@context.private_data[:composed] || @context.private_data["composed"]`.
|
|
139
|
+
- **Path normalization**: `path = Array(step_name).map(&:to_sym)`. A one-element path is a bare name.
|
|
140
|
+
- **Bare names** keep today's `validate_continue_step!`.
|
|
141
|
+
- **Paths**:
|
|
142
|
+
- raise `Error::ValidationError, "Cannot resume: #{path.inspect} is not a pending interrupt; pending: #{ready_interrupt_steps.inspect}"` unless `ready_interrupt_steps.include?(path)`;
|
|
143
|
+
- then walk `composed_contexts[...][:context]` down `path[0..-2]` to get `target_context`, and take `step_config = target_context.reactor_class.steps[path.last]`.
|
|
144
|
+
- **Callers to update**:
|
|
145
|
+
- `validate_continue_payload` takes `step_config` and an attempts key: `path.join(".")` for a path, the Symbol for a bare name;
|
|
146
|
+
- `set_result` goes to `target_context` with `path.last`;
|
|
147
|
+
- `background_resume?` reads `step_config`.
|
|
148
|
+
- Root status, save and `resume_execution` stay as they are.
|
|
149
|
+
- [X] T023 [US2] Run `bundle exec rspec spec/ruby_reactor/interrupt_in_compose_spec.rb spec/ruby_reactor/interrupt_spec.rb spec/ruby_reactor/multiple_interrupts_spec.rb spec/ruby_reactor/interrupt_background_resume_spec.rb spec/integration/interrupt_validation_spec.rb spec/integration/interrupt_max_attempts_spec.rb`. T011–T019 must pass with no regression.
|
|
150
|
+
- [X] T024 [US2] In `spec/map/map_compose_fan_out_spec.rb`, extend the T006 example: `InterruptRoot.continue(id:, payload: {}, step_name: [:c, :approve])`, `drain`, then expect `"completed"` and `r2` == `"r2-value"` (FR-015).
|
|
151
|
+
|
|
152
|
+
**Checkpoint**: The MVP (US1 + US2) is usable: pause and resume through a compose.
|
|
153
|
+
|
|
154
|
+
---
|
|
155
|
+
|
|
156
|
+
## Phase 5: User Story 3 - Undo or cancel a run paused inside a child (Priority: P2)
|
|
157
|
+
|
|
158
|
+
**Goal**: Undo reaches the child's completed steps, cancel blocks resumes, and `max_attempts`
|
|
159
|
+
exhaustion rolls back from the root (FR-012, FR-013, US2-AS5).
|
|
160
|
+
|
|
161
|
+
**Independent Test**: Pause `Root`, then `Root.undo(id)`. The log tail is `%w[undo:child.c1
|
|
162
|
+
undo:r1]`, each once, and the status is `"cancelled"`.
|
|
163
|
+
|
|
164
|
+
### Tests for User Story 3 (write first, confirm failing)
|
|
165
|
+
|
|
166
|
+
- [X] T025 [US3] In `spec/ruby_reactor/interrupt_in_compose_spec.rb`, describe "undo and cancel":
|
|
167
|
+
- Undoing a paused `Root` logs `undo:child.c1` before `undo:r1`, each exactly once, and ends `"cancelled"` (US3-AS1).
|
|
168
|
+
- Undoing a paused `Top` reaches `undo:child.c1` too.
|
|
169
|
+
- [X] T026 [US3] In the same file, assert that a `Root` paused, resumed to completion, then undone logs each of `undo:child.c2`, `undo:child.c1`, `undo:r2` and `undo:r1` exactly once. This proves the partial-run entry was replaced, not duplicated (R-04).
|
|
170
|
+
- [X] T027 [US3] In the same file, assert that `Root.cancel(id:, reason: "no")` on a paused root makes a later `continue` with `[:fulfil, :approve]` raise `ValidationError` matching `/cancelled/`, with no `run:` logged (US3-AS3, FR-013).
|
|
171
|
+
- [X] T028 [US3] In the same file, assert that continuing the `ValidatedChild` root (`max_attempts: 1`) with `{ ok: "x" }` (US2-AS5, FR-010):
|
|
172
|
+
- returns a `Failure`;
|
|
173
|
+
- logs `undo:child.c1` before `undo:r1`;
|
|
174
|
+
- leaves the stored status `"failed"`.
|
|
175
|
+
- [X] T029 [P] [US3] In `spec/map/map_compose_fan_out_spec.rb`, under "rollback through the root", add an example (US3-AS2). Pause `InterruptRoot`, `RollbackRecorder.reset!`, `InterruptRoot.undo(id)`, `drain`, then expect:
|
|
176
|
+
- `"cancelled"`;
|
|
177
|
+
- `element_undos.size == 8`, each once;
|
|
178
|
+
- `undo:child.c1` then `undo:r1` last.
|
|
179
|
+
|
|
180
|
+
### Implementation for User Story 3
|
|
181
|
+
|
|
182
|
+
- [X] T030 [US3] In `lib/ruby_reactor/executor/result_handler.rb`, `handle_interrupted` also pushes `{ step: step_config, arguments: step_config.rollback_arguments(resolved_arguments), result: RubyReactor.Success(nil) }` with `@compensation_manager.add_to_undo_stack` when `step_config.undoes_partial_run?`. This is the same entry as `StepExecutor#track_interrupted_construct`. Pass `resolved_arguments` through from `handle_step_result` (R-04).
|
|
183
|
+
- [X] T031 [US3] In `lib/ruby_reactor/executor/compensation_manager.rb`, `add_to_undo_stack` pops the top entry first when it belongs to the same step (`undo_stack.last && undo_stack.last[:step].name == step_info[:step].name`). Add a `ponytail:` comment naming the assumption: a step's entry is on top when it re-runs, because a run stops at a pause (R-04).
|
|
184
|
+
- [X] T032 [US3] Run `bundle exec rspec spec/ruby_reactor/interrupt_in_compose_spec.rb spec/map/map_compose_fan_out_spec.rb spec/ruby_reactor/interrupt_undo_spec.rb spec/ruby_reactor/rollback`. T025–T029 must pass with no regression.
|
|
185
|
+
|
|
186
|
+
**Checkpoint**: Saga integrity holds for nested pauses.
|
|
187
|
+
|
|
188
|
+
---
|
|
189
|
+
|
|
190
|
+
## Phase 6: User Story 4 - Test a nested pause with the RSpec helpers (Priority: P3)
|
|
191
|
+
|
|
192
|
+
**Goal**: `be_paused_at`, `have_ready_interrupts`, `ready_interrupt_steps` and `resume(step:)`
|
|
193
|
+
accept paths (FR-014).
|
|
194
|
+
|
|
195
|
+
**Independent Test**: `test_reactor(InterruptInComposeSpec::Root)`. Expect:
|
|
196
|
+
|
|
197
|
+
- `be_paused_at([:fulfil, :approve])` passes;
|
|
198
|
+
- `be_paused_at(:approve)` fails, with a message containing `[:fulfil, :approve]`;
|
|
199
|
+
- `resume(payload: { ok: true })` reaches `be_success`.
|
|
200
|
+
|
|
201
|
+
### Tests for User Story 4 (write first, confirm failing)
|
|
202
|
+
|
|
203
|
+
- [X] T033 [US4] In `spec/ruby_reactor/rspec/helpers_spec.rb`, add a "nested interrupts" context using `InterruptInComposeSpec::Root` (US4-AS1–AS3):
|
|
204
|
+
- `subject.ready_interrupt_steps == [[:fulfil, :approve]]`;
|
|
205
|
+
- `be_paused_at([:fulfil, :approve])` passes, and `have_ready_interrupts([:fulfil, :approve])` passes;
|
|
206
|
+
- `expect { expect(subject).to be_paused_at(:approve) }.to fail_with(/\[:fulfil, :approve\]/)`;
|
|
207
|
+
- `resume(payload: { ok: true })` with no step, and `resume(step: [:fulfil, :approve], …)`, both end `be_success`.
|
|
208
|
+
|
|
209
|
+
Require the fixture file, or move the `InterruptInComposeSpec` module to `spec/support/interrupt_in_compose_fixtures.rb` if it is needed from two files.
|
|
210
|
+
|
|
211
|
+
### Implementation for User Story 4
|
|
212
|
+
|
|
213
|
+
- [X] T034 [P] [US4] In `lib/ruby_reactor/rspec/test_subject.rb`:
|
|
214
|
+
- `ready_interrupt_steps` delegates to `@reactor_instance.ready_interrupt_steps`, and the duplicated graph code goes;
|
|
215
|
+
- `determine_resume_step` normalizes `step` (`step.is_a?(Array) ? step.map(&:to_sym) : step.to_sym`) before the `include?` check, and prints `step.inspect` in its error.
|
|
216
|
+
- [X] T035 [P] [US4] In `lib/ruby_reactor/rspec/matchers.rb`:
|
|
217
|
+
- `be_paused_at` and `have_ready_interrupts` normalize each expected name the same way;
|
|
218
|
+
- sorting in `have_ready_interrupts` uses `sort_by(&:inspect)`, because Symbols and Arrays do not compare;
|
|
219
|
+
- failure messages print names with `inspect` instead of `":#{s}"`.
|
|
220
|
+
- [X] T036 [US4] Run `bundle exec rspec spec/ruby_reactor/rspec` and the demo interrupt specs (`demo_app/spec/reactors/form_interrupt_reactor_spec.rb`, `webhook_interrupt_reactor_spec.rb`) per quickstart.md §2. T033 must pass with no regression.
|
|
221
|
+
|
|
222
|
+
**Checkpoint**: All four stories work independently.
|
|
223
|
+
|
|
224
|
+
---
|
|
225
|
+
|
|
226
|
+
## Phase 7: Polish & Cross-Cutting Concerns
|
|
227
|
+
|
|
228
|
+
### Map element guard (FR-017, R-08)
|
|
229
|
+
|
|
230
|
+
- [X] T037 Add specs that an interrupt inside a map element fails that element with a message matching `/not supported inside a map element/`, never `NoMethodError`:
|
|
231
|
+
- in `spec/map/map_compose_fan_out_spec.rb`, an element reactor that composes a child with an `interrupt`, run fan-out under `for_each_async_backend`, with the root ending `"failed"`;
|
|
232
|
+
- in `spec/ruby_reactor/interrupt_in_compose_spec.rb`, the same element run as an inline map.
|
|
233
|
+
- [X] T038 In `lib/ruby_reactor/executor/step_executor.rb`, `handle_interrupt_step` returns `RubyReactor::Failure("interrupt :#{step_config.name} is not supported inside a map element", step_name: step_config.name, reactor_name: @reactor_class.name, retryable: false)` when `@context` or any `parent_context` up the chain has `map_metadata`. Use a small private `inside_map_element?` loop. T037 must pass.
|
|
234
|
+
|
|
235
|
+
### Demo app (Constitution VI)
|
|
236
|
+
|
|
237
|
+
- [X] T039 [P] Create `demo_app/app/reactors/manager_approval_reactor.rb`: `ManagerApprovalReactor`, with class-based steps.
|
|
238
|
+
- `input :order_id`;
|
|
239
|
+
- `step :reserve_stock`, with an `undo` that prints/records the release;
|
|
240
|
+
- `interrupt :wait_for_manager` (`wait_for :reserve_stock`, `correlation_id { |ctx| "approval-#{ctx.inputs[:order_id]}" }`, `validate_payload { required(:approved).filled(:bool) }`);
|
|
241
|
+
- `step :confirm_reservation`, which fails when `approved` is false.
|
|
242
|
+
- [X] T040 [P] Create `demo_app/app/reactors/composed_approval_reactor.rb`: `ComposedApprovalReactor`, with class-based steps.
|
|
243
|
+
- `input :order_id`;
|
|
244
|
+
- `step :charge_card`, with an `undo` refund;
|
|
245
|
+
- `compose :approval, ManagerApprovalReactor` (`argument :order_id, input(:order_id)`);
|
|
246
|
+
- `step :ship`, after `:approval`.
|
|
247
|
+
- [X] T041 Add `demo:composed_interrupt` to `demo_app/lib/tasks/demo_reactors.rake`, with `desc` and `[:environment, :flush_redis]`. It prints:
|
|
248
|
+
- the run's status, root id and `ready_interrupt_steps`;
|
|
249
|
+
- an approved `continue` by correlation id with `step_name: [:approval, :wait_for_manager]`, and its final status;
|
|
250
|
+
- a second run undone while paused, its status, and that `reserve_stock` and `charge_card` were undone.
|
|
251
|
+
- [X] T042 Create `demo_app/spec/reactors/composed_approval_reactor_spec.rb` (`type: :reactor`), using only the shipped surface:
|
|
252
|
+
- `test_reactor`, `be_paused_at([:approval, :wait_for_manager])`, `have_ready_interrupts`;
|
|
253
|
+
- `resume(step: [:approval, :wait_for_manager], payload: { approved: true })` then `be_success` and `have_run_step(:ship).after(:approval)`;
|
|
254
|
+
- a rejected resume ending `be_failure`.
|
|
255
|
+
|
|
256
|
+
Run it locally per quickstart.md §2.
|
|
257
|
+
- [X] T043 Run the Docker acceptance `docker compose … run --rm --no-deps demo-app bash -c "bin/rails db:prepare && bin/rails demo:composed_interrupt"` with an isolated compose project (quickstart.md §2). Record the output in the PR description.
|
|
258
|
+
|
|
259
|
+
### Documentation (Constitution Development Workflow)
|
|
260
|
+
|
|
261
|
+
- [X] T044 [P] Update `documentation/composition.md` rule 6: replace "An `interrupt` inside a composed child is not supported yet, with or without a map." with the nested pause behavior, linking to interrupts.md: the root pauses, is resumed by path, and undo reaches the child.
|
|
262
|
+
- [X] T045 [P] Add "Interrupts inside composed reactors" to `documentation/interrupts.md`, after "Resuming Execution". Use class-based examples. Cover:
|
|
263
|
+
- the path form, why it is an Array, and `ready_interrupt_steps`;
|
|
264
|
+
- correlation id on the root class;
|
|
265
|
+
- background resume running in the root's worker;
|
|
266
|
+
- undo and cancel;
|
|
267
|
+
- the composed-child guard;
|
|
268
|
+
- interrupts inside map elements being unsupported.
|
|
269
|
+
- [X] T046 [P] Add one line in `README.md` § *Interrupts (Pause & Resume)* noting that interrupts work inside `compose`d children, resumed by path, with a link to the new interrupts.md section.
|
|
270
|
+
- [X] T047 [P] Add a *Features* entry under *Unreleased* in `CHANGELOG.md`: interrupts inside composed children; Array `step_name` paths; `Reactor#ready_interrupt_steps`; matcher path support; and the map-element interrupt now failing clearly.
|
|
271
|
+
- [X] T048 [P] Remove the "Interrupt inside a composed child" section from `specs/future_improvements.md`.
|
|
272
|
+
|
|
273
|
+
### Gate
|
|
274
|
+
|
|
275
|
+
- [X] T049 Run `bundle exec rspec` and `bundle exec rubocop` at the repo root, and the full `demo_app` spec suite per quickstart.md §2. All must be green. If an async example flakes, rerun it alone first (shared test Redis).
|
|
276
|
+
|
|
277
|
+
---
|
|
278
|
+
|
|
279
|
+
## Dependencies & Execution Order
|
|
280
|
+
|
|
281
|
+
- **Setup (T001–T002)** comes first.
|
|
282
|
+
- **US1 (T003–T010)** depends on Setup, and blocks US2, US3 and US4: nothing can be resumed, undone or matched until the root pauses.
|
|
283
|
+
- **US2 (T011–T024)** depends on US1.
|
|
284
|
+
- **US3 (T025–T032)** depends on US1. T028 (`max_attempts` exhaustion) also needs US2's T022 to resolve the nested step config, so implement it after T022.
|
|
285
|
+
- **US4 (T033–T036)** depends on US2's T021 (`Reactor#ready_interrupt_steps`).
|
|
286
|
+
- **Polish**:
|
|
287
|
+
- the map guard (T037–T038) depends on US1 only;
|
|
288
|
+
- the demo (T039–T043) depends on US1–US4;
|
|
289
|
+
- the docs (T044–T048) can be drafted after US2 and finalized last;
|
|
290
|
+
- T049 runs last.
|
|
291
|
+
|
|
292
|
+
Within each story, the tests are written and seen failing before the implementation tasks.
|
|
293
|
+
|
|
294
|
+
## Parallel Opportunities
|
|
295
|
+
|
|
296
|
+
- T002 can run alongside T001.
|
|
297
|
+
- In US1, T007, T008 and T009 touch three different files.
|
|
298
|
+
- In US2, T020 can run alongside T021/T022 (different file). T021 and T022 share `reactor.rb`, so they run one after the other.
|
|
299
|
+
- In US3, T029 is in a different spec file from T025–T028.
|
|
300
|
+
- In US4, T034 and T035 touch different files.
|
|
301
|
+
- In Polish, T039 and T040 run in parallel, as do T044–T048.
|
|
302
|
+
|
|
303
|
+
```text
|
|
304
|
+
# US1 implementation, after T003–T006 fail:
|
|
305
|
+
T007 compose_step.rb | T008 retry_manager.rb | T009 result_handler.rb
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
## Implementation Strategy
|
|
309
|
+
|
|
310
|
+
1. **MVP = US1 + US2** (both P1). Pausing alone is a dead end: ship them together. Stop after T024
|
|
311
|
+
and validate with quickstart.md §1, rows 1–6.
|
|
312
|
+
2. **US3** next. Undo while paused is a saga guarantee (Constitution II), so it must land before
|
|
313
|
+
release, not as a follow-up.
|
|
314
|
+
3. **US4** after that: test-surface convenience.
|
|
315
|
+
4. **Polish**: map guard, demo with the Docker run, docs, then the full gate.
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Specification Quality Checklist: Rollback and Resume Follow-ups
|
|
2
|
+
|
|
3
|
+
**Purpose**: Validate specification completeness and quality before proceeding to planning
|
|
4
|
+
**Created**: 2026-10-08
|
|
5
|
+
**Feature**: [spec.md](../spec.md)
|
|
6
|
+
|
|
7
|
+
## Content Quality
|
|
8
|
+
|
|
9
|
+
- [x] No implementation details (languages, frameworks, APIs)
|
|
10
|
+
- [x] Focused on user value and business needs
|
|
11
|
+
- [x] Written for non-technical stakeholders
|
|
12
|
+
- [x] All mandatory sections completed
|
|
13
|
+
|
|
14
|
+
## Requirement Completeness
|
|
15
|
+
|
|
16
|
+
- [x] No [NEEDS CLARIFICATION] markers remain
|
|
17
|
+
- [x] Requirements are testable and unambiguous
|
|
18
|
+
- [x] Success criteria are measurable
|
|
19
|
+
- [x] Success criteria are technology-agnostic (no implementation details)
|
|
20
|
+
- [x] All acceptance scenarios are defined
|
|
21
|
+
- [x] Edge cases are identified
|
|
22
|
+
- [x] Scope is clearly bounded
|
|
23
|
+
- [x] Dependencies and assumptions identified
|
|
24
|
+
|
|
25
|
+
## Feature Readiness
|
|
26
|
+
|
|
27
|
+
- [x] All functional requirements have clear acceptance criteria
|
|
28
|
+
- [x] User scenarios cover primary flows
|
|
29
|
+
- [x] Feature meets measurable outcomes defined in Success Criteria
|
|
30
|
+
- [x] No implementation details leak into specification
|
|
31
|
+
|
|
32
|
+
## Notes
|
|
33
|
+
|
|
34
|
+
- The gem's public API (`continue`, `Reactor.run`, `Reactor.undo`, `map`, `undo_all`,
|
|
35
|
+
`DispatchResult`) is the user-facing surface of a library, so naming it is not an implementation
|
|
36
|
+
detail (same convention as specs 008 and 009). Internal classes, storage keys and lock mechanics
|
|
37
|
+
are kept out; the one mechanism mentioned (a liveness lock for caller-process runs) is an example
|
|
38
|
+
in Assumptions, explicitly left to the plan.
|
|
39
|
+
- No clarification markers: each item's direction came from `specs/future_improvements.md`. Defaults
|
|
40
|
+
chosen where it offered two options:
|
|
41
|
+
- US5 accepts a resume for another pending interrupt, rather than returning a retryable error,
|
|
42
|
+
matching US3's hand-off.
|
|
43
|
+
- US1/US2 get targeted fixes; the full fenced-writes proposal is out of scope.
|
|
44
|
+
- `undo_all` leaves partially-run (aborted) elements to per-element replay.
|
|
45
|
+
|
|
46
|
+
`/speckit-clarify` can revisit any of these.
|
|
47
|
+
- Validation pass 1: all items pass.
|
|
@@ -0,0 +1,123 @@
|
|
|
1
|
+
# Contract: Public API Surface
|
|
2
|
+
|
|
3
|
+
Changes to what gem users call, receive, or observe. Internals appear only where a spec pins them.
|
|
4
|
+
|
|
5
|
+
## 1. `Reactor.continue` / `Reactor#continue` / `Reactor.continue_by_correlation_id`
|
|
6
|
+
|
|
7
|
+
Signatures are unchanged: `continue(id:, payload:, step_name:, idempotency_key: nil)` on the class,
|
|
8
|
+
`continue(payload:, step_name:, idempotency_key: nil)` on an instance.
|
|
9
|
+
|
|
10
|
+
### Outcomes
|
|
11
|
+
|
|
12
|
+
| Situation | Today | After 010 |
|
|
13
|
+
| --- | --- | --- |
|
|
14
|
+
| Valid payload, run `paused`, gates free | resumes inline and returns its result (`Success`, `Failure`, `InterruptResult` or `DispatchResult`) | unchanged |
|
|
15
|
+
| Valid payload, the reactor's `with_lock` / `with_semaphore` is held by another run | raises `Lock::AcquisitionError` / `Semaphore::AcquisitionError`; the run stays `paused` | **returns `DispatchResult`**; the run is `running`; a Worker resumes it once the lock frees (FR-009, FR-010) |
|
|
16
|
+
| Valid payload, `resume: :background` interrupt | returns `DispatchResult` | unchanged |
|
|
17
|
+
| Valid payload for a ready interrupt, run `running` (another resume, or the run is executing) | raises `ValidationError` "the reactor is running, not paused" | **returns `DispatchResult`**; the payload is applied once the run is free (FR-017–FR-019) |
|
|
18
|
+
| Second resume of the same interrupt (concurrent or later) | may run twice (inline mode) or race | **raises `ValidationError`** "Cannot resume: interrupt :x was already resumed" (FR-015, FR-020) |
|
|
19
|
+
| Invalid payload | the class method raises `InputValidationError`; the instance method returns `Failure(invalid_payload: true)`; one attempt counted | unchanged; the attempt is now counted in its own record (data-model §3) |
|
|
20
|
+
| Invalid payload at `max_attempts` | undoes, then saves `failed` (two unlocked writes) | same outcome, under the run's lock (R-08, R-09); may raise `Lock::AcquisitionError` if another execution holds the run for more than 5s |
|
|
21
|
+
| Interrupt not ready | raises `ValidationError` | unchanged |
|
|
22
|
+
| Run finished, `aborted`, `rolling_back`, or cancelled | raises `ValidationError` | unchanged (008 FR-032) |
|
|
23
|
+
|
|
24
|
+
A `DispatchResult` from a hand-off carries `execution_id: <run id>`. In inline job-testing mode, a
|
|
25
|
+
hand-off that already finished returns the reloaded result, as `resume: :background` does today
|
|
26
|
+
(`check_for_inline_completion`).
|
|
27
|
+
|
|
28
|
+
**Migration** (CHANGELOG): a `rescue Lock::AcquisitionError` / `Semaphore::AcquisitionError` around
|
|
29
|
+
`continue` no longer fires; the resume was accepted. Callers that treat any non-`Failure` result as
|
|
30
|
+
"resumed" need no change.
|
|
31
|
+
|
|
32
|
+
### Web dashboard API
|
|
33
|
+
|
|
34
|
+
`POST /api/reactors/:id/continue`:
|
|
35
|
+
|
|
36
|
+
- a `DispatchResult` answers `{ success: true, message: "Resume accepted" }`;
|
|
37
|
+
- an "already resumed" rejection answers 422 with the error.
|
|
38
|
+
|
|
39
|
+
## 2. `Reactor.undo(id)` on an `aborted` run
|
|
40
|
+
|
|
41
|
+
The signature is unchanged.
|
|
42
|
+
|
|
43
|
+
- If the run was interrupted during its failing step's `compensate`, that `compensate` runs again
|
|
44
|
+
first, with the recorded arguments and failure, before the completed steps are undone (FR-022 to
|
|
45
|
+
FR-025).
|
|
46
|
+
- The `reason` passed to the re-run `compensate`:
|
|
47
|
+
- the original `String`, if the failure was a string;
|
|
48
|
+
- an `RubyReactor::Error::RecordedFailure` (`StandardError`) otherwise, whose `message` is the
|
|
49
|
+
original message and whose `original_class` is the original class name.
|
|
50
|
+
- A failing re-run is reported as any compensation failure is: a `:compensate` trace entry, a
|
|
51
|
+
`:failed_compensation` middleware event, and a rollback failure. The undo stack is still replayed.
|
|
52
|
+
The run still ends `cancelled` ("Undo triggered"), as manual undo does today.
|
|
53
|
+
|
|
54
|
+
Dashboard API (`GET /api/reactors/:id`): an `aborted` run whose `compensate` did not finish adds
|
|
55
|
+
`"pending_compensation": { "step": "<name>" }`. The GUI shows it on the run's detail view.
|
|
56
|
+
|
|
57
|
+
## 3. `map ... undo_all`
|
|
58
|
+
|
|
59
|
+
```ruby
|
|
60
|
+
map :refunds do
|
|
61
|
+
source input(:payments)
|
|
62
|
+
argument :payment, element(:refunds)
|
|
63
|
+
step :charge, ChargeStep
|
|
64
|
+
returns :charge
|
|
65
|
+
fan_out batch_size: 100
|
|
66
|
+
|
|
67
|
+
# Called once per rollback of this map, with the completed elements' results
|
|
68
|
+
# (a lazy Enumerable, in index order), INSTEAD of each element's own undos.
|
|
69
|
+
undo_all do |completed_results|
|
|
70
|
+
Payments.bulk_refund(completed_results.map { |charge| charge[:id] })
|
|
71
|
+
end
|
|
72
|
+
end
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
| Rule | Behavior |
|
|
76
|
+
| --- | --- |
|
|
77
|
+
| Declaration | optional; at most once per map; needs a block. Otherwise `Error::ValidationError` at class definition |
|
|
78
|
+
| When it is called | every rollback of the map: an atomic map's failure, a later step's failure, a manual undo |
|
|
79
|
+
| Argument | a lazy `Enumerable` of the results of the elements that completed, in index order; never every element's state at once |
|
|
80
|
+
| Not included | failed elements (they roll themselves back), halted, skipped |
|
|
81
|
+
| Not called | no element completed |
|
|
82
|
+
| Inline map with an `aborted` element (an interrupted run) | that element replays its own undos first; it is not passed |
|
|
83
|
+
| Return | `RubyReactor::Failure` or a raise is a failure; anything else is success |
|
|
84
|
+
| Failure | one rollback failure `{ step: <map>, kind: :undo_all, reason:, message: }`; the steps before the map are still undone |
|
|
85
|
+
| Fan-out | no element rollback jobs, no rollback records |
|
|
86
|
+
| Repetition | once per map rollback; at least once if the executing process dies mid-call |
|
|
87
|
+
|
|
88
|
+
## 4. Synchronous `Reactor.run`
|
|
89
|
+
|
|
90
|
+
The signature and results are unchanged. The run holds its liveness lock (`async:<id>`) while it
|
|
91
|
+
executes in the caller's process (R-01). Observable effects:
|
|
92
|
+
|
|
93
|
+
- `RubyReactor::Sweeper` / `RubyReactor.sweep_once` no longer re-enqueue it while it executes.
|
|
94
|
+
- `Reactor.undo(id)` of that run waits up to 5s, then raises `Lock::AcquisitionError`, as for a run
|
|
95
|
+
executing in a worker today.
|
|
96
|
+
|
|
97
|
+
## 5. Worker behavior (background jobs)
|
|
98
|
+
|
|
99
|
+
| Change | Observable effect |
|
|
100
|
+
| --- | --- |
|
|
101
|
+
| It takes the run's liveness lock before it reads the run (waits up to 2s) | a Worker enqueued by a caller still finishing its hand-off starts within about 2s instead of racing it |
|
|
102
|
+
| It returns on terminal statuses, and on a `paused` run without claimed payloads | stray or duplicate Workers do nothing |
|
|
103
|
+
| Admitted runs are not limited by `lock_snooze_max_attempts` | a resumed run waiting on a reactor lock or semaphore is never marked `failed` without rollback; it logs `ruby_reactor.resume.waiting` once at the limit |
|
|
104
|
+
|
|
105
|
+
## 6. Structured logs (key=value)
|
|
106
|
+
|
|
107
|
+
| Event | Level | Fields |
|
|
108
|
+
| --- | --- | --- |
|
|
109
|
+
| `ruby_reactor.resume.deferred` | info | `reactor`, `context_id`, `step`, `reason` (`lock`, `semaphore`, `run_busy`, `background`), `key` |
|
|
110
|
+
| `ruby_reactor.resume.waiting` | warn | `reactor`, `context_id`, `error` (names the contended key), `snooze_count` |
|
|
111
|
+
| `ruby_reactor.map.rollback.undo_all.started` | info | `reactor`, `context_id`, `map_step`, `count` |
|
|
112
|
+
| `ruby_reactor.map.rollback.undo_all.completed` | info | `reactor`, `context_id`, `map_step`, `count`, `failed` |
|
|
113
|
+
|
|
114
|
+
## 7. RSpec surface (`lib/ruby_reactor/rspec/`)
|
|
115
|
+
|
|
116
|
+
| Addition | Contract |
|
|
117
|
+
| --- | --- |
|
|
118
|
+
| `TestSubject#resume(payload:, step:, process_jobs: nil)` | Accepts a `paused` **or `running`** run with a ready interrupt, mirroring `continue`. `process_jobs: false` leaves a hand-off pending; `nil` keeps the subject's setting. |
|
|
119
|
+
| `be_resume_deferred` | Passes when the subject's last `resume` returned a `DispatchResult` and the run is `running`. |
|
|
120
|
+
| `have_run_undo_all(:map_step)` / `.with_elements(n)` | Passes when the run's execution trace has an `:undo_all` entry for that map, with a count of `n`. |
|
|
121
|
+
|
|
122
|
+
The existing `hold_lock`, `be_paused_at`, `have_ready_interrupts`, `have_rollback_failure` and
|
|
123
|
+
`be_success` / `be_failure` are unchanged.
|
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
# Contract: Run Ownership and Resume Protocol
|
|
2
|
+
|
|
3
|
+
The sequences the implementation must follow and the invariants the specs pin. The records are
|
|
4
|
+
defined in [data-model.md](../data-model.md). This extends the invariants I-1 to I-8 of
|
|
5
|
+
`specs/009-distributed-map-undo/contracts/rollback-protocol.md`.
|
|
6
|
+
|
|
7
|
+
## Invariants
|
|
8
|
+
|
|
9
|
+
- **J-1: a live run holds its lock.** While a run executes anywhere (a worker, the caller's process
|
|
10
|
+
through `execute`, `continue`, or `undo`), its `async:<id>` lock is held and auto-extended.
|
|
11
|
+
Exception: inline job-testing mode, which takes no context lock (R-14). The sweeper re-enqueues
|
|
12
|
+
only `running` and `rolling_back` runs with no lock holder.
|
|
13
|
+
- **J-2: lock, then load.** A Worker, `continue` and `Reactor#undo` read the run only after taking
|
|
14
|
+
its lock. A synchronous `execute` creates the run, so it has nothing older to read.
|
|
15
|
+
- **J-3: save before release** (009 I-6, extended to `execute`). Every holder persists its final
|
|
16
|
+
state before its lock count reaches zero.
|
|
17
|
+
- **J-4: one claim per interrupt.** At most one payload is accepted per interrupt per run (the
|
|
18
|
+
`SET NX` claim). Rejected resumes write nothing.
|
|
19
|
+
- **J-5: claims are applied only by the lock owner.** A claimed payload enters the run's context
|
|
20
|
+
only in the execution holding `async:<id>`, at the start of `resume_execution`. `continue` never
|
|
21
|
+
writes the context without the lock.
|
|
22
|
+
- **J-6: an accepted resume is not stranded.** After a successful claim, either the claiming
|
|
23
|
+
`continue` owns the run and resumes it, or a Worker for the run is enqueued. A Worker that finds
|
|
24
|
+
the run `paused` with an unapplied claim resumes it.
|
|
25
|
+
- **J-7: validated once.** A claimed payload is never validated again (FR-011).
|
|
26
|
+
- **J-8: no escalation after admission.** An admitted run is never marked `failed` by snooze
|
|
27
|
+
escalation (R-07).
|
|
28
|
+
- **J-9: no `compensate` is lost.** An `aborted` run whose failing step's `compensate` had not
|
|
29
|
+
returned keeps a record of it. Manual undo runs it before the undo stack, at each level where it
|
|
30
|
+
was recorded.
|
|
31
|
+
|
|
32
|
+
## P-1: Synchronous run with a fan-out hand-off (US1, US2)
|
|
33
|
+
|
|
34
|
+
```text
|
|
35
|
+
caller process element jobs / collector Worker(run)
|
|
36
|
+
────────────── ──────────────────────── ───────────
|
|
37
|
+
Reactor.run → Executor#execute
|
|
38
|
+
take async:<id> (wait 0, auto-extend) ◀── sweeper sees it held: skip
|
|
39
|
+
steps…; MapStep#run dispatches, returns DispatchResult
|
|
40
|
+
elements run; all settled →
|
|
41
|
+
enqueue Worker(run) ─────────▶ take async:<id> (wait ≤ 2s)
|
|
42
|
+
ensure: save final state (J-3) …blocked…
|
|
43
|
+
release async:<id> ──────────────────────────────────────────────────▶ acquired
|
|
44
|
+
return DispatchResult load (sees caller's final save, J-2)
|
|
45
|
+
resume_execution (re-entrant) → adopt map → …
|
|
46
|
+
save; release
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
If the caller takes more than 2s to release, the Worker snoozes as a `ContextLockContention`
|
|
50
|
+
(uncapped) and retries. It has loaded and written nothing.
|
|
51
|
+
|
|
52
|
+
## P-2: Resume while the reactor's lock is held (US3)
|
|
53
|
+
|
|
54
|
+
```text
|
|
55
|
+
webhook → Reactor.continue(id, payload, :approve)
|
|
56
|
+
find (snapshot) → status paused ✓, :approve ready ✓
|
|
57
|
+
validate payload ✓ (in this process)
|
|
58
|
+
SET NX resume:approve ✓
|
|
59
|
+
take async:<id> ✓ → reload → Executor(owner).resume_execution
|
|
60
|
+
apply claims → status running → with_lock contended → ensure: save → raise Lock::AcquisitionError
|
|
61
|
+
rescue: release async:<id> → enqueue Worker(run) → log resume.deferred(reason=lock)
|
|
62
|
+
return DispatchResult
|
|
63
|
+
Worker(run): lock, then load → running → resume_execution → apply claims (none left) → with_lock
|
|
64
|
+
contended → snooze (uncapped: admitted) … lock freed → acquire → steps run → final save
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## P-3: Two resumes of the same interrupt (US4)
|
|
68
|
+
|
|
69
|
+
```text
|
|
70
|
+
caller A caller B
|
|
71
|
+
find: paused find: paused
|
|
72
|
+
validate ✓ validate ✓
|
|
73
|
+
SET NX resume:approve ✓ SET NX resume:approve ✗ → raise "already resumed" (nothing written)
|
|
74
|
+
take lock, reload, resume …
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
The same holds in inline job-testing mode: the claim does not depend on the context lock.
|
|
78
|
+
|
|
79
|
+
## P-4: Resuming a second interrupt while the first executes (US5)
|
|
80
|
+
|
|
81
|
+
```text
|
|
82
|
+
caller A (approve_a) caller B (approve_b) Worker(run)
|
|
83
|
+
SET NX resume:approve_a ✓; take async:<id> ✓
|
|
84
|
+
reload; resume_execution: apply claim a
|
|
85
|
+
running; steps after a… find: running; approve_b ready ✓
|
|
86
|
+
validate ✓; SET NX resume:approve_b ✓
|
|
87
|
+
take async:<id> ✗ (held by A)
|
|
88
|
+
enqueue Worker(run); log deferred(run_busy)
|
|
89
|
+
return DispatchResult
|
|
90
|
+
reaches approve_b (no result) → pauses
|
|
91
|
+
save paused; release ───────────────────────────────────────────────────────────▶ take lock; load: paused
|
|
92
|
+
claim b unapplied → resume_execution
|
|
93
|
+
apply claim b → steps → completed
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
- If A's run reaches its end before B's claim, it pauses at `approve_b`, and the Worker resumes it.
|
|
97
|
+
- If A's run fails and rolls back first, the Worker loads a `failed` run and returns. Claim b is
|
|
98
|
+
never applied (FR-021).
|
|
99
|
+
|
|
100
|
+
## P-5: Manual undo of an aborted run (US6)
|
|
101
|
+
|
|
102
|
+
```text
|
|
103
|
+
caller process: Step X fails → handle_step_failure records @pending{step X, args, error}
|
|
104
|
+
compensate(X) … SIGTERM → rescue Exception → mark_aborted:
|
|
105
|
+
status aborted; rollback = {trigger: failure, step: X, compensated: false, arguments, error}
|
|
106
|
+
ensure: save; release async:<id>
|
|
107
|
+
operator: Reactor.undo(id)
|
|
108
|
+
take async:<id> (wait 5s) → reload (J-2)
|
|
109
|
+
rollback merged {trigger: undo, step: X, compensated: false, arguments, error}
|
|
110
|
+
Executor#undo_all → compensate_pending!: compensate(X, recorded error, recorded args)
|
|
111
|
+
→ compensated: true
|
|
112
|
+
→ rollback_completed_steps (newest first)
|
|
113
|
+
status cancelled ("Undo triggered"); save; release
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
A composed child or inline map element that was cut off records the same, on its own context. Its
|
|
117
|
+
level's `undo_all` (called through `ComposeStep#undo` or `Map::ElementRollback`) runs it before
|
|
118
|
+
that level's undo stack.
|
|
119
|
+
|
|
120
|
+
## P-6: Map rollback through `undo_all` (US7)
|
|
121
|
+
|
|
122
|
+
```text
|
|
123
|
+
executor rollback reaches the map's undo entry (or compensates a failed atomic map)
|
|
124
|
+
MapStep#compensate → undo_all_block declared → bulk_rollback
|
|
125
|
+
fan-out: report indexes with no result slot; enum = result slots (Success values, index order)
|
|
126
|
+
inline: pass 1 → aborted elements → Map::ElementRollback (per-element replay)
|
|
127
|
+
enum = completed element contexts, one at a time, index order
|
|
128
|
+
count == 0 → skip the call
|
|
129
|
+
log undo_all.started → block.call(enum) → trace :undo_all → log undo_all.completed
|
|
130
|
+
failure → rollback failure {step: map, kind: :undo_all}
|
|
131
|
+
return Success / Failure(rollback_failures) → the executor pops the entry → steps before the map
|
|
132
|
+
```
|