ruby_reactor 0.7.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (91) hide show
  1. checksums.yaml +4 -4
  2. data/.claude/skills/demo-app-e2e-verify/SKILL.md +226 -0
  3. data/.claude/skills/speckit-demo-tests/SKILL.md +144 -0
  4. data/.release-please-manifest.json +1 -1
  5. data/.specify/feature.json +1 -1
  6. data/.specify/memory/constitution.md +92 -15
  7. data/.specify/templates/plan-template.md +4 -0
  8. data/.specify/templates/tasks-template.md +8 -1
  9. data/CHANGELOG.md +151 -0
  10. data/CLAUDE.md +2 -2
  11. data/README.md +149 -40
  12. data/lib/ruby_reactor/context.rb +9 -2
  13. data/lib/ruby_reactor/context_serializer.rb +13 -0
  14. data/lib/ruby_reactor/dsl/interrupt_builder.rb +6 -0
  15. data/lib/ruby_reactor/dsl/lockable.rb +2 -2
  16. data/lib/ruby_reactor/dsl/reactor.rb +36 -18
  17. data/lib/ruby_reactor/dsl/step_builder.rb +95 -2
  18. data/lib/ruby_reactor/dsl/template_helpers.rb +13 -5
  19. data/lib/ruby_reactor/dsl/validation_helpers.rb +17 -0
  20. data/lib/ruby_reactor/error/input_validation_error.rb +4 -0
  21. data/lib/ruby_reactor/error/step_failure_error.rb +10 -3
  22. data/lib/ruby_reactor/executor/compensation_manager.rb +58 -45
  23. data/lib/ruby_reactor/executor/ordered_lock_support.rb +9 -9
  24. data/lib/ruby_reactor/executor/result_handler.rb +44 -13
  25. data/lib/ruby_reactor/executor/retry_manager.rb +6 -2
  26. data/lib/ruby_reactor/executor/step_executor.rb +27 -14
  27. data/lib/ruby_reactor/executor.rb +20 -15
  28. data/lib/ruby_reactor/map/element_executor.rb +7 -1
  29. data/lib/ruby_reactor/map/helpers.rb +9 -7
  30. data/lib/ruby_reactor/map/result_enumerator.rb +2 -0
  31. data/lib/ruby_reactor/map/sweeper.rb +1 -1
  32. data/lib/ruby_reactor/max_retries_exhausted_failure.rb +3 -2
  33. data/lib/ruby_reactor/open_telemetry.rb +7 -4
  34. data/lib/ruby_reactor/ordered_lock.rb +3 -3
  35. data/lib/ruby_reactor/reactor.rb +9 -12
  36. data/lib/ruby_reactor/rspec/matchers.rb +64 -17
  37. data/lib/ruby_reactor/rspec/test_subject.rb +8 -8
  38. data/lib/ruby_reactor/step/async_reactor_step.rb +159 -162
  39. data/lib/ruby_reactor/step/compose_step.rb +56 -75
  40. data/lib/ruby_reactor/step/input_contract.rb +128 -0
  41. data/lib/ruby_reactor/step/map_step.rb +178 -215
  42. data/lib/ruby_reactor/step.rb +119 -18
  43. data/lib/ruby_reactor/step_signals.rb +37 -0
  44. data/lib/ruby_reactor/step_worker.rb +25 -10
  45. data/lib/ruby_reactor/storage/adapter.rb +4 -0
  46. data/lib/ruby_reactor/storage/redis_adapter.rb +1 -72
  47. data/lib/ruby_reactor/storage/redis_reactor_scan.rb +116 -0
  48. data/lib/ruby_reactor/template/result.rb +9 -2
  49. data/lib/ruby_reactor/utils/fetch_indifferent.rb +13 -0
  50. data/lib/ruby_reactor/version.rb +1 -1
  51. data/lib/ruby_reactor/web/api.rb +23 -6
  52. data/lib/ruby_reactor/web/public/assets/index-BQvIWPdx.css +1 -0
  53. data/lib/ruby_reactor/web/public/assets/index-Dw4KV4QY.js +22 -0
  54. data/lib/ruby_reactor/web/public/index.html +2 -2
  55. data/lib/ruby_reactor.rb +61 -9
  56. data/specs/002-step-input-contracts/checklists/requirements.md +49 -0
  57. data/specs/002-step-input-contracts/contracts/dsl-surface.md +193 -0
  58. data/specs/002-step-input-contracts/data-model.md +115 -0
  59. data/specs/002-step-input-contracts/plan.md +165 -0
  60. data/specs/002-step-input-contracts/quickstart.md +170 -0
  61. data/specs/002-step-input-contracts/research.md +233 -0
  62. data/specs/002-step-input-contracts/spec.md +359 -0
  63. data/specs/002-step-input-contracts/tasks.md +367 -0
  64. data/specs/004-inheritable-step-class/checklists/requirements.md +40 -0
  65. data/specs/004-inheritable-step-class/contracts/step-lifecycle.md +85 -0
  66. data/specs/004-inheritable-step-class/data-model.md +116 -0
  67. data/specs/004-inheritable-step-class/plan.md +174 -0
  68. data/specs/004-inheritable-step-class/quickstart.md +112 -0
  69. data/specs/004-inheritable-step-class/research.md +308 -0
  70. data/specs/004-inheritable-step-class/spec.md +316 -0
  71. data/specs/004-inheritable-step-class/tasks.md +258 -0
  72. data/specs/deferred-003-step-lock-declarations/checklists/requirements.md +51 -0
  73. data/specs/deferred-003-step-lock-declarations/contracts/dsl-surface.md +154 -0
  74. data/specs/deferred-003-step-lock-declarations/data-model.md +131 -0
  75. data/specs/deferred-003-step-lock-declarations/plan.md +166 -0
  76. data/specs/deferred-003-step-lock-declarations/quickstart.md +169 -0
  77. data/specs/deferred-003-step-lock-declarations/research.md +196 -0
  78. data/specs/deferred-003-step-lock-declarations/spec.md +447 -0
  79. data/specs/deferred-003-step-lock-declarations/tasks.md +572 -0
  80. data/specs/possible_feature.md +22 -0
  81. metadata +34 -11
  82. data/lib/ruby_reactor/web/public/assets/index-B46p-M6K.css +0 -1
  83. data/lib/ruby_reactor/web/public/assets/index-DPmP4yXT.js +0 -22
  84. data/specs/001-background-async-steps/checklists/requirements.md +0 -39
  85. data/specs/001-background-async-steps/contracts/public-dsl.md +0 -154
  86. data/specs/001-background-async-steps/data-model.md +0 -117
  87. data/specs/001-background-async-steps/plan.md +0 -168
  88. data/specs/001-background-async-steps/quickstart.md +0 -102
  89. data/specs/001-background-async-steps/research.md +0 -150
  90. data/specs/001-background-async-steps/spec.md +0 -146
  91. data/specs/001-background-async-steps/tasks.md +0 -271
@@ -0,0 +1,572 @@
1
+ ---
2
+
3
+ description: "Task list for Step-Scoped Coordination"
4
+ ---
5
+
6
+ # Tasks: Step-Scoped Coordination
7
+
8
+ **Input**: Design documents from `specs/003-step-lock-declarations/`
9
+
10
+ **Prerequisites**: plan.md, spec.md, research.md, data-model.md, contracts/dsl-surface.md, quickstart.md
11
+
12
+ **Tests**: REQUIRED. Constitution III makes test-first mandatory: write each story's specs first,
13
+ watch them fail, then implement. Real Redis always (`docker compose up -d redis-test`). Any
14
+ worker/park claim uses the live-Sidekiq lane (`spec/support/real_async_backend.rb`,
15
+ `RealAsyncBackend.start_sidekiq!`). `Sidekiq::Testing.inline!` is never allowed in these specs:
16
+ it re-enters the worker inside the frame that holds the lock.
17
+
18
+ **Organization**: Grouped by user story. The plan's Phase 2 outline maps as: plan 1 → Phase 2,
19
+ plan 2 → US1/US2, plan 3 → US4, plan 4 → US3, plan 6 → US5, plan 5 → US6, plan 7 → US7/US8,
20
+ plan 8 → US5 (cont.), plan 9 → Polish.
21
+
22
+ ## Format: `[ID] [P?] [Story] Description`
23
+
24
+ - **[P]**: Can run in parallel (different files, no dependency on an incomplete task)
25
+ - **[Story]**: User story the task belongs to (US1–US8)
26
+
27
+ ## Findings from the code that the tasks below encode
28
+
29
+ Read these before starting. Each was found in the current code and is not in plan.md.
30
+
31
+ 1. **Registry pop is wrong for nesting.** `Executor#release_locks` (`lib/ruby_reactor/executor.rb:612,621`)
32
+ calls `held_lock_keys.delete(key)`, which removes *every* occurrence. With a reactor and a
33
+ step both holding K, the step's release would erase the reactor's entry, and the async
34
+ deadlock guard would stop seeing K. Pop one occurrence instead (T031).
35
+ 2. **Retry attempts would be charged for contention.** `RetryManager#prepare_retry_attempt`
36
+ increments `retry_context.step_attempts` before every attempt. A contention park must give
37
+ that increment back, or a busy key eats the retry budget meant for real failures (T024).
38
+ 3. **`InterruptBuilder < StepBuilder`.** Once `StepBuilder` gets the five macros, interrupt
39
+ steps inherit them. They have to be overridden to raise (D6) (T008).
40
+ 4. **`Context#with_step` restores `current_step` in `ensure`.** When a contention error unwinds
41
+ out of the step, `current_step` goes back to its old value, which is `nil` for a first step.
42
+ The park path must set `@context.current_step` *after* unwinding (the rescue in
43
+ `safe_execute_step_sync`). Otherwise the redelivery looks like a first run
44
+ (`Executor#first_execution?`) and consumes the reactor-level rate limit and period gate a
45
+ second time (T025).
46
+ 5. **`async_step` defers argument resolution** (`step_executor.rb:81`). The dispatcher never
47
+ resolves the step's args, so it cannot compute the step's key without extra work. The
48
+ dispatch-time guard resolves args only when that cannot block (T034).
49
+ 6. **The async_step worker has no delayed re-enqueue.** Routers expose `perform_step_async`
50
+ only. Parking an `async_step` body needs `perform_step_in` on both routers (T036).
51
+ 7. **Contention ceiling and delay**: reuse the existing `lock_snooze_max_attempts`,
52
+ `lock_snooze_base_delay`, `lock_snooze_jitter` config, and `Worker#compute_snooze_delay`'s
53
+ hint logic. They mean the same thing and satisfy FR-017 with no new config. The contention
54
+ counter lives on `RetryContext` (plan/research D4), not in `private_data`. data-model.md's
55
+ ContentionState table is superseded on that one point.
56
+
57
+ ## Shared names used across tasks
58
+
59
+ - `RubyReactor::Executor::StepCoordination`, in `lib/ruby_reactor/executor/step_coordination.rb`
60
+ (Zeitwerk autoloads it).
61
+ - `StepCoordination::KeyError < RubyReactor::Error::Base`: key proc raised, or returned nil/empty.
62
+ - `StepCoordination::Contended < StandardError`: raised on contention. Carries `primitive`,
63
+ `key`, `step_name`, `reactor_name`, `retry_after_seconds`, and `original` (the underlying
64
+ `Lock::AcquisitionError` / `Semaphore::AcquisitionError` / `RateLimit::ExceededError` /
65
+ `OrderedLock::WaitError`). Message: `"<Reactor> step :<step> could not acquire <primitive> '<key>': <original message>"`.
66
+ - Coordination source for a step: `step_config` for inline declarations, falling back to
67
+ `step_config.impl` for class declarations. The fallback lives in `StepConfig`'s readers (T007),
68
+ so callers only ever read `step_config.lock_config` and the other four.
69
+
70
+ ---
71
+
72
+ ## Phase 1: Setup (Shared Infrastructure)
73
+
74
+ **Purpose**: Test harness shared by every concurrency spec in this feature
75
+
76
+ - [ ] T001 Bring up `redis-test` and run `bundle exec rspec` and `bundle exec rubocop` on the untouched branch. Save the pass/fail counts to `specs/003-step-lock-declarations/baseline.txt`; SC-012 is checked against it at the end
77
+ - [ ] T002 [P] Create `spec/support/step_coordination_helpers.rb` with an `OverlapRecorder`. `enter(tag)` and `leave(tag)` RPUSH `"#{tag}:#{Process.pid}:#{Thread.current.object_id}:#{Process.clock_gettime(Process::CLOCK_REALTIME)}"` to the Redis list `step_coord:trace:<run_id>`, so the recorder works across threads and the live Sidekiq process. Add `max_concurrency(tag)` and `overlapped?(tag_a, tag_b)`, both computed from the list. Include the helper in specs tagged `:step_coordination`
78
+ - [ ] T003 [P] Create `spec/support/reactors/step_coordination_reactors.rb` with the fixture step classes and reactors shared by the specs and the live worker. Examples: `LockedChargeStep` (`with_lock { |a| "acct:#{a[:account_id]}" }`), `SleepyStep`, `RecordingStep`. Every body calls `OverlapRecorder`. `spec/support/sidekiq_boot.rb` already requires `reactors/*.rb`, so the live worker loads them. Confirm the recorder is reachable there: require `step_coordination_helpers.rb` from the fixture file, not from `spec_helper`
79
+ - [ ] T004 Create the `spec/ruby_reactor/step_coordination/` directory. In `spec/support/step_coordination_helpers.rb`, tag every example group in it `:step_coordination` via `RSpec.configure { |c| c.define_derived_metadata(file_path: %r{/step_coordination/}) { |m| m[:step_coordination] = true } }`, so T002's helper is included automatically
80
+
81
+ ---
82
+
83
+ ## Phase 2: Foundational (Declaration surface, no enforcement)
84
+
85
+ **Purpose**: Steps can declare the five primitives, and the declarations can be inspected. Nothing is enforced yet. Every story depends on this phase.
86
+
87
+ **⚠️ CRITICAL**: No user story work can begin until this phase is complete
88
+
89
+ - [ ] T005 [P] Write `spec/ruby_reactor/step_coordination/declaration_spec.rb` and confirm it fails. Cover:
90
+ - (a) A class including `RubyReactor::Step` can call all five macros; the per-macro readers (`lock_config`, etc.) return the same hashes the reactor form builds.
91
+ - (b) `with_rate_limit(:name, limit: 1)` still raises `ArgumentError`, and `with_period(every: :bogus)` still raises at class load.
92
+ - (c) A subclass inherits the parent's configs, and redeclaring a primitive replaces the parent's.
93
+ - (d) `declares_coordination?` is false for a bare step and true once any macro is used.
94
+ - (e) `coordination_declarations` returns `{ lock: {...}, semaphore: {...} }` with only the declared keys.
95
+ - (f) An inline `step :x do with_lock { ... } end` exposes `lock_config` on its `StepConfig`.
96
+ - (g) `StepConfig#lock_config` falls back to `impl.lock_config` when the block declares nothing, and the block wins when both declare.
97
+ - (h) `with_lock` inside an `interrupt` block raises at class definition, and the message names reactor-level coordination as the alternative.
98
+ - (i) Reactor-level `lock_config` is unchanged for an existing reactor.
99
+ - [ ] T006 Make steps host the macros in `lib/ruby_reactor/step.rb`: in `Step.included`, also `base.extend(RubyReactor::Dsl::Lockable::ClassMethods)`. `Lockable::ClassMethods#inherited` already propagates configs to subclasses. Leave the macro definitions untouched
100
+ - [ ] T007 Add introspection to `lib/ruby_reactor/dsl/lockable.rb`: `coordination_declarations` (hash of the five configs, nils compacted, keys `:lock, :semaphore, :rate_limit, :period, :ordered_lock`) and `declares_coordination?` (`!coordination_declarations.empty?`). Reactors get these too, which is additive. In `lib/ruby_reactor/dsl/step_builder.rb`:
101
+ - `StepBuilder` does `include RubyReactor::Dsl::Lockable::ClassMethods`, so the macros become builder instance methods.
102
+ - `build` passes `lock_config:`, `semaphore_config:`, `rate_limit_config:`, `period_config:`, `ordered_lock_config:` into `StepConfig`.
103
+ - `StepConfig` stores them and defines each reader as `@lock_config || (impl.lock_config if impl.respond_to?(:lock_config))`, and likewise for the other four.
104
+ - `StepConfig` also defines `coordination_declarations` and `declares_coordination?` over those readers.
105
+ - [ ] T008 In `lib/ruby_reactor/dsl/interrupt_builder.rb`, override `with_lock`, `with_semaphore`, `with_rate_limit`, `with_period`, `with_ordered_lock` so each raises `RubyReactor::Error::ValidationError` (research D6). Message: `"interrupt :#{@name} cannot declare step-level coordination: its body is split across a pause, so a hold would span the gap. Declare it on the reactor (with_lock etc.) instead."`
106
+ - [ ] T009 Create the `lib/ruby_reactor/executor/step_coordination.rb` skeleton with no enforcement yet.
107
+ - Define `KeyError` and `Contended` per "Shared names".
108
+ - `initialize(step_config:, arguments:, context:, reactor_class:, middlewares:, owner: nil, park: true)`.
109
+ - `#key_for(config)`: call `config[:key_proc].call(arguments)`. Raise `KeyError` naming the step and the cause if the proc raises, or if the result is nil or `to_s.empty?`.
110
+ - `#owner`: the explicit `owner:`, else `(context.root_context || context).context_id`, else `SecureRandom.uuid` when the context is nil.
111
+ - `#wait_for(configured)`: return `0` when `park && context&.inline_async_execution`, else `configured`. This mirrors `Executor#contention_wait`.
112
+ - `#push_key(key)` / `#pop_key(key)` on `root.private_data[:held_lock_keys] ||= []`. Pop removes ONE occurrence via `index` + `delete_at`.
113
+ - `#around_run { }` and `#around_rollback { }` just `yield` for now.
114
+ - Add `self.none?(step_config)`, returning `!step_config.respond_to?(:declares_coordination?) || !step_config.declares_coordination?`, so the executor can skip construction entirely (plan: one nil check per step)
115
+ - [ ] T010 Run `declaration_spec.rb` and get it green. Run the full `bundle exec rspec`: no existing spec may change result (FR-027)
116
+
117
+ **Checkpoint**: Declarations exist and are inspectable; runtime behavior is unchanged
118
+
119
+ ---
120
+
121
+ ## Phase 3: User Story 1 - A step class declares the lock it needs (Priority: P1) 🎯 MVP
122
+
123
+ **Goal**: `with_lock` on a step class is taken right before the step body and released right after, keyed on the step's resolved arguments.
124
+
125
+ **Independent Test**: Two concurrent executions with the same key never overlap inside the step body. With different keys, they do overlap.
126
+
127
+ ### Tests for User Story 1 ⚠️ (write first, confirm failing)
128
+
129
+ - [ ] T011 [P] [US1] Write `spec/ruby_reactor/step_coordination/lock_spec.rb` against real Redis, using two `Thread`s running sync `Reactor.run`, each with `with_lock(wait: 5)` so the loser blocks instead of failing. Cover:
130
+ - (1) Same key: `OverlapRecorder.max_concurrency(:charge) == 1` across 20 iterations (SC-001).
131
+ - (2) Different keys: `overlapped?` is true.
132
+ - (3) After a successful locked step, the next step observes `expect("acct:1").not_to be_locked` (SC-003).
133
+ - (4) The step body returns `Failure` and the key is free afterwards.
134
+ - (5) The step body raises and the key is free afterwards.
135
+ - (6) Reactor input `account_id: 1` plus a step `argument :account_id, input(:account_id), transform: ->(v) { v + 100 }` locks `"acct:101"`, not `"acct:1"`.
136
+ - (7) The key proc raises, or returns nil, or returns `""`: the reactor result is a Failure naming the step and the cause, and the body's recorder tag never appears (SC-011, FR-007).
137
+ - (8) A step suppressed by `where { false }` leaves no lock and no `:lock_acquired` event (FR-012).
138
+ - (9) `with_lock(ttl: 1, auto_extend: true)` with a 2.5s body: a second thread with `wait: 0` gets contention throughout (FR-013).
139
+ - (10) `fork` a child that takes the step lock and sleeps, `Process.kill("KILL", pid)`, then after `ttl` seconds a new run succeeds (SC-008).
140
+ - (11) With the coordination Redis unreachable (point the adapter at `redis://127.0.0.1:1` for this example only, and restore it in `after`), the step fails with the connection error as cause and the body never runs.
141
+
142
+ ### Implementation for User Story 1
143
+
144
+ - [ ] T012 [US1] Implement exclusive-lock acquisition in `StepCoordination#around_run`. When `step_config.lock_config` is present:
145
+ - Build `RubyReactor::Lock.new(key, owner:, ttl: config[:ttl], wait: wait_for(config[:wait]), auto_extend: config.fetch(:auto_extend, true))`.
146
+ - `acquire`, then `push_key(key)`, then `middlewares.on(:lock_acquired, key, context)`.
147
+ - On `Lock::AcquisitionError`: `middlewares.on(:lock_failed, key, e, context)`, then raise `Contended` with `primitive: :lock`.
148
+ - Yield. In `ensure`: release, logging and never raising (copy `Executor#release_one`), then `pop_key(key)` and `middlewares.on(:lock_released, key, context)`.
149
+ - [ ] T013 [US1] Wire enforcement into `lib/ruby_reactor/executor/step_executor.rb`. In both `execute_step_sync` and `execute_step_sync_without_result_handling`, replace the bare `run_step_implementation(step_config, resolved_arguments)` call AFTER `validate_step_arguments` with a private `run_coordinated(step_config, resolved_arguments)`:
150
+ - Call `run_step_implementation` directly when `StepCoordination.none?(step_config)`.
151
+ - Otherwise build `StepCoordination.new(step_config:, arguments: resolved_arguments, context: @context, reactor_class: @reactor_class, middlewares: @middlewares)` and call `.around_run { run_step_implementation(...) }`.
152
+ - The guard check and validation stay before this point, so nothing is taken for a step that will not run (FR-012, D3).
153
+ - [ ] T014 [US1] In `StepExecutor#safe_execute_step_sync`, add `rescue StepCoordination::KeyError => e` BEFORE the generic `StandardError` rescue. Return `RubyReactor::Failure(e, retryable: false, step_name: step_config.name, reactor_name: @reactor_class.name, step_arguments: resolved_arguments, inputs: @context.inputs)`. Also add a temporary `rescue StepCoordination::Contended => e` returning the same shape with `exception_class: e.original.class.name`. US3 replaces this with the park/fail split
154
+ - [ ] T015 [US1] Run `lock_spec.rb` until green. Fix `StepCoordination` or the wiring, not the spec
155
+
156
+ **Checkpoint**: MVP. A class step's `with_lock` serializes that step in-process
157
+
158
+ ---
159
+
160
+ ## Phase 4: User Story 2 - Only the step is locked, not the whole workflow (Priority: P1)
161
+
162
+ **Goal**: Steps around a locked step keep overlapping across concurrent executions.
163
+
164
+ **Independent Test**: In an eight-step reactor whose third step locks a shared key, steps 1–2 and 4–8 overlap across two runs, and only step 3 serializes.
165
+
166
+ ### Tests for User Story 2 ⚠️
167
+
168
+ - [ ] T016 [P] [US2] Write `spec/ruby_reactor/step_coordination/scope_spec.rb`. Build an eight-step fixture reactor in `spec/support/reactors/step_coordination_reactors.rb`: each step is a `RecordingStep` subclass that sleeps 0.2s, and step 3 declares `with_lock(wait: 10) { "shared" }`. Run two sync executions on threads and assert:
169
+ - `OverlapRecorder.overlapped?` is true for steps 1, 2, 4..8 across the runs.
170
+ - `max_concurrency(:step3) == 1`.
171
+ - Total wall time is under `8 * 0.2 * 2` (SC-002).
172
+ - Second example: while run B waits on step 3, run A's step 4 starts before B's step 3 starts (US2-2).
173
+
174
+ ### Implementation for User Story 2
175
+
176
+ - [ ] T017 [US2] Run `scope_spec.rb`. If it fails, the cause is a hold outliving its step. Check that `StepCoordination` is local to `run_coordinated` (no ivar on `StepExecutor`) and that no reactor-level lock is acquired for a reactor that declares none. Fix in `lib/ruby_reactor/executor/step_executor.rb` / `lib/ruby_reactor/executor/step_coordination.rb`
177
+
178
+ **Checkpoint**: US1 and US2 green. The feature delivers its core value synchronously
179
+
180
+ ---
181
+
182
+ ## Phase 5: User Story 3 - Contention parks the execution instead of failing it (Priority: P1)
183
+
184
+ **Goal**: In a worker, losing contention requeues the execution at that step (`RetryQueuedResult`), with no compensation and bounded attempts. Synchronously it waits, then fails, and rollback runs.
185
+
186
+ **Independent Test**: Two worker-backed executions on one key both complete. The same pair run synchronously gives the loser a contention Failure.
187
+
188
+ ### Tests for User Story 3 ⚠️
189
+
190
+ - [ ] T018 [P] [US3] Write `spec/ruby_reactor/step_coordination/contention_spec.rb`, live-Sidekiq lane (`RealAsyncBackend.start_sidekiq!`, skip with reason if it cannot boot). Cover:
191
+ - (1) Two `background`-dispatched runs of a reactor whose `:charge` step is locked on the same key, with a 1s body: both contexts reach `completed`, the second's `execution_trace` has a `type: :contention_park, step: :charge` entry, and neither trace has a `:compensate` or `:undo` entry (SC-004, US3-1, US3-2).
192
+ - (2) The `:charge` body recorder tag appears exactly once per execution.
193
+ - (3) `config.lock_snooze_max_attempts = 2`, with the key held externally for the whole test (take a `RubyReactor::Lock` with a foreign owner in the spec process, since this is a lib spec, not demo): the context ends `failed`, and the failure message contains `"contention"`, the key, and `2` (FR-017, US3-5).
194
+ - (4) A step with `retries max_attempts: 2` that is contended 3 times and then fails for real still gets 2 failure attempts: `retry_context.attempts_for_step(:charge) == 2` (Finding 2).
195
+ - (5) Sync path: the key is held externally and `with_lock(wait: 1)`. `Reactor.run` returns a Failure after about 1s. Its message names the reactor class, `:charge`, and the key, and `exception_class == "RubyReactor::Lock::AcquisitionError"`. A prior step with `compensate` has run its compensation (US3-4, FR-016).
196
+ - (6) A reactor with reactor-level `with_lock` on K1 and step `:charge` locked on K2, contended in a worker: across the park gap K1 stays held by the root context id (`expect("K1").to be_locked.by(ctx_id)`), and the `:lock_acquired` count for K1 is exactly 1, recorded through a fixture middleware that RPUSHes events to Redis (US3-3, FR-018).
197
+ - (7) The contended step is the reactor's FIRST step, and the reactor has `with_rate_limit(limit: 1, period: :minute)`: the parked-then-resumed run still succeeds, meaning the rate limit was not charged twice (Finding 4).
198
+ - (8) A `map` whose iteration step is locked on one key: every element completes (map elements park through `perform_map_element_in`).
199
+
200
+ ### Implementation for User Story 3
201
+
202
+ - [ ] T019 [P] [US3] Add a contention counter to `lib/ruby_reactor/retry_context.rb`:
203
+ - `attr_accessor :contention_attempts` (Hash, string step keys), initialized `{}`.
204
+ - Methods `increment_contention_for_step(name)`, `contention_attempts_for_step(name)`, `clear_contention_for_step(name)`.
205
+ - Include it in `reset`, `serialize_for_retry` (`contention_attempts:`), and `deserialize_from_retry` (`data["contention_attempts"] || {}`).
206
+ - [ ] T020 [P] [US3] Extract `Worker#compute_snooze_delay` and `#hinted_retry?` (`lib/ruby_reactor/worker.rb`) into public module functions `RubyReactor::Worker.snooze_delay(config, error)` / `.hinted_retry?(error)`. Keep the instance methods delegating so worker behavior is unchanged. `StepCoordination::Contended#retry_after_seconds` returns `original.retry_after_seconds` if it responds to it, so the hint logic works on the wrapper. Keep the `OrderedLock::WaitError` exclusion by checking `original`
207
+ - [ ] T021 [US3] In `lib/ruby_reactor/executor/retry_manager.rb`, split `requeue_job_for_step_retry` into:
208
+ - `requeue_job(step_config, delay)`: everything after the `delay =` line, taking `delay` as a parameter. This keeps the map-element branch.
209
+ - The existing method, which now computes the backoff and calls `requeue_job`.
210
+ - [ ] T022 [US3] Add public `RetryManager#park_for_contention(step_config, contended, reactor_class)`:
211
+ - Increment `retry_context.contention_attempts` for the step.
212
+ - If `config.lock_snooze_max_attempts != :infinity` and the count exceeds it, return `RubyReactor::Failure(<message "Step ':x' gave up on <primitive> '<key>' after N contention attempts">, retryable: false, step_name:, reactor_name:, exception_class: contended.original.class.name)`.
213
+ - Otherwise set `delay = RubyReactor::Worker.snooze_delay(config, contended)` and `retry_context.next_retry_at = Time.now + delay`, call `requeue_job(step_config, delay)`, and return `RetryQueuedResult.new(step_config.name, retry_context.attempts_for_step(step_config.name), retry_context.next_retry_at)`.
214
+ - Uncapped cases: `OrderedLock::WaitError` originals, the same exemption `Worker#handle_snooze` makes.
215
+ - [ ] T023 [US3] Make sure `RetryManager#handle_retry_result` passes a `RetryQueuedResult` returned from the block through unchanged. It already does; add a comment saying contention parks rely on this
216
+ - [ ] T024 [US3] Give back the failure-retry attempt on a contention park (Finding 2): add `RetryContext#decrement_attempt_for_step(name)` (floor 0) and call it at the start of `park_for_contention` before anything else, so `prepare_retry_attempt`'s increment for this round is undone
217
+ - [ ] T025 [US3] Replace T014's temporary `Contended` rescue in `StepExecutor#safe_execute_step_sync` with the park/fail split:
218
+ - (a) Set `@context.current_step = step_config.name`. `with_step`'s ensure has already restored the old value (Finding 4).
219
+ - (b) Append trace `{ type: :contention_park, step:, primitive: contended.primitive, key: contended.key, attempt: }`. Set `@context.private_data[:step_contention] = { step:, primitive:, key:, attempts:, next_attempt_at: }` for operators (US7).
220
+ - (c) When `@context.inline_async_execution` is set, return `@retry_manager.park_for_contention(step_config, e, @reactor_class)`.
221
+ - (d) Otherwise return the sync contention `Failure(e, retryable: false, exception_class: e.original.class.name, step_name:, reactor_name:, step_arguments:, inputs:)`.
222
+ - On successful acquisition (in `around_run`, after all primitives are taken), clear `private_data[:step_contention]` and call `retry_context.clear_contention_for_step`.
223
+ - [ ] T026 [US3] Keep reactor-level holds across a contention park (FR-018, US3-3). In `Executor#resume_execution` (`lib/ruby_reactor/executor.rb`), after `@result` is computed: if `@result.is_a?(RetryQueuedResult) && @context.private_data[:step_contention]`, call `park_held_primitives!`, which sets `@parked` so the ensure skips `release_locks`. Then confirm that `consume_parked_primitives!` / reattach runs on the redelivery path that `requeue_job` → `Worker#perform` → `resume_execution` takes. If reattach is only wired for the `AsyncResultPending` redelivery, generalize the check to any resume that finds `private_data[:parked_primitives]`
224
+ - [ ] T027 [US3] Verify the map path in `lib/ruby_reactor/map/element_executor.rb`: a `RetryQueuedResult` from a contention park must take the existing "async retry requeued this element" branch. Adjust the condition only if it keys on something other than the result class
225
+ - [ ] T028 [US3] Run `contention_spec.rb` until green, then re-run `lock_spec.rb` and `scope_spec.rb`
226
+
227
+ **Checkpoint**: Contention parks in workers and fails synchronously, bounded, with no spurious compensation
228
+
229
+ ---
230
+
231
+ ## Phase 6: User Story 4 - Re-entrancy behaves exactly as nested workflows already do (Priority: P1)
232
+
233
+ **Goal**: Step holds follow the nested-reactor rules. They are owned by the root context, nesting is counted, they share one registry, and a hand-off that would deadlock is refused before dispatch. They are also honored on the `async_step` worker and on direct `Step.run`.
234
+
235
+ **Independent Test**: Reactor lock K, then step lock K, then compose child lock K: this completes. Handing K-declaring work to another process is refused, naming K.
236
+
237
+ **Depends on**: US3's `RubyReactor::Worker.snooze_delay` (T020), for T036.
238
+
239
+ ### Tests for User Story 4 ⚠️
240
+
241
+ - [ ] T029 [P] [US4] Write `spec/ruby_reactor/step_coordination/reentrancy_spec.rb` and cover:
242
+ - (1) Reactor `with_lock { "k:#{i[:id]}" }` plus a step locking the same key completes with no wait (`wait: 0`) (US4-1).
243
+ - (2) A locked step whose body calls `ChildReactor.run` (sync compose-style) where the child has `with_lock` on K completes (US4-2, SC-006).
244
+ - (3) Nested holds: inside the innermost body record `lock_info`. After the step releases but while the reactor still holds K, `expect("k:1").to be_locked` and `root.private_data[:held_lock_keys].count("k:1") == 1` (US4-3, FR-020, Finding 1).
245
+ - (4) Two steps in one reactor locking the same key run in order and complete.
246
+ - (5) A locked step's body dispatches `async_reactor` whose child `with_lock`s K: the result is a Failure whose message names K and the dispatching reactor, and no job is enqueued (US4-4, SC-007).
247
+ - (6) The reactor holds K and a later `async_step` whose step class declares `with_lock` on K: refused at dispatch with the same message shape, and no StepWorker job (FR-022).
248
+ - (7) Live-Sidekiq: an `async_step` with `with_lock` on K. While its body runs in the worker, `expect("K").to be_locked` and the owner is NOT the root context id (ownership never crosses the hand-off, US4-5). Two such dispatches on the same K never overlap (SC-005).
249
+ - (8) `LockedChargeStep.run({ account_id: 1 }, nil)` called directly while K is held externally fails with the contention error after `wait`. When K is free it takes and releases K (FR-023).
250
+ - (9) A step whose executor already took K does not double-take through the direct-invocation wrapper: exactly one `:lock_acquired` event.
251
+
252
+ ### Implementation for User Story 4
253
+
254
+ - [ ] T030 [US4] Confirm the owner rule in `StepCoordination#owner` (T009): root context id for executor-driven runs, a per-call UUID when `context` is nil. No code change expected; add the `# re-entrancy: same owner as every reactor in this execution tree` comment
255
+ - [ ] T031 [US4] Fix `Executor#release_locks` in `lib/ruby_reactor/executor.rb` (Finding 1). Replace both `held_lock_keys.delete(key)` calls with a single-occurrence pop (`i = held_lock_keys.index(key); held_lock_keys.delete_at(i) if i`). Extract it as private `pop_held_lock_key(key)` so `StepCoordination#pop_key` has the same semantics
256
+ - [ ] T032 [US4] Also push the step's semaphore key to the registry when `limit == 1`, in `StepCoordination`, matching `Executor#acquire_semaphore`. The semaphore itself is implemented in US5 (T043); this task is the registry rule only, written so T043 calls it
257
+ - [ ] T033 [US4] Make the deadlock guard reusable from `lib/ruby_reactor/step/async_reactor_step.rb`:
258
+ - Turn `held_lock_keys(context)` and `deadlock_message(key, child_class, context)` into public class methods.
259
+ - Generalize the message's `"async_reactor dispatch of"` prefix to take a `kind:` (`"async_reactor"` / `"async_step"`) and name the dispatched step.
260
+ - Add remedy text for async_step: "run the step inline (drop async_step) if it belongs inside the critical section".
261
+ - [ ] T034 [US4] Add a dispatch-time guard in `lib/ruby_reactor/executor/async_step_dispatch.rb#dispatch_async_step`, before `record_async_step_dispatch`:
262
+ - Skip unless the root's held keys are non-empty and `step_config.lock_config` (or a `semaphore_config` with `limit == 1`) is declared.
263
+ - Resolve the step's arguments non-blocking (Finding 5): if any argument source is a `RubyReactor::Template::Result` whose target has an `:async_step_ref`/`:async_reactor_ref` entry in `@context.composed_contexts` and no recorded intermediate result, skip the check and log `event="ruby_reactor.step_coordination.guard_skipped"` with reactor/step/execution_id.
264
+ - Otherwise compute the keys (a `StepCoordination::KeyError` fails the dispatching step) and, on a collision, return `RubyReactor.Failure(Step::AsyncReactorStep.deadlock_message(...))` without enqueueing or writing the dispatch record.
265
+ - [ ] T035 [US4] Enforce coordination in the async_step worker, `lib/ruby_reactor/step_worker.rb`. In `run_step`, wrap each `execute_step_body` attempt in `StepCoordination.new(step_config:, arguments:, context:, reactor_class: context.reactor_class, middlewares: context.middlewares || RubyReactor::MiddlewareRunner.new([]), owner: @coordination_owner ||= SecureRandom.uuid, park: true).around_run { ... }`. The owner is per job and NEVER the root id (US4-5, D5 "ownership never crosses a hand-off"). The context is already marked `inline_async_execution`, so `wait_for` returns 0. Rescue `StepCoordination::KeyError` into a Failure record through `complete`
266
+ - [ ] T036 [US4] Park the async_step body on contention (Finding 6):
267
+ - Add `perform_step_in(delay, root_context_id:, reactor_class_name:, step_context_id:, step_name:, contention_attempts: 0)` to `lib/ruby_reactor/adapters/sidekiq/router.rb` (`StepWorker.perform_in`) and `lib/ruby_reactor/adapters/active_job/router.rb` (`set(wait: delay).perform_later`), passing `"contention_attempts"` in the payload.
268
+ - Make `StepWorker.slice_arguments` / `initialize` accept it.
269
+ - In `StepWorker#run_step`, rescue `StepCoordination::Contended`. Over `lock_snooze_max_attempts` (unless `OrderedLock::WaitError`), `complete` a Failure naming the step, key, and attempts. Otherwise call `perform_step_in(RubyReactor::Worker.snooze_delay(config, e), ..., contention_attempts: n + 1)`, `log(:info, "parked", key:, attempt:)`, and return WITHOUT calling `complete`, so the Step Result Record stays `dispatched` and readers keep waiting.
270
+ - [ ] T037 [US4] Honor declarations on direct invocation (FR-023) in `lib/ruby_reactor/step.rb`:
271
+ - Define `Step::DirectCoordination` with `def run(arguments, context = nil)`. It calls `super` when `!declares_coordination?` or when `Thread.current[:ruby_reactor_coordinated]&.include?(self)`.
272
+ - Otherwise it does `StepCoordination.new(step_config: self, arguments:, context: (context if context.is_a?(RubyReactor::Context)), reactor_class: nil, middlewares: RubyReactor::MiddlewareRunner.new([]), park: false).around_run { super }`.
273
+ - `Step.included` does `base.singleton_class.prepend(DirectCoordination)`. Because a subclass's own `def self.run` would sit ahead of the parent's prepend, also prepend in an `inherited` hook on `Step::ClassMethods` (call `super` first).
274
+ - In `StepCoordination#around_run`, push `step_config.impl` (when present) onto `Thread.current[:ruby_reactor_coordinated] ||= []` for the duration of the yield and pop in `ensure`. That way an executor-driven run is never coordinated twice (test (9)).
275
+ - Here `step_config:` is the step class itself. It already has the five readers (T006), but its `name` is the class name (a String), not a step symbol. In `StepCoordination`, derive attribution as `step_name = step_config.is_a?(RubyReactor::Dsl::StepConfig) ? step_config.name : step_config.name.to_s`, and use `step_name` everywhere a step name appears (events, errors, trace).
276
+ - [ ] T038 [US4] Run `reentrancy_spec.rb` until green. Re-run `spec/ruby_reactor/dsl/async_reactor_locks_spec.rb` and `spec/ruby_reactor/integration/locking_spec.rb` unchanged (SC-012)
277
+
278
+ **Checkpoint**: All P1 stories green. Reactor-level behavior is unchanged
279
+
280
+ ---
281
+
282
+ ## Phase 7: User Story 5 - The whole coordination family is available per step (Priority: P2)
283
+
284
+ **Goal**: Semaphore, rate limit, and period (dedup) work at step level, in the fixed order of contract §3. The ordered lock is Phase 11.
285
+
286
+ **Independent Test**: Each primitive declared on a step matches the reactor-level behavior narrowed to the step.
287
+
288
+ ### Tests for User Story 5 ⚠️
289
+
290
+ - [ ] T039 [P] [US5] Write `spec/ruby_reactor/step_coordination/primitives_spec.rb` (semaphore, rate limit, period, ordering) and cover:
291
+ - (1) `with_semaphore(limit: 2, wait: 10)`: 5 threads, `max_concurrency == 2`, all succeed (US5-1).
292
+ - (2) `limit: 1` semaphore key appears in `held_lock_keys` during the body.
293
+ - (3) `with_rate_limit(limit: 2, period: :minute)`: 3 sync runs give 2 successes and a 3rd Failure with `exception_class == "RubyReactor::RateLimit::ExceededError"`. In the live-Sidekiq lane the 3rd parks with `next_retry_at` about `retry_after_seconds` (US5-2).
294
+ - (4) `with_rate_limit(:registered)` uses `config.rate_limits`. An unknown name gives a non-retryable Failure, not a park.
295
+ - (5) `with_period(every: :hour)`: the second run in the bucket gives the step `be_skipped` (reason `:period`), later steps run, and the reactor result is Success, not Halt (FR-003, US5-3).
296
+ - (6) The bucket is not marked when the step fails.
297
+ - (7) Period plus lock: two threads racing the same bucket give exactly one body execution. The re-check under the lock closes the race (D3 step 6).
298
+ - (8) A step declaring both lock and semaphore, and two different steps declaring the same pair, run concurrently without deadlock, and the release order is semaphore then lock, asserted through recorded `:semaphore_released` before `:lock_released` events (FR-008).
299
+
300
+ ### Implementation for User Story 5
301
+
302
+ - [ ] T040 [US5] Restructure `StepCoordination#around_run` into the fixed order (contract §3, D3) so each primitive is a small private method:
303
+ - ordered gate (stub until T060)
304
+ - `period_seen?` fast check, which returns `RubyReactor.Skipped(nil, reason: :period, step_name:)` without yielding
305
+ - rate limit
306
+ - lock
307
+ - semaphore
308
+ - period re-check under the hold
309
+ - yield
310
+ - mark period on a plain Success (not `Skipped`, not `Halt`)
311
+ - Release in reverse in nested `ensure`s. Add a `# order: see contracts/dsl-surface.md §3` comment
312
+ - [ ] T041 [US5] Implement the rate limit in `StepCoordination`, mirroring `Executor#check_rate_limit`:
313
+ - Named config: `key_base = name.to_s` and `limits = RubyReactor.configuration.rate_limits.fetch(name)`. `UnknownLimitError` propagates as a normal non-retryable Failure, not `Contended`.
314
+ - Inline config: `key_base = key_for(config)`.
315
+ - `RubyReactor::RateLimit.new(key_base, limits:).check_and_increment!`. `ExceededError` becomes `Contended(primitive: :rate_limit)`.
316
+ - [ ] T042 [US5] Implement the period gate in `StepCoordination` using `RubyReactor::Period.key(key_for(config), config[:every])`, `storage_adapter.period_seen?`, and `period_mark(key, RubyReactor::Period.ttl_seconds(every))`, mirroring `Executor#check_period_gate` / `#mark_period_on_success`. The mark happens only after the body returns a plain Success
317
+ - [ ] T043 [US5] Implement the semaphore in `StepCoordination`:
318
+ - `RubyReactor::Semaphore.new(key, limit:, wait: wait_for(config[:wait]))`, then `acquire` and `middlewares.on(:semaphore_acquired, key, limit, context)`, then the T032 registry push when `limit == 1`.
319
+ - Failure: `:semaphore_failed` event, then raise `Contended(primitive: :semaphore)`.
320
+ - Release: `:semaphore_released`, then pop.
321
+ - [ ] T044 [US5] Run `primitives_spec.rb` (non-ordered sections) until green
322
+
323
+ **Checkpoint**: Four of five primitives at parity
324
+
325
+ ---
326
+
327
+ ## Phase 8: User Story 6 - Coordination is re-taken to undo the work it protected (Priority: P2)
328
+
329
+ **Goal**: `compensate` and `undo` of a step run under that step's lock and semaphore, keyed from the same arguments. Rate limit, period, and ordered lock never gate them.
330
+
331
+ **Independent Test**: With a compensation that sleeps, a concurrent execution cannot enter the step's forward body until the compensation releases.
332
+
333
+ ### Tests for User Story 6 ⚠️
334
+
335
+ - [ ] T045 [P] [US6] Write `spec/ruby_reactor/step_coordination/rollback_spec.rb` and cover:
336
+ - (1) Step `:charge` has `with_lock`, succeeds, then a later step fails. During `:charge`'s `undo`, the recorder shows it holding `"acct:1"` (`lock_info` owner == root id) (US6-1).
337
+ - (2) The undo sleeps 1s while a second thread runs the same reactor: `overlapped?(:charge_undo, :charge_run) == false` (SC-009, US6-2).
338
+ - (3) The same for `compensate`, when `:charge` itself fails.
339
+ - (4) `with_semaphore(limit: 1)` is re-taken for undo.
340
+ - (5) A step with `with_rate_limit(limit: 1, period: :minute)` and `with_period` whose quota and bucket are already exhausted still runs its undo/compensate (US6-3, FR-025).
341
+ - (6) The key is held externally during rollback with `wait: 0`: compensate yields `Error::CompensationError` whose message names the key, and undo leaves an `:undo_failure` trace entry naming the key. Neither is silently skipped (US6-4, FR-026).
342
+ - (7) The live-Sidekiq lane: a compensation in a worker never parks. It waits the configured `wait` and then reports.
343
+
344
+ ### Implementation for User Story 6
345
+
346
+ - [ ] T046 [US6] Implement `StepCoordination#around_rollback { }`:
347
+ - Take only lock then semaphore (release in reverse), using the configured `wait` directly, NOT `wait_for`. Rollback never parks: the execution is already mid-failure.
348
+ - Use the same `key_for(arguments)`, and the same owner, so it nests with any live reactor hold.
349
+ - On `Lock::AcquisitionError` / `Semaphore::AcquisitionError`, return `RubyReactor::Failure("could not re-acquire <primitive> '<key>' for rollback of :<step>: <msg>", retryable: false, step_name:)` without yielding.
350
+ - A `KeyError` returns the same shape.
351
+ - [ ] T047 [US6] Wrap both call sites in `lib/ruby_reactor/executor/compensation_manager.rb`, the `catch(StepSignals::TAG)` bodies inside `compensate_step` and `undo_step`, in `StepCoordination.new(step_config:, arguments:, context: @context, reactor_class: @context.reactor_class, middlewares:).around_rollback { ... }` unless `StepCoordination.none?(step_config)`. The existing `Failure` branches then report acquisition failure through `:failed_compensation` → `CompensationError`, and through `:failed_undo` → the trace, with no new reporting code
352
+ - [ ] T048 [US6] Run `rollback_spec.rb` until green
353
+
354
+ **Checkpoint**: Forward and rollback paths are both protected
355
+
356
+ ---
357
+
358
+ ## Phase 9: User Story 7 - Operators can see step coordination (Priority: P2)
359
+
360
+ **Goal**: Step holds and parks are visible in events, logs, the execution trace, and the dashboard, attributed to the step. A park is never shown as a failure.
361
+
362
+ **Independent Test**: While a step lock is held and another execution is parked on it, the dashboard payload and logs identify the step, key, and holder.
363
+
364
+ ### Tests for User Story 7 ⚠️
365
+
366
+ - [ ] T049 [P] [US7] Write `spec/ruby_reactor/step_coordination/observability_spec.rb` and cover:
367
+ - (1) A recording middleware receives `:lock_acquired`, `:lock_released`, and `:lock_failed` with the key, and `context.current_step == :charge` at each call (FR-028, US7-3). The same holds for semaphore events.
368
+ - (2) A park emits one log line matching `event="ruby_reactor.step_coordination.parked" reactor=... step=:charge key="acct:1" primitive=:lock attempt=1 execution_id=...` (constitution IV: key=value).
369
+ - (3) A parked context has `status` not `:failed`, has `private_data[:step_contention]` naming step/key/primitive, and emits no `:failed_step` for the park (US7-4).
370
+ - (4) `RubyReactor::Web::CoordinationSerializer` output for a context whose `:charge` ran includes `steps: [{ step: "charge", primitive: "lock", key: "acct:1", state: ..., owner: ... }]`. For a parked context it includes `waiting: { step:, key:, primitive:, attempts:, next_attempt_at: }` (US7-1, US7-2, SC-010).
371
+ - (5) The sync contention Failure's `step_name`, `reactor_name`, and message carry the reactor, step, and key (US7-2).
372
+
373
+ ### Implementation for User Story 7
374
+
375
+ - [ ] T050 [US7] Emit `:failed_step` correctly around parks in `StepExecutor#execute_step` (`lib/ruby_reactor/executor/step_executor.rb`). A `RetryQueuedResult` from a contention park must go to `:complete_step`, as it does today for retry requeues; confirm this and add a comment. Add the structured `parked` log line in the T025 park branch, using the existing `log_async_event` formatting (extend it to accept extra key=value fields)
376
+ - [ ] T051 [US7] Attribute step events in `lib/ruby_reactor/open_telemetry.rb`: where lock/semaphore events are handled, add a `ruby_reactor.step` span attribute from `context.current_step` when present. If those events are not handled there, skip it and note that in the task
377
+ - [ ] T052 [US7] Extend `lib/ruby_reactor/web/coordination_serializer.rb` with `build_steps(reactor_class, context)`:
378
+ - For each `reactor_class.steps` entry with `declares_coordination?`, find the latest `execution_trace` entry `type: :run, step: name` and compute the key from its `arguments` with the step's `key_proc`, rescuing to `key: nil, error:`.
379
+ - Report lock state through the existing `build_lock` / `build_semaphore` helpers, and `state: "pending"` for steps not yet reached.
380
+ - Add `waiting:` from `context.private_data[:step_contention]`.
381
+ - Call it from the existing `build` call site in `lib/ruby_reactor/web/api.rb` and pass the context.
382
+ - [ ] T053 [US7] Render step-level entries in the dashboard coordination panel under `lib/ruby_reactor/web/public/`: one row per step with step/primitive/key/state/owner, plus a "waiting on" badge for `waiting`, visually distinct from failure
383
+ - [ ] T054 [US7] Add a matcher `have_contended_at(step_name)` to `lib/ruby_reactor/rspec/matchers.rb`. It matches a result/context whose `execution_trace` contains `type: :contention_park, step: step_name`, with an optional `.on(key)` chain. The demo spec needs it (Constitution VI), and it doubles as the shipped way to assert US7-4
384
+ - [ ] T055 [US7] Run `observability_spec.rb` until green
385
+
386
+ **Checkpoint**: Operators can see step, key, holder, and park status
387
+
388
+ ---
389
+
390
+ ## Phase 10: User Story 8 - Inline steps can declare coordination too (Priority: P3)
391
+
392
+ **Goal**: The inline `step :x do with_lock { } end` behaves exactly like the class form.
393
+
394
+ **Independent Test**: The US1 lock scenarios pass with an inline-declared step.
395
+
396
+ ### Tests for User Story 8 ⚠️
397
+
398
+ - [ ] T056 [P] [US8] Write `spec/ruby_reactor/step_coordination/inline_spec.rb`:
399
+ - Extract `lock_spec.rb` scenarios (1)–(5) into `shared_examples "step-scoped lock"`, parameterized by a `let(:reactor_class)`, and run them for a class-declared step and an inline-declared step with a `run` block.
400
+ - Add: moving the inline declaration verbatim into a step class gives identical recorder output.
401
+ - Add: an inline `compensate` block runs under the lock (US6 via `step_config`).
402
+
403
+ ### Implementation for User Story 8
404
+
405
+ - [ ] T057 [US8] Run `inline_spec.rb`. The inline path is expected to work already via T007 + T013 (`run_block` goes through `run_step_implementation`). Fix only real gaps, such as `StepCoordination` reading `step_config.name` for attribution when there is no `impl`
406
+
407
+ **Checkpoint**: US1–US8 complete, apart from the step-level ordered lock
408
+
409
+ ---
410
+
411
+ ## Phase 11: User Story 5 (cont.) - Step-level ordered lock (Priority: P2, separable — cut first if schedule tightens)
412
+
413
+ **Goal**: `with_ordered_lock` on a step sequences executions at that step in arrival order. With `strict: true`, a failed earlier position short-circuits later positions at that step (`Skipped`), and the rest of the workflow continues.
414
+
415
+ **Independent Test**: Executions reaching the step out of order pass through it in arrival order. A failed position makes later positions skip the step.
416
+
417
+ ### Tests for User Story 5 (cont.) ⚠️
418
+
419
+ - [ ] T058 [P] [US5] Add an ordered section to `spec/ruby_reactor/step_coordination/primitives_spec.rb`, live-Sidekiq lane. Cover:
420
+ - (1) 5 background runs of a reactor whose FIRST step has `with_ordered_lock { "seq" }` and a random 0–0.3s body: the recorder shows step bodies in nonce order and never overlapping (US5-4).
421
+ - (2) A later unordered step of run N may overlap run N+1's ordered step ("surrounding steps are unaffected").
422
+ - (3) `strict: true` with position 2 failing: positions 3+ have the ordered step `be_skipped` with reason `:ordered_lock_chain_failed`, and their later steps still run (FR-004, US5-5).
423
+ - (4) `strict: false`: every position runs.
424
+ - (5) A contention redelivery reuses the same nonce: `private_data[:step_ordered_locks]["<step>"][:nonce]` is unchanged across parks.
425
+ - (6) Poison-pill: a position that never arrives is advanced past after `poison_pill_timeout`.
426
+ - (7) `have_ordered_lock_next` / `have_ordered_lock_last_completed` / `be_ordered_lock_drained` work on the step's key.
427
+ - (8) The ordered lock is NOT re-taken for rollback (data-model rollback table).
428
+
429
+ ### Implementation for User Story 5 (cont.)
430
+
431
+ - [ ] T059 [US5] Document the weaker guarantee on the macro itself in `lib/ruby_reactor/dsl/lockable.rb#with_ordered_lock`'s doc comment: "On a step, the position is assigned when the execution first REACHES the step (its key reads step arguments), so executions are ordered by arrival at that step, not by enqueue. Identical to the reactor form only when the step is first in its reactor." (research D8, contract §1)
432
+ - [ ] T060 [US5] Implement the ordered gate as the first stage of `StepCoordination#around_run`:
433
+ - On first arrival, `RubyReactor::OrderedLock.assign(key, ttl:)`, stashing `{ key:, nonce:, epoch:, poison_pill_timeout:, ttl:, strict: }` in `context.private_data[:step_ordered_locks][step_name.to_s]`. Reuse it when present (redelivery).
434
+ - Gate using the same `OrderedLock` API that `Executor::OrderedLockSupport#check_ordered_lock_gate` uses. Out of turn, raise `Contended(primitive: :ordered_lock, original: WaitError)`; it parks uncapped (T022). Synchronously the gate is not retried: it fails as contention.
435
+ - With `strict` and a failed chain, return `RubyReactor.Skipped(nil, reason: :ordered_lock_chain_failed, step_name:)` without taking anything else.
436
+ - Take nothing else while waiting for a turn (D3 step 1).
437
+ - [ ] T061 [US5] Add a per-step heartbeat and advance-on-terminal in `StepCoordination`:
438
+ - Start a heartbeat thread for the duration of the step body, reusing the refresh call and interval from `OrderedLockSupport#start_ordered_lock_heartbeat` (extract a shared class method rather than copy if it is more than a few lines).
439
+ - In `ensure`, when the step reached a terminal result (a Success, Skipped, or Failure value, and NOT a `Contended` raise), call `OrderedLockSupport.advance_with_retry(info, failed: result.is_a?(RubyReactor::Failure))` and delete the per-step stash.
440
+ - [ ] T062 [US5] Exclude the ordered lock from `around_rollback` (T046 already takes only lock/semaphore; add a spec-referenced comment), and exclude it from the async_step dispatch guard (T034 only considers lock and limit-1 semaphore)
441
+ - [ ] T063 [US5] Run the ordered section of `primitives_spec.rb` until green
442
+
443
+ **Checkpoint**: Five-primitive parity
444
+
445
+ ---
446
+
447
+ ## Phase 12: Polish & Cross-Cutting Concerns
448
+
449
+ **Purpose**: Docs, demo-app proof (Constitution VI), full-suite verification
450
+
451
+ - [ ] T064 [P] Update `documentation/locks_and_semaphores.md` with a new "Step-scoped coordination" section:
452
+ - The five macros on a class step and an inline step.
453
+ - The fixed acquisition order (contract §3).
454
+ - The entry-point table (contract §3).
455
+ - Contention on each path: parks in a worker, bounded by `lock_snooze_max_attempts`; waits then fails synchronously.
456
+ - Re-entrancy rules and the dispatch refusal.
457
+ - The rollback table.
458
+ - `with_period` skipping the step, not halting.
459
+ - The ordered-lock arrival caveat.
460
+ - "Reactor level for 'this whole workflow is exclusive'; step level for 'this one operation is exclusive'" (FR-031).
461
+ - [ ] T065 [P] Update `README.md`: add a short step-level `with_lock` example to the locking section, linking to `documentation/locks_and_semaphores.md`
462
+ - [ ] T066 [P] Add a `CHANGELOG.md` entry under `Features`: "Steps can declare `with_lock`, `with_semaphore`, `with_rate_limit`, `with_period`, `with_ordered_lock`, keyed on their own arguments" (MINOR, additive)
463
+ - [ ] T067 [P] Create `demo_app/app/reactors/step_lock_demo_reactor.rb` (class `StepLockDemoReactor`) with class-based steps only:
464
+ - `AuditStep` (unlocked, records), `ChargeStep` (`with_lock(wait: 2) { |a| "demo:acct:#{a[:account_id]}" }`, with `compensate` and `undo`), `NotifyStep` (unlocked), and a `fail_after_charge` input that makes a later step fail, to exercise compensation under the lock.
465
+ - [ ] T068 Register `demo:step_lock` in `demo_app/lib/tasks/demo_reactors.rake` with a `desc` and `[:environment, :flush_redis]`. It prints three sections:
466
+ - (1) Serialized: two threads, same account. Print enter/leave times of `:charge` showing no overlap.
467
+ - (2) Contended: two `background` runs on the same account. Wait for both to finish, then print both `completed` and the contention-park trace entry of the loser.
468
+ - (3) Compensated: run with `fail_after_charge: true` and print the undo trace showing the lock held (`be_locked` equivalent printed via `RubyReactor::Lock` info, owner = context id).
469
+ - [ ] T069 Write `demo_app/spec/reactors/step_lock_demo_reactor_spec.rb` (`type: :reactor`) using ONLY `lib/ruby_reactor/rspec.rb` surface:
470
+ - `test_reactor`, `be_success`, `have_run_step(:notify).after(:charge)`, `be_failure`, `be_locked`, `have_contended_at(:charge)` (T054), `drain_async_jobs`.
471
+ - If holding a key from outside a reactor cannot be expressed with the shipped surface, add a helper `hold_lock(key, owner: "spec") { ... }` to `lib/ruby_reactor/rspec/helpers.rb` in this same change, with a spec in `spec/ruby_reactor/rspec/helpers_spec.rb`. Never hand-roll Redis calls in the demo spec.
472
+ - [ ] T070 Check `docker-compose.yml`: `demo-redis` and `sidekiq` must be present and wired so `docker compose run --rm demo-app bin/rails demo:step_lock` needs no manual setup (Constitution VI.4). Add a missing service or env var in the same change
473
+ - [ ] T071 Run `docker compose run --rm demo-app bundle exec rspec spec/reactors/step_lock_demo_reactor_spec.rb` and `docker compose run --rm demo-app bin/rails demo:step_lock`. Confirm the three printed sections match quickstart Scenario 6 (SC-013)
474
+ - [ ] T072 Run full `bundle exec rspec` and `bundle exec rubocop` (no `--disable-pending-cops`). Compare against `specs/003-step-lock-declarations/baseline.txt` (T001): no previously passing example may fail (SC-012). Then delete `baseline.txt`
475
+ - [ ] T073 Walk the quickstart.md acceptance checklist (SC-001…SC-013), map each row to the spec example that proves it, and fix any gap
476
+ - [ ] T074 Update the `Complexity Tracking` table in `specs/003-step-lock-declarations/plan.md` if the ordered-lock phase was cut or deferred. Mark data-model.md's `ContentionState` as superseded by `RetryContext#contention_attempts` + `lock_snooze_*` config (Finding 7)
477
+
478
+ ---
479
+
480
+ ## Dependencies & Execution Order
481
+
482
+ ### Phase Dependencies
483
+
484
+ - **Setup (Phase 1)**: none
485
+ - **Foundational (Phase 2)**: depends on Setup. BLOCKS all stories
486
+ - **US1 (Phase 3)**: depends on Phase 2
487
+ - **US2 (Phase 4)**: depends on US1 (it verifies US1's scope)
488
+ - **US3 (Phase 5)**: depends on US1 (it replaces T014's temporary rescue)
489
+ - **US4 (Phase 6)**: depends on US1. T036 depends on US3's T020
490
+ - **US5 (Phase 7)**: depends on US1. T043 uses US4's T032 registry rule
491
+ - **US6 (Phase 8)**: depends on US1. The semaphore re-take in T046 needs US5's T043
492
+ - **US7 (Phase 9)**: depends on US3 (park trace/state) and US1
493
+ - **US8 (Phase 10)**: depends on US1. Its compensate example needs US6
494
+ - **US5 cont. (Phase 11)**: depends on US3 (the park path) and US5 (the fixed-order skeleton)
495
+ - **Polish (Phase 12)**: T067–T071 need US1, US3, US6, and T054. Docs can start after US4
496
+
497
+ ### Story graph
498
+
499
+ ```text
500
+ Setup → Foundational → US1 ─┬─> US2
501
+ ├─> US3 ─┬─> US4 (T036)
502
+ │ ├─> US7
503
+ │ └─> US5-ordered (Phase 11)
504
+ ├─> US4 ──> US5 (T032→T043) ──> US6 ──> US8 (compensate example)
505
+ └─> US8 (core)
506
+ └──> Polish
507
+ ```
508
+
509
+ ### Within each story
510
+
511
+ - The spec task comes first and must fail before implementation (Constitution III)
512
+ - `StepCoordination` changes come before call-site wiring
513
+ - The story's run-until-green task closes the phase
514
+
515
+ ### Parallel opportunities
516
+
517
+ - T002, T003 (Setup): different files
518
+ - T005 alongside T006–T009 authoring (spec vs lib)
519
+ - Every story's spec-writing task ([P]) can be drafted as soon as Phase 2 lands. Write
520
+ `lock_spec`, `scope_spec`, `contention_spec`, `reentrancy_spec`, `primitives_spec`,
521
+ `rollback_spec`, `observability_spec`, and `inline_spec` in parallel
522
+ - T019 (retry_context.rb) and T020 (worker.rb) in parallel
523
+ - T064, T065, T066, T067 (docs and demo reactor) in parallel
524
+ - Note: most lib tasks touch `step_coordination.rb` or `step_executor.rb`, so lib work within and
525
+ across stories is mostly sequential. Parallelism is in specs and docs, not lib
526
+
527
+ ---
528
+
529
+ ## Parallel Example: after Phase 2
530
+
531
+ ```bash
532
+ # Draft all failing specs at once (different files):
533
+ Task: "T011 lock_spec.rb"
534
+ Task: "T016 scope_spec.rb"
535
+ Task: "T018 contention_spec.rb"
536
+ Task: "T029 reentrancy_spec.rb"
537
+ Task: "T039 primitives_spec.rb"
538
+ Task: "T045 rollback_spec.rb"
539
+
540
+ # Within US3:
541
+ Task: "T019 contention counter in lib/ruby_reactor/retry_context.rb"
542
+ Task: "T020 extract snooze_delay in lib/ruby_reactor/worker.rb"
543
+ ```
544
+
545
+ ---
546
+
547
+ ## Implementation Strategy
548
+
549
+ ### MVP (US1 only)
550
+
551
+ Phases 1 → 2 → 3. Stop and validate `lock_spec.rb`. A class step's `with_lock` then serializes
552
+ that step synchronously, which is the literal request.
553
+
554
+ ### Incremental delivery
555
+
556
+ 1. MVP (US1), then US2 (proves the value over reactor-level locks)
557
+ 2. US3: worker parking. Required before anyone runs this in Sidekiq in production
558
+ 3. US4: re-entrancy and hand-off refusal. With US1–US4 all P1 stories are done, and this is the
559
+ release candidate for `with_lock`-only
560
+ 4. US5 (semaphore/rate/period), US6 (rollback), US7 (observability), US8 (inline)
561
+ 5. Phase 11 (ordered lock): separable, and the first to cut (plan Complexity Tracking)
562
+ 6. Polish: the demo-app artifacts are required before merge (Constitution VI)
563
+
564
+ ---
565
+
566
+ ## Notes
567
+
568
+ - `[P]` = different files, no dependency on an incomplete task
569
+ - Commit after each phase checkpoint
570
+ - Never use `Sidekiq::Testing.inline!` in `spec/ruby_reactor/step_coordination/`
571
+ - The line numbers cited are from branch `step_validations` at commit `845b010`. Re-locate by
572
+ method name if they have drifted
@@ -0,0 +1,22 @@
1
+ # Future improvements for review
2
+
3
+ ## Guard
4
+
5
+ I want to add another hook feature to the step: `guard`
6
+ A guard concept is to execute after validations and before run.
7
+ example:
8
+
9
+ ```ruby
10
+ class SendEmail < RubyReactor::Step
11
+ input :email, :string, format?: /\A[^@\s]+@[^@\s]+\z/
12
+
13
+ def guard
14
+ fail!("Prevent spamming") if EmailService.sent_today?(inputs[:email])
15
+ success! # optional
16
+ end
17
+
18
+ def run
19
+ # do the work
20
+ end
21
+ end
22
+ ```