ruby_reactor 0.8.2 → 0.8.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.claude/skills/speckit-review/SKILL.md +324 -0
- data/.release-please-manifest.json +1 -1
- data/.specify/extensions.yml +10 -0
- data/.specify/feature.json +1 -1
- data/.specify/workflows/speckit/workflow.yml +13 -1
- data/.specify/workflows/workflow-registry.json +2 -2
- data/CHANGELOG.md +82 -0
- data/CLAUDE.md +2 -2
- data/README.md +35 -2
- data/lib/ruby_reactor/adapters/active_job/router.rb +19 -0
- data/lib/ruby_reactor/adapters/sidekiq/router.rb +21 -0
- data/lib/ruby_reactor/context.rb +26 -0
- data/lib/ruby_reactor/context_serializer.rb +4 -2
- data/lib/ruby_reactor/dsl/interrupt_builder.rb +14 -0
- data/lib/ruby_reactor/dsl/lockable.rb +76 -21
- data/lib/ruby_reactor/dsl/step_builder.rb +112 -1
- data/lib/ruby_reactor/error/async_result_pending.rb +1 -1
- data/lib/ruby_reactor/error/execution_parked.rb +16 -0
- data/lib/ruby_reactor/error/reactor_contention_park.rb +26 -0
- data/lib/ruby_reactor/error/step_contention_park.rb +26 -0
- data/lib/ruby_reactor/executor/async_step_dispatch.rb +109 -3
- data/lib/ruby_reactor/executor/compensation_manager.rb +99 -17
- data/lib/ruby_reactor/executor/ordered_lock_support.rb +76 -44
- data/lib/ruby_reactor/executor/result_handler.rb +31 -11
- data/lib/ruby_reactor/executor/retry_manager.rb +9 -1
- data/lib/ruby_reactor/executor/step_coordination.rb +788 -0
- data/lib/ruby_reactor/executor/step_executor.rb +115 -11
- data/lib/ruby_reactor/executor.rb +90 -20
- data/lib/ruby_reactor/map/element_executor.rb +24 -2
- data/lib/ruby_reactor/map/helpers.rb +35 -11
- data/lib/ruby_reactor/max_retries_exhausted_failure.rb +2 -2
- data/lib/ruby_reactor/open_telemetry.rb +61 -24
- data/lib/ruby_reactor/retry_context.rb +31 -2
- data/lib/ruby_reactor/rspec/helpers.rb +15 -0
- data/lib/ruby_reactor/rspec/matchers.rb +92 -0
- data/lib/ruby_reactor/rspec/test_subject.rb +7 -1
- data/lib/ruby_reactor/step/async_reactor_step.rb +40 -24
- data/lib/ruby_reactor/step/compose_step.rb +14 -3
- data/lib/ruby_reactor/step.rb +49 -7
- data/lib/ruby_reactor/step_sweeper.rb +29 -1
- data/lib/ruby_reactor/step_worker.rb +260 -37
- data/lib/ruby_reactor/version.rb +1 -1
- data/lib/ruby_reactor/web/api.rb +72 -7
- data/lib/ruby_reactor/web/coordination_serializer.rb +120 -2
- data/lib/ruby_reactor/web/public/assets/{index-Dw4KV4QY.js → index-CeZU-ESu.js} +9 -9
- data/lib/ruby_reactor/web/public/index.html +1 -1
- data/lib/ruby_reactor/worker.rb +56 -30
- data/lib/ruby_reactor.rb +27 -5
- data/specs/future_improvements.md +250 -0
- metadata +8 -28
- data/specs/002-step-input-contracts/checklists/requirements.md +0 -49
- data/specs/002-step-input-contracts/contracts/dsl-surface.md +0 -193
- data/specs/002-step-input-contracts/data-model.md +0 -115
- data/specs/002-step-input-contracts/plan.md +0 -165
- data/specs/002-step-input-contracts/quickstart.md +0 -170
- data/specs/002-step-input-contracts/research.md +0 -233
- data/specs/002-step-input-contracts/spec.md +0 -359
- data/specs/002-step-input-contracts/tasks.md +0 -367
- data/specs/004-inheritable-step-class/checklists/requirements.md +0 -40
- data/specs/004-inheritable-step-class/contracts/step-lifecycle.md +0 -85
- data/specs/004-inheritable-step-class/data-model.md +0 -116
- data/specs/004-inheritable-step-class/plan.md +0 -174
- data/specs/004-inheritable-step-class/quickstart.md +0 -112
- data/specs/004-inheritable-step-class/research.md +0 -308
- data/specs/004-inheritable-step-class/spec.md +0 -316
- data/specs/004-inheritable-step-class/tasks.md +0 -258
- data/specs/active_job.md +0 -259
- data/specs/deferred-003-step-lock-declarations/checklists/requirements.md +0 -51
- data/specs/deferred-003-step-lock-declarations/contracts/dsl-surface.md +0 -154
- data/specs/deferred-003-step-lock-declarations/data-model.md +0 -131
- data/specs/deferred-003-step-lock-declarations/plan.md +0 -166
- data/specs/deferred-003-step-lock-declarations/quickstart.md +0 -169
- data/specs/deferred-003-step-lock-declarations/research.md +0 -196
- data/specs/deferred-003-step-lock-declarations/spec.md +0 -447
- data/specs/deferred-003-step-lock-declarations/tasks.md +0 -572
- data/specs/possible_feature.md +0 -22
|
@@ -1,166 +0,0 @@
|
|
|
1
|
-
# Implementation Plan: Step-Scoped Coordination
|
|
2
|
-
|
|
3
|
-
**Branch**: `step_validations` | **Date**: 2026-09-10 | **Spec**: [spec.md](./spec.md)
|
|
4
|
-
|
|
5
|
-
**Input**: Feature specification from `specs/003-step-lock-declarations/spec.md`
|
|
6
|
-
|
|
7
|
-
## Summary
|
|
8
|
-
|
|
9
|
-
Let a step declare its own coordination — exclusivity, concurrency ceiling, rate ceiling,
|
|
10
|
-
dedup window, strict ordering — keyed on the step's own resolved arguments, so one step of a
|
|
11
|
-
workflow can be serialized without serializing the workflow.
|
|
12
|
-
|
|
13
|
-
The declaration surface already exists: `Dsl::Lockable`'s five macros are a self-contained
|
|
14
|
-
module whose only contract is "a key proc that receives a hash". Steps host it unchanged and
|
|
15
|
-
pass their arguments where the reactor passes its inputs. Enforcement is a `StepCoordination`
|
|
16
|
-
object the executor wraps around the step body, in the same fixed order the reactor uses.
|
|
17
|
-
|
|
18
|
-
Contention parks the execution rather than failing it, reusing `requeue_job_for_step_retry` —
|
|
19
|
-
which already persists the context with `current_step` set and re-enqueues — with contention
|
|
20
|
-
attempts counted separately from failure retries. Synchronously there is no queue to park
|
|
21
|
-
into, so that path waits then fails.
|
|
22
|
-
|
|
23
|
-
Re-entrancy reuses every existing nested-workflow primitive unchanged: holds owned by the root
|
|
24
|
-
context id, the adapter's nesting count, the `held_lock_keys` registry, and the dispatch-time
|
|
25
|
-
deadlock guard — which already covers step holds, since it reads that registry.
|
|
26
|
-
|
|
27
|
-
Design decisions and evidence: [research.md](./research.md).
|
|
28
|
-
|
|
29
|
-
## Technical Context
|
|
30
|
-
|
|
31
|
-
**Language/Version**: Ruby >= 3.0.0
|
|
32
|
-
|
|
33
|
-
**Primary Dependencies**: redis ~> 5.0 (every primitive is Redis-backed), sidekiq ~> 7.0
|
|
34
|
-
(the park-and-retry path), zeitwerk ~> 2.6. No new dependency.
|
|
35
|
-
|
|
36
|
-
**Storage**: Redis. Step holds use the same key spaces, TTLs, and Lua primitives as reactor
|
|
37
|
-
holds (`storage/redis_locking.rb`, `storage/redis_ordered_locking.rb`). New per-step state is
|
|
38
|
-
confined to `context.private_data` (contention counter, per-step ordered-lock nonce), which
|
|
39
|
-
already round-trips through `ContextSerializer`.
|
|
40
|
-
|
|
41
|
-
**Testing**: RSpec against real Redis (constitution III). Concurrency claims need genuine
|
|
42
|
-
parallelism — overlap detection across processes/threads, not mocked timing. The park-and-
|
|
43
|
-
retry path must be exercised with a real Sidekiq worker, not `Sidekiq::Testing.inline!`.
|
|
44
|
-
|
|
45
|
-
**Target Platform**: Ruby library, sync and Sidekiq-backed async execution
|
|
46
|
-
|
|
47
|
-
**Project Type**: Library / DSL
|
|
48
|
-
|
|
49
|
-
**Performance Goals**: A step declaring nothing pays one nil check per step. A step declaring
|
|
50
|
-
coordination pays the same Redis round-trips the reactor-level equivalent pays today, moved
|
|
51
|
-
from once-per-run to once-per-step-execution. The point of the feature is that the critical
|
|
52
|
-
section shrinks, so end-to-end throughput under contention should improve, not regress.
|
|
53
|
-
|
|
54
|
-
**Constraints**: Additive and SemVer-MINOR — no existing reactor changes behavior. Coordination
|
|
55
|
-
must never be held across a process hand-off. The critical section must stay minimal:
|
|
56
|
-
acquisition happens after guards and after argument validation.
|
|
57
|
-
|
|
58
|
-
**Scale/Scope**: ~10 library files touched, 2-3 new, plus demo-app artifacts and docs. The
|
|
59
|
-
ordered-lock phase is roughly the weight of the other four primitives combined.
|
|
60
|
-
|
|
61
|
-
## Constitution Check
|
|
62
|
-
|
|
63
|
-
*GATE: passed before Phase 0. Re-checked after Phase 1 design — see below.*
|
|
64
|
-
|
|
65
|
-
| Principle | Assessment |
|
|
66
|
-
|---|---|
|
|
67
|
-
| **I. Gem-First Design** | ✅ Entirely inside `lib/`. Redis and Sidekiq usage stays behind the existing adapter and router boundaries; callers still provide their own connections. |
|
|
68
|
-
| **II. Saga Pattern Integrity** | ✅ The strongest alignment in this feature. Coordination is re-taken for compensate/undo (FR-024), so rollback of a protected operation is protected too — closing a race the reactor-level lock leaves open whenever rollback outlives the reactor's own hold. Contention parks rather than fails, so routine contention never triggers spurious compensation. Nothing changes which steps run or in what order (FR-014). |
|
|
69
|
-
| **III. Test-First with Real Infrastructure** | ✅ Non-negotiable here: every claim is a concurrency claim. Real Redis, real Sidekiq for the park path. `Sidekiq::Testing.inline!` is explicitly wrong for this feature — it re-enters the worker synchronously inside the holding frame (the reason `acquire_context_lock` skips itself under it, `executor.rb:470`). |
|
|
70
|
-
| **IV. Observability by Default** | ✅ FR-028/FR-029. Events carry the step name; the dashboard's coordination view learns step-level state; a contention-parked execution is distinguishable from a failed one, reusing the `:snooze_reactor` precedent that already keeps snooze rounds from reading as phantom failures. |
|
|
71
|
-
| **V. Simplicity and SemVer** | ⚠️ Justified. MINOR and fully additive — a step declaring nothing is unaffected. But the scope is five primitives where the request was one, and YAGNI applies to four of them; the step-level ordered lock in particular invents a guarantee (order-of-arrival) weaker than the one its name implies. Recorded in Complexity Tracking, sequenced last, and flagged as the first thing to cut. |
|
|
72
|
-
| **VI. Demo-App Proof of Feature** | ✅ Blocking work: example reactor + `demo:` rake task + spec using only shipped matchers + `docker compose run`. `be_locked`, `have_available_tokens`, `have_held_tokens`, `have_rate_limit_count`, `be_period_marked`, and the ordered-lock matchers already exist; step-scoped assertions are expected to need at least one addition (a step-attributed hold), which goes into `lib/ruby_reactor/rspec/` in the same change rather than being worked around. |
|
|
73
|
-
|
|
74
|
-
**Post-design re-check**: no new violations. No new dependency, no new storage primitive, no
|
|
75
|
-
new failure shape — the design routes a second declaration site into mechanisms that already
|
|
76
|
-
exist. The two carried items are scope (five primitives) and the ordered-lock guarantee gap.
|
|
77
|
-
|
|
78
|
-
## Project Structure
|
|
79
|
-
|
|
80
|
-
### Documentation (this feature)
|
|
81
|
-
|
|
82
|
-
```text
|
|
83
|
-
specs/003-step-lock-declarations/
|
|
84
|
-
├── plan.md # This file
|
|
85
|
-
├── spec.md # Feature specification
|
|
86
|
-
├── research.md # Phase 0 — current-state findings and design decisions
|
|
87
|
-
├── data-model.md # Phase 1 — declaration/hold entities and lifecycle
|
|
88
|
-
├── quickstart.md # Phase 1 — how to run and verify
|
|
89
|
-
├── contracts/
|
|
90
|
-
│ └── dsl-surface.md # Phase 1 — public DSL, semantics per primitive, errors
|
|
91
|
-
├── checklists/
|
|
92
|
-
│ └── requirements.md # Spec quality checklist (complete)
|
|
93
|
-
└── tasks.md # Phase 2 — /speckit-tasks output, NOT created here
|
|
94
|
-
```
|
|
95
|
-
|
|
96
|
-
### Source Code (repository root)
|
|
97
|
-
|
|
98
|
-
```text
|
|
99
|
-
lib/ruby_reactor/
|
|
100
|
-
├── dsl/
|
|
101
|
-
│ ├── lockable.rb # unchanged module, now also hosted by steps
|
|
102
|
-
│ └── step_builder.rb # + the five macros for inline steps; refuse on
|
|
103
|
-
│ # interrupt steps (D6); config onto StepConfig
|
|
104
|
-
├── step.rb # + host Lockable macros; introspection (D1)
|
|
105
|
-
├── executor/
|
|
106
|
-
│ ├── step_coordination.rb # NEW — acquire/release in fixed order, contention
|
|
107
|
-
│ │ # handling, park decision (D2, D3, D4)
|
|
108
|
-
│ ├── step_executor.rb # wrap the step body in StepCoordination
|
|
109
|
-
│ ├── retry_manager.rb # contention requeue + separate contention counter (D4)
|
|
110
|
-
│ └── compensation_manager.rb # re-take exclusion primitives for compensate/undo (FR-024)
|
|
111
|
-
├── step/
|
|
112
|
-
│ └── async_reactor_step.rb # deadlock guard also covers async_step dispatch (D5)
|
|
113
|
-
├── step_worker.rb # coordination around the worker-side step body
|
|
114
|
-
├── retry_context.rb # + contention attempt counter
|
|
115
|
-
├── web/coordination_serializer.rb # + step-level coordination state (D9)
|
|
116
|
-
└── rspec/matchers.rb # + step-attributed hold assertions as needed
|
|
117
|
-
|
|
118
|
-
spec/ruby_reactor/
|
|
119
|
-
├── step_coordination/lock_spec.rb # NEW — US1, US2 (real concurrency)
|
|
120
|
-
├── step_coordination/contention_spec.rb # NEW — US3 both paths, bounded retries
|
|
121
|
-
├── step_coordination/reentrancy_spec.rb # NEW — US4 incl. dispatch refusal
|
|
122
|
-
├── step_coordination/primitives_spec.rb # NEW — US5, one per primitive
|
|
123
|
-
├── step_coordination/rollback_spec.rb # NEW — US6
|
|
124
|
-
└── step_coordination/inline_spec.rb # NEW — US8 equivalence
|
|
125
|
-
|
|
126
|
-
demo_app/
|
|
127
|
-
├── app/reactors/step_lock_demo_reactor.rb # NEW — serialized, contended, compensated
|
|
128
|
-
├── lib/tasks/demo_reactors.rake # + demo:step_lock
|
|
129
|
-
└── spec/reactors/step_lock_demo_reactor_spec.rb # NEW — shipped matchers only
|
|
130
|
-
|
|
131
|
-
documentation/locks_and_semaphores.md # step-scoped forms, when to prefer which
|
|
132
|
-
README.md, CHANGELOG.md
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
**Structure Decision**: Existing layout kept. One new library file carries the feature
|
|
136
|
-
(`executor/step_coordination.rb`); everything else is an edit to the file that already owns
|
|
137
|
-
the concern. Specs get a `step_coordination/` directory because they are concurrency tests
|
|
138
|
-
with shared harness needs, not unit tests scattered across existing files.
|
|
139
|
-
|
|
140
|
-
## Phase 2 outline (for `/speckit-tasks`)
|
|
141
|
-
|
|
142
|
-
Dependency-ordered. Phases 1-6 deliver US1-US4 and US6-US8 in full.
|
|
143
|
-
|
|
144
|
-
1. **Declaration surface** — host `Lockable` on `Step` and `StepBuilder`, introspection,
|
|
145
|
-
refuse on interrupt steps. No enforcement yet.
|
|
146
|
-
2. **Exclusive lock enforcement (US1, US2)** — `StepCoordination` around the step body,
|
|
147
|
-
acquire/release, keep-alive, guard skip, key-computation failure. Real-concurrency specs.
|
|
148
|
-
3. **Re-entrancy (US4)** — root-context owner, registry push/pop, `async_step` dispatch guard.
|
|
149
|
-
4. **Contention (US3)** — requeue park, contention counter and ceiling, sync wait-then-fail.
|
|
150
|
-
5. **Rollback (US6)** — re-take exclusion primitives for compensate/undo; verify rate/dedup are
|
|
151
|
-
not applied.
|
|
152
|
-
6. **Remaining narrowing primitives (US5 partial)** — semaphore, rate limit, period-skips-step.
|
|
153
|
-
7. **Observability + inline steps (US7, US8)** — events, dashboard, matcher additions, inline
|
|
154
|
-
equivalence.
|
|
155
|
-
8. **Step-level ordered lock (US5 remainder)** — per-step nonce, heartbeat, advance-on-terminal,
|
|
156
|
-
strict chain skip. Last, and separable: see Complexity Tracking.
|
|
157
|
-
9. **Docs + demo** — `documentation/locks_and_semaphores.md`, README, CHANGELOG, demo reactor +
|
|
158
|
-
rake + spec, docker acceptance run.
|
|
159
|
-
|
|
160
|
-
## Complexity Tracking
|
|
161
|
-
|
|
162
|
-
| Violation | Why Needed | Simpler Alternative Rejected Because |
|
|
163
|
-
|-----------|------------|--------------------------------------|
|
|
164
|
-
| Five primitives at step level where the request named one (Principle V / YAGNI) | Explicit user decision after being shown the narrower option. Parity means an author never has to ask which primitives "work" on a step. | Shipping `with_lock` alone covers the stated use case and every acceptance scenario in US1-US4. It was offered and declined. The four extra primitives are sequenced after the core so the schedule can still absorb them being cut. |
|
|
165
|
-
| Step-level ordered lock provides a weaker guarantee than its reactor-level namesake | Included in the user's "all five" decision. Sequencing at step arrival is still useful for a step that sits first in its reactor, where arrival order equals enqueue order. | The reactor-level guarantee cannot be reproduced: the nonce would have to be assigned at enqueue, but the key expression reads arguments that do not exist until the step is reached (research Finding 5). Mitigation is documentation on the macro plus a demo that shows arrival ordering explicitly — not a silent redefinition of the word "ordered". |
|
|
166
|
-
| Contention behaves differently in a worker (park) than synchronously (wait, then fail) | Direct consequence of the chosen park-and-retry behavior; a synchronous run has no queue to park into. | Failing on both paths is simpler and was the recommended option; it was declined. `contention_wait` already encodes this exact split for reactor-level holds, so the divergence is inherited rather than invented, and it is one branch in one method. |
|
|
@@ -1,169 +0,0 @@
|
|
|
1
|
-
# Quickstart: Step-Scoped Coordination
|
|
2
|
-
|
|
3
|
-
**Feature**: `specs/003-step-lock-declarations/` | **Date**: 2026-09-10
|
|
4
|
-
|
|
5
|
-
How to run and verify this feature. DSL details: [contracts/dsl-surface.md](./contracts/dsl-surface.md).
|
|
6
|
-
Design rationale: [research.md](./research.md).
|
|
7
|
-
|
|
8
|
-
## Prerequisites
|
|
9
|
-
|
|
10
|
-
Every claim here is a concurrency claim, so real infrastructure is mandatory — mocked Redis or
|
|
11
|
-
`Sidekiq::Testing.inline!` cannot prove any of it. Inline testing mode is actively wrong for
|
|
12
|
-
this feature: it re-enters the worker synchronously inside the frame that already holds the
|
|
13
|
-
lock.
|
|
14
|
-
|
|
15
|
-
```bash
|
|
16
|
-
docker compose up -d redis-test # gem suite (port 6780)
|
|
17
|
-
docker compose up -d demo-redis sidekiq # demo app + a real worker
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
## Scenario 1 — one step serializes, the workflow does not (US1, US2)
|
|
21
|
-
|
|
22
|
-
```ruby
|
|
23
|
-
class ChargeStep < RubyReactor::Step
|
|
24
|
-
input :account_id
|
|
25
|
-
input :amount
|
|
26
|
-
|
|
27
|
-
with_lock { |args| "acct:#{args[:account_id]}" }
|
|
28
|
-
|
|
29
|
-
def run
|
|
30
|
-
Success(charge!(inputs))
|
|
31
|
-
end
|
|
32
|
-
end
|
|
33
|
-
|
|
34
|
-
class PaymentReactor < RubyReactor::Reactor
|
|
35
|
-
input :account_id
|
|
36
|
-
input :amount
|
|
37
|
-
|
|
38
|
-
step :audit, AuditStep # unlocked
|
|
39
|
-
step :charge, ChargeStep # locked on the account
|
|
40
|
-
step :notify, NotifyStep # unlocked
|
|
41
|
-
end
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
**Expected**: two concurrent runs with the same `account_id` never overlap inside `:charge`;
|
|
45
|
-
`:audit` and `:notify` of both runs overlap freely. Two runs with different `account_id`
|
|
46
|
-
overlap everywhere.
|
|
47
|
-
|
|
48
|
-
```bash
|
|
49
|
-
bundle exec rspec spec/ruby_reactor/step_coordination/lock_spec.rb
|
|
50
|
-
```
|
|
51
|
-
|
|
52
|
-
## Scenario 2 — contention parks instead of failing (US3)
|
|
53
|
-
|
|
54
|
-
```ruby
|
|
55
|
-
# Two worker-backed executions, same key:
|
|
56
|
-
PaymentReactor.run(account_id: 1, amount: 10) # via background dispatch
|
|
57
|
-
PaymentReactor.run(account_id: 1, amount: 20)
|
|
58
|
-
|
|
59
|
-
# => both complete successfully; the second after the first released.
|
|
60
|
-
# No compensation ran. The second was requeued, not failed.
|
|
61
|
-
```
|
|
62
|
-
|
|
63
|
-
Synchronously there is no queue to park into:
|
|
64
|
-
|
|
65
|
-
```ruby
|
|
66
|
-
PaymentReactor.run(account_id: 1, amount: 20) # in-process, key held elsewhere
|
|
67
|
-
# => Failure(Lock::AcquisitionError, reactor:, step: :charge, key: "acct:1")
|
|
68
|
-
# prior steps compensated, as with any step failure
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
```bash
|
|
72
|
-
bundle exec rspec spec/ruby_reactor/step_coordination/contention_spec.rb
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
Also asserted there: contention attempts are bounded, counted separately from failure retries,
|
|
76
|
-
and an execution over the ceiling reports contention rather than snoozing forever.
|
|
77
|
-
|
|
78
|
-
## Scenario 3 — re-entrancy matches nested reactors (US4)
|
|
79
|
-
|
|
80
|
-
```ruby
|
|
81
|
-
class OuterReactor < RubyReactor::Reactor
|
|
82
|
-
input :id
|
|
83
|
-
with_lock { |i| "k:#{i[:id]}" } # reactor holds it
|
|
84
|
-
step :work, LockingStep # step declares the same key
|
|
85
|
-
end
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
**Expected**: completes without waiting on itself; the key becomes available to other
|
|
89
|
-
executions only after the outermost release.
|
|
90
|
-
|
|
91
|
-
Where ownership cannot cross a process boundary, the hand-off is refused up front:
|
|
92
|
-
|
|
93
|
-
```ruby
|
|
94
|
-
# execution holds "k:1", then dispatches work that declares "k:1"
|
|
95
|
-
# => Failure at dispatch naming the key, the holder, and how to restructure.
|
|
96
|
-
# Never a silent wait.
|
|
97
|
-
```
|
|
98
|
-
|
|
99
|
-
```bash
|
|
100
|
-
bundle exec rspec spec/ruby_reactor/step_coordination/reentrancy_spec.rb
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
## Scenario 4 — rollback runs under the same exclusivity (US6)
|
|
104
|
-
|
|
105
|
-
```ruby
|
|
106
|
-
# :charge succeeds holding "acct:1", a later step fails, rollback reaches :charge
|
|
107
|
-
# => the compensation runs holding "acct:1"
|
|
108
|
-
# => a concurrent execution cannot enter :charge's forward work while it runs
|
|
109
|
-
```
|
|
110
|
-
|
|
111
|
-
Rate ceilings and dedup windows are deliberately *not* applied to compensation — cleanup is
|
|
112
|
-
never suppressed by a forward-work quota.
|
|
113
|
-
|
|
114
|
-
```bash
|
|
115
|
-
bundle exec rspec spec/ruby_reactor/step_coordination/rollback_spec.rb
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
## Scenario 5 — the other primitives (US5)
|
|
119
|
-
|
|
120
|
-
```bash
|
|
121
|
-
bundle exec rspec spec/ruby_reactor/step_coordination/primitives_spec.rb
|
|
122
|
-
```
|
|
123
|
-
|
|
124
|
-
Asserts, one per primitive:
|
|
125
|
-
|
|
126
|
-
- semaphore: at most N inside the step's work per key
|
|
127
|
-
- rate limit: further executions of the step contend rather than exceed the rate
|
|
128
|
-
- dedup window: the **step** is skipped and the workflow continues — the reactor is not halted
|
|
129
|
-
- ordered lock: executions pass through the step in sequence; stop-the-line short-circuits that
|
|
130
|
-
step for later positions
|
|
131
|
-
|
|
132
|
-
## Scenario 6 — end to end against real infrastructure
|
|
133
|
-
|
|
134
|
-
```bash
|
|
135
|
-
docker compose run --rm demo-app bin/rails demo:step_lock
|
|
136
|
-
```
|
|
137
|
-
|
|
138
|
-
**Expected output**: the serialized path (two executions, non-overlapping step bodies), the
|
|
139
|
-
contended path (one parked and retried, both completing), and the compensated path (rollback
|
|
140
|
-
holding the same key).
|
|
141
|
-
|
|
142
|
-
## Full suite
|
|
143
|
-
|
|
144
|
-
```bash
|
|
145
|
-
docker compose up -d redis-test
|
|
146
|
-
bundle exec rspec
|
|
147
|
-
bundle exec rubocop
|
|
148
|
-
|
|
149
|
-
docker compose run --rm demo-app bundle exec rspec spec/reactors/step_lock_demo_reactor_spec.rb
|
|
150
|
-
docker compose run --rm demo-app bin/rails demo:step_lock
|
|
151
|
-
```
|
|
152
|
-
|
|
153
|
-
## Acceptance checklist
|
|
154
|
-
|
|
155
|
-
| # | Claim | Verified by |
|
|
156
|
-
|---|---|---|
|
|
157
|
-
| SC-001 | Same-key step bodies never overlap | Scenario 1, sustained concurrent run |
|
|
158
|
-
| SC-002 | Unrelated steps still overlap | Scenario 1 |
|
|
159
|
-
| SC-003 | Released within one step boundary, all outcomes | Scenario 1 + failure/raise cases |
|
|
160
|
-
| SC-004 | Contention costs zero compensations | Scenario 2 |
|
|
161
|
-
| SC-005 | Worker path protected identically to in-process | Scenarios 1 and 6 |
|
|
162
|
-
| SC-006 | Nested holds on one key complete without self-waiting | Scenario 3 |
|
|
163
|
-
| SC-007 | Deadlocking hand-offs refused at dispatch | Scenario 3 |
|
|
164
|
-
| SC-008 | Killed holder frees the key without operator action | kill-process test in lock_spec |
|
|
165
|
-
| SC-009 | Compensation runs under the same exclusivity | Scenario 4 |
|
|
166
|
-
| SC-010 | Operator can see step, key, holder; park ≠ failure | dashboard + log assertions |
|
|
167
|
-
| SC-011 | Uncomputable key never runs the work | lock_spec |
|
|
168
|
-
| SC-012 | Existing reactor-level coordination tests unchanged | full suite |
|
|
169
|
-
| SC-013 | Demo shows serialized, contended, compensated | Scenario 6 |
|
|
@@ -1,196 +0,0 @@
|
|
|
1
|
-
# Phase 0 Research: Step-Scoped Coordination
|
|
2
|
-
|
|
3
|
-
**Feature**: `specs/003-step-lock-declarations/` | **Date**: 2026-09-10
|
|
4
|
-
|
|
5
|
-
Findings come from reading the current implementation. File references are to the state of
|
|
6
|
-
`step_validations` at the time of writing.
|
|
7
|
-
|
|
8
|
-
## Current state
|
|
9
|
-
|
|
10
|
-
| Concern | Where it lives |
|
|
11
|
-
|---|---|
|
|
12
|
-
| Declaration DSL | `Dsl::Lockable::ClassMethods` — `with_lock`, `with_semaphore`, `with_rate_limit`, `with_period`, `with_ordered_lock`. Included into `Reactor` only. |
|
|
13
|
-
| Acquisition | `Executor#acquire_locks` → `check_rate_limit`, `acquire_exclusive_lock`, `acquire_semaphore` (`executor.rb:350-360`) |
|
|
14
|
-
| Key derivation | `config[:key_proc].call(@context.inputs)` — reactor inputs |
|
|
15
|
-
| Owner | root context id (`executor.rb:501`) — the basis of re-entrancy |
|
|
16
|
-
| Held-key registry | `root.private_data[:held_lock_keys]` |
|
|
17
|
-
| Deadlock guard | `Step::AsyncReactorStep.detect_lock_deadlock` (`async_reactor_step.rb:65`) — refuses dispatch when the child declares a key the parent holds |
|
|
18
|
-
| Park / resume | `park_held_primitives!` + `consume_parked_primitives!`; `Lock#detach` / `#reattach` |
|
|
19
|
-
| Contention split | `Executor#contention_wait` (`executor.rb:566`) — `0` inside a worker (snooze instead of blocking), configured wait otherwise |
|
|
20
|
-
| Step retry requeue | `RetryManager#requeue_job_for_step_retry` — sets `current_step`, persists root context, `perform_in(delay, …)`, returns `RetryQueuedResult` |
|
|
21
|
-
| Ordered lock | `Executor::OrderedLockSupport` — nonce assigned at enqueue in `Reactor#run`, stashed in `private_data[:ordered_lock]`, gate at execute/resume, advance on terminal reactor result, heartbeat thread |
|
|
22
|
-
| Compensation | `CompensationManager#compensate_step` / `#undo_step` |
|
|
23
|
-
| Dashboard | `Web::CoordinationSerializer` — reads `reactor_class.lock_config` etc. |
|
|
24
|
-
|
|
25
|
-
### Finding 1 — the DSL needs no redesign, only a second host
|
|
26
|
-
|
|
27
|
-
`Dsl::Lockable::ClassMethods` is already a self-contained module of five macros whose only
|
|
28
|
-
contract is "a key proc that receives a hash". Reactor passes `context.inputs`; a step would
|
|
29
|
-
pass its resolved arguments. The declaration surface is reusable as-is, including its
|
|
30
|
-
`inherited` hook for subclass propagation.
|
|
31
|
-
|
|
32
|
-
### Finding 2 — `contention_wait` already encodes the sync/worker split the spec asks for
|
|
33
|
-
|
|
34
|
-
FR-015/FR-016's split is not new policy: `contention_wait` returns `0` inside a worker so the
|
|
35
|
-
job snoozes rather than blocking a thread, and the configured wait outside. Step-level
|
|
36
|
-
coordination gets the correct behavior on both paths by reusing it.
|
|
37
|
-
|
|
38
|
-
### Finding 3 — parking an execution at a step already exists
|
|
39
|
-
|
|
40
|
-
`requeue_job_for_step_retry` persists the root context with `current_step` set and enqueues a
|
|
41
|
-
delayed job; the redelivery resumes at that step. It is reached today only from retry-on-
|
|
42
|
-
failure, but nothing in it is failure-specific. Step contention can reuse it verbatim with a
|
|
43
|
-
contention delay, which is why FR-015 ("park, don't fail") costs one call rather than a new
|
|
44
|
-
mechanism.
|
|
45
|
-
|
|
46
|
-
### Finding 4 — the deadlock guard is keyed on the registry, not on reactors
|
|
47
|
-
|
|
48
|
-
`detect_lock_deadlock` reads `held_lock_keys` from the root context and compares against the
|
|
49
|
-
*child's* declared keys. Because step-held keys will land in the same registry, the existing
|
|
50
|
-
guard covers "a step holds K, its body dispatches an async_reactor that wants K" with no
|
|
51
|
-
change. It needs extending only to know about `async_step` dispatch, where the dispatched
|
|
52
|
-
step class may itself declare K.
|
|
53
|
-
|
|
54
|
-
### Finding 5 — ordered lock is structurally reactor-shaped
|
|
55
|
-
|
|
56
|
-
The nonce is assigned in `Reactor#run` at enqueue time, before any step exists, and the
|
|
57
|
-
advance fires on the *reactor's* terminal result. Its guarantee is "executions run in the
|
|
58
|
-
order they were enqueued". A step-level equivalent cannot assign at enqueue, because the key
|
|
59
|
-
expression reads arguments that are not resolved until the step is reached — so its guarantee
|
|
60
|
-
degrades to "in the order executions reached this step", which is a materially weaker promise.
|
|
61
|
-
See D8; this is the one primitive whose step-scoped meaning is not a simple narrowing.
|
|
62
|
-
|
|
63
|
-
### Finding 6 — a step holding coordination across a park is nearly unreachable
|
|
64
|
-
|
|
65
|
-
Argument resolution (including any blocking wait on an async result) happens *before*
|
|
66
|
-
acquisition, and a step whose body is dispatched elsewhere never acquires in the dispatching
|
|
67
|
-
process. The one construct that could hold coordination across a park is an interrupt step,
|
|
68
|
-
whose body is split across a pause. See D6.
|
|
69
|
-
|
|
70
|
-
---
|
|
71
|
-
|
|
72
|
-
## Decisions
|
|
73
|
-
|
|
74
|
-
### D1 — Steps host the existing `Lockable` macros unchanged
|
|
75
|
-
|
|
76
|
-
`RubyReactor::Step::ClassMethods` gains the same five macros by reusing
|
|
77
|
-
`Dsl::Lockable::ClassMethods`; `Dsl::StepBuilder` gains them for inline steps. The key proc
|
|
78
|
-
receives the step's resolved arguments instead of reactor inputs.
|
|
79
|
-
|
|
80
|
-
**Rationale**: Finding 1. One declaration surface, one set of option semantics, one place to
|
|
81
|
-
document. Authors already know the macros.
|
|
82
|
-
|
|
83
|
-
**Alternatives rejected**: a parallel `step_lock` vocabulary — two names for one concept.
|
|
84
|
-
|
|
85
|
-
### D2 — Enforcement is driven by the executor, not by a wrapper on the step
|
|
86
|
-
|
|
87
|
-
A single `StepCoordination` object wraps the step body. The executor drives it, because three
|
|
88
|
-
requirements are outside a step class's reach: parking the execution on contention (needs the
|
|
89
|
-
requeue path), skipping acquisition for a guard-suppressed step, and re-taking coordination
|
|
90
|
-
during rollback.
|
|
91
|
-
|
|
92
|
-
FR-023 (direct invocation) is served by the same object, entered with `park: false` — with no
|
|
93
|
-
execution to park, contention waits and then fails, exactly as the synchronous path does.
|
|
94
|
-
|
|
95
|
-
**Rationale**: this deliberately differs from `002`'s decision to enforce input contracts in a
|
|
96
|
-
prepended `run`. Validation is a pure function of the arguments; coordination is a property of
|
|
97
|
-
the execution — it parks it, releases it, and must survive into rollback. The two belong at
|
|
98
|
-
different layers, and saying so explicitly is cheaper than discovering it later.
|
|
99
|
-
|
|
100
|
-
### D3 — Fixed acquisition order, mirroring the reactor's
|
|
101
|
-
|
|
102
|
-
1. Ordered-lock gate (nothing else held while waiting for a turn — the existing hold-and-wait
|
|
103
|
-
guard in `OrderedLockSupport`)
|
|
104
|
-
2. Period gate, fast path
|
|
105
|
-
3. Rate limit
|
|
106
|
-
4. Exclusive lock
|
|
107
|
-
5. Semaphore
|
|
108
|
-
6. Period gate, re-check under the lock (closes the both-passed race, same as
|
|
109
|
-
`executor.rb:113`)
|
|
110
|
-
|
|
111
|
-
Released in reverse. Acquisition happens after guards and after argument validation — a step
|
|
112
|
-
that will fail validation must not first take a lock (FR-012, and it keeps the critical
|
|
113
|
-
section minimal).
|
|
114
|
-
|
|
115
|
-
### D4 — Contention parks via the step-retry requeue path
|
|
116
|
-
|
|
117
|
-
On `Lock::AcquisitionError`, `Semaphore::AcquisitionError`, `RateLimit::ExceededError`, or
|
|
118
|
-
`OrderedLock::WaitError`:
|
|
119
|
-
|
|
120
|
-
- **In a worker** (`context.inline_async_execution`): call the requeue path with a contention
|
|
121
|
-
delay and return `RetryQueuedResult`. The delay uses the primitive's own hint where it has
|
|
122
|
-
one (`retry_after_seconds` for rate limits) and a configured contention backoff otherwise.
|
|
123
|
-
- **Synchronously**: `contention_wait` has already blocked for the configured wait, so the
|
|
124
|
-
error propagates as an ordinary step failure and rollback proceeds.
|
|
125
|
-
|
|
126
|
-
Contention attempts are counted separately from failure retries (`retry_context` gains a
|
|
127
|
-
contention counter), bounded by a configurable ceiling; exceeding it converts the park into a
|
|
128
|
-
contention failure (FR-017). Counting contention against the step's `retries` budget would let
|
|
129
|
-
a busy key exhaust the retries meant for genuine failures.
|
|
130
|
-
|
|
131
|
-
**Alternatives rejected**: failing on both paths — the user's chosen behavior is park-and-
|
|
132
|
-
retry; the sync fallback exists only because there is no queue to park into.
|
|
133
|
-
|
|
134
|
-
### D5 — Re-entrancy reuses every existing primitive verbatim
|
|
135
|
-
|
|
136
|
-
- **Owner** is the root context id, so a step's hold nests inside its reactor's hold on the
|
|
137
|
-
same key and inside any ancestor's (FR-019).
|
|
138
|
-
- **Nesting count** is the adapter's existing re-entrancy count; the key frees at zero (FR-020).
|
|
139
|
-
- **Registry**: step-held keys are pushed to and popped from `root.private_data[:held_lock_keys]`
|
|
140
|
-
exactly as reactor-held keys are (FR-021).
|
|
141
|
-
- **Deadlock guard**: covered for `async_reactor` with no change (Finding 4); extended so
|
|
142
|
-
`async_step` dispatch also checks the dispatched step class's declared keys against the
|
|
143
|
-
registry (FR-022).
|
|
144
|
-
- **Direct invocation** has no context, so the owner is a per-call UUID and no re-entrancy
|
|
145
|
-
applies.
|
|
146
|
-
|
|
147
|
-
### D6 — Coordination on an interrupt step is refused at declaration
|
|
148
|
-
|
|
149
|
-
An interrupt step's body is split across a pause, so it is the one construct where a
|
|
150
|
-
step-level hold could span a park (Finding 6). Rather than extend `park_held_primitives!` to
|
|
151
|
-
carry step-level holds across gaps — new state, new reattach path, new failure mode — v1
|
|
152
|
-
raises at declaration time with a message pointing at reactor-level coordination for that case.
|
|
153
|
-
|
|
154
|
-
Everything FR-018 requires of parked executions continues to work: it describes coordination
|
|
155
|
-
the *execution* holds, which is reactor-level and unchanged.
|
|
156
|
-
|
|
157
|
-
### D7 — Step-level period skips the step
|
|
158
|
-
|
|
159
|
-
Reactor-level `with_period` halts the whole reactor when the bucket is already marked. At step
|
|
160
|
-
level that would kill workflows over one deduplicated step, so the step returns `Skipped` and
|
|
161
|
-
the workflow continues (FR-003). The bucket is marked when the step's work succeeds.
|
|
162
|
-
|
|
163
|
-
### D8 — Step-level ordered lock: nonce assigned on first arrival, sequenced last
|
|
164
|
-
|
|
165
|
-
The nonce is assigned when an execution first reaches the step, stashed per-step in
|
|
166
|
-
`private_data`, and reused across contention redeliveries. The gate advances when the step
|
|
167
|
-
reaches a terminal result. Strict chain failure short-circuits that step with `Skipped` rather
|
|
168
|
-
than halting the reactor (FR-004).
|
|
169
|
-
|
|
170
|
-
**Honest caveat**: per Finding 5 this delivers "ordered by arrival at the step", not "ordered
|
|
171
|
-
by enqueue". For a reactor whose first step is the ordered one, the two coincide; the further
|
|
172
|
-
into a workflow the step sits, the weaker the guarantee. This must be documented on the macro
|
|
173
|
-
itself, not just in a spec.
|
|
174
|
-
|
|
175
|
-
**Sequencing**: this primitive is roughly the same implementation weight as the other four
|
|
176
|
-
combined — per-step nonce state, per-step heartbeat, per-step advance-on-terminal, poison-pill
|
|
177
|
-
and strict-chain handling at step granularity. It is the last phase, and it is the piece to
|
|
178
|
-
cut first if the schedule tightens: phases 1-6 deliver the whole of US1-US4 and US6-US8 without
|
|
179
|
-
it. Recorded in Complexity Tracking.
|
|
180
|
-
|
|
181
|
-
### D9 — Observability extends the existing surfaces
|
|
182
|
-
|
|
183
|
-
Middleware events gain the step name; `Web::CoordinationSerializer` learns to read step-level
|
|
184
|
-
configs alongside reactor-level ones; a contention-parked execution is reported distinctly
|
|
185
|
-
from a failure, reusing the `:snooze_reactor` precedent (`executor.rb:166`) so a snooze round
|
|
186
|
-
does not appear as a phantom failure.
|
|
187
|
-
|
|
188
|
-
## Open risks
|
|
189
|
-
|
|
190
|
-
| Risk | Mitigation |
|
|
191
|
-
|---|---|
|
|
192
|
-
| Contention behaves differently sync vs. worker | Inherent to the chosen behavior; `contention_wait` makes it one branch, and it is called out in the macro's own documentation. |
|
|
193
|
-
| A busy key snoozes an execution indefinitely | Bounded contention counter (D4), separate from the failure-retry budget. |
|
|
194
|
-
| Step-level ordered lock's weaker guarantee is mistaken for the reactor-level one | Documented on the macro; demo shows arrival-order explicitly. |
|
|
195
|
-
| Lock TTL shorter than a slow step's work | Auto-extend applies to step holds exactly as to reactor holds. |
|
|
196
|
-
| Compensation stalls on a contended key | Compensation waits then reports (FR-026); it never parks, because rollback is already mid-failure. |
|