ruby_reactor 0.6.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. checksums.yaml +4 -4
  2. data/.release-please-manifest.json +1 -1
  3. data/.specify/feature.json +1 -1
  4. data/CHANGELOG.md +114 -1
  5. data/CLAUDE.md +5 -0
  6. data/README.md +131 -16
  7. data/lib/ruby_reactor/adapters/active_job/router.rb +21 -6
  8. data/lib/ruby_reactor/adapters/active_job/step_worker.rb +20 -0
  9. data/lib/ruby_reactor/adapters/active_job/worker.rb +6 -1
  10. data/lib/ruby_reactor/adapters/sidekiq/router.rb +21 -6
  11. data/lib/ruby_reactor/adapters/sidekiq/step_worker.rb +17 -0
  12. data/lib/ruby_reactor/adapters/sidekiq/worker.rb +5 -2
  13. data/lib/ruby_reactor/async_waiter.rb +113 -0
  14. data/lib/ruby_reactor/configuration.rb +30 -1
  15. data/lib/ruby_reactor/context.rb +21 -0
  16. data/lib/ruby_reactor/context_serializer.rb +2 -0
  17. data/lib/ruby_reactor/dsl/async_macros.rb +176 -0
  18. data/lib/ruby_reactor/dsl/async_reactor_builder.rb +66 -0
  19. data/lib/ruby_reactor/dsl/compose_builder.rb +23 -4
  20. data/lib/ruby_reactor/dsl/interrupt_builder.rb +3 -2
  21. data/lib/ruby_reactor/dsl/interrupt_step_config.rb +7 -1
  22. data/lib/ruby_reactor/dsl/lockable.rb +1 -1
  23. data/lib/ruby_reactor/dsl/map_builder.rb +1 -2
  24. data/lib/ruby_reactor/dsl/reactor.rb +29 -10
  25. data/lib/ruby_reactor/dsl/step_builder.rb +30 -11
  26. data/lib/ruby_reactor/error/async_result_pending.rb +21 -0
  27. data/lib/ruby_reactor/error/async_wait_timeout_error.rb +10 -0
  28. data/lib/ruby_reactor/error/deprecated_dsl_error.rb +11 -0
  29. data/lib/ruby_reactor/executor/async_step_dispatch.rb +110 -0
  30. data/lib/ruby_reactor/executor/ordered_lock_support.rb +2 -2
  31. data/lib/ruby_reactor/executor/result_handler.rb +12 -1
  32. data/lib/ruby_reactor/executor/retry_manager.rb +9 -5
  33. data/lib/ruby_reactor/executor/step_executor.rb +58 -11
  34. data/lib/ruby_reactor/executor.rb +162 -8
  35. data/lib/ruby_reactor/lock.rb +19 -0
  36. data/lib/ruby_reactor/map/element_executor.rb +6 -1
  37. data/lib/ruby_reactor/map/helpers.rb +1 -1
  38. data/lib/ruby_reactor/map/result_enumerator.rb +5 -1
  39. data/lib/ruby_reactor/map/result_summary.rb +63 -0
  40. data/lib/ruby_reactor/open_telemetry.rb +1 -1
  41. data/lib/ruby_reactor/reactor.rb +25 -2
  42. data/lib/ruby_reactor/rspec/sidekiq_helpers.rb +2 -1
  43. data/lib/ruby_reactor/rspec/step_executor_patch.rb +2 -2
  44. data/lib/ruby_reactor/rspec/test_subject.rb +66 -11
  45. data/lib/ruby_reactor/semaphore.rb +10 -0
  46. data/lib/ruby_reactor/step/async_reactor_step.rb +207 -0
  47. data/lib/ruby_reactor/step/compose_step.rb +1 -1
  48. data/lib/ruby_reactor/step/map_step.rb +1 -1
  49. data/lib/ruby_reactor/step_sweeper.rb +72 -0
  50. data/lib/ruby_reactor/step_worker.rb +260 -0
  51. data/lib/ruby_reactor/storage/adapter.rb +22 -1
  52. data/lib/ruby_reactor/storage/redis_adapter.rb +17 -13
  53. data/lib/ruby_reactor/storage/redis_locking.rb +7 -0
  54. data/lib/ruby_reactor/storage/redis_pub_sub.rb +31 -0
  55. data/lib/ruby_reactor/storage/redis_step_results.rb +49 -0
  56. data/lib/ruby_reactor/sweeper.rb +7 -1
  57. data/lib/ruby_reactor/sweeper_job.rb +1 -0
  58. data/lib/ruby_reactor/template/result.rb +151 -5
  59. data/lib/ruby_reactor/version.rb +1 -1
  60. data/lib/ruby_reactor/web/api.rb +94 -16
  61. data/lib/ruby_reactor/web/public/assets/index-B46p-M6K.css +1 -0
  62. data/lib/ruby_reactor/web/public/assets/index-DPmP4yXT.js +22 -0
  63. data/lib/ruby_reactor/web/public/index.html +2 -2
  64. data/lib/ruby_reactor/worker.rb +53 -3
  65. data/lib/ruby_reactor.rb +24 -3
  66. data/specs/001-background-async-steps/checklists/requirements.md +39 -0
  67. data/specs/001-background-async-steps/contracts/public-dsl.md +154 -0
  68. data/specs/001-background-async-steps/data-model.md +117 -0
  69. data/specs/001-background-async-steps/plan.md +168 -0
  70. data/specs/001-background-async-steps/quickstart.md +102 -0
  71. data/specs/001-background-async-steps/research.md +150 -0
  72. data/specs/001-background-async-steps/spec.md +146 -0
  73. data/specs/001-background-async-steps/tasks.md +271 -0
  74. data/specs/active_job.md +1 -1
  75. metadata +27 -3
  76. data/lib/ruby_reactor/web/public/assets/index-CCnNVQy5.css +0 -1
  77. data/lib/ruby_reactor/web/public/assets/index-D7IBZvos.js +0 -21
data/lib/ruby_reactor/web/public/index.html CHANGED
@@ -5,8 +5,8 @@
5
5
  <link rel="icon" type="image/svg+xml" href="./vite.svg" />
6
6
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
7
7
  <title>ui</title>
8
- <script type="module" crossorigin src="./assets/index-D7IBZvos.js"></script>
9
- <link rel="stylesheet" crossorigin href="./assets/index-CCnNVQy5.css">
8
+ <script type="module" crossorigin src="./assets/index-DPmP4yXT.js"></script>
9
+ <link rel="stylesheet" crossorigin href="./assets/index-B46p-M6K.css">
10
10
  </head>
11
11
  <body>
12
12
  <div id="root"></div>
data/lib/ruby_reactor/worker.rb CHANGED
@@ -8,6 +8,36 @@ module RubyReactor
8
8
  # `Adapters::ActiveJob::Compat` on ActiveJob::Base) — nothing here references
9
9
  # a specific backend.
10
10
  module Worker
11
+ TERMINAL_STATUSES = %w[completed failed cancelled skipped].freeze
12
+
13
+ # Last line of observability when a job burns its whole retry budget on an
14
+ # infrastructure failure and the backend then discards it (Sidekiq runs
15
+ # with `dead: false`): without this, the context would stay "running"
16
+ # forever with zero surface anywhere, and every reader would wait out its
17
+ # full timeout. Called from the backends' retries-exhausted hooks with the
18
+ # job's own args. Best-effort — never raises back into the backend.
19
+ def self.record_retries_exhausted(args, exception)
20
+ context_id, reactor_class_name = args
21
+ return unless context_id
22
+
23
+ reactor_class_name ||= RubyReactor.reactor_storage_name(nil)
24
+ storage = RubyReactor.configuration.storage_adapter
25
+ data = storage.retrieve_context(context_id, reactor_class_name)
26
+ return if data.nil? || TERMINAL_STATUSES.include?((data["status"] || data[:status]).to_s)
27
+
28
+ data["status"] = "failed"
29
+ data["failure_reason"] = {
30
+ "message" => "job retries exhausted: #{exception.class.name}: #{exception.message}",
31
+ "exception_class" => exception.class.name
32
+ }
33
+ storage.store_context(context_id, JSON.generate(data), reactor_class_name)
34
+ storage.publish(RubyReactor.async_reactor_channel(context_id), "failed")
35
+ rescue StandardError => e
36
+ RubyReactor.configuration.logger.error(
37
+ "RubyReactor: could not record retries-exhausted failure for #{context_id}: #{e.class.name}: #{e.message}"
38
+ )
39
+ end
40
+
11
41
  # Identity-only payload: storage is the source of truth. Rehydrate the live
12
42
  # context from storage by id, then resume. A nil read means the context was
13
43
  # swept, expired, or already terminal-and-collected — nothing to resume.
@@ -62,7 +92,8 @@ module RubyReactor
62
92
  rescue RubyReactor::Lock::AcquisitionError,
63
93
  RubyReactor::Semaphore::AcquisitionError,
64
94
  RubyReactor::RateLimit::ExceededError,
65
- RubyReactor::OrderedLock::WaitError => e
95
+ RubyReactor::OrderedLock::WaitError,
96
+ RubyReactor::Error::AsyncResultPending => e
66
97
  # Snooze on expected concurrency, rate, or ordering contention.
67
98
  # OrderedLock::WaitError carries a poison-pill-derived retry hint,
68
99
  # consumed by compute_snooze_delay below. We avoid the framework's native
@@ -89,7 +120,16 @@ module RubyReactor
89
120
  rescue NameError
90
121
  # If not found, try to find it in the current namespace
91
122
  # This is a fallback for test environments
92
- context.reactor_class = reactor_class_name.constantize if reactor_class_name.respond_to?(:constantize)
123
+ begin
124
+ context.reactor_class = reactor_class_name.constantize if reactor_class_name.respond_to?(:constantize)
125
+ rescue NameError
126
+ # Leave reactor_class nil: the caller's guard fails the context with
127
+ # a durable record. Letting this second NameError escape would burn
128
+ # the job's whole retry budget on an error retries can never fix,
129
+ # then vanish — leaving the context "running" forever and any reader
130
+ # waiting out its full timeout.
131
+ nil
132
+ end
93
133
  end
94
134
  end
95
135
 
@@ -107,8 +147,12 @@ module RubyReactor
107
147
  # duplicate of the *same* execution may wait arbitrarily long for the
108
148
  # live original to finish (e.g. a sweeper re-enqueue racing a slow but
109
149
  # alive worker). Capping it would fail a legitimately-waiting duplicate.
150
+ # A parked async wait is likewise uncapped HERE: its bound is
151
+ # `async_park_timeout`, enforced against `dispatched_at` at the wait
152
+ # site — counting snoozes would double-bound it with the wrong unit.
110
153
  capped = !(error.is_a?(RubyReactor::OrderedLock::WaitError) ||
111
- error.is_a?(RubyReactor::Lock::ContextLockContention))
154
+ error.is_a?(RubyReactor::Lock::ContextLockContention) ||
155
+ error.is_a?(RubyReactor::Error::AsyncResultPending))
112
156
 
113
157
  if capped && max != :infinity && snooze_count >= max
114
158
  escalate_snooze(context, snooze_count, error)
@@ -203,6 +247,12 @@ module RubyReactor
203
247
  payload,
204
248
  reactor_class_name
205
249
  )
250
+ # Written first, signalled second — wake any reader blocked (or parked)
251
+ # on this execution so it fails fast with the real cause instead of
252
+ # waiting out its timeout.
253
+ RubyReactor.configuration.storage_adapter.publish(
254
+ RubyReactor.async_reactor_channel(context_id), "failed"
255
+ )
206
256
  rescue StandardError => e
207
257
  # Don't let a persistence failure mask the original deserialization error.
208
258
  RubyReactor.configuration.logger.error(
data/lib/ruby_reactor.rb CHANGED
@@ -3,6 +3,7 @@
3
3
  require "zeitwerk"
4
4
  require "pathname"
5
5
  require "securerandom"
6
+ require "time"
6
7
  require_relative "ruby_reactor/registry"
7
8
  require_relative "ruby_reactor/utils/code_extractor"
8
9
  require_relative "ruby_reactor/dsl/lockable" # Add this
@@ -292,8 +293,10 @@ module RubyReactor
292
293
  end
293
294
  end
294
295
 
295
- # Async result for background job execution
296
- class AsyncResult
296
+ # Sentinel returned when a step's work is handed off to a worker job and is
297
+ # not yet resolved — produced by `background`, `async_step`, `async_reactor`,
298
+ # and map's async element dispatch alike.
299
+ class DispatchResult
297
300
  attr_reader :job_id, :intermediate_results, :execution_id
298
301
 
299
302
  def initialize(job_id:, intermediate_results: {}, execution_id: nil)
@@ -348,6 +351,23 @@ module RubyReactor
348
351
  # (A truly anonymous class can't be reconstituted by name in another process,
349
352
  # so cross-process resume of one is inherently unsupported; this only keeps
350
353
  # the keys self-consistent within a process — e.g. inline tests.)
354
+ # Completion-signal channels for the notified wait. One place, because
355
+ # the publishing side and the waiting side must agree exactly and they live in
356
+ # different files (StepWorker / Executor vs Template::Result).
357
+ def self.async_step_channel(context_id, step_name)
358
+ "rr:done:#{context_id}:#{step_name}"
359
+ end
360
+
361
+ def self.async_reactor_channel(execution_id)
362
+ "rr:done:#{execution_id}"
363
+ end
364
+
365
+ # Liveness lock for one dispatched `async_step`. Held by StepWorker for the
366
+ # life of the unit, so StepSweeper can tell a slow unit from a lost job.
367
+ def self.async_step_lock_key(context_id, step_name)
368
+ "async_step:#{context_id}:#{step_name}"
369
+ end
370
+
351
371
  def self.reactor_storage_name(reactor_class)
352
372
  return "AnonymousReactor" if reactor_class.nil?
353
373
 
@@ -407,7 +427,8 @@ module RubyReactor
407
427
  limit ||= configuration.sweeper_limit
408
428
  {
409
429
  reactors: Sweeper.run_once(limit: limit),
410
- maps: Map::Sweeper.run_once(limit: limit)
430
+ maps: Map::Sweeper.run_once(limit: limit),
431
+ async_steps: StepSweeper.run_once(limit: limit)
411
432
  }
412
433
  end
413
434
 
data/specs/001-background-async-steps/checklists/requirements.md ADDED
@@ -0,0 +1,39 @@
1
+ # Specification Quality Checklist: Background Execution & Real Async Steps
2
+
3
+ **Purpose**: Validate specification completeness and quality before proceeding to planning
4
+ **Created**: 2026-08-16
5
+ **Feature**: [spec.md](../spec.md)
6
+
7
+ ## Content Quality
8
+
9
+ - [x] No implementation details (languages, frameworks, APIs)
10
+ - [x] Focused on user value and business needs
11
+ - [x] Written for non-technical stakeholders
12
+ - [x] All mandatory sections completed
13
+
14
+ ## Requirement Completeness
15
+
16
+ - [x] No [NEEDS CLARIFICATION] markers remain
17
+ - [x] Requirements are testable and unambiguous
18
+ - [x] Success criteria are measurable
19
+ - [x] Success criteria are technology-agnostic (no implementation details)
20
+ - [x] All acceptance scenarios are defined
21
+ - [x] Edge cases are identified
22
+ - [x] Scope is clearly bounded
23
+ - [x] Dependencies and assumptions identified
24
+
25
+ ## Feature Readiness
26
+
27
+ - [x] All functional requirements have clear acceptance criteria
28
+ - [x] User scenarios cover primary flows
29
+ - [x] Feature meets measurable outcomes defined in Success Criteria
30
+ - [x] No implementation details leak into specification
31
+
32
+ ## Notes
33
+
34
+ - Items marked incomplete require spec updates before `/speckit-clarify` or `/speckit-plan`.
35
+ - Note: the spec names concrete terms (`Sidekiq`, `ActiveJob`, `Redis`) only inside the **Assumptions** section, where they describe existing constraints already codified in the project constitution, not new implementation choices being introduced by this feature. All requirement- and success-criteria-level language remains technology-agnostic.
36
+ - 2026-08-16 (post-plan review): added FR-014/SC-006 covering dashboard visibility for `async_step`/`async_reactor`, a gap found while reviewing plan.md against the constitution's Observability principle. Re-validated against this checklist — all items still pass.
37
+ - 2026-08-21 (final consistency pass): fixed stale "blocking-poll" wording left from the superseded wait design (plan Summary/tree, contract, data-model); fixed the async_step dispatch ordering in data-model (record + ref written BEFORE enqueue, matching the F2 rule its own Step Result Record row states); fixed "record absent = pending" to "record still `dispatched` = pending"; reconciled plan Scale/Scope with the tree (dependency_graph.rb unchanged; executor.rb added for the completion-signal publish); aligned FR-015 with contract/research on single-slot semaphores. All items re-verified passing.
38
+ - 2026-08-20 (wait-mechanism & locks review): FR-005 revised from tight-poll to notified wait (record-first publish, subscribe-first check, fallback re-check — durable record stays the source of truth, timeout bound unchanged); added FR-015 (no lock-owner sharing across the async boundary + dispatch-time same-key deadlock guard; reentrancy stays compose-only) and FR-016 (async_reactor dispatch reuses the full pre-enqueue sequence: child input validation + ordered-lock nonce, a gap found during the locks review). Clarifications Session 2026-08-20 records both decisions. All checklist items re-verified passing.
39
+ - 2026-08-16 (full-artifact review): fixed FR-006 (claimed a nonexistent "existing step-result storage"; now: new per-step record for `async_step`, own execution record for `async_reactor`); clarified FR-003 scope (compose's `async` flag is removed too — it sets the same flag; map's `async` is a different mechanism and stays); added edge cases for `returns` × async unit, `background` × whole-reactor `async true`, and a paused awaited `async_reactor` child. Contract additionally pins failure-read semantics for `result()` on async units and lock-window behavior. All checklist items re-verified passing.
data/specs/001-background-async-steps/contracts/public-dsl.md ADDED
@@ -0,0 +1,154 @@
1
+ # Contract: Public DSL Surface
2
+
3
+ This gem's "interface" is its Ruby DSL, consumed by host applications that subclass `RubyReactor::Reactor`. This contract documents the exact public surface this feature adds, changes, or removes, so implementation and tests can be checked against it directly.
4
+
5
+ ## Removed
6
+
7
+ ```ruby
8
+ class MyReactor < RubyReactor::Reactor
9
+ step :do_thing do
10
+ async true # REMOVED — raises a definition-time error (FR-003)
11
+ end
12
+ end
13
+ ```
14
+
15
+ ```ruby
16
+ class MyReactor < RubyReactor::Reactor
17
+ compose :sub_flow, SubReactor do
18
+ async true # ALSO REMOVED — same StepConfig hand-off flag, same definition-time error
19
+ end
20
+ end
21
+ ```
22
+
23
+ **Error contract**: attempting to call `async` inside a `step` **or `compose`** block MUST raise a `RubyReactor::Error::ValidationError` (or a new dedicated error subclass — implementation's choice, but it MUST be raised at reactor **class-definition** time, not at `run` time) whose message names both the removed syntax and its replacement(s): `background after:`/`background before:`, `async_step`, `async_reactor`. (`ComposeBuilder#async` sets the very `StepConfig` flag this feature removes — `dsl/compose_builder.rb:62` — so it cannot survive; the exact migration for a compose that handed off is `background before: :<that compose step>`, which reproduces the old semantics precisely — the flagged step and everything after it moved to the worker — without the author having to identify a predecessor.)
24
+
25
+ **Not removed**: the `async` option inside a `map` block (`MapBuilder#async`, `dsl/map_builder.rb:43`) is a map-internal element-dispatch mode passed as a step *argument* — the map's own `StepConfig` is hardcoded `async: false` (`map_builder.rb:111`), so it does not touch the removed flag and keeps working unchanged.
26
+
27
+ ## Added: `background after:` / `background before:`
28
+
29
+ ```ruby
30
+ class MyReactor < RubyReactor::Reactor
31
+ step :first
32
+ step :second
33
+ background after: :second # :second is the LAST step in the calling process
34
+ step :third
35
+ end
36
+
37
+ class EquivalentInLinearFlow < RubyReactor::Reactor
38
+ step :first
39
+ step :second
40
+ background before: :third # :third is the FIRST step in the worker
41
+ step :third
42
+ end
43
+ ```
44
+
45
+ - **Signature**: `self.background(after: nil, before: nil)` — reactor class macro, exactly one keyword supplied.
46
+ - **The two forms** name one cut point from opposite sides, and each carries a *guarantee about the step it names*:
47
+ - `after: :x` — `:x` runs in the calling process, and is the last step to do so.
48
+ - `before: :x` — `:x` runs in the worker, and is the first step to do so; it never executes in the calling process.
49
+
50
+ In a linear reactor where `:third` immediately follows `:second`, `after: :second` and `before: :third` are equivalent. In a branching workflow they are not, and the author picks whichever step they actually need pinned.
51
+ - **Constraints** (all raise at class-definition time, FR-002):
52
+ - at most one `background` declaration per reactor class;
53
+ - the named step must be defined in the class (either keyword);
54
+ - the named step must not be an `interrupt` (either keyword) — an interrupt re-enters the reactor from a foreground process, so `after:` would silently never fire on resume and `before:` would enqueue a worker that instantly pauses and swallows the `InterruptResult`; the error points to `interrupt :name, resume: :background` as the correct spelling;
55
+ - exactly one of `after:`/`before:` — supplying both, or neither, raises;
56
+ - combining `background` with whole-reactor `async true` raises (the hand-off point would be silently meaningless inside a reactor that already runs entirely in a worker — see spec Edge Cases).
57
+ - **Runtime contract**: hand-off is triggered by *reaching the named step*, not by the declaration's lexical position — the declaration may sit anywhere in the class body. For `after: :x`, the trigger fires when `:x` completes; for `before: :x`, it fires when `:x` is selected to run, and `:x` is left unexecuted for the worker to run. Either way: checkpoint, enqueue the remainder via `configuration.async_router`, return an `DispatchResult` to the caller. In a DAG with parallel branches, any independent step that became ready and executed before the trigger fired has already run in the calling process; everything not yet executed at the trigger moment runs in the worker — this caveat is identical for both forms. Compensation for worker-side step failures works exactly as it does for any same-process step failure — `background` only changes *where* code runs, not the saga/compensation contract (US1 acceptance scenarios 1-2). Inside the worker the hand-off never re-triggers (the existing `inline_async_execution` guard).
58
+ - **Never-reached trigger**: if the named step is skipped by a `where`/guard condition, or the reactor fails before reaching it, the hand-off simply never fires and the run completes in the calling process. No step is stranded — the hand-off only ever relocates *remaining* work.
59
+ - **`before:` naming the first step** is legal, and is not the same as whole-reactor `async true`: every step body runs in the worker, but input validation still happens in the calling process, so invalid inputs fail the caller synchronously instead of failing inside a worker.
60
+
61
+ ## Added: `async_step`
62
+
63
+ ```ruby
64
+ class MyReactor < RubyReactor::Reactor
65
+ async_step :send_email do
66
+ argument :to, input(:email)
67
+ run { |args| Mailer.send(args[:to]) }
68
+ end
69
+
70
+ step :do_something_same_thread do
71
+ run { do_work }
72
+ end
73
+
74
+ step :check_email do
75
+ argument :email, result(:send_email) # blocks (notified wait, bounded by timeout) until :send_email is done
76
+ run { |args| ... }
77
+ end
78
+ end
79
+ ```
80
+
81
+ - **Signature**: `self.async_step(name, impl = nil, &block)` — same call shape as `step`, builds the same `StepConfig` fields (`argument`, `run`, `compensate`, `undo`, `validate_args`, `validate_output`, `retries`, etc. all still work identically inside the block).
82
+ - **Runtime contract**:
83
+ - Dispatches the step's work to an independent worker job; does **not** halt the calling reactor's execution of other ready steps that don't depend on it (US2 acceptance scenario 1).
84
+ - Any step that references `result(:async_step_name)` blocks — notified wait: woken by the completion signal the finishing worker publishes after its durable write, with a coarse fallback re-check of the durable record, bounded overall by `Configuration#async_wait_timeout` (spec Clarifications, Session 2026-08-20) — until the async step's terminal result is available. **Read semantics**: on `Success`, the reader receives the same deserialized raw value a same-process step's result would produce (US2 acceptance scenario 2, FR-006); on `Failure`, the reader receives the `Failure` object itself as the argument value — a same-process step's failure would have halted the reactor before any reader ran, so there is no sync-behavior to mirror here, and injecting the `Failure` is what lets the reader "see the failure and decide" per the spec's clarified compensation model (US2 acceptance scenario 3). This matches `async_reactor`'s wrapped-result-on-inspection pattern.
85
+ - `returns :async_step_name` raises at class-definition time — the reactor's return value must come from a same-process step (spec Edge Cases).
86
+ - Dispatch is **not** suppressed inside a worker: an `async_step` declared after a `background` hand-off point (or reached during a worker resume) still dispatches to its own independent job — the existing `inline_async_execution` guard suppresses only the *hand-off* re-trigger, never `async_step`/`async_reactor` dispatch (spec Edge Cases).
87
+ - Reactor-level `lock`/`semaphore`/`rate_limit` windows are held by the process executing the reactor's own steps — the async step's work runs *outside* those windows (in its own job, which acquires nothing). A step body that needs mutual exclusion must arrange it itself.
88
+ - If the async step fails and no later step reads its result, the parent reactor's compensation is **not** automatically triggered (US2 acceptance scenario 3, FR-011). A later step that does read the result and observes failure may itself return `Failure` to trigger compensation.
89
+ - `compensate`/`undo` blocks declared on an `async_step` still register normally — they only run if the step's own failure is surfaced into the parent's compensation path via the opt-in mechanism above, never automatically.
90
+ - A reference to the dispatched unit is recorded on the parent's own context (`composed_contexts[:send_email] = { type: :async_step_ref, ... }`) at dispatch time, and the web dashboard renders `send_email` as an `async_step`-typed node (FR-008, FR-014).
91
+
92
+ ## Added: `async_reactor`
93
+
94
+ ```ruby
95
+ class MyReactor < RubyReactor::Reactor
96
+ async_reactor :create_profile, CreateProfileReactor # fire-and-forget, no downstream reference
97
+
98
+ async_reactor :create_account, CreateAccountReactor do
99
+ argument :user_id, input(:user_id)
100
+ end
101
+
102
+ step :verify_all do
103
+ argument :account, result(:create_account) # blocks until create_account finishes
104
+ run do |args, context|
105
+ if args[:account].success?
106
+ Success(args[:account].value)
107
+ else
108
+ Failure(args[:account].error) # opt-in compensation trigger
109
+ end
110
+ end
111
+ end
112
+ end
113
+ ```
114
+
115
+ - **Signature**: `self.async_reactor(name, child_reactor_class, &block)` — `argument` inside the block maps parent-visible sources to the child reactor's inputs, same shape as `compose`.
116
+ - **Dispatch contract** (the part that runs in the parent's process, FR-015/FR-016):
117
+ - Dispatch applies the same pre-enqueue safeguards as a top-level async run: the child's inputs are validated and, if the child declares `with_ordered_lock`, its ordering nonce is assigned at enqueue. A child-input validation failure fails **the dispatching step** (normal saga handling in the parent) — this is a dispatch failure, not a child-execution failure, and is deliberately outside FR-009's no-auto-compensation rule.
118
+ - Deadlock guard: if the child declares an exclusive `lock` (or a `semaphore` with `limit: 1`) whose resolved key equals one the dispatching execution currently holds, the dispatch step fails immediately with an error naming the lock key and both reactor classes. The error message MUST enumerate the three remediations, ranked: (1) use `compose` if the child belongs in the parent's critical section and its result is needed — the wait means the work is sequential anyway; (2) narrow the lock keys if parent and child actually protect different resources; (3) restructure so the locked reactor never reads the child's result — fire-and-forget with verification in the child itself or in a successor reactor outside the lock window. Lock ownership is never shared across the async boundary (parent and child run concurrently — shared ownership would break mutual exclusion); owner-based reentrancy remains for `compose` only. Transitive cross-execution cycles are out of the guard's reach (undetectable at dispatch) — documentation advises consistent key-acquisition order, with the FR-005 timeout as backstop.
119
+ - **Runtime contract**:
120
+ - Dispatches `child_reactor_class` to run independently via `configuration.async_router`, linked to the parent by the child's `execution_id` for traceability/logging (FR-008, US3 acceptance scenario 4) — never added to the parent's compensation graph (FR-009).
121
+ - If nothing in the parent reads `result(:name)`, the child's eventual failure never affects the parent (US3 acceptance scenario 1).
122
+ - If a later step reads `result(:name)`, it blocks (same notified-wait policy as `async_step` — the child publishes its completion signal after its terminal save) until the child reactor reaches a terminal state, then receives the child's actual `Success`/`Failure` result object (not the enqueue-time `DispatchResult`), and may inspect `.success?`/`.value`/`.error` to decide whether to itself return `Failure` (US3 acceptance scenarios 2-3, FR-010).
123
+ - A reference is recorded on the parent's own context (`composed_contexts[:create_profile] = { type: :async_reactor_ref, execution_id:, reactor_class_name:, ... }`) at dispatch time — the web dashboard renders `create_profile`/`create_account` as `async_reactor`-typed nodes and lets an operator open the linked child execution, the same drill-down `compose`/`map` already offer (FR-008, FR-014, US3 acceptance scenario 4).
124
+ - `returns :async_reactor_name` raises at class-definition time, same as for `async_step`.
125
+ - A child that *pauses* at an interrupt step is not terminal: a reader keeps polling and hits the FR-005 timeout unless the child is resumed within the bound (spec Edge Cases). The child is an ordinary independently-recoverable execution — the existing sweeper/durability machinery covers its crash recovery with no new mechanism.
126
+
127
+ ## Added: `interrupt ..., resume: :background`
128
+
129
+ ```ruby
130
+ class MyReactor < RubyReactor::Reactor
131
+ interrupt :webhook, resume: :background do
132
+ validate_payload { required(:status).filled(:string) }
133
+ end
134
+
135
+ step :heavy_work do
136
+ argument :status, result(:webhook)
137
+ run { |args| ... } # runs in a worker, never in the process that delivered the payload
138
+ end
139
+ end
140
+ ```
141
+
142
+ - **Signature**: `self.interrupt(name, resume: :inline, &block)` — `:inline` (today's behavior, the default) or `:background`. Any other value raises `RubyReactor::Error::ValidationError` at class-definition time.
143
+ - **Why it exists**: `background after:/before:` is edge-triggered on reaching one named step, but an interrupt re-enters the reactor from a foreground process (`continue` runs in whatever process delivered the payload). `resume: :background` makes that re-entry itself the hand-off: the delivering process validates and stores the payload, then enqueues the remainder.
144
+ - **Runtime contract**:
145
+ - Payload validation (`validate_payload`), attempt counting, and `max_attempts` exhaustion (undo + failed) all run **synchronously in the calling process**, before anything is enqueued — validate in the caller, execute in the worker, the same split as `background before: <first step>` and FR-016. An invalid payload enqueues nothing and leaves the reactor paused.
146
+ - On a valid payload: result stored, context persisted (job payload is identity-only), `before_async_enqueue` middleware fires, remainder enqueued via `configuration.async_router`, and `continue` returns an `DispatchResult` (with `execution_id`) instead of the final result.
147
+ - The worker resumes through the existing path: rehydrate by id, `resume_execution`, interrupt's stored result found, remaining steps run under `inline_async_execution` (so a `background` hand-off point never re-triggers there).
148
+
149
+ ## Unchanged (explicitly out of scope, called out to prevent accidental regression)
150
+
151
+ - Reactor-level `async true` ("Full Reactor Async") — `self.class.async?`, `lib/ruby_reactor/dsl/reactor.rb:44-50`. (Its only new interaction: combining it with `background after:` is a definition-time error, see above.)
152
+ - `compose` itself — synchronous, fully compensation-linked nested execution, untouched. (Its `async` flag is removed — see the Removed section — but everything else about `compose` is unchanged.)
153
+ - `map`'s dispatch/collection machinery and its full DSL including the map-internal `async` element-dispatch option — reused as an architectural pattern (see research.md) but untouched.
154
+ - `result(:name)` for a **synchronous** step's result — resolves exactly as it does today (`Template::Result#resolve`), with zero added latency; the new notified-wait path only activates for `async_step`/`async_reactor` references.
data/specs/001-background-async-steps/data-model.md ADDED
@@ -0,0 +1,117 @@
1
+ # Data Model: Background Execution & Real Async Steps
2
+
3
+ This is a library feature — "entities" are DSL/runtime constructs and the storage records backing them, not application data.
4
+
5
+ ## Background Hand-off Point
6
+
7
+ Reactor-class-level declaration, one per reactor.
8
+
9
+ | Field | Type | Notes |
10
+ |---|---|---|
11
+ | `mode` | Symbol — `:after` or `:before` | Which side of the cut point the declaration named. Derived from which keyword the author supplied. |
12
+ | `step` | Symbol | The named step. For `:after`, the last step to run in the calling process; for `:before`, the first step to run in the worker. Must reference a step defined in the same reactor (validated at class-definition time, FR-002). |
13
+
14
+ Exposed to the runtime, `TestSubject`, and the dashboard as a single normalized reader — `background_handoff → { mode:, step: }` — never as a one-sided `background_after`. One concept with two trigger positions, not two parallel features: every consumer branches on `mode`, so no consumer can be accidentally implemented for `after:` only.
15
+
16
+ **Storage**: not persisted as its own record — it compiles into which step reaching which position triggers the `StepExecutor#handle_async_step`-style enqueue. Enforced-single via a class-level guard.
17
+
18
+ **Validation rules** (all definition-time errors):
19
+ - Exactly zero or one `background` declaration per reactor class (FR-002).
20
+ - Exactly one of `after:`/`before:` per declaration — both or neither raises (FR-002).
21
+ - The named step must exist in `steps` at the time `background` is evaluated, or at class-definition-close time if steps can be declared afterward (implementation detail for tasks phase).
22
+ - `background` combined with whole-reactor `async true` is rejected — the hand-off point would be silently meaningless inside a reactor that already runs entirely in a worker (spec Edge Cases).
23
+ - `returns` naming an `async_step` or `async_reactor` is rejected — the return value must come from a same-process step (spec Edge Cases).
24
+
25
+ ## Async Step
26
+
27
+ A step declared with `async_step :name` (or `async_step :name, ImplClass`) instead of `step`.
28
+
29
+ | Field | Type | Notes |
30
+ |---|---|---|
31
+ | `name` | Symbol | Step name, same namespace as regular steps. |
32
+ | `arguments` / `run_block` / `impl` | (existing `StepConfig` fields) | Same shape as a regular step — `async_step` is a `StepConfig` with a dispatch-mode marker, not a new config type. |
33
+ | dispatch-mode marker | Boolean/Symbol | Distinguishes "run inline" vs "dispatch as an independent unit" at `StepExecutor#execute_step` time. |
34
+
35
+ **Lifecycle** (state machine, tracked via the new Step Result Record below, keyed by `(context_id, step_name)`):
36
+
37
+ ```
38
+ dispatched -> running -> completed(Success)
39
+ -> completed(Failure)
40
+ ```
41
+
42
+ - `dispatched`: parent process has, synchronously and in this order (durable-write-before-enqueue, F2): (1) written the Step Result Record with status `dispatched` and the `composed_contexts[step_name] = { type: :async_step_ref, name:, dispatched_at: }` reference onto its own context (see Async Step ↔ context linkage below), (2) enqueued the `StepWorker` job, (3) marked the step graph-complete for scheduling purposes (siblings may now proceed). No result exists yet.
43
+ - `running`/`completed`: opaque to the parent process except through the Step Result Record; the parent only observes "record carries a terminal value" or "record still `dispatched`" (still-pending — keep waiting, subject to FR-005's timeout). A record *absent* entirely means the step was never dispatched — `result()` does not wait in that case. The parent may reach its own terminal state while a record is still `dispatched`; that is the fire-and-forget contract (FR-018), and the parent's status makes no claim about the unit's outcome.
44
+
45
+ **Relationships**: An `async_step` is a dependency-graph node like any other step — other steps that declare `argument :x, result(:async_step_name)` get an automatic DAG edge (existing `DependencyGraph#add_step` behavior, unchanged) and, per FR-005, enter the notified wait for the terminal record when they resolve that argument.
46
+
47
+ **Async Step ↔ context linkage (FR-008, FR-014)**: the *reference* (not the result) lives in `context.composed_contexts[step_name]`, the same field `compose`/`map` already populate for their own children — see research.md decision 8. The dashboard's existing `hydrate_composed_contexts` pipeline (`lib/ruby_reactor/web/api.rb`) is extended with a branch for `type: :async_step_ref` that resolves the Step Result Record to show current status/result, mirroring how it already resolves `:map_ref`.
48
+
49
+ ## Async Reactor
50
+
51
+ A step declared with `async_reactor :name, ChildReactorClass`.
52
+
53
+ | Field | Type | Notes |
54
+ |---|---|---|
55
+ | `name` | Symbol | Step name in the parent. |
56
+ | `child_reactor_class` | Class | Must be a `RubyReactor::Reactor` subclass. |
57
+ | `argument_mappings` | Hash | Same shape as `compose`'s `argument_mappings` — maps parent-visible sources to the child's inputs. |
58
+
59
+ **Dispatch-time behavior** (FR-015, FR-016 — 2026-08-20 session):
60
+ - Dispatch reuses the full pre-enqueue sequence of a top-level async run (child input validation → ordered-lock nonce assignment where the child declares one → persist child context → enqueue), never raw `perform_async` — see research.md decision 3. A validation failure fails the dispatching step itself (normal parent saga handling), distinct from child-execution failure (FR-009).
61
+ - Deadlock guard: the child's `lock_config[:key_proc]` (and `semaphore_config` with `limit: 1`) is resolved against the mapped child inputs; a key matching one the dispatching execution currently holds fails the dispatch step immediately with an error naming the key and both reactor classes (research.md decision 9). No lock-owner sharing across the async boundary — reentrancy stays `compose`-only.
62
+
63
+ **Relationships**:
64
+ - Linked to the parent via `context.composed_contexts[step_name] = { type: :async_reactor_ref, name:, execution_id:, reactor_class_name:, dispatched_at: }` — written synchronously by the dispatching step, same field and pattern `compose`/`map` already use (research.md decision 8). Unlike `async_step`, no separate Step Result Record is needed for the *outcome*: the child is a normal, independently addressable `Reactor` with its own context row, so its terminal result is reached via the existing `storage.retrieve_context(execution_id, reactor_class_name)` / `ChildReactorClass.find(execution_id)` — the same lookup any other reactor execution uses.
65
+ - **Not** added to the parent's compensation graph — no `compensate`/`undo` block is registered for this step (see spec Clarifications: compensation is opt-in via a later step reading the result, never automatic).
66
+ - FR-008/FR-014: this `composed_contexts` entry is what the web dashboard's `hydrate_composed_contexts` reads to render the reference and drill into the child's own step structure (`build_structure` recursion, same as `compose`/`map`'s `nested_structure`).
67
+
68
+ ## Step Result Record (new storage-level entity)
69
+
70
+ The durable record backing `async_step` completion. (`async_reactor` needs no equivalent record — per the relationship above, its outcome is simply its own context row, reached by execution id through the existing `retrieve_context`/`find` path.) This avoids what would otherwise be a race-prone write into the parent's context blob from a worker running concurrently with the still-executing parent process.
71
+
72
+ | Field | Type | Notes |
73
+ |---|---|---|
74
+ | `context_id` | String (UUID) | The **parent** reactor's context id — the bucket is scoped per parent execution. |
75
+ | `step_name` | Symbol/String | The `async_step`'s name within that parent. |
76
+ | `status` | Enum: `dispatched`, `completed` | `dispatched` written synchronously **before** the job is enqueued (same checkpoint-before-enqueue ordering the existing hand-off uses, F2) — so a crash after enqueue can never find a job with no record, and this record doubles as the **re-attach marker** (FR-017): on recovery/resume the dispatch path finds a record in any status and skips enqueue entirely, marking the node graph-complete as the original dispatch did, rather than duplicating the side effect. The `async_reactor` equivalent is the `:async_reactor_ref` entry in `composed_contexts` (research.md decision 10). `completed` written by the step's own worker. |
77
+ | `serialized_result` | String (via `ContextSerializer.serialize_value`) | The step's `Success`/`Failure` value, same serialization the existing map-result bucket uses. |
78
+ | `reactor_class_name` | String | Needed for storage-key namespacing, mirrors every other storage primitive's `reactor_class_name` parameter. |
79
+
80
+ **Storage interface additions** (`RubyReactor::Storage::Adapter`, implemented by `RedisAdapter`):
81
+
82
+ ```ruby
83
+ store_step_result(context_id, step_name, serialized_result, reactor_class_name)
84
+ retrieve_step_result(context_id, step_name, reactor_class_name)
85
+ ```
86
+
87
+ Modeled directly on the existing `store_map_result(map_id, index, serialized_result, reactor_class_name, strict_ordering:)` / `retrieve_map_results(...)` pair (`lib/ruby_reactor/storage/adapter.rb:14-20`) — same TTL policy as `context_ttl` (records must not outlive the parent context's own retention window).
88
+
89
+ **Retention across a fire-and-forget parent (FR-018)**: the worker loads the *parent* context by id, so the parent must outlive the dispatched unit — including the common case where the parent completes immediately and nothing ever waits on the unit. Dispatch therefore refreshes the parent context's TTL, and the record is stamped with the same window. A worker that still finds no parent context (swept, or beyond the window) writes a `completed`/`Failure` record for its unit and logs it per FR-012 rather than raising — an unhandled raise would only hand the job to the backend's retry machinery to fail identically N more times. See research.md decision 10.
90
+
91
+ ## Completion Signal (new, ephemeral — not a stored entity)
92
+
93
+ The wake-up channel for FR-005's notified wait (research.md decision 4). Pure latency optimization: at-most-once, unpersisted, never load-bearing — every path falls back to the durable record above (or the child's context row).
94
+
95
+ | Channel | Published by | When |
96
+ |---|---|---|
97
+ | `rr:done:<parent_context_id>:<step_name>` | the `async_step`'s StepWorker | after `store_step_result` write |
98
+ | `rr:done:<child_execution_id>` | the `async_reactor` child's executor | after its terminal context save (unconditional — no-subscriber publish is near-free) |
99
+
100
+ Uses the existing, currently-unused `Storage::Adapter#publish`/`#subscribe` primitives (`adapter.rb:38-44`, implemented at `redis_adapter.rb:177-183`). Ordering contract: durable write **before** publish; waiter subscribes **before** its first record check; waiter re-checks the record on a coarse fallback interval. The subscriber MUST use a dedicated Redis connection (`SUBSCRIBE` puts a connection into subscriber mode — blocking the shared client would poison all other storage calls).
101
+
102
+ ## Configuration additions
103
+
104
+ | Knob | Default | Notes |
105
+ |---|---|---|
106
+ | `Configuration#async_wait_timeout` | `30` (seconds) | Seconds a `result()` notified wait will block before failing the referencing step with a timeout. Single global value — no per-reactor/per-reference override (Clarifications, Question 3). Rationale for 30s in research.md decision 5. |
107
+
108
+ **Derived (not configurable)**: the notified wait's fallback re-check interval is `async_wait_timeout / 10`, clamped to `1..5` seconds (3s at the default). It is a latency backstop for a lost signal, not a tuning surface — the clamp guarantees ≥10 re-checks inside any bound, so a dropped notification costs at most ~10% of the timeout. See research.md decision 4.
109
+
110
+ ## State/behavior changes to existing entities
111
+
112
+ - **`StepConfig`** (`lib/ruby_reactor/dsl/step_builder.rb`): the `async`/`async?` accessor is removed; using `async true` inside a `step` block raises a definition-time error naming the replacement DSL (FR-003). `ComposeBuilder#async` (`dsl/compose_builder.rb:31-33`) is **removed too** — it sets the very `StepConfig` `async:` flag being deleted (`compose_builder.rb:62`), so it cannot survive the removal; it raises the same definition-time error (migration: `background before: :<that compose step>`, which reproduces the old flag's semantics exactly). `MapBuilder#async` (`dsl/map_builder.rb:43`) is genuinely unaffected: it is a map-internal element-dispatch mode passed as a step *argument*, and the map's own `StepConfig` is hardcoded `async: false` (`map_builder.rb:111`) — it never touched the removed flag. (An earlier pass of this document had the compose/map carve-outs backwards; corrected after verifying both builders.)
113
+ - **`DependencyGraph`**: no schema change; `complete_step` is now called for an `async_step` at dispatch time rather than at true completion — a deliberate, documented divergence from every other step type, captured here so it isn't mistaken for a bug during implementation review.
114
+ - **Reactor-class DSL** (`lib/ruby_reactor/dsl/reactor.rb`): three new class macros — `background(after: nil, before: nil)`, `async_step(name, impl = nil, &block)`, `async_reactor(name, child_reactor_class, &block)` — alongside the existing `step`, `compose`, `map`, `interrupt`. The existing reactor-level `async`/`async?` (whole-reactor async) is unchanged.
115
+ - **`Context#composed_contexts`**: gains two new `type:` tags in its value union — `:async_step_ref` and `:async_reactor_ref` — alongside the existing `:composed` and `:map_ref`. No schema/serialization change (it's already a generic `Hash`, already serialized/deserialized as-is); this is purely a new convention for the `type:` field's allowed values, consumed by `Web::API.hydrate_composed_contexts` and by the new `TestSubject#async_step`/`#async_reactor` traversal helpers (mirroring `#composed`/`#map`).
116
+ - **`Web::API`** (`lib/ruby_reactor/web/api.rb`): `determine_step_type` gains `async_step`/`async_reactor` branches (replacing the removed `config.async?` branch); `build_structure` drops the per-step `async:` field and instead exposes the reactor's single hand-off point once, as the normalized `{ mode:, step: }` pair (so the dashboard can mark the cut regardless of which side declared it); `hydrate_composed_contexts` gains resolution branches for the two new ref types, mirroring `hydrate_map_ref`. See research.md decision 8.
117
+ - **GUI** (`gui/src/components/DagVisualizer.tsx`, `StepInspector.tsx`): need new rendering cases for the `'async_step'`/`'async_reactor'` step types surfaced by the API above — a task-phase implementation item, called out here so it isn't missed (constitution Principle IV: dashboard must stay current with the reactor state model).