ruby_reactor 0.5.4 → 0.7.0

data/specs/001-background-async-steps/data-model.md

@@ -0,0 +1,117 @@
+# Data Model: Background Execution & Real Async Steps
+
+This is a library feature — "entities" are DSL/runtime constructs and the storage records backing them, not application data.
+
+## Background Hand-off Point
+
+Reactor-class-level declaration, one per reactor.
+
+| Field | Type | Notes |
+|---|---|---|
+| `mode` | Symbol — `:after` or `:before` | Which side of the cut point the declaration named. Derived from which keyword the author supplied. |
+| `step` | Symbol | The named step. For `:after`, the last step to run in the calling process; for `:before`, the first step to run in the worker. Must reference a step defined in the same reactor (validated at class-definition time, FR-002). |
+
+Exposed to the runtime, `TestSubject`, and the dashboard as a single normalized reader — `background_handoff → { mode:, step: }` — never as a one-sided `background_after`. One concept with two trigger positions, not two parallel features: every consumer branches on `mode`, so no consumer can be accidentally implemented for `after:` only.
+
+**Storage**: not persisted as its own record — it compiles into which step reaching which position triggers the `StepExecutor#handle_async_step`-style enqueue. Enforced-single via a class-level guard.
+
+**Validation rules** (all definition-time errors):
+- Exactly zero or one `background` declaration per reactor class (FR-002).
+- Exactly one of `after:`/`before:` per declaration — both or neither raises (FR-002).
+- The named step must exist in `steps` at the time `background` is evaluated, or at class-definition-close time if steps can be declared afterward (implementation detail for tasks phase).
+- `background` combined with whole-reactor `async true` is rejected — the hand-off point would be silently meaningless inside a reactor that already runs entirely in a worker (spec Edge Cases).
+- `returns` naming an `async_step` or `async_reactor` is rejected — the return value must come from a same-process step (spec Edge Cases).
+
+## Async Step
+
+A step declared with `async_step :name` (or `async_step :name, ImplClass`) instead of `step`.
+
+| Field | Type | Notes |
+|---|---|---|
+| `name` | Symbol | Step name, same namespace as regular steps. |
+| `arguments` / `run_block` / `impl` | (existing `StepConfig` fields) | Same shape as a regular step — `async_step` is a `StepConfig` with a dispatch-mode marker, not a new config type. |
+| dispatch-mode marker | Boolean/Symbol | Distinguishes "run inline" vs "dispatch as an independent unit" at `StepExecutor#execute_step` time. |
+
+**Lifecycle** (state machine, tracked via the new Step Result Record below, keyed by `(context_id, step_name)`):
+
+```
+dispatched -> running -> completed(Success)
+                       -> completed(Failure)
+```
+
+- `dispatched`: parent process has, synchronously and in this order (durable-write-before-enqueue, F2): (1) written the Step Result Record with status `dispatched` and the `composed_contexts[step_name] = { type: :async_step_ref, name:, dispatched_at: }` reference onto its own context (see Async Step ↔ context linkage below), (2) enqueued the `StepWorker` job, (3) marked the step graph-complete for scheduling purposes (siblings may now proceed). No result exists yet.
+- `running`/`completed`: opaque to the parent process except through the Step Result Record; the parent only observes "record carries a terminal value" or "record still `dispatched`" (still-pending — keep waiting, subject to FR-005's timeout). A record *absent* entirely means the step was never dispatched — `result()` does not wait in that case. The parent may reach its own terminal state while a record is still `dispatched`; that is the fire-and-forget contract (FR-018), and the parent's status makes no claim about the unit's outcome.
+
+**Relationships**: An `async_step` is a dependency-graph node like any other step — other steps that declare `argument :x, result(:async_step_name)` get an automatic DAG edge (existing `DependencyGraph#add_step` behavior, unchanged) and, per FR-005, enter the notified wait for the terminal record when they resolve that argument.
+
+**Async Step ↔ context linkage (FR-008, FR-014)**: the *reference* (not the result) lives in `context.composed_contexts[step_name]`, the same field `compose`/`map` already populate for their own children — see research.md decision 8. The dashboard's existing `hydrate_composed_contexts` pipeline (`lib/ruby_reactor/web/api.rb`) is extended with a branch for `type: :async_step_ref` that resolves the Step Result Record to show current status/result, mirroring how it already resolves `:map_ref`.
+
+## Async Reactor
+
+A step declared with `async_reactor :name, ChildReactorClass`.
+
+| Field | Type | Notes |
+|---|---|---|
+| `name` | Symbol | Step name in the parent. |
+| `child_reactor_class` | Class | Must be a `RubyReactor::Reactor` subclass. |
+| `argument_mappings` | Hash | Same shape as `compose`'s `argument_mappings` — maps parent-visible sources to the child's inputs. |
+
+**Dispatch-time behavior** (FR-015, FR-016 — 2026-08-20 session):
+- Dispatch reuses the full pre-enqueue sequence of a top-level async run (child input validation → ordered-lock nonce assignment where the child declares one → persist child context → enqueue), never raw `perform_async` — see research.md decision 3. A validation failure fails the dispatching step itself (normal parent saga handling), distinct from child-execution failure (FR-009).
+- Deadlock guard: the child's `lock_config[:key_proc]` (and `semaphore_config` with `limit: 1`) is resolved against the mapped child inputs; a key matching one the dispatching execution currently holds fails the dispatch step immediately with an error naming the key and both reactor classes (research.md decision 9). No lock-owner sharing across the async boundary — reentrancy stays `compose`-only.
+
+**Relationships**:
+- Linked to the parent via `context.composed_contexts[step_name] = { type: :async_reactor_ref, name:, execution_id:, reactor_class_name:, dispatched_at: }` — written synchronously by the dispatching step, same field and pattern `compose`/`map` already use (research.md decision 8). Unlike `async_step`, no separate Step Result Record is needed for the *outcome*: the child is a normal, independently addressable `Reactor` with its own context row, so its terminal result is reached via the existing `storage.retrieve_context(execution_id, reactor_class_name)` / `ChildReactorClass.find(execution_id)` — the same lookup any other reactor execution uses.
+- **Not** added to the parent's compensation graph — no `compensate`/`undo` block is registered for this step (see spec Clarifications: compensation is opt-in via a later step reading the result, never automatic).
+- FR-008/FR-014: this `composed_contexts` entry is what the web dashboard's `hydrate_composed_contexts` reads to render the reference and drill into the child's own step structure (`build_structure` recursion, same as `compose`/`map`'s `nested_structure`).
+
+## Step Result Record (new storage-level entity)
+
+The durable record backing `async_step` completion. (`async_reactor` needs no equivalent record — per the relationship above, its outcome is simply its own context row, reached by execution id through the existing `retrieve_context`/`find` path.) This avoids what would otherwise be a race-prone write into the parent's context blob from a worker running concurrently with the still-executing parent process.
+
+| Field | Type | Notes |
+|---|---|---|
+| `context_id` | String (UUID) | The **parent** reactor's context id — the bucket is scoped per parent execution. |
+| `step_name` | Symbol/String | The `async_step`'s name within that parent. |
+| `status` | Enum: `dispatched`, `completed` | `dispatched` written synchronously **before** the job is enqueued (same checkpoint-before-enqueue ordering the existing hand-off uses, F2) — so a crash after enqueue can never find a job with no record, and this record doubles as the **re-attach marker** (FR-017): on recovery/resume the dispatch path finds a record in any status and skips enqueue entirely, marking the node graph-complete as the original dispatch did, rather than duplicating the side effect. The `async_reactor` equivalent is the `:async_reactor_ref` entry in `composed_contexts` (research.md decision 10). `completed` written by the step's own worker. |
+| `serialized_result` | String (via `ContextSerializer.serialize_value`) | The step's `Success`/`Failure` value, same serialization the existing map-result bucket uses. |
+| `reactor_class_name` | String | Needed for storage-key namespacing, mirrors every other storage primitive's `reactor_class_name` parameter. |
+
+**Storage interface additions** (`RubyReactor::Storage::Adapter`, implemented by `RedisAdapter`):
+
+```ruby
+store_step_result(context_id, step_name, serialized_result, reactor_class_name)
+retrieve_step_result(context_id, step_name, reactor_class_name)
+```
+
+Modeled directly on the existing `store_map_result(map_id, index, serialized_result, reactor_class_name, strict_ordering:)` / `retrieve_map_results(...)` pair (`lib/ruby_reactor/storage/adapter.rb:14-20`) — same TTL policy as `context_ttl` (records must not outlive the parent context's own retention window).
+
+**Retention across a fire-and-forget parent (FR-018)**: the worker loads the *parent* context by id, so the parent must outlive the dispatched unit — including the common case where the parent completes immediately and nothing ever waits on the unit. Dispatch therefore refreshes the parent context's TTL, and the record is stamped with the same window. A worker that still finds no parent context (swept, or beyond the window) writes a `completed`/`Failure` record for its unit and logs it per FR-012 rather than raising — an unhandled raise would only hand the job to the backend's retry machinery to fail identically N more times. See research.md decision 10.
+
+## Completion Signal (new, ephemeral — not a stored entity)
+
+The wake-up channel for FR-005's notified wait (research.md decision 4). Pure latency optimization: at-most-once, unpersisted, never load-bearing — every path falls back to the durable record above (or the child's context row).
+
+| Channel | Published by | When |
+|---|---|---|
+| `rr:done:<parent_context_id>:<step_name>` | the `async_step`'s StepWorker | after `store_step_result` write |
+| `rr:done:<child_execution_id>` | the `async_reactor` child's executor | after its terminal context save (unconditional — no-subscriber publish is near-free) |
+
+Uses the existing, currently-unused `Storage::Adapter#publish`/`#subscribe` primitives (`adapter.rb:38-44`, implemented at `redis_adapter.rb:177-183`). Ordering contract: durable write **before** publish; waiter subscribes **before** its first record check; waiter re-checks the record on a coarse fallback interval. The subscriber MUST use a dedicated Redis connection (`SUBSCRIBE` puts a connection into subscriber mode — blocking the shared client would poison all other storage calls).
+
+## Configuration additions
+
+| Knob | Default | Notes |
+|---|---|---|
+| `Configuration#async_wait_timeout` | `30` (seconds) | Seconds a `result()` notified wait will block before failing the referencing step with a timeout. Single global value — no per-reactor/per-reference override (Clarifications, Question 3). Rationale for 30s in research.md decision 5. |
+
+**Derived (not configurable)**: the notified wait's fallback re-check interval is `async_wait_timeout / 10`, clamped to `1..5` seconds (3s at the default). It is a latency backstop for a lost signal, not a tuning surface — the clamp guarantees ≥10 re-checks inside any bound, so a dropped notification costs at most ~10% of the timeout. See research.md decision 4.
+
+## State/behavior changes to existing entities
+
+- **`StepConfig`** (`lib/ruby_reactor/dsl/step_builder.rb`): the `async`/`async?` accessor is removed; using `async true` inside a `step` block raises a definition-time error naming the replacement DSL (FR-003). `ComposeBuilder#async` (`dsl/compose_builder.rb:31-33`) is **removed too** — it sets the very `StepConfig` `async:` flag being deleted (`compose_builder.rb:62`), so it cannot survive the removal; it raises the same definition-time error (migration: `background before: :<that compose step>`, which reproduces the old flag's semantics exactly). `MapBuilder#async` (`dsl/map_builder.rb:43`) is genuinely unaffected: it is a map-internal element-dispatch mode passed as a step *argument*, and the map's own `StepConfig` is hardcoded `async: false` (`map_builder.rb:111`) — it never touched the removed flag. (An earlier pass of this document had the compose/map carve-outs backwards; corrected after verifying both builders.)
+- **`DependencyGraph`**: no schema change; `complete_step` is now called for an `async_step` at dispatch time rather than at true completion — a deliberate, documented divergence from every other step type, captured here so it isn't mistaken for a bug during implementation review.
+- **Reactor-class DSL** (`lib/ruby_reactor/dsl/reactor.rb`): three new class macros — `background(after: nil, before: nil)`, `async_step(name, impl = nil, &block)`, `async_reactor(name, child_reactor_class, &block)` — alongside the existing `step`, `compose`, `map`, `interrupt`. The existing reactor-level `async`/`async?` (whole-reactor async) is unchanged.
+- **`Context#composed_contexts`**: gains two new `type:` tags in its value union — `:async_step_ref` and `:async_reactor_ref` — alongside the existing `:composed` and `:map_ref`. No schema/serialization change (it's already a generic `Hash`, already serialized/deserialized as-is); this is purely a new convention for the `type:` field's allowed values, consumed by `Web::API.hydrate_composed_contexts` and by the new `TestSubject#async_step`/`#async_reactor` traversal helpers (mirroring `#composed`/`#map`).
+- **`Web::API`** (`lib/ruby_reactor/web/api.rb`): `determine_step_type` gains `async_step`/`async_reactor` branches (replacing the removed `config.async?` branch); `build_structure` drops the per-step `async:` field and instead exposes the reactor's single hand-off point once, as the normalized `{ mode:, step: }` pair (so the dashboard can mark the cut regardless of which side declared it); `hydrate_composed_contexts` gains resolution branches for the two new ref types, mirroring `hydrate_map_ref`. See research.md decision 8.
+- **GUI** (`gui/src/components/DagVisualizer.tsx`, `StepInspector.tsx`): need new rendering cases for the `'async_step'`/`'async_reactor'` step types surfaced by the API above — a task-phase implementation item, called out here so it isn't missed (constitution Principle IV: dashboard must stay current with the reactor state model).

data/specs/001-background-async-steps/plan.md

@@ -0,0 +1,168 @@
+# Implementation Plan: Background Execution & Real Async Steps
+
+**Branch**: `001-background-async-steps` | **Date**: 2026-08-16 | **Spec**: [spec.md](./spec.md)
+
+**Input**: Feature specification from `/specs/001-background-async-steps/spec.md`
+
+## Summary
+
+Replace the confusing per-step `async: true` flag (only the first flagged step in a reactor actually takes effect — the rest are silently ignored) with a single, unambiguous reactor-level `background` declaration that hands off all remaining steps to an independent worker job. The cut point is nameable from either side — `background after: :second` (that step is the last to run in the calling process) or `background before: :third` (that step is the first to run in the worker) — which coincide in a linear chain but pin different steps in a DAG. Add two genuinely new capabilities on top of that: `async_step`, whose unit of work is dispatched to run in its own independent worker while the rest of the reactor keeps executing in the calling process; and `async_reactor`, which dispatches a whole nested reactor to run independently, linked to the parent by execution id but excluded from the parent's compensation graph. Both `async_step` and `async_reactor` results are consumed via the existing `result(:name)` template helper, which gains a notified wait (completion signal via the storage adapter's pub/sub, durable-record-first, bounded fallback re-check — see research.md decision 4) when the referenced work hasn't finished yet. Per the clarified spec, neither `async_step` nor `async_reactor` failures auto-compensate the parent — compensation only happens if a later step explicitly reads the result and decides to fail. Lock ownership is never shared across the async boundary; a same-key collision between parent and child fails at dispatch (FR-015), and dispatch reuses the full pre-enqueue sequence including child input validation (FR-016). All dispatch continues to go through the existing pluggable `configuration.async_router` (Sidekiq or ActiveJob), unchanged.
+
+## Technical Context
+
+**Language/Version**: Ruby >= 3.0.0 (per constitution's Technical Constraints)
+
+**Primary Dependencies**: `sidekiq` and/or `activejob` (pluggable via `RubyReactor::Configuration#async_router`, already introduced by the ActiveJob Support feature on `main`), `redis` (storage), `dry-validation` (input validation DSL, unaffected by this feature)
+
+**Storage**: Redis via `RubyReactor::Storage::RedisAdapter` (implements `RubyReactor::Storage::Adapter`). This feature adds one new storage primitive pair for per-step async results (see data-model.md) modeled directly on the existing `store_map_result` / `retrieve_map_results` pair used by `map`.
+
+**Testing**: RSpec (mandatory per constitution), in two lanes (research.md decision 6):
+
+- *Unit lane* — the existing `RubyReactor::RSpec::TestSubject` DSL (`test_reactor`, `have_run_step`, `drain_async_jobs`/`process_pending_jobs` via `AsyncTestHelpers`) over the in-memory queue fakes, for everything that does not involve a caller blocked on `result()`: definition-time guards, dispatch shape, `composed_contexts` refs, read semantics against an already-terminal record.
+- *Orchestration lane* — a **real worker consuming a real queue**, required by constitution Principle III for every async orchestration path (a caller blocked in the notified wait while an independent worker completes the work; the interleaving is not reproducible under an in-process fake, which only runs jobs when the spec itself calls `drain_async_jobs` — and the spec is blocked). Sidekiq backend: a live `sidekiq` process booted against the test Redis. ActiveJob backend: the `:async` adapter (a real queue runner, not the `:test` fake).
+
+No parallel test *framework* — per explicit instruction, only the gem's built-in spec helpers are used; the orchestration lane is one new shared context (`spec/support/real_async_backend.rb`) plus new helper methods on the existing `AsyncTestHelpers`/`TestSubject` modules, and the `TestSubject` interceptor logic is updated where it hardcodes the old per-step `async?` flag (`prepare_execution_class`/`apply_mock_interceptor`).
+
+**Target Platform**: Server-side Ruby (gem consumed by Rails/Sinatra-style host apps); demo validated against `demo_app/` (Rails app already in the repo)
+
+**Project Type**: Library (Ruby gem) with a bundled `demo_app/` integration example — matches the constitution's Gem-First Design principle
+
+**Performance Goals**: Not a throughput-sensitive change — dispatch overhead should stay within the same order of magnitude as the existing `map` per-element dispatch path it reuses patterns from. No new SLO introduced.
+
+**Constraints**: Blocking waits on `result()` (FR-005) are notified waits — completion signal via the storage adapter's existing (currently unused) `publish`/`subscribe` primitives, durable-record-first ordering, subscribe-before-check on the waiter, coarse fallback re-check, all bounded by a single library-wide `Configuration#async_wait_timeout` (new knob; see research.md decision 4 and data-model.md "Completion Signal"); the subscriber uses a dedicated Redis connection. Never an unbounded wait. Lock ownership is never shared across the async boundary (FR-015 dispatch-time deadlock guard; reentrancy stays `compose`-only). Must not change behavior of the untouched reactor-level `async` flag (`self.class.async?`, "Full Reactor Async"), of `compose`'s execution/compensation semantics (its `async` flag is removed with the shared `StepConfig` flag, per FR-003 — see contracts/public-dsl.md), or of `map` (including its map-internal `async` element-dispatch option, which is a different mechanism and stays).
+
+**Scale/Scope**: Single-gem change; touches DSL (`dsl/reactor.rb`, `dsl/step_builder.rb`, `dsl/compose_builder.rb`), executor (`executor/step_executor.rb` for dispatch, `executor.rb` for the completion-signal publish on terminal save; `dependency_graph.rb` itself unchanged), a new `async_reactor` dispatch step (`step/async_reactor_step.rb`), a framework-agnostic async-step worker body (`step_worker.rb`) plus its two adapter bindings and a `perform_step_async` entry point on both routers (mirroring `Adapters::{Sidekiq,ActiveJob}::MapElementWorker`), the new `async_waiter.rb` shared wait core, two new error classes, `context.rb` (documented `composed_contexts` ref types), `template/result.rb`, `storage/adapter.rb` + `storage/redis_adapter.rb`, `configuration.rb`, RSpec helpers plus the new real-worker shared context in `spec/support/`, the bundled web dashboard (`lib/ruby_reactor/web/api.rb` + `gui/src/components/{DagVisualizer,StepInspector}.tsx`, per FR-014), `demo_app/`, `documentation/`, `README.md`.
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+- **I. Gem-First Design**: PASS. New DSL (`background`, `async_step`, `async_reactor`) ships in `lib/ruby_reactor/dsl/`; the new worker class ships in both `lib/ruby_reactor/adapters/sidekiq/` and `lib/ruby_reactor/adapters/active_job/`, mirroring the existing adapter-isolation pattern — no host-application coupling, Sidekiq/ActiveJob stay optional per the existing pluggable-router mechanism.
+- **II. Saga Pattern Integrity (NON-NEGOTIABLE)**: PASS, with a deliberate, spec-clarified narrowing. `background` hand-off (either form) preserves full compensation — it is the same underlying mechanism as today's step-level hand-off, just triggered once, unambiguously, and nameable from either side. `async_step`/`async_reactor` do NOT auto-compensate on failure — this was explicitly clarified with the user (see spec Clarifications) as an intentional escape hatch mirroring the existing `compose`/whole-reactor-async model, not a silent gap: a later step that reads the result via `result(:name)` can always trigger compensation itself, so no failure is ever unrecoverable, it is opt-in rather than automatic. This mirrors the already-shipped behavior of `async_reactor`-adjacent whole-reactor `async true` composition today, so it is not a new category of principle exception.
+- **III. Test-First with Real Infrastructure**: PASS, after correction. Tests are written first and run against real Redis. An earlier pass of this plan claimed PASS while resting the *entire* async-orchestration test strategy on `Sidekiq::Testing.fake!` + in-process `drain_async_jobs` and the ActiveJob `:test` adapter — those are mocked Sidekiq/queue state, which Principle III forbids on async orchestration paths, and they also cannot reproduce the behavior under test (a caller blocked in the notified wait cannot call `drain_async_jobs`, so the job would never run and every awaited-result spec would time out rather than pass). Corrected: every spec exercising a blocked caller plus a concurrently-completing worker runs in the orchestration lane — a live `sidekiq` process against the test Redis, and the ActiveJob `:async` adapter — via the shared context in `spec/support/real_async_backend.rb`. The in-memory fakes are retained strictly for unit-level assertions with no blocked caller (definition-time guards, dispatch shape, read semantics against an already-terminal record), which is exactly the carve-out Principle III allows.
+- **IV. Observability by Default**: PASS, and specifically checked against "The web dashboard MUST remain current with the reactor state model": FR-012 requires machine-parseable (key=value/JSON) structured log entries for hand-off/dispatch/completion carrying reactor name, step name, and execution id — plus, on any failure entry, the failure reason and redacted inputs, which Principle IV mandates and an earlier pass of FR-012 omitted; this matters more here than elsewhere because the fire-and-forget model (FR-009/FR-011) can leave a worker-side failure with no other surface. Implemented following the existing `middlewares.on(:before_async_enqueue, ...)` pattern; FR-008/FR-014 additionally require the async link to be recorded on the parent's own context (reusing `composed_contexts`, the same field `compose`/`map` already use — research.md decision 8) and rendered/drillable in `Web::API` + the `gui/` dashboard, not just logged. An earlier pass of this plan only logged the link and missed the dashboard requirement; corrected after review.
+- **V. Simplicity and Semantic Versioning**: PASS. This is a MAJOR (breaking) change to the public step DSL (removal of per-step `async`), documented per FR-013. The design deliberately reuses three existing mechanisms (step-level hand-off in `StepExecutor#handle_async_step`, `map`'s per-unit dispatch-and-collect pattern, and the existing `DispatchResult`/`result()` template mechanism) rather than inventing a fourth. No speculative per-reactor/per-reference timeout override is introduced (resolved via clarification) — a single global config value only, added when a second real use case exists.
+
+No violations requiring the Complexity Tracking table.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/001-background-async-steps/
+├── plan.md              # This file (/speckit-plan command output)
+├── research.md          # Phase 0 output (/speckit-plan command)
+├── data-model.md         # Phase 1 output (/speckit-plan command)
+├── quickstart.md        # Phase 1 output (/speckit-plan command)
+├── contracts/            # Phase 1 output (/speckit-plan command)
+│   └── public-dsl.md
+└── tasks.md              # Phase 2 output (/speckit-tasks command - NOT created by /speckit-plan)
+```
+
+### Source Code (repository root)
+
+```text
+lib/ruby_reactor/
+├── dsl/
+│   ├── reactor.rb                  # ADD: `background(after:/before:)`, `async_step`, `async_reactor`
+│   │                                #      class macros + definition-time guards (dup background,
+│   │                                #      unknown step, both-or-neither key, background×async-true,
+│   │                                #      returns×async-unit); whole-reactor `async` stays
+│   ├── step_builder.rb             # REMOVE: `async` step-level flag (raise on use, FR-003)
+│   └── compose_builder.rb          # REMOVE: `async` flag too — it sets the same StepConfig flag
+│                                    #         (raise on use, FR-003); map_builder.rb untouched
+├── executor/
+│   └── step_executor.rb            # REPLACE: `handle_async_step` keyed off the reactor's single
+│                                    #          hand-off point (post-execution check for `after:`,
+│                                    #          pre-execution check for `before:`), not per-step `async?`;
+│                                    #          ADD: async_step dispatch (fire-and-continue, marks the
+│                                    #          node complete in the DependencyGraph at dispatch time so
+│                                    #          siblings aren't blocked) and async_reactor dispatch;
+│                                    #          both re-attach instead of re-dispatching when a durable
+│                                    #          dispatch marker already exists (FR-017), and extend the
+│                                    #          parent context's retention window at dispatch (FR-018)
+├── executor.rb                     # ADD: publish completion signal after terminal context save
+│                                    #      (async_reactor completer side, research.md decision 4)
+├── async_waiter.rb                 # NEW: shared FR-005 wait core — subscribe-first, durable check,
+│                                    #      signal wake + coarse fallback re-check, timeout bound
+├── step_worker.rb                  # NEW: framework-agnostic async_step worker body (load parent
+│                                    #      context, resolve args, run, store record, publish signal;
+│                                    #      missing-parent-context handling per FR-018)
+├── context.rb                      # UPDATE: document the two new `composed_contexts` ref types
+│                                    #         (`:async_step_ref`, `:async_reactor_ref`) — no
+│                                    #         serialization change
+├── dependency_graph.rb             # UNCHANGED (no schema change; dispatch-time complete_step is
+│                                    # called by the executor, see data-model.md)
+├── error/
+│   ├── async_wait_timeout_error.rb # NEW: raised when an FR-005 wait exceeds its bound
+│   └── deprecated_dsl_error.rb     # NEW: definition-time error for the removed `async` flag (FR-003)
+├── step/
+│   └── async_reactor_step.rb       # NEW: async_reactor dispatch step — full pre-enqueue sequence
+│                                    #      (FR-016) + FR-015 deadlock guard, no compensate/undo
+├── adapters/
+│   ├── sidekiq/step_worker.rb      # NEW: independent one-shot worker for a single async_step
+│   ├── sidekiq/router.rb           # ADD: `perform_step_async` dispatch entry point
+│   ├── active_job/step_worker.rb   # NEW: ActiveJob counterpart
+│   └── active_job/router.rb        # ADD: `perform_step_async` dispatch entry point
+├── template/
+│   └── result.rb                   # UPDATE: notified wait (subscribe-first, record check, fallback
+│                                    #         re-check, timeout) when the referenced async result
+│                                    #         is not yet available (FR-005, research.md decision 4)
+├── storage/
+│   ├── adapter.rb                  # ADD: `store_step_result` / `retrieve_step_result` interface methods
+│   │                                #      (async_step outcome only — async_reactor reuses the existing
+│   │                                #      retrieve_context/find path, no new primitive needed there)
+│   └── redis_adapter.rb            # implement them (mirrors store_map_result/retrieve_map_results);
+│                                    # ADD dedicated-connection subscribe (SUBSCRIBE must not block
+│                                    # the shared client)
+├── configuration.rb                # ADD: `async_wait_timeout` config knob (default 30s)
+├── web/
+│   └── api.rb                      # UPDATE: `determine_step_type`/`build_structure` (drop removed
+│                                    #         `config.async?` field, expose the normalized
+│                                    #         `background_handoff { mode:, step: }` pair once per
+│                                    #         reactor, add async_step/async_reactor branches) and
+│                                    #         `hydrate_composed_contexts` (resolve the two new
+│                                    #         composed_contexts ref types) — FR-008, FR-014
+└── rspec/
+    └── test_subject.rb             # UPDATE: `prepare_execution_class`/interceptors to understand the
+                                     #         normalized `background_handoff`/`async_step`/
+                                     #         `async_reactor` instead of the removed per-step `async?`;
+                                     #         ADD `#async_step`/`#async_reactor` traversal helpers
+                                     #         mirroring `#composed`/`#map`
+
+gui/src/components/
+├── DagVisualizer.tsx               # UPDATE: render 'async_step'/'async_reactor' step types (FR-014)
+└── StepInspector.tsx               # UPDATE: same
+
+spec/ruby_reactor/                  # gem's own unit/integration specs (mirrors lib/ layout above)
+spec/support/reactors/              # new fixture reactors for background/async_step/async_reactor
+spec/support/async_backends.rb      # NEW: shared context parameterizing a group over both backends
+spec/support/real_async_backend.rb  # NEW: orchestration lane — live `sidekiq` process against the
+                                     #      test Redis / ActiveJob `:async` adapter (Principle III)
+
+demo_app/app/reactors/
+├── full_async_reactor.rb           # UNCHANGED (whole-reactor async, out of scope)
+├── partial_async_reactor.rb        # REPLACED by `background_demo_reactor.rb`, a `background after:`
+│                                    # example (old per-step `async true` syntax is removed, FR-003);
+│                                    # migrated in Phase 2 with the rest of the call sites, since the
+│                                    # old syntax stops parsing the moment the flag is removed
+├── async_step_demo_reactor.rb      # NEW: demonstrates async_step + result() wait (send_email example)
+└── async_reactor_demo_reactor.rb   # NEW: demonstrates async_reactor fire-and-forget vs. awaited
+demo_app/spec/reactors/             # matching specs, using the same built-in TestSubject helpers
+
+documentation/
+├── async_reactors.md               # REWRITE "Step-Level Async" to `background after:`/`before:`;
+│                                    # ADD sections for `async_step` and `async_reactor`
+└── composition.md                  # cross-reference `async_reactor` vs. `compose`
+demo_app/documentation/             # kept in sync with the same edits (duplicate copy, see research.md)
+
+README.md                           # rewrite "Step-Level Async" subsection, add async_step/async_reactor
+CHANGELOG.md                        # breaking-change entry (FR-013)
+```
+
+**Structure Decision**: Single-project Ruby gem layout (existing `lib/ruby_reactor/**`, `spec/**`, plus the bundled `demo_app/` Rails integration example and `documentation/**`). No new top-level directories — every change lands inside the existing module boundaries (`dsl/`, `executor/`, `adapters/`, `template/`, `storage/`, `rspec/`), consistent with Principle I (Gem-First Design).
+
+## Complexity Tracking
+
+*No Constitution Check violations — table not needed.*

data/specs/001-background-async-steps/quickstart.md

@@ -0,0 +1,102 @@
+# Quickstart: Validating Background Execution & Real Async Steps
+
+Prerequisites: repo checked out on this branch, `bundle install` run (repo root and `demo_app/`), Redis running (`docker-compose up -d` at repo root; the gem's suite reads `RUBY_REACTOR_TEST_REDIS_URL`, defaulting to `redis://localhost:6780` — see `spec/spec_helper.rb:13`). Real Redis, not mocked, per the constitution's storage requirement.
+
+## 1. Gem-level unit/integration specs
+
+```bash
+bundle exec rspec spec/ruby_reactor/dsl/reactor_background_spec.rb        # background after:/before:
+bundle exec rspec spec/ruby_reactor/dsl/async_step_spec.rb                # async_step
+bundle exec rspec spec/ruby_reactor/dsl/async_reactor_spec.rb             # async_reactor
+bundle exec rspec spec/ruby_reactor/rspec/test_subject_async_spec.rb      # TestSubject support for the new DSL
+```
+
+(Exact spec file names are placeholders for the tasks phase — see contracts/public-dsl.md for the acceptance scenarios each must cover, taken directly from spec.md's User Stories 1–3.)
+
+Expected outcomes, mapped to spec.md acceptance scenarios:
+
+- A reactor with `background after: :second` runs `:first`/`:second` inline and `:third` via a dispatched job; `MyReactor.run` returns an `DispatchResult`; `TestSubject`'s `drain_async_jobs` completes it (US1 scenario 1, SC-001).
+- The same reactor declared `background before: :third` behaves identically in this linear case, and `:third` never executes in the calling process (US1 scenario 2). In a branching fixture the two forms pin different steps — assert the specific step each one guarantees.
+- A `background` declaration that is duplicated, names an unknown step, carries both `after:` and `before:`, carries neither, or sits on a whole-reactor-`async` reactor each raise at class-definition time (US1 scenarios 4-5, FR-002).
+- A reactor still using `step { async true }` raises a `ValidationError` at class-definition time, not at run time (US1 scenario 3, SC-004).
+- An `async_step` reactor: a sibling step with no dependency on it completes without waiting; a step declaring `argument :x, result(:async_step_name)` receives the correct value once available (US2, SC-002).
+- An `async_step` failure with no downstream reader does NOT flip the parent to `failed`/trigger compensation; a downstream reader that inspects the failure and returns `Failure` DOES trigger compensation (US2 acceptance scenario 3, Clarifications).
+- An `async_reactor` with no reader: forcing the child to fail does not compensate the parent. An `async_reactor` with a reader: the reader's `run` block sees the child's real `Success`/`Failure` and can choose to propagate (US3, SC-003).
+- A `result()` reference to a never-completing async unit fails with a timeout error, not an indefinite hang (SC-005) — verify by pointing `Configuration#async_wait_timeout` at a short value in the spec and never draining the corresponding job.
+- The notified wait is race-free (FR-005, Session 2026-08-20 clarification): a completion that lands *before* the waiter subscribes is still found (subscribe-then-check), and a dropped signal is caught by the fallback re-check — verify with a spec that completes the async unit before the reader step runs, and one that publishes nothing and relies on the record alone.
+- An `async_reactor` whose child declares the same `lock` key the parent holds fails at dispatch with an error naming the key and both reactors (FR-015) — not a wait-then-timeout.
+- Dispatching an `async_reactor` with invalid child inputs fails the dispatching step in the parent (FR-016), while a child that fails *during execution* still never auto-compensates the parent (FR-009).
+
+## 2. Sidekiq AND ActiveJob backends both pass
+
+Per Assumptions (spec.md) and Technical Context (plan.md), this feature must not hardcode a backend. Run the relevant spec files twice — once with the default (`Sidekiq::Testing.fake!`) and once with `config.async_router = RubyReactor::Adapters::ActiveJob::Router` + `ActiveJob::Base.queue_adapter = :test` — however the existing test suite's backend-parameterization convention already does this (check `spec/support/` for a shared-example/shared-context wrapping both backends before inventing a new one).
+
+## 3. `demo_app` end-to-end
+
+```bash
+cd demo_app
+bin/rails db:test:prepare   # if needed
+bundle exec rspec spec/reactors/background_demo_reactor_spec.rb
+bundle exec rspec spec/reactors/async_step_demo_reactor_spec.rb
+bundle exec rspec spec/reactors/async_reactor_demo_reactor_spec.rb
+```
+
+These exercise the new example reactors (`app/reactors/*.rb`, replacing `partial_async_reactor.rb`'s old syntax) through the full Rails/Sidekiq(or ActiveJob) stack the demo app wires up, giving a real (not just unit-tested) confirmation that the feature works end-to-end — the closest thing this library has to a "run it in a browser" check for a non-UI gem.
+
+## 4. Manual smoke check (optional, for a human reviewer)
+
+```ruby
+# bin/console or demo_app's bin/console
+result = MyReactor.run(...)
+result.class            # => RubyReactor::DispatchResult (background) or RubyReactor::Success (no background)
+RubyReactor::RSpec::AsyncTestHelpers.drain_async_jobs if defined?(RubyReactor::RSpec) # in a test/console context
+MyReactor.find(result.execution_id).result
+```
+
+## 5. Web dashboard visibility (FR-008, FR-014, SC-006)
+
+```bash
+bundle exec rspec spec/ruby_reactor/web/api_spec.rb   # determine_step_type / hydrate_composed_contexts additions
+cd gui && npm run dev                                  # or npm test if component tests exist
+```
+
+Run a reactor with an `async_step` and one with an `async_reactor` (e.g. the new `demo_app` example reactors from step 3), open the dashboard, and confirm: the step renders with a distinct `async_step`/`async_reactor` badge (not falling back to generic `step`), and for `async_reactor`, clicking through opens the linked child execution — the same drill-down `compose`/`map` already provide. This is a UI-affecting change (constitution: "For UI or frontend changes, start the dev server and use the feature in a browser before reporting the task as complete") — do not report FR-014 done from passing specs alone.
+
+## 6. Documentation review
+
+- `documentation/async_reactors.md` and its duplicate `demo_app/documentation/async_reactors.md` render correctly and no longer show `step { async true }` as the recommended step-level pattern.
+- `README.md`'s "Step-Level Async" subsection reflects `background after:`/`before:`.
+- `CHANGELOG.md` has a breaking-change entry under the correct semantic heading (constitution Development Workflow requirement).
+
+## Done criteria
+
+All specs above pass; `bundle exec rubocop` is clean (constitution requirement, no `--disable-pending-cops`); `demo_app`'s specs pass against both configured backends if the demo app is wired to test both (verify via its `config/` — otherwise document which single backend the demo exercises).
+
+---
+
+## Validation run — 2026-08-22
+
+| Step | Result |
+|---|---|
+| 1. Gem specs | **733 examples, 0 failures**, 1 pre-existing pending |
+| 2. Both backends | Every async spec group runs twice via `for_each_async_backend` (Sidekiq fake mode + ActiveJob `:test`); the blocked-caller groups run twice via `for_each_real_async_backend` (a **live sidekiq process** against the test Redis + the ActiveJob `:async` adapter) |
+| 3. `demo_app` | **94 examples, 0 failures** (13 of them the new async examples) |
+| 5. Dashboard — API | Driven through the real `Web::Application` Rack app for a reactor carrying all three new constructs: `async_step`/`async_reactor` step types render as themselves (not generic `step`), the dead per-step `async` field is gone, `background_handoff` is exposed once per reactor, the async_reactor node carries `nested_structure` plus a child `execution_id` that resolves to an addressable execution, and the `async_step` ref hydrates to its Step Result Record |
+| 5. Dashboard — GUI | `tsc -b` clean, `npm run build` succeeds, 36 component tests pass, `eslint` at its pre-existing baseline |
+| 5. Dashboard — browser | **Not performed.** See below. |
+| 6. Documentation | Both trees rewritten and kept in sync; no remaining example anywhere shows the removed `step { async true }` |
+| Rubocop | **216 files, no offenses** (no `--disable-pending-cops`) |
+
+### Open item: visual browser confirmation
+
+Step 5 asks for a human to open the dashboard and confirm the new nodes render
+and the async_reactor drill-down opens the child execution. That was **not**
+done: the implementing session was non-interactive with no browser available.
+
+Everything the browser check would depend on is verified programmatically above
+— the payload shape the components read, the components' own tests, the
+typecheck, and a clean production build — but the rendered result itself has not
+been looked at. **A reviewer should still open the dashboard once** (`cd gui &&
+npm run dev` against a reactor with an `async_step` and an `async_reactor`, e.g.
+the new `demo_app` examples) and confirm the node styling and the child link,
+before considering FR-014 closed.