ruby_reactor 0.8.4 → 0.8.6

data/specs/009-distributed-map-undo/plan.md

@@ -0,0 +1,323 @@
+# Implementation Plan: Distributed Map Rollback and Bounded Fan-out
+
+**Branch**: `distributed_map_undo` | **Date**: 2026-09-30 | **Spec**: [spec.md](spec.md)
+
+**Input**: Feature specification from `specs/009-distributed-map-undo/spec.md`
+
+**Revised 2026-09-30 after `/speckit-analyze`**:
+
+- per-throw back-pressure bound;
+- started-element coverage;
+- the handed-off handshake;
+- requeue on a contended element lock;
+- aggregation dedup and symbols;
+- the `cancel` guard;
+- the GUI status surfaces;
+- one reactor per demo file;
+- the justified `inline!` spec.
+
+## Summary
+
+Four changes to `map`, one shared mechanism:
+
+1. **Distributed rollback (US1, P1)**. A fan-out map's compensate/undo stops loading every element
+   into one process.
+   - It starts a map rollback: one job per started element (only completed ones undo anything),
+     with the forward back-pressure mechanism. No throw enqueues more than `batch_size` (R-02,
+     R-06).
+   - The rollback then hands off with `Error::RollbackHandedOff` (R-04). The undo stack is the
+     resume cursor, the run becomes `rolling_back`, and it saves under its lock.
+   - The owner's `Worker` is resumed once, and only after the hand-off is saved. Whichever of the
+     last-settling job and the hand-off handler sees both "settled" and `handed_off` claims the
+     signal (R-05). `Executor#resume_rollback` then adopts the settled map, undoes the steps before
+     it and finalizes with the same `Failure` the inline path produces (I-7).
+   - Inline maps keep rolling back in process, reading element states in chunks of 100 (R-01).
+2. **Fan-out map inside a composed child (US2, P2)**. The collector stops resuming contexts. It
+   signals the **owner** run, the top-level `root_context || context`, and `MapStep#run` adopts the
+   settled outcome on re-entry at any depth (R-03). This also removes the collector's
+   failure-branch write: map failures now roll back through the executor's normal path.
+3. **Default batch size 50 (US3, P2)**. `Map::DEFAULT_BATCH_SIZE`, stored in the map metadata with
+   `atomic`. This fixes the sweeper re-dispatch losing both (R-10).
+4. **`atomic` (US4, P3)**. It renames `fail_fast`, which stays as a warned alias. Legacy job
+   payloads are normalized in one place (R-11).
+
+Supporting changes:
+
+- `Reactor#undo` holds the run's context lock, and neither undo nor cancel is accepted on a
+  `rolling_back` run (R-12, FR-027).
+- An element rollback job requeues itself on a contended element lock before reporting it in
+  flight (R-07).
+- `resume_execution` saves before releasing that lock (R-13).
+- Map undo records carry no arguments (R-14).
+- Observability: `rolling_back` status, map rollback progress in the dashboard, and structured log
+  lines (R-15).
+
+## Technical Context
+
+**Language/Version**: Ruby >= 3.0
+
+**Primary Dependencies**: Sidekiq and ActiveJob adapters, Redis, dry-validation. Optional
+OpenTelemetry middleware. GUI: React/Vite (`gui/`, built into `lib/ruby_reactor/web/public`). No
+new dependencies.
+
+**Storage**: Redis through `Storage::RedisAdapter`. New data (see [data-model.md](data-model.md)):
+
+- Context: the `rollback` hash and the `rolling_back` status.
+- Map metadata: `owner_context_id`, `owner_reactor_class_name`, `batch_size`, `atomic`.
+- `map:<id>:owner_signalled`.
+- The `map:<id>:rollback:{metadata,offset,results,indexes,handed_off,signalled}` records.
+
+The element-context index write is deduplicated at the source.
+
+**Testing**: RSpec against real Redis (`redis://localhost:6780`), with Sidekiq fake mode plus
+`drain_async_jobs` for every async path. One new `inline!` spec, for the inline-mode settle check;
+it is justified in Complexity Tracking.
+
+- **Oracle** (I-7): the same reactor with an inline map and with a fan-out map.
+- **Fault injection**: an interruption during an element undo, a dropped rollback job, a dropped
+  owner resume.
+- **Scale**: `:slow` tag, 10k elements.
+- **Demo specs**: shipped matchers only, plus a new `be_rolling_back`.
+- **SC-002**: a demo benchmark rake task on Docker Sidekiq at `-c 10` (quickstart §5).
+
+**Target Platform**: Ruby gem (MRI), Sidekiq/ActiveJob workers, Rails demo app in Docker.
+
+**Project Type**: Library (gem) with a bundled dashboard.
+
+**Performance Goals**:
+
+- No rollback job loads more than one element context.
+- The coordinating owner holds no per-element state beyond the failures it reports (R-09).
+- With 10 workers, rolling back 10k elements is at least 5× faster than the serial in-process
+  rollback of the same elements, the inline-map path (SC-002).
+- Enqueue burst at most `B` per throw. Outstanding jobs are not bounded: the trigger is
+  position-based, as forward (R-02).
+- Forward fan-out gains one job hop per map completion (collector → owner Worker, R-03).
+
+**Constraints**:
+
+- Single-writer rule (I-1): the collector and element rollback jobs never write a reactor context.
+- A hand-off saves before its lock is released (I-6, R-13).
+- The undo stack pops only after an undo returns (I-5, 008 R-16).
+- `Error::Rescuable` must not match `RollbackHandedOff` (R-04).
+- Legacy in-flight payloads (`fail_fast`, metadata without owner ids) keep working (S-6).
+- An owner resume from a rollback requires `rollback:handed_off`, which is set after the hand-off
+  save (I-2, R-05).
+- A `rolling_back` run is never cancelled or undone again (I-8, FR-027).
+
+**Scale/Scope**:
+
+- About 24 library files touched. New files: `map.rb`, `error/rollback_handed_off.rb`,
+  `map/element_rollback.rb`, one `map_element_rollback_worker.rb` per adapter, and
+  `dsl/definition_warnings.rb` (extracted).
+- 9 new spec files and about 14 updated, plus 2 new spec support files.
+- Demo:
+  - 7 reactor files, one reactor per file: `DistributedRefundDemoReactor`,
+    `DistributedRefundElementReactor`, `ComposedFanOutDemoReactor`, `ComposedFanOutChildReactor`,
+    `ComposedFanOutItemReactor`, `DefaultBatchFanOutDemoReactor`, `InlineRefundBenchmarkReactor`;
+  - 4 rake tasks plus the `map_execution_undo` group;
+  - 3 demo specs.
+- 7 documentation files, CHANGELOG, and `future_improvements.md`. The constitution's worker-path
+  drift (C5) was already amended to 1.3.1 during analysis remediation.
+- GUI: 5 components plus `lib/reactors.ts`.
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+| Principle | Status | Note |
+| --- | --- | --- |
+| I. Gem-First Design | PASS | Everything is in `lib/` behind the existing DSL. The new workers live in the existing adapter modules; the Sidekiq and ActiveJob routers both gain the method. No host coupling. |
+| II. Saga Pattern Integrity | PASS (the feature strengthens it) | Large-map rollback stops being bounded by one job's time and memory. Step-level reverse order is kept (I-4). Every hand-off has a recovery path: the sweepers for lost jobs and resumes (S-5), and resume from the last saved undo after a crash (R-07). Fixes a hang that left composed runs with no recovery path. |
+| III. Test-First with Real Infrastructure | PASS with one justified deviation | Each story's specs are written first and fail first, against real Redis, including the FR-011 log-shape spec (T020(g)). Async orchestration uses fake mode plus drain. There is one new `inline!` spec (T041), justified in Complexity Tracking. Existing specs that already use `inline!` gain no new `inline!` blocks. |
+| IV. Observability by Default | PASS | `rolling_back` appears in the state model, the scan, and every dashboard status surface (badge, status groups, live and per-class filters, detail colour). Map rollback progress (`total/settled/outstanding/failed`) is shown in the API, and in the GUI while `rolling_back`. Structured key=value log lines cover rollback start, element and completion. Element undo middleware events keep firing. Rollback failures keep `map_step` / `element_index`, and the symbol values are restored. |
+| V. Simplicity & SemVer | PASS with notes | See Complexity Tracking for the two mechanisms added. Code is deleted: `Helpers#resume_parent_execution` and its helpers, and the collector's failure branch. No config knob for the default batch size. `fail_fast` is aliased, not removed. MINOR (R-18), with CHANGELOG notes for the default batch size, `rolling_back`, the extra job hop and the deprecation. |
+| VI. Demo-App Proof of Feature | PASS (planned) | One reactor per file, named after its class, so a fresh Sidekiq process autoloads an element class by name (`eager_load = false`). Demos: `DistributedRefundDemoReactor` (fan-out, a later failure, distributed rollback with progress), `ComposedFanOutDemoReactor` (compose → fan-out, happy and failure paths) and `DefaultBatchFanOutDemoReactor` (default batch size), each with its element and child reactors in their own files. Rake `demo:distributed_map_rollback`, `demo:composed_fan_out` and `demo:default_batch_size`, grouped as `demo:map_execution_undo` in `demo:all`, plus `demo:map_rollback_benchmark`. Matcher-only specs; `be_rolling_back` and the ActiveJob `PendingJob#worker_class` alias are added to `lib/ruby_reactor/rspec/`. `ar_map_reactor_not_fail.rb` migrates to `atomic false`. No new Docker service. |
+
+- [x] Documentation impact identified (research R-17):
+  - **README.md**: map section (~880–930), statuses, Durability.
+  - **documentation/**: `data_pipelines.md`, `background_and_async.md`, `composition.md`,
+    `core_concepts.md`, `testing.md`.
+  - **demo_app/documentation/data_pipelines.md**: kept in sync.
+  - **CHANGELOG.md**.
+  - **specs/future_improvements.md**: remove the fixed items.
+  - Carried into tasks.md as a required task per user story.
+
+**Post-design re-check (after Phase 1)**: PASS.
+
+- New public surface:
+  - one DSL method (`atomic`);
+  - one status;
+  - two router methods with their worker per adapter;
+  - one matcher and one test-helper alias;
+  - the dashboard `rollback` field;
+  - the `cancel` / `undo` guard on `rolling_back`.
+- New internals: one control signal and one per-element rollback class, shared by the job and the
+  inline path.
+- The complexity is justified below.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/009-distributed-map-undo/
+├── plan.md                        # This file
+├── research.md                    # Phase 0: R-01..R-18
+├── data-model.md                  # Phase 1: records, statuses, outcomes
+├── quickstart.md                  # Phase 1: validation guide
+├── contracts/
+│   ├── api-surface.md             # DSL, statuses, undo, routers/workers, dashboard, logs, matcher
+│   └── rollback-protocol.md       # Invariants I-1..I-8, sequences S-1..S-6
+├── checklists/requirements.md
+└── tasks.md                       # Phase 2 (/speckit-tasks)
+```
+
+### Source Code (repository root)
+
+```text
+lib/ruby_reactor/
+├── error/rollback_handed_off.rb        # new: control signal carrying map_id (R-04)
+├── error/rescuable.rb                  # === false for RollbackHandedOff (R-04)
+├── map.rb                              # new Zeitwerk namespace file: DEFAULT_BATCH_SIZE = 50,
+│                                       #   ROLLBACK_CHUNK = 100 (R-10, R-01)
+├── map/element_rollback.rb             # new: roll back one element (lock with requeue on contention,
+│                                       #   per-entry checkpoint, outcome, handed_off-gated owner signal);
+│                                       #   used by the job and the inline path (R-05, R-07)
+├── map/collector.rb                    # settled → SET NX owner_signalled → enqueue owner Worker;
+│                                       #   no context writes; legacy fallback to parent ids (R-03, S-6)
+├── map/helpers.rb                      # delete resume_parent_execution & co.; add normalize_arguments
+│                                       #   (fail_fast → atomic) (R-03, R-11)
+├── map/dispatcher.rb                   # default batch size; atomic; rollback batch dispatch
+│                                       #   (claim positions, LRANGE, enqueue) (R-02, R-06, R-10)
+├── map/element_executor.rb             # register the id only for a fresh context; atomic (R-06, R-11)
+├── map/sweeper.rb                      # rollback pass (missing positions, lost batch trigger);
+│                                       #   owner lock for recollect (R-05, S-5)
+├── step/map_step.rb                    # run: adopt on re-entry; owner ids + batch_size +
+│                                       #   atomic in metadata; store the owner tree before dispatch.
+│                                       #   compensate/undo: distributed start / settle / aggregate /
+│                                       #   hand-off; inline chunked; aggregation dedups unavailable
+│                                       #   entries and re-symbolizes. rollback_arguments → {} (R-01..R-10, R-14)
+├── step/compose_step.rb                # none expected; covered by the specs (S-3)
+├── executor.rb                         # rescue RollbackHandedOff in execute/resume_execution; handed_off
+│                                       #   handshake; undo_all records/restores its level's failures; new
+│                                       #   resume_rollback; save before releasing the context lock (R-04, R-05, R-13)
+├── executor/compensation_manager.rb    # track the pending failing step + compensated flag; per-entry
+│                                       #   callback on rollback_completed_steps (R-04, R-07)
+├── executor/result_handler.rb          # undo entry uses step_config.rollback_arguments (R-14)
+├── executor/retry_manager.rb           # atomic in element requeue args (R-11)
+├── dsl/map_builder.rb                  # atomic; fail_fast alias + warning; both → error (R-11)
+├── dsl/definition_warnings.rb          # new: warn_definition extracted from StepBuilder (R-11)
+├── dsl/step_builder.rb                 # include DefinitionWarnings; StepConfig#rollback_arguments (R-11, R-14)
+├── context.rb / context_serializer.rb  # `rollback` field (R-04)
+├── reactor.rb                          # undo: context lock, reject rolling_back, hand-off → handshake,
+│                                       #   skip cancel; cancel rejects rolling_back (R-12, FR-027)
+├── worker.rb                           # rolling_back → resume_rollback; non-terminal (R-04, R-15)
+├── sweeper.rb                          # re-enqueue rolling_back without an async: lock (R-05)
+├── storage/redis_adapter.rb            # metadata fields; owner signal; rollback records; LRANGE by range
+├── storage/redis_reactor_scan.rb       # rolling_back known status (R-15)
+├── adapters/{sidekiq,active_job}/router.rb                       # perform_map_element_rollback_async / _in
+├── adapters/{sidekiq,active_job}/map_element_rollback_worker.rb  # new
+├── web/api.rb                          # map_ref hydration: rollback {total, settled, outstanding, failed} (R-15)
+└── rspec/{matchers,test_subject,sidekiq_helpers,active_job_helpers}.rb
+                                        # be_rolling_back; drain on rolling_back; register rollback worker;
+                                        #   PendingJob#worker_class alias (R-16)
+
+gui/src/lib/reactors.ts                              # STATUS_GROUPS.running += rolling_back
+gui/src/components/StatusBadge.tsx                   # amber + undo icon for rolling_back
+gui/src/components/{LiveView,ReactorClassInstances}.tsx  # rolling_back filter option
+gui/src/components/ReactorDetail.tsx                 # rolling_back colour
+gui/src/components/StepInspector.tsx                 # rollback progress while rolling_back
+lib/ruby_reactor/web/public/                         # rebuilt bundle
+
+spec/
+├── support/{map_rollback_fixtures,queue_probe}.rb           # new helpers
+├── ruby_reactor/executor/context_lock_save_order_spec.rb    # new: R-13 regression
+├── ruby_reactor/executor/resume_rollback_spec.rb            # new: hand-off + resume (stub construct)
+├── ruby_reactor/rollback/distributed_map_rollback_spec.rb   # new: US1 incl. oracle, back pressure, undo,
+│                                                            #   cancel guard, log shape, inline mode (T041)
+├── ruby_reactor/rollback/map_rollback_recovery_spec.rb      # new: fault injection, contention requeue, sweepers
+├── ruby_reactor/rspec/be_rolling_back_spec.rb               # new
+├── map/map_owner_resume_spec.rb                             # new: R-03 at root level
+├── map/map_legacy_payload_spec.rb                           # new: fail_fast payloads (FR-023)
+├── map/map_compose_fan_out_spec.rb                          # new: US2 incl. rollback through the root
+├── map/map_atomic_spec.rb                              # new: US4
+├── ruby_reactor/rollback/map_rollback_spec.rb               # updated: both paths, chunked inline, nested map
+├── ruby_reactor/rollback/map_scale_spec.rb                  # updated: one element per job
+├── ruby_reactor/map/{sweeper,dispatcher}_spec.rb            # updated: rollback pass, owner lock, rollback batches
+├── ruby_reactor/storage/redis_adapter_spec.rb               # updated: metadata, owner signal, rollback records
+├── ruby_reactor/sweeper_spec.rb                             # updated: rolling_back
+├── map/map_batch_size_spec.rb                               # updated: default 50, metadata
+├── map/map_recovery_spec.rb                                 # updated: re-dispatch keeps batch_size/atomic
+└── map/{fail_fast→atomic_inline, map_fail_fast→map_atomic_async, map_retry, map_async_retry}_spec.rb
+
+demo_app/
+├── app/reactors/distributed_refund_demo_reactor.rb          # one reactor per file (Constitution VI.1)
+├── app/reactors/distributed_refund_element_reactor.rb       #   + its charge step class
+├── app/reactors/inline_refund_benchmark_reactor.rb          # same element, fan_out false (SC-002 baseline)
+├── app/reactors/composed_fan_out_demo_reactor.rb
+├── app/reactors/composed_fan_out_child_reactor.rb
+├── app/reactors/composed_fan_out_item_reactor.rb
+├── app/reactors/default_batch_fan_out_demo_reactor.rb
+├── app/reactors/ar_map_reactor_not_fail.rb                  # fail_fast false → atomic false
+├── lib/tasks/demo_reactors.rake   # demo:distributed_map_rollback, :composed_fan_out, :default_batch_size,
+│                                  #   :map_execution_undo (in demo:all), :map_rollback_benchmark
+└── spec/reactors/{distributed_refund,composed_fan_out,default_batch_fan_out}_demo_reactor_spec.rb
+
+README.md, documentation/*.md, demo_app/documentation/data_pipelines.md, CHANGELOG.md,
+specs/future_improvements.md   # per R-17
+.specify/memory/constitution.md  # amended to 1.3.1 during analysis remediation (C5): worker path
+```
+
+**Structure Decision**: single gem project, existing `lib/ruby_reactor` layout. No new directories
+under `lib/`. The new rollback specs sit next to 008's in `spec/ruby_reactor/rollback/`, and the
+map DSL/dispatch specs in `spec/map/`.
+
+## Delivery Order
+
+The stories are independently testable. tasks.md is the authoritative order. The owner-resume
+mechanism is foundational, because a failed fan-out map's rollback must run through the executor
+(not the collector) before US1 can hand it off:
+
+1. **Foundation** (blocks every story):
+   - map metadata owner ids, `batch_size`, `atomic` (R-10);
+   - `Map::DEFAULT_BATCH_SIZE`;
+   - save before releasing the context lock (R-13), with its regression spec;
+   - `StepConfig#rollback_arguments` (R-14);
+   - `normalize_arguments` (R-11 internals);
+   - **owner resume + adopt at root level (R-03)**: the collector rewrite and `MapStep#run`
+     re-entry, deleting `resume_parent_execution`.
+2. **US1: distributed rollback**:
+   - a. `RollbackHandedOff`, `Rescuable` exclusion, the `rollback` context field, `rolling_back`
+     status, executor rescue plus `resume_rollback`, `Worker` branch. Spec: a stub construct that
+     hands off once, resumed by `Worker`.
+   - b. `Map::ElementRollback`, with per-entry checkpoints (R-07).
+   - c. Storage rollback records and dispatcher rollback batches; router methods and workers.
+   - d. `MapStep#compensate` / `#undo`, both distributed and inline chunked. Oracle and
+     back-pressure specs.
+   - e. `Reactor#undo` lock, hand-off and the `cancel` guard (R-12).
+   - f. Handshake-gated owner signal and contention requeue (R-05, R-07).
+   - g. Recovery: the sweepers (S-5) and fault-injection specs.
+   - h. Observability: logs, API field, every GUI status surface, matcher.
+3. **US2: composed child**:
+   - store the owner tree before dispatch;
+   - compose specs (happy, interrupt, nested, rollback through the root, manual undo from the root);
+   - child `undo_all` hand-off (S-3; needs US1 for the distributed variant).
+4. **US3: default batch size**:
+   - the fallback in `MapStep` and `Dispatcher`;
+   - specs for 500, 20 and explicit elements.
+5. **US4: `atomic`**:
+   - the DSL, the warning, the both-declared error;
+   - migrate internal specs and the demo;
+   - a legacy payload spec.
+6. **Demo, docs, CHANGELOG** per story, closing with the Docker acceptance run and the SC-002
+   benchmark.
+
+## Complexity Tracking
+
+| Addition | Why needed | Simpler alternative rejected because |
+| --- | --- | --- |
+| Resumable rollback (`RollbackHandedOff`, context `rollback` state, `rolling_back`, `Executor#resume_rollback`) | FR-001 and FR-004: element rollbacks run in other jobs, and the steps before the map must wait for them without holding a worker. | **Waiting in place** holds a worker for the whole rollback, hits job timeouts, and turns a shutdown into `aborted`. **A sentinel return value** needs plumbing through six call sites. **Snooze-polling** reloads the parent blob on every poll. The undo stack already gives the resume cursor, so the addition is the signal plus about five persisted fields (R-04). |
+| Owner resume + adopt-on-re-entry (collector signals the owner; `MapStep#run` adopts) | FR-013 to FR-015 (the composed-child hang), and the same resume path US1 needs. | **Collector writes the root** breaks the single-writer rule. **Special-casing composed children** leaves two completion paths and the collector's failure-branch write. This change deletes more than it adds (R-03). |
+| Map rollback records (6 keys, including `handed_off`) | Per-position idempotent outcomes, atomic batch claims, an owner signal that is exactly-once and never early (the handshake), and naming unavailable indexes at bounded memory. | **Reusing the forward results hash** would mix forward and rollback outcomes for the same index, and the forward slots are still read by the `ResultEnumerator` of a map that completed. **Relying on the `async:` lock alone** to order the owner resume fails in inline job mode, where the lock is skipped and the owner would run nested on a stale blob. |
+| One new `Sidekiq::Testing.inline!` spec (T041), a deviation from Constitution III | The subject is inline-mode support itself: R-05's claim that a map rollback settling synchronously never hands off and never enqueues a nested owner resume. It is a spec edge case (Edge Cases, "Inline job-testing mode"). | **Fake mode plus drain** cannot express it: jobs never run inside dispatch there, so the path under test never executes. It is scoped to one example. The existing `inline!` specs gain no new `inline!` blocks. |

data/specs/009-distributed-map-undo/pr-description.md

@@ -0,0 +1,78 @@
+# PR description draft: 009 Distributed Map Rollback and Bounded Fan-out
+
+## Regression bar (T003, before any change, at `c9711cc1`)
+
+- `bundle exec rspec spec/map spec/ruby_reactor/rollback spec/ruby_reactor/map spec/compose_spec.rb spec/single_worker_map_spec.rb`:
+  **192 examples, 0 failures**.
+- `bundle exec ruby specs/007-execution-flow-analysis/evidence/run.rb | tail -1`:
+  **64 scenarios, 64 match, 0 mismatch**.
+
+## `RollbackHandedOff` rescue audit (T078)
+
+`RollbackHandedOff` is an `Error::Base` (a `StandardError`), so every broad rescue between
+`MapStep#compensate`/`#undo` and the handlers (`Executor`, `Reactor#undo`) was checked:
+
+- `rescue Error::Rescuable` in `CompensationManager#compensate_step_body` / `#undo_step`,
+  `StepExecutor#safe_execute_step_sync`, `StepCoordination`, `StepBuilder#resolve_arguments`,
+  `MapStep#collect_results`: pass it through, since `Rescuable.===` excludes it.
+- `Executor#aborting_on_interruption`: re-raises it without marking the run `aborted`.
+- `StepExecutor#execute_step`'s `rescue Exception`: emits its step event and re-raises;
+  `track_interrupted_construct` ignores it.
+- `rescue StandardError` in `StepCoordination` (lock/semaphore release, ordered-lock advance,
+  `chain_failed?`), `OrderedLockSupport`, `MiddlewareRunner#on`, `Executor#publish_completion_signal`
+  / `#release_one`, `Worker` (retries-exhausted and deserialization helpers), `Reactor` reloads and
+  `Map::Sweeper`: none wraps a rollback call, so none can see it.
+
+No explicit re-raise was needed.
+
+## Results
+
+- Gem suite: `bundle exec rspec` → **1479 examples, 0 failures, 2 pending** (the pending pair is
+  US2-AS2, an interrupt inside a composed child, unsupported on main; see research R-03).
+  `bundle exec rspec --tag slow` → 2 examples, 0 failures (10,000-element inline and fan-out
+  rollbacks).
+- T003 regression set: 192 → now part of the above, all green.
+- 007 evidence harness: **64 scenarios, 64 match**. S-map-12's expected order changed (009 R-08:
+  element undos run in their own jobs, after the elements' independent `async_step` units);
+  the harness drains `MapElementRollbackWorker` too.
+- `bundle exec rubocop`: no offenses.
+- GUI: `npm test` 68 passed; bundle rebuilt into `lib/ruby_reactor/web/public/`.
+- Demo specs (`REDIS_URL=redis://localhost:6780/5`): 145 examples, 0 failures.
+- Docker acceptance (isolated project `rr_009`, real Sidekiq): `demo:map_execution_undo`
+  prints SUCCESS for all four scenarios (distributed refund with a `rolling_back` progress line,
+  composed fan-out happy and failure paths, default batch size 50 with 120 collected).
+
+## Constitution re-check (T086)
+
+- Docs (R-17): README, `data_pipelines.md`, `background_and_async.md`, `composition.md`,
+  `core_concepts.md`, `testing.md`, `demo_app/documentation/data_pipelines.md`,
+  `specs/future_improvements.md`.
+- CHANGELOG: Features (default batch size, distributed rollback, `rolling_back`, router methods,
+  matcher, undo lock), Bug Fixes (composed fan-out, owner resume, sweeper re-dispatch, save before
+  release, undo record, `exception_class`), Deprecations (`fail_fast` → `atomic`).
+- Demo per story, one reactor per file: US1 `DistributedRefundDemoReactor` +
+  `DistributedRefundElementReactor` (+ `InlineRefundBenchmarkReactor`), US2
+  `ComposedFanOutDemoReactor` + `ComposedFanOutChildReactor` + `ComposedFanOutItemReactor`, US3
+  `DefaultBatchFanOutDemoReactor` + `DefaultBatchNumberReactor`, US4 `ar_map_reactor_not_fail.rb`
+  on `atomic false`. Rake: `demo:distributed_map_rollback`, `demo:composed_fan_out`,
+  `demo:default_batch_size`, grouped as `demo:map_execution_undo` (in `demo:all`), plus
+  `demo:map_rollback_benchmark`. Specs use the shipped surface (`be_rolling_back` added).
+- `git diff main -- spec | grep '^+.*inline!'`: none besides T041's
+  `Sidekiq::Testing.inline! { reactor.run(items) }` in `distributed_map_rollback_spec.rb`.
+
+## SC-002 benchmark (T084) — not demonstrated
+
+Run in the isolated `rr_009` stack, `demo-sidekiq -c 10`, `demo:map_rollback_benchmark[10000]`
+(each side timed from `:notify`'s failure to the end of the run):
+
+```
+serial (inline map):                  14.57s, 17286 refunds
+distributed (fan-out, batch_size 10): 118.60s, 22714 refunds, failed
+ratio: 0.1x
+```
+
+Invalid: both sides refunded more than 10,000 elements. `demo:flush_redis` restarts the recovery
+sweeper, which (a) re-enqueued the in-process inline run into a worker (known follow-up "Sweeper
+re-enqueues in-progress inline runs"), and (b) re-dispatched not-yet-thrown element jobs of the
+fan-out map (review finding F1), so elements ran and were refunded twice and the distributed run
+shared Sidekiq with that duplicate work. Rerun after fixing F1, with the sweeper disabled.

data/specs/009-distributed-map-undo/quickstart.md

@@ -0,0 +1,154 @@
+# Quickstart: Validate Distributed Map Rollback and Bounded Fan-out
+
+A guide to proving each user story end to end. The behavior is specified in
+[contracts/](contracts/), and the records in [data-model.md](data-model.md).
+
+## 0. Prerequisites
+
+- Ruby >= 3.0 and `bundle install`.
+- Test Redis reachable at `redis://localhost:6780`. The suite refuses to start without it.
+- Docker, for the demo acceptance runs and the SC-002 benchmark only.
+- If other suites share the test Redis, run the new specs alone first (known flake under load).
+
+## 1. User Story 1: distributed rollback (P1)
+
+```bash
+bundle exec rspec spec/ruby_reactor/rollback/distributed_map_rollback_spec.rb
+bundle exec rspec spec/ruby_reactor/rollback/map_rollback_recovery_spec.rb
+bundle exec rspec spec/ruby_reactor/rollback/map_rollback_spec.rb     # 008 coverage, now via both paths
+```
+
+Expected:
+
+| Scenario | Observable outcome |
+| --- | --- |
+| Fan-out map of 20 elements, `batch_size 5`, a later step fails. This is the spec's 1,000/50 scenario scaled down; §1 Scale covers 10,000/50. | Mid-drain (once rollback jobs are pending): `be_rolling_back`. While draining: no job enqueues more than 5 rollback jobs (`max_burst`); queue depth is not bounded (R-02). After draining: `be_failure`; every element undone once; the steps before the map undone after the last element. |
+| Same reactor with an inline map (oracle, I-7) | Same final `Failure` message, step and `rollback_failures` set as the fan-out run. |
+| Element 7 fails under atomic | Every started element gets a rollback job. Element 7's job reports `not_needed` and undoes nothing. Skipped indexes get no job. |
+| An element's lock is held while its rollback job runs | The job requeues itself, then completes once the lock is free, with no `element_in_flight` failure. After `lock_snooze_max_attempts` requeues it reports `element_in_flight`. |
+| An element context expired (row deleted) | Exactly one `context_unavailable` entry for that index, as with the inline map. |
+| An element reactor containing an inline (nested) map | The nested map is rolled back inline inside the outer element's rollback job; the log shows nested undos before the outer element's earlier steps. |
+| One element's undo raises | The run still undoes the steps before the map. `rollback_failures` holds that element's entry with `map_step` and `element_index`. |
+| Interruption raised in an element's second undo, then the job is redelivered | The first undo ran once. The second may run twice. The run finishes its rollback. |
+| Rollback job dropped, then `Map::Sweeper.run_once` | The position is re-dispatched and the run finishes. |
+| Owner resume dropped, then `RubyReactor::Sweeper.run_once` | The owner is re-enqueued and the run finishes. |
+| `Reactor.undo(id)` on a completed run with a fan-out map | Returns with the run `rolling_back`; after draining, `cancelled`. A second `Reactor.undo` while rolling back raises `ValidationError`, and so does `Reactor.cancel`. |
+| Inline map rollback | Elements are read 100 at a time, highest index first. Same outcome as before. |
+
+Scale (slow):
+
+```bash
+bundle exec rspec spec/ruby_reactor/rollback/map_scale_spec.rb --tag slow
+```
+
+Expected: 10,000 elements roll back, and each rollback job loads exactly one element context.
+
+## 2. User Story 2: fan-out map inside a composed child (P2)
+
+```bash
+bundle exec rspec spec/map/map_compose_fan_out_spec.rb
+```
+
+Expected (drain all jobs):
+
+- Root composes Child; Child has `map … fan_out batch_size: 1`. The root is `be_success` with the
+  expected result. Before this change it stayed `running`.
+- The same with an interrupt after the map in Child: `be_paused_at(...)`, then `resume`, then
+  `be_success`.
+- A root step after the compose fails: elements rolled back (distributed), then Child's earlier
+  steps, then Root's. The root is `be_failure`.
+- A Child element fails under atomic: the root (not only Child's record) is `be_failure`.
+- Compose nested two levels deep: the top-level root is `be_success`.
+- `Reactor.undo(root id)` on a completed root whose composed child ran a fan-out map: the root is
+  `rolling_back`, then `cancelled`. Every element and every earlier step is undone exactly once.
+- A root step after the compose fails, and the child step after the map has an undo that fails.
+  That failure is recorded before the child hands off, and it still appears in the root's final
+  `rollback_failures` after the resume.
+- The same child step with a failing undo, reached by a manual undo instead: the failure is in the
+  trace, as today for manual undo.
+
+## 3. User Story 3: default batch size (P2)
+
+```bash
+bundle exec rspec spec/map/map_batch_size_spec.rb
+```
+
+Expected:
+
+- `fan_out` with no batch size, 500 elements: no job enqueues more than 50 element jobs
+  (`max_burst`), and all 500 outcomes are collected.
+- 20 elements: all 20 enqueued at once.
+- `batch_size: 200`: 200 per throw.
+- Map metadata stores `batch_size` and `atomic`. `Map::Sweeper` re-dispatch keeps both.
+
+## 4. User Story 4: `atomic` (P3)
+
+```bash
+bundle exec rspec spec/map/map_atomic_spec.rb spec/map/map_atomic_async_spec.rb spec/map/atomic_inline_spec.rb
+```
+
+Expected:
+
+- No declaration: one element failure fails the map and rolls back the completed elements.
+- `atomic false`: the map completes with one `Failure` among the results.
+- `fail_fast false`: the same behavior, plus exactly one stderr line starting
+  `[RubyReactor] DEPRECATION:` that names `atomic`.
+- Both declared: `ValidationError` at class definition.
+- An element job payload carrying `"fail_fast" => true` (legacy) is processed as atomic.
+
+## 5. Demo app acceptance (Constitution VI)
+
+Locally, without Docker (test Redis DB 5; run `bin/rails db:prepare` once first):
+
+```bash
+cd demo_app
+REDIS_URL=redis://localhost:6780/5 RAILS_ENV=test bundle exec rspec \
+  spec/reactors/distributed_refund_demo_reactor_spec.rb \
+  spec/reactors/composed_fan_out_demo_reactor_spec.rb \
+  spec/reactors/default_batch_fan_out_demo_reactor_spec.rb spec/reactors/ar_map_reactor_not_fail_spec.rb
+```
+
+Real Sidekiq, in an isolated compose project. `docker-compose.yml` hard-codes container names, so a
+second worktree needs its own project plus an override that gives unique `container_name`s and
+`ports: !reset []`:
+
+```bash
+docker compose -p rr_009 -f docker-compose.yml -f override.yml up -d --build demo-redis demo-sidekiq
+docker compose -p rr_009 -f docker-compose.yml -f override.yml run --rm --no-deps demo-app \
+  bash -c "bin/rails db:prepare && bin/rails demo:map_execution_undo"
+```
+
+Expected output, in order:
+
+- `DistributedRefundDemoReactor`: dispatch, then a `rolling_back` line with progress, then
+  `failed`, with refunds equal to charges.
+- `ComposedFanOutDemoReactor`: `completed` on the happy path, and `failed` with everything
+  unshipped on the failure path.
+- `DefaultBatchFanOutDemoReactor`: map metadata `batch_size 50`, and 120 elements collected.
+
+### SC-002 benchmark (manual)
+
+Run `demo-sidekiq` with `command: bundle exec sidekiq -c 10` in the override, then:
+
+```bash
+docker compose -p rr_009 -f docker-compose.yml -f override.yml run --rm --no-deps demo-app \
+  bin/rails "demo:map_rollback_benchmark[10000]"
+```
+
+The task prints two times and their ratio, each timed from the moment `:notify` fails (the
+rollback's start) to the end of the run, so the forward run is not counted:
+
+- the serial baseline: the inline-map rollback of the same 10,000 elements, through
+  `InlineRefundBenchmarkReactor`, which is today's serial algorithm;
+- the distributed rollback time.
+
+Pass: the distributed time is 5× faster or better (SC-002).
+
+## 6. Full checks before PR
+
+```bash
+bundle exec rspec
+bundle exec rubocop
+cd gui && npm test && npm run build   # rolling_back in badge/groups/filters + map rollback progress
+grep -rn "fail_fast" lib gui/src documentation README.md demo_app/app demo_app/documentation   # only the deprecated alias, its warning, and legacy normalization remain
+```