ruby_reactor 0.8.4 → 0.8.6

data/specs/009-distributed-map-undo/checklists/requirements.md

@@ -0,0 +1,53 @@
+# Specification Quality Checklist: Distributed Map Rollback and Bounded Fan-out
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-09-30
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- The product is a developer library, so its stakeholders are developers: DSL names (`fan_out`,
+  `batch_size`, `fail_fast`, `atomic`) are the user-facing surface and are kept on purpose.
+  The spec names no internal classes, storage keys or job backends.
+- No clarification markers. Defaults taken instead, recorded in Assumptions:
+  - `fail_fast` becomes a deprecated alias, not removed (MINOR release);
+  - 50 is a fixed library default with no global setting (per-map `batch_size` overrides it);
+  - rollback mirrors execution mode (inline maps stay in process, with bounded reads);
+  - no ordering guarantee across elements' rollbacks.
+- FR-025/FR-026 (docs, CHANGELOG, demo app) are there because the constitution requires them for
+  every feature, not as implementation detail.
+- 2026-09-30, after /speckit-plan: FR-002, FR-006 and FR-008 and SC-001, SC-003 and SC-004
+  reworded to match research R-02 (the back-pressure bound per throw), R-07 (the at-least-once
+  window for the undo that was cut off) and R-08 (start order). Re-validated: all items still pass.
+- 2026-09-30, after /speckit-analyze. All 27 findings were addressed:
+  - spec: US1-AS2, FR-002, FR-003, FR-007, FR-009, FR-018, SC-001, SC-002 and SC-004, the Edge
+    Cases and Key Entities were revised, and FR-027 (a single rollback per run) was added;
+  - plan, research, data-model, contracts, quickstart and tasks were updated to match;
+  - the constitution was amended to 1.3.1 (the worker path).
+
+  Re-validated: all items still pass, and there are no clarification markers.

data/specs/009-distributed-map-undo/contracts/api-surface.md

@@ -0,0 +1,125 @@
+# Contract: Public API Surface
+
+What users of the gem see change. Internal classes are listed only where a custom setup (router,
+worker registration, dashboards) touches them.
+
+## Map DSL
+
+```ruby
+map :charge_orders, ChargeOrderReactor do
+  source result(:orders)
+  argument :order, element(:charge_orders)
+
+  fan_out batch_size: 200   # batch_size optional; 50 when omitted
+  atomic               # default; `atomic false` returns per-element outcomes
+  collect { |results| ... }
+end
+```
+
+| Declaration | Behavior | Errors / warnings |
+| --- | --- | --- |
+| `atomic` / `atomic true` | Default. Any element failure fails the map, no new element starts, and every completed element is rolled back. | — |
+| `atomic false` | The map completes with every element's outcome (a `ResultEnumerator` of `Success` / `Failure`). Failed elements roll themselves back; succeeded ones are kept. | — |
+| `fail_fast(x)` | Same as `atomic(x)`. | One line on stderr per call site: `[RubyReactor] DEPRECATION: <file:line> <Reactor> map :<name> declares fail_fast. Use atomic (same meaning). Removal no earlier than the next MAJOR.` |
+| `fail_fast` and `atomic` on one map | — | `RubyReactor::Error::ValidationError` at class definition: "map :<name> declares both fail_fast and atomic; declare only atomic". |
+| `fan_out` (no `batch_size`) | No throw enqueues more than 50 element jobs; the next throw fires when the previous throw's last element finishes. | — |
+| `fan_out batch_size: n` | As today; `n` must be a positive Integer. | `ValidationError` as today. |
+| `batch_size n` on an inline map | No effect, as today. | — |
+
+## Rollback behavior (observable)
+
+| Situation | Before | After |
+| --- | --- | --- |
+| A fan-out map fails (atomic), or a later step fails, or `Reactor.undo` reaches a completed fan-out map | Every completed element is undone serially, in one process, with all element states loaded at once. | One background job per started element; only completed elements undo anything, the others report `not_needed`. No throw enqueues more than `batch_size`. The run is `rolling_back` until every element has reported, then the steps before the map are undone and the run ends `failed` (or `cancelled` for `undo`). |
+| An inline map is rolled back | In process, all element states loaded at once. | In process, element states read 100 at a time, highest index first. |
+| The final `Failure` | Carries `rollback_failures` with `map_step` / `element_index`. | Identical shape and content. The oracle is the same run with an inline map. |
+| A worker dies mid-element-rollback | Not applicable (inline). | The element resumes after its last recorded undo. The undo that was cut off may run again, so undos must stay idempotent, as the docs already require. |
+
+## Statuses
+
+| Status | New? | Terminal? | Meaning |
+| --- | --- | --- | --- |
+| `rolling_back` | **new** | no | A rollback handed off at a fan-out map and is waiting for element rollback jobs. |
+
+Dashboards and code that filter status strings see this one new value.
+
+## `Reactor.undo(id)` / `Reactor#undo` / `Reactor.cancel`
+
+| Case | Behavior |
+| --- | --- |
+| No fan-out map on the undo path | As today: undone in the caller's process, then `cancel(reason: "Undo triggered")`. |
+| A completed fan-out map on the undo path | The steps after the map are undone in the caller's process. Element rollbacks are dispatched, the run becomes `rolling_back`, and the call returns. A worker finishes the rollback and applies `cancelled`. |
+| The run's `async:` lock is held (a live run or rollback) | Raises `RubyReactor::Lock::AcquisitionError` after waiting up to 5 s. The caller retries. |
+| The run is already `rolling_back` | Raises `RubyReactor::Error::ValidationError`: "rollback already in progress" (FR-027). |
+| `Reactor.cancel(id:, reason:)` on a `rolling_back` run | Raises `RubyReactor::Error::ValidationError`: "rollback in progress; cannot cancel" (FR-027). Cancel on any other status behaves as today. |
+
+## Composition
+
+A `compose`d child (at any depth) whose steps include a `fan_out` map now works:
+
+- the top-level run is resumed when the map settles, and finishes;
+- rollback travels through the top-level run.
+
+Nothing to declare.
+
+## Background workers and routers
+
+Both shipped routers gain two methods, and each adapter gains one worker class:
+
+| Adapter | Router methods | Worker class |
+| --- | --- | --- |
+| Sidekiq | `perform_map_element_rollback_async(**args)`, `perform_map_element_rollback_in(delay, **args)` | `RubyReactor::Adapters::Sidekiq::MapElementRollbackWorker` |
+| ActiveJob | `perform_map_element_rollback_async(**args)`, `perform_map_element_rollback_in(delay, **args)` | `RubyReactor::Adapters::ActiveJob::MapElementRollbackWorker` |
+
+`perform_map_element_rollback_in` is the requeue a job uses when the element's lock is contended
+(R-07).
+
+Args (JSON-safe):
+
+- `map_id`, `position`, `element_context_id`, `reactor_class_info`;
+- `parent_reactor_class_name`, `step_name`, `batch_size`;
+- `owner_context_id`, `owner_reactor_class_name`;
+- `attempt`: 0 on dispatch, incremented on each contended-lock requeue.
+
+The worker runs on the same queue as `MapElementWorker`.
+
+`MapCollectorWorker` keeps its class and arguments. A collector job enqueued before the upgrade is
+handled by the new body: it signals the owner instead of resuming the parent.
+
+## Dashboard API
+
+`GET /api/reactors/:id`: a hydrated map reference whose map has rollback records gains:
+
+```json
+{ "rollback": { "total": 1000, "settled": 350, "outstanding": 650, "failed": 2 } }
+```
+
+`outstanding = total - settled`. The records outlive the run by the durability TTL, so the field is
+present after the rollback finishes too; clients decide from the run's status.
+
+GUI:
+
+- `rolling_back` appears in amber with an undo icon everywhere statuses are shown or filtered: the
+  status badge, the "running" status group, and the live and per-class filters.
+- The step inspector shows "Rolling back 350/1000 (650 outstanding, 2 failed)" **only while the
+  run is `rolling_back`**.
+
+## Structured logs
+
+key=value, one line each, through `RubyReactor.configuration.logger`:
+
+```text
+event=ruby_reactor.map.rollback.started   reactor=<owner class> context_id=<owner id> map_step=<step> total=<n> batch_size=<b>
+event=ruby_reactor.map.rollback.element   reactor=<owner class> context_id=<owner id> map_step=<step> index=<i> outcome=<outcome> failures=<count>
+event=ruby_reactor.map.rollback.completed reactor=<owner class> context_id=<owner id> map_step=<step> total=<n> failed=<count>
+```
+
+## RSpec test surface
+
+| Addition | Use |
+| --- | --- |
+| `be_rolling_back` matcher | `expect(subject).to be_rolling_back`, for a run handed off at a fan-out map and not drained yet. |
+| `PendingJob#worker_class` on the ActiveJob helper | Alias of `job_class`, so `pending_async_jobs.first.worker_class` works on both backends for step-wise draining. |
+
+`drain_async_jobs` drains `MapElementRollbackWorker` like any other adapter worker. There is no API
+change.

data/specs/009-distributed-map-undo/contracts/rollback-protocol.md

@@ -0,0 +1,157 @@
+# Contract: Map Completion and Distributed Rollback Protocol
+
+The sequences the implementation must follow, and the invariants the specs pin. The record names
+are defined in [data-model.md](../data-model.md).
+
+## Invariants
+
+- **I-1: single writer.** A context is written only by the execution that owns it: the top-level
+  run's worker holding `async:<owner id>`, or the caller's process during `Reactor#undo`, which
+  holds the same lock. The collector and element rollback jobs write only map records and element
+  contexts.
+- **I-2: the owner resumes once per signal, and never before its hand-off is saved.** Each settle
+  signal (`owner_signalled`, `rollback:signalled`) enqueues the owner's `Worker` at most once. A
+  rollback signal also requires `rollback:handed_off`, which the handler sets after saving (R-05).
+  Extra resumes from the sweepers are harmless because every construct adopts idempotently on
+  re-entry.
+- **I-3: no double dispatch.** A map dispatches its elements once (`map_operations[step]` is set
+  before dispatch), and starts its rollback once (`rollback:metadata` is created with `HSETNX`).
+- **I-4: step-level saga order.** Steps before a map are undone only after its rollback has
+  settled. Within an element, steps are undone in reverse order.
+- **I-5: resume cursor.** An undo entry leaves the stack only after its undo returns. A hand-off
+  leaves the map entry, and every entry below it, on the stack.
+- **I-6: save before release.** Every hand-off (a forward `DispatchResult` or a rollback hand-off)
+  persists the context before the owner's `async:` lock is released.
+- **I-7: oracle.** For the same reactor and failure, the final `Failure` and the set of undone
+  elements are the same whether the map is inline or fan-out. That includes `rollback_failures`
+  entries with symbol values, and each unavailable element reported once.
+- **I-8: no terminal status mid-rollback.** A `rolling_back` run cannot be cancelled or undone
+  again (FR-027). Only `resume_rollback` moves it to `failed` / `cancelled`.
+
+## S-1: A fan-out map completes (any composition depth)
+
+```text
+owner worker (holds async:<owner>)           element jobs            collector
+─────────────────────────────────           ────────────            ─────────
+MapStep#run (first time)
+  map_operations[step] = map_id
+  store owner tree + child snapshot
+  initialize map metadata (owner ids, batch_size, atomic)
+  Dispatcher: enqueue batch 0 (≤ B)
+  return DispatchResult ── executor saves, then releases async: (I-6)
+                                              run, store result slot,
+                                              position kB-1 → next batch,
+                                              counter hits 0 → enqueue collector
+                                                                     all slots present?
+                                                                     SET NX owner_signalled
+                                                                     enqueue Worker(owner)
+Worker(owner) takes async:<owner>
+  resume_execution → … → ComposeStep#run → child resume_execution
+  MapStep#run (re-entry: map_operations[step] present)
+    settled + failed_context_id → return the element's Failure (+ its rollback_failures)
+    settled                     → return Success(collect(ResultEnumerator)); record started
+    not settled                 → return DispatchResult (no dispatch)
+  the executor continues: result recorded, undo entry {step, arguments: {}} (R-14)
+```
+
+## S-2: Rollback reaches a completed fan-out map
+
+A later step failed, the map itself failed (atomic), or `Reactor#undo` was called.
+
+```text
+owner (worker, or caller during undo; holds async:<owner>)
+  CompensationManager: compensate the failing step, then pop entries newest first …
+  MapStep#undo (map_operations[step] present)
+    HSETNX rollback:metadata (total = LLEN element_contexts, batch_size, owner ids)
+      created → log rollback.started; claim batch 0 (INCRBY offset B); enqueue ≤ B element rollback jobs
+      existed → dispatch nothing (I-3)
+    settled?  (HLEN results == total)
+      yes → aggregate (HSCAN failures + started-index check) → return Success / Failure(rollback_failures)
+      no  → raise RollbackHandedOff
+  ▲ unwinds without popping (I-5), through Rescuable sites untouched
+  composed child executor / child Executor#undo_all: record child `rollback` (incl. its failures so far), re-raise
+  top-level executor:      record `rollback`, status rolling_back, SAVE (in rescue, lock held),
+                           SET rollback:handed_off, then: settled && SET NX rollback:signalled → enqueue Worker(owner);
+                           release
+  Reactor#undo:            same, and skip `cancel`
+
+element rollback job (position p, attempt a)
+  load element context (by id from the job args)
+    missing → outcome context_unavailable (index null)
+  take map_element:<map>:<index> (wait 2 s)
+    held and a < lock_snooze_max_attempts → perform_map_element_rollback_in(delay, attempt: a+1); stop (no outcome)
+    held and a = max                      → outcome element_in_flight
+  status completed|aborted → rollback_completed_steps, saving the element after each pop
+                             → undone | failed(failures)
+  otherwise (failed / halted / superseded running) → not_needed
+  HSET results[p]; SADD indexes index; log rollback.element
+  p ≡ B-1 (mod B) → claim the next batch
+  HLEN results == total && EXISTS rollback:handed_off && SET NX rollback:signalled → enqueue Worker(owner)
+
+Worker(owner), status rolling_back → Executor#resume_rollback (takes async:<owner>)
+  restore rollback_failures from `rollback.failures`
+  unless compensated: handle the failing step's compensate again (constructs adopt)
+  rollback_completed_steps → MapStep#undo re-entry → settled → aggregate → pop → continue
+  finalize: failure → the same Failure as the inline path (I-7) | undo → cancelled
+  clear `rollback`, save, release
+```
+
+A second fan-out map lower in the undo stack repeats S-2 from inside `resume_rollback`. The
+status stays `rolling_back`.
+
+**Aggregation** (`MapStep#undo` re-entry once settled): `HSCAN` failures and re-symbolize them.
+Then report unavailable elements once: named started-but-unseen indexes when `started` is known,
+otherwise one entry per nil-index outcome (DM §5).
+
+**Inline test mode** (all jobs run inside dispatch): every outcome is stored before `MapStep#undo`
+checks settled, and `handed_off` never exists, so no job enqueues the owner. `#undo` aggregates
+synchronously and never raises the hand-off.
+
+## S-3: The failing step is inside a composed child
+
+```text
+root: ComposeStep#run → child executes → child step fails
+  child handle_step_failure: compensate the child step, then pop child entries … child MapStep#undo → RollbackHandedOff
+  child executor: record child.rollback {trigger: failure, step: <child step>, compensated: true, failure: <child Failure>}, re-raise
+  root StepExecutor: passes it through (not Rescuable)
+  root executor: root.rollback {trigger: failure, step: <compose step>, compensated: false, failure: <child Failure>}
+                 status rolling_back, save (the child is embedded), release
+… element rollbacks settle → Worker(root)
+root resume_rollback: compensated false → handle_step_failure(compose step)
+  ComposeStep#compensate → child executor undo_all (restores child.rollback failures) → child stack:
+    map entry → adopt → pop → the rest of the child → clear child.rollback
+  → then the root's own completed steps → finalize with the saved failure
+```
+
+The same path runs when a **manual undo from the root** reaches a composed child: root
+`Reactor#undo` → root `undo_all` → `ComposeStep#undo` → child `undo_all` → child `MapStep#undo`
+hands off. Child `undo_all` records its failures and re-raises, and `Reactor#undo` records the root
+`rollback {trigger: undo}`. On resume, child `undo_all` restores its failures, so none are lost.
+
+## S-4: Inline map rollback (no hand-off)
+
+```text
+MapStep#undo (map_operations[step] absent)
+  for each chunk of 100 positions from the tail of element_contexts:
+    load contexts → Map::ElementRollback (same class as the job, in process) → collect failures, indexes
+  started-index check → Success / Failure(rollback_failures)
+```
+
+## S-5: Recovery
+
+| Lost thing | Detected by | Action |
+| --- | --- | --- |
+| Element rollback job (crash before storing the outcome) | `Map::Sweeper` rollback pass: a position `< offset` with no outcome and no live element lock | Re-enqueue that position. The element resumes after its last saved undo. |
+| Batch trigger (the job at `kB-1` died after storing its outcome) | `Map::Sweeper`: offset `< total` and every claimed position settled | Claim and enqueue the next batch. |
+| Owner resume (after `rollback:signalled` was set) | `RubyReactor::Sweeper`: `rolling_back`, no `async:` lock | Re-enqueue the owner. `resume_rollback` adopts the settled map. |
+| Owner resume (after `owner_signalled` was set) | `RubyReactor::Sweeper`: `running`, no `async:` lock (existing) | Re-enqueue the owner. `MapStep#run` adopts. |
+| Collector | `Map::Sweeper` recollect (existing), with the owner's `async:` lock checked instead of the parent's | Re-trigger the collector. `owner_signalled` keeps it to one resume. |
+
+## S-6: Legacy in-flight work across the upgrade
+
+| Payload | Handling |
+| --- | --- |
+| Element / dispatcher / collector / retry job carrying `fail_fast` | `Map::Helpers.normalize_arguments` maps it to `atomic`. |
+| Map metadata without `owner_context_id` (a map started before the upgrade) | Collector falls back to `parent_context_id` / `parent_reactor_class_name`. This is correct for root-level maps. A map inside a composed child started before the upgrade keeps the old hang; documented in CHANGELOG. |
+| Map metadata without `batch_size` | The rollback uses `DEFAULT_BATCH_SIZE`. |
+| Element index with duplicate ids | Safe: an empty undo stack reports `undone`, and a concurrent duplicate waits on the element lock or reports `element_in_flight`. |

data/specs/009-distributed-map-undo/data-model.md

@@ -0,0 +1,151 @@
+# Data Model: Distributed Map Rollback and Bounded Fan-out
+
+**Feature**: [spec.md](spec.md) | **Research**: [research.md](research.md)
+
+All records live in Redis through `Storage::RedisAdapter`. Every key expires with
+`durability_ttl`, like the existing map keys. `<P>` is the class name of the context that ran the
+map (the parent), as in today's map keys; `<map_id>` is `"<parent context id>:<map step>"`.
+
+## 1. Context (changed)
+
+| Field | Type | Change | Notes |
+| --- | --- | --- | --- |
+| `status` | string | new value `rolling_back` | A rollback handed off at a fan-out map and has not finished. Non-terminal. |
+| `rollback` | hash or nil | **new** | Where this execution level's rollback stands. Set only while `rolling_back`; cleared when the rollback finishes. A composed child keeps its own copy inside the root blob. |
+| `failure_reason` | Failure | unchanged | Holds the final `Failure` once the rollback finishes. |
+| `undo_stack` | array | unchanged | Already pops one entry per completed undo (008 R-16); this is the rollback's resume cursor. |
+| `map_operations[step]` | string | meaning extended | Present means the map was dispatched: `MapStep#run` adopts instead of dispatching again, and `#undo` rolls back distributed (R-01, R-03). |
+
+### `rollback` hash
+
+| Key | Type | Meaning |
+| --- | --- | --- |
+| `trigger` | `"failure"` \| `"undo"` | What started the rollback: a step failure, or `Reactor#undo`. |
+| `step` | string or nil | The failing step (`failure` trigger). For a compose whose child handed off, this is the compose step. |
+| `compensated` | bool | Whether `step`'s own compensate has finished. False when the hand-off came from compensating `step` itself (a failed fan-out map, or a compose whose child is mid-rollback). |
+| `failure` | serialized `Failure` or nil | The triggering failure. `resume_rollback` rebuilds the `StepFailureError` from it. |
+| `failures` | array of rollback-failure entries | Rollback failures collected before the hand-off, restored into `CompensationManager#rollback_failures` on resume. |
+
+### Status transitions
+
+```text
+running ──step failure──▶ (rollback in process) ──hand-off at fan-out map──▶ rolling_back
+rolling_back ──owner resumed, map settled, rest undone──▶ failed      (trigger: failure)
+rolling_back ──owner resumed, map settled, rest undone──▶ cancelled   (trigger: undo)
+completed | failed | paused ──Reactor.undo──▶ (undo in caller) ──hand-off──▶ rolling_back
+rolling_back ──another hand-off (a second fan-out map lower in the stack)──▶ rolling_back
+```
+
+Rules:
+
+- `Worker#perform` on `rolling_back` calls `Executor#resume_rollback` and never resumes forward.
+- `Reactor.undo` on `rolling_back` raises `ValidationError` (FR-027).
+- `Reactor.cancel` on `rolling_back` raises `ValidationError` (FR-027). `cancelled` is terminal and
+  would strand the steps before the map.
+- `RubyReactor::Sweeper` re-enqueues `rolling_back` when no `async:` lock is held.
+- A parent waiting on an `async_reactor` child keeps waiting while the child is `rolling_back`,
+  because the status is non-terminal.
+
+## 2. Map metadata (changed)
+
+Key: `reactor:<P>:map:<map_id>:metadata` (hash, existing).
+
+| Field | Change | Notes |
+| --- | --- | --- |
+| `owner_context_id` | **new** | The top-level run: `(context.root_context \|\| context).context_id`. The collector and the rollback resume this run's `Worker` (R-03, R-05). |
+| `owner_reactor_class_name` | **new** | Storage name of the owner's class. |
+| `batch_size` | **new** | The effective batch size: declared, or `Map::DEFAULT_BATCH_SIZE` (50). Read by forward re-dispatch and by the rollback. |
+| `atomic` | **new** | Replaces the never-stored `fail_fast`. Read by `Dispatcher.requeue_index`. |
+| everything else | unchanged | `count`, `strict_ordering`, `reactor_class_info`, `parent_context_id`, `step_name`, the nested-map fields. |
+
+## 3. Map owner signal (new)
+
+Key: `reactor:<P>:map:<map_id>:owner_signalled`, a string set with `SET NX` (TTL `durability_ttl`).
+
+The collector claims it once the map has settled, then enqueues the owner's `Worker`. Only one
+collector delivery ever enqueues.
+
+## 4. Element-context index (write rule changed)
+
+Key: `reactor:<P>:map:<map_id>:element_contexts` (list, existing).
+
+- **Forward write**: `ElementExecutor` pushes the context id only for a **fresh** element context,
+  not on a parked or retried re-entry (R-06). The inline map path is unchanged: one push per
+  element.
+- **Rollback read**: positions counted from the tail (position 0 is the last pushed id), one
+  `LRANGE` per claimed batch.
+
+## 5. Map rollback records (new)
+
+| Key suffix (`reactor:<P>:map:<map_id>:rollback…`) | Type | Contents |
+| --- | --- | --- |
+| `:metadata` | hash, created with `HSETNX` | `total` (LLEN of the element index at start), `batch_size`, `step_name`, `owner_context_id`, `owner_reactor_class_name`, `reactor_class_info` (element class), `started_at`. Its presence means "a rollback of this map exists" (FR-027). |
+| `:offset` | integer | The next unclaimed position. `INCRBY batch_size` claims a batch (R-02). |
+| `:results` | hash, position → JSON outcome | One element rollback outcome per position (below). `HLEN == total` means settled. |
+| `:indexes` | set of integers | Element indexes that a job saw. Used to name started-but-never-seen indexes (R-09). |
+| `:handed_off` | string, `SET` | Set by the top-level hand-off handler **after** it saved the `rolling_back` context. Until it exists, no job enqueues the owner (R-05). Never set when the map settles synchronously (inline mode). |
+| `:signalled` | string, `SET NX` | Claimed by whichever side observes both "settled" and `handed_off` second: the job storing an outcome, or the handler right after setting `handed_off`. The claimer enqueues the owner's `Worker` once (R-05). |
+
+### Element rollback outcome (value in `:results`)
+
+```json
+{ "index": 42, "outcome": "undone", "failures": [] }
+```
+
+| `outcome` | When | Contributes to the final `rollback_failures` |
+| --- | --- | --- |
+| `undone` | The element was `completed` or `aborted` and its undo stack was replayed with no failure, or the stack was already empty (a duplicate). | nothing |
+| `failed` | Replayed, but one or more undos failed. | each entry of `failures`, tagged `map_step`, `element_index` |
+| `not_needed` | The element started but has nothing to undo: `failed` (it rolled itself back), `halted`, or still `running` from a superseded delivery. Skipped elements never started, so they have no position and no outcome. | nothing |
+| `element_in_flight` | The element's `map_element:` lock was still held after the job requeued itself `lock_snooze_max_attempts` times (R-07): a genuinely live forward duplicate. A contended lock before that only requeues the job; no outcome is stored. | one entry, `reason: :element_in_flight` (as 008) |
+| `context_unavailable` | The element context row is gone (expired). `index` is null. | only when `started` is unknown (below) |
+
+**Aggregation** (owner, on re-entry once settled; R-09):
+
+1. `HSCAN` outcomes in chunks of 500, and keep the `failures` of `failed` and `element_in_flight`
+   outcomes.
+2. **Re-symbolize** each entry, because JSON round-trips symbols as strings. Keys become symbols;
+   the `step`, `kind`, `reason` and `map_step` values become symbols; `element_index` stays an
+   Integer or nil. The entries then equal the inline path's (FR-005).
+3. **Unavailable elements, each reported once**:
+   - `started` known: add one `context_unavailable` entry per index in `0...started` missing from
+     `:indexes`, and ignore nil-index `context_unavailable` outcomes, which describe the same
+     elements;
+   - `started` unknown: add one unnamed entry per nil-index `context_unavailable` outcome.
+
+   This mirrors 008's `report_unavailable`, which uses one form or the other, never both.
+
+### Rollback-failure entry (unchanged shape, 008)
+
+```ruby
+{ step: :charge, kind: :undo, key: nil, reason: :raised, message: "...",
+  map_step: :charge_orders, element_index: 42 }
+```
+
+## 6. Element context during rollback (write rule changed)
+
+The element rollback job saves the element context after **each** popped undo entry (R-07), under
+the element's `map_element:<map_id>:<index>` lock. A redelivered job therefore resumes after the
+last saved entry. Status stays as it was (`completed` / `aborted`); an empty undo stack is the
+"already undone" signal.
+
+## 7. Constants
+
+| Constant | Value | Where |
+| --- | --- | --- |
+| `Map::DEFAULT_BATCH_SIZE` | 50 | Forward dispatch and rollback when no `batch_size` is declared. |
+| `Map::ROLLBACK_CHUNK` | 100 | Elements loaded per read in inline map rollback. |
+| `MapStep::ELEMENT_LOCK_WAIT` | 2 s | Existing. The rollback's wait on an element's liveness lock, per attempt. |
+| `lock_snooze_max_attempts` / `lock_snooze_base_delay` | configuration (existing) | How many times, and how far apart, an element rollback job requeues itself on a contended element lock before reporting `element_in_flight` (R-07). |
+| `Reactor::UNDO_LOCK_WAIT` | 5 s | Manual undo's wait on the run's `async:` lock (R-12). |
+
+## 8. DSL value (renamed)
+
+| DSL | Stored as | Default |
+| --- | --- | --- |
+| `atomic(enabled = true)` | `MapStep` input `atomic` | `true` |
+| `fail_fast(enabled = true)` (deprecated) | same input; one warning per call site | — |
+| `fan_out(enabled = true, batch_size: nil)` | `batch_size` input; the effective value is resolved at dispatch | 50 when omitted |
+
+Legacy job payload key `fail_fast` is read as `atomic` by `Map::Helpers.normalize_arguments`
+(FR-023).