active_durable 0.6.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d6ce41040ab6f0cb0036bfe36b30b831a1f91a7da099b9f9e3914568716ef6e6
-  data.tar.gz: ac2223d523ec0edf3470b8b9b66b5a892d2521156ab0446e874feff3104c59ff
+  metadata.gz: 47ef8eb3bb5a133221ed52ce914ec952ff7f73f552b5352d4558b96e6738458c
+  data.tar.gz: af132b5a177125726a6aed8699e7cf4391c70e9e4fbac566531ab87c3204f998
 SHA512:
-  metadata.gz: c95a763c1808a55ceebf13c02aa1e53f29830601b2547f6175c6bb375f1b78b9a5c5cd02aa2da505afac8989259f2047f588448aef34ce3bdb98a61dc2037e85
-  data.tar.gz: f12d9dff80e6c7e77bdaa40cd3683356bfc443b04eb442d59a5ca2e8e3da5cd9ecf2f0f97df180014c41c723b49ffb6b80fcfc111833f02994856fef460f61c3
+  metadata.gz: fe0584f97ce0594ee30c2186b8a10ac91e57505172b758cf8e667b290a6cb387e0da3566db685c8070798bdab84f32f225a7f2f1787aabc9b106bbe069ceb11e
+  data.tar.gz: 863396b2acc54f007f65c80379e3bcbac0512eb05888d3a91372cddfd88333e5b3536d25798949eaf2200a85fd021ae540a65ffd4d1d1078716e0221247a609b

data/CHANGELOG.md

@@ -5,7 +5,105 @@ All notable changes to this project are documented here. The format follows
 [Semantic Versioning](https://semver.org/). Schema changes ship as new migrations: after updating the gem, run
 `bin/rails generate active_durable:upgrade` and `bin/rails db:migrate`.
 
-## [Unreleased]
+## [0.8.0] - 2026-10-22
+
+For sagas that live long and talk to people, such as a WhatsApp agent that books appointments: a mailbox for
+messages, a fresh notebook for long conversations, and the tenant in every run.
+
+Upgrading from 0.7: `bin/rails generate active_durable:upgrade && bin/rails db:migrate`. It adds three nullable
+columns (`durable_executions.context`, `durable_executions.continued_as` with its index, `durable_steps.signal_name`);
+nothing is rewritten. Note the first change below: hash keys now come back sorted.
+
+### Added
+
+- `flow.continue_as_new(**input)` closes the execution and starts a new one of the same recipe (its latest version)
+  with that input, the same context and the signals nobody read yet, in one transaction. Ids follow the chain
+  `order-1`, `order-1~2`, `order-1~3`. The new `continued` status; `Durable.find(id, latest: true)`.
+  `Durable.signal` to any id of the chain reaches the newest execution, also during the hand-off.
+- `flow.wait_for(name, signal:, all:)`: several waits share one signal name, like a mailbox, and `all: true` takes
+  every pending signal at once, returning their payloads in order.
+- `config.context`: an object with `capture` (when an execution starts) and `wrap(context) { }` (around every run,
+  undo, hook and `flow.parallel` branch). `Durable.start(..., context: {...})` gives it explicitly. Reruns keep it.
+- `ActiveDurable::Testing.drain(id, expire_waits: false)` stops at a wait instead of fast-forwarding to its
+  timeout, to test a conversation turn by turn. `drain` follows `continue_as_new`.
+- The dashboard shows `continued` executions and links to the one that carries on.
+
+### Changed
+
+- Hash keys in step results, inputs, signal payloads and `flow.parallel` results are sorted, on the first run and
+  when read back. PostgreSQL's jsonb and MySQL's JSON store keys shorter-first, so a replay used to read a hash whose
+  keys were in another order than the first run returned. That broke anything that compares bytes, such as an AI
+  provider that checks the conversation it is sent again. Executions in flight during the upgrade see their keys in
+  a new order once.
+- `Durable.signal` locks the execution row while it writes the signal.
+- `Testing.drain` delivers each signal given in `signals:` once per call, instead of every time a step of that name
+  waits.
+- `Durable.start` takes `context:`, so `context` can no longer be an input key.
+- `ActiveDurable.retry`, `compensate` and `rerun` on a continued execution raise, naming the one that carries on.
+- `ActiveDurable.prune` deletes a continued execution only together with the one that carried it on.
+
+## [0.7.0] - 2026-10-07
+
+Fixes from an architecture and security review. Most of them stop a saga from paying twice or undoing the wrong
+thing.
+
+Upgrading from 0.6: `bin/rails generate active_durable:upgrade && bin/rails db:migrate`. The new migration only
+changes MySQL databases: it rebuilds the three tables and blocks writes to them while it runs, so run it at a quiet
+time, or prune first. Note the behavior changes below: more errors block instead of undoing, `retry` no longer
+resets every failed step, rerun ids are random, and `Durable.start` refuses some ids.
+
+### Changed
+
+- More errors count as bugs and block the saga instead of being retried and undone: `ArgumentError`, `TypeError`,
+  `IndexError` and `KeyError`, `FrozenError`, `ZeroDivisionError`, `RangeError`, `NoMatchingPatternError`,
+  `LocalJumpError`, `RegexpError` and `EncodingError`, besides `NameError` and `NoMethodError`. The list is the new
+  `config.code_errors`, which takes classes or names, so an app can add its own errors or drop one.
+- A step that blocks is written in the notebook as `blocked`, and runs again after `ActiveDurable.retry`. Rescuing
+  its error in the recipe no longer lets the saga carry on.
+- `ActiveDurable.retry` gives a fresh set of attempts only to the step that blocked the saga (with its whole
+  `flow.parallel`) and to steps that hit a bug. Compensating, it still resets every failed undo.
+- Rerun ids are `<id>~rerun-<random>` instead of `<id>~rerun-<count>`.
+- `ActiveDurable.compensate` refuses `running` executions, even after their lease ran out.
+- `Durable.start` refuses ids that are blank, contain `/`, a NUL character or bytes that are not UTF-8.
+- The serializer converts binary strings that are valid UTF-8 (such as `Net::HTTP` bodies) and strings in other
+  encodings to UTF-8, and rejects NUL characters and invalid bytes, in values and keys.
+
+### Fixed
+
+- `retry` ran again a failed step the recipe had already handled: with a fallback (Stripe fails, the recipe pays
+  with PayPal) and a later bug, retrying charged twice. `rerun` did the same; it now copies the failed steps before
+  the chosen one too.
+- A step whose result the database refused (a NUL on PostgreSQL, bytes that are not UTF-8 on MySQL and SQLite, any
+  failed write) counted as a failed attempt: it ran again, then the saga was undone without undoing it. It now
+  blocks.
+- Undoing a saga by hand skipped a step declared `undo_on_failure` that was between attempts or blocked, and the
+  finished branches of a `flow.parallel` stopped halfway.
+- An operator could undo a saga while its worker was still inside a step longer than `lease_duration`.
+- A pending signal nobody was waiting for (a duplicate webhook) made the saga enqueue itself again forever.
+- Signals sent to a blocked saga were refused and lost.
+- `rerun` lost the undos of the `flow.parallel` branches before the chosen step, and from a branch it ran nothing;
+  the dashboard no longer offers branches.
+- On MySQL, ids and step names that differ only in case or accents were treated as the same. The new migration
+  makes those columns `utf8mb4_bin`.
+- `raise ActiveRecord::Rollback` inside `flow.transaction`, an undo or a hook counted as success.
+- On Rails 6.1 and 7.0 with MySQL, two concurrent `start` calls with the same id inside the app's transaction made
+  the second raise `RecordNotFound`.
+- Rerun ids collided after pruning, and could give a new rerun the tickets of a pruned one.
+- An id with `/` broke the whole dashboard list.
+- `LoadError`, `NotImplementedError` and `SystemStackError` left the saga running, retried forever by the sweeper.
+- A bug in an undo used up all its attempts before blocking; it now blocks at once.
+- Error messages are cleaned up (invalid bytes, NUL) before they are stored.
+- README: how to run Solid Queue in development, and the sweeper and cleanup entries go under the `production:` key
+  Rails already wrote in `config/recurring.yml`. Pasting a second `production:` key silently dropped Rails' own
+  `clear_solid_queue_finished_jobs` task.
+
+### Added
+
+- Benchmarks in `benchmarks/`: the cost of a step and of resuming a long saga (`throughput.rb`), and many worker
+  processes on the same sagas, optionally killing one with SIGKILL every second (`load.rb`, `CHAOS=1`). The README
+  has a "Performance" section with the results on PostgreSQL, MySQL and SQLite.
+- README: what counts as a bug, `config.code_errors`, and new limits: ids are forgotten after pruning, the lease and
+  slow steps, clocks, and secrets in error messages.
 
 ## [0.6.0] - 2026-10-06
 

data/README.md

@@ -161,12 +161,22 @@ config.solid_queue.connects_to = { database: { writing: :queue } }
   The server runs its own, sleeps and retries included, but a saga started from the console, `bin/rails runner`, a
   rake task or `db/seeds.rb` is lost when that process exits, and scheduled wake-ups are lost when the server
   restarts. A minute later, `bin/rails active_durable:sweep` picks them up: with `:async` it runs them right there.
-  Or run Solid Queue in development too, with `bin/jobs` next to the server.
+  Or run Solid Queue in development too, so jobs live in the database: give `development` a `queue` database in
+  `config/database.yml` (like the one `production` has, with `migrations_paths: db/queue_migrate`), add the two
+  lines below, run `bin/rails db:prepare`, and start `bin/jobs` next to the server.
+
+```ruby
+# config/environments/development.rb
+config.active_job.queue_adapter = :solid_queue
+config.solid_queue.connects_to = { database: { writing: :queue } }
+```
 
 ### 3. The sweeper
 
 The safety net: every minute it enqueues executions that lost their job, because the process died between the
-commit and the enqueue or a worker died holding a lease. With Solid Queue:
+commit and the enqueue or a worker died holding a lease. With Solid Queue, add these entries under the
+`production:` key Rails already wrote in `config/recurring.yml` (a second `production:` key would silently replace
+it), and under a `development:` key too if you run Solid Queue in development:
 
 ```yaml
 # config/recurring.yml
@@ -202,6 +212,8 @@ ActiveDurable.configure do |config|
   config.parallel_concurrency = 4       # threads per flow.parallel
   config.sweep_grace = 1.minute         # the sweeper leaves executions this young alone
   config.keep_finished_for = 30.days    # then ActiveDurable::PruneJob deletes finished ones
+  config.code_errors += ["Payments::Misconfigured"] # your own errors that mean a bug (see "A bug is not a failure")
+  config.context = TenantContext        # who the work is for, in every run (see "Context")
 
   # Who may open the dashboard outside development and test. With Devise (HTTP basic auth: see Dashboard):
   config.dashboard_authorize = ->(controller) { controller.request.env["warden"]&.user&.admin? }
@@ -402,7 +414,8 @@ Three rules follow from replaying the recipe:
 | `flow.pivot(name) { \|ticket\| ... }` | the step after which there is no going back | retried, then the saga is undone |
 | `flow.parallel(name) { \|branches\| ... }` | several steps at the same time | each branch like a step |
 | `flow.sleep(name, 3.days)` | waiting without holding a worker | — |
-| `flow.wait_for(name, timeout:)` | waiting for `Durable.signal` | a timeout fails the saga |
+| `flow.wait_for(name, timeout:)` | waiting for `Durable.signal` (`signal:` and `all:` for a mailbox) | a timeout fails the saga |
+| `flow.continue_as_new(**input)` | a long-lived saga, such as a chat, that starts a fresh notebook | refused inside a step or with undos pending |
 | `flow.on(:completed) { ... }` | updating your own records when the saga ends (also `:compensated`) | blocks; a retry runs the hook again |
 
 Steps take `undo:`, `retry:` (`3`, `false` or `{ attempts:, backoff: }`) and `undo_on_failure:`.
@@ -428,10 +441,33 @@ When a step runs out of attempts or calls `flow.abort!`, every finished step is
 stopped. An undo receives `(result, undo_ticket, step_ticket)` and takes as many as it declares: `-> { ... }` takes
 none. Any object that responds to `call` works too, such as `Payments.method(:refund)`.
 
-**A bug is not a failure.** Anything else the recipe raises, and a `NameError` or `NoMethodError` inside a step,
-blocks the execution instead of undoing it: a typo in a deploy must never refund your customers. Fix the code and
-call `ActiveDurable.retry(id)`, or press Retry in the dashboard, and the saga carries on from where it stopped. To
-reject the work for a business reason outside a step, call `flow.abort!(reason)`.
+**A bug is not a failure.** Undoing a saga refunds money and releases stock, so only a step that failed for good or
+`flow.abort!` triggers it. These block the execution instead, without retrying, because a typo in a deploy must never
+refund your customers:
+
+- anything the recipe raises outside a step;
+- inside a step, an error that means the code is wrong: `NameError` (and `NoMethodError`), `ArgumentError`,
+  `TypeError`, `IndexError` (and `KeyError`), `FrozenError`, `ZeroDivisionError`, `RangeError`,
+  `NoMatchingPatternError`, `LocalJumpError`, `RegexpError`, `EncodingError`, and the errors outside
+  `StandardError` such as `LoadError`, `NotImplementedError` and `SystemStackError`;
+- a step whose result cannot be stored (a NUL character, bytes that are not UTF-8): it already acted.
+
+Fix the code and call `ActiveDurable.retry(id)`, or press Retry in the dashboard, and the saga carries on from where it
+stopped. Rescuing the error in the recipe does not help: the saga stops at the next step and blocks anyway. To reject
+the work for a business reason, call `flow.abort!(reason)`.
+
+The list is `config.code_errors`. Add your own errors, as a class or as a name (a name also matches subclasses, and
+works before the gem that defines it is loaded), or drop one:
+
+```ruby
+ActiveDurable.configure do |config|
+  config.code_errors << "Payments::Misconfigured"
+  config.code_errors -= ["ArgumentError"] # if a gem you call raises it for bad input
+end
+```
+
+An `ArgumentError` from bad data, like an invalid date, blocks too: a person decides. To treat one of these errors as
+a failure in a single step, rescue it inside the step and raise another error, or call `flow.abort!`.
 
 A step that failed is not undone, because it did not happen. The exception is a step whose failure may hide a
 success, like a charge whose answer timed out: declare `undo_on_failure: true` and its undo runs with `nil` as the
@@ -492,7 +528,7 @@ For progress before the end (paid, shipped), write a step: `flow.transaction(:ma
 <br>
 
 `flow.sleep` writes the wake-up time down and releases the worker. `flow.wait_for` does the same until a signal
-arrives. Signals can arrive before the saga starts waiting.
+arrives. Signals can arrive before the saga starts waiting, or while it is blocked: they are kept until it does.
 
 ```ruby
 kyc = flow.wait_for(:kyc_done, timeout: 2.hours)
@@ -503,6 +539,18 @@ Durable.signal("loan-42", :kyc_done, verified: true)
 
 Pass `id:` to `Durable.start` to choose the execution id (the call becomes idempotent).
 
+**A mailbox.** By default the signal has the step's name. A conversation waits for the customer on every turn, so
+each turn gets its own step name and all of them share one signal name with `signal:`. A message that arrives while
+the saga is busy waits in the mailbox for the next turn. With `all: true` the wait takes every pending message at
+once and returns them in order, so five messages in a row are one turn, not five:
+
+```ruby
+messages = flow.wait_for("reply.#{turn}", signal: :message, all: true, timeout: 1.day)
+
+# in the WhatsApp webhook
+Durable.signal(conversation.execution_id, :message, text: params[:text])
+```
+
 </details>
 
 <details>
@@ -528,6 +576,57 @@ the finished branches are undone in the order they finished. Give your connectio
 
 </details>
 
+<details>
+<summary><b>Context: the tenant in every run</b></summary>
+
+<br>
+
+A saga runs in a worker, days after the request that started it, so `Current.tenant` and friends are gone.
+`config.context` carries them: `capture` runs when the execution starts and its JSON is stored with it; `wrap` runs
+around every run of it, including undos, hooks and each `flow.parallel` branch thread.
+
+```ruby
+module TenantContext
+  def self.capture = { "tenant_id" => Current.tenant_id }
+  def self.wrap(context, &run) = Current.set(tenant_id: context["tenant_id"], &run)
+end
+
+ActiveDurable.configure { |config| config.context = TenantContext }
+
+Durable.start(:checkout, order_id: order.id)                                  # captures the current tenant
+Durable.start(:welcome, tenant_id: t.id, context: { "tenant_id" => t.id })   # or gives it explicitly
+```
+
+`wrap` must call the block once. Executions started without a context get `{}`.
+
+</details>
+
+<details>
+<summary><b>Long conversations: continue as new</b></summary>
+
+<br>
+
+Every run replays the recipe from the top, reading the notebook. A chat that has gone on for 200 turns reads 1,000
+entries before it can answer. `flow.continue_as_new` closes the execution and starts a fresh one of the same recipe
+(its latest version) with the input you give it, usually a summary, plus the same context and the messages nobody
+read yet. All of it happens in one transaction:
+
+```ruby
+Durable.define(:chat) do |flow, conversation_id:, turn: 0, summary: nil|
+  # ... the turn ...
+  flow.continue_as_new(conversation_id: conversation_id, turn: turn + 1, summary: summary) if turn % 20 == 19
+end
+```
+
+The new execution's id is the old one plus `~2` (then `~3`), so its steps get new tickets. `Durable.signal` to any id
+of the chain reaches the newest execution, even one sent during the hand-off, and
+`Durable.find(id, latest: true)` returns it. Compute the input from step results, so a replay computes the same.
+`continue_as_new` is refused inside a step and while undos are pending (those steps could never be undone); after
+`flow.pivot` it is allowed. The old execution's hooks do not run, and it is pruned together with the one that
+carries on.
+
+</details>
+
 <details>
 <summary><b>Changing a recipe while sagas are in flight</b></summary>
 
@@ -547,6 +646,91 @@ New executions use the highest version; each execution keeps the one it started
 
 </details>
 
+## Example: a WhatsApp assistant that books appointments
+
+First in plain words, then the code.
+
+**What it does.** A customer writes to a bakery on WhatsApp: "Hi, can I come on Friday to order a cake?". An
+assistant (an AI that can see the calendar) answers, offers the free times and books the one the customer picks. The
+conversation can take two minutes or two days. The customer may send three messages in a row, or disappear and come
+back tomorrow. Meanwhile your servers restart, you deploy, a process dies.
+
+**The notebook.** Think of a cook who ticks every step of a recipe in a notebook. When someone interrupts, the cook
+does not trust their memory: they reread the notebook from the top, skip every step with a tick, and carry on from
+the first one without it. ActiveDurable does the same with your code. Each step's result is written in your database,
+and every time the saga wakes up (a message arrives, a timer rings) it rereads the recipe and skips what is done. That
+is why the assistant never books twice, even if the server dies right after booking: the booking has its tick. (A
+step that died halfway runs again with the same ticket, so the calendar and WhatsApp can recognize a repeat.)
+
+**Turns.** A turn is one exchange: the customer writes, the assistant answers. Each turn adds three ticks: the
+customer's messages, the AI's answer (which may book) and the WhatsApp message sent back. The AI's answer is written
+down too, so rereading never asks the AI again: no double cost, and the very same text.
+
+**The mailbox.** Every message goes to one mailbox, `:message`. If the customer writes while the assistant is
+thinking, the message waits there for the next turn. If they send five in a row, the assistant takes all five at
+once: one turn, not five.
+
+**A fresh page.** After 20 turns the notebook has 60 ticks to reread every time something happens. Instead of making
+the same page longer, `continue_as_new` starts a new page with a short summary ("Ana wants a cake on Friday, saw the
+times at 10 and 3"), keeping the messages nobody read yet.
+
+**Who it is for.** One app serves many bakeries. `config.context` writes down which one the conversation belongs to
+when it starts, and gives it back on every run, so each step sees the right calendar.
+
+```ruby
+# config/initializers/active_durable.rb
+module TenantContext
+  def self.capture = { "tenant_id" => Current.tenant_id }
+  def self.wrap(context, &run) = Current.set(tenant_id: context["tenant_id"], &run)
+end
+ActiveDurable.configure { |config| config.context = TenantContext }
+
+# app/sagas/booking_assistant_saga.rb
+BookingAssistantSaga = Durable.define(:booking_assistant) do |flow, conversation_id:, turn: 0, summary: nil|
+  conversation = Conversation.find(conversation_id)
+  history = summary ? [{ "role" => "user", "content" => "What happened so far: #{summary}" }] : []
+  booked = nil
+
+  20.times do |i|
+    n = turn + i
+    # 1. The customer's messages: every one that arrived since the last turn.
+    messages = flow.wait_for("customer.#{n}", signal: :message, all: true)
+    history << { "role" => "user", "content" => messages.map { |m| m["text"] }.join("\n") }
+
+    # 2. The AI's answer. When it decides to book, it calls Calendar.book(slot, key: ticket): booked once.
+    reply = flow.step("assistant.#{n}") { |ticket| BookingAssistant.answer(history, ticket: ticket) }
+    history << { "role" => "assistant", "content" => reply["text"] }
+
+    # 3. The answer. If the process dies right after sending it, this step runs again with the same ticket:
+    #    WhatsApp.send_text keeps the ticket with the message it sent and skips sending it twice.
+    flow.step("send.#{n}") { |ticket| WhatsApp.send_text(conversation.phone, reply["text"], key: ticket) }
+
+    if reply["booked"]
+      booked = reply
+      break
+    end
+  end
+  next booked if booked
+
+  # 20 turns without a booking: a fresh page with a summary.
+  summary = flow.step("summary.#{turn}") { BookingAssistant.summarize(history) }
+  flow.continue_as_new(conversation_id: conversation_id, turn: turn + 20, summary: summary["text"])
+end
+
+# app/controllers/whatsapp_webhooks_controller.rb
+def create
+  conversation = Conversation.open_for(params[:from]) # a new one once the last one ended; tenant: Current.tenant_id
+  id = "chat-#{conversation.id}"
+  Durable.start(:booking_assistant, id: id, conversation_id: conversation.id) # idempotent: starts it once
+  Durable.signal(id, :message, text: params[:text]) # reaches the newest page of the chain
+  head :ok
+end
+```
+
+To close conversations where the customer went quiet, give the wait a `timeout: 1.day` and rescue
+`ActiveDurable::StepFailed`. To test it turn by turn: `Testing.drain(id, expire_waits: false)`, then `Durable.signal`,
+then drain again.
+
 ## Dashboard
 
 ```ruby
@@ -585,14 +769,21 @@ end
 ## Operating sagas
 
 ```ruby
-ActiveDurable.retry("checkout-7")                                   # blocked: try again where it stopped
+ActiveDurable.retry("checkout-7")                                   # blocked: try again the step that blocked it
 ActiveDurable.compensate("checkout-7", reason: "customer cancelled") # undo everything (only before the pivot)
 ActiveDurable.rerun("checkout-7", from: :ship)                      # a new execution reusing steps before :ship
 ActiveDurable.prune(older_than: 30.days)                            # delete finished executions and their notebook
 ```
 
-All three refuse an execution a worker is running right now. A rerun runs the chosen step and the following ones
-again with new tickets, so they have effects again; a blocked original becomes `superseded`.
+`retry` gives the step that blocked the saga a fresh set of attempts. Failed steps the recipe already handled stay
+failed: if Stripe failed and the recipe paid with PayPal instead, a retry never runs Stripe again.
+
+A rerun (`<id>~rerun-<random>`) keeps the steps before the chosen one, failed ones and parallel branches included,
+and runs the chosen step and the following ones again with new tickets, so they have effects again. A blocked
+original becomes `superseded`.
+
+None of them touches an execution a worker is running. `compensate` refuses a `running` execution even after its
+lease ran out: the worker may still be inside a slow step, and if it died, the sweeper resumes it.
 
 ## Testing: the crash tester
 
@@ -613,7 +804,8 @@ point, kills it right there, and finishes it with a new worker. It is the fastes
 idempotent.
 
 `ActiveDurable::Testing.drain(id, signals: { name => payload })` runs an execution synchronously, fast-forwarding
-sleeps, retries and expired leases.
+sleeps, retries and expired leases, and follows `continue_as_new`. It fast-forwards wait timeouts too; to test a
+conversation turn by turn, pass `expire_waits: false`, send a signal and drain again.
 
 ## Observability
 
@@ -679,6 +871,28 @@ Every feature works on every version. Things your app may need on older Rails, u
 | Temporal | the Temporal server | written by hand | a Temporal cluster |
 | **ActiveDurable** | **your database** | **yes, in reverse, with a point of no return** | **nothing extra** |
 
+## Performance
+
+Measured on an Apple M1 Ultra with Ruby 4.0.7 and Rails 8.1, each database on the same machine. The scripts are in
+[`benchmarks/`](benchmarks/README.md), so you can run them on yours.
+
+| | PostgreSQL 16 | MySQL 9.6 | SQLite 3 |
+| --- | --- | --- | --- |
+| A step (`flow.step`), one process | 1.8 ms | 3.0 ms | 0.45 ms |
+| A step (`flow.transaction`), one process | 2.0 ms | 2.8 ms | 0.45 ms |
+| Resuming a saga with 1,000 finished steps | 17 ms | 21 ms | 9 ms |
+| 2,000 sagas of 5 steps, 8 worker processes | 6.9 s (290 sagas/s) | 8.1 s (246 sagas/s) | 1,000 sagas, 4 workers: 6.5 s |
+| The same, killing a worker with SIGKILL every second | 9.2 s, 9 workers killed | 13.2 s, 13 killed | — |
+| Duplicate charges | 0 | 0 | 0 |
+
+A step costs about four queries: the lease check that fences the write, the notebook row and the commit that makes
+it durable. In the load tests each call to the outside world takes 5 ms, so a saga spends most of its time waiting,
+as in a real app. When a worker dies mid-step, another one runs that step again with the same ticket: on PostgreSQL 2
+of the 2,000 charges were sent twice, and the idempotency key made them one.
+
+The same test in a Rails 8 app with Solid Queue: 300 checkouts, every Solid Queue process killed with SIGKILL twice
+while sagas were halfway through. All of them settled, the refunds and hooks ran once, and no order was charged twice.
+
 ## Guarantees and limits
 
 - A step runs **at least once**; with an idempotency key it has its effect once. `flow.transaction` runs exactly
@@ -687,7 +901,19 @@ Every feature works on every version. Things your app may need on older Rails, u
 - A hook (`flow.on`) runs once; exactly once if it only touches your database.
 - A bug never undoes a saga: an error in the code blocks it until you fix it and call `ActiveDurable.retry`.
 - Sagas do not isolate each other: two sagas can see each other's intermediate states.
+- `Durable.start(id:)` is idempotent while the execution exists. Once it is pruned (`keep_finished_for`, 30 days),
+  the same id starts a new saga with the same tickets: keep finished executions longer than a webhook can be
+  delivered again.
+- The lease is renewed on each notebook write. A step longer than `lease_duration` can run twice at the same time,
+  with the same ticket. Keep the workers' clocks in sync (NTP): the lease compares times written by different
+  machines.
+- Error messages are stored in the notebook and reach the dashboard, the logs, the `blocked` event and
+  OpenTelemetry: keep secrets and personal data out of them.
+- Ids cannot be blank or contain `/`. Ids and step names compare exactly, also on MySQL.
 - `flow.transaction` is atomic only when the notebook lives in the same database as your data.
+- Step results, inputs and signals come back with their hash keys sorted, on the first run and on every replay, since
+  PostgreSQL and MySQL do not keep the order you wrote them in.
+- A replay reads the whole notebook. For sagas that grow without end, such as a chat, use `flow.continue_as_new`.
 
 ## Development
 

data/app/controllers/active_durable/executions_controller.rb

@@ -24,6 +24,7 @@ module ActiveDurable
       steps = @execution.steps.to_a
       @steps = steps
       @forward_steps = steps.select(&:forward?).sort_by { |step| step.position.to_i }
+      @rerun_steps = @forward_steps.select(&:position) # a parallel branch cannot be a starting point
       @undo_steps = steps.select(&:undo?)
       @hook_steps = steps.select(&:hook?)
       @signals = @execution.signals.to_a
@@ -32,7 +33,7 @@ module ActiveDurable
 
     def retry_now
       ActiveDurable.retry(params[:id])
-      redirect_to execution_path(params[:id]), notice: "Retrying. Failed steps got a fresh set of attempts."
+      redirect_to execution_path(params[:id]), notice: "Retrying. The step that blocked it got a fresh set of attempts."
     end
 
     def compensate

data/app/helpers/active_durable/dashboard_helper.rb

@@ -8,14 +8,14 @@ module ActiveDurable
 
     STATUS_TONES = {
       "completed" => "good", "compensated" => "undo", "blocked" => "bad", "superseded" => "muted",
-      "running" => "busy", "pending" => "busy", "sleeping" => "wait", "waiting" => "wait",
+      "continued" => "muted", "running" => "busy", "pending" => "busy", "sleeping" => "wait", "waiting" => "wait",
       "retrying" => "wait", "failed" => "bad", "undone" => "undo"
     }.freeze
 
     STATUS_WORDS = {
       "pending" => "Queued", "running" => "Running", "sleeping" => "Sleeping", "waiting" => "Waiting",
       "completed" => "Completed", "compensated" => "Undone", "blocked" => "Needs a person",
-      "superseded" => "Superseded"
+      "superseded" => "Superseded", "continued" => "Continued"
     }.freeze
 
     def status_pill(status)
@@ -57,6 +57,13 @@ module ActiveDurable
       safe_join(name.to_s.split(/(?<=_)/), tag.wbr)
     end
 
+    # Ids with a slash cannot be routed; versions before 0.7 accepted them, so they are listed without a link.
+    def execution_link(id, **options)
+      return tag.span(id, title: "Ids with a slash have no page; use the console", **options) if id.include?("/")
+
+      link_to(id, execution_path(id), **options)
+    end
+
     def ticket_for(execution, step)
       "#{execution.id}:#{step.name}"
     end
@@ -75,7 +82,7 @@ module ActiveDurable
     end
 
     def can_compensate?(execution, steps)
-      %w[blocked pending running sleeping waiting].include?(execution.status) && !execution.compensating &&
+      %w[blocked pending sleeping waiting].include?(execution.status) && !execution.compensating &&
         steps.none? { |step| step.kind == "pivot" && step.completed? }
     end
 
@@ -109,7 +116,8 @@ module ActiveDurable
 
     def state_word(state)
       {
-        "completed" => "done", "failed" => "failed", "retrying" => "retrying", "waiting" => "waiting for a signal",
+        "completed" => "done", "failed" => "failed", "blocked" => "blocked", "retrying" => "retrying",
+        "waiting" => "waiting for a signal",
         "sleeping" => "sleeping", "undone" => "undone", "next" => "up next", "running" => "running"
       }.fetch(state.to_s, state.to_s)
     end

data/app/views/active_durable/executions/index.html.erb

@@ -43,7 +43,7 @@
           <% items = track_items(execution, @steps_by_execution.fetch(execution.id, [])) %>
           <tr data-id="<%= execution.id %>" data-status="<%= execution.status %>">
             <td class="exec">
-              <%= link_to execution.id, execution_path(execution), class: "exec-link" %>
+              <%= execution_link(execution.id, class: "exec-link") %>
               <span class="recipe"><%= execution.recipe %>, version <%= execution.recipe_version %></span>
             </td>
             <td>

data/app/views/active_durable/executions/show.html.erb

@@ -15,7 +15,10 @@
       Started <%= when_text(@execution.created_at) %>, last change <%= when_text(@execution.updated_at) %>.
       <% if @execution.wake_at %>Wakes <%= when_text(@execution.wake_at) %>.<% end %>
       <% if @execution.forked_from %>
-        Rerun of <%= link_to @execution.forked_from, execution_path(@execution.forked_from), class: "mono" %>.
+        Rerun of <%= execution_link(@execution.forked_from, class: "mono") %>.
+      <% end %>
+      <% if @execution.continued? %>
+        Continued as <%= execution_link(@execution.continued_as, class: "mono") %>.
       <% end %>
     </p>
   </div>
@@ -29,7 +32,7 @@
   </section>
 <% end %>
 
-<% if can_retry?(@execution) || can_compensate?(@execution, @forward_steps) || (can_rerun?(@execution) && @forward_steps.any?) %>
+<% if can_retry?(@execution) || can_compensate?(@execution, @forward_steps) || (can_rerun?(@execution) && @rerun_steps.any?) %>
   <section class="actions" aria-label="Actions">
     <% if can_retry?(@execution) %>
       <%= button_to retry_execution_path(@execution), class: "primary", form_class: "inline" do %>Retry<% end %>
@@ -40,10 +43,10 @@
         Undo everything
       <% end %>
     <% end %>
-    <% if can_rerun?(@execution) && @forward_steps.any? %>
+    <% if can_rerun?(@execution) && @rerun_steps.any? %>
       <%= form_with url: rerun_execution_path(@execution), method: :post, class: "inline", local: true do %>
         <label for="rerun-from">Run again from</label>
-        <%= select_tag :from, options_for_select(@forward_steps.map(&:name), rerun_default(@forward_steps)), id: "rerun-from" %>
+        <%= select_tag :from, options_for_select(@rerun_steps.map(&:name), rerun_default(@rerun_steps)), id: "rerun-from" %>
         <button type="submit">Rerun</button>
       <% end %>
       <p class="note">A rerun is a new execution. Steps before the one you choose are reused; that step and the ones
@@ -65,7 +68,7 @@
         <% items.each_with_index do |item, index| %>
           <% if index.positive? %>
             <% previous = items[index - 1] %>
-            <% if previous.kind == "pivot" && previous.state != "failed" %>
+            <% if previous.kind == "pivot" && !%w[failed blocked].include?(previous.state) %>
               <span class="gate" style="--i: <%= index %>"><span>point of no return</span></span>
             <% end %>
             <% wire = if item.state == "undone" || (backwards && previous.state == "undone") then "back"
@@ -199,7 +202,7 @@
       <section class="panel">
         <h2>Reruns</h2>
         <% @reruns.each do |rerun| %>
-          <p><%= link_to rerun.id, execution_path(rerun), class: "mono" %> <%= status_pill(rerun.status) %></p>
+          <p><%= execution_link(rerun.id, class: "mono") %> <%= status_pill(rerun.status) %></p>
         <% end %>
       </section>
     <% end %>

data/app/views/layouts/active_durable/application.html.erb

@@ -33,7 +33,7 @@
     .tone-good, .st-completed { --c: var(--jade); --cb: var(--jade-bg); }
     .tone-wait, .st-retrying, .st-waiting, .st-sleeping { --c: var(--amber); --cb: var(--amber-bg); }
     .tone-busy, .st-running, .st-next { --c: var(--cobalt); --cb: var(--cobalt-bg); }
-    .tone-bad, .st-failed { --c: var(--ruby); --cb: var(--ruby-bg); }
+    .tone-bad, .st-failed, .st-blocked { --c: var(--ruby); --cb: var(--ruby-bg); }
     .tone-undo, .st-undone { --c: var(--violet); --cb: var(--violet-bg); }
     .tone-muted { --c: var(--slate); --cb: var(--slate-bg); }
 
@@ -140,7 +140,7 @@
                animation-delay: calc(var(--i, 0) * 45ms); }
     .mini .b.st-completed { background: var(--c); }
     .mini .b.k-pivot { transform: rotate(45deg) scale(.86); animation-name: pop-pivot; }
-    .mini .b.st-failed { animation: pop .35s both, shake .5s .5s ease 2; }
+    .mini .b.st-failed, .mini .b.st-blocked { animation: pop .35s both, shake .5s .5s ease 2; }
     .mini .b.st-undone { background: repeating-linear-gradient(135deg, var(--cb) 0 3px, var(--c) 3px 5px); }
     .mini .b.st-next { border-style: dashed; background: transparent; animation: pop .35s both, beat 1.2s ease-in-out infinite; }
     .mini .b.st-retrying, .mini .b.st-sleeping { animation: pop .35s both, beat 2.2s ease-in-out infinite; }
@@ -194,7 +194,7 @@
     .block.st-completed { --cb: color-mix(in srgb, var(--jade-bg) 85%, var(--surface)); }
     .block.k-pivot { border-width: 3px; }
     .block.k-pivot .kind { color: var(--amber); }
-    .block.st-failed { animation: drop .55s both, shake .5s .7s ease 2; box-shadow: 0 7px 0 var(--c2), 0 0 0 6px color-mix(in srgb, var(--ruby) 18%, transparent); }
+    .block.st-failed, .block.st-blocked { animation: drop .55s both, shake .5s .7s ease 2; box-shadow: 0 7px 0 var(--c2), 0 0 0 6px color-mix(in srgb, var(--ruby) 18%, transparent); }
     .block.st-undone { background: repeating-linear-gradient(135deg, var(--violet-bg) 0 10px, color-mix(in srgb, var(--violet) 22%, var(--violet-bg)) 10px 20px); }
     .block.st-undone h3 { text-decoration: line-through; text-decoration-thickness: 2px; }
     .block.st-next { background: transparent; border-style: dashed; box-shadow: none; animation: drop .55s both, beat 1.4s ease-in-out infinite; }