active_durable 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d6ce41040ab6f0cb0036bfe36b30b831a1f91a7da099b9f9e3914568716ef6e6
-  data.tar.gz: ac2223d523ec0edf3470b8b9b66b5a892d2521156ab0446e874feff3104c59ff
+  metadata.gz: ca565450baafb042b5ec363d2d7c101f24edd3281b180d43c173eaeff8dd8df8
+  data.tar.gz: 96ceecb07cf0d85b0d8aaf604dececee56301983d16055de408e533c53059da6
 SHA512:
-  metadata.gz: c95a763c1808a55ceebf13c02aa1e53f29830601b2547f6175c6bb375f1b78b9a5c5cd02aa2da505afac8989259f2047f588448aef34ce3bdb98a61dc2037e85
-  data.tar.gz: f12d9dff80e6c7e77bdaa40cd3683356bfc443b04eb442d59a5ca2e8e3da5cd9ecf2f0f97df180014c41c723b49ffb6b80fcfc111833f02994856fef460f61c3
+  metadata.gz: a6dafa621dba36aac0429fc264922d876d378390aa7fccebf05a085bbcb1f2e001d7bf57d202075299e47cbbb20d29af86df62000adca22367bec76ab18dd16b
+  data.tar.gz: 7b4c90ddb5104811ab17be9532fe2c4ca029bae139eeeaafdef1bd04a222fa4c2896c34b2ce44376d3ff13222473e76276a991c125fcdbe057937d7e6a649896

data/CHANGELOG.md

@@ -5,7 +5,68 @@ All notable changes to this project are documented here. The format follows
 [Semantic Versioning](https://semver.org/). Schema changes ship as new migrations: after updating the gem, run
 `bin/rails generate active_durable:upgrade` and `bin/rails db:migrate`.
 
-## [Unreleased]
+## [0.7.0] - 2026-10-07
+
+Fixes from an architecture and security review. Most of them stop a saga from paying twice or undoing the wrong
+thing.
+
+Upgrading from 0.6: `bin/rails generate active_durable:upgrade && bin/rails db:migrate`. The new migration only
+changes MySQL databases: it rebuilds the three tables and blocks writes to them while it runs, so run it at a quiet
+time, or prune first. Note the behavior changes below: more errors block instead of undoing, `retry` no longer
+resets every failed step, rerun ids are random, and `Durable.start` refuses some ids.
+
+### Changed
+
+- More errors count as bugs and block the saga instead of being retried and undone: `ArgumentError`, `TypeError`,
+  `IndexError` and `KeyError`, `FrozenError`, `ZeroDivisionError`, `RangeError`, `NoMatchingPatternError`,
+  `LocalJumpError`, `RegexpError` and `EncodingError`, besides `NameError` and `NoMethodError`. The list is the new
+  `config.code_errors`, which takes classes or names, so an app can add its own errors or drop one.
+- A step that blocks is written in the notebook as `blocked`, and runs again after `ActiveDurable.retry`. Rescuing
+  its error in the recipe no longer lets the saga carry on.
+- `ActiveDurable.retry` gives a fresh set of attempts only to the step that blocked the saga (with its whole
+  `flow.parallel`) and to steps that hit a bug. Compensating, it still resets every failed undo.
+- Rerun ids are `<id>~rerun-<random>` instead of `<id>~rerun-<count>`.
+- `ActiveDurable.compensate` refuses `running` executions, even after their lease ran out.
+- `Durable.start` refuses ids that are blank, contain `/`, a NUL character or bytes that are not UTF-8.
+- The serializer converts binary strings that are valid UTF-8 (such as `Net::HTTP` bodies) and strings in other
+  encodings to UTF-8, and rejects NUL characters and invalid bytes, in values and keys.
+
+### Fixed
+
+- `retry` ran again a failed step the recipe had already handled: with a fallback (Stripe fails, the recipe pays
+  with PayPal) and a later bug, retrying charged twice. `rerun` did the same; it now copies the failed steps before
+  the chosen one too.
+- A step whose result the database refused (a NUL on PostgreSQL, bytes that are not UTF-8 on MySQL and SQLite, any
+  failed write) counted as a failed attempt: it ran again, then the saga was undone without undoing it. It now
+  blocks.
+- Undoing a saga by hand skipped a step declared `undo_on_failure` that was between attempts or blocked, and the
+  finished branches of a `flow.parallel` stopped halfway.
+- An operator could undo a saga while its worker was still inside a step longer than `lease_duration`.
+- A pending signal nobody was waiting for (a duplicate webhook) made the saga enqueue itself again forever.
+- Signals sent to a blocked saga were refused and lost.
+- `rerun` lost the undos of the `flow.parallel` branches before the chosen step, and from a branch it ran nothing;
+  the dashboard no longer offers branches.
+- On MySQL, ids and step names that differ only in case or accents were treated as the same. The new migration
+  makes those columns `utf8mb4_bin`.
+- `raise ActiveRecord::Rollback` inside `flow.transaction`, an undo or a hook counted as success.
+- On Rails 6.1 and 7.0 with MySQL, two concurrent `start` calls with the same id inside the app's transaction made
+  the second raise `RecordNotFound`.
+- Rerun ids collided after pruning, and could give a new rerun the tickets of a pruned one.
+- An id with `/` broke the whole dashboard list.
+- `LoadError`, `NotImplementedError` and `SystemStackError` left the saga running, retried forever by the sweeper.
+- A bug in an undo used up all its attempts before blocking; it now blocks at once.
+- Error messages are cleaned up (invalid bytes, NUL) before they are stored.
+- README: how to run Solid Queue in development, and the sweeper and cleanup entries go under the `production:` key
+  Rails already wrote in `config/recurring.yml`. Pasting a second `production:` key silently dropped Rails' own
+  `clear_solid_queue_finished_jobs` task.
+
+### Added
+
+- Benchmarks in `benchmarks/`: the cost of a step and of resuming a long saga (`throughput.rb`), and many worker
+  processes on the same sagas, optionally killing one with SIGKILL every second (`load.rb`, `CHAOS=1`). The README
+  has a "Performance" section with the results on PostgreSQL, MySQL and SQLite.
+- README: what counts as a bug, `config.code_errors`, and new limits: ids are forgotten after pruning, the lease and
+  slow steps, clocks, and secrets in error messages.
 
 ## [0.6.0] - 2026-10-06
 

data/README.md

@@ -161,12 +161,22 @@ config.solid_queue.connects_to = { database: { writing: :queue } }
   The server runs its own, sleeps and retries included, but a saga started from the console, `bin/rails runner`, a
   rake task or `db/seeds.rb` is lost when that process exits, and scheduled wake-ups are lost when the server
   restarts. A minute later, `bin/rails active_durable:sweep` picks them up: with `:async` it runs them right there.
-  Or run Solid Queue in development too, with `bin/jobs` next to the server.
+  Or run Solid Queue in development too, so jobs live in the database: give `development` a `queue` database in
+  `config/database.yml` (like the one `production` has, with `migrations_paths: db/queue_migrate`), add the two
+  lines below, run `bin/rails db:prepare`, and start `bin/jobs` next to the server.
+
+```ruby
+# config/environments/development.rb
+config.active_job.queue_adapter = :solid_queue
+config.solid_queue.connects_to = { database: { writing: :queue } }
+```
 
 ### 3. The sweeper
 
 The safety net: every minute it enqueues executions that lost their job, because the process died between the
-commit and the enqueue or a worker died holding a lease. With Solid Queue:
+commit and the enqueue or a worker died holding a lease. With Solid Queue, add these entries under the
+`production:` key Rails already wrote in `config/recurring.yml` (a second `production:` key would silently replace
+it), and under a `development:` key too if you run Solid Queue in development:
 
 ```yaml
 # config/recurring.yml
@@ -202,6 +212,7 @@ ActiveDurable.configure do |config|
   config.parallel_concurrency = 4       # threads per flow.parallel
   config.sweep_grace = 1.minute         # the sweeper leaves executions this young alone
   config.keep_finished_for = 30.days    # then ActiveDurable::PruneJob deletes finished ones
+  config.code_errors += ["Payments::Misconfigured"] # your own errors that mean a bug (see "A bug is not a failure")
 
   # Who may open the dashboard outside development and test. With Devise (HTTP basic auth: see Dashboard):
   config.dashboard_authorize = ->(controller) { controller.request.env["warden"]&.user&.admin? }
@@ -428,10 +439,33 @@ When a step runs out of attempts or calls `flow.abort!`, every finished step is
 stopped. An undo receives `(result, undo_ticket, step_ticket)` and takes as many as it declares: `-> { ... }` takes
 none. Any object that responds to `call` works too, such as `Payments.method(:refund)`.
 
-**A bug is not a failure.** Anything else the recipe raises, and a `NameError` or `NoMethodError` inside a step,
-blocks the execution instead of undoing it: a typo in a deploy must never refund your customers. Fix the code and
-call `ActiveDurable.retry(id)`, or press Retry in the dashboard, and the saga carries on from where it stopped. To
-reject the work for a business reason outside a step, call `flow.abort!(reason)`.
+**A bug is not a failure.** Undoing a saga refunds money and releases stock, so only a step that failed for good or
+`flow.abort!` triggers it. These block the execution instead, without retrying, because a typo in a deploy must never
+refund your customers:
+
+- anything the recipe raises outside a step;
+- inside a step, an error that means the code is wrong: `NameError` (and `NoMethodError`), `ArgumentError`,
+  `TypeError`, `IndexError` (and `KeyError`), `FrozenError`, `ZeroDivisionError`, `RangeError`,
+  `NoMatchingPatternError`, `LocalJumpError`, `RegexpError`, `EncodingError`, and the errors outside
+  `StandardError` such as `LoadError`, `NotImplementedError` and `SystemStackError`;
+- a step whose result cannot be stored (a NUL character, bytes that are not UTF-8): it already acted.
+
+Fix the code and call `ActiveDurable.retry(id)`, or press Retry in the dashboard, and the saga carries on from where it
+stopped. Rescuing the error in the recipe does not help: the saga stops at the next step and blocks anyway. To reject
+the work for a business reason, call `flow.abort!(reason)`.
+
+The list is `config.code_errors`. Add your own errors, as a class or as a name (a name also matches subclasses, and
+works before the gem that defines it is loaded), or drop one:
+
+```ruby
+ActiveDurable.configure do |config|
+  config.code_errors << "Payments::Misconfigured"
+  config.code_errors -= ["ArgumentError"] # if a gem you call raises it for bad input
+end
+```
+
+An `ArgumentError` from bad data, like an invalid date, blocks too: a person decides. To treat one of these errors as
+a failure in a single step, rescue it inside the step and raise another error, or call `flow.abort!`.
 
 A step that failed is not undone, because it did not happen. The exception is a step whose failure may hide a
 success, like a charge whose answer timed out: declare `undo_on_failure: true` and its undo runs with `nil` as the
@@ -492,7 +526,7 @@ For progress before the end (paid, shipped), write a step: `flow.transaction(:ma
 <br>
 
 `flow.sleep` writes the wake-up time down and releases the worker. `flow.wait_for` does the same until a signal
-arrives. Signals can arrive before the saga starts waiting.
+arrives. Signals can arrive before the saga starts waiting, or while it is blocked: they are kept until it does.
 
 ```ruby
 kyc = flow.wait_for(:kyc_done, timeout: 2.hours)
@@ -585,14 +619,21 @@ end
 ## Operating sagas
 
 ```ruby
-ActiveDurable.retry("checkout-7")                                   # blocked: try again where it stopped
+ActiveDurable.retry("checkout-7")                                   # blocked: try again the step that blocked it
 ActiveDurable.compensate("checkout-7", reason: "customer cancelled") # undo everything (only before the pivot)
 ActiveDurable.rerun("checkout-7", from: :ship)                      # a new execution reusing steps before :ship
 ActiveDurable.prune(older_than: 30.days)                            # delete finished executions and their notebook
 ```
 
-All three refuse an execution a worker is running right now. A rerun runs the chosen step and the following ones
-again with new tickets, so they have effects again; a blocked original becomes `superseded`.
+`retry` gives the step that blocked the saga a fresh set of attempts. Failed steps the recipe already handled stay
+failed: if Stripe failed and the recipe paid with PayPal instead, a retry never runs Stripe again.
+
+A rerun (`<id>~rerun-<random>`) keeps the steps before the chosen one, failed ones and parallel branches included,
+and runs the chosen step and the following ones again with new tickets, so they have effects again. A blocked
+original becomes `superseded`.
+
+None of them touches an execution a worker is running. `compensate` refuses a `running` execution even after its
+lease ran out: the worker may still be inside a slow step, and if it died, the sweeper resumes it.
 
 ## Testing: the crash tester
 
@@ -679,6 +720,28 @@ Every feature works on every version. Things your app may need on older Rails, u
 | Temporal | the Temporal server | written by hand | a Temporal cluster |
 | **ActiveDurable** | **your database** | **yes, in reverse, with a point of no return** | **nothing extra** |
 
+## Performance
+
+Measured on an Apple M1 Ultra with Ruby 4.0.7 and Rails 8.1, each database on the same machine. The scripts are in
+[`benchmarks/`](benchmarks/README.md), so you can run them on yours.
+
+| | PostgreSQL 16 | MySQL 9.6 | SQLite 3 |
+| --- | --- | --- | --- |
+| A step (`flow.step`), one process | 1.8 ms | 3.0 ms | 0.45 ms |
+| A step (`flow.transaction`), one process | 2.0 ms | 2.8 ms | 0.45 ms |
+| Resuming a saga with 1,000 finished steps | 17 ms | 21 ms | 9 ms |
+| 2,000 sagas of 5 steps, 8 worker processes | 6.9 s (290 sagas/s) | 8.1 s (246 sagas/s) | 1,000 sagas, 4 workers: 6.5 s |
+| The same, killing a worker with SIGKILL every second | 9.2 s, 9 workers killed | 13.2 s, 13 killed | — |
+| Duplicate charges | 0 | 0 | 0 |
+
+A step costs about four queries: the lease check that fences the write, the notebook row and the commit that makes
+it durable. In the load tests each call to the outside world takes 5 ms, so a saga spends most of its time waiting,
+as in a real app. When a worker dies mid-step, another one runs that step again with the same ticket: on PostgreSQL 2
+of the 2,000 charges were sent twice, and the idempotency key made them one.
+
+The same test in a Rails 8 app with Solid Queue: 300 checkouts, every Solid Queue process killed with SIGKILL twice
+while sagas were halfway through. All of them settled, the refunds and hooks ran once, and no order was charged twice.
+
 ## Guarantees and limits
 
 - A step runs **at least once**; with an idempotency key it has its effect once. `flow.transaction` runs exactly
@@ -687,6 +750,15 @@ Every feature works on every version. Things your app may need on older Rails, u
 - A hook (`flow.on`) runs once; exactly once if it only touches your database.
 - A bug never undoes a saga: an error in the code blocks it until you fix it and call `ActiveDurable.retry`.
 - Sagas do not isolate each other: two sagas can see each other's intermediate states.
+- `Durable.start(id:)` is idempotent while the execution exists. Once it is pruned (`keep_finished_for`, 30 days),
+  the same id starts a new saga with the same tickets: keep finished executions longer than a webhook can be
+  delivered again.
+- The lease is renewed on each notebook write. A step longer than `lease_duration` can run twice at the same time,
+  with the same ticket. Keep the workers' clocks in sync (NTP): the lease compares times written by different
+  machines.
+- Error messages are stored in the notebook and reach the dashboard, the logs, the `blocked` event and
+  OpenTelemetry: keep secrets and personal data out of them.
+- Ids cannot be blank or contain `/`. Ids and step names compare exactly, also on MySQL.
 - `flow.transaction` is atomic only when the notebook lives in the same database as your data.
 
 ## Development

data/app/controllers/active_durable/executions_controller.rb

@@ -24,6 +24,7 @@ module ActiveDurable
       steps = @execution.steps.to_a
       @steps = steps
       @forward_steps = steps.select(&:forward?).sort_by { |step| step.position.to_i }
+      @rerun_steps = @forward_steps.select(&:position) # a parallel branch cannot be a starting point
       @undo_steps = steps.select(&:undo?)
       @hook_steps = steps.select(&:hook?)
       @signals = @execution.signals.to_a
@@ -32,7 +33,7 @@ module ActiveDurable
 
     def retry_now
       ActiveDurable.retry(params[:id])
-      redirect_to execution_path(params[:id]), notice: "Retrying. Failed steps got a fresh set of attempts."
+      redirect_to execution_path(params[:id]), notice: "Retrying. The step that blocked it got a fresh set of attempts."
     end
 
     def compensate

data/app/helpers/active_durable/dashboard_helper.rb

@@ -57,6 +57,13 @@ module ActiveDurable
       safe_join(name.to_s.split(/(?<=_)/), tag.wbr)
     end
 
+    # Ids with a slash cannot be routed; versions before 0.7 accepted them, so they are listed without a link.
+    def execution_link(id, **options)
+      return tag.span(id, title: "Ids with a slash have no page; use the console", **options) if id.include?("/")
+
+      link_to(id, execution_path(id), **options)
+    end
+
     def ticket_for(execution, step)
       "#{execution.id}:#{step.name}"
     end
@@ -75,7 +82,7 @@ module ActiveDurable
     end
 
     def can_compensate?(execution, steps)
-      %w[blocked pending running sleeping waiting].include?(execution.status) && !execution.compensating &&
+      %w[blocked pending sleeping waiting].include?(execution.status) && !execution.compensating &&
         steps.none? { |step| step.kind == "pivot" && step.completed? }
     end
 
@@ -109,7 +116,8 @@ module ActiveDurable
 
     def state_word(state)
       {
-        "completed" => "done", "failed" => "failed", "retrying" => "retrying", "waiting" => "waiting for a signal",
+        "completed" => "done", "failed" => "failed", "blocked" => "blocked", "retrying" => "retrying",
+        "waiting" => "waiting for a signal",
         "sleeping" => "sleeping", "undone" => "undone", "next" => "up next", "running" => "running"
       }.fetch(state.to_s, state.to_s)
     end

data/app/views/active_durable/executions/index.html.erb

@@ -43,7 +43,7 @@
           <% items = track_items(execution, @steps_by_execution.fetch(execution.id, [])) %>
           <tr data-id="<%= execution.id %>" data-status="<%= execution.status %>">
             <td class="exec">
-              <%= link_to execution.id, execution_path(execution), class: "exec-link" %>
+              <%= execution_link(execution.id, class: "exec-link") %>
               <span class="recipe"><%= execution.recipe %>, version <%= execution.recipe_version %></span>
             </td>
             <td>

data/app/views/active_durable/executions/show.html.erb

@@ -15,7 +15,7 @@
       Started <%= when_text(@execution.created_at) %>, last change <%= when_text(@execution.updated_at) %>.
       <% if @execution.wake_at %>Wakes <%= when_text(@execution.wake_at) %>.<% end %>
       <% if @execution.forked_from %>
-        Rerun of <%= link_to @execution.forked_from, execution_path(@execution.forked_from), class: "mono" %>.
+        Rerun of <%= execution_link(@execution.forked_from, class: "mono") %>.
       <% end %>
     </p>
   </div>
@@ -29,7 +29,7 @@
   </section>
 <% end %>
 
-<% if can_retry?(@execution) || can_compensate?(@execution, @forward_steps) || (can_rerun?(@execution) && @forward_steps.any?) %>
+<% if can_retry?(@execution) || can_compensate?(@execution, @forward_steps) || (can_rerun?(@execution) && @rerun_steps.any?) %>
   <section class="actions" aria-label="Actions">
     <% if can_retry?(@execution) %>
       <%= button_to retry_execution_path(@execution), class: "primary", form_class: "inline" do %>Retry<% end %>
@@ -40,10 +40,10 @@
         Undo everything
       <% end %>
     <% end %>
-    <% if can_rerun?(@execution) && @forward_steps.any? %>
+    <% if can_rerun?(@execution) && @rerun_steps.any? %>
       <%= form_with url: rerun_execution_path(@execution), method: :post, class: "inline", local: true do %>
         <label for="rerun-from">Run again from</label>
-        <%= select_tag :from, options_for_select(@forward_steps.map(&:name), rerun_default(@forward_steps)), id: "rerun-from" %>
+        <%= select_tag :from, options_for_select(@rerun_steps.map(&:name), rerun_default(@rerun_steps)), id: "rerun-from" %>
         <button type="submit">Rerun</button>
       <% end %>
       <p class="note">A rerun is a new execution. Steps before the one you choose are reused; that step and the ones
@@ -65,7 +65,7 @@
         <% items.each_with_index do |item, index| %>
           <% if index.positive? %>
             <% previous = items[index - 1] %>
-            <% if previous.kind == "pivot" && previous.state != "failed" %>
+            <% if previous.kind == "pivot" && !%w[failed blocked].include?(previous.state) %>
               <span class="gate" style="--i: <%= index %>"><span>point of no return</span></span>
             <% end %>
             <% wire = if item.state == "undone" || (backwards && previous.state == "undone") then "back"
@@ -199,7 +199,7 @@
       <section class="panel">
         <h2>Reruns</h2>
         <% @reruns.each do |rerun| %>
-          <p><%= link_to rerun.id, execution_path(rerun), class: "mono" %> <%= status_pill(rerun.status) %></p>
+          <p><%= execution_link(rerun.id, class: "mono") %> <%= status_pill(rerun.status) %></p>
         <% end %>
       </section>
     <% end %>

data/app/views/layouts/active_durable/application.html.erb

@@ -33,7 +33,7 @@
     .tone-good, .st-completed { --c: var(--jade); --cb: var(--jade-bg); }
     .tone-wait, .st-retrying, .st-waiting, .st-sleeping { --c: var(--amber); --cb: var(--amber-bg); }
     .tone-busy, .st-running, .st-next { --c: var(--cobalt); --cb: var(--cobalt-bg); }
-    .tone-bad, .st-failed { --c: var(--ruby); --cb: var(--ruby-bg); }
+    .tone-bad, .st-failed, .st-blocked { --c: var(--ruby); --cb: var(--ruby-bg); }
     .tone-undo, .st-undone { --c: var(--violet); --cb: var(--violet-bg); }
     .tone-muted { --c: var(--slate); --cb: var(--slate-bg); }
 
@@ -140,7 +140,7 @@
                animation-delay: calc(var(--i, 0) * 45ms); }
     .mini .b.st-completed { background: var(--c); }
     .mini .b.k-pivot { transform: rotate(45deg) scale(.86); animation-name: pop-pivot; }
-    .mini .b.st-failed { animation: pop .35s both, shake .5s .5s ease 2; }
+    .mini .b.st-failed, .mini .b.st-blocked { animation: pop .35s both, shake .5s .5s ease 2; }
     .mini .b.st-undone { background: repeating-linear-gradient(135deg, var(--cb) 0 3px, var(--c) 3px 5px); }
     .mini .b.st-next { border-style: dashed; background: transparent; animation: pop .35s both, beat 1.2s ease-in-out infinite; }
     .mini .b.st-retrying, .mini .b.st-sleeping { animation: pop .35s both, beat 2.2s ease-in-out infinite; }
@@ -194,7 +194,7 @@
     .block.st-completed { --cb: color-mix(in srgb, var(--jade-bg) 85%, var(--surface)); }
     .block.k-pivot { border-width: 3px; }
     .block.k-pivot .kind { color: var(--amber); }
-    .block.st-failed { animation: drop .55s both, shake .5s .7s ease 2; box-shadow: 0 7px 0 var(--c2), 0 0 0 6px color-mix(in srgb, var(--ruby) 18%, transparent); }
+    .block.st-failed, .block.st-blocked { animation: drop .55s both, shake .5s .7s ease 2; box-shadow: 0 7px 0 var(--c2), 0 0 0 6px color-mix(in srgb, var(--ruby) 18%, transparent); }
     .block.st-undone { background: repeating-linear-gradient(135deg, var(--violet-bg) 0 10px, color-mix(in srgb, var(--violet) 22%, var(--violet-bg)) 10px 20px); }
     .block.st-undone h3 { text-decoration: line-through; text-decoration-thickness: 2px; }
     .block.st-next { background: transparent; border-style: dashed; box-shadow: none; animation: drop .55s both, beat 1.4s ease-in-out infinite; }

data/lib/active_durable/configuration.rb

@@ -39,6 +39,19 @@ module ActiveDurable
     # development and test only: closed in production, staging and any other environment.
     attr_accessor :dashboard_authorize
 
+    # Errors that mean the code is wrong, not the outside world. A step that raises one of them is neither retried
+    # nor undone: the execution is blocked until the code is fixed and someone calls ActiveDurable.retry. Classes
+    # or class names; a name also matches subclasses and needs no loaded gem. Add your own with
+    # `config.code_errors << "Payments::Misconfigured"`, or drop one with `config.code_errors -= ["ArgumentError"]`.
+    # Errors outside StandardError (LoadError, NotImplementedError, SystemStackError) always count as bugs.
+    attr_accessor :code_errors
+
+    # The default {#code_errors}. NameError covers NoMethodError, and IndexError covers KeyError.
+    DEFAULT_CODE_ERRORS = %w[
+      NameError ArgumentError TypeError IndexError FrozenError ZeroDivisionError RangeError
+      NoMatchingPatternError LocalJumpError RegexpError EncodingError
+    ].freeze
+
     attr_writer :logger
 
     def initialize
@@ -53,6 +66,7 @@ module ActiveDurable
       @parallel_concurrency = 4
       @clock = -> { Time.current }
       @dashboard_authorize = nil
+      @code_errors = DEFAULT_CODE_ERRORS.dup
       @logger = nil
     end
 

data/lib/active_durable/errors.rb

@@ -18,7 +18,10 @@ module ActiveDurable
   class RecipeChanged < Error; end
 
   # A step returned something that cannot be stored in the notebook as JSON.
-  class NotSerializable < Error; end
+  class NotSerializable < Error
+    # @api private
+    attr_accessor :step_name
+  end
 
   # Raise it inside a step to reject the work for a business reason: no retries, straight to compensation.
   class Abort < Error; end
@@ -37,8 +40,15 @@ module ActiveDurable
     end
   end
 
-  # An undo kept failing after all its attempts. The execution is blocked for a human to review.
-  class UndoFailed < Error; end
+  # An undo kept failing after all its attempts, or hit a bug. The execution is blocked for a human to review.
+  class UndoFailed < Error
+    attr_reader :step_name
+
+    def initialize(message = nil, step_name: nil)
+      @step_name = step_name
+      super(message)
+    end
+  end
 
   # A flow.on(:completed) or flow.on(:compensated) hook raised. The execution is blocked; ActiveDurable.retry runs
   # the hook again once it is fixed.
@@ -51,9 +61,10 @@ module ActiveDurable
     end
   end
 
-  # A step whose code cannot run as written: it raised NameError or NoMethodError (a typo, a missing class or
-  # method). Retrying cannot fix it and undoing the saga would punish customers for a bug, so the execution is
-  # blocked until the code is fixed and someone calls ActiveDurable.retry.
+  # A step whose code cannot run as written: it raised one of {Configuration#code_errors} (a typo, a missing
+  # key, a wrong argument). Retrying cannot fix it and undoing the saga would punish customers for a bug, so the
+  # execution is blocked until the code is fixed and someone calls ActiveDurable.retry. Rescuing it in the recipe
+  # does not help: the saga stops at the next step and blocks anyway.
   class CodeError < Error
     attr_reader :step_name
 
@@ -63,6 +74,17 @@ module ActiveDurable
     end
   end
 
+  # A step ran, but the database refused to record its result. Running it again could repeat its effect and
+  # undoing the saga would skip it, so the execution is blocked at that step.
+  class CheckpointFailed < Error
+    attr_reader :step_name
+
+    def initialize(step_name, error)
+      @step_name = step_name
+      super("step :#{step_name} ran, but its result could not be recorded: #{error.class}: #{error.message}")
+    end
+  end
+
   # Control flow signals. They inherit from Exception on purpose: a `rescue => e` inside
   # user code must not swallow them, otherwise a lost lease could keep writing.
   #
@@ -79,13 +101,36 @@ module ActiveDurable
   # @api private
   class StopForward < ControlFlow; end
 
-  # Serializes an exception for the notebook and the dashboard.
+  # Serializes an exception for the notebook and the dashboard. The message is cleaned up first: it may quote the
+  # very bytes the database refused.
   def self.dump_error(error, step: nil)
     {
       "class" => error.class.name,
-      "message" => error.message.to_s[0, 2000],
+      "message" => storable_text(error.message)[0, 2000],
       "step" => step,
       "at" => now.utc.iso8601(6)
     }.compact
   end
+
+  # Whether an error means the code is wrong rather than the outside world: it is one of config.code_errors (or a
+  # subclass), or one of the errors Ruby raises outside StandardError, such as LoadError or SystemStackError.
+  #
+  # @api private
+  def self.code_error?(error)
+    return true unless error.is_a?(StandardError)
+
+    names = config.code_errors.map { |item| item.is_a?(Module) ? item.name : item.to_s }
+    error.class.ancestors.any? { |ancestor| names.include?(ancestor.name) }
+  end
+
+  # @api private
+  def self.storable_text(text)
+    text = text.to_s
+    text = if [Encoding::UTF_8, Encoding::BINARY, Encoding::US_ASCII].include?(text.encoding)
+             text.dup.force_encoding(Encoding::UTF_8).scrub("?")
+           else
+             text.encode(Encoding::UTF_8, invalid: :replace, undef: :replace, replace: "?")
+           end
+    text.delete("\u0000")
+  end
 end

data/lib/active_durable/execution.rb

@@ -11,6 +11,8 @@ module ActiveDurable
     # Statuses where nothing happens without a person: done, undone, blocked (needs {ActiveDurable.retry},
     # {ActiveDurable.compensate} or a fix), or replaced by a rerun.
     TERMINAL = %w[completed compensated blocked superseded].freeze
+    # Statuses that never change again: signals are refused and ActiveDurable.prune may delete them.
+    FINISHED = %w[completed compensated superseded].freeze
     # Every status.
     STATUSES = (ACTIVE + TERMINAL).freeze
 
@@ -42,6 +44,11 @@ module ActiveDurable
       TERMINAL.include?(status)
     end
 
+    # @return [Boolean] whether it will never change again (blocked is not finished: it can be retried)
+    def finished?
+      FINISHED.include?(status)
+    end
+
     # The forward steps in recipe order (undo and hook entries excluded).
     def notebook
       steps.where.not(kind: %w[undo hook]).reorder(:position).to_a

data/lib/active_durable/flow.rb

@@ -32,6 +32,12 @@ module ActiveDurable
     # @api private
     attr_reader :undo_stack
 
+    # The bug (or unrecorded result) that stopped this run. Once set, no other step runs and the execution is
+    # blocked, even if the recipe rescued the error.
+    #
+    # @api private
+    attr_reader :blocked_by
+
     # @api private
     def initialize(runner, compensating:)
       @runner = runner
@@ -43,6 +49,7 @@ module ActiveDurable
       @seen = {}
       @undo_stack = []
       @hooks = {}
+      @blocked_by = nil
     end
 
     # @return [String] the id of the execution this recipe is running for
@@ -206,7 +213,7 @@ module ActiveDurable
       signal = SignalRecord.next_for(execution_id, name)
       return consume_signal(signal, name, position).deep_dup if signal
 
-      if entry.nil?
+      unless entry&.waiting? # first time here, or retried after a timeout: wait again, with a new deadline
         deadline = timeout && (now + timeout)
         @notebook.wait!(name, kind: "wait", position: position, wake_at: deadline)
         @runner.suspend!(deadline, "waiting")
@@ -226,6 +233,7 @@ module ActiveDurable
     #
     # @api private
     def finish!
+      raise blocked_by if blocked_by
       return if compensating?
 
       missing = @notebook.forward_entries.reject { |entry| @seen.key?(entry.name) }
@@ -267,7 +275,11 @@ module ActiveDurable
         remember_failure(name, kind, undo, options)
         raise StepFailed.new(name, entry.error&.fetch("message", nil))
       end
-      raise StopForward, name if compensating?
+      if compensating?
+        # Stopped between attempts or on a bug: it may have acted, so undo_on_failure applies.
+        remember_failure(name, kind, undo, options) if entry&.unfinished?
+        raise StopForward, name
+      end
 
       @runner.suspend!(entry.wake_at, "sleeping") if entry&.retrying? && entry.wake_at && entry.wake_at > now
 
@@ -280,28 +292,51 @@ module ActiveDurable
       ActiveDurable.crash_point(:before_step, name)
       result = ActiveDurable.instrument("step", execution_id: execution_id, step: name, kind: kind) do
         if kind == "transaction"
-          @notebook.transaction { record_result(name, kind, position, block.call(ticket)) }
+          @notebook.transaction(records: name) { record_result(name, kind, position, block.call(ticket)) }
         else
           record_result(name, kind, position, block.call(ticket))
         end
       end
       ActiveDurable.crash_point(:after_record, name)
       result
-    rescue NotSerializable, InvalidRecipe
-      raise
-    rescue NameError => e # NoMethodError too: a bug in the code, not a failure of the outside world
-      raise CodeError.new(name, e)
-    rescue StandardError => e
+    rescue Abort => e
+      handle_failure(name, kind, position, entry, undo, options, e)
+    rescue NotSerializable, InvalidRecipe, CheckpointFailed => e
+      block_step!(name, kind, position, entry, e)
+    rescue StandardError, ScriptError, SystemStackError => e
+      raise if blocked_by # a nested step already blocked the run
+
+      return block_step!(name, kind, position, entry, CodeError.new(name, e)) if ActiveDurable.code_error?(e)
+
       handle_failure(name, kind, position, entry, undo, options, e)
     end
 
+    # Outside a transaction the step has already acted when its result is recorded, so a write the database
+    # refuses is not a failure of the step: it blocks. Inside flow.transaction it rolled back with the step.
     def record_result(name, kind, position, value)
       result = Serializer.normalize(value, "the result of :#{name}")
       ActiveDurable.crash_point(:after_call, name)
-      @notebook.complete!(name, kind: kind, position: position, result: result)
+      begin
+        @notebook.complete!(name, kind: kind, position: position, result: result)
+      rescue StandardError => e
+        raise if kind == "transaction"
+
+        raise CheckpointFailed.new(name, e)
+      end
       result
     end
 
+    # Retrying cannot fix a bug and undoing the saga would punish customers for it: the step is written down as
+    # blocked, and nothing else runs until someone fixes the code and calls ActiveDurable.retry. It keeps its
+    # attempts, since a bug is not a failure of the outside world.
+    def block_step!(name, kind, position, entry, error)
+      error.step_name ||= name if error.respond_to?(:step_name=)
+      @blocked_by = error
+      @notebook.block!(name, kind: kind, position: position, attempts: entry&.attempts || 0,
+                             error: ActiveDurable.dump_error(error, step: name))
+      raise error
+    end
+
     def handle_failure(name, kind, position, entry, undo, options, error)
       attempts = (entry&.attempts || 0) + 1
       default = @pivoted ? config.after_pivot_attempts : config.step_attempts
@@ -348,6 +383,8 @@ module ActiveDurable
     end
 
     def visit!(name, kind)
+      raise blocked_by if blocked_by
+
       name = name.to_s
       raise InvalidRecipe, "step names cannot be blank" if name.empty?
       raise InvalidRecipe, "step names cannot end in ':undo' (#{name})" if name.end_with?(":undo")