active_durable 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +131 -0
- data/LICENSE.txt +21 -0
- data/README.md +630 -0
- data/Rakefile +12 -0
- data/app/controllers/active_durable/application_controller.rb +25 -0
- data/app/controllers/active_durable/executions_controller.rb +59 -0
- data/app/helpers/active_durable/dashboard_helper.rb +163 -0
- data/app/views/active_durable/executions/index.html.erb +90 -0
- data/app/views/active_durable/executions/show.html.erb +197 -0
- data/app/views/layouts/active_durable/application.html.erb +422 -0
- data/config/routes.rb +13 -0
- data/lib/active_durable/configuration.rb +59 -0
- data/lib/active_durable/engine.rb +24 -0
- data/lib/active_durable/errors.rb +61 -0
- data/lib/active_durable/execution.rb +43 -0
- data/lib/active_durable/flow.rb +277 -0
- data/lib/active_durable/flow_parallel.rb +200 -0
- data/lib/active_durable/lease.rb +50 -0
- data/lib/active_durable/notebook.rb +100 -0
- data/lib/active_durable/open_telemetry.rb +94 -0
- data/lib/active_durable/operations.rb +111 -0
- data/lib/active_durable/parallel.rb +54 -0
- data/lib/active_durable/record.rb +13 -0
- data/lib/active_durable/registry.rb +89 -0
- data/lib/active_durable/retry_policy.rb +42 -0
- data/lib/active_durable/run_job.rb +12 -0
- data/lib/active_durable/runner.rb +157 -0
- data/lib/active_durable/serializer.rb +40 -0
- data/lib/active_durable/signal_record.rb +20 -0
- data/lib/active_durable/step.rb +34 -0
- data/lib/active_durable/sweep_job.rb +12 -0
- data/lib/active_durable/sweeper.rb +22 -0
- data/lib/active_durable/testing.rb +118 -0
- data/lib/active_durable/version.rb +5 -0
- data/lib/active_durable.rb +145 -0
- data/lib/generators/active_durable/install/install_generator.rb +27 -0
- data/lib/generators/active_durable/install/templates/create_active_durable_tables.rb.tt +60 -0
- data/lib/tasks/active_durable.rake +24 -0
- data/sig/active_durable.rbs +4 -0
- metadata +134 -0
data/README.md
ADDED
|
@@ -0,0 +1,630 @@
|
|
|
1
|
+
<p align="center"><b>English</b> · <a href="README.es.md">Español</a></p>
|
|
2
|
+
|
|
3
|
+
<p align="center">
|
|
4
|
+
<img src="docs/assets/hero.svg" alt="ActiveDurable: durable sagas for Rails. Finish the work or undo it in order, even if the server dies halfway." width="100%">
|
|
5
|
+
</p>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
<a href="https://github.com/williamromero/active_durable/actions/workflows/main.yml"><img src="https://github.com/williamromero/active_durable/actions/workflows/main.yml/badge.svg" alt="CI"></a>
|
|
9
|
+
<img src="https://img.shields.io/badge/ruby-3.1%2B-CC342D?logo=ruby&logoColor=white" alt="Ruby 3.1 and newer">
|
|
10
|
+
<img src="https://img.shields.io/badge/rails-6.1%2B-D30001?logo=rubyonrails&logoColor=white" alt="Rails 6.1 and newer">
|
|
11
|
+
<img src="https://img.shields.io/badge/PostgreSQL%20%C2%B7%20MySQL%20%C2%B7%20SQLite-tested-3DD6A0" alt="PostgreSQL, MySQL and SQLite">
|
|
12
|
+
<img src="https://img.shields.io/badge/no%20Redis-no%20extra%20servers-7EA6FF" alt="No Redis, no extra servers">
|
|
13
|
+
<a href="LICENSE.txt"><img src="https://img.shields.io/badge/license-MIT-B08CFF" alt="MIT license"></a>
|
|
14
|
+
</p>
|
|
15
|
+
|
|
16
|
+
<p align="center">
|
|
17
|
+
<a href="#quick-start">Quick start</a> ·
|
|
18
|
+
<a href="#in-a-rails-app">In a Rails app</a> ·
|
|
19
|
+
<a href="#how-it-works">How it works</a> ·
|
|
20
|
+
<a href="#the-building-blocks">Building blocks</a> ·
|
|
21
|
+
<a href="#dashboard">Dashboard</a> ·
|
|
22
|
+
<a href="#testing-the-crash-tester">Crash tester</a> ·
|
|
23
|
+
<a href="#compatibility">Compatibility</a>
|
|
24
|
+
</p>
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
A checkout reserves stock, charges a card, ships a parcel and sends an email. `ActiveRecord::Base.transaction`
|
|
29
|
+
can roll back your tables, but **a ROLLBACK cannot reach Stripe**. If the server dies after the charge, or the
|
|
30
|
+
carrier refuses the parcel, you end up with money taken and no order.
|
|
31
|
+
|
|
32
|
+
**ActiveDurable** turns that flow into a durable saga stored in your own database:
|
|
33
|
+
|
|
34
|
+
- **Nothing is done twice.** Every finished step is written down. After a crash, another worker continues where
|
|
35
|
+
the first one stopped.
|
|
36
|
+
- **Nothing is left half done.** If a step fails for good, the steps that finished are undone, last one first.
|
|
37
|
+
- **Some things cannot be undone.** Mark the point of no return; after it, steps are retried instead.
|
|
38
|
+
- **Nothing extra to run.** Active Record and Active Job: no Redis, no workflow server.
|
|
39
|
+
|
|
40
|
+
## See it happen
|
|
41
|
+
|
|
42
|
+
<p align="center">
|
|
43
|
+
<img src="docs/assets/crash.svg" alt="Animation: a checkout runs two steps, the server dies, a new worker reads the notebook, skips the finished steps and Stripe charges only once" width="100%">
|
|
44
|
+
</p>
|
|
45
|
+
|
|
46
|
+
<p align="center">
|
|
47
|
+
<img src="docs/assets/undo.svg" alt="Animation: the dispatch step fails before the point of no return, so the charge is refunded and then the stock is released" width="100%">
|
|
48
|
+
</p>
|
|
49
|
+
|
|
50
|
+
## Quick start
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
bundle add active_durable
|
|
54
|
+
bin/rails generate active_durable:install
|
|
55
|
+
bin/rails db:migrate
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Write the recipe once, in `app/sagas/`:
|
|
59
|
+
|
|
60
|
+
```ruby
|
|
61
|
+
# app/sagas/checkout_saga.rb
|
|
62
|
+
CheckoutSaga = Durable.define(:checkout) do |flow, order_id:|
|
|
63
|
+
order = Order.find(order_id)
|
|
64
|
+
|
|
65
|
+
# Touches only your database: committed together with its checkpoint, so it runs exactly once.
|
|
66
|
+
flow.transaction :reserve_stock, undo: -> { order.release_stock! } do
|
|
67
|
+
order.reserve_stock!
|
|
68
|
+
{ "reserved" => true }
|
|
69
|
+
end
|
|
70
|
+
|
|
71
|
+
# Talks to the outside world: the ticket is an idempotency key that never changes for this step.
|
|
72
|
+
payment = flow.step :charge, undo: ->(charge, ticket) { Payments.refund(charge, ticket) } do |ticket|
|
|
73
|
+
Payments.charge(order, ticket)
|
|
74
|
+
rescue Stripe::CardError => e
|
|
75
|
+
flow.abort!(e.message) # a declined card is not retried: the stock is released right away
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# The point of no return: before it failures are undone, after it steps are retried.
|
|
79
|
+
flow.pivot :dispatch do |ticket|
|
|
80
|
+
{ "tracking" => Carrier.ship(order.id, reference: ticket).tracking_number, "payment" => payment["id"] }
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
flow.step(:confirmation_email) { OrderMailer.shipped(order.id).deliver_now && true }
|
|
84
|
+
flow.sleep(:wait_for_delivery, 3.days) # no worker is held while it sleeps
|
|
85
|
+
flow.step(:ask_for_review) { ReviewMailer.ask(order.id).deliver_now && true }
|
|
86
|
+
end
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
The calls to Stripe live in a plain module, doing and undoing side by side. The recipe hands them the ticket:
|
|
90
|
+
|
|
91
|
+
```ruby
|
|
92
|
+
# app/services/payments.rb
|
|
93
|
+
module Payments
|
|
94
|
+
def self.charge(order, ticket)
|
|
95
|
+
intent = Stripe::PaymentIntent.create(
|
|
96
|
+
{ amount: order.total_cents, currency: "usd", customer: order.user.stripe_id, confirm: true },
|
|
97
|
+
{ idempotency_key: ticket }
|
|
98
|
+
)
|
|
99
|
+
{ "id" => intent.id } # written in the notebook, so it must fit in JSON
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
def self.refund(charge, ticket)
|
|
103
|
+
Stripe::Refund.create({ payment_intent: charge["id"] }, { idempotency_key: ticket })
|
|
104
|
+
end
|
|
105
|
+
end
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
Start it in the same transaction that creates the order. The saga is saved with the order and the job is enqueued
|
|
109
|
+
only after the commit, so a crash in between cannot lose it:
|
|
110
|
+
|
|
111
|
+
```ruby
|
|
112
|
+
Order.transaction do
|
|
113
|
+
order = Order.create!(order_params)
|
|
114
|
+
Durable.start(:checkout, order_id: order.id)
|
|
115
|
+
end
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
To run it for real, with a job backend, the sweeper, the initializer, the dashboard and tests, follow
|
|
119
|
+
[In a Rails app](#in-a-rails-app).
|
|
120
|
+
|
|
121
|
+
> `Durable` is a short alias for `ActiveDurable`. It is skipped if your app already defines a `Durable` constant.
|
|
122
|
+
|
|
123
|
+
## In a Rails app
|
|
124
|
+
|
|
125
|
+
Everything a Rails app needs, in order. Steps 1 to 5 are done once; step 6 is the code you write for each saga.
|
|
126
|
+
|
|
127
|
+
### 1. Install
|
|
128
|
+
|
|
129
|
+
Run the three commands from the [quick start](#quick-start). The migration creates `durable_executions`,
|
|
130
|
+
`durable_steps` and `durable_signals` in your **primary database**, next to your models: `flow.transaction` runs
|
|
131
|
+
exactly once only because the notebook and your data commit together. A queue in its own database, like Solid
|
|
132
|
+
Queue's in Rails 8, is fine.
|
|
133
|
+
|
|
134
|
+
### 2. A job backend
|
|
135
|
+
|
|
136
|
+
ActiveDurable runs on Active Job, so it uses the backend you already have. New Rails 8 apps come with Solid Queue;
|
|
137
|
+
on older apps, `bundle add solid_queue` and `bin/rails solid_queue:install` write these lines. With Sidekiq or
|
|
138
|
+
GoodJob, set their adapter instead.
|
|
139
|
+
|
|
140
|
+
```ruby
|
|
141
|
+
# config/environments/production.rb
|
|
142
|
+
config.active_job.queue_adapter = :solid_queue
|
|
143
|
+
config.solid_queue.connects_to = { database: { writing: :queue } }
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
- **Workers must listen to the saga queue.** It is `:default` unless you change `config.queue_name`; if you do, add
|
|
147
|
+
it to your workers (Solid Queue's `config/queue.yml`, Sidekiq's `-q`).
|
|
148
|
+
- **In development** Rails' default `:async` adapter runs jobs inside the server, sleeps and retries included. With
|
|
149
|
+
Solid Queue, run `bin/jobs` next to the server.
|
|
150
|
+
|
|
151
|
+
### 3. The sweeper
|
|
152
|
+
|
|
153
|
+
The safety net: every minute it enqueues executions that lost their job, because the process died between the
|
|
154
|
+
commit and the enqueue or a worker died holding a lease. With Solid Queue:
|
|
155
|
+
|
|
156
|
+
```yaml
|
|
157
|
+
# config/recurring.yml
|
|
158
|
+
production:
|
|
159
|
+
active_durable_sweep:
|
|
160
|
+
class: ActiveDurable::SweepJob
|
|
161
|
+
schedule: every minute
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
With another backend, schedule `ActiveDurable::SweepJob` in its own scheduler (GoodJob cron, sidekiq-cron), or run
|
|
165
|
+
`bin/rails active_durable:sweep` from cron.
|
|
166
|
+
|
|
167
|
+
### 4. The initializer
|
|
168
|
+
|
|
169
|
+
The settings, with their defaults:
|
|
170
|
+
|
|
171
|
+
```ruby
|
|
172
|
+
# config/initializers/active_durable.rb
|
|
173
|
+
ActiveDurable.configure do |config|
|
|
174
|
+
config.queue_name = :default # the queue of ActiveDurable::RunJob and SweepJob
|
|
175
|
+
config.lease_duration = 5.minutes # longer than your slowest step
|
|
176
|
+
config.step_attempts = 3 # before the pivot, then undo
|
|
177
|
+
config.after_pivot_attempts = 25 # after the pivot, then block
|
|
178
|
+
config.undo_attempts = 10 # then block
|
|
179
|
+
config.backoff = ->(attempt) { [2**attempt, 3600].min } # or [5, 30, 300], or a number
|
|
180
|
+
config.parallel_concurrency = 4 # threads per flow.parallel
|
|
181
|
+
config.sweep_grace = 1.minute # the sweeper leaves executions this young alone
|
|
182
|
+
|
|
183
|
+
# Who may open the dashboard outside development and test. With Devise (HTTP basic auth: see Dashboard):
|
|
184
|
+
config.dashboard_authorize = ->(controller) { controller.request.env["warden"]&.user&.admin? }
|
|
185
|
+
end
|
|
186
|
+
|
|
187
|
+
# Tell someone when a saga needs a person.
|
|
188
|
+
ActiveSupport::Notifications.subscribe("blocked.active_durable") do |event|
|
|
189
|
+
Sentry.capture_message("Saga blocked", extra: event.payload)
|
|
190
|
+
end
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
For traces, see [OpenTelemetry](#observability).
|
|
194
|
+
|
|
195
|
+
### 5. Routes
|
|
196
|
+
|
|
197
|
+
```ruby
|
|
198
|
+
# config/routes.rb
|
|
199
|
+
mount ActiveDurable::Engine => "/durable"
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
### 6. Your code
|
|
203
|
+
|
|
204
|
+
Each recipe lives in `app/sagas/<name>_saga.rb` and is assigned to `<Name>Saga`, so a worker can load
|
|
205
|
+
`:checkout` from `CheckoutSaga` by its name.
|
|
206
|
+
|
|
207
|
+
```text
|
|
208
|
+
app/
|
|
209
|
+
sagas/checkout_saga.rb the recipe
|
|
210
|
+
services/payments.rb charge and refund, side by side
|
|
211
|
+
services/place_order.rb creates the order and starts the saga
|
|
212
|
+
controllers/orders_controller.rb calls PlaceOrder and answers right away
|
|
213
|
+
```
|
|
214
|
+
|
|
215
|
+
The controller never calls Stripe. It starts the saga and answers at once; a job runs the steps.
|
|
216
|
+
|
|
217
|
+
```ruby
|
|
218
|
+
# app/services/place_order.rb
|
|
219
|
+
class PlaceOrder
|
|
220
|
+
def self.call(params)
|
|
221
|
+
Order.transaction do
|
|
222
|
+
order = Order.create!(params)
|
|
223
|
+
Durable.start(:checkout, id: "checkout-#{order.id}", order_id: order.id)
|
|
224
|
+
order
|
|
225
|
+
end
|
|
226
|
+
end
|
|
227
|
+
end
|
|
228
|
+
|
|
229
|
+
# app/controllers/orders_controller.rb
|
|
230
|
+
class OrdersController < ApplicationController
|
|
231
|
+
def create
|
|
232
|
+
redirect_to PlaceOrder.call(order_params)
|
|
233
|
+
end
|
|
234
|
+
|
|
235
|
+
def show
|
|
236
|
+
@order = Order.find(params[:id])
|
|
237
|
+
end
|
|
238
|
+
end
|
|
239
|
+
|
|
240
|
+
# app/models/order.rb
|
|
241
|
+
class Order < ApplicationRecord
|
|
242
|
+
def checkout
|
|
243
|
+
Durable.find("checkout-#{id}")
|
|
244
|
+
end
|
|
245
|
+
end
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
The `id:` ties the saga to its order, so the order page can show where the saga is:
|
|
249
|
+
|
|
250
|
+
```erb
|
|
251
|
+
<%# app/views/orders/show.html.erb %>
|
|
252
|
+
<% case @order.checkout.status %>
|
|
253
|
+
<% when "completed" %> Your order is confirmed.
|
|
254
|
+
<% when "compensated" %> We could not complete it and refunded you.
|
|
255
|
+
<% when "blocked" %> We are looking into it.
|
|
256
|
+
<% else %> Processing…
|
|
257
|
+
<% end %>
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
The customer does not see "card declined" in the same response: the page says "Processing…" and updates itself
|
|
261
|
+
with polling or Turbo Streams. In exchange, nobody is ever charged for half an order.
|
|
262
|
+
|
|
263
|
+
### 7. The job
|
|
264
|
+
|
|
265
|
+
You do not write one. Once the transaction commits, ActiveDurable enqueues its own `ActiveDurable::RunJob` with the
|
|
266
|
+
execution id, on the Active Job backend you already use (Solid Queue, Sidekiq, GoodJob…). Every time the saga wakes
|
|
267
|
+
up, after a sleep, a retry or a signal, it enqueues that job again. Retries belong to each step and are written in
|
|
268
|
+
the notebook, not to the job: if your backend retries the job too, the copy finds the lease taken and returns.
|
|
269
|
+
|
|
270
|
+
| You want to | Do this |
|
|
271
|
+
| --- | --- |
|
|
272
|
+
| choose the queue | `config.queue_name = :sagas` |
|
|
273
|
+
| set a priority or other job options | the same as for any job, in an initializer: `ActiveDurable::RunJob.queue_with_priority 10` |
|
|
274
|
+
| retry a step more or fewer times | `retry:` on the step, or `config.step_attempts` |
|
|
275
|
+
| stop retrying a business failure | `flow.abort!`, like the declined card above |
|
|
276
|
+
| see what is running | the [dashboard](#dashboard), or `ActiveDurable::RunJob` in your backend's UI |
|
|
277
|
+
| hear about a stuck saga | the `blocked.active_durable` [event](#observability) |
|
|
278
|
+
| run a saga inline in tests | `ActiveDurable::Testing.drain(id)` |
|
|
279
|
+
| start or wake sagas from your own jobs | call `Durable.start` or `Durable.signal` there |
|
|
280
|
+
|
|
281
|
+
> Do not wrap `Durable.start` in a job of your own. The order and its saga would no longer be saved together, and a
|
|
282
|
+
> crash between the two would leave an order without a saga.
|
|
283
|
+
|
|
284
|
+
### 8. Tests
|
|
285
|
+
|
|
286
|
+
```ruby
|
|
287
|
+
# spec/rails_helper.rb
|
|
288
|
+
require "active_durable/testing"
|
|
289
|
+
|
|
290
|
+
RSpec.configure do |config|
|
|
291
|
+
config.before { ActiveDurable::Testing.reset! } # forgets simulated time and crash hooks
|
|
292
|
+
end
|
|
293
|
+
```
|
|
294
|
+
|
|
295
|
+
```ruby
|
|
296
|
+
# spec/services/place_order_spec.rb
|
|
297
|
+
it "charges once and confirms the order" do
|
|
298
|
+
order = PlaceOrder.call(order_params)
|
|
299
|
+
|
|
300
|
+
expect(ActiveDurable::Testing.drain(order.checkout.id).status).to eq("completed")
|
|
301
|
+
end
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
`drain` runs the saga right there, without a worker. Stub Stripe as you already do, then let the
|
|
305
|
+
[crash tester](#testing-the-crash-tester) kill the saga at every point.
|
|
306
|
+
|
|
307
|
+
### 9. Before going to production
|
|
308
|
+
|
|
309
|
+
- [ ] Workers are running and listen to `config.queue_name`.
|
|
310
|
+
- [ ] The sweeper runs every minute.
|
|
311
|
+
- [ ] `dashboard_authorize` is set; without it the dashboard answers 403.
|
|
312
|
+
- [ ] `lease_duration` is longer than your slowest step.
|
|
313
|
+
- [ ] Every step that calls an outside service passes the ticket as its idempotency key.
|
|
314
|
+
- [ ] The crash tester passes for every recipe.
|
|
315
|
+
|
|
316
|
+
## How it works
|
|
317
|
+
|
|
318
|
+
Every execution has a **notebook**: one row per step, with its status and its result. A worker takes the
|
|
319
|
+
execution with a lease, then runs the recipe from the top. For each step it looks at the notebook first:
|
|
320
|
+
|
|
321
|
+
| The notebook says | The worker |
|
|
322
|
+
| --- | --- |
|
|
323
|
+
| ✔ done | does not run the step and returns the recorded result |
|
|
324
|
+
| nothing yet | runs the step and writes the result down |
|
|
325
|
+
| retrying, failed or waiting | waits, retries, compensates or blocks, as described below |
|
|
326
|
+
|
|
327
|
+
```mermaid
|
|
328
|
+
stateDiagram-v2
|
|
329
|
+
direction LR
|
|
330
|
+
[*] --> pending: Durable.start
|
|
331
|
+
pending --> running: a worker takes the lease
|
|
332
|
+
running --> sleeping: flow.sleep or a retry
|
|
333
|
+
running --> waiting: flow.wait_for
|
|
334
|
+
sleeping --> running: wake-up time
|
|
335
|
+
waiting --> running: Durable.signal
|
|
336
|
+
running --> completed: every step done
|
|
337
|
+
running --> compensated: failure before the pivot, undos ran
|
|
338
|
+
running --> blocked: needs a person
|
|
339
|
+
blocked --> pending: ActiveDurable.retry
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
Three rules follow from replaying the recipe:
|
|
343
|
+
|
|
344
|
+
1. **Everything that changes the world goes inside a step.** Code outside steps runs again on every replay:
|
|
345
|
+
reading is fine; writing, charging or sending is not.
|
|
346
|
+
2. **Step results are JSON.** They come back from the notebook with string keys, and the first run returns the same
|
|
347
|
+
JSON so it behaves exactly like a replay: `payment["id"]`, never `payment[:id]`.
|
|
348
|
+
3. **Step names are keys.** Each step needs a unique name within its recipe.
|
|
349
|
+
|
|
350
|
+
## The building blocks
|
|
351
|
+
|
|
352
|
+
| Call | Use it for | When it fails |
|
|
353
|
+
| --- | --- | --- |
|
|
354
|
+
| `flow.step(name) { \|ticket\| ... }` | anything that talks to the outside world | retried, then the saga is undone |
|
|
355
|
+
| `flow.transaction(name) { ... }` | changes to your own database only (exactly once) | rolled back with its checkpoint, retried, then undone |
|
|
356
|
+
| `flow.pivot(name) { \|ticket\| ... }` | the step after which there is no going back | retried, then the saga is undone |
|
|
357
|
+
| `flow.parallel(name) { \|branches\| ... }` | several steps at the same time | each branch like a step |
|
|
358
|
+
| `flow.sleep(name, 3.days)` | waiting without holding a worker | — |
|
|
359
|
+
| `flow.wait_for(name, timeout:)` | waiting for `Durable.signal` | a timeout fails the saga |
|
|
360
|
+
|
|
361
|
+
Steps take `undo:`, `retry:` (`3`, `false` or `{ attempts:, backoff: }`) and `undo_on_failure:`.
|
|
362
|
+
|
|
363
|
+
<details>
|
|
364
|
+
<summary><b>Tickets: a step that runs twice still has an effect once</b></summary>
|
|
365
|
+
|
|
366
|
+
<br>
|
|
367
|
+
|
|
368
|
+
Each step receives a ticket, `"<execution id>:<step name>"`, that is the same every time the step runs. If a worker
|
|
369
|
+
dies after calling Stripe but before writing the result down, the step runs again; with the ticket as the
|
|
370
|
+
idempotency key, Stripe answers with the first result instead of charging twice. Undos get their own ticket,
|
|
371
|
+
`"...:<step name>:undo"`.
|
|
372
|
+
|
|
373
|
+
</details>
|
|
374
|
+
|
|
375
|
+
<details>
|
|
376
|
+
<summary><b>Undo in reverse</b></summary>
|
|
377
|
+
|
|
378
|
+
<br>
|
|
379
|
+
|
|
380
|
+
When a step runs out of attempts, raises `ActiveDurable::Abort`, or the recipe raises, every finished step is undone,
|
|
381
|
+
last one first. Each undo is written in the notebook too, so a crash in the middle of undoing resumes where it
|
|
382
|
+
stopped. An undo receives `(result, undo_ticket, step_ticket)` and takes as many as it declares: `-> { ... }` takes
|
|
383
|
+
none. Any object that responds to `call` works too, such as `Payments.method(:refund)`.
|
|
384
|
+
|
|
385
|
+
A step that failed is not undone, because it did not happen. The exception is a step whose failure may hide a
|
|
386
|
+
success, like a charge whose answer timed out: declare `undo_on_failure: true` and its undo runs with `nil` as the
|
|
387
|
+
result, so it can look the outcome up with the step's ticket.
|
|
388
|
+
|
|
389
|
+
To take another path instead, rescue the failure in the recipe:
|
|
390
|
+
|
|
391
|
+
```ruby
|
|
392
|
+
begin
|
|
393
|
+
flow.step(:charge_with_stripe, retry: 2) { |ticket| ... }
|
|
394
|
+
rescue ActiveDurable::StepFailed
|
|
395
|
+
flow.step(:charge_with_paypal) { |ticket| ... }
|
|
396
|
+
end
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
</details>
|
|
400
|
+
|
|
401
|
+
<details>
|
|
402
|
+
<summary><b>The point of no return</b></summary>
|
|
403
|
+
|
|
404
|
+
<br>
|
|
405
|
+
|
|
406
|
+
A shipped parcel or a wire transfer cannot be undone. Mark that step with `flow.pivot`. Before it, a failure undoes
|
|
407
|
+
everything. After it, steps cannot declare `undo:` and are retried with backoff (`config.after_pivot_attempts`,
|
|
408
|
+
25 by default); if they still fail, the execution is **blocked** for a person.
|
|
409
|
+
|
|
410
|
+
</details>
|
|
411
|
+
|
|
412
|
+
<details>
|
|
413
|
+
<summary><b>Sleeping and waiting for signals</b></summary>
|
|
414
|
+
|
|
415
|
+
<br>
|
|
416
|
+
|
|
417
|
+
`flow.sleep` writes the wake-up time down and releases the worker. `flow.wait_for` does the same until a signal
|
|
418
|
+
arrives. Signals can arrive before the saga starts waiting.
|
|
419
|
+
|
|
420
|
+
```ruby
|
|
421
|
+
kyc = flow.wait_for(:kyc_done, timeout: 2.hours)
|
|
422
|
+
|
|
423
|
+
# in the webhook controller
|
|
424
|
+
Durable.signal("loan-42", :kyc_done, verified: true)
|
|
425
|
+
```
|
|
426
|
+
|
|
427
|
+
Pass `id:` to `Durable.start` to choose the execution id (the call becomes idempotent).
|
|
428
|
+
|
|
429
|
+
</details>
|
|
430
|
+
|
|
431
|
+
<details>
|
|
432
|
+
<summary><b>Parallel branches</b></summary>
|
|
433
|
+
|
|
434
|
+
<br>
|
|
435
|
+
|
|
436
|
+
```ruby
|
|
437
|
+
reservations = flow.parallel :reserve_stock do |branches|
|
|
438
|
+
order.warehouses.each do |warehouse|
|
|
439
|
+
branches.step warehouse.code, undo: ->(r, ticket) { warehouse.release(r["id"], key: ticket) } do |ticket|
|
|
440
|
+
{ "id" => warehouse.reserve(order.items_for(warehouse), key: ticket) }
|
|
441
|
+
end
|
|
442
|
+
end
|
|
443
|
+
end
|
|
444
|
+
reservations # => { "MEX" => { "id" => ... }, "GDL" => { "id" => ... } }
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
Each branch runs in its own thread (`config.parallel_concurrency`, 4 by default), has its own notebook row
|
|
448
|
+
(`reserve_stock/MEX`), ticket and retries. After a crash only unfinished branches run again; if one fails for good,
|
|
449
|
+
the finished branches are undone in the order they finished. Give your connection pool at least
|
|
450
|
+
`parallel_concurrency + 1` connections.
|
|
451
|
+
|
|
452
|
+
</details>
|
|
453
|
+
|
|
454
|
+
<details>
|
|
455
|
+
<summary><b>Changing a recipe while sagas are in flight</b></summary>
|
|
456
|
+
|
|
457
|
+
<br>
|
|
458
|
+
|
|
459
|
+
If a replay reaches a step the notebook did not record, ActiveDurable blocks that execution with
|
|
460
|
+
`ActiveDurable::RecipeChanged` and names both steps, instead of guessing. To change a recipe safely, keep the old
|
|
461
|
+
one and add a version:
|
|
462
|
+
|
|
463
|
+
```ruby
|
|
464
|
+
CheckoutSaga = Durable.define(:checkout, version: 2) { |flow, order_id:| ... }
|
|
465
|
+
Durable.define(:checkout, version: 1) { |flow, order_id:| ... } # keep until nothing uses it
|
|
466
|
+
```
|
|
467
|
+
|
|
468
|
+
New executions use the highest version; each execution keeps the one it started with.
|
|
469
|
+
`bin/rails active_durable:versions` lists the versions unfinished executions still use.
|
|
470
|
+
|
|
471
|
+
</details>
|
|
472
|
+
|
|
473
|
+
## Dashboard
|
|
474
|
+
|
|
475
|
+
```ruby
|
|
476
|
+
# config/routes.rb
|
|
477
|
+
mount ActiveDurable::Engine => "/durable"
|
|
478
|
+
```
|
|
479
|
+
|
|
480
|
+
<table>
|
|
481
|
+
<tr>
|
|
482
|
+
<td width="50%"><img src="docs/assets/dashboard-list.png" alt="Dashboard: executions by status, each drawn as a row of blocks"></td>
|
|
483
|
+
<td width="50%"><img src="docs/assets/dashboard-saga.png" alt="Dashboard: one saga step by step, with the point of no return and a sleeping step"></td>
|
|
484
|
+
</tr>
|
|
485
|
+
<tr>
|
|
486
|
+
<td colspan="2"><img src="docs/assets/dashboard-undone.png" alt="Dashboard: a saga undone in reverse after the dispatch step failed"></td>
|
|
487
|
+
</tr>
|
|
488
|
+
</table>
|
|
489
|
+
|
|
490
|
+
Every saga is drawn as a row of blocks, and every animation means something: a running step beats, a waiting one
|
|
491
|
+
pings like a radar, a sleeping one fills a ring until it wakes, a failed one shakes, undone ones are striped and
|
|
492
|
+
wired backwards. Countdowns are live, and live mode refreshes the list and flashes the rows that changed. It has
|
|
493
|
+
buttons to retry, undo everything or run a saga again from a chosen step.
|
|
494
|
+
|
|
495
|
+
It needs no asset pipeline and works in `rails new --api` apps, with its own session for CSRF protection. Outside
|
|
496
|
+
development and test it stays **closed** until you decide who can open it:
|
|
497
|
+
|
|
498
|
+
```ruby
|
|
499
|
+
# config/initializers/active_durable.rb
|
|
500
|
+
ActiveDurable.config.dashboard_authorize = lambda do |controller|
|
|
501
|
+
controller.authenticate_or_request_with_http_basic do |user, password|
|
|
502
|
+
ActiveSupport::SecurityUtils.secure_compare(user, ENV.fetch("DURABLE_USER")) &
|
|
503
|
+
ActiveSupport::SecurityUtils.secure_compare(password, ENV.fetch("DURABLE_PASSWORD"))
|
|
504
|
+
end
|
|
505
|
+
end
|
|
506
|
+
```
|
|
507
|
+
|
|
508
|
+
## Operating sagas
|
|
509
|
+
|
|
510
|
+
```ruby
|
|
511
|
+
ActiveDurable.retry("checkout-7") # blocked: try again where it stopped
|
|
512
|
+
ActiveDurable.compensate("checkout-7", reason: "customer cancelled") # undo everything (only before the pivot)
|
|
513
|
+
ActiveDurable.rerun("checkout-7", from: :ship) # a new execution reusing steps before :ship
|
|
514
|
+
```
|
|
515
|
+
|
|
516
|
+
All three refuse an execution a worker is running right now. A rerun runs the chosen step and the following ones
|
|
517
|
+
again with new tickets, so they have effects again; a blocked original becomes `superseded`.
|
|
518
|
+
|
|
519
|
+
## Testing: the crash tester
|
|
520
|
+
|
|
521
|
+
```ruby
|
|
522
|
+
require "active_durable/testing"
|
|
523
|
+
|
|
524
|
+
it "survives a crash at any point" do
|
|
525
|
+
ActiveDurable::Testing.crash_everywhere(:checkout, order_id: order.id) do |execution, point|
|
|
526
|
+
expect(execution.status).to eq("completed")
|
|
527
|
+
expect(FakeStripe.charges.size).to eq(1)
|
|
528
|
+
end
|
|
529
|
+
end
|
|
530
|
+
```
|
|
531
|
+
|
|
532
|
+
`crash_everywhere` runs the saga once to find every point where a process could die (before each step, after the
|
|
533
|
+
call but before the checkpoint, after the checkpoint, and the same for undos). Then it runs a fresh execution per
|
|
534
|
+
point, kills it right there, and finishes it with a new worker. It is the fastest way to find a step that is not
|
|
535
|
+
idempotent.
|
|
536
|
+
|
|
537
|
+
`ActiveDurable::Testing.drain(id, signals: { name => payload })` runs an execution synchronously, fast-forwarding
|
|
538
|
+
sleeps, retries and expired leases.
|
|
539
|
+
|
|
540
|
+
## Observability
|
|
541
|
+
|
|
542
|
+
<details>
|
|
543
|
+
<summary><b>Events and OpenTelemetry</b></summary>
|
|
544
|
+
|
|
545
|
+
<br>
|
|
546
|
+
|
|
547
|
+
Subscribe to `blocked.active_durable` to page someone:
|
|
548
|
+
|
|
549
|
+
```ruby
|
|
550
|
+
ActiveSupport::Notifications.subscribe("blocked.active_durable") do |event|
|
|
551
|
+
Sentry.capture_message("Saga blocked", extra: event.payload)
|
|
552
|
+
end
|
|
553
|
+
```
|
|
554
|
+
|
|
555
|
+
| Event | Payload |
|
|
556
|
+
| --- | --- |
|
|
557
|
+
| `execution` / `step` / `compensation` / `undo` | `execution_id` (and `recipe`, `step`, `kind`) |
|
|
558
|
+
| `completed` / `compensated` | `execution_id`, `recipe` |
|
|
559
|
+
| `blocked` | `execution_id`, `recipe`, `error` |
|
|
560
|
+
| `retried` / `compensation_requested` / `rerun` | operator actions |
|
|
561
|
+
|
|
562
|
+
With `opentelemetry-sdk` configured, every worker run becomes a span with its steps, undos and compensation nested
|
|
563
|
+
inside, parallel branches included:
|
|
564
|
+
|
|
565
|
+
```ruby
|
|
566
|
+
require "active_durable/open_telemetry"
|
|
567
|
+
ActiveDurable::OpenTelemetry.install!
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
</details>
|
|
571
|
+
|
|
572
|
+
The settings are in [the initializer](#4-the-initializer).
|
|
573
|
+
|
|
574
|
+
## Compatibility
|
|
575
|
+
|
|
576
|
+
Every combination below runs the full test suite in CI, against **PostgreSQL**, **MySQL 8+** and **SQLite 3**.
|
|
577
|
+
|
|
578
|
+
| | Rails 6.1 | Rails 7.0 | Rails 7.1 | Rails 7.2 | Rails 8.0 | Rails 8.1 |
|
|
579
|
+
| --- | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
580
|
+
| **Ruby 3.1** | ✔ | ✔ | ✔ | ✔ | needs Ruby 3.2 | needs Ruby 3.2 |
|
|
581
|
+
| **Ruby 3.2** | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
|
|
582
|
+
| **Ruby 3.3** | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
|
|
583
|
+
| **Ruby 3.4** | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
|
|
584
|
+
| **Ruby 4.0** | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |
|
|
585
|
+
|
|
586
|
+
Every feature works on every version. Things your app may need on older Rails, unrelated to ActiveDurable:
|
|
587
|
+
|
|
588
|
+
- **MySQL on Rails 6.1 and 7.0** uses the `mysql2` adapter (`trilogy` ships with Active Record 7.1+).
|
|
589
|
+
- **Rails 6.1 on Ruby 3.4+** needs `base64`, `benchmark`, `bigdecimal`, `drb`, `logger`, `mutex_m`, `observer`
|
|
590
|
+
and `ostruct` in the Gemfile: Rails 6.1 uses them, and Ruby no longer ships them by default.
|
|
591
|
+
- **`unknown keyword: quirks_mode`** comes from some Active Support versions (seen with 7.1 and 8.0) and json 3:
|
|
592
|
+
add `gem "json", "< 3"`.
|
|
593
|
+
|
|
594
|
+
## How it compares
|
|
595
|
+
|
|
596
|
+
| | Keeps progress in | Undoes steps | Needs |
|
|
597
|
+
| --- | --- | --- | --- |
|
|
598
|
+
| Active Job Continuations (Rails 8.1) | the job (a cursor) | no | nothing extra |
|
|
599
|
+
| ChronoForge | your database | not in its docs | nothing extra |
|
|
600
|
+
| ruby_reactor | Redis | yes | Redis and Sidekiq |
|
|
601
|
+
| Temporal | the Temporal server | written by hand | a Temporal cluster |
|
|
602
|
+
| **ActiveDurable** | **your database** | **yes, in reverse, with a point of no return** | **nothing extra** |
|
|
603
|
+
|
|
604
|
+
## Guarantees and limits
|
|
605
|
+
|
|
606
|
+
- A step runs **at least once**; with an idempotency key it has its effect once. `flow.transaction` runs exactly
|
|
607
|
+
once, because its change and its checkpoint commit together.
|
|
608
|
+
- One worker at a time per execution: taking an execution and every write are fenced by a lease token.
|
|
609
|
+
- Sagas do not isolate each other: two sagas can see each other's intermediate states.
|
|
610
|
+
- `flow.transaction` is atomic only when the notebook lives in the same database as your data.
|
|
611
|
+
|
|
612
|
+
## Development
|
|
613
|
+
|
|
614
|
+
```bash
|
|
615
|
+
bundle install
|
|
616
|
+
bundle exec rspec # PostgreSQL, Rails 8.1
|
|
617
|
+
DB=mysql bundle exec rspec # MySQL
|
|
618
|
+
DB=sqlite3 bundle exec rspec # SQLite
|
|
619
|
+
BUNDLE_GEMFILE=gemfiles/rails-7.1.gemfile bundle exec rspec # any Rails in gemfiles/
|
|
620
|
+
bundle exec rubocop
|
|
621
|
+
bin/demo # the dashboard with sample sagas
|
|
622
|
+
ruby docs/assets/generate.rb # rebuild the animated SVGs of this README
|
|
623
|
+
```
|
|
624
|
+
|
|
625
|
+
This README exists in two languages: every change goes to `README.md` and `README.es.md` (see `CONTRIBUTING.md`).
|
|
626
|
+
The design notes, in Spanish, are in `docs/`.
|
|
627
|
+
|
|
628
|
+
## License
|
|
629
|
+
|
|
630
|
+
MIT. See [LICENSE.txt](LICENSE.txt).
|
data/Rakefile
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module ActiveDurable
|
|
4
|
+
# Base controller for the dashboard. It does not inherit from your ApplicationController on purpose:
|
|
5
|
+
# the dashboard must work the same in full and API-only apps.
|
|
6
|
+
class ApplicationController < ActionController::Base
|
|
7
|
+
protect_from_forgery with: :exception
|
|
8
|
+
layout "active_durable/application"
|
|
9
|
+
helper ActiveDurable::DashboardHelper
|
|
10
|
+
|
|
11
|
+
before_action :authorize_dashboard!
|
|
12
|
+
|
|
13
|
+
private
|
|
14
|
+
|
|
15
|
+
def authorize_dashboard!
|
|
16
|
+
rule = ActiveDurable.config.dashboard_authorize
|
|
17
|
+
# Rails.env.local? only exists since Rails 7.1; before that it silently answers false.
|
|
18
|
+
allowed = rule ? rule.call(self) : Rails.env.development? || Rails.env.test?
|
|
19
|
+
return if performed? || allowed
|
|
20
|
+
|
|
21
|
+
render plain: "The ActiveDurable dashboard is closed. Set ActiveDurable.config.dashboard_authorize " \
|
|
22
|
+
"to decide who can open it.", status: :forbidden
|
|
23
|
+
end
|
|
24
|
+
end
|
|
25
|
+
end
|