postburner 0.8.0 → 1.0.0.pre.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +0 -22
- data/README.md +1219 -238
- data/Rakefile +1 -1
- data/app/concerns/postburner/callbacks.rb +286 -0
- data/app/models/postburner/job.rb +735 -46
- data/app/models/postburner/tracked_job.rb +83 -0
- data/bin/postburner +91 -0
- data/bin/rails +14 -0
- data/config/environment.rb +3 -0
- data/config/postburner.yml +22 -0
- data/config/postburner.yml.example +142 -0
- data/lib/postburner/active_job/adapter.rb +163 -0
- data/lib/postburner/active_job/execution.rb +109 -0
- data/lib/postburner/active_job/payload.rb +157 -0
- data/lib/postburner/beanstalkd.rb +97 -0
- data/lib/postburner/configuration.rb +202 -0
- data/lib/postburner/connection.rb +113 -0
- data/lib/postburner/engine.rb +1 -1
- data/lib/postburner/queue_config.rb +151 -0
- data/lib/postburner/strategies/immediate_test_queue.rb +133 -0
- data/lib/postburner/strategies/nice_queue.rb +85 -0
- data/lib/postburner/strategies/null_queue.rb +132 -0
- data/lib/postburner/strategies/queue.rb +119 -0
- data/lib/postburner/strategies/test_queue.rb +128 -0
- data/lib/postburner/time_helpers.rb +75 -0
- data/lib/postburner/tracked.rb +171 -0
- data/lib/postburner/version.rb +1 -1
- data/lib/postburner/workers/base.rb +210 -0
- data/lib/postburner/workers/worker.rb +480 -0
- data/lib/postburner.rb +433 -4
- metadata +66 -17
- data/lib/postburner/tube.rb +0 -53
data/README.md
CHANGED
|
@@ -1,365 +1,1346 @@
|
|
|
1
1
|
# Postburner
|
|
2
|
-
An ActiveRecord layer on top of [Backburner](https://github.com/nesquena/backburner) for inspecting and auditing the
|
|
3
|
-
queue, especially for delayed jobs. It isn't meant to be outperform other queues, but be safe (and inspectable).
|
|
4
2
|
|
|
5
|
-
|
|
6
|
-
ActiveJob processor for mailers, active storage, and the like. Use a
|
|
7
|
-
`Postburner::Job` for things that you want to track. See [Comparison to Backburner](#comparison-to-backburner) for more.
|
|
3
|
+
Fast Beanstalkd-backed job queue with **optional PostgreSQL audit trail** for ActiveJob.
|
|
8
4
|
|
|
9
|
-
Postburner
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
differences. See [Comparison to Que](#comparison-to-que) for more.
|
|
5
|
+
Postburner provides dual-mode job execution:
|
|
6
|
+
- **Default jobs**: Fast execution via Beanstalkd only (no PostgreSQL overhead)
|
|
7
|
+
- **Tracked jobs**: Full audit trail with logs, timing, errors, and statistics
|
|
13
8
|
|
|
14
|
-
|
|
9
|
+
Built for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
|
|
10
|
+
|
|
11
|
+
## Features
|
|
12
|
+
|
|
13
|
+
- **ActiveJob native** - Works seamlessly with Rails, ActionMailer, ActiveStorage
|
|
14
|
+
- **Dual-mode execution** - Default or tracked (database backed)
|
|
15
|
+
- **Rich audit trail** - Logs, timing, errors, retry tracking (tracked jobs only)
|
|
16
|
+
- **ActiveRecord** - Query jobs with ActiveRecord (Tracked jobs only)
|
|
17
|
+
- **Beanstalkd** - Fast, reliable queue separate from your database, peristent storage
|
|
18
|
+
- **Process isolation** - Forking workers with optional threading for throughput
|
|
19
|
+
- **Test-friendly** - Inline execution without Beanstalkd in tests
|
|
20
|
+
|
|
21
|
+
## Quick Start
|
|
22
|
+
|
|
23
|
+
**Installation:**
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
sudo apt-get install beanstalkd # OR brew install beanstalkd
|
|
27
|
+
|
|
28
|
+
# Start beanstalkd (in-memory only)
|
|
29
|
+
beanstalkd -l 127.0.0.1 -p 11300
|
|
30
|
+
|
|
31
|
+
# OR with persistence (recommended for production)
|
|
32
|
+
mkdir -p /var/lib/beanstalkd
|
|
33
|
+
beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
|
|
34
|
+
```
|
|
15
35
|
|
|
16
36
|
```ruby
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
queue_max_job_retries 0 # don't retry
|
|
37
|
+
# Gemfile
|
|
38
|
+
gem 'postburner', '~> 1.0.0.pre.1'
|
|
39
|
+
```
|
|
21
40
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
41
|
+
```bash
|
|
42
|
+
bundle install
|
|
43
|
+
rails generate postburner:install
|
|
44
|
+
rails db:migrate
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**Configuration:**
|
|
48
|
+
|
|
49
|
+
```ruby
|
|
50
|
+
# config/application.rb
|
|
51
|
+
config.active_job.queue_adapter = :postburner
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
```yaml
|
|
55
|
+
# config/postburner.yml
|
|
56
|
+
production:
|
|
57
|
+
beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
|
|
58
|
+
worker_type: simple
|
|
59
|
+
queues:
|
|
60
|
+
default: {}
|
|
61
|
+
critical: {}
|
|
62
|
+
mailers: {}
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
**Usage:**
|
|
66
|
+
|
|
67
|
+
```ruby
|
|
68
|
+
# Default job (fast, no PostgreSQL overhead)
|
|
69
|
+
class SendEmail < ApplicationJob
|
|
70
|
+
def perform(user_id)
|
|
71
|
+
UserMailer.welcome(user_id).deliver_now
|
|
25
72
|
end
|
|
26
73
|
end
|
|
27
74
|
|
|
28
|
-
#
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
#
|
|
33
|
-
#
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
75
|
+
# Default job with Beanstalkd configuration
|
|
76
|
+
class SendEmail < ApplicationJob
|
|
77
|
+
include Postburner::Beanstalkd
|
|
78
|
+
|
|
79
|
+
queue_priority 100 # Custom priority
|
|
80
|
+
queue_ttr 300 # 5 minute timeout
|
|
81
|
+
|
|
82
|
+
def perform(user_id)
|
|
83
|
+
UserMailer.welcome(user_id).deliver_now
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
# Tracked job (full audit trail, includes Beanstalkd automatically)
|
|
88
|
+
class ProcessPayment < ApplicationJob
|
|
89
|
+
include Postburner::Tracked # ← Opt-in to tracking (includes Beanstalkd)
|
|
90
|
+
|
|
91
|
+
queue_priority 0 # Highest priority
|
|
92
|
+
queue_ttr 600 # 10 minute timeout
|
|
37
93
|
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
94
|
+
def perform(payment_id)
|
|
95
|
+
log "Processing payment #{payment_id}"
|
|
96
|
+
Payment.find(payment_id).process!
|
|
97
|
+
log! "Payment processed successfully"
|
|
98
|
+
end
|
|
99
|
+
end
|
|
41
100
|
|
|
42
|
-
#
|
|
43
|
-
|
|
44
|
-
# `:delay` takes priority over `:at`takes priority over `:at` because the
|
|
45
|
-
# beanstalkd protocol uses uses `delay`
|
|
46
|
-
RunDonation.create!(args: {donation_id: 123}).queue! delay: 1.hour
|
|
47
|
-
=> {:status=>"INSERTED", :id=>"1141"}
|
|
101
|
+
# Run worker
|
|
102
|
+
bin/postburner --config config/postburner.yml --env production
|
|
48
103
|
```
|
|
49
104
|
|
|
50
|
-
|
|
105
|
+
## Table of Contents
|
|
51
106
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
107
|
+
- [Why Beanstalkd?](#why-beanstalkd)
|
|
108
|
+
- [Installation](#installation)
|
|
109
|
+
- [Usage](#usage)
|
|
110
|
+
- [Standard Jobs](#standard-jobs)
|
|
111
|
+
- [Tracked Jobs](#tracked-jobs)
|
|
112
|
+
- [Postburner::Job](#postburnerjob)
|
|
113
|
+
- [Workers](#workers)
|
|
114
|
+
- [Configuration](#configuration)
|
|
115
|
+
- [Queue Strategies](#queue-strategies)
|
|
116
|
+
- [Testing](#testing)
|
|
117
|
+
- [Job Management](#job-management)
|
|
118
|
+
- [Beanstalkd Integration](#beanstalkd-integration)
|
|
119
|
+
- [Web UI](#web-ui)
|
|
56
120
|
|
|
57
|
-
|
|
58
|
-
|
|
121
|
+
## Why Beanstalkd?
|
|
122
|
+
|
|
123
|
+
Beanstalkd is a simple, fast, and reliable queue system. It is a good choice for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
|
|
124
|
+
|
|
125
|
+
The [protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt) reads more like a README than a protocol. Check it out and you will instantly understand how it works.
|
|
126
|
+
|
|
127
|
+
### Binlogs
|
|
128
|
+
|
|
129
|
+
Beanstalkd lets you persist jobs to disk in case of a crash or restart. Just restart beanstalkd and the jobs will be back in the queue.
|
|
130
|
+
|
|
131
|
+
**Setup:**
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
# Create binlog directory
|
|
135
|
+
sudo mkdir -p /var/lib/beanstalkd
|
|
136
|
+
sudo chown beanstalkd:beanstalkd /var/lib/beanstalkd # If running as service
|
|
137
|
+
|
|
138
|
+
# Start beanstalkd with binlog persistence
|
|
139
|
+
beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
|
|
59
140
|
```
|
|
60
141
|
|
|
61
|
-
|
|
62
|
-
```ruby
|
|
63
|
-
# get the beanstalkd job id
|
|
64
|
-
job.bkid
|
|
65
|
-
=> 1104
|
|
142
|
+
**Configuration options:**
|
|
66
143
|
|
|
67
|
-
|
|
68
|
-
|
|
144
|
+
```bash
|
|
145
|
+
# Basic persistence
|
|
146
|
+
beanstalkd -b /var/lib/beanstalkd
|
|
69
147
|
|
|
70
|
-
#
|
|
71
|
-
|
|
148
|
+
# With custom binlog file size (default varies)
|
|
149
|
+
beanstalkd -b /var/lib/beanstalkd -s 10485760 # 10MB per binlog file
|
|
72
150
|
|
|
73
|
-
#
|
|
74
|
-
|
|
151
|
+
# Reduce fsync frequency for better performance (trades safety for speed)
|
|
152
|
+
beanstalkd -b /var/lib/beanstalkd -f 200 # fsync at most every 200ms (default: 50ms)
|
|
75
153
|
|
|
76
|
-
#
|
|
77
|
-
|
|
154
|
+
# Never fsync (maximum performance, higher risk of data loss on power failure)
|
|
155
|
+
beanstalkd -b /var/lib/beanstalkd -F
|
|
156
|
+
```
|
|
78
157
|
|
|
79
|
-
|
|
80
|
-
|
|
158
|
+
**Options:**
|
|
159
|
+
- `-b <dir>` - Enable binlog persistence in specified directory
|
|
160
|
+
- `-s <bytes>` - Maximum size of each binlog file (requires `-b`)
|
|
161
|
+
- `-f <ms>` - Call fsync at most once every N milliseconds (requires `-b`, default: 50ms)
|
|
162
|
+
- `-F` - Never call fsync (requires `-b`, maximum performance but higher data loss risk)
|
|
163
|
+
|
|
164
|
+
**Note:** The fsync options control the trade-off between performance and safety. A power failure could result in loss of jobs added within the fsync interval.
|
|
165
|
+
|
|
166
|
+
### Priority
|
|
167
|
+
|
|
168
|
+
Priority controls the order in which jobs are processed from the queue. Beanstalkd always processes the highest priority (lowest number) jobs first.
|
|
169
|
+
|
|
170
|
+
**Priority Range:** 0 to 4,294,967,295 (unsigned 32-bit integer)
|
|
171
|
+
- `0` = Highest priority (processed first)
|
|
172
|
+
- `4,294,967,295` = Lowest priority (processed last)
|
|
173
|
+
- Default: `65536` (configurable in `config/postburner.yml`)
|
|
174
|
+
|
|
175
|
+
When a worker reserves a job, Beanstalkd returns the highest priority job available in the tube. Jobs with the same priority are processed in FIFO order.
|
|
176
|
+
|
|
177
|
+
**ActiveJob with Postburner::Beanstalkd:**
|
|
178
|
+
|
|
179
|
+
```ruby
|
|
180
|
+
class ProcessPayment < ApplicationJob
|
|
181
|
+
include Postburner::Beanstalkd
|
|
182
|
+
|
|
183
|
+
queue_as :critical
|
|
184
|
+
queue_priority 0 # Highest priority - processed immediately
|
|
185
|
+
end
|
|
81
186
|
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
c.beanstalk.tubes.to_a
|
|
88
|
-
c.beanstalk.tubes.to_a.map{|t| c.tubes[t.name].peek(:buried)}
|
|
89
|
-
c.beanstalk.tubes['ss.development.caching'].stats
|
|
90
|
-
c.beanstalk.tubes['ss.development.caching'].peek(:buried).kick
|
|
91
|
-
c.beanstalk.tubes['ss.development.caching'].kick(3)
|
|
92
|
-
c.close
|
|
93
|
-
|
|
94
|
-
# automatically close
|
|
95
|
-
Postburner.connected do |connection|
|
|
96
|
-
# do stuff with connection
|
|
187
|
+
class SendEmail < ApplicationJob
|
|
188
|
+
include Postburner::Beanstalkd
|
|
189
|
+
|
|
190
|
+
queue_as :mailers
|
|
191
|
+
queue_priority 1000 # Lower priority - processed after critical jobs
|
|
97
192
|
end
|
|
98
193
|
```
|
|
99
194
|
|
|
100
|
-
|
|
195
|
+
**Postburner::Job:**
|
|
101
196
|
|
|
102
|
-
### Basic model fields
|
|
103
197
|
```ruby
|
|
104
|
-
|
|
105
|
-
|
|
198
|
+
class CriticalJob < Postburner::Job
|
|
199
|
+
queue 'critical'
|
|
200
|
+
queue_priority 0 # Highest priority
|
|
201
|
+
|
|
202
|
+
def perform(args)
|
|
203
|
+
# Critical business logic
|
|
204
|
+
end
|
|
205
|
+
end
|
|
206
|
+
|
|
207
|
+
class BackgroundTask < Postburner::Job
|
|
208
|
+
queue 'default'
|
|
209
|
+
queue_priority 5000 # Lower priority
|
|
210
|
+
|
|
211
|
+
def perform(args)
|
|
212
|
+
# Non-urgent background work
|
|
213
|
+
end
|
|
214
|
+
end
|
|
215
|
+
```
|
|
106
216
|
|
|
107
|
-
|
|
108
|
-
job.bkid
|
|
217
|
+
**Recommended Priority Ranges:**
|
|
109
218
|
|
|
110
|
-
|
|
111
|
-
|
|
219
|
+
| Priority Range | Use Case | Examples |
|
|
220
|
+
|---------------|----------|----------|
|
|
221
|
+
| 0-256 | Critical business operations | Payments, transactions, real-time notifications |
|
|
222
|
+
| 256-4096 | High priority tasks | Password resets, order confirmations |
|
|
223
|
+
| 4096-65536 | Standard background jobs | Email sending, data exports (65536 is default) |
|
|
224
|
+
| 65536-262144 | Low priority maintenance | Cache warming, log cleanup |
|
|
225
|
+
| 262144+ | Deferred/bulk operations | Analytics, batch reports |
|
|
112
226
|
|
|
113
|
-
|
|
114
|
-
job.type
|
|
227
|
+
**Configuration:**
|
|
115
228
|
|
|
116
|
-
|
|
117
|
-
job.args
|
|
229
|
+
Set default priority in `config/postburner.yml`:
|
|
118
230
|
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
231
|
+
```yaml
|
|
232
|
+
production:
|
|
233
|
+
default_priority: 65536 # Default for jobs without explicit priority
|
|
234
|
+
default_ttr: 300
|
|
122
235
|
```
|
|
123
236
|
|
|
124
|
-
|
|
237
|
+
**Dynamic Priority (Postburner::Job only):**
|
|
238
|
+
|
|
125
239
|
```ruby
|
|
126
|
-
|
|
127
|
-
|
|
240
|
+
class ProcessOrder < Postburner::Job
|
|
241
|
+
queue 'orders'
|
|
242
|
+
queue_priority 100 # Default
|
|
243
|
+
|
|
244
|
+
before_enqueue :adjust_priority
|
|
128
245
|
|
|
129
|
-
|
|
130
|
-
|
|
246
|
+
def perform(args)
|
|
247
|
+
order = Order.find(args['order_id'])
|
|
248
|
+
order.process!
|
|
249
|
+
end
|
|
131
250
|
|
|
132
|
-
|
|
133
|
-
job.processing_at
|
|
251
|
+
private
|
|
134
252
|
|
|
135
|
-
|
|
136
|
-
|
|
253
|
+
def adjust_priority
|
|
254
|
+
if args['urgent']
|
|
255
|
+
self.queue_priority = 0 # Override to highest priority
|
|
256
|
+
end
|
|
257
|
+
end
|
|
258
|
+
end
|
|
137
259
|
|
|
138
|
-
#
|
|
139
|
-
|
|
260
|
+
# Usage
|
|
261
|
+
ProcessOrder.create!(args: { 'order_id' => 123, 'urgent' => true }).queue!
|
|
262
|
+
```
|
|
140
263
|
|
|
141
|
-
|
|
142
|
-
job.lag
|
|
264
|
+
### Time to Run (TTR)
|
|
143
265
|
|
|
144
|
-
|
|
145
|
-
job.duration
|
|
266
|
+
TTR (Time-to-Run) is the maximum duration in seconds that a worker has to process a reserved job before Beanstalkd automatically releases it back to the ready queue.
|
|
146
267
|
|
|
147
|
-
|
|
148
|
-
|
|
268
|
+
1. Worker reserves a job from Beanstalkd
|
|
269
|
+
2. TTR countdown begins immediately upon reservation
|
|
270
|
+
3. If job completes within TTR, worker deletes/buries the job (success)
|
|
271
|
+
4. If TTR expires before completion, Beanstalkd automatically releases the job back to ready queue
|
|
272
|
+
5. Another worker can then reserve and process the same job
|
|
149
273
|
|
|
150
|
-
|
|
151
|
-
job.error_count
|
|
274
|
+
This mechanism protects against workers crashing or hanging—jobs won't be stuck indefinitely.
|
|
152
275
|
|
|
153
|
-
|
|
154
|
-
|
|
276
|
+
**TTR Range:** 1 to 4,294,967,295 seconds (unsigned 32-bit integer)
|
|
277
|
+
- Minimum: `1` second (Beanstalkd silently increases 0 to 1)
|
|
278
|
+
- Default: `300` seconds (5 minutes, configurable in `config/postburner.yml`)
|
|
155
279
|
|
|
156
|
-
|
|
157
|
-
job.attempts
|
|
280
|
+
**ActiveJob with Postburner::Beanstalkd:**
|
|
158
281
|
|
|
159
|
-
|
|
160
|
-
|
|
282
|
+
```ruby
|
|
283
|
+
class ProcessPayment < ApplicationJob
|
|
284
|
+
include Postburner::Beanstalkd
|
|
161
285
|
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
286
|
+
queue_as :critical
|
|
287
|
+
queue_priority 0
|
|
288
|
+
queue_ttr 300 # 5 minutes to complete
|
|
289
|
+
end
|
|
165
290
|
|
|
166
|
-
|
|
291
|
+
class QuickEmail < ApplicationJob
|
|
292
|
+
include Postburner::Beanstalkd
|
|
167
293
|
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
294
|
+
queue_as :mailers
|
|
295
|
+
queue_ttr 60 # 1 minute for fast email jobs
|
|
296
|
+
end
|
|
297
|
+
|
|
298
|
+
class LongRunningReport < ApplicationJob
|
|
299
|
+
include Postburner::Beanstalkd
|
|
300
|
+
|
|
301
|
+
queue_as :reports
|
|
302
|
+
queue_ttr 3600 # 1 hour for complex reports
|
|
303
|
+
end
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
**Postburner::Job:**
|
|
171
307
|
|
|
172
308
|
```ruby
|
|
173
|
-
class
|
|
174
|
-
queue '
|
|
175
|
-
|
|
176
|
-
queue_max_job_retries 0 # don't retry
|
|
309
|
+
class DataImport < Postburner::Job
|
|
310
|
+
queue 'imports'
|
|
311
|
+
queue_ttr 1800 # 30 minutes (equivalent to queue_ttr)
|
|
177
312
|
|
|
178
313
|
def perform(args)
|
|
179
|
-
#
|
|
180
|
-
|
|
314
|
+
# Long-running import logic
|
|
315
|
+
end
|
|
316
|
+
end
|
|
317
|
+
```
|
|
181
318
|
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
319
|
+
**Extending TTR with `extend!` (Touch Command):**
|
|
320
|
+
|
|
321
|
+
For jobs that may take longer than expected, you can extend the TTR dynamically using `extend!`. This calls Beanstalkd's `touch` command, which resets the TTR countdown.
|
|
322
|
+
|
|
323
|
+
**Available for:**
|
|
324
|
+
- `Postburner::Job` subclasses (always available via `bk.extend!`)
|
|
325
|
+
- `Postburner::Tracked` ActiveJob classes (includes `extend!` method)
|
|
326
|
+
|
|
327
|
+
```ruby
|
|
328
|
+
class ProcessImport < ApplicationJob
|
|
329
|
+
include Postburner::Tracked # Includes Beanstalkd and enables extend!
|
|
330
|
+
|
|
331
|
+
queue_as :imports
|
|
332
|
+
queue_ttr 300 # 5 minutes initial TTR
|
|
333
|
+
|
|
334
|
+
def perform(file_id)
|
|
335
|
+
file = ImportFile.find(file_id)
|
|
336
|
+
|
|
337
|
+
file.each_batch(size: 100) do |batch|
|
|
338
|
+
batch.each do |row|
|
|
339
|
+
process_row(row)
|
|
340
|
+
end
|
|
341
|
+
|
|
342
|
+
extend! # Reset TTR to 300 seconds from now
|
|
343
|
+
log "Processed batch, extended TTR"
|
|
186
344
|
end
|
|
187
345
|
end
|
|
188
346
|
end
|
|
189
347
|
```
|
|
190
348
|
|
|
191
|
-
|
|
349
|
+
**For Postburner::Job (via `bk` accessor):**
|
|
192
350
|
|
|
193
351
|
```ruby
|
|
194
|
-
|
|
352
|
+
class LargeDataProcessor < Postburner::Job
|
|
353
|
+
queue 'processing'
|
|
354
|
+
queue_ttr 600 # 10 minutes
|
|
195
355
|
|
|
196
|
-
|
|
197
|
-
|
|
356
|
+
def perform(args)
|
|
357
|
+
dataset = Dataset.find(args['dataset_id'])
|
|
358
|
+
|
|
359
|
+
dataset.each_chunk do |chunk|
|
|
360
|
+
process_chunk(chunk)
|
|
361
|
+
|
|
362
|
+
bk.extend! # Reset TTR via beanstalkd job accessor
|
|
363
|
+
log "Chunk processed, TTR extended"
|
|
364
|
+
end
|
|
365
|
+
end
|
|
366
|
+
end
|
|
367
|
+
```
|
|
368
|
+
|
|
369
|
+
**`extend!` and Beanstalkd `touch`**
|
|
370
|
+
|
|
371
|
+
- Calls Beanstalkd's `touch` command on the reserved job
|
|
372
|
+
- Resets the TTR countdown to the original TTR value from the current moment
|
|
373
|
+
- Example: Job has `queue_ttr 300`. After 200 seconds, calling `extend!` gives you another 300 seconds (not just 100)
|
|
374
|
+
- Can be called multiple times during job execution
|
|
375
|
+
- Useful for iterative processing where each iteration is quick but total time is unpredictable
|
|
376
|
+
|
|
377
|
+
**Configuration:**
|
|
378
|
+
|
|
379
|
+
Set default TTR in `config/postburner.yml`:
|
|
380
|
+
|
|
381
|
+
```yaml
|
|
382
|
+
production:
|
|
383
|
+
default_priority: 65536
|
|
384
|
+
default_ttr: 300 # 5 minutes default for all jobs
|
|
198
385
|
```
|
|
199
386
|
|
|
200
|
-
|
|
201
|
-
|
|
387
|
+
**Recommended TTR Values:**
|
|
388
|
+
|
|
389
|
+
| Job Type | TTR | Reasoning |
|
|
390
|
+
|----------|-----|-----------|
|
|
391
|
+
| Email sending | 60-120s | Network operations, should be fast |
|
|
392
|
+
| API calls | 30-60s | External services, fast or timeout |
|
|
393
|
+
| File processing | 600-1800s | Variable based on file size, use `extend!` |
|
|
394
|
+
| Reports | 1800-3600s | Complex queries and aggregations |
|
|
395
|
+
| Imports/Exports | 3600s+ | Large datasets, use `extend!` in loops |
|
|
202
396
|
|
|
203
|
-
|
|
204
|
-
prettier - or follow the suggestion above and use your own.
|
|
397
|
+
**Best Practices:**
|
|
205
398
|
|
|
206
|
-
|
|
399
|
+
1. **Set realistic TTR values** based on expected job duration plus buffer
|
|
400
|
+
2. **Use `extend!` for iterative work** where total time is unpredictable but each iteration is bounded
|
|
401
|
+
3. **Don't set TTR too high** - it delays recovery from crashed workers
|
|
402
|
+
4. **Monitor TTR timeouts** - frequent timeouts indicate jobs need more time or optimization
|
|
403
|
+
5. **For Default jobs** (ActiveJob without `Postburner::Beanstalkd`), include the module to set custom TTR
|
|
207
404
|
|
|
208
|
-
|
|
209
|
-
underscores upon instrospection of `stats`. See example. Reccommend using
|
|
210
|
-
names without hyphens / dashes.
|
|
405
|
+
**What happens on TTR timeout:**
|
|
211
406
|
|
|
212
407
|
```ruby
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
#
|
|
220
|
-
#
|
|
221
|
-
|
|
408
|
+
# Job starts processing
|
|
409
|
+
job = ProcessImport.perform_later(file_id)
|
|
410
|
+
|
|
411
|
+
# Worker crashes or hangs after 200 seconds
|
|
412
|
+
# At 300 seconds (TTR), Beanstalkd automatically:
|
|
413
|
+
# 1. Releases job back to ready queue
|
|
414
|
+
# 2. Increments the job's reserve count
|
|
415
|
+
# 3. Makes job available for another worker
|
|
416
|
+
|
|
417
|
+
# Another worker picks up the job and retries
|
|
418
|
+
# (ActiveJob retry_on/discard_on handlers still apply)
|
|
222
419
|
```
|
|
223
420
|
|
|
224
421
|
## Installation
|
|
225
422
|
|
|
226
|
-
|
|
423
|
+
### 1. Install Beanstalkd
|
|
424
|
+
|
|
425
|
+
First [install beanstalkd](https://beanstalkd.github.io/download.html):
|
|
426
|
+
|
|
227
427
|
```bash
|
|
228
428
|
sudo apt-get install beanstalkd
|
|
429
|
+
brew install beanstalkd # for macOS/Linux
|
|
430
|
+
|
|
431
|
+
# Start beanstalkd (in-memory only)
|
|
432
|
+
beanstalkd -l 127.0.0.1 -p 11300
|
|
433
|
+
|
|
434
|
+
# OR with persistence (recommended for production)
|
|
435
|
+
mkdir -p /var/lib/beanstalkd
|
|
436
|
+
beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
|
|
229
437
|
```
|
|
230
438
|
|
|
231
|
-
|
|
439
|
+
### 2. Add Gem
|
|
440
|
+
|
|
232
441
|
```ruby
|
|
233
|
-
|
|
442
|
+
# Gemfile
|
|
443
|
+
gem 'postburner', '~> 1.0.0.pre.1'
|
|
234
444
|
```
|
|
235
445
|
|
|
236
|
-
Install...
|
|
237
446
|
```bash
|
|
238
|
-
|
|
447
|
+
bundle install
|
|
239
448
|
```
|
|
240
449
|
|
|
241
|
-
|
|
450
|
+
### 3. Install Migration
|
|
451
|
+
|
|
452
|
+
```bash
|
|
453
|
+
rails generate postburner:install
|
|
454
|
+
rails db:migrate
|
|
242
455
|
```
|
|
243
|
-
|
|
244
|
-
|
|
456
|
+
|
|
457
|
+
This creates the `postburner_jobs` table for tracked jobs.
|
|
458
|
+
|
|
459
|
+
### 4. Configure ActiveJob
|
|
460
|
+
|
|
461
|
+
```ruby
|
|
462
|
+
# config/application.rb
|
|
463
|
+
config.active_job.queue_adapter = :postburner
|
|
464
|
+
```
|
|
465
|
+
|
|
466
|
+
### 5. Create Worker Configuration
|
|
467
|
+
|
|
468
|
+
```bash
|
|
469
|
+
cp config/postburner.yml.example config/postburner.yml
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
Edit `config/postburner.yml` for your environment (see [Configuration](#configuration)).
|
|
473
|
+
|
|
474
|
+
## Usage
|
|
475
|
+
|
|
476
|
+
### Default Jobs
|
|
477
|
+
|
|
478
|
+
Default jobs execute quickly via Beanstalkd without PostgreSQL overhead. Perfect for emails, cache warming, notifications, etc.
|
|
479
|
+
|
|
480
|
+
```ruby
|
|
481
|
+
class SendWelcomeEmail < ApplicationJob
|
|
482
|
+
queue_as :mailers
|
|
483
|
+
|
|
484
|
+
def perform(user_id)
|
|
485
|
+
UserMailer.welcome(user_id).deliver_now
|
|
486
|
+
end
|
|
487
|
+
end
|
|
488
|
+
|
|
489
|
+
# Enqueue immediately
|
|
490
|
+
SendWelcomeEmail.perform_later(123)
|
|
491
|
+
|
|
492
|
+
# Enqueue with delay
|
|
493
|
+
SendWelcomeEmail.set(wait: 1.hour).perform_later(123)
|
|
494
|
+
|
|
495
|
+
# Enqueue at specific time
|
|
496
|
+
SendWelcomeEmail.set(wait_until: Date.tomorrow.noon).perform_later(123)
|
|
245
497
|
```
|
|
246
498
|
|
|
247
|
-
|
|
248
|
-
|
|
499
|
+
**Overview:**
|
|
500
|
+
- Fast execution (no database writes)
|
|
501
|
+
- Low overhead
|
|
502
|
+
- Standard ActiveJob retry_on/discard_on support
|
|
503
|
+
- No audit trail
|
|
504
|
+
- No logging or statistics
|
|
505
|
+
|
|
506
|
+
#### Configuring Beanstalkd Priority and TTR
|
|
507
|
+
|
|
508
|
+
For default jobs that need custom Beanstalkd configuration, include `Postburner::Beanstalkd`:
|
|
249
509
|
|
|
250
|
-
Because `Postburner` is built on `Backburner`, add a `config/initializers/backburner.rb` with option found [here](https://github.com/nesquena/backburner#configuration).
|
|
251
510
|
```ruby
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
config.default_worker = Backburner::Workers::Simple
|
|
263
|
-
config.logger = Logger.new(STDOUT)
|
|
264
|
-
config.primary_queue = "backburner-jobs"
|
|
265
|
-
config.priority_labels = { :custom => 50, :useless => 1000 }
|
|
266
|
-
config.reserve_timeout = nil
|
|
267
|
-
config.job_serializer_proc = lambda { |body| JSON.dump(body) }
|
|
268
|
-
config.job_parser_proc = lambda { |body| JSON.parse(body) }
|
|
511
|
+
class SendWelcomeEmail < ApplicationJob
|
|
512
|
+
include Postburner::Beanstalkd
|
|
513
|
+
|
|
514
|
+
queue_as :mailers
|
|
515
|
+
queue_priority 100 # Lower = higher priority (0 is highest)
|
|
516
|
+
queue_ttr 300 # Time-to-run in seconds (5 minutes)
|
|
517
|
+
|
|
518
|
+
def perform(user_id)
|
|
519
|
+
UserMailer.welcome(user_id).deliver_now
|
|
520
|
+
end
|
|
269
521
|
end
|
|
270
522
|
```
|
|
271
523
|
|
|
272
|
-
|
|
524
|
+
**Configuration options:**
|
|
525
|
+
- `queue_priority` - Beanstalkd priority (0-4294967295, lower = higher priority)
|
|
526
|
+
- `queue_ttr` - Time-to-run in seconds before job times out
|
|
527
|
+
|
|
528
|
+
Jobs without `Postburner::Beanstalkd` use defaults from `config/postburner.yml`:
|
|
529
|
+
- `default_priority: 65536`
|
|
530
|
+
- `default_ttr: 300`
|
|
531
|
+
|
|
532
|
+
### Tracked Jobs
|
|
533
|
+
|
|
534
|
+
Tracked jobs store full execution details in PostgreSQL, providing comprehensive audit trails (i.e. logging, timing, errors, retry tracking) for critical operations.
|
|
535
|
+
|
|
536
|
+
**Note:** `Postburner::Tracked` automatically includes `Postburner::Beanstalkd`, giving you access to `queue_priority`, `queue_ttr`, `bk`, and `extend!`.
|
|
537
|
+
|
|
538
|
+
```ruby
|
|
539
|
+
class ProcessPayment < ApplicationJob
|
|
540
|
+
include Postburner::Tracked # ← Opt-in to PostgreSQL tracking (includes Beanstalkd)
|
|
541
|
+
|
|
542
|
+
queue_as :critical
|
|
543
|
+
queue_priority 0 # Highest priority (Beanstalkd included automatically)
|
|
544
|
+
queue_ttr 600 # 10 minute timeout
|
|
545
|
+
retry_on StandardError, wait: :exponentially_longer, attempts: 5
|
|
546
|
+
|
|
547
|
+
def perform(payment_id)
|
|
548
|
+
payment = Payment.find(payment_id)
|
|
549
|
+
|
|
550
|
+
log "Starting payment processing for $#{payment.amount}"
|
|
551
|
+
|
|
552
|
+
begin
|
|
553
|
+
payment.charge!
|
|
554
|
+
log! "Payment charged successfully"
|
|
555
|
+
rescue PaymentError => e
|
|
556
|
+
log_exception!(e)
|
|
557
|
+
raise
|
|
558
|
+
end
|
|
559
|
+
|
|
560
|
+
log "Payment complete", level: :info
|
|
561
|
+
end
|
|
562
|
+
end
|
|
273
563
|
```
|
|
274
|
-
|
|
275
|
-
|
|
564
|
+
|
|
565
|
+
**Tracking provides:**
|
|
566
|
+
- Complete execution history (queued_at, processing_at, processed_at)
|
|
567
|
+
- Custom logs with `log` and `log!`
|
|
568
|
+
- Exception tracking with `log_exception()` and `log_exception!()`
|
|
569
|
+
- Timing statistics (lag, duration)
|
|
570
|
+
- Retry attempts tracking
|
|
571
|
+
- Query with ActiveRecord
|
|
572
|
+
- Foreign key relationships
|
|
573
|
+
- Beanstalkd operations via `bk` accessor
|
|
574
|
+
- TTR extension via `extend!` method
|
|
575
|
+
|
|
576
|
+
**Query tracked jobs:**
|
|
577
|
+
|
|
578
|
+
```ruby
|
|
579
|
+
# Find by state
|
|
580
|
+
Postburner::TrackedJob.where(processed_at: nil)
|
|
581
|
+
Postburner::TrackedJob.where.not(removed_at: nil)
|
|
582
|
+
|
|
583
|
+
# Find with errors
|
|
584
|
+
Postburner::TrackedJob.where("error_count > 0")
|
|
585
|
+
|
|
586
|
+
# Statistics
|
|
587
|
+
Postburner::TrackedJob.average(:duration) # Average execution time
|
|
588
|
+
Postburner::TrackedJob.maximum(:lag) # Worst lag
|
|
589
|
+
|
|
590
|
+
# Inspect execution
|
|
591
|
+
job = Postburner::TrackedJob.last
|
|
592
|
+
job.logs # Array of log entries with timestamps
|
|
593
|
+
job.errata # Array of exceptions with backtraces
|
|
594
|
+
job.attempts # Array of attempt timestamps
|
|
595
|
+
job.duration # Execution time in milliseconds
|
|
596
|
+
job.lag # Queue lag in milliseconds
|
|
276
597
|
```
|
|
277
598
|
|
|
278
|
-
|
|
279
|
-
directly.
|
|
599
|
+
### Direct Postburner::Job Usage
|
|
280
600
|
|
|
281
|
-
|
|
601
|
+
Direct `Postburner::Job` subclasses are **always tracked**:
|
|
282
602
|
|
|
283
603
|
```ruby
|
|
284
|
-
# app/jobs/run_donation.rb
|
|
285
604
|
class RunDonation < Postburner::Job
|
|
286
605
|
queue 'critical'
|
|
287
|
-
queue_priority 0
|
|
288
|
-
queue_max_job_retries 0
|
|
606
|
+
queue_priority 0
|
|
607
|
+
queue_max_job_retries 0
|
|
289
608
|
|
|
290
|
-
def
|
|
291
|
-
|
|
292
|
-
|
|
609
|
+
def perform(args)
|
|
610
|
+
donation = Donation.find(args['donation_id'])
|
|
611
|
+
donation.process!
|
|
612
|
+
log "Processed donation #{donation.id}"
|
|
293
613
|
end
|
|
294
614
|
end
|
|
615
|
+
|
|
616
|
+
# Create and enqueue
|
|
617
|
+
job = RunDonation.create!(args: { 'donation_id' => 123 })
|
|
618
|
+
job.queue!
|
|
619
|
+
|
|
620
|
+
# With delay
|
|
621
|
+
job.queue!(delay: 1.hour)
|
|
622
|
+
|
|
623
|
+
# At specific time
|
|
624
|
+
job.queue!(at: 2.days.from_now)
|
|
295
625
|
```
|
|
296
626
|
|
|
297
|
-
|
|
627
|
+
#### Instance-Level Queue Configuration
|
|
298
628
|
|
|
299
|
-
|
|
300
|
-
Postburner adds:
|
|
301
|
-
- Database Jobs for inspection, linking, auditing, removal (and deletion)
|
|
302
|
-
- Direct access to associated [Beanstalk](https://beanstalkd.github.io/) (via [beaneater](https://github.com/beanstalkd/beaneater))
|
|
303
|
-
- Job Statistics (lag, attempts, logs, tracked errors)
|
|
304
|
-
- Convenience methods to clear tubes, stats, and connections for Beanstalk.
|
|
629
|
+
Override queue priority and TTR per job instance for dynamic behavior:
|
|
305
630
|
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
631
|
+
```ruby
|
|
632
|
+
# Set priority during creation
|
|
633
|
+
job = RunDonation.create!(
|
|
634
|
+
args: { 'donation_id' => 123 },
|
|
635
|
+
queue_priority: 1500, # Override class-level priority
|
|
636
|
+
queue_ttr: 300 # Override class-level TTR
|
|
637
|
+
)
|
|
638
|
+
job.queue!
|
|
309
639
|
|
|
310
|
-
|
|
311
|
-
|
|
640
|
+
# Set dynamically after creation
|
|
641
|
+
job = RunDonation.create!(args: { 'donation_id' => 456 })
|
|
642
|
+
job.queue_priority = 2000
|
|
643
|
+
job.queue_ttr = 300
|
|
644
|
+
job.queue!
|
|
312
645
|
|
|
313
|
-
|
|
646
|
+
# Use in before_enqueue callback for conditional behavior
|
|
647
|
+
class ProcessOrder < Postburner::Job
|
|
648
|
+
queue 'orders'
|
|
649
|
+
queue_priority 100 # Default priority
|
|
650
|
+
queue_ttr 120 # Default TTR
|
|
314
651
|
|
|
315
|
-
|
|
316
|
-
However, if you need something faster and backed with ACID compliance, check out Que.
|
|
652
|
+
before_enqueue :set_priority_based_on_urgency
|
|
317
653
|
|
|
318
|
-
|
|
319
|
-
|
|
654
|
+
def perform(args)
|
|
655
|
+
order = Order.find(args['order_id'])
|
|
656
|
+
order.process!
|
|
657
|
+
end
|
|
320
658
|
|
|
321
|
-
|
|
322
|
-
be simple so the logic and history can be transparent.
|
|
659
|
+
private
|
|
323
660
|
|
|
324
|
-
|
|
325
|
-
|
|
661
|
+
def set_priority_based_on_urgency
|
|
662
|
+
if args['express_shipping']
|
|
663
|
+
self.queue_priority = 0 # High priority for express orders
|
|
664
|
+
self.queue_ttr = 600 # Allow 10 minutes to complete
|
|
665
|
+
else
|
|
666
|
+
self.queue_priority = 1000 # Low priority for standard orders
|
|
667
|
+
self.queue_ttr = 120 # Standard 2 minute timeout
|
|
668
|
+
end
|
|
669
|
+
end
|
|
670
|
+
end
|
|
326
671
|
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
672
|
+
# Express order gets high priority automatically
|
|
673
|
+
ProcessOrder.create!(args: { 'order_id' => 789, 'express_shipping' => true }).queue!
|
|
674
|
+
```
|
|
675
|
+
|
|
676
|
+
**Overview:**
|
|
677
|
+
- Same as Tracked jobs
|
|
678
|
+
- Access the ActiveRecord Postburner::Job directly.
|
|
679
|
+
|
|
680
|
+
## Workers
|
|
681
|
+
|
|
682
|
+
Postburner uses named worker configurations to support different deployment patterns. Each worker can have different fork/thread settings and process different queues, enabling flexible production deployments.
|
|
683
|
+
|
|
684
|
+
### Named Workers Configuration
|
|
685
|
+
|
|
686
|
+
Configure multiple named workers with different concurrency profiles:
|
|
687
|
+
|
|
688
|
+
```yaml
|
|
689
|
+
production:
|
|
690
|
+
beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
|
|
691
|
+
|
|
692
|
+
workers:
|
|
693
|
+
# Heavy, memory-intensive jobs - more processes, fewer threads
|
|
694
|
+
imports:
|
|
695
|
+
default_forks: 4
|
|
696
|
+
default_threads: 1
|
|
697
|
+
default_gc_limit: 500
|
|
698
|
+
queues:
|
|
699
|
+
- imports
|
|
700
|
+
- data_processing
|
|
701
|
+
|
|
702
|
+
# General jobs - fewer processes, many threads
|
|
703
|
+
general:
|
|
704
|
+
default_forks: 2
|
|
705
|
+
default_threads: 100
|
|
706
|
+
default_gc_limit: 5000
|
|
707
|
+
queues:
|
|
708
|
+
- default
|
|
709
|
+
- mailers
|
|
710
|
+
- notifications
|
|
711
|
+
```
|
|
712
|
+
|
|
713
|
+
### Puma-Style Architecture
|
|
714
|
+
|
|
715
|
+
The worker supports two modes based on fork configuration:
|
|
716
|
+
- **`forks: 0`** - Single process with thread pools (development/staging)
|
|
717
|
+
- **`forks: 1+`** - Multiple processes with thread pools (production, Puma-style)
|
|
718
|
+
|
|
719
|
+
This gives you a natural progression from simple single-threaded development to high-concurrency production.
|
|
720
|
+
|
|
721
|
+
### Configuration Examples
|
|
722
|
+
|
|
723
|
+
**Development (single worker, single-threaded):**
|
|
724
|
+
```yaml
|
|
725
|
+
development:
|
|
726
|
+
beanstalk_url: beanstalk://localhost:11300
|
|
727
|
+
|
|
728
|
+
workers:
|
|
729
|
+
default:
|
|
730
|
+
# Defaults: forks=0, threads=1, gc_limit=nil
|
|
731
|
+
queues:
|
|
732
|
+
- default
|
|
733
|
+
- mailers
|
|
734
|
+
```
|
|
735
|
+
|
|
736
|
+
**Staging (single worker, multi-threaded):**
|
|
737
|
+
```yaml
|
|
738
|
+
staging:
|
|
739
|
+
beanstalk_url: beanstalk://localhost:11300
|
|
740
|
+
|
|
741
|
+
workers:
|
|
742
|
+
default:
|
|
743
|
+
default_threads: 10
|
|
744
|
+
default_gc_limit: 5000
|
|
745
|
+
queues:
|
|
746
|
+
- critical
|
|
747
|
+
- default
|
|
748
|
+
- mailers
|
|
749
|
+
```
|
|
750
|
+
|
|
751
|
+
**Production (multiple workers with different profiles):**
|
|
752
|
+
```yaml
|
|
753
|
+
production:
|
|
754
|
+
beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
|
|
755
|
+
|
|
756
|
+
workers:
|
|
757
|
+
imports:
|
|
758
|
+
default_forks: 4 # 4 processes
|
|
759
|
+
default_threads: 1 # 1 thread per process = 4 concurrent jobs
|
|
760
|
+
default_gc_limit: 500
|
|
761
|
+
queues:
|
|
762
|
+
- imports
|
|
763
|
+
- data_processing
|
|
764
|
+
|
|
765
|
+
general:
|
|
766
|
+
default_forks: 2 # 2 processes
|
|
767
|
+
default_threads: 100 # 100 threads per process = 200 concurrent jobs
|
|
768
|
+
default_gc_limit: 5000
|
|
769
|
+
queues:
|
|
770
|
+
- default
|
|
771
|
+
- mailers
|
|
772
|
+
- notifications
|
|
773
|
+
```
|
|
774
|
+
|
|
775
|
+
### Running Workers
|
|
776
|
+
|
|
777
|
+
**Single worker (auto-selected):**
|
|
778
|
+
```bash
|
|
779
|
+
bin/postburner
|
|
780
|
+
```
|
|
781
|
+
If only one worker is defined, it's automatically selected.
|
|
782
|
+
|
|
783
|
+
**Multiple workers (must specify):**
|
|
784
|
+
```bash
|
|
785
|
+
bin/postburner --worker imports # Run the 'imports' worker
|
|
786
|
+
bin/postburner --worker general # Run the 'general' worker
|
|
787
|
+
```
|
|
788
|
+
|
|
789
|
+
**Filter queues (override config):**
|
|
790
|
+
```bash
|
|
791
|
+
bin/postburner --worker general --queues default,mailers # Only process specific queues
|
|
792
|
+
```
|
|
330
793
|
|
|
331
|
-
###
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
794
|
+
### Running Workers in Separate Processes
|
|
795
|
+
|
|
796
|
+
For production deployments, run different workers in separate OS processes for isolation and resource allocation:
|
|
797
|
+
|
|
798
|
+
**Docker Compose example:**
|
|
799
|
+
```yaml
|
|
800
|
+
services:
|
|
801
|
+
# Process 1: Imports worker (forks=4, threads=1)
|
|
802
|
+
worker-imports:
|
|
803
|
+
build: .
|
|
804
|
+
command: bin/postburner --worker imports
|
|
805
|
+
environment:
|
|
806
|
+
RAILS_ENV: production
|
|
807
|
+
BEANSTALK_URL: beanstalk://beanstalkd:11300
|
|
808
|
+
depends_on:
|
|
809
|
+
- beanstalkd
|
|
810
|
+
- postgres
|
|
811
|
+
|
|
812
|
+
# Process 2: General worker (forks=2, threads=100)
|
|
813
|
+
worker-general:
|
|
814
|
+
build: .
|
|
815
|
+
command: bin/postburner --worker general
|
|
816
|
+
environment:
|
|
817
|
+
RAILS_ENV: production
|
|
818
|
+
BEANSTALK_URL: beanstalk://beanstalkd:11300
|
|
819
|
+
depends_on:
|
|
820
|
+
- beanstalkd
|
|
821
|
+
- postgres
|
|
822
|
+
```
|
|
823
|
+
|
|
824
|
+
**Procfile example (Heroku/Foreman):**
|
|
825
|
+
```
|
|
826
|
+
worker_imports: bin/postburner --worker imports
|
|
827
|
+
worker_general: bin/postburner --worker general
|
|
828
|
+
```
|
|
829
|
+
|
|
830
|
+
### Worker Architecture
|
|
831
|
+
|
|
832
|
+
**Single Process Mode (forks: 0):**
|
|
833
|
+
```
|
|
834
|
+
Main Process
|
|
835
|
+
├─ Queue 'critical' Thread Pool (1 thread)
|
|
836
|
+
├─ Queue 'default' Thread Pool (10 threads)
|
|
837
|
+
└─ Queue 'mailers' Thread Pool (5 threads)
|
|
838
|
+
```
|
|
345
839
|
|
|
840
|
+
**Multi-Process Mode (forks: 1+):**
|
|
841
|
+
```
|
|
842
|
+
Parent Process
|
|
843
|
+
├─ Fork 0 (queue: critical)
|
|
844
|
+
│ └─ Thread 1
|
|
845
|
+
├─ Fork 0 (queue: default)
|
|
846
|
+
│ ├─ Thread 1-10
|
|
847
|
+
├─ Fork 1 (queue: default) # Puma-style: multiple forks of same queue
|
|
848
|
+
│ ├─ Thread 1-10
|
|
849
|
+
├─ Fork 2 (queue: default)
|
|
850
|
+
│ └─ Thread 1-10
|
|
851
|
+
├─ Fork 3 (queue: default)
|
|
852
|
+
│ └─ Thread 1-10
|
|
853
|
+
│ └─ Total: 40 concurrent jobs for 'default' (4 forks × 10 threads)
|
|
854
|
+
└─ Fork 0 (queue: mailers)
|
|
855
|
+
└─ Thread 1-5
|
|
856
|
+
```
|
|
346
857
|
|
|
347
|
-
###
|
|
858
|
+
### Benefits
|
|
859
|
+
|
|
860
|
+
**Single Process Mode (forks: 0):**
|
|
861
|
+
- Simplest deployment
|
|
862
|
+
- Easy debugging
|
|
863
|
+
- Lower memory footprint
|
|
864
|
+
- Best for development and moderate load
|
|
865
|
+
|
|
866
|
+
**Multi-Process Mode (forks: 1+):**
|
|
867
|
+
- Horizontal scaling per queue
|
|
868
|
+
- Process isolation
|
|
869
|
+
- Automatic memory cleanup via GC limits
|
|
870
|
+
- Crash isolation (one fork down doesn't affect others)
|
|
871
|
+
- Best for production high-volume workloads
|
|
872
|
+
|
|
873
|
+
### GC Limits
|
|
874
|
+
|
|
875
|
+
Set `default_gc_limit` per worker to automatically restart after processing N jobs.
|
|
876
|
+
|
|
877
|
+
- Worker processes N jobs
|
|
878
|
+
- Worker exits with code 99
|
|
879
|
+
- In single-process mode (forks: 0): process manager restarts the entire process
|
|
880
|
+
- In multi-process mode (forks: 1+): parent process automatically restarts just that fork
|
|
881
|
+
|
|
882
|
+
```yaml
|
|
883
|
+
production:
|
|
884
|
+
workers:
|
|
885
|
+
imports:
|
|
886
|
+
default_forks: 4
|
|
887
|
+
default_threads: 1
|
|
888
|
+
default_gc_limit: 500 # Restart after 500 jobs (memory-intensive)
|
|
889
|
+
queues:
|
|
890
|
+
- imports
|
|
891
|
+
- data_processing
|
|
892
|
+
|
|
893
|
+
general:
|
|
894
|
+
default_forks: 2
|
|
895
|
+
default_threads: 100
|
|
896
|
+
default_gc_limit: 5000 # Restart after 5000 jobs
|
|
897
|
+
queues:
|
|
898
|
+
- default
|
|
899
|
+
- mailers
|
|
900
|
+
```
|
|
348
901
|
|
|
902
|
+
**Why?**
|
|
903
|
+
- Prevents memory bloat in long-running workers
|
|
904
|
+
- Automatic cleanup without manual intervention
|
|
905
|
+
- Works seamlessly with Docker and Kubernetes restarts
|
|
906
|
+
|
|
907
|
+
## Configuration
|
|
908
|
+
|
|
909
|
+
### YAML Configuration
|
|
910
|
+
|
|
911
|
+
```yaml
|
|
912
|
+
# config/postburner.yml
|
|
913
|
+
default: &default
|
|
914
|
+
beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
|
|
915
|
+
|
|
916
|
+
development:
|
|
917
|
+
<<: *default
|
|
918
|
+
# Defaults: forks: 0, threads: 1, gc_limit: nil
|
|
919
|
+
queues:
|
|
920
|
+
default: {}
|
|
921
|
+
mailers: {}
|
|
922
|
+
|
|
923
|
+
test:
|
|
924
|
+
<<: *default
|
|
925
|
+
# Defaults: forks: 0, threads: 1, gc_limit: nil
|
|
926
|
+
queues:
|
|
927
|
+
default: {}
|
|
928
|
+
|
|
929
|
+
staging:
|
|
930
|
+
<<: *default
|
|
931
|
+
default_threads: 10 # Multi-threaded, single process
|
|
932
|
+
default_gc_limit: 5000
|
|
933
|
+
queues:
|
|
934
|
+
default: {}
|
|
935
|
+
mailers: {}
|
|
936
|
+
|
|
937
|
+
production:
|
|
938
|
+
<<: *default
|
|
939
|
+
default_forks: 4 # Puma-style: multiple processes
|
|
940
|
+
default_threads: 10 # 10 threads per fork
|
|
941
|
+
default_gc_limit: 5000
|
|
942
|
+
|
|
943
|
+
queues:
|
|
944
|
+
critical:
|
|
945
|
+
forks: 1
|
|
946
|
+
threads: 1 # 1 concurrent job
|
|
947
|
+
gc_limit: 100
|
|
948
|
+
|
|
949
|
+
default:
|
|
950
|
+
forks: 4
|
|
951
|
+
threads: 10 # 40 total concurrent jobs (4 × 10)
|
|
952
|
+
gc_limit: 1000
|
|
953
|
+
|
|
954
|
+
mailers:
|
|
955
|
+
forks: 2
|
|
956
|
+
threads: 5 # 10 total email senders (2 × 5)
|
|
957
|
+
gc_limit: 500
|
|
349
958
|
```
|
|
350
|
-
|
|
351
|
-
|
|
959
|
+
|
|
960
|
+
### Queue Configuration Methods
|
|
961
|
+
|
|
962
|
+
For `Postburner::Job` subclasses (and backwards compatibility):
|
|
963
|
+
|
|
964
|
+
```ruby
|
|
965
|
+
class CriticalJob < Postburner::Job
|
|
966
|
+
queue 'critical' # Beanstalkd tube name
|
|
967
|
+
queue_priority 0 # Lower = higher priority (0 is highest)
|
|
968
|
+
queue_ttr 300 # TTR (time-to-run) in seconds
|
|
969
|
+
queue_max_job_retries 3 # Max retry attempts
|
|
970
|
+
queue_retry_delay 5 # Fixed delay: 5 seconds between retries
|
|
971
|
+
|
|
972
|
+
def perform(args)
|
|
973
|
+
# ...
|
|
974
|
+
end
|
|
975
|
+
end
|
|
352
976
|
```
|
|
353
977
|
|
|
354
|
-
|
|
978
|
+
**Exponential backoff with proc:**
|
|
355
979
|
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
Where <nac> is an authorized key with push capabilities.
|
|
980
|
+
```ruby
|
|
981
|
+
class BackgroundTask < Postburner::Job
|
|
982
|
+
queue 'default'
|
|
983
|
+
queue_max_job_retries 5
|
|
984
|
+
queue_retry_delay ->(retries) { 2 ** retries } # 2s, 4s, 8s, 16s, 32s
|
|
362
985
|
|
|
986
|
+
def perform(args)
|
|
987
|
+
# ...
|
|
988
|
+
end
|
|
989
|
+
end
|
|
990
|
+
```
|
|
991
|
+
|
|
992
|
+
### ActiveJob Configuration
|
|
993
|
+
|
|
994
|
+
For ActiveJob classes, use standard ActiveJob configuration:
|
|
995
|
+
|
|
996
|
+
```ruby
|
|
997
|
+
class MyJob < ApplicationJob
|
|
998
|
+
include Postburner::Tracked # Optional: for audit trail
|
|
999
|
+
|
|
1000
|
+
queue_as :default
|
|
1001
|
+
|
|
1002
|
+
retry_on StandardError, wait: :exponentially_longer, attempts: 5
|
|
1003
|
+
retry_on CustomError, wait: 10.seconds
|
|
1004
|
+
|
|
1005
|
+
discard_on ActiveJob::DeserializationError
|
|
1006
|
+
|
|
1007
|
+
def perform(user_id)
|
|
1008
|
+
# ...
|
|
1009
|
+
end
|
|
1010
|
+
end
|
|
1011
|
+
```
|
|
1012
|
+
|
|
1013
|
+
## Callbacks
|
|
1014
|
+
|
|
1015
|
+
Postburner provides lifecycle callbacks:
|
|
1016
|
+
|
|
1017
|
+
```ruby
|
|
1018
|
+
class ProcessPayment < ApplicationJob
|
|
1019
|
+
include Postburner::Tracked
|
|
1020
|
+
|
|
1021
|
+
before_enqueue :validate_payment
|
|
1022
|
+
after_enqueue :send_confirmation
|
|
1023
|
+
|
|
1024
|
+
before_attempt :log_attempt # Runs on every retry
|
|
1025
|
+
after_attempt :update_metrics
|
|
1026
|
+
|
|
1027
|
+
before_processing :acquire_lock
|
|
1028
|
+
after_processing :release_lock
|
|
1029
|
+
|
|
1030
|
+
after_processed :send_receipt # Only on success
|
|
1031
|
+
|
|
1032
|
+
def perform(payment_id)
|
|
1033
|
+
# ...
|
|
1034
|
+
end
|
|
1035
|
+
|
|
1036
|
+
private
|
|
1037
|
+
|
|
1038
|
+
def validate_payment
|
|
1039
|
+
raise "Invalid" unless payment_valid?
|
|
1040
|
+
end
|
|
1041
|
+
|
|
1042
|
+
def send_receipt
|
|
1043
|
+
PaymentMailer.receipt(arguments.first).deliver_later
|
|
1044
|
+
end
|
|
1045
|
+
end
|
|
1046
|
+
```
|
|
1047
|
+
|
|
1048
|
+
**Available callbacks:**
|
|
1049
|
+
- `before_enqueue`, `around_enqueue`, `after_enqueue` - When queued
|
|
1050
|
+
- `before_attempt`, `around_attempt`, `after_attempt` - Every execution (including retries)
|
|
1051
|
+
- `before_processing`, `around_processing`, `after_processing` - During execution
|
|
1052
|
+
- `after_processed` - Only after successful completion
|
|
1053
|
+
|
|
1054
|
+
## Queue Strategies
|
|
1055
|
+
|
|
1056
|
+
Postburner uses different strategies to control job execution. These affect `Postburner::Job` subclasses (not ActiveJob classes).
|
|
1057
|
+
|
|
1058
|
+
| Strategy | When to Use | Behavior | Requires Beanstalkd |
|
|
1059
|
+
|----------|-------------|----------|---------------------|
|
|
1060
|
+
| **NiceQueue** (default) | Production | Async via Beanstalkd, gracefully re-queues premature jobs | Yes |
|
|
1061
|
+
| **Queue** | Production (strict) | Async via Beanstalkd, raises error on premature execution | Yes |
|
|
1062
|
+
| **TestQueue** | Testing with explicit time control | Inline execution, raises error for scheduled jobs | No |
|
|
1063
|
+
| **ImmediateTestQueue** | Testing with automatic time travel | Inline execution, auto time-travels for scheduled jobs | No |
|
|
1064
|
+
| **NullQueue** | Batch processing / deferred execution | Jobs created but not queued, manual execution | No |
|
|
1065
|
+
|
|
1066
|
+
```ruby
|
|
1067
|
+
# Switch strategies
|
|
1068
|
+
Postburner.nice_async_strategy! # Default production (NiceQueue)
|
|
1069
|
+
Postburner.async_strategy! # Strict production (Queue)
|
|
1070
|
+
Postburner.inline_test_strategy! # Testing (TestQueue)
|
|
1071
|
+
Postburner.inline_immediate_test_strategy! # Testing with time travel
|
|
1072
|
+
Postburner.null_strategy! # Deferred execution
|
|
1073
|
+
```
|
|
1074
|
+
|
|
1075
|
+
**Note:** These strategies only affect `Postburner::Job` subclasses. ActiveJob classes execute according to the ActiveJob adapter configuration.
|
|
1076
|
+
|
|
1077
|
+
## Testing
|
|
1078
|
+
|
|
1079
|
+
Postburner provides test-friendly execution modes that don't require Beanstalkd.
|
|
1080
|
+
|
|
1081
|
+
### Automatic Test Mode
|
|
1082
|
+
|
|
1083
|
+
In Rails test environments, Postburner automatically uses inline execution:
|
|
1084
|
+
|
|
1085
|
+
```ruby
|
|
1086
|
+
# test/test_helper.rb - automatic!
|
|
1087
|
+
Postburner.testing? # => true in tests
|
|
1088
|
+
```
|
|
1089
|
+
|
|
1090
|
+
### Testing Default Jobs (ActiveJob)
|
|
1091
|
+
|
|
1092
|
+
Use standard ActiveJob test helpers:
|
|
1093
|
+
|
|
1094
|
+
```ruby
|
|
1095
|
+
class MyTest < ActiveSupport::TestCase
|
|
1096
|
+
test "job executes" do
|
|
1097
|
+
SendEmail.perform_later(123)
|
|
1098
|
+
# Job executes immediately in test mode
|
|
1099
|
+
end
|
|
1100
|
+
|
|
1101
|
+
test "job with delay" do
|
|
1102
|
+
travel 1.hour do
|
|
1103
|
+
SendEmail.set(wait: 1.hour).perform_later(123)
|
|
1104
|
+
end
|
|
1105
|
+
end
|
|
1106
|
+
end
|
|
1107
|
+
```
|
|
1108
|
+
|
|
1109
|
+
### Testing Tracked Jobs
|
|
1110
|
+
|
|
1111
|
+
Tracked jobs create database records you can inspect:
|
|
1112
|
+
|
|
1113
|
+
```ruby
|
|
1114
|
+
test "tracked job logs execution" do
|
|
1115
|
+
ProcessPayment.perform_later(456)
|
|
1116
|
+
|
|
1117
|
+
job = Postburner::TrackedJob.last
|
|
1118
|
+
assert job.processed_at.present?
|
|
1119
|
+
assert_includes job.logs.map { |l| l[1]['message'] }, "Processing payment 456"
|
|
1120
|
+
end
|
|
1121
|
+
```
|
|
1122
|
+
|
|
1123
|
+
### Testing Legacy Postburner::Job
|
|
1124
|
+
|
|
1125
|
+
```ruby
|
|
1126
|
+
test "processes immediately" do
|
|
1127
|
+
job = RunDonation.create!(args: { 'donation_id' => 123 })
|
|
1128
|
+
job.queue!
|
|
1129
|
+
|
|
1130
|
+
assert job.reload.processed_at
|
|
1131
|
+
end
|
|
1132
|
+
|
|
1133
|
+
test "scheduled job with time travel" do
|
|
1134
|
+
job = RunDonation.create!(args: { 'donation_id' => 123 })
|
|
1135
|
+
|
|
1136
|
+
travel_to(2.hours.from_now) do
|
|
1137
|
+
job.queue!(delay: 2.hours)
|
|
1138
|
+
assert job.reload.processed_at
|
|
1139
|
+
end
|
|
1140
|
+
end
|
|
1141
|
+
```
|
|
1142
|
+
|
|
1143
|
+
## Job Management
|
|
1144
|
+
|
|
1145
|
+
### Canceling Jobs
|
|
1146
|
+
|
|
1147
|
+
**For Postburner::Job subclasses:**
|
|
1148
|
+
|
|
1149
|
+
```ruby
|
|
1150
|
+
job = RunDonation.create!(args: { 'donation_id' => 123 })
|
|
1151
|
+
job.queue!(delay: 1.hour)
|
|
1152
|
+
|
|
1153
|
+
# Soft delete (recommended) - removes from queue, sets removed_at
|
|
1154
|
+
job.remove!
|
|
1155
|
+
job.removed_at # => 2025-11-19 12:34:56 UTC
|
|
1156
|
+
job.persisted? # => true (still in database for audit)
|
|
1157
|
+
|
|
1158
|
+
# Hard delete - removes from both queue and database
|
|
1159
|
+
job.destroy!
|
|
1160
|
+
|
|
1161
|
+
# Delete from Beanstalkd only (keeps database record)
|
|
1162
|
+
job.delete!
|
|
1163
|
+
```
|
|
1164
|
+
|
|
1165
|
+
**For tracked ActiveJob classes:**
|
|
1166
|
+
|
|
1167
|
+
```ruby
|
|
1168
|
+
# Find the TrackedJob record
|
|
1169
|
+
job = Postburner::TrackedJob.find(123)
|
|
1170
|
+
job.remove! # Cancel execution
|
|
1171
|
+
```
|
|
1172
|
+
|
|
1173
|
+
### Retrying Buried Jobs
|
|
1174
|
+
|
|
1175
|
+
Jobs that fail repeatedly get "buried" in Beanstalkd:
|
|
1176
|
+
|
|
1177
|
+
```ruby
|
|
1178
|
+
job = Postburner::Job.find(123)
|
|
1179
|
+
job.kick! # Moves buried job back to ready queue
|
|
1180
|
+
```
|
|
1181
|
+
|
|
1182
|
+
### Inspecting Job State
|
|
1183
|
+
|
|
1184
|
+
```ruby
|
|
1185
|
+
job = Postburner::TrackedJob.find(123)
|
|
1186
|
+
|
|
1187
|
+
# Timestamps
|
|
1188
|
+
job.queued_at # When queued to Beanstalkd
|
|
1189
|
+
job.run_at # Scheduled execution time
|
|
1190
|
+
job.attempting_at # First attempt started
|
|
1191
|
+
job.processing_at # Current/last processing started
|
|
1192
|
+
job.processed_at # Completed successfully
|
|
1193
|
+
job.removed_at # Soft deleted
|
|
1194
|
+
|
|
1195
|
+
# Statistics
|
|
1196
|
+
job.lag # Milliseconds between scheduled and actual execution
|
|
1197
|
+
job.duration # Milliseconds to execute
|
|
1198
|
+
job.attempt_count # Number of attempts
|
|
1199
|
+
job.error_count # Number of errors
|
|
1200
|
+
|
|
1201
|
+
# Audit trail
|
|
1202
|
+
job.logs # Array of log entries with timestamps
|
|
1203
|
+
job.errata # Array of exceptions with backtraces
|
|
1204
|
+
job.attempts # Array of attempt timestamps
|
|
1205
|
+
```
|
|
1206
|
+
|
|
1207
|
+
## Beanstalkd Integration
|
|
1208
|
+
|
|
1209
|
+
Direct access to Beanstalkd for advanced operations:
|
|
1210
|
+
|
|
1211
|
+
```ruby
|
|
1212
|
+
# Get Beanstalkd job ID
|
|
1213
|
+
job.bkid # => 12345
|
|
1214
|
+
|
|
1215
|
+
# Access Beaneater job object
|
|
1216
|
+
job.beanstalk_job.stats
|
|
1217
|
+
# => {"id"=>12345, "tube"=>"critical", "state"=>"ready", ...}
|
|
1218
|
+
|
|
1219
|
+
# Connection management
|
|
1220
|
+
Postburner.connected do |conn|
|
|
1221
|
+
conn.tubes.to_a # List all tubes
|
|
1222
|
+
conn.tubes['postburner.production.critical'].stats
|
|
1223
|
+
conn.tubes['postburner.production.critical'].kick(10) # Kick 10 buried jobs
|
|
1224
|
+
end
|
|
1225
|
+
```
|
|
1226
|
+
|
|
1227
|
+
|
|
1228
|
+
## Web UI
|
|
1229
|
+
|
|
1230
|
+
Mount the inspection interface:
|
|
1231
|
+
|
|
1232
|
+
```ruby
|
|
1233
|
+
# config/routes.rb
|
|
1234
|
+
mount Postburner::Engine => "/postburner"
|
|
1235
|
+
|
|
1236
|
+
# Only in development
|
|
1237
|
+
mount Postburner::Engine => "/postburner" if Rails.env.development?
|
|
1238
|
+
```
|
|
1239
|
+
|
|
1240
|
+
Add your own authentication:
|
|
1241
|
+
|
|
1242
|
+
```ruby
|
|
1243
|
+
# config/routes.rb
|
|
1244
|
+
authenticate :user, ->(user) { user.admin? } do
|
|
1245
|
+
mount Postburner::Engine => "/postburner"
|
|
1246
|
+
end
|
|
1247
|
+
```
|
|
1248
|
+
|
|
1249
|
+
## Deployment
|
|
1250
|
+
|
|
1251
|
+
### Docker
|
|
1252
|
+
|
|
1253
|
+
```dockerfile
|
|
1254
|
+
# Dockerfile.worker
|
|
1255
|
+
FROM ruby:3.3
|
|
1256
|
+
|
|
1257
|
+
WORKDIR /app
|
|
1258
|
+
COPY . .
|
|
1259
|
+
RUN bundle install
|
|
1260
|
+
|
|
1261
|
+
CMD ["bundle", "exec", "postburner", "--config", "config/postburner.yml", "--env", "production"]
|
|
1262
|
+
```
|
|
1263
|
+
|
|
1264
|
+
```yaml
|
|
1265
|
+
# docker-compose.yml
|
|
1266
|
+
services:
|
|
1267
|
+
beanstalkd:
|
|
1268
|
+
image: schickling/beanstalkd
|
|
1269
|
+
command: ["-l", "0.0.0.0", "-p", "11300", "-b", "/var/lib/beanstalkd"]
|
|
1270
|
+
ports:
|
|
1271
|
+
- "11300:11300"
|
|
1272
|
+
volumes:
|
|
1273
|
+
- beanstalkd_data:/var/lib/beanstalkd
|
|
1274
|
+
|
|
1275
|
+
worker:
|
|
1276
|
+
build:
|
|
1277
|
+
context: .
|
|
1278
|
+
dockerfile: Dockerfile.worker
|
|
1279
|
+
depends_on:
|
|
1280
|
+
- beanstalkd
|
|
1281
|
+
- postgres
|
|
1282
|
+
environment:
|
|
1283
|
+
BEANSTALK_URL: beanstalk://beanstalkd:11300
|
|
1284
|
+
DATABASE_URL: postgres://postgres@postgres/myapp_production
|
|
1285
|
+
|
|
1286
|
+
volumes:
|
|
1287
|
+
beanstalkd_data:
|
|
1288
|
+
```
|
|
1289
|
+
|
|
1290
|
+
## Migration from v0.x
|
|
1291
|
+
|
|
1292
|
+
Key changes in v1.0:
|
|
1293
|
+
|
|
1294
|
+
### Removed
|
|
1295
|
+
- Backburner dependency
|
|
1296
|
+
- `config/initializers/backburner.rb`
|
|
1297
|
+
- Backburner worker (`bundle exec backburner`)
|
|
1298
|
+
|
|
1299
|
+
### Added
|
|
1300
|
+
- ActiveJob adapter (first-class Rails integration)
|
|
1301
|
+
- Default jobs (Beanstalkd only, no PostgreSQL)
|
|
1302
|
+
- `bin/postburner` executable
|
|
1303
|
+
- `config/postburner.yml` configuration
|
|
1304
|
+
- Multiple worker types (Simple, Forking, ThreadsOnFork)
|
|
1305
|
+
|
|
1306
|
+
### Migration Steps
|
|
1307
|
+
|
|
1308
|
+
1. **Update Gemfile:**
|
|
1309
|
+
```ruby
|
|
1310
|
+
gem 'postburner', '~> 1.0.0.pre.1'
|
|
1311
|
+
```
|
|
1312
|
+
|
|
1313
|
+
2. **Remove Backburner config:**
|
|
1314
|
+
```bash
|
|
1315
|
+
rm config/initializers/backburner.rb
|
|
1316
|
+
```
|
|
1317
|
+
|
|
1318
|
+
3. **Create Postburner config:**
|
|
1319
|
+
```bash
|
|
1320
|
+
cp config/postburner.yml.example config/postburner.yml
|
|
1321
|
+
```
|
|
1322
|
+
|
|
1323
|
+
4. **Update ActiveJob adapter:**
|
|
1324
|
+
```ruby
|
|
1325
|
+
# config/application.rb
|
|
1326
|
+
config.active_job.queue_adapter = :postburner # was :backburner
|
|
1327
|
+
```
|
|
1328
|
+
|
|
1329
|
+
5. **Update worker command:**
|
|
1330
|
+
```bash
|
|
1331
|
+
# Old
|
|
1332
|
+
bundle exec backburner
|
|
1333
|
+
|
|
1334
|
+
# New
|
|
1335
|
+
bundle exec postburner --config config/postburner.yml
|
|
1336
|
+
```
|
|
1337
|
+
|
|
1338
|
+
6. **Existing Jobs:** Direct `Postburner::Job` subclasses continue to work without changes!
|
|
1339
|
+
|
|
1340
|
+
## Contributing
|
|
1341
|
+
|
|
1342
|
+
Submit a pull request. Follow project conventions. Be nice.
|
|
363
1343
|
|
|
364
1344
|
## License
|
|
1345
|
+
|
|
365
1346
|
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|