postburner 0.8.0 → 1.0.0.pre.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -1,365 +1,1346 @@
1
1
  # Postburner
2
- An ActiveRecord layer on top of [Backburner](https://github.com/nesquena/backburner) for inspecting and auditing the
3
- queue, especially for delayed jobs. It isn't meant to be outperform other queues, but be safe (and inspectable).
4
2
 
5
- It is meant to be complementary to [Backburner](https://github.com/nesquena/backburner). Use Backburner as the default
6
- ActiveJob processor for mailers, active storage, and the like. Use a
7
- `Postburner::Job` for things that you want to track. See [Comparison to Backburner](#comparison-to-backburner) for more.
3
+ Fast Beanstalkd-backed job queue with **optional PostgreSQL audit trail** for ActiveJob.
8
4
 
9
- Postburner meant to be a replacement/upgrade for [Que](https://github.com/que-rb/que).
10
- If you need something faster, check out Que - we love it! Postburner is built for a slightly different purpose: to be
11
- simple, safe, and inspectable. All of which Que has (or can be added), but are included by default with some architecture
12
- differences. See [Comparison to Que](#comparison-to-que) for more.
5
+ Postburner provides dual-mode job execution:
6
+ - **Default jobs**: Fast execution via Beanstalkd only (no PostgreSQL overhead)
7
+ - **Tracked jobs**: Full audit trail with logs, timing, errors, and statistics
13
8
 
14
- ## Usage
9
+ Built for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
10
+
11
+ ## Features
12
+
13
+ - **ActiveJob native** - Works seamlessly with Rails, ActionMailer, ActiveStorage
14
+ - **Dual-mode execution** - Default or tracked (database backed)
15
+ - **Rich audit trail** - Logs, timing, errors, retry tracking (tracked jobs only)
16
+ - **ActiveRecord** - Query jobs with ActiveRecord (Tracked jobs only)
17
+ - **Beanstalkd** - Fast, reliable queue separate from your database, peristent storage
18
+ - **Process isolation** - Forking workers with optional threading for throughput
19
+ - **Test-friendly** - Inline execution without Beanstalkd in tests
20
+
21
+ ## Quick Start
22
+
23
+ **Installation:**
24
+
25
+ ```bash
26
+ sudo apt-get install beanstalkd # OR brew install beanstalkd
27
+
28
+ # Start beanstalkd (in-memory only)
29
+ beanstalkd -l 127.0.0.1 -p 11300
30
+
31
+ # OR with persistence (recommended for production)
32
+ mkdir -p /var/lib/beanstalkd
33
+ beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
34
+ ```
15
35
 
16
36
  ```ruby
17
- class RunDonation < Postburner::Job
18
- queue 'critical'
19
- queue_priority 0 # 0 is highest priority
20
- queue_max_job_retries 0 # don't retry
37
+ # Gemfile
38
+ gem 'postburner', '~> 1.0.0.pre.1'
39
+ ```
21
40
 
22
- def perform(args)
23
- # do long tasks here
24
- # also have access to self.args
41
+ ```bash
42
+ bundle install
43
+ rails generate postburner:install
44
+ rails db:migrate
45
+ ```
46
+
47
+ **Configuration:**
48
+
49
+ ```ruby
50
+ # config/application.rb
51
+ config.active_job.queue_adapter = :postburner
52
+ ```
53
+
54
+ ```yaml
55
+ # config/postburner.yml
56
+ production:
57
+ beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
58
+ worker_type: simple
59
+ queues:
60
+ default: {}
61
+ critical: {}
62
+ mailers: {}
63
+ ```
64
+
65
+ **Usage:**
66
+
67
+ ```ruby
68
+ # Default job (fast, no PostgreSQL overhead)
69
+ class SendEmail < ApplicationJob
70
+ def perform(user_id)
71
+ UserMailer.welcome(user_id).deliver_now
25
72
  end
26
73
  end
27
74
 
28
- # RunDonation#create! is the `ActiveRecord` method, so it returns a model,
29
- # you can manipulate, add columns to the table to, reference with foreign
30
- # keys, etc.
31
- job = RunDonation.create!(args: {donation_id: 123})
32
- # NOTE Make sure use use an after_commit or after_save_commit, etc to avoid
33
- # any race conditions of Postburner trying to process before a required
34
- # database mutation is commited.
35
- job.queue!
36
- => {:status=>"INSERTED", :id=>"1139"}
75
+ # Default job with Beanstalkd configuration
76
+ class SendEmail < ApplicationJob
77
+ include Postburner::Beanstalkd
78
+
79
+ queue_priority 100 # Custom priority
80
+ queue_ttr 300 # 5 minute timeout
81
+
82
+ def perform(user_id)
83
+ UserMailer.welcome(user_id).deliver_now
84
+ end
85
+ end
86
+
87
+ # Tracked job (full audit trail, includes Beanstalkd automatically)
88
+ class ProcessPayment < ApplicationJob
89
+ include Postburner::Tracked # ← Opt-in to tracking (includes Beanstalkd)
90
+
91
+ queue_priority 0 # Highest priority
92
+ queue_ttr 600 # 10 minute timeout
37
93
 
38
- # queue for later with `:at`
39
- RunDonation.create!(args: {donation_id: 123}).queue! at: Time.zone.now + 2.days
40
- => {:status=>"INSERTED", :id=>"1140"}
94
+ def perform(payment_id)
95
+ log "Processing payment #{payment_id}"
96
+ Payment.find(payment_id).process!
97
+ log! "Payment processed successfully"
98
+ end
99
+ end
41
100
 
42
- # queue for later with `:delay`
43
- #
44
- # `:delay` takes priority over `:at`takes priority over `:at` because the
45
- # beanstalkd protocol uses uses `delay`
46
- RunDonation.create!(args: {donation_id: 123}).queue! delay: 1.hour
47
- => {:status=>"INSERTED", :id=>"1141"}
101
+ # Run worker
102
+ bin/postburner --config config/postburner.yml --env production
48
103
  ```
49
104
 
50
- ### Mailers
105
+ ## Table of Contents
51
106
 
52
- ```ruby
53
- j = Postburner::Mailer.
54
- delivery(UserMailer, :welcome)
55
- .with(name: 'Freddy')
107
+ - [Why Beanstalkd?](#why-beanstalkd)
108
+ - [Installation](#installation)
109
+ - [Usage](#usage)
110
+ - [Standard Jobs](#standard-jobs)
111
+ - [Tracked Jobs](#tracked-jobs)
112
+ - [Postburner::Job](#postburnerjob)
113
+ - [Workers](#workers)
114
+ - [Configuration](#configuration)
115
+ - [Queue Strategies](#queue-strategies)
116
+ - [Testing](#testing)
117
+ - [Job Management](#job-management)
118
+ - [Beanstalkd Integration](#beanstalkd-integration)
119
+ - [Web UI](#web-ui)
56
120
 
57
- j.queue!
58
- => {:status=>"INSERTED", :id=>"1139"}
121
+ ## Why Beanstalkd?
122
+
123
+ Beanstalkd is a simple, fast, and reliable queue system. It is a good choice for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
124
+
125
+ The [protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt) reads more like a README than a protocol. Check it out and you will instantly understand how it works.
126
+
127
+ ### Binlogs
128
+
129
+ Beanstalkd lets you persist jobs to disk in case of a crash or restart. Just restart beanstalkd and the jobs will be back in the queue.
130
+
131
+ **Setup:**
132
+
133
+ ```bash
134
+ # Create binlog directory
135
+ sudo mkdir -p /var/lib/beanstalkd
136
+ sudo chown beanstalkd:beanstalkd /var/lib/beanstalkd # If running as service
137
+
138
+ # Start beanstalkd with binlog persistence
139
+ beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
59
140
  ```
60
141
 
61
- ### [Beaneater](https://github.com/beanstalkd/beaneater) and [beanstalkd](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt) attributes and methods
62
- ```ruby
63
- # get the beanstalkd job id
64
- job.bkid
65
- => 1104
142
+ **Configuration options:**
66
143
 
67
- # get the beaneater job, call any beaneater methods on this object
68
- job.beanstalk_job
144
+ ```bash
145
+ # Basic persistence
146
+ beanstalkd -b /var/lib/beanstalkd
69
147
 
70
- # get the beanstald stats
71
- job.beanstalk_job.stats
148
+ # With custom binlog file size (default varies)
149
+ beanstalkd -b /var/lib/beanstalkd -s 10485760 # 10MB per binlog file
72
150
 
73
- # kick the beanstalk job
74
- job.kick!
151
+ # Reduce fsync frequency for better performance (trades safety for speed)
152
+ beanstalkd -b /var/lib/beanstalkd -f 200 # fsync at most every 200ms (default: 50ms)
75
153
 
76
- # delete beankstalkd job, retain the job model
77
- job.delete!
154
+ # Never fsync (maximum performance, higher risk of data loss on power failure)
155
+ beanstalkd -b /var/lib/beanstalkd -F
156
+ ```
78
157
 
79
- # delete beankstalkd job, retain the job model, but set `removed_at` on model.
80
- job.remove!
158
+ **Options:**
159
+ - `-b <dir>` - Enable binlog persistence in specified directory
160
+ - `-s <bytes>` - Maximum size of each binlog file (requires `-b`)
161
+ - `-f <ms>` - Call fsync at most once every N milliseconds (requires `-b`, default: 50ms)
162
+ - `-F` - Never call fsync (requires `-b`, maximum performance but higher data loss risk)
163
+
164
+ **Note:** The fsync options control the trade-off between performance and safety. A power failure could result in loss of jobs added within the fsync interval.
165
+
166
+ ### Priority
167
+
168
+ Priority controls the order in which jobs are processed from the queue. Beanstalkd always processes the highest priority (lowest number) jobs first.
169
+
170
+ **Priority Range:** 0 to 4,294,967,295 (unsigned 32-bit integer)
171
+ - `0` = Highest priority (processed first)
172
+ - `4,294,967,295` = Lowest priority (processed last)
173
+ - Default: `65536` (configurable in `config/postburner.yml`)
174
+
175
+ When a worker reserves a job, Beanstalkd returns the highest priority job available in the tube. Jobs with the same priority are processed in FIFO order.
176
+
177
+ **ActiveJob with Postburner::Beanstalkd:**
178
+
179
+ ```ruby
180
+ class ProcessPayment < ApplicationJob
181
+ include Postburner::Beanstalkd
182
+
183
+ queue_as :critical
184
+ queue_priority 0 # Highest priority - processed immediately
185
+ end
81
186
 
82
- # or simply remove the model, which will clean up the beanstalkd job in a before_destroy hook
83
- job.destroy # OR job.destroy!
84
-
85
- # get a cached Backburner connection and inspect it (or even use it directly)
86
- c = Postburner.connection
87
- c.beanstalk.tubes.to_a
88
- c.beanstalk.tubes.to_a.map{|t| c.tubes[t.name].peek(:buried)}
89
- c.beanstalk.tubes['ss.development.caching'].stats
90
- c.beanstalk.tubes['ss.development.caching'].peek(:buried).kick
91
- c.beanstalk.tubes['ss.development.caching'].kick(3)
92
- c.close
93
-
94
- # automatically close
95
- Postburner.connected do |connection|
96
- # do stuff with connection
187
+ class SendEmail < ApplicationJob
188
+ include Postburner::Beanstalkd
189
+
190
+ queue_as :mailers
191
+ queue_priority 1000 # Lower priority - processed after critical jobs
97
192
  end
98
193
  ```
99
194
 
100
- Read about the [beanstalkd protocol](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt).
195
+ **Postburner::Job:**
101
196
 
102
- ### Basic model fields
103
197
  ```ruby
104
- # ActiveRecord primary key
105
- job.id
198
+ class CriticalJob < Postburner::Job
199
+ queue 'critical'
200
+ queue_priority 0 # Highest priority
201
+
202
+ def perform(args)
203
+ # Critical business logic
204
+ end
205
+ end
206
+
207
+ class BackgroundTask < Postburner::Job
208
+ queue 'default'
209
+ queue_priority 5000 # Lower priority
210
+
211
+ def perform(args)
212
+ # Non-urgent background work
213
+ end
214
+ end
215
+ ```
106
216
 
107
- # int id of the beankstalkd job
108
- job.bkid
217
+ **Recommended Priority Ranges:**
109
218
 
110
- # string uuid
111
- job.sid
219
+ | Priority Range | Use Case | Examples |
220
+ |---------------|----------|----------|
221
+ | 0-256 | Critical business operations | Payments, transactions, real-time notifications |
222
+ | 256-4096 | High priority tasks | Password resets, order confirmations |
223
+ | 4096-65536 | Standard background jobs | Email sending, data exports (65536 is default) |
224
+ | 65536-262144 | Low priority maintenance | Cache warming, log cleanup |
225
+ | 262144+ | Deferred/bulk operations | Analytics, batch reports |
112
226
 
113
- # ActiveRecord STI job subtype
114
- job.type
227
+ **Configuration:**
115
228
 
116
- # jsonb arguments for use in job
117
- job.args
229
+ Set default priority in `config/postburner.yml`:
118
230
 
119
- # time job should run - not intended to be changed
120
- # TODO run_at should be readonly after create
121
- job.run_at
231
+ ```yaml
232
+ production:
233
+ default_priority: 65536 # Default for jobs without explicit priority
234
+ default_ttr: 300
122
235
  ```
123
236
 
124
- ### Job statistics
237
+ **Dynamic Priority (Postburner::Job only):**
238
+
125
239
  ```ruby
126
- # when job was inserted into beankstalkd
127
- job.queued_at
240
+ class ProcessOrder < Postburner::Job
241
+ queue 'orders'
242
+ queue_priority 100 # Default
243
+
244
+ before_enqueue :adjust_priority
128
245
 
129
- # last time attempted
130
- job.attempting_at
246
+ def perform(args)
247
+ order = Order.find(args['order_id'])
248
+ order.process!
249
+ end
131
250
 
132
- # last time processing started
133
- job.processing_at
251
+ private
134
252
 
135
- # when completed
136
- job.processed_at
253
+ def adjust_priority
254
+ if args['urgent']
255
+ self.queue_priority = 0 # Override to highest priority
256
+ end
257
+ end
258
+ end
137
259
 
138
- # when removed, may be nil
139
- job.removed_at
260
+ # Usage
261
+ ProcessOrder.create!(args: { 'order_id' => 123, 'urgent' => true }).queue!
262
+ ```
140
263
 
141
- # lag in ms from run_at/queued_at to attempting_at
142
- job.lag
264
+ ### Time to Run (TTR)
143
265
 
144
- # duration of processing in ms
145
- job.duration
266
+ TTR (Time-to-Run) is the maximum duration in seconds that a worker has to process a reserved job before Beanstalkd automatically releases it back to the ready queue.
146
267
 
147
- # number of attempts
148
- job.attempt_count
268
+ 1. Worker reserves a job from Beanstalkd
269
+ 2. TTR countdown begins immediately upon reservation
270
+ 3. If job completes within TTR, worker deletes/buries the job (success)
271
+ 4. If TTR expires before completion, Beanstalkd automatically releases the job back to ready queue
272
+ 5. Another worker can then reserve and process the same job
149
273
 
150
- # number of errors (length of errata)
151
- job.error_count
274
+ This mechanism protects against workers crashing or hanging—jobs won't be stuck indefinitely.
152
275
 
153
- # number of log entries (length of logs)
154
- job.log_count
276
+ **TTR Range:** 1 to 4,294,967,295 seconds (unsigned 32-bit integer)
277
+ - Minimum: `1` second (Beanstalkd silently increases 0 to 1)
278
+ - Default: `300` seconds (5 minutes, configurable in `config/postburner.yml`)
155
279
 
156
- # array of attempting_at times
157
- job.attempts
280
+ **ActiveJob with Postburner::Beanstalkd:**
158
281
 
159
- # array of errors
160
- job.errata
282
+ ```ruby
283
+ class ProcessPayment < ApplicationJob
284
+ include Postburner::Beanstalkd
161
285
 
162
- # array of log messages
163
- job.logs
164
- ```
286
+ queue_as :critical
287
+ queue_priority 0
288
+ queue_ttr 300 # 5 minutes to complete
289
+ end
165
290
 
166
- ### Job logging and exceptions
291
+ class QuickEmail < ApplicationJob
292
+ include Postburner::Beanstalkd
167
293
 
168
- Optionally, you can:
169
- 1. Add log messages to the job during processing to `logs`
170
- 1. Add log your own exceptions to `errata`
294
+ queue_as :mailers
295
+ queue_ttr 60 # 1 minute for fast email jobs
296
+ end
297
+
298
+ class LongRunningReport < ApplicationJob
299
+ include Postburner::Beanstalkd
300
+
301
+ queue_as :reports
302
+ queue_ttr 3600 # 1 hour for complex reports
303
+ end
304
+ ```
305
+
306
+ **Postburner::Job:**
171
307
 
172
308
  ```ruby
173
- class RunDonation < Postburner::Job
174
- queue 'critical'
175
- queue_priority 0 # 0 is highest priority
176
- queue_max_job_retries 0 # don't retry
309
+ class DataImport < Postburner::Job
310
+ queue 'imports'
311
+ queue_ttr 1800 # 30 minutes (equivalent to queue_ttr)
177
312
 
178
313
  def perform(args)
179
- # log at task, defaults to `:info`, but `:debug`, `:warning`, `:error`
180
- log "Log bad condition...", level: :error
314
+ # Long-running import logic
315
+ end
316
+ end
317
+ ```
181
318
 
182
- begin
183
- # danger
184
- rescue Exception => e
185
- log_exception e
319
+ **Extending TTR with `extend!` (Touch Command):**
320
+
321
+ For jobs that may take longer than expected, you can extend the TTR dynamically using `extend!`. This calls Beanstalkd's `touch` command, which resets the TTR countdown.
322
+
323
+ **Available for:**
324
+ - `Postburner::Job` subclasses (always available via `bk.extend!`)
325
+ - `Postburner::Tracked` ActiveJob classes (includes `extend!` method)
326
+
327
+ ```ruby
328
+ class ProcessImport < ApplicationJob
329
+ include Postburner::Tracked # Includes Beanstalkd and enables extend!
330
+
331
+ queue_as :imports
332
+ queue_ttr 300 # 5 minutes initial TTR
333
+
334
+ def perform(file_id)
335
+ file = ImportFile.find(file_id)
336
+
337
+ file.each_batch(size: 100) do |batch|
338
+ batch.each do |row|
339
+ process_row(row)
340
+ end
341
+
342
+ extend! # Reset TTR to 300 seconds from now
343
+ log "Processed batch, extended TTR"
186
344
  end
187
345
  end
188
346
  end
189
347
  ```
190
348
 
191
- ### Optionally, mount the engine
349
+ **For Postburner::Job (via `bk` accessor):**
192
350
 
193
351
  ```ruby
194
- mount Postburner::Engine => "/postburner"
352
+ class LargeDataProcessor < Postburner::Job
353
+ queue 'processing'
354
+ queue_ttr 600 # 10 minutes
195
355
 
196
- # mount only for development inspection
197
- mount Postburner::Engine => "/postburner" if Rails.env.development?
356
+ def perform(args)
357
+ dataset = Dataset.find(args['dataset_id'])
358
+
359
+ dataset.each_chunk do |chunk|
360
+ process_chunk(chunk)
361
+
362
+ bk.extend! # Reset TTR via beanstalkd job accessor
363
+ log "Chunk processed, TTR extended"
364
+ end
365
+ end
366
+ end
367
+ ```
368
+
369
+ **`extend!` and Beanstalkd `touch`**
370
+
371
+ - Calls Beanstalkd's `touch` command on the reserved job
372
+ - Resets the TTR countdown to the original TTR value from the current moment
373
+ - Example: Job has `queue_ttr 300`. After 200 seconds, calling `extend!` gives you another 300 seconds (not just 100)
374
+ - Can be called multiple times during job execution
375
+ - Useful for iterative processing where each iteration is quick but total time is unpredictable
376
+
377
+ **Configuration:**
378
+
379
+ Set default TTR in `config/postburner.yml`:
380
+
381
+ ```yaml
382
+ production:
383
+ default_priority: 65536
384
+ default_ttr: 300 # 5 minutes default for all jobs
198
385
  ```
199
386
 
200
- [Open the controller](https://guides.rubyonrails.org/engines.html#overriding-models-and-controllers)
201
- to add your own authentication or changes - or just create your own routes, controllers, and views.
387
+ **Recommended TTR Values:**
388
+
389
+ | Job Type | TTR | Reasoning |
390
+ |----------|-----|-----------|
391
+ | Email sending | 60-120s | Network operations, should be fast |
392
+ | API calls | 30-60s | External services, fast or timeout |
393
+ | File processing | 600-1800s | Variable based on file size, use `extend!` |
394
+ | Reports | 1800-3600s | Complex queries and aggregations |
395
+ | Imports/Exports | 3600s+ | Large datasets, use `extend!` in loops |
202
396
 
203
- [Override the views](https://guides.rubyonrails.org/engines.html#overriding-views) to make them
204
- prettier - or follow the suggestion above and use your own.
397
+ **Best Practices:**
205
398
 
206
- ## Known Issues
399
+ 1. **Set realistic TTR values** based on expected job duration plus buffer
400
+ 2. **Use `extend!` for iterative work** where total time is unpredictable but each iteration is bounded
401
+ 3. **Don't set TTR too high** - it delays recovery from crashed workers
402
+ 4. **Monitor TTR timeouts** - frequent timeouts indicate jobs need more time or optimization
403
+ 5. **For Default jobs** (ActiveJob without `Postburner::Beanstalkd`), include the module to set custom TTR
207
404
 
208
- 1. Beaneater and/or Beanstalkd seems to transform a `tube` `name` with hyphens to
209
- underscores upon instrospection of `stats`. See example. Reccommend using
210
- names without hyphens / dashes.
405
+ **What happens on TTR timeout:**
211
406
 
212
407
  ```ruby
213
- (main):001:0> Postburner.connection.tubes["backburner.worker.queue.cat-dog"].put("{}", delay: 1.week.to_i)
214
- => {:status=>"INSERTED", :id=>"9"}
215
- irb(main):002:0> Postburner.connection.jobs.find(9)
216
- => #<Beaneater::Job id=9 body="{}">
217
- irb(main):003:0> Postburner.connection.jobs.find(9).stats
218
- => #<Beaneater::StatStruct id=9, tube="backburner.worker.queue.cat_dog", state="delayed", pri=65536, age=23, delay=604800, ttr=120, time_left=604776, file=0, reserves=0, timeouts=0, releases=0, buries=0, kicks=0>
219
- #
220
- # NOTE 'cat-dog' on line 1 and 'cat_dog' on line 3.
221
- #
408
+ # Job starts processing
409
+ job = ProcessImport.perform_later(file_id)
410
+
411
+ # Worker crashes or hangs after 200 seconds
412
+ # At 300 seconds (TTR), Beanstalkd automatically:
413
+ # 1. Releases job back to ready queue
414
+ # 2. Increments the job's reserve count
415
+ # 3. Makes job available for another worker
416
+
417
+ # Another worker picks up the job and retries
418
+ # (ActiveJob retry_on/discard_on handlers still apply)
222
419
  ```
223
420
 
224
421
  ## Installation
225
422
 
226
- First [install beanstalkd](https://beanstalkd.github.io/download.html). On Debian-based systems, that goes like this:
423
+ ### 1. Install Beanstalkd
424
+
425
+ First [install beanstalkd](https://beanstalkd.github.io/download.html):
426
+
227
427
  ```bash
228
428
  sudo apt-get install beanstalkd
429
+ brew install beanstalkd # for macOS/Linux
430
+
431
+ # Start beanstalkd (in-memory only)
432
+ beanstalkd -l 127.0.0.1 -p 11300
433
+
434
+ # OR with persistence (recommended for production)
435
+ mkdir -p /var/lib/beanstalkd
436
+ beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
229
437
  ```
230
438
 
231
- Then add to your Gemfile.
439
+ ### 2. Add Gem
440
+
232
441
  ```ruby
233
- gem 'postburner'
442
+ # Gemfile
443
+ gem 'postburner', '~> 1.0.0.pre.1'
234
444
  ```
235
445
 
236
- Install...
237
446
  ```bash
238
- $ bundle
447
+ bundle install
239
448
  ```
240
449
 
241
- After bundling, install the migration.
450
+ ### 3. Install Migration
451
+
452
+ ```bash
453
+ rails generate postburner:install
454
+ rails db:migrate
242
455
  ```
243
- # install migration, possible to edit and add attributes or indexes as needed.
244
- $ bundle exec rails g postburner:install
456
+
457
+ This creates the `postburner_jobs` table for tracked jobs.
458
+
459
+ ### 4. Configure ActiveJob
460
+
461
+ ```ruby
462
+ # config/application.rb
463
+ config.active_job.queue_adapter = :postburner
464
+ ```
465
+
466
+ ### 5. Create Worker Configuration
467
+
468
+ ```bash
469
+ cp config/postburner.yml.example config/postburner.yml
470
+ ```
471
+
472
+ Edit `config/postburner.yml` for your environment (see [Configuration](#configuration)).
473
+
474
+ ## Usage
475
+
476
+ ### Default Jobs
477
+
478
+ Default jobs execute quickly via Beanstalkd without PostgreSQL overhead. Perfect for emails, cache warming, notifications, etc.
479
+
480
+ ```ruby
481
+ class SendWelcomeEmail < ApplicationJob
482
+ queue_as :mailers
483
+
484
+ def perform(user_id)
485
+ UserMailer.welcome(user_id).deliver_now
486
+ end
487
+ end
488
+
489
+ # Enqueue immediately
490
+ SendWelcomeEmail.perform_later(123)
491
+
492
+ # Enqueue with delay
493
+ SendWelcomeEmail.set(wait: 1.hour).perform_later(123)
494
+
495
+ # Enqueue at specific time
496
+ SendWelcomeEmail.set(wait_until: Date.tomorrow.noon).perform_later(123)
245
497
  ```
246
498
 
247
- Inspect `XXX_create_postburner_jobs.rb`. (The template is [here](-/blob/master/lib/generators/postburner/install/templates/migrations/create_postburner_jobs.rb.erb)).The required attributes are in there, but add more if you would like to use them,
248
- or do it in a separate migration. By default, several indexes are added
499
+ **Overview:**
500
+ - Fast execution (no database writes)
501
+ - Low overhead
502
+ - Standard ActiveJob retry_on/discard_on support
503
+ - No audit trail
504
+ - No logging or statistics
505
+
506
+ #### Configuring Beanstalkd Priority and TTR
507
+
508
+ For default jobs that need custom Beanstalkd configuration, include `Postburner::Beanstalkd`:
249
509
 
250
- Because `Postburner` is built on `Backburner`, add a `config/initializers/backburner.rb` with option found [here](https://github.com/nesquena/backburner#configuration).
251
510
  ```ruby
252
- Backburner.configure do |config|
253
- config.beanstalk_url = "beanstalk://127.0.0.1"
254
- config.tube_namespace = "some.app.production"
255
- config.namespace_separator = "."
256
- config.on_error = lambda { |e| puts e }
257
- config.max_job_retries = 3 # default 0 retries
258
- config.retry_delay = 2 # default 5 seconds
259
- config.retry_delay_proc = lambda { |min_retry_delay, num_retries| min_retry_delay + (num_retries ** 3) }
260
- config.default_priority = 65536
261
- config.respond_timeout = 120
262
- config.default_worker = Backburner::Workers::Simple
263
- config.logger = Logger.new(STDOUT)
264
- config.primary_queue = "backburner-jobs"
265
- config.priority_labels = { :custom => 50, :useless => 1000 }
266
- config.reserve_timeout = nil
267
- config.job_serializer_proc = lambda { |body| JSON.dump(body) }
268
- config.job_parser_proc = lambda { |body| JSON.parse(body) }
511
+ class SendWelcomeEmail < ApplicationJob
512
+ include Postburner::Beanstalkd
513
+
514
+ queue_as :mailers
515
+ queue_priority 100 # Lower = higher priority (0 is highest)
516
+ queue_ttr 300 # Time-to-run in seconds (5 minutes)
517
+
518
+ def perform(user_id)
519
+ UserMailer.welcome(user_id).deliver_now
520
+ end
269
521
  end
270
522
  ```
271
523
 
272
- Finally, set `Backburner` for `ActiveJob`
524
+ **Configuration options:**
525
+ - `queue_priority` - Beanstalkd priority (0-4294967295, lower = higher priority)
526
+ - `queue_ttr` - Time-to-run in seconds before job times out
527
+
528
+ Jobs without `Postburner::Beanstalkd` use defaults from `config/postburner.yml`:
529
+ - `default_priority: 65536`
530
+ - `default_ttr: 300`
531
+
532
+ ### Tracked Jobs
533
+
534
+ Tracked jobs store full execution details in PostgreSQL, providing comprehensive audit trails (i.e. logging, timing, errors, retry tracking) for critical operations.
535
+
536
+ **Note:** `Postburner::Tracked` automatically includes `Postburner::Beanstalkd`, giving you access to `queue_priority`, `queue_ttr`, `bk`, and `extend!`.
537
+
538
+ ```ruby
539
+ class ProcessPayment < ApplicationJob
540
+ include Postburner::Tracked # ← Opt-in to PostgreSQL tracking (includes Beanstalkd)
541
+
542
+ queue_as :critical
543
+ queue_priority 0 # Highest priority (Beanstalkd included automatically)
544
+ queue_ttr 600 # 10 minute timeout
545
+ retry_on StandardError, wait: :exponentially_longer, attempts: 5
546
+
547
+ def perform(payment_id)
548
+ payment = Payment.find(payment_id)
549
+
550
+ log "Starting payment processing for $#{payment.amount}"
551
+
552
+ begin
553
+ payment.charge!
554
+ log! "Payment charged successfully"
555
+ rescue PaymentError => e
556
+ log_exception!(e)
557
+ raise
558
+ end
559
+
560
+ log "Payment complete", level: :info
561
+ end
562
+ end
273
563
  ```
274
- # config/application.rb
275
- config.active_job.queue_adapter = :backburner
564
+
565
+ **Tracking provides:**
566
+ - Complete execution history (queued_at, processing_at, processed_at)
567
+ - Custom logs with `log` and `log!`
568
+ - Exception tracking with `log_exception()` and `log_exception!()`
569
+ - Timing statistics (lag, duration)
570
+ - Retry attempts tracking
571
+ - Query with ActiveRecord
572
+ - Foreign key relationships
573
+ - Beanstalkd operations via `bk` accessor
574
+ - TTR extension via `extend!` method
575
+
576
+ **Query tracked jobs:**
577
+
578
+ ```ruby
579
+ # Find by state
580
+ Postburner::TrackedJob.where(processed_at: nil)
581
+ Postburner::TrackedJob.where.not(removed_at: nil)
582
+
583
+ # Find with errors
584
+ Postburner::TrackedJob.where("error_count > 0")
585
+
586
+ # Statistics
587
+ Postburner::TrackedJob.average(:duration) # Average execution time
588
+ Postburner::TrackedJob.maximum(:lag) # Worst lag
589
+
590
+ # Inspect execution
591
+ job = Postburner::TrackedJob.last
592
+ job.logs # Array of log entries with timestamps
593
+ job.errata # Array of exceptions with backtraces
594
+ job.attempts # Array of attempt timestamps
595
+ job.duration # Execution time in milliseconds
596
+ job.lag # Queue lag in milliseconds
276
597
  ```
277
598
 
278
- Postburner may later provide an adapter, but we recommend using `Postburner::Job` classes
279
- directly.
599
+ ### Direct Postburner::Job Usage
280
600
 
281
- Add jobs to `app/jobs/`. There currently is no generator.
601
+ Direct `Postburner::Job` subclasses are **always tracked**:
282
602
 
283
603
  ```ruby
284
- # app/jobs/run_donation.rb
285
604
  class RunDonation < Postburner::Job
286
605
  queue 'critical'
287
- queue_priority 0 # 0 is highest priority
288
- queue_max_job_retries 0 # don't retry
606
+ queue_priority 0
607
+ queue_max_job_retries 0
289
608
 
290
- def process(args)
291
- # do long tasks here
292
- # also have access to self.args
609
+ def perform(args)
610
+ donation = Donation.find(args['donation_id'])
611
+ donation.process!
612
+ log "Processed donation #{donation.id}"
293
613
  end
294
614
  end
615
+
616
+ # Create and enqueue
617
+ job = RunDonation.create!(args: { 'donation_id' => 123 })
618
+ job.queue!
619
+
620
+ # With delay
621
+ job.queue!(delay: 1.hour)
622
+
623
+ # At specific time
624
+ job.queue!(at: 2.days.from_now)
295
625
  ```
296
626
 
297
- ### Comparison to Backburner
627
+ #### Instance-Level Queue Configuration
298
628
 
299
- Compared to plain [Backburner](https://github.com/nesquena/backburner),
300
- Postburner adds:
301
- - Database Jobs for inspection, linking, auditing, removal (and deletion)
302
- - Direct access to associated [Beanstalk](https://beanstalkd.github.io/) (via [beaneater](https://github.com/beanstalkd/beaneater))
303
- - Job Statistics (lag, attempts, logs, tracked errors)
304
- - Convenience methods to clear tubes, stats, and connections for Beanstalk.
629
+ Override queue priority and TTR per job instance for dynamic behavior:
305
630
 
306
- Otherwise, Postburner tries to be a super simple layer on `Backburner::Queue`
307
- and `ActiveRecord`. Every tool with either of those are available in
308
- `Postburner::Job`s.
631
+ ```ruby
632
+ # Set priority during creation
633
+ job = RunDonation.create!(
634
+ args: { 'donation_id' => 123 },
635
+ queue_priority: 1500, # Override class-level priority
636
+ queue_ttr: 300 # Override class-level TTR
637
+ )
638
+ job.queue!
309
639
 
310
- Comes with a mountable interface that can be password protected with whatever
311
- authentication you use in your project.
640
+ # Set dynamically after creation
641
+ job = RunDonation.create!(args: { 'donation_id' => 456 })
642
+ job.queue_priority = 2000
643
+ job.queue_ttr = 300
644
+ job.queue!
312
645
 
313
- ### Comparison to Que
646
+ # Use in before_enqueue callback for conditional behavior
647
+ class ProcessOrder < Postburner::Job
648
+ queue 'orders'
649
+ queue_priority 100 # Default priority
650
+ queue_ttr 120 # Default TTR
314
651
 
315
- Postburner meant to be a replacement/upgrade for [Que](https://github.com/que-rb/que).
316
- However, if you need something faster and backed with ACID compliance, check out Que.
652
+ before_enqueue :set_priority_based_on_urgency
317
653
 
318
- Postburner has some additional features such as retained jobs after processing,
319
- stats, per job logging, etc.
654
+ def perform(args)
655
+ order = Order.find(args['order_id'])
656
+ order.process!
657
+ end
320
658
 
321
- Postburner is meant to be simpler than `Que`. Que is incredible, but jobs should
322
- be simple so the logic and history can be transparent.
659
+ private
323
660
 
324
- ## Contributing
325
- Submit a pull request. Follow conventions of the project. Be nice.
661
+ def set_priority_based_on_urgency
662
+ if args['express_shipping']
663
+ self.queue_priority = 0 # High priority for express orders
664
+ self.queue_ttr = 600 # Allow 10 minutes to complete
665
+ else
666
+ self.queue_priority = 1000 # Low priority for standard orders
667
+ self.queue_ttr = 120 # Standard 2 minute timeout
668
+ end
669
+ end
670
+ end
326
671
 
327
- ### V1 TODO
328
- - Basic tests
329
- - Add Authentication modules for engine mount.
672
+ # Express order gets high priority automatically
673
+ ProcessOrder.create!(args: { 'order_id' => 789, 'express_shipping' => true }).queue!
674
+ ```
675
+
676
+ **Overview:**
677
+ - Same as Tracked jobs
678
+ - Access the ActiveRecord Postburner::Job directly.
679
+
680
+ ## Workers
681
+
682
+ Postburner uses named worker configurations to support different deployment patterns. Each worker can have different fork/thread settings and process different queues, enabling flexible production deployments.
683
+
684
+ ### Named Workers Configuration
685
+
686
+ Configure multiple named workers with different concurrency profiles:
687
+
688
+ ```yaml
689
+ production:
690
+ beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
691
+
692
+ workers:
693
+ # Heavy, memory-intensive jobs - more processes, fewer threads
694
+ imports:
695
+ default_forks: 4
696
+ default_threads: 1
697
+ default_gc_limit: 500
698
+ queues:
699
+ - imports
700
+ - data_processing
701
+
702
+ # General jobs - fewer processes, many threads
703
+ general:
704
+ default_forks: 2
705
+ default_threads: 100
706
+ default_gc_limit: 5000
707
+ queues:
708
+ - default
709
+ - mailers
710
+ - notifications
711
+ ```
712
+
713
+ ### Puma-Style Architecture
714
+
715
+ The worker supports two modes based on fork configuration:
716
+ - **`forks: 0`** - Single process with thread pools (development/staging)
717
+ - **`forks: 1+`** - Multiple processes with thread pools (production, Puma-style)
718
+
719
+ This gives you a natural progression from simple single-threaded development to high-concurrency production.
720
+
721
+ ### Configuration Examples
722
+
723
+ **Development (single worker, single-threaded):**
724
+ ```yaml
725
+ development:
726
+ beanstalk_url: beanstalk://localhost:11300
727
+
728
+ workers:
729
+ default:
730
+ # Defaults: forks=0, threads=1, gc_limit=nil
731
+ queues:
732
+ - default
733
+ - mailers
734
+ ```
735
+
736
+ **Staging (single worker, multi-threaded):**
737
+ ```yaml
738
+ staging:
739
+ beanstalk_url: beanstalk://localhost:11300
740
+
741
+ workers:
742
+ default:
743
+ default_threads: 10
744
+ default_gc_limit: 5000
745
+ queues:
746
+ - critical
747
+ - default
748
+ - mailers
749
+ ```
750
+
751
+ **Production (multiple workers with different profiles):**
752
+ ```yaml
753
+ production:
754
+ beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
755
+
756
+ workers:
757
+ imports:
758
+ default_forks: 4 # 4 processes
759
+ default_threads: 1 # 1 thread per process = 4 concurrent jobs
760
+ default_gc_limit: 500
761
+ queues:
762
+ - imports
763
+ - data_processing
764
+
765
+ general:
766
+ default_forks: 2 # 2 processes
767
+ default_threads: 100 # 100 threads per process = 200 concurrent jobs
768
+ default_gc_limit: 5000
769
+ queues:
770
+ - default
771
+ - mailers
772
+ - notifications
773
+ ```
774
+
775
+ ### Running Workers
776
+
777
+ **Single worker (auto-selected):**
778
+ ```bash
779
+ bin/postburner
780
+ ```
781
+ If only one worker is defined, it's automatically selected.
782
+
783
+ **Multiple workers (must specify):**
784
+ ```bash
785
+ bin/postburner --worker imports # Run the 'imports' worker
786
+ bin/postburner --worker general # Run the 'general' worker
787
+ ```
788
+
789
+ **Filter queues (override config):**
790
+ ```bash
791
+ bin/postburner --worker general --queues default,mailers # Only process specific queues
792
+ ```
330
793
 
331
- ### V1+ TODO
332
- - Install generator
333
- - Sub to backburner
334
- - Job generator
335
- - Build file in app/jobs
336
- - Inherit from Postburner::Job
337
- - Add before/after/around hooks
338
- - Add destroy, and remove actions on show page
339
- - Clear tubes.
340
- - Document how/when to use activerecord hooks
341
- - Document how/when to use backburner hooks
342
- - Document how/when to use postburner hooks
343
- - Add logging with Job.args in backburner logs
344
- - MAYBE - ActiveJob adapter
794
+ ### Running Workers in Separate Processes
795
+
796
+ For production deployments, run different workers in separate OS processes for isolation and resource allocation:
797
+
798
+ **Docker Compose example:**
799
+ ```yaml
800
+ services:
801
+ # Process 1: Imports worker (forks=4, threads=1)
802
+ worker-imports:
803
+ build: .
804
+ command: bin/postburner --worker imports
805
+ environment:
806
+ RAILS_ENV: production
807
+ BEANSTALK_URL: beanstalk://beanstalkd:11300
808
+ depends_on:
809
+ - beanstalkd
810
+ - postgres
811
+
812
+ # Process 2: General worker (forks=2, threads=100)
813
+ worker-general:
814
+ build: .
815
+ command: bin/postburner --worker general
816
+ environment:
817
+ RAILS_ENV: production
818
+ BEANSTALK_URL: beanstalk://beanstalkd:11300
819
+ depends_on:
820
+ - beanstalkd
821
+ - postgres
822
+ ```
823
+
824
+ **Procfile example (Heroku/Foreman):**
825
+ ```
826
+ worker_imports: bin/postburner --worker imports
827
+ worker_general: bin/postburner --worker general
828
+ ```
829
+
830
+ ### Worker Architecture
831
+
832
+ **Single Process Mode (forks: 0):**
833
+ ```
834
+ Main Process
835
+ ├─ Queue 'critical' Thread Pool (1 thread)
836
+ ├─ Queue 'default' Thread Pool (10 threads)
837
+ └─ Queue 'mailers' Thread Pool (5 threads)
838
+ ```
345
839
 
840
+ **Multi-Process Mode (forks: 1+):**
841
+ ```
842
+ Parent Process
843
+ ├─ Fork 0 (queue: critical)
844
+ │ └─ Thread 1
845
+ ├─ Fork 0 (queue: default)
846
+ │ ├─ Thread 1-10
847
+ ├─ Fork 1 (queue: default) # Puma-style: multiple forks of same queue
848
+ │ ├─ Thread 1-10
849
+ ├─ Fork 2 (queue: default)
850
+ │ └─ Thread 1-10
851
+ ├─ Fork 3 (queue: default)
852
+ │ └─ Thread 1-10
853
+ │ └─ Total: 40 concurrent jobs for 'default' (4 forks × 10 threads)
854
+ └─ Fork 0 (queue: mailers)
855
+ └─ Thread 1-5
856
+ ```
346
857
 
347
- ### Running in Development
858
+ ### Benefits
859
+
860
+ **Single Process Mode (forks: 0):**
861
+ - Simplest deployment
862
+ - Easy debugging
863
+ - Lower memory footprint
864
+ - Best for development and moderate load
865
+
866
+ **Multi-Process Mode (forks: 1+):**
867
+ - Horizontal scaling per queue
868
+ - Process isolation
869
+ - Automatic memory cleanup via GC limits
870
+ - Crash isolation (one fork down doesn't affect others)
871
+ - Best for production high-volume workloads
872
+
873
+ ### GC Limits
874
+
875
+ Set `default_gc_limit` per worker to automatically restart after processing N jobs.
876
+
877
+ - Worker processes N jobs
878
+ - Worker exits with code 99
879
+ - In single-process mode (forks: 0): process manager restarts the entire process
880
+ - In multi-process mode (forks: 1+): parent process automatically restarts just that fork
881
+
882
+ ```yaml
883
+ production:
884
+ workers:
885
+ imports:
886
+ default_forks: 4
887
+ default_threads: 1
888
+ default_gc_limit: 500 # Restart after 500 jobs (memory-intensive)
889
+ queues:
890
+ - imports
891
+ - data_processing
892
+
893
+ general:
894
+ default_forks: 2
895
+ default_threads: 100
896
+ default_gc_limit: 5000 # Restart after 5000 jobs
897
+ queues:
898
+ - default
899
+ - mailers
900
+ ```
348
901
 
902
+ **Why?**
903
+ - Prevents memory bloat in long-running workers
904
+ - Automatic cleanup without manual intervention
905
+ - Works seamlessly with Docker and Kubernetes restarts
906
+
907
+ ## Configuration
908
+
909
+ ### YAML Configuration
910
+
911
+ ```yaml
912
+ # config/postburner.yml
913
+ default: &default
914
+ beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
915
+
916
+ development:
917
+ <<: *default
918
+ # Defaults: forks: 0, threads: 1, gc_limit: nil
919
+ queues:
920
+ default: {}
921
+ mailers: {}
922
+
923
+ test:
924
+ <<: *default
925
+ # Defaults: forks: 0, threads: 1, gc_limit: nil
926
+ queues:
927
+ default: {}
928
+
929
+ staging:
930
+ <<: *default
931
+ default_threads: 10 # Multi-threaded, single process
932
+ default_gc_limit: 5000
933
+ queues:
934
+ default: {}
935
+ mailers: {}
936
+
937
+ production:
938
+ <<: *default
939
+ default_forks: 4 # Puma-style: multiple processes
940
+ default_threads: 10 # 10 threads per fork
941
+ default_gc_limit: 5000
942
+
943
+ queues:
944
+ critical:
945
+ forks: 1
946
+ threads: 1 # 1 concurrent job
947
+ gc_limit: 100
948
+
949
+ default:
950
+ forks: 4
951
+ threads: 10 # 40 total concurrent jobs (4 × 10)
952
+ gc_limit: 1000
953
+
954
+ mailers:
955
+ forks: 2
956
+ threads: 5 # 10 total email senders (2 × 5)
957
+ gc_limit: 500
349
958
  ```
350
- cd test/dummy
351
- bundle exec backburner
959
+
960
+ ### Queue Configuration Methods
961
+
962
+ For `Postburner::Job` subclasses (and backwards compatibility):
963
+
964
+ ```ruby
965
+ class CriticalJob < Postburner::Job
966
+ queue 'critical' # Beanstalkd tube name
967
+ queue_priority 0 # Lower = higher priority (0 is highest)
968
+ queue_ttr 300 # TTR (time-to-run) in seconds
969
+ queue_max_job_retries 3 # Max retry attempts
970
+ queue_retry_delay 5 # Fixed delay: 5 seconds between retries
971
+
972
+ def perform(args)
973
+ # ...
974
+ end
975
+ end
352
976
  ```
353
977
 
354
- ### Releasing
978
+ **Exponential backoff with proc:**
355
979
 
356
- 1. `gem bump -v minor -t`
357
- Where <minor> can be: major|minor|patch|pre|release or a version number
358
- 2. Edit `CHANGELOG.md` with details from `git log --oneline`
359
- 3. `git commit --amend`
360
- 4. `gem release -k nac -p`
361
- Where <nac> is an authorized key with push capabilities.
980
+ ```ruby
981
+ class BackgroundTask < Postburner::Job
982
+ queue 'default'
983
+ queue_max_job_retries 5
984
+ queue_retry_delay ->(retries) { 2 ** retries } # 2s, 4s, 8s, 16s, 32s
362
985
 
986
+ def perform(args)
987
+ # ...
988
+ end
989
+ end
990
+ ```
991
+
992
+ ### ActiveJob Configuration
993
+
994
+ For ActiveJob classes, use standard ActiveJob configuration:
995
+
996
+ ```ruby
997
+ class MyJob < ApplicationJob
998
+ include Postburner::Tracked # Optional: for audit trail
999
+
1000
+ queue_as :default
1001
+
1002
+ retry_on StandardError, wait: :exponentially_longer, attempts: 5
1003
+ retry_on CustomError, wait: 10.seconds
1004
+
1005
+ discard_on ActiveJob::DeserializationError
1006
+
1007
+ def perform(user_id)
1008
+ # ...
1009
+ end
1010
+ end
1011
+ ```
1012
+
1013
+ ## Callbacks
1014
+
1015
+ Postburner provides lifecycle callbacks:
1016
+
1017
+ ```ruby
1018
+ class ProcessPayment < ApplicationJob
1019
+ include Postburner::Tracked
1020
+
1021
+ before_enqueue :validate_payment
1022
+ after_enqueue :send_confirmation
1023
+
1024
+ before_attempt :log_attempt # Runs on every retry
1025
+ after_attempt :update_metrics
1026
+
1027
+ before_processing :acquire_lock
1028
+ after_processing :release_lock
1029
+
1030
+ after_processed :send_receipt # Only on success
1031
+
1032
+ def perform(payment_id)
1033
+ # ...
1034
+ end
1035
+
1036
+ private
1037
+
1038
+ def validate_payment
1039
+ raise "Invalid" unless payment_valid?
1040
+ end
1041
+
1042
+ def send_receipt
1043
+ PaymentMailer.receipt(arguments.first).deliver_later
1044
+ end
1045
+ end
1046
+ ```
1047
+
1048
+ **Available callbacks:**
1049
+ - `before_enqueue`, `around_enqueue`, `after_enqueue` - When queued
1050
+ - `before_attempt`, `around_attempt`, `after_attempt` - Every execution (including retries)
1051
+ - `before_processing`, `around_processing`, `after_processing` - During execution
1052
+ - `after_processed` - Only after successful completion
1053
+
1054
+ ## Queue Strategies
1055
+
1056
+ Postburner uses different strategies to control job execution. These affect `Postburner::Job` subclasses (not ActiveJob classes).
1057
+
1058
+ | Strategy | When to Use | Behavior | Requires Beanstalkd |
1059
+ |----------|-------------|----------|---------------------|
1060
+ | **NiceQueue** (default) | Production | Async via Beanstalkd, gracefully re-queues premature jobs | Yes |
1061
+ | **Queue** | Production (strict) | Async via Beanstalkd, raises error on premature execution | Yes |
1062
+ | **TestQueue** | Testing with explicit time control | Inline execution, raises error for scheduled jobs | No |
1063
+ | **ImmediateTestQueue** | Testing with automatic time travel | Inline execution, auto time-travels for scheduled jobs | No |
1064
+ | **NullQueue** | Batch processing / deferred execution | Jobs created but not queued, manual execution | No |
1065
+
1066
+ ```ruby
1067
+ # Switch strategies
1068
+ Postburner.nice_async_strategy! # Default production (NiceQueue)
1069
+ Postburner.async_strategy! # Strict production (Queue)
1070
+ Postburner.inline_test_strategy! # Testing (TestQueue)
1071
+ Postburner.inline_immediate_test_strategy! # Testing with time travel
1072
+ Postburner.null_strategy! # Deferred execution
1073
+ ```
1074
+
1075
+ **Note:** These strategies only affect `Postburner::Job` subclasses. ActiveJob classes execute according to the ActiveJob adapter configuration.
1076
+
1077
+ ## Testing
1078
+
1079
+ Postburner provides test-friendly execution modes that don't require Beanstalkd.
1080
+
1081
+ ### Automatic Test Mode
1082
+
1083
+ In Rails test environments, Postburner automatically uses inline execution:
1084
+
1085
+ ```ruby
1086
+ # test/test_helper.rb - automatic!
1087
+ Postburner.testing? # => true in tests
1088
+ ```
1089
+
1090
+ ### Testing Default Jobs (ActiveJob)
1091
+
1092
+ Use standard ActiveJob test helpers:
1093
+
1094
+ ```ruby
1095
+ class MyTest < ActiveSupport::TestCase
1096
+ test "job executes" do
1097
+ SendEmail.perform_later(123)
1098
+ # Job executes immediately in test mode
1099
+ end
1100
+
1101
+ test "job with delay" do
1102
+ travel 1.hour do
1103
+ SendEmail.set(wait: 1.hour).perform_later(123)
1104
+ end
1105
+ end
1106
+ end
1107
+ ```
1108
+
1109
+ ### Testing Tracked Jobs
1110
+
1111
+ Tracked jobs create database records you can inspect:
1112
+
1113
+ ```ruby
1114
+ test "tracked job logs execution" do
1115
+ ProcessPayment.perform_later(456)
1116
+
1117
+ job = Postburner::TrackedJob.last
1118
+ assert job.processed_at.present?
1119
+ assert_includes job.logs.map { |l| l[1]['message'] }, "Processing payment 456"
1120
+ end
1121
+ ```
1122
+
1123
+ ### Testing Legacy Postburner::Job
1124
+
1125
+ ```ruby
1126
+ test "processes immediately" do
1127
+ job = RunDonation.create!(args: { 'donation_id' => 123 })
1128
+ job.queue!
1129
+
1130
+ assert job.reload.processed_at
1131
+ end
1132
+
1133
+ test "scheduled job with time travel" do
1134
+ job = RunDonation.create!(args: { 'donation_id' => 123 })
1135
+
1136
+ travel_to(2.hours.from_now) do
1137
+ job.queue!(delay: 2.hours)
1138
+ assert job.reload.processed_at
1139
+ end
1140
+ end
1141
+ ```
1142
+
1143
+ ## Job Management
1144
+
1145
+ ### Canceling Jobs
1146
+
1147
+ **For Postburner::Job subclasses:**
1148
+
1149
+ ```ruby
1150
+ job = RunDonation.create!(args: { 'donation_id' => 123 })
1151
+ job.queue!(delay: 1.hour)
1152
+
1153
+ # Soft delete (recommended) - removes from queue, sets removed_at
1154
+ job.remove!
1155
+ job.removed_at # => 2025-11-19 12:34:56 UTC
1156
+ job.persisted? # => true (still in database for audit)
1157
+
1158
+ # Hard delete - removes from both queue and database
1159
+ job.destroy!
1160
+
1161
+ # Delete from Beanstalkd only (keeps database record)
1162
+ job.delete!
1163
+ ```
1164
+
1165
+ **For tracked ActiveJob classes:**
1166
+
1167
+ ```ruby
1168
+ # Find the TrackedJob record
1169
+ job = Postburner::TrackedJob.find(123)
1170
+ job.remove! # Cancel execution
1171
+ ```
1172
+
1173
+ ### Retrying Buried Jobs
1174
+
1175
+ Jobs that fail repeatedly get "buried" in Beanstalkd:
1176
+
1177
+ ```ruby
1178
+ job = Postburner::Job.find(123)
1179
+ job.kick! # Moves buried job back to ready queue
1180
+ ```
1181
+
1182
+ ### Inspecting Job State
1183
+
1184
+ ```ruby
1185
+ job = Postburner::TrackedJob.find(123)
1186
+
1187
+ # Timestamps
1188
+ job.queued_at # When queued to Beanstalkd
1189
+ job.run_at # Scheduled execution time
1190
+ job.attempting_at # First attempt started
1191
+ job.processing_at # Current/last processing started
1192
+ job.processed_at # Completed successfully
1193
+ job.removed_at # Soft deleted
1194
+
1195
+ # Statistics
1196
+ job.lag # Milliseconds between scheduled and actual execution
1197
+ job.duration # Milliseconds to execute
1198
+ job.attempt_count # Number of attempts
1199
+ job.error_count # Number of errors
1200
+
1201
+ # Audit trail
1202
+ job.logs # Array of log entries with timestamps
1203
+ job.errata # Array of exceptions with backtraces
1204
+ job.attempts # Array of attempt timestamps
1205
+ ```
1206
+
1207
+ ## Beanstalkd Integration
1208
+
1209
+ Direct access to Beanstalkd for advanced operations:
1210
+
1211
+ ```ruby
1212
+ # Get Beanstalkd job ID
1213
+ job.bkid # => 12345
1214
+
1215
+ # Access Beaneater job object
1216
+ job.beanstalk_job.stats
1217
+ # => {"id"=>12345, "tube"=>"critical", "state"=>"ready", ...}
1218
+
1219
+ # Connection management
1220
+ Postburner.connected do |conn|
1221
+ conn.tubes.to_a # List all tubes
1222
+ conn.tubes['postburner.production.critical'].stats
1223
+ conn.tubes['postburner.production.critical'].kick(10) # Kick 10 buried jobs
1224
+ end
1225
+ ```
1226
+
1227
+
1228
+ ## Web UI
1229
+
1230
+ Mount the inspection interface:
1231
+
1232
+ ```ruby
1233
+ # config/routes.rb
1234
+ mount Postburner::Engine => "/postburner"
1235
+
1236
+ # Only in development
1237
+ mount Postburner::Engine => "/postburner" if Rails.env.development?
1238
+ ```
1239
+
1240
+ Add your own authentication:
1241
+
1242
+ ```ruby
1243
+ # config/routes.rb
1244
+ authenticate :user, ->(user) { user.admin? } do
1245
+ mount Postburner::Engine => "/postburner"
1246
+ end
1247
+ ```
1248
+
1249
+ ## Deployment
1250
+
1251
+ ### Docker
1252
+
1253
+ ```dockerfile
1254
+ # Dockerfile.worker
1255
+ FROM ruby:3.3
1256
+
1257
+ WORKDIR /app
1258
+ COPY . .
1259
+ RUN bundle install
1260
+
1261
+ CMD ["bundle", "exec", "postburner", "--config", "config/postburner.yml", "--env", "production"]
1262
+ ```
1263
+
1264
+ ```yaml
1265
+ # docker-compose.yml
1266
+ services:
1267
+ beanstalkd:
1268
+ image: schickling/beanstalkd
1269
+ command: ["-l", "0.0.0.0", "-p", "11300", "-b", "/var/lib/beanstalkd"]
1270
+ ports:
1271
+ - "11300:11300"
1272
+ volumes:
1273
+ - beanstalkd_data:/var/lib/beanstalkd
1274
+
1275
+ worker:
1276
+ build:
1277
+ context: .
1278
+ dockerfile: Dockerfile.worker
1279
+ depends_on:
1280
+ - beanstalkd
1281
+ - postgres
1282
+ environment:
1283
+ BEANSTALK_URL: beanstalk://beanstalkd:11300
1284
+ DATABASE_URL: postgres://postgres@postgres/myapp_production
1285
+
1286
+ volumes:
1287
+ beanstalkd_data:
1288
+ ```
1289
+
1290
+ ## Migration from v0.x
1291
+
1292
+ Key changes in v1.0:
1293
+
1294
+ ### Removed
1295
+ - Backburner dependency
1296
+ - `config/initializers/backburner.rb`
1297
+ - Backburner worker (`bundle exec backburner`)
1298
+
1299
+ ### Added
1300
+ - ActiveJob adapter (first-class Rails integration)
1301
+ - Default jobs (Beanstalkd only, no PostgreSQL)
1302
+ - `bin/postburner` executable
1303
+ - `config/postburner.yml` configuration
1304
+ - Multiple worker types (Simple, Forking, ThreadsOnFork)
1305
+
1306
+ ### Migration Steps
1307
+
1308
+ 1. **Update Gemfile:**
1309
+ ```ruby
1310
+ gem 'postburner', '~> 1.0.0.pre.1'
1311
+ ```
1312
+
1313
+ 2. **Remove Backburner config:**
1314
+ ```bash
1315
+ rm config/initializers/backburner.rb
1316
+ ```
1317
+
1318
+ 3. **Create Postburner config:**
1319
+ ```bash
1320
+ cp config/postburner.yml.example config/postburner.yml
1321
+ ```
1322
+
1323
+ 4. **Update ActiveJob adapter:**
1324
+ ```ruby
1325
+ # config/application.rb
1326
+ config.active_job.queue_adapter = :postburner # was :backburner
1327
+ ```
1328
+
1329
+ 5. **Update worker command:**
1330
+ ```bash
1331
+ # Old
1332
+ bundle exec backburner
1333
+
1334
+ # New
1335
+ bundle exec postburner --config config/postburner.yml
1336
+ ```
1337
+
1338
+ 6. **Existing Jobs:** Direct `Postburner::Job` subclasses continue to work without changes!
1339
+
1340
+ ## Contributing
1341
+
1342
+ Submit a pull request. Follow project conventions. Be nice.
363
1343
 
364
1344
  ## License
1345
+
365
1346
  The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).