postburner 1.0.0.pre.2 → 1.0.0.pre.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +248 -167
- data/app/concerns/postburner/commands.rb +143 -0
- data/app/concerns/postburner/execution.rb +190 -0
- data/app/concerns/postburner/insertion.rb +170 -0
- data/app/concerns/postburner/logging.rb +181 -0
- data/{lib/postburner/queue_config.rb → app/concerns/postburner/properties.rb} +90 -26
- data/app/concerns/postburner/statistics.rb +125 -0
- data/app/models/postburner/job.rb +15 -757
- data/app/models/postburner/mailer.rb +2 -2
- data/app/views/postburner/jobs/show.html.haml +2 -2
- data/config/postburner.yml.example +32 -30
- data/lib/generators/postburner/install/templates/config/postburner.yml +32 -30
- data/lib/postburner/active_job/adapter.rb +9 -9
- data/lib/postburner/beanstalkd.rb +18 -39
- data/lib/postburner/configuration.rb +41 -14
- data/lib/postburner/strategies/queue.rb +19 -9
- data/lib/postburner/version.rb +1 -1
- data/lib/postburner/workers/worker.rb +27 -28
- data/lib/postburner.rb +0 -1
- metadata +7 -2
data/README.md
CHANGED
|
@@ -1,86 +1,36 @@
|
|
|
1
1
|
# Postburner
|
|
2
2
|
|
|
3
|
-
Fast Beanstalkd-backed job queue with **optional PostgreSQL
|
|
3
|
+
Fast Beanstalkd-backed job queue with **optional PostgreSQL records** for ActiveJob.
|
|
4
4
|
|
|
5
5
|
Postburner provides dual-mode job execution:
|
|
6
|
-
- **
|
|
7
|
-
- **Tracked jobs**:
|
|
6
|
+
- **Fast jobs**: Fast execution via Beanstalkd
|
|
7
|
+
- **Tracked jobs**: Audited jobs logs, timing, errors, and statistics
|
|
8
8
|
|
|
9
9
|
Built for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
|
|
10
10
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
- **ActiveJob native** - Works seamlessly with Rails, ActionMailer, ActiveStorage
|
|
14
|
-
- **Dual-mode execution** - Default or tracked (database backed)
|
|
15
|
-
- **Rich audit trail** - Logs, timing, errors, retry tracking (tracked jobs only)
|
|
16
|
-
- **ActiveRecord** - Query jobs with ActiveRecord (Tracked jobs only)
|
|
17
|
-
- **Beanstalkd** - Fast, reliable queue separate from your database, peristent storage
|
|
18
|
-
- **Process isolation** - Forking workers with optional threading for throughput
|
|
19
|
-
- **Test-friendly** - Inline execution without Beanstalkd in tests
|
|
20
|
-
|
|
21
|
-
## Quick Start
|
|
22
|
-
|
|
23
|
-
**Installation:**
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
sudo apt-get install beanstalkd # OR brew install beanstalkd
|
|
27
|
-
|
|
28
|
-
# Start beanstalkd (in-memory only)
|
|
29
|
-
beanstalkd -l 127.0.0.1 -p 11300
|
|
30
|
-
|
|
31
|
-
# OR with persistence (recommended for production)
|
|
32
|
-
mkdir -p /var/lib/beanstalkd
|
|
33
|
-
beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
```ruby
|
|
37
|
-
# Gemfile
|
|
38
|
-
gem 'postburner', '~> 1.0.0.pre.1'
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
```bash
|
|
42
|
-
bundle install
|
|
43
|
-
rails generate postburner:install
|
|
44
|
-
rails db:migrate
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
**Configuration:**
|
|
48
|
-
|
|
49
|
-
```ruby
|
|
50
|
-
# config/application.rb
|
|
51
|
-
config.active_job.queue_adapter = :postburner
|
|
52
|
-
```
|
|
53
|
-
|
|
54
|
-
```yaml
|
|
55
|
-
# config/postburner.yml
|
|
56
|
-
production:
|
|
57
|
-
beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
|
|
58
|
-
worker_type: simple
|
|
59
|
-
queues:
|
|
60
|
-
default: {}
|
|
61
|
-
critical: {}
|
|
62
|
-
mailers: {}
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
**Usage:**
|
|
11
|
+
Depends on Beanstalkd, ActiveRecord, ActiveJob, Postgres.
|
|
66
12
|
|
|
67
13
|
```ruby
|
|
68
14
|
# Default job (fast, no PostgreSQL overhead)
|
|
69
|
-
class
|
|
15
|
+
class SendSms < ApplicationJob
|
|
70
16
|
def perform(user_id)
|
|
71
|
-
|
|
17
|
+
user = User.find(user_id)
|
|
18
|
+
TextMessage.welcome(
|
|
19
|
+
to: user.phone_number,
|
|
20
|
+
body: "Welcome to our app!"
|
|
21
|
+
).deliver_now
|
|
72
22
|
end
|
|
73
23
|
end
|
|
74
24
|
|
|
75
25
|
# Default job with Beanstalkd configuration
|
|
76
|
-
class
|
|
77
|
-
include Postburner::Beanstalkd
|
|
26
|
+
class DoSomething < ApplicationJob
|
|
27
|
+
include Postburner::Beanstalkd # optional, allow access to beanstalkd
|
|
78
28
|
|
|
79
|
-
|
|
80
|
-
|
|
29
|
+
priority 5000 # 0=highest, 65536=default, can set per job
|
|
30
|
+
ttr 30 # 30 second timeout
|
|
81
31
|
|
|
82
32
|
def perform(user_id)
|
|
83
|
-
|
|
33
|
+
# Do something
|
|
84
34
|
end
|
|
85
35
|
end
|
|
86
36
|
|
|
@@ -88,8 +38,8 @@ end
|
|
|
88
38
|
class ProcessPayment < ApplicationJob
|
|
89
39
|
include Postburner::Tracked # ← Opt-in to tracking (includes Beanstalkd)
|
|
90
40
|
|
|
91
|
-
|
|
92
|
-
|
|
41
|
+
priority 0 # Highest priority
|
|
42
|
+
ttr 600 # 10 minute timeout
|
|
93
43
|
|
|
94
44
|
def perform(payment_id)
|
|
95
45
|
log "Processing payment #{payment_id}"
|
|
@@ -99,7 +49,82 @@ class ProcessPayment < ApplicationJob
|
|
|
99
49
|
end
|
|
100
50
|
|
|
101
51
|
# Run worker
|
|
102
|
-
|
|
52
|
+
bundle exec postburner --worker default
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## Why
|
|
56
|
+
|
|
57
|
+
Postburner supports Async, Queues, Delayed, Priorities, Timeouts, and Retries from the [Backend Matrix](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters.html). But uniquely, priorities are per job, in addition to the class level. Timeouts are per job and class level as well, and can be extended dynamically.
|
|
58
|
+
|
|
59
|
+
Postburner is inspired by the [Backburner](https://github.com/nesquena/backburner) gem i.e. "burner", Postgres i.e. "Post", and the database backends (SolidQueue, Que, etc), and the in memory/redis backends (Sidekiq, Resque, etc). And puma's concurrency model.
|
|
60
|
+
|
|
61
|
+
Thus old-school [beanstalkd](https://beanstalkd.github.io/) is used with PostgreSQL to cover all the cases we care about.
|
|
62
|
+
- Fast when you want it (light, ephemeral jobs like delayed turbo_stream rendering, communications)
|
|
63
|
+
- Tracked when you need it (critical operations like payments, and processes.
|
|
64
|
+
- Able to record jobs before and after execution in PostgreSQL, with foreign keys and constraints.
|
|
65
|
+
- Store the jobs outside of the database, but also persist them to disk for disaster recovery (beanstalkd binlogs)
|
|
66
|
+
- Introspect the jobs either with ActiveRecord or Beanstalkd.
|
|
67
|
+
- Only one worker type, that can be single/multi-process, with optional threading, and optional GC (Garbage Collection) limits (kill fork after processing N jobs).
|
|
68
|
+
- Easy testing.
|
|
69
|
+
|
|
70
|
+
## Features
|
|
71
|
+
|
|
72
|
+
- **ActiveJob native** - Works seamlessly with Rails, ActionMailer, ActiveStorage
|
|
73
|
+
- **Dual-mode execution** - Default or tracked (database backed)
|
|
74
|
+
- **Rich audit trail** - Logs, timing, errors, retry tracking (tracked jobs only)
|
|
75
|
+
- **ActiveRecord** - Query jobs with ActiveRecord (Tracked jobs only)
|
|
76
|
+
- **Beanstalkd** - Fast, reliable queue separate from your database, peristent storage
|
|
77
|
+
- **Process isolation** - Forking workers with optional threading for throughput
|
|
78
|
+
- **Test-friendly** - Inline execution without Beanstalkd in tests
|
|
79
|
+
|
|
80
|
+
## Quick Start
|
|
81
|
+
|
|
82
|
+
```ruby
|
|
83
|
+
# Gemfile
|
|
84
|
+
gem 'postburner', '~> 1.0.0.pre.3'
|
|
85
|
+
|
|
86
|
+
# config/application.rb
|
|
87
|
+
config.active_job.queue_adapter = :postburner
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
```yaml
|
|
91
|
+
# config/postburner.yml
|
|
92
|
+
development: # <- environment config, i.e. defaults
|
|
93
|
+
beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
|
|
94
|
+
default_forks: 2
|
|
95
|
+
default_threads: 10
|
|
96
|
+
default_gc_limit: 500
|
|
97
|
+
|
|
98
|
+
workers: # <- worker config, i.e. overrides
|
|
99
|
+
default:
|
|
100
|
+
threads: 16 # Overrides default_threads
|
|
101
|
+
queues:
|
|
102
|
+
- default
|
|
103
|
+
- mailers
|
|
104
|
+
critical:
|
|
105
|
+
forks: 4 # Overrides default_forks
|
|
106
|
+
threads: 8 # Overrides default_threads
|
|
107
|
+
gc_limit: 24 # Overrides default_gc_limit
|
|
108
|
+
queues:
|
|
109
|
+
- payments
|
|
110
|
+
slow:
|
|
111
|
+
forks: 4 # Overrides default_forks
|
|
112
|
+
threads: 1 # Overrides default_threads
|
|
113
|
+
gc_limit: 1 # Overrides default_gc_limit
|
|
114
|
+
queues:
|
|
115
|
+
- imports
|
|
116
|
+
- video
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
sudo apt-get install beanstalkd # OR brew install beanstalkd
|
|
121
|
+
|
|
122
|
+
beanstalkd -l 127.0.0.1 -p 11300 # Start beanstalkd
|
|
123
|
+
|
|
124
|
+
bundle exec rails generate postburner:install
|
|
125
|
+
bundle exec rails db:migrate
|
|
126
|
+
|
|
127
|
+
bundle exec postburner # start
|
|
103
128
|
```
|
|
104
129
|
|
|
105
130
|
## Table of Contents
|
|
@@ -122,14 +147,42 @@ bin/postburner --config config/postburner.yml --env production
|
|
|
122
147
|
|
|
123
148
|
Beanstalkd is a simple, fast, and reliable queue system. It is a good choice for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
|
|
124
149
|
|
|
125
|
-
The [protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt) reads more like a README than a protocol. Check it out and you will instantly understand how it works.
|
|
150
|
+
The [protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt) reads more like a README than a protocol. Check it out and you will instantly understand how it works. Here is a help
|
|
151
|
+
|
|
152
|
+
Here is a picture of the typical job lifecycle:
|
|
153
|
+
|
|
154
|
+
```
|
|
155
|
+
put reserve delete
|
|
156
|
+
-----> [READY] ---------> [RESERVED] --------> *poof*`
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
Here is a picture with more possibilities:
|
|
160
|
+
|
|
161
|
+
```
|
|
162
|
+
put with delay release with delay
|
|
163
|
+
----------------> [DELAYED] <------------.
|
|
164
|
+
| |
|
|
165
|
+
| (time passes) |
|
|
166
|
+
| |
|
|
167
|
+
put v reserve | delete
|
|
168
|
+
-----------------> [READY] ---------> [RESERVED] --------> *poof*
|
|
169
|
+
^ ^ | |
|
|
170
|
+
| \ release | |
|
|
171
|
+
| `-------------' |
|
|
172
|
+
| |
|
|
173
|
+
| kick |
|
|
174
|
+
| |
|
|
175
|
+
| bury |
|
|
176
|
+
[BURIED] <---------------'
|
|
177
|
+
|
|
|
178
|
+
| delete
|
|
179
|
+
`--------> *poof*
|
|
180
|
+
```
|
|
126
181
|
|
|
127
182
|
### Binlogs
|
|
128
183
|
|
|
129
184
|
Beanstalkd lets you persist jobs to disk in case of a crash or restart. Just restart beanstalkd and the jobs will be back in the queue.
|
|
130
185
|
|
|
131
|
-
**Setup:**
|
|
132
|
-
|
|
133
186
|
```bash
|
|
134
187
|
# Create binlog directory
|
|
135
188
|
sudo mkdir -p /var/lib/beanstalkd
|
|
@@ -139,7 +192,7 @@ sudo chown beanstalkd:beanstalkd /var/lib/beanstalkd # If running as service
|
|
|
139
192
|
beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
|
|
140
193
|
```
|
|
141
194
|
|
|
142
|
-
**
|
|
195
|
+
**Other options:**
|
|
143
196
|
|
|
144
197
|
```bash
|
|
145
198
|
# Basic persistence
|
|
@@ -155,7 +208,7 @@ beanstalkd -b /var/lib/beanstalkd -f 200 # fsync at most every 200ms (default:
|
|
|
155
208
|
beanstalkd -b /var/lib/beanstalkd -F
|
|
156
209
|
```
|
|
157
210
|
|
|
158
|
-
**
|
|
211
|
+
**Beanstalkd switches:**
|
|
159
212
|
- `-b <dir>` - Enable binlog persistence in specified directory
|
|
160
213
|
- `-s <bytes>` - Maximum size of each binlog file (requires `-b`)
|
|
161
214
|
- `-f <ms>` - Call fsync at most once every N milliseconds (requires `-b`, default: 50ms)
|
|
@@ -181,14 +234,14 @@ class ProcessPayment < ApplicationJob
|
|
|
181
234
|
include Postburner::Beanstalkd
|
|
182
235
|
|
|
183
236
|
queue_as :critical
|
|
184
|
-
|
|
237
|
+
priority 0 # Highest priority - processed immediately
|
|
185
238
|
end
|
|
186
239
|
|
|
187
240
|
class SendEmail < ApplicationJob
|
|
188
241
|
include Postburner::Beanstalkd
|
|
189
242
|
|
|
190
243
|
queue_as :mailers
|
|
191
|
-
|
|
244
|
+
priority 1000 # Lower priority - processed after critical jobs
|
|
192
245
|
end
|
|
193
246
|
```
|
|
194
247
|
|
|
@@ -197,7 +250,7 @@ end
|
|
|
197
250
|
```ruby
|
|
198
251
|
class CriticalJob < Postburner::Job
|
|
199
252
|
queue 'critical'
|
|
200
|
-
|
|
253
|
+
priority 0 # Highest priority
|
|
201
254
|
|
|
202
255
|
def perform(args)
|
|
203
256
|
# Critical business logic
|
|
@@ -206,7 +259,7 @@ end
|
|
|
206
259
|
|
|
207
260
|
class BackgroundTask < Postburner::Job
|
|
208
261
|
queue 'default'
|
|
209
|
-
|
|
262
|
+
priority 5000 # Lower priority
|
|
210
263
|
|
|
211
264
|
def perform(args)
|
|
212
265
|
# Non-urgent background work
|
|
@@ -239,7 +292,7 @@ production:
|
|
|
239
292
|
```ruby
|
|
240
293
|
class ProcessOrder < Postburner::Job
|
|
241
294
|
queue 'orders'
|
|
242
|
-
|
|
295
|
+
priority 100 # Default
|
|
243
296
|
|
|
244
297
|
before_enqueue :adjust_priority
|
|
245
298
|
|
|
@@ -252,7 +305,7 @@ class ProcessOrder < Postburner::Job
|
|
|
252
305
|
|
|
253
306
|
def adjust_priority
|
|
254
307
|
if args['urgent']
|
|
255
|
-
self.
|
|
308
|
+
self.priority = 0 # Override to highest priority
|
|
256
309
|
end
|
|
257
310
|
end
|
|
258
311
|
end
|
|
@@ -284,22 +337,22 @@ class ProcessPayment < ApplicationJob
|
|
|
284
337
|
include Postburner::Beanstalkd
|
|
285
338
|
|
|
286
339
|
queue_as :critical
|
|
287
|
-
|
|
288
|
-
|
|
340
|
+
priority 0
|
|
341
|
+
ttr 300 # 5 minutes to complete
|
|
289
342
|
end
|
|
290
343
|
|
|
291
344
|
class QuickEmail < ApplicationJob
|
|
292
345
|
include Postburner::Beanstalkd
|
|
293
346
|
|
|
294
347
|
queue_as :mailers
|
|
295
|
-
|
|
348
|
+
ttr 60 # 1 minute for fast email jobs
|
|
296
349
|
end
|
|
297
350
|
|
|
298
351
|
class LongRunningReport < ApplicationJob
|
|
299
352
|
include Postburner::Beanstalkd
|
|
300
353
|
|
|
301
354
|
queue_as :reports
|
|
302
|
-
|
|
355
|
+
ttr 3600 # 1 hour for complex reports
|
|
303
356
|
end
|
|
304
357
|
```
|
|
305
358
|
|
|
@@ -308,7 +361,7 @@ end
|
|
|
308
361
|
```ruby
|
|
309
362
|
class DataImport < Postburner::Job
|
|
310
363
|
queue 'imports'
|
|
311
|
-
|
|
364
|
+
ttr 1800 # 30 minutes (equivalent to queue_ttr)
|
|
312
365
|
|
|
313
366
|
def perform(args)
|
|
314
367
|
# Long-running import logic
|
|
@@ -329,7 +382,7 @@ class ProcessImport < ApplicationJob
|
|
|
329
382
|
include Postburner::Tracked # Includes Beanstalkd and enables extend!
|
|
330
383
|
|
|
331
384
|
queue_as :imports
|
|
332
|
-
|
|
385
|
+
ttr 300 # 5 minutes initial TTR
|
|
333
386
|
|
|
334
387
|
def perform(file_id)
|
|
335
388
|
file = ImportFile.find(file_id)
|
|
@@ -351,7 +404,7 @@ end
|
|
|
351
404
|
```ruby
|
|
352
405
|
class LargeDataProcessor < Postburner::Job
|
|
353
406
|
queue 'processing'
|
|
354
|
-
|
|
407
|
+
ttr 600 # 10 minutes
|
|
355
408
|
|
|
356
409
|
def perform(args)
|
|
357
410
|
dataset = Dataset.find(args['dataset_id'])
|
|
@@ -370,7 +423,7 @@ end
|
|
|
370
423
|
|
|
371
424
|
- Calls Beanstalkd's `touch` command on the reserved job
|
|
372
425
|
- Resets the TTR countdown to the original TTR value from the current moment
|
|
373
|
-
- Example: Job has `
|
|
426
|
+
- Example: Job has `ttr 300`. After 200 seconds, calling `extend!` gives you another 300 seconds (not just 100)
|
|
374
427
|
- Can be called multiple times during job execution
|
|
375
428
|
- Useful for iterative processing where each iteration is quick but total time is unpredictable
|
|
376
429
|
|
|
@@ -512,8 +565,8 @@ class SendWelcomeEmail < ApplicationJob
|
|
|
512
565
|
include Postburner::Beanstalkd
|
|
513
566
|
|
|
514
567
|
queue_as :mailers
|
|
515
|
-
|
|
516
|
-
|
|
568
|
+
priority 100 # Lower = higher priority (0 is highest)
|
|
569
|
+
ttr 300 # Time-to-run in seconds (5 minutes)
|
|
517
570
|
|
|
518
571
|
def perform(user_id)
|
|
519
572
|
UserMailer.welcome(user_id).deliver_now
|
|
@@ -540,8 +593,8 @@ class ProcessPayment < ApplicationJob
|
|
|
540
593
|
include Postburner::Tracked # ← Opt-in to PostgreSQL tracking (includes Beanstalkd)
|
|
541
594
|
|
|
542
595
|
queue_as :critical
|
|
543
|
-
|
|
544
|
-
|
|
596
|
+
priority 0 # Highest priority (Beanstalkd included automatically)
|
|
597
|
+
ttr 600 # 10 minute timeout
|
|
545
598
|
retry_on StandardError, wait: :exponentially_longer, attempts: 5
|
|
546
599
|
|
|
547
600
|
def perform(payment_id)
|
|
@@ -603,8 +656,8 @@ Direct `Postburner::Job` subclasses are **always tracked**:
|
|
|
603
656
|
```ruby
|
|
604
657
|
class RunDonation < Postburner::Job
|
|
605
658
|
queue 'critical'
|
|
606
|
-
|
|
607
|
-
|
|
659
|
+
priority 0
|
|
660
|
+
max_retries 0
|
|
608
661
|
|
|
609
662
|
def perform(args)
|
|
610
663
|
donation = Donation.find(args['donation_id'])
|
|
@@ -639,15 +692,15 @@ job.queue!
|
|
|
639
692
|
|
|
640
693
|
# Set dynamically after creation
|
|
641
694
|
job = RunDonation.create!(args: { 'donation_id' => 456 })
|
|
642
|
-
job.
|
|
643
|
-
job.
|
|
695
|
+
job.priority = 2000
|
|
696
|
+
job.ttr = 300
|
|
644
697
|
job.queue!
|
|
645
698
|
|
|
646
699
|
# Use in before_enqueue callback for conditional behavior
|
|
647
700
|
class ProcessOrder < Postburner::Job
|
|
648
701
|
queue 'orders'
|
|
649
|
-
|
|
650
|
-
|
|
702
|
+
priority 100 # Default priority
|
|
703
|
+
ttr 120 # Default TTR
|
|
651
704
|
|
|
652
705
|
before_enqueue :set_priority_based_on_urgency
|
|
653
706
|
|
|
@@ -660,11 +713,11 @@ class ProcessOrder < Postburner::Job
|
|
|
660
713
|
|
|
661
714
|
def set_priority_based_on_urgency
|
|
662
715
|
if args['express_shipping']
|
|
663
|
-
self.
|
|
664
|
-
self.
|
|
716
|
+
self.priority = 0 # High priority for express orders
|
|
717
|
+
self.ttr = 600 # Allow 10 minutes to complete
|
|
665
718
|
else
|
|
666
|
-
self.
|
|
667
|
-
self.
|
|
719
|
+
self.priority = 1000 # Low priority for standard orders
|
|
720
|
+
self.ttr = 120 # Standard 2 minute timeout
|
|
668
721
|
end
|
|
669
722
|
end
|
|
670
723
|
end
|
|
@@ -686,24 +739,25 @@ Postburner uses named worker configurations to support different deployment patt
|
|
|
686
739
|
Configure multiple named workers with different concurrency profiles:
|
|
687
740
|
|
|
688
741
|
```yaml
|
|
689
|
-
production:
|
|
742
|
+
production: # <- environment config, i.e. defaults
|
|
690
743
|
beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
|
|
744
|
+
default_forks: 2
|
|
745
|
+
default_threads: 10
|
|
746
|
+
default_gc_limit: 5000
|
|
691
747
|
|
|
692
|
-
workers:
|
|
748
|
+
workers: # <- worker config, i.e. overrides
|
|
693
749
|
# Heavy, memory-intensive jobs - more processes, fewer threads
|
|
694
750
|
imports:
|
|
695
|
-
|
|
696
|
-
|
|
697
|
-
|
|
751
|
+
forks: 4 # Overrides default_forks
|
|
752
|
+
threads: 1 # Overrides default_threads
|
|
753
|
+
gc_limit: 500 # Overrides default_gc_limit
|
|
698
754
|
queues:
|
|
699
755
|
- imports
|
|
700
756
|
- data_processing
|
|
701
757
|
|
|
702
758
|
# General jobs - fewer processes, many threads
|
|
703
759
|
general:
|
|
704
|
-
|
|
705
|
-
default_threads: 100
|
|
706
|
-
default_gc_limit: 5000
|
|
760
|
+
threads: 100 # Overrides default_threads (forks uses default_forks=2)
|
|
707
761
|
queues:
|
|
708
762
|
- default
|
|
709
763
|
- mailers
|
|
@@ -722,12 +776,12 @@ This gives you a natural progression from simple single-threaded development to
|
|
|
722
776
|
|
|
723
777
|
**Development (single worker, single-threaded):**
|
|
724
778
|
```yaml
|
|
725
|
-
development:
|
|
779
|
+
development: # <- environment config
|
|
726
780
|
beanstalk_url: beanstalk://localhost:11300
|
|
727
781
|
|
|
728
|
-
workers:
|
|
782
|
+
workers: # <- worker config
|
|
729
783
|
default:
|
|
730
|
-
#
|
|
784
|
+
# Uses env defaults: forks=0, threads=1, gc_limit=nil
|
|
731
785
|
queues:
|
|
732
786
|
- default
|
|
733
787
|
- mailers
|
|
@@ -735,13 +789,14 @@ development:
|
|
|
735
789
|
|
|
736
790
|
**Staging (single worker, multi-threaded):**
|
|
737
791
|
```yaml
|
|
738
|
-
staging:
|
|
792
|
+
staging: # <- environment config
|
|
739
793
|
beanstalk_url: beanstalk://localhost:11300
|
|
794
|
+
default_threads: 10
|
|
795
|
+
default_gc_limit: 5000
|
|
740
796
|
|
|
741
|
-
workers:
|
|
797
|
+
workers: # <- worker config
|
|
742
798
|
default:
|
|
743
|
-
|
|
744
|
-
default_gc_limit: 5000
|
|
799
|
+
# Uses env defaults: default_threads=10, default_gc_limit=5000
|
|
745
800
|
queues:
|
|
746
801
|
- critical
|
|
747
802
|
- default
|
|
@@ -750,22 +805,25 @@ staging:
|
|
|
750
805
|
|
|
751
806
|
**Production (multiple workers with different profiles):**
|
|
752
807
|
```yaml
|
|
753
|
-
production:
|
|
808
|
+
production: # <- environment config, i.e. defaults
|
|
754
809
|
beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
|
|
810
|
+
default_forks: 2
|
|
811
|
+
default_threads: 10
|
|
812
|
+
default_gc_limit: 5000
|
|
755
813
|
|
|
756
|
-
workers:
|
|
814
|
+
workers: # <- worker config, i.e. overrides
|
|
757
815
|
imports:
|
|
758
|
-
|
|
759
|
-
|
|
760
|
-
|
|
816
|
+
forks: 4 # Overrides default_forks (4 processes)
|
|
817
|
+
threads: 1 # Overrides default_threads (1 thread per process = 4 concurrent jobs)
|
|
818
|
+
gc_limit: 500 # Overrides default_gc_limit
|
|
761
819
|
queues:
|
|
762
820
|
- imports
|
|
763
821
|
- data_processing
|
|
764
822
|
|
|
765
823
|
general:
|
|
766
|
-
default_forks
|
|
767
|
-
|
|
768
|
-
default_gc_limit
|
|
824
|
+
# forks uses default_forks=2 (2 processes)
|
|
825
|
+
threads: 100 # Overrides default_threads (100 threads per process = 200 concurrent jobs)
|
|
826
|
+
# gc_limit uses default_gc_limit=5000
|
|
769
827
|
queues:
|
|
770
828
|
- default
|
|
771
829
|
- mailers
|
|
@@ -872,7 +930,7 @@ Parent Process
|
|
|
872
930
|
|
|
873
931
|
### GC Limits
|
|
874
932
|
|
|
875
|
-
Set `default_gc_limit` per worker to automatically restart after processing N jobs.
|
|
933
|
+
Set `default_gc_limit` at environment level or `gc_limit` per worker to automatically restart after processing N jobs.
|
|
876
934
|
|
|
877
935
|
- Worker processes N jobs
|
|
878
936
|
- Worker exits with code 99
|
|
@@ -880,20 +938,23 @@ Set `default_gc_limit` per worker to automatically restart after processing N jo
|
|
|
880
938
|
- In multi-process mode (forks: 1+): parent process automatically restarts just that fork
|
|
881
939
|
|
|
882
940
|
```yaml
|
|
883
|
-
production:
|
|
884
|
-
|
|
941
|
+
production: # <- environment config
|
|
942
|
+
default_forks: 2
|
|
943
|
+
default_threads: 10
|
|
944
|
+
default_gc_limit: 5000
|
|
945
|
+
|
|
946
|
+
workers: # <- worker config
|
|
885
947
|
imports:
|
|
886
|
-
|
|
887
|
-
|
|
888
|
-
|
|
948
|
+
forks: 4 # Overrides default_forks
|
|
949
|
+
threads: 1 # Overrides default_threads
|
|
950
|
+
gc_limit: 500 # Overrides default_gc_limit (restart after 500 jobs, memory-intensive)
|
|
889
951
|
queues:
|
|
890
952
|
- imports
|
|
891
953
|
- data_processing
|
|
892
954
|
|
|
893
955
|
general:
|
|
894
|
-
default_forks
|
|
895
|
-
|
|
896
|
-
default_gc_limit: 5000 # Restart after 5000 jobs
|
|
956
|
+
# Uses default_forks=2, default_threads=100, default_gc_limit=5000 (restart after 5000 jobs)
|
|
957
|
+
threads: 100 # Overrides default_threads
|
|
897
958
|
queues:
|
|
898
959
|
- default
|
|
899
960
|
- mailers
|
|
@@ -913,48 +974,64 @@ production:
|
|
|
913
974
|
default: &default
|
|
914
975
|
beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
|
|
915
976
|
|
|
916
|
-
development:
|
|
977
|
+
development: # <- environment config
|
|
917
978
|
<<: *default
|
|
918
|
-
#
|
|
919
|
-
|
|
920
|
-
|
|
921
|
-
|
|
979
|
+
# Env defaults: forks=0, threads=1, gc_limit=nil
|
|
980
|
+
|
|
981
|
+
workers: # <- worker config
|
|
982
|
+
default:
|
|
983
|
+
queues:
|
|
984
|
+
- default
|
|
985
|
+
- mailers
|
|
922
986
|
|
|
923
|
-
test:
|
|
987
|
+
test: # <- environment config
|
|
924
988
|
<<: *default
|
|
925
|
-
#
|
|
926
|
-
|
|
927
|
-
|
|
989
|
+
# Env defaults: forks=0, threads=1, gc_limit=nil
|
|
990
|
+
|
|
991
|
+
workers: # <- worker config
|
|
992
|
+
default:
|
|
993
|
+
queues:
|
|
994
|
+
- default
|
|
928
995
|
|
|
929
|
-
staging:
|
|
996
|
+
staging: # <- environment config
|
|
930
997
|
<<: *default
|
|
931
998
|
default_threads: 10 # Multi-threaded, single process
|
|
932
999
|
default_gc_limit: 5000
|
|
933
|
-
queues:
|
|
934
|
-
default: {}
|
|
935
|
-
mailers: {}
|
|
936
1000
|
|
|
937
|
-
|
|
1001
|
+
workers: # <- worker config
|
|
1002
|
+
default:
|
|
1003
|
+
# Uses env defaults: default_threads=10, default_gc_limit=5000
|
|
1004
|
+
queues:
|
|
1005
|
+
- default
|
|
1006
|
+
- mailers
|
|
1007
|
+
|
|
1008
|
+
production: # <- environment config, i.e. defaults
|
|
938
1009
|
<<: *default
|
|
939
|
-
default_forks:
|
|
940
|
-
default_threads: 10 #
|
|
941
|
-
default_gc_limit: 5000
|
|
1010
|
+
default_forks: 2 # Default for workers
|
|
1011
|
+
default_threads: 10 # Default for workers
|
|
1012
|
+
default_gc_limit: 5000 # Default for workers
|
|
942
1013
|
|
|
943
|
-
|
|
1014
|
+
workers: # <- worker config, i.e. overrides
|
|
944
1015
|
critical:
|
|
945
|
-
forks: 1
|
|
946
|
-
threads: 1 # 1 concurrent job
|
|
947
|
-
gc_limit: 100
|
|
1016
|
+
forks: 1 # Overrides default_forks
|
|
1017
|
+
threads: 1 # Overrides default_threads (1 concurrent job)
|
|
1018
|
+
gc_limit: 100 # Overrides default_gc_limit
|
|
1019
|
+
queues:
|
|
1020
|
+
- critical
|
|
948
1021
|
|
|
949
1022
|
default:
|
|
950
|
-
forks: 4
|
|
951
|
-
threads: 10 # 40 total concurrent jobs
|
|
952
|
-
gc_limit: 1000
|
|
1023
|
+
forks: 4 # Overrides default_forks
|
|
1024
|
+
threads: 10 # Overrides default_threads (40 total concurrent jobs: 4 × 10)
|
|
1025
|
+
gc_limit: 1000 # Overrides default_gc_limit
|
|
1026
|
+
queues:
|
|
1027
|
+
- default
|
|
953
1028
|
|
|
954
1029
|
mailers:
|
|
955
|
-
forks
|
|
956
|
-
threads: 5 # 10 total email senders
|
|
957
|
-
gc_limit: 500
|
|
1030
|
+
# forks uses default_forks=2
|
|
1031
|
+
threads: 5 # Overrides default_threads (10 total email senders: 2 × 5)
|
|
1032
|
+
gc_limit: 500 # Overrides default_gc_limit
|
|
1033
|
+
queues:
|
|
1034
|
+
- mailers
|
|
958
1035
|
```
|
|
959
1036
|
|
|
960
1037
|
### Queue Configuration Methods
|
|
@@ -964,10 +1041,10 @@ For `Postburner::Job` subclasses (and backwards compatibility):
|
|
|
964
1041
|
```ruby
|
|
965
1042
|
class CriticalJob < Postburner::Job
|
|
966
1043
|
queue 'critical' # Beanstalkd tube name
|
|
967
|
-
|
|
968
|
-
|
|
969
|
-
|
|
970
|
-
|
|
1044
|
+
priority 0 # Lower = higher priority (0 is highest)
|
|
1045
|
+
ttr 300 # TTR (time-to-run) in seconds
|
|
1046
|
+
max_retries 3 # Max retry attempts
|
|
1047
|
+
retry_delay 5 # Fixed delay: 5 seconds between retries
|
|
971
1048
|
|
|
972
1049
|
def perform(args)
|
|
973
1050
|
# ...
|
|
@@ -980,8 +1057,8 @@ end
|
|
|
980
1057
|
```ruby
|
|
981
1058
|
class BackgroundTask < Postburner::Job
|
|
982
1059
|
queue 'default'
|
|
983
|
-
|
|
984
|
-
|
|
1060
|
+
max_retries 5
|
|
1061
|
+
retry_delay ->(retries) { 2 ** retries } # 2s, 4s, 8s, 16s, 32s
|
|
985
1062
|
|
|
986
1063
|
def perform(args)
|
|
987
1064
|
# ...
|
|
@@ -1213,7 +1290,7 @@ Direct access to Beanstalkd for advanced operations:
|
|
|
1213
1290
|
job.bkid # => 12345
|
|
1214
1291
|
|
|
1215
1292
|
# Access Beaneater job object
|
|
1216
|
-
job.
|
|
1293
|
+
job.bk.stats
|
|
1217
1294
|
# => {"id"=>12345, "tube"=>"critical", "state"=>"ready", ...}
|
|
1218
1295
|
|
|
1219
1296
|
# Connection management
|
|
@@ -1341,6 +1418,10 @@ Key changes in v1.0:
|
|
|
1341
1418
|
|
|
1342
1419
|
Submit a pull request. Follow project conventions. Be nice.
|
|
1343
1420
|
|
|
1421
|
+
There is a CLAUDE.md file for guidance when using Claude Code. Please use it or contribute to it. Do NOT reference AI tools in commits or code. Do not co-author with AI tools. Pull requests will be rejected if they violate these rules.
|
|
1422
|
+
|
|
1423
|
+
We encourage AI tools, but do not vibe, as the code must look like it was written by a human. Code that contains AI agent idioms will be rejected. Code that doesn't follow the project conventions will be rejected.
|
|
1424
|
+
|
|
1344
1425
|
## License
|
|
1345
1426
|
|
|
1346
1427
|
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|