backburner-allq 1.0.3 → 1.0.9
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +2 -131
- data/HOOKS.md +1 -1
- data/README.md +22 -56
- data/TODO +0 -4
- data/backburner-allq.gemspec +2 -2
- data/deploy.sh +1 -1
- data/lib/backburner/allq_wrapper.rb +23 -31
- data/lib/backburner/configuration.rb +2 -2
- data/lib/backburner/connection.rb +4 -8
- data/lib/backburner/job.rb +1 -6
- data/lib/backburner/version.rb +1 -1
- data/lib/backburner/worker.rb +13 -12
- data/lib/backburner/workers/threading.rb +1 -1
- data/lib/backburner/workers/threads_on_fork.rb +1 -1
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 252fc4e5aa793b391b70aa0d4947d88c7d3c1ad3df7807824a9b7c351937caff
|
4
|
+
data.tar.gz: b1f459dc189399a59d26bebae015b10377916fd250edf66a987cbb8bcb264ca6
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 8ce217f35c0b62b80e707264c253a479e828f0d5a075b145c63d558c4f525d1928d6fbb0f4e247adcff71fe398bec213ecbf75d67ae83c3794413f49b2832cec
|
7
|
+
data.tar.gz: 4229e711046a0f055fda7992708be107e1e53d6dc9eca834bfade289734188d46b7e86c13d567eec14bce6fcd263943f9c7e6202e4ab2c254759ee34ec8b80da
|
data/CHANGELOG.md
CHANGED
@@ -1,133 +1,4 @@
|
|
1
1
|
# CHANGELOG
|
2
2
|
|
3
|
-
## Version 1.
|
4
|
-
|
5
|
-
* TBD
|
6
|
-
|
7
|
-
## Version 1.4.1 (June 10 2017)
|
8
|
-
|
9
|
-
* Fix warning for constant ::Fixnum is deprecated (@amatsuda)
|
10
|
-
|
11
|
-
## Version 1.4.0 (May 13 2017)
|
12
|
-
|
13
|
-
* Fix unit tests to be more consistent (@eltone)
|
14
|
-
* Ensure job supports body hash with symbol keys (@eltone)
|
15
|
-
* Add support for custom serialization formats (@eltone)
|
16
|
-
* Log the params when a job timeout occurs (@nathantsoi)
|
17
|
-
|
18
|
-
## Version 1.3.1 (April 21 2016)
|
19
|
-
|
20
|
-
* Addition of thread-pool-based concurrency (@contentfree)
|
21
|
-
|
22
|
-
## Version 1.3.0 (February 05 2016)
|
23
|
-
|
24
|
-
* Enqueue command now responds with beanstalk response details
|
25
|
-
|
26
|
-
## Version 1.2.0 (November 01 2015)
|
27
|
-
|
28
|
-
* FIX Made connections to beanstalkd more resilient (@contentfree)
|
29
|
-
|
30
|
-
## Version 1.2.0.pre (October 24 2015)
|
31
|
-
|
32
|
-
* FIX Replace static Beaneater connection with individual connections per worker instance/thread (@contentfree)
|
33
|
-
* FIX Beaneater connections try really hard to repair themselves if disconnected accidentally (@contentfree)
|
34
|
-
* NEW Event hook for workers: on_reconnect (@contentfree)
|
35
|
-
|
36
|
-
## Version 1.1.0 (September 14 2015)
|
37
|
-
|
38
|
-
* NEW Ability to configure namespace separator (@bfolkens)
|
39
|
-
* NEW Avoid timeouts altogether by setting queue_respond_timeout to 0 (@zacviandier)
|
40
|
-
* NEW Event hooks for on_retry and on_bury (@contentfree)
|
41
|
-
* NEW Support lambdas for queue names (@contentfree)
|
42
|
-
* NEW Allow for control of delay calculation (@contentfree)
|
43
|
-
* NEW Ability to specify environment when running the CLI (@contentfree)
|
44
|
-
* NEW Control default async behavior of methods (@contentfree)
|
45
|
-
|
46
|
-
## Version 1.0.0 (April 26 2015)
|
47
|
-
|
48
|
-
* NEW Updating to Beaneater 1.0 (@alup)
|
49
|
-
|
50
|
-
## Version 0.4.6 (October 26 2014)
|
51
|
-
|
52
|
-
* NEW Add job to on_error handler if the handler has a 4th argument (@Nitrodist)
|
53
|
-
* NEW Use a timeout when looking for a job to reserve (@EasyPost)
|
54
|
-
* NEW Support configuring settings on threads on fork class (@silentshade)
|
55
|
-
* FIX queue override by existing queues (@silentshade)
|
56
|
-
* FIX Use thread to log exit message (@silentshade)
|
57
|
-
|
58
|
-
## Version 0.4.5 (December 16 2013)
|
59
|
-
|
60
|
-
* FIX #47 Create a backburner connection per thread (Thanks @thcrock)
|
61
|
-
|
62
|
-
## Version 0.4.4 (October 27 2013)
|
63
|
-
|
64
|
-
* NEW #51 Added ability to set per-queue default ttr's (Thanks @ryanjohns)
|
65
|
-
|
66
|
-
## Version 0.4.3 (July 19 2013)
|
67
|
-
|
68
|
-
* FIX #44 Additional fix to issue introduced in 0.4.2
|
69
|
-
* FIX #45 More graceful shutdown using Kernel.exit and rescuing SystemExit. (Thanks @ryanjohns)
|
70
|
-
|
71
|
-
## Version 0.4.2 (July 3 2013)
|
72
|
-
|
73
|
-
* FIX #44 Properly retry to connect to beanstalkd when connection fails.
|
74
|
-
|
75
|
-
## Version 0.4.1 (June 28 2013)
|
76
|
-
|
77
|
-
* FIX #43 Properly support CLI options and smart load the app environment.
|
78
|
-
|
79
|
-
## Version 0.4.0 (June 28 2013)
|
80
|
-
|
81
|
-
NOTE: This is the start of working with @bradgessler to improve backburner and merge with quebert
|
82
|
-
|
83
|
-
* NEW #26 #27 Remove need for Queue mixin, allow plain ruby objects
|
84
|
-
* NEW Default all jobs to a single general queue rather than separate queues
|
85
|
-
* NEW Add support for named priorities, allowing shorthand names for priority values
|
86
|
-
|
87
|
-
## Version 0.3.4 (April 23 2013)
|
88
|
-
|
89
|
-
* FIX #22 Adds signal handlers for worker to manage proper shutdown (Thanks @tkiley)
|
90
|
-
|
91
|
-
## Version 0.3.3 (April 19 2013)
|
92
|
-
|
93
|
-
* Fix naming conflict rename 'config' to 'queue_config'
|
94
|
-
|
95
|
-
## Version 0.3.2 (Jan 23 2013)
|
96
|
-
|
97
|
-
* Bump version of beaneater to 0.3.0 (better socket handling)
|
98
|
-
|
99
|
-
## Version 0.3.1 (Dec 28 2012)
|
100
|
-
|
101
|
-
* Adds basic forking processing strategy and rake tasks (Thanks @danielfarrell)
|
102
|
-
|
103
|
-
## Version 0.3.0 (Nov 14 2012)
|
104
|
-
|
105
|
-
* Major update with support for a 'threads_on_fork' processing strategy (Thanks @ShadowBelmolve)
|
106
|
-
* Different workers have different rake tasks (Thanks @ShadowBelmolve)
|
107
|
-
* Added processing strategy specific examples i.e stress.rb and adds new unit tests. (Thanks @ShadowBelmolve)
|
108
|
-
|
109
|
-
## Version 0.2.6 (Nov 12 2012)
|
110
|
-
|
111
|
-
* Upgrade to beaneater 0.2.0
|
112
|
-
|
113
|
-
## Version 0.2.5 (Nov 9 2012)
|
114
|
-
|
115
|
-
* Add support for multiple worker processing strategies through subclassing.
|
116
|
-
|
117
|
-
## Version 0.2.0 (Nov 7 2012)
|
118
|
-
|
119
|
-
* Add new plugin hooks feature (see HOOKS.md)
|
120
|
-
|
121
|
-
## Version 0.1.2 (Nov 7 2012)
|
122
|
-
|
123
|
-
* Adds ability to specify a custom logger.
|
124
|
-
* Adds job retry configuration and worker support.
|
125
|
-
|
126
|
-
## Version 0.1.1 (Nov 6 2012)
|
127
|
-
|
128
|
-
* Fix issue with timed out reserves
|
129
|
-
|
130
|
-
## Version 0.1.0 (Nov 4 2012)
|
131
|
-
|
132
|
-
* Switch to beaneater as new ruby beanstalkd client
|
133
|
-
* Add support for array of connections in `beanstalk_url`
|
3
|
+
## Version 1.0.9 (September 9 2020)
|
4
|
+
Branched from backburner to use Allq instead
|
data/HOOKS.md
CHANGED
@@ -6,7 +6,7 @@ In many cases you can use a hook rather than mess around with Backburner's inter
|
|
6
6
|
## Job Hooks
|
7
7
|
|
8
8
|
Hooks are transparently adapted from [Resque](https://github.com/resque/resque/blob/master/docs/HOOKS.md), so
|
9
|
-
if you are familiar with their hook API, now you can use nearly the same ones with
|
9
|
+
if you are familiar with their hook API, now you can use nearly the same ones with allq and backburner!
|
10
10
|
|
11
11
|
There are a variety of hooks available that are triggered during the lifecycle of a job:
|
12
12
|
|
data/README.md
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
# Backburner [![Build Status](https://travis-ci.org/nesquena/backburner.svg?branch=master)](https://travis-ci.org/nesquena/backburner)
|
2
2
|
|
3
|
-
Backburner is a [
|
3
|
+
Backburner-allq is a [allq](http://www.allqueue.com)-powered job queue that can handle a very high volume of jobs.
|
4
4
|
You create background jobs and place them on multiple work queues to be processed later.
|
5
5
|
|
6
|
-
Processing background jobs reliably has never been easier than with
|
6
|
+
Processing background jobs reliably has never been easier than with allq and Backburner. This gem works with any ruby-based
|
7
7
|
web framework, but is especially suited for use with [Sinatra](http://sinatrarb.com), [Padrino](http://padrinorb.com) and Rails.
|
8
8
|
|
9
|
-
If you want to use
|
9
|
+
If you want to use allq for your job processing, consider using Backburner.
|
10
10
|
Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads.
|
11
|
-
Persistent queues are supported when
|
11
|
+
Persistent queues are supported when allq persistence mode is enabled.
|
12
12
|
|
13
13
|
Backburner supports multiple queues, job priorities, delays, and timeouts. In addition,
|
14
14
|
Backburner has robust support for retrying failed jobs, handling error cases,
|
@@ -26,28 +26,14 @@ libraries under the hood. Every job queue requires a queue store that jobs are p
|
|
26
26
|
In the case of Resque, jobs are processed through **Redis**, a persistent key-value store. In the case of DelayedJob, jobs are processed through
|
27
27
|
**ActiveRecord** and a database such as PostgreSQL.
|
28
28
|
|
29
|
-
The work queue underlying these gems tells you infinitely more about the differences than anything else.
|
30
|
-
Beanstalk is probably the best solution for job queues available today for many reasons.
|
31
|
-
The real question then is... "Why Beanstalk?".
|
32
29
|
|
33
|
-
|
30
|
+
You will quickly see that **allq** is an underrated but incredible project that is extremely well-suited as a job queue.
|
31
|
+
Significantly better suited for this task than Redis or a database.
|
34
32
|
|
35
|
-
|
36
|
-
|
37
|
-
Adam Wiggins posted [an excellent comparison](http://adam.herokuapp.com/past/2010/4/24/beanstalk_a_simple_and_fast_queueing_backend/).
|
33
|
+
A single instance of Allq is perfectly capable of handling thousands of jobs a second (or more, depending on your job size)
|
34
|
+
because it is an in-memory, event-driven system.
|
38
35
|
|
39
|
-
|
40
|
-
Significantly better suited for this task than Redis or a database. Beanstalk is a simple,
|
41
|
-
and a very fast work queue service rolled into a single binary - it is the memcached of work queues.
|
42
|
-
Originally built to power the backend for the 'Causes' Facebook app, it is a mature and production ready open source project.
|
43
|
-
[PostRank](http://www.postrank.com) uses beanstalk to reliably process millions of jobs a day.
|
44
|
-
|
45
|
-
A single instance of Beanstalk is perfectly capable of handling thousands of jobs a second (or more, depending on your job size)
|
46
|
-
because it is an in-memory, event-driven system. Powered by libevent under the hood,
|
47
|
-
it requires zero setup (launch and forget, à la memcached), optional log based persistence, an easily parsed ASCII protocol,
|
48
|
-
and a rich set of tools for job management that go well beyond a simple FIFO work queue.
|
49
|
-
|
50
|
-
Beanstalkd supports the following features out of the box:
|
36
|
+
Allq supports the following features out of the box:
|
51
37
|
|
52
38
|
| Feature | Description |
|
53
39
|
| ------- | ------------------------------- |
|
@@ -59,27 +45,14 @@ Beanstalkd supports the following features out of the box:
|
|
59
45
|
| **Persistence** | Jobs are stored in memory for speed, but logged to disk for safe keeping. |
|
60
46
|
| **Federation** | Horizontal scalability provided through federation by the client. |
|
61
47
|
| **Error Handling** | Bury any job which causes an error for later debugging and inspection.|
|
48
|
+
| **Fair Queueing** | Allows you to shard work across queues for fair client queuing.|
|
49
|
+
| **Workflows** | Allows you tell Job C to wait for Job A and Job B to finish first.|
|
62
50
|
|
63
|
-
Keep in mind that these features are supported out of the box with
|
64
|
-
In the end, **
|
51
|
+
Keep in mind that these features are supported out of the box with allq and require no special code within this gem to support.
|
52
|
+
In the end, **allq is the ideal job queue** while also being ridiculously easy to install and setup.
|
65
53
|
|
66
54
|
## Installation
|
67
55
|
|
68
|
-
First, you probably want to [install beanstalkd](http://kr.github.com/beanstalkd/download.html), which powers the job queues.
|
69
|
-
Depending on your platform, this should be as simple as (for Ubuntu):
|
70
|
-
|
71
|
-
$ sudo apt-get install beanstalkd
|
72
|
-
|
73
|
-
Add this line to your application's Gemfile:
|
74
|
-
|
75
|
-
gem 'backburner'
|
76
|
-
|
77
|
-
And then execute:
|
78
|
-
|
79
|
-
$ bundle
|
80
|
-
|
81
|
-
Or install it yourself as:
|
82
|
-
|
83
56
|
$ gem install backburner
|
84
57
|
|
85
58
|
## Configuration ##
|
@@ -88,7 +61,7 @@ Backburner is extremely simple to setup. Just configure basic settings for backb
|
|
88
61
|
|
89
62
|
```ruby
|
90
63
|
Backburner.configure do |config|
|
91
|
-
config.beanstalk_url = "
|
64
|
+
config.beanstalk_url = "allq://127.0.0.1:8090"
|
92
65
|
config.tube_namespace = "some.app.production"
|
93
66
|
config.namespace_separator = "."
|
94
67
|
config.on_error = lambda { |e| puts e }
|
@@ -112,7 +85,7 @@ The key options available are:
|
|
112
85
|
|
113
86
|
| Option | Description |
|
114
87
|
| ----------------- | ------------------------------- |
|
115
|
-
| `
|
88
|
+
| `allq_url` | Address for allq connection i.e 'allq://127.0.0.1:8090' |
|
116
89
|
| `tube_namespace` | Prefix used for all tubes related to this backburner queue. |
|
117
90
|
| `namespace_separator` | Separator used for namespace and queue name |
|
118
91
|
| `on_error` | Lambda invoked with the error whenever any job in the system fails. |
|
@@ -129,15 +102,10 @@ The key options available are:
|
|
129
102
|
| `job_serializer_proc` | Lambda serializes a job body to a string to write to the task |
|
130
103
|
| `job_parser_proc` | Lambda parses a task body string to a hash |
|
131
104
|
|
132
|
-
## Breaking Changes
|
133
|
-
|
134
|
-
Before **v0.4.0**: Jobs were placed into default queues based on the name of the class creating the queue. i.e NewsletterJob would
|
135
|
-
be put into a 'newsletter-job' queue. As of 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs"
|
136
|
-
unless otherwise specified.
|
137
105
|
|
138
106
|
## Usage
|
139
107
|
|
140
|
-
Backburner allows you to create jobs and place them onto any number of
|
108
|
+
Backburner allows you to create jobs and place them onto any number of allq tubes, and later pull those jobs off the tubes and
|
141
109
|
process them asynchronously with a worker.
|
142
110
|
|
143
111
|
### Enqueuing Jobs ###
|
@@ -165,7 +133,7 @@ class NewsletterJob
|
|
165
133
|
|
166
134
|
# optional, defaults to respond_timeout in config
|
167
135
|
def self.queue_respond_timeout
|
168
|
-
300 # number of seconds before job times out, 0 to avoid timeout. NB: A timeout of 1 second will likely lead to race conditions between Backburner and
|
136
|
+
300 # number of seconds before job times out, 0 to avoid timeout. NB: A timeout of 1 second will likely lead to race conditions between Backburner and allq and should be avoided
|
169
137
|
end
|
170
138
|
|
171
139
|
# optional, defaults to retry_delay_proc in config
|
@@ -387,15 +355,15 @@ This is why our examples use object IDs instead of passing around objects.
|
|
387
355
|
|
388
356
|
### Named Priorities
|
389
357
|
|
390
|
-
As of v0.4.0, Backburner has support for named priorities.
|
358
|
+
As of v0.4.0, Backburner has support for named priorities. allq priorities are numerical but
|
391
359
|
backburner supports a mapping between a word and a numerical value. The following priorities are
|
392
|
-
available by default: `high` is 0, `medium` is
|
360
|
+
available by default: `high` is 0, `medium` is 4, and `low` is 9.
|
393
361
|
|
394
362
|
Priorities can be customized with:
|
395
363
|
|
396
364
|
```ruby
|
397
365
|
Backburner.configure do |config|
|
398
|
-
config.priority_labels = { :custom =>
|
366
|
+
config.priority_labels = { :custom => 1, :useful => 5 }
|
399
367
|
# or append to default priorities with
|
400
368
|
# config.priority_labels = Backburner::Configuration::PRIORITY_LABELS.merge(:foo => 5)
|
401
369
|
end
|
@@ -557,8 +525,7 @@ Check out [HOOKS.md](https://github.com/nesquena/backburner/blob/master/HOOKS.md
|
|
557
525
|
|
558
526
|
### Workers in Production
|
559
527
|
|
560
|
-
Once you have Backburner setup in your application, starting workers is really easy.
|
561
|
-
is installed, your best bet is to use the built-in rake task that comes with Backburner. Simply add the task to your Rakefile:
|
528
|
+
Once you have Backburner setup in your application, starting workers is really easy. Simply add the task to your Rakefile:
|
562
529
|
|
563
530
|
```ruby
|
564
531
|
# Rakefile
|
@@ -609,8 +576,7 @@ backburner -e $environment
|
|
609
576
|
|
610
577
|
#### Reconnecting
|
611
578
|
|
612
|
-
In Backburner, if the
|
613
|
-
After several retries, if the connection is still not able to be made, a `Beaneater::NotConnected` exception will be raised.
|
579
|
+
In Backburner, if the allq connection is temporarily severed, several retries to establish the connection will be attempted.
|
614
580
|
You can manually catch this exception, and attempt another manual retry using `Backburner::Worker.retry_connection!`.
|
615
581
|
|
616
582
|
### Web Front-end
|
data/TODO
CHANGED
data/backburner-allq.gemspec
CHANGED
@@ -4,9 +4,9 @@ require File.expand_path('../lib/backburner/version', __FILE__)
|
|
4
4
|
Gem::Specification.new do |s|
|
5
5
|
s.authors = ["Jason Malcolm"]
|
6
6
|
s.email = ["jason@blitline.com"]
|
7
|
-
s.description = %q{
|
7
|
+
s.description = %q{Allq background job processing made easy}
|
8
8
|
s.summary = %q{Reliable allq background job processing made easy for Ruby and Sinatra}
|
9
|
-
s.homepage = "http://github.com/
|
9
|
+
s.homepage = "http://github.com/blitline-dev/backburner-allq"
|
10
10
|
|
11
11
|
s.files = `git ls-files`.split($\)
|
12
12
|
s.executables = s.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
|
data/deploy.sh
CHANGED
@@ -79,7 +79,7 @@ module Backburner
|
|
79
79
|
end
|
80
80
|
|
81
81
|
class AllQWrapper
|
82
|
-
def initialize(url =
|
82
|
+
def initialize(url = 'localhost:8090')
|
83
83
|
allq_conf = Allq::Configuration.new do |config|
|
84
84
|
config.host = url
|
85
85
|
end
|
@@ -127,35 +127,30 @@ module Backburner
|
|
127
127
|
return nil if job.body.nil?
|
128
128
|
|
129
129
|
job.body = Base64.decode64(job.body) if job
|
130
|
-
job_obj =
|
130
|
+
job_obj = Backburner::AllQJob.new(self, job)
|
131
131
|
job_obj
|
132
132
|
end
|
133
133
|
|
134
134
|
def get(tube_name = 'default')
|
135
135
|
job = nil
|
136
|
-
|
137
|
-
job = @client.job_get(tube_name)
|
138
|
-
end
|
139
|
-
puts "Allq http get delta: #{delt} #{tube_name} #{job}" if delt.to_f > 1.5
|
136
|
+
job = @client.job_get(tube_name)
|
140
137
|
|
141
|
-
@recent_times.push(delt.to_f)
|
142
|
-
@recent_times.shift if @recent_times.size > 2
|
143
138
|
# Inplace decode
|
144
139
|
job.body = Base64.decode64(job.body) if job&.body
|
145
140
|
|
146
|
-
job_obj =
|
141
|
+
job_obj = Backburner::AllQJob.new(self, job)
|
147
142
|
job_obj
|
148
143
|
rescue StandardError => ex
|
149
144
|
if ex.message == "Couldn't resolve host name"
|
150
|
-
|
145
|
+
puts("COUDNT RESOLVE HOST NAME------ SHOULD REBOOT")
|
151
146
|
else
|
152
|
-
|
147
|
+
puts(ex)
|
153
148
|
end
|
154
149
|
end
|
155
150
|
|
156
151
|
def close
|
157
152
|
rescue StandardError => ex
|
158
|
-
|
153
|
+
puts(ex)
|
159
154
|
end
|
160
155
|
|
161
156
|
def map_priority(app_priority)
|
@@ -178,9 +173,9 @@ module Backburner
|
|
178
173
|
end
|
179
174
|
|
180
175
|
def log_result(job_result)
|
181
|
-
|
176
|
+
puts("ALLQ-HTTP-JOB-ID=#{job_result.job_id}")
|
182
177
|
rescue StandardError => ex
|
183
|
-
|
178
|
+
puts(ex)
|
184
179
|
end
|
185
180
|
|
186
181
|
def build_new_job(body, options)
|
@@ -241,30 +236,27 @@ module Backburner
|
|
241
236
|
begin
|
242
237
|
Timeout.timeout(10) do
|
243
238
|
if body && body.to_s.include?('["default"]')
|
244
|
-
|
239
|
+
puts "PUTTING DEFAULT! #{caller.inspect}"
|
245
240
|
end
|
246
241
|
|
247
|
-
|
248
|
-
|
249
|
-
|
250
|
-
|
251
|
-
|
252
|
-
|
253
|
-
result = @client.job_post(new_job)
|
254
|
-
end
|
255
|
-
raise 'PUT returned nil' if result.nil? || result.to_s == ''
|
242
|
+
if is_parent
|
243
|
+
new_job = build_new_parent_job(body, options)
|
244
|
+
result = @client.parent_job_post(new_job)
|
245
|
+
else
|
246
|
+
new_job = build_new_job(body, options)
|
247
|
+
result = @client.job_post(new_job)
|
256
248
|
end
|
257
|
-
|
249
|
+
raise 'PUT returned nil' if result.nil? || result.to_s == ''
|
258
250
|
end
|
259
251
|
rescue Timeout::Error
|
260
|
-
|
252
|
+
puts('ALLQ_PUT_TIMEOUT')
|
261
253
|
sleep(5)
|
262
254
|
retry_count += 1
|
263
255
|
retry if retry_count < 4
|
264
256
|
raise 'Failed to put on allq, we are investigating the problem, please try again'
|
265
257
|
rescue StandardError => ex
|
266
|
-
|
267
|
-
|
258
|
+
puts('Failed to ALLQ PUT')
|
259
|
+
puts(ex)
|
268
260
|
retry_count += 1
|
269
261
|
sleep(5)
|
270
262
|
retry if retry_count < 4
|
@@ -290,7 +282,7 @@ module Backburner
|
|
290
282
|
end
|
291
283
|
final_stats
|
292
284
|
rescue StandardError => ex
|
293
|
-
|
285
|
+
puts(ex)
|
294
286
|
{}
|
295
287
|
end
|
296
288
|
|
@@ -300,7 +292,7 @@ module Backburner
|
|
300
292
|
count = tube_stats['ready'].to_i if tube_stats && tube_stats['ready']
|
301
293
|
count
|
302
294
|
rescue StandardError => ex
|
303
|
-
|
295
|
+
puts(ex)
|
304
296
|
-1
|
305
297
|
end
|
306
298
|
|
@@ -308,7 +300,7 @@ module Backburner
|
|
308
300
|
result = get_ready_by_tube('default')
|
309
301
|
result.to_i
|
310
302
|
rescue StandardError => ex
|
311
|
-
|
303
|
+
puts(ex)
|
312
304
|
0
|
313
305
|
end
|
314
306
|
end
|
@@ -21,8 +21,8 @@ module Backburner
|
|
21
21
|
attr_accessor :job_parser_proc # proc to parse a job body from a string
|
22
22
|
|
23
23
|
def initialize
|
24
|
-
@allq_url = "127.0.0.1:
|
25
|
-
@tube_namespace = ""
|
24
|
+
@allq_url = "allq://127.0.0.1:8091"
|
25
|
+
@tube_namespace = "backallq"
|
26
26
|
@namespace_separator = "."
|
27
27
|
@default_priority = 5
|
28
28
|
@respond_timeout = 120
|
@@ -1,7 +1,7 @@
|
|
1
1
|
require 'delegate'
|
2
2
|
|
3
3
|
module Backburner
|
4
|
-
class Connection
|
4
|
+
class Connection
|
5
5
|
class BadURL < RuntimeError; end
|
6
6
|
|
7
7
|
attr_accessor :url, :allq_wrapper
|
@@ -27,7 +27,6 @@ module Backburner
|
|
27
27
|
def close
|
28
28
|
@allq_wrapper.close if @allq_wrapper
|
29
29
|
@allq_wrapper = nil
|
30
|
-
__setobj__(@allq_wrapper)
|
31
30
|
end
|
32
31
|
|
33
32
|
# Determines if the connection to allq is currently open
|
@@ -67,7 +66,7 @@ module Backburner
|
|
67
66
|
begin
|
68
67
|
yield
|
69
68
|
|
70
|
-
rescue
|
69
|
+
rescue Exception => e
|
71
70
|
if retry_count > 0
|
72
71
|
reconnect!
|
73
72
|
retry_count -= 1
|
@@ -80,8 +79,6 @@ module Backburner
|
|
80
79
|
end
|
81
80
|
end
|
82
81
|
|
83
|
-
protected
|
84
|
-
|
85
82
|
# Attempt to ensure we're connected to allq if the missing method is
|
86
83
|
# present in the delegate and we haven't shut down the connection on purpose
|
87
84
|
# @raise [Beaneater::NotConnected] If allq fails to connect after multiple attempts.
|
@@ -93,8 +90,7 @@ module Backburner
|
|
93
90
|
# Connects to a allq queue
|
94
91
|
# @raise Beaneater::NotConnected if the connection cannot be established
|
95
92
|
def connect!
|
96
|
-
@allq_wrapper = Backburner::
|
97
|
-
__setobj__(@allq_wrapper)
|
93
|
+
@allq_wrapper = Backburner::AllQWrapper.new(allq_addresses)
|
98
94
|
@allq_wrapper
|
99
95
|
end
|
100
96
|
|
@@ -102,7 +98,7 @@ module Backburner
|
|
102
98
|
pri = (opt[:pri] || 5).to_i
|
103
99
|
delay = opt[:delay].to_i
|
104
100
|
ttr = (opt[:ttr] || 600).to_i
|
105
|
-
@allq_wrapper.put2(data, pri, ttr, delay)
|
101
|
+
@allq_wrapper.put2(data, pri, ttr, tube_name, delay)
|
106
102
|
end
|
107
103
|
|
108
104
|
def get(tube_name)
|
data/lib/backburner/job.rb
CHANGED
@@ -48,12 +48,7 @@ module Backburner
|
|
48
48
|
return false unless res
|
49
49
|
# Execute the job
|
50
50
|
@hooks.around_hook_events(job_name, :around_perform, *args) do
|
51
|
-
|
52
|
-
# a) ttr == 0, to support never timing out
|
53
|
-
# b) ttr == 1, so that we don't accidentally set it to never time out
|
54
|
-
# NB: A ttr of 1 will likely result in race conditions between
|
55
|
-
# Backburner and beanstalkd and should probably be avoided
|
56
|
-
timeout_job_after(task.ttr > 1 ? task.ttr - 1 : task.ttr) { job_class.perform(*args) }
|
51
|
+
job_class.perform(*args)
|
57
52
|
end
|
58
53
|
task.delete
|
59
54
|
# Invoke after perform hook
|
data/lib/backburner/version.rb
CHANGED
data/lib/backburner/worker.rb
CHANGED
@@ -40,7 +40,8 @@ module Backburner
|
|
40
40
|
connection.retryable do
|
41
41
|
tube_name = expand_tube_name(queue || job_class)
|
42
42
|
serialized_data = Backburner.configuration.job_serializer_proc.call(data)
|
43
|
-
|
43
|
+
puts "Tube name #{tube_name}"
|
44
|
+
response = connection.put(tube_name, serialized_data, :pri => pri, :delay => delay, :ttr => ttr)
|
44
45
|
end
|
45
46
|
return nil unless Backburner::Hooks.invoke_hook_events(job_class, :after_enqueue, *args)
|
46
47
|
ensure
|
@@ -129,7 +130,8 @@ module Backburner
|
|
129
130
|
def work_one_job(conn = connection)
|
130
131
|
begin
|
131
132
|
job = reserve_job(conn)
|
132
|
-
rescue
|
133
|
+
rescue Exception => e
|
134
|
+
sleep(rand*3)
|
133
135
|
return
|
134
136
|
end
|
135
137
|
|
@@ -147,20 +149,19 @@ module Backburner
|
|
147
149
|
return
|
148
150
|
end
|
149
151
|
|
150
|
-
# NB: There's a slight chance here that the connection to
|
152
|
+
# NB: There's a slight chance here that the connection to allq has
|
151
153
|
# gone down between the time we reserved / processed the job and here.
|
152
|
-
num_retries = job.
|
154
|
+
num_retries = job.releases
|
153
155
|
max_job_retries = resolve_max_job_retries(job.job_class)
|
154
156
|
retry_status = "failed: attempt #{num_retries+1} of #{max_job_retries+1}"
|
155
|
-
|
156
|
-
|
157
|
-
|
158
|
-
job.retry(num_retries + 1, delay)
|
159
|
-
self.log_job_end(job.name, "#{retry_status}, retrying in #{delay}s") if job_started_at
|
160
|
-
else # retries failed, bury
|
157
|
+
retry_delay = resolve_retry_delay(job.job_class)
|
158
|
+
delay = resolve_retry_delay_proc(job.job_class).call(retry_delay, num_retries) rescue retry_delay
|
159
|
+
if num_retries + 1 > max_job_retries
|
161
160
|
job.bury
|
162
|
-
|
161
|
+
else
|
162
|
+
job.release(delay)
|
163
163
|
end
|
164
|
+
self.log_job_end(job.name, "#{retry_status}, retrying in #{delay}s") if job_started_at
|
164
165
|
|
165
166
|
handle_error(e, job.name, job.args, job)
|
166
167
|
end
|
@@ -170,7 +171,7 @@ module Backburner
|
|
170
171
|
|
171
172
|
# Return a new connection instance
|
172
173
|
def new_connection
|
173
|
-
Connection.new(Backburner.configuration.
|
174
|
+
Connection.new(Backburner.configuration.allq_url) { |conn| Backburner::Hooks.invoke_hook_events(self, :on_reconnect, conn) }
|
174
175
|
end
|
175
176
|
|
176
177
|
# Reserve a job from the watched queues
|
@@ -105,7 +105,7 @@ module Backburner
|
|
105
105
|
# Process the tube settings
|
106
106
|
# This overrides @tubes_data set by process_tube_names method. So a tube has name 'super_job:5'
|
107
107
|
# and the tube class has setting queue_jobs_limit 10, the result limit will be 10
|
108
|
-
# If the tube is known by existing
|
108
|
+
# If the tube is known by existing allq queue, but not by class - skip it
|
109
109
|
#
|
110
110
|
def process_tube_options
|
111
111
|
Backburner::Worker.known_queue_classes.each do |queue|
|
@@ -105,7 +105,7 @@ module Backburner
|
|
105
105
|
# Process the tube settings
|
106
106
|
# This overrides @tubes_data set by process_tube_names method. So a tube has name 'super_job:5:20:10'
|
107
107
|
# and the tube class has setting queue_jobs_limit 10, the result limit will be 10
|
108
|
-
# If the tube is known by existing
|
108
|
+
# If the tube is known by existing allq queue, but not by class - skip it
|
109
109
|
#
|
110
110
|
def process_tube_options
|
111
111
|
Backburner::Worker.known_queue_classes.each do |queue|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: backburner-allq
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.
|
4
|
+
version: 1.0.9
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jason Malcolm
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-09-
|
11
|
+
date: 2020-09-09 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: allq_rest
|
@@ -100,7 +100,7 @@ dependencies:
|
|
100
100
|
- - ">="
|
101
101
|
- !ruby/object:Gem::Version
|
102
102
|
version: '0'
|
103
|
-
description:
|
103
|
+
description: Allq background job processing made easy
|
104
104
|
email:
|
105
105
|
- jason@blitline.com
|
106
106
|
executables:
|
@@ -169,7 +169,7 @@ files:
|
|
169
169
|
- test/workers/simple_worker_test.rb
|
170
170
|
- test/workers/threading_worker_test.rb
|
171
171
|
- test/workers/threads_on_fork_worker_test.rb
|
172
|
-
homepage: http://github.com/
|
172
|
+
homepage: http://github.com/blitline-dev/backburner-allq
|
173
173
|
licenses:
|
174
174
|
- MIT
|
175
175
|
metadata: {}
|