quebert 3.0.2 → 3.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.ruby-version +1 -1
- data/.travis.yml +1 -1
- data/README.md +42 -21
- data/lib/quebert/controller/beanstalk.rb +2 -2
- data/lib/quebert/version.rb +1 -1
- metadata +3 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 107451b4a0b3a15d3e42285f65c1c53be6700c52
|
4
|
+
data.tar.gz: ce2f9fd96496d6701eadad9abd94acc33a7fa6d4
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 06fe749f66920ee7ebb436b72d82c3d0f2f68049cf5dfbba10c9ca3e7ac76eebfb60644cab07634d23fb9d8863b9b5c16880c45b3191f91a6ee187604fc2adea
|
7
|
+
data.tar.gz: 3523257233d8d1f7732be791ae48a6bb7f359ba063231908c4c5957e9411333b48ea32ecd70daec03a0dd915847b087abf4247f21b33c78fadab3575e44e5b84
|
data/.ruby-version
CHANGED
@@ -1 +1 @@
|
|
1
|
-
2.
|
1
|
+
2.2.5
|
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -1,35 +1,39 @@
|
|
1
1
|
# Quebert
|
2
2
|
|
3
|
-
[](https://travis-ci.org/polleverywhere/quebert) [](https://travis-ci.org/polleverywhere/quebert) [](https://codeclimate.com/github/polleverywhere/quebert)
|
4
4
|
|
5
|
-
|
5
|
+
Quebert is a ruby background worker library that works with the very fast and simple [beanstalkd](http://kr.github.io/beanstalkd/) deamon.
|
6
6
|
|
7
|
-
|
7
|
+
## Why Quebert?
|
8
8
|
|
9
|
-
Because it has really low latency. Other Ruby queuing frameworks, like DJ or Resque, have to poll their queue servers periodicly. You could think of it as a "pull" queue. Quebert is a "push" queue. It maintains a persistent connection with beanstalkd and when is enqueud, its instantly pushed to the workers and executed.
|
9
|
+
Because it has really low latency. Other Ruby queuing frameworks, like [DJ](https://github.com/collectiveidea/delayed_job) or [Resque](https://github.com/resque/resque), have to poll their queue servers periodicly. You could think of it as a "pull" queue. Quebert is a "push" queue. It maintains a persistent connection with beanstalkd and when is enqueud, its instantly pushed to the workers and executed.
|
10
10
|
|
11
|
-
|
11
|
+
[Sidekiq](http://sidekiq.org) uses Redis's "push" primitives so it has low latency, but it doesn't support class reloading in a development environment. Sidekiq is also threaded, which means there are no guarantees of reliability when running non-threadsafe code.
|
12
12
|
|
13
|
-
|
13
|
+
[Backburner](https://github.com/nesquena/backburner) is very similar to Quebert. It offers more options for concurrency (threading, forking, etc.) than queubert but lacks pluggable back-ends, which means you'll be stubbing and mocking async calls.
|
14
14
|
|
15
|
-
|
15
|
+
## Who uses it?
|
16
16
|
|
17
|
-
|
18
|
-
* Rails/ActiveRecord integration similar to async_observer
|
19
|
-
* Pluggable exception handling (for Hoptoad integration)
|
20
|
-
* Run workers with pid, log, and config files. These do not daemonize (do it yourself punk!)
|
21
|
-
* Provide custom hooks to be called before, after & around jobs are run
|
17
|
+
Quebert is a serious project. Its used in a production environment at [Poll Everywhere](https://www.polleverywhere.com/) to handle everything from SMS message processing to account downgrades.
|
22
18
|
|
23
|
-
|
19
|
+
## Features
|
20
|
+
|
21
|
+
* **Multiple back-ends** (InProcess, Sync, and Beanstalk)
|
22
|
+
* **Rails/ActiveRecord integration** similar to async_observer
|
23
|
+
* **Pluggable exception handling** (for Hoptoad integration)
|
24
|
+
* **Run workers with pid, log, and config files**. These do not daemonize (do it yourself punk!)
|
25
|
+
* **Custom hooks** that may be called before, after & around jobs are run
|
26
|
+
|
27
|
+
Some features that are currently *missing* that I will soon add include:
|
24
28
|
|
25
29
|
* Rails plugin support (The AR integrations have to be done manually today)
|
26
30
|
* Auto-detecting serializers. Enhanced ClassRegistry to more efficiently look up serializers for objects.
|
27
31
|
|
28
|
-
|
32
|
+
## How to use
|
29
33
|
|
30
34
|
There are two ways to enqueue jobs with Quebert: through the Job itself, provided you set a default back-end for the job, or put it on the backend.
|
31
35
|
|
32
|
-
|
36
|
+
### Jobs
|
33
37
|
|
34
38
|
Quebert includes a Job class so you can implement how you want certain types of Jobs performed.
|
35
39
|
|
@@ -52,7 +56,10 @@ Quebert.backend.put WackyMathWizard.new(1, 2, 3)
|
|
52
56
|
Or drop it in right from the job:
|
53
57
|
|
54
58
|
```ruby
|
59
|
+
# Run job right away!
|
55
60
|
WackyMathWizard.new(4, 5, 6).enqueue
|
61
|
+
# Run a lower priority job in 10 seconds for a max of 120 seconds
|
62
|
+
WackyMathWizard.new(10, 10, 10).enqueue(ttr: 120, priority: 100, delay: 10)
|
56
63
|
```
|
57
64
|
|
58
65
|
Then perform the jobs!
|
@@ -60,9 +67,10 @@ Then perform the jobs!
|
|
60
67
|
```ruby
|
61
68
|
Quebert.backend.reserve.perform # => 6
|
62
69
|
Quebert.backend.reserve.perform # => 15
|
70
|
+
Quebert.backend.reserve.perform # => 30
|
63
71
|
```
|
64
72
|
|
65
|
-
|
73
|
+
### Rails integration
|
66
74
|
|
67
75
|
config/quebert.yml:
|
68
76
|
|
@@ -83,7 +91,7 @@ Quebert.config.from_hash(Rails.application.config.quebert)
|
|
83
91
|
Quebert.config.logger = Rails.logger
|
84
92
|
```
|
85
93
|
|
86
|
-
|
94
|
+
### Job & Worker hooks
|
87
95
|
|
88
96
|
Quebert has support for providing custom hooks to be called before, after & around your jobs are being run.
|
89
97
|
A common example is making sure that any active ActiveRecord database connections are put back on the connection pool after a job is done:
|
@@ -104,7 +112,7 @@ Quebert.config.around_job do |job|
|
|
104
112
|
end
|
105
113
|
```
|
106
114
|
|
107
|
-
|
115
|
+
### Async sender
|
108
116
|
|
109
117
|
Take any ol' class and include the Quebert::AsyncSender.
|
110
118
|
|
@@ -162,13 +170,13 @@ Quebert.backend.reserve.perform # => "waazup Coraline!"
|
|
162
170
|
|
163
171
|
* Only basic data types are included for serialization. Serializers may be customized to include support for different types.
|
164
172
|
|
165
|
-
|
173
|
+
### Backends
|
166
174
|
|
167
175
|
* Beanstalk: Enqueue jobs in a beanstalkd service. The workers run in a separate process. Typically used in production environments.
|
168
176
|
* Sync: Perform jobs immediately upon enqueuing. Typically used in testing environments.
|
169
177
|
* InProcess: Enqueue jobs in an in-memory array. A worker will need to reserve a job to perform.
|
170
178
|
|
171
|
-
|
179
|
+
### Multiple queues
|
172
180
|
|
173
181
|
To start a worker pointed at a non-default queue (e.g., a Quebert "tube"), start the process with `-q`:
|
174
182
|
|
@@ -190,16 +198,29 @@ class FooJob < Quebert::Job
|
|
190
198
|
end
|
191
199
|
```
|
192
200
|
|
193
|
-
|
201
|
+
### Setting job defaults
|
202
|
+
|
203
|
+
A `Quebert::Job` is a Plain Ol' Ruby Object. The defaults of a job, including its `ttr`, `queue_name`, and `delay` may be overridden in a super class as follows:
|
194
204
|
|
195
205
|
```ruby
|
206
|
+
# Assuming you're in Rails or using ActiveSupport
|
196
207
|
class FooJob < Quebert::Job
|
197
208
|
def ttr
|
198
209
|
5.minutes
|
199
210
|
end
|
200
211
|
|
212
|
+
def delay
|
213
|
+
30.seconds
|
214
|
+
end
|
215
|
+
|
216
|
+
def queue_name
|
217
|
+
"long-running-delayed-jobs"
|
218
|
+
end
|
219
|
+
|
201
220
|
def perform(args)
|
202
221
|
# ...
|
203
222
|
end
|
204
223
|
end
|
205
224
|
```
|
225
|
+
|
226
|
+
Take a look at the [`Quebert::Job` class](https://github.com/polleverywhere/quebert/blob/master/lib/quebert/job.rb) code for more details on methods you may ovveride.
|
@@ -22,7 +22,7 @@ module Quebert
|
|
22
22
|
end
|
23
23
|
|
24
24
|
def perform
|
25
|
-
logger.error(job) { "Performing with args #{job.args.inspect}" }
|
25
|
+
logger.error(job) { "Performing #{job.class.name} with args #{job.args.inspect}" }
|
26
26
|
logger.error(job) { "Beanstalk Job Stats: #{beanstalk_job.stats.inspect}" }
|
27
27
|
|
28
28
|
result = false
|
@@ -31,7 +31,7 @@ module Quebert
|
|
31
31
|
beanstalk_job.delete
|
32
32
|
end
|
33
33
|
|
34
|
-
logger.
|
34
|
+
logger.info(job) { "Completed in #{(time*1000*1000).to_i/1000.to_f} ms\n" }
|
35
35
|
result
|
36
36
|
rescue Job::Delete
|
37
37
|
logger.info(job) { "Deleting job" }
|
data/lib/quebert/version.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: quebert
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 3.0.
|
4
|
+
version: 3.0.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Brad Gessler
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: bin
|
12
12
|
cert_chain: []
|
13
|
-
date:
|
13
|
+
date: 2016-11-29 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: json
|
@@ -131,7 +131,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
131
131
|
version: '0'
|
132
132
|
requirements: []
|
133
133
|
rubyforge_project: quebert
|
134
|
-
rubygems_version: 2.
|
134
|
+
rubygems_version: 2.4.5.1
|
135
135
|
signing_key:
|
136
136
|
specification_version: 4
|
137
137
|
summary: A worker queue framework built around beanstalkd
|
@@ -149,4 +149,3 @@ test_files:
|
|
149
149
|
- spec/support/jobs.rb
|
150
150
|
- spec/support_spec.rb
|
151
151
|
- spec/worker_spec.rb
|
152
|
-
has_rdoc:
|