que 0.7.1 → 0.7.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -15
- data/CHANGELOG.md +4 -0
- data/README.md +58 -51
- data/docs/using_sequel.md +1 -1
- data/lib/que/version.rb +1 -1
- data/lib/que/worker.rb +11 -7
- data/spec/adapters/active_record_spec.rb +2 -0
- metadata +34 -32
checksums.yaml
CHANGED
@@ -1,15 +1,7 @@
|
|
1
|
-
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
metadata.gz: !binary |-
|
9
|
-
OGVmMTc2MzNlYjkyNTQwYzhiNDZmMDJlOTdhMDViOGFhYzYzMzgzMGQ0MzY5
|
10
|
-
YzY1OTIzYjg5YmZjZDcyNjJkOWM2ZmY4MWFjNjQ0YmYzODQ4NmM5Y2I0Yjhi
|
11
|
-
Y2Q0YTkwYTk4NDlmNDM1Y2E5OWJlYWE1OWE2MDY5YzIxYTQ0ZWE=
|
12
|
-
data.tar.gz: !binary |-
|
13
|
-
NzlmZWZmY2RjZjc0OGJkNzJhODA5NWY5ZmZkYzY0M2VlMTk5M2NkODQyMmQ1
|
14
|
-
OWYzNjExYmMxMTdmYzQ5NGQ1NTBlNzY3NTA2MjQ5ZDQwZDM3M2FmMTY2ODUz
|
15
|
-
OTVjZjA0ZGVhM2VmZTFkMTViMmI2M2ZmYWM3M2U0N2Y4N2QyNmU=
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 87a3655974bbd2d2524ccfc8527bbb949d1a1e4b
|
4
|
+
data.tar.gz: 183e2c344f8e25c3961def7cf9f2f1a6e86235d7
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: fe9145ad097ba517f9a1978b8061bb263df0b83c1295c7673da02d757642b08371295bef139ad910e1c00dbd66edd2949dc3a3b26a097c8b50a3d91011cf0efb
|
7
|
+
data.tar.gz: 0e048d4c3c6cfdd135c0eb353c74677409eb2e9d83c7137652bf049ab3a5f305526f10e230ef0209607f33d418f3ddfa77eb8e7414e494844e2885e94de10710
|
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -4,16 +4,16 @@
|
|
4
4
|
|
5
5
|
Que is a queue for Ruby and PostgreSQL that manages jobs using [advisory locks](http://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS), which gives it several advantages over other RDBMS-backed queues:
|
6
6
|
|
7
|
-
* **Concurrency** - Workers don't block each other when trying to lock jobs, as often occurs with "SELECT FOR UPDATE"-style locking. This allows for very high throughput with a large number of workers.
|
8
|
-
* **Efficiency** - Locks are held in memory, so locking a job doesn't incur a disk write. These first two points are what limit performance with other queues - all workers trying to lock jobs have to wait behind one that's persisting its UPDATE on a locked_at column to disk (and the disks of however many other servers your database is synchronously replicating to). Under heavy load, Que's bottleneck is CPU, not I/O.
|
9
|
-
* **Safety** - If a Ruby process dies, the jobs it's working won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
|
7
|
+
* **Concurrency** - Workers don't block each other when trying to lock jobs, as often occurs with "SELECT FOR UPDATE"-style locking. This allows for very high throughput with a large number of workers.
|
8
|
+
* **Efficiency** - Locks are held in memory, so locking a job doesn't incur a disk write. These first two points are what limit performance with other queues - all workers trying to lock jobs have to wait behind one that's persisting its UPDATE on a locked_at column to disk (and the disks of however many other servers your database is synchronously replicating to). Under heavy load, Que's bottleneck is CPU, not I/O.
|
9
|
+
* **Safety** - If a Ruby process dies, the jobs it's working won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
|
10
10
|
|
11
11
|
Additionally, there are the general benefits of storing jobs in Postgres, alongside the rest of your data, rather than in Redis or a dedicated queue:
|
12
12
|
|
13
|
-
* **Transactional Control** - Queue a job along with other changes to your database, and it'll commit or rollback with everything else. If you're using ActiveRecord or Sequel, Que can piggyback on their connections, so setup is simple and jobs are protected by the transactions you're already using.
|
14
|
-
* **Atomic Backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), there's no risk of jobs falling through the cracks during a recovery.
|
15
|
-
* **Fewer Dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
|
16
|
-
* **Security** - Postgres' support for SSL connections keeps your data safe in transport, for added protection when you're running workers on cloud platforms that you can't completely control.
|
13
|
+
* **Transactional Control** - Queue a job along with other changes to your database, and it'll commit or rollback with everything else. If you're using ActiveRecord or Sequel, Que can piggyback on their connections, so setup is simple and jobs are protected by the transactions you're already using.
|
14
|
+
* **Atomic Backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), there's no risk of jobs falling through the cracks during a recovery.
|
15
|
+
* **Fewer Dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
|
16
|
+
* **Security** - Postgres' support for SSL connections keeps your data safe in transport, for added protection when you're running workers on cloud platforms that you can't completely control.
|
17
17
|
|
18
18
|
Que's primary goal is reliability. You should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the docs on [how to write a reliable job](https://github.com/chanks/que/blob/master/docs/writing_reliable_jobs.md)).
|
19
19
|
|
@@ -21,10 +21,9 @@ Que's secondary goal is performance. It won't be able to match the speed or thro
|
|
21
21
|
|
22
22
|
Que also includes a worker pool, so that multiple threads can process jobs in the same process. It can even do this in the background of your web process - if you're running on Heroku, for example, you don't need to run a separate worker dyno.
|
23
23
|
|
24
|
-
*Please keep an eye out for problems when running Que in production. It's still new compared to other RDBMS-backed queues, and there may be issues that haven't been ironed out yet. Bug reports are welcome.*
|
25
|
-
|
26
24
|
Que is tested on Ruby 2.0, Rubinius and JRuby (with the `jruby-pg` gem, which is [not yet functional with ActiveRecord](https://github.com/chanks/que/issues/4#issuecomment-29561356)). It requires Postgres 9.2+ for the JSON datatype.
|
27
25
|
|
26
|
+
|
28
27
|
## Installation
|
29
28
|
|
30
29
|
Add this line to your application's Gemfile:
|
@@ -39,70 +38,78 @@ Or install it yourself as:
|
|
39
38
|
|
40
39
|
$ gem install que
|
41
40
|
|
41
|
+
|
42
42
|
## Usage
|
43
43
|
|
44
44
|
The following assumes you're using Rails 4.0 and ActiveRecord. *Que hasn't been tested with versions of Rails before 4.0, and may or may not work with them.* See the [/docs directory](https://github.com/chanks/que/blob/master/docs) for instructions on using Que [outside of Rails](https://github.com/chanks/que/blob/master/docs/advanced_setup.md), and with [Sequel](https://github.com/chanks/que/blob/master/docs/using_sequel.md) or [no ORM](https://github.com/chanks/que/blob/master/docs/using_plain_connections.md), among other things.
|
45
45
|
|
46
46
|
First, generate and run a migration for the job table.
|
47
47
|
|
48
|
-
rails generate que:install
|
49
|
-
rake db:migrate
|
48
|
+
$ bin/rails generate que:install
|
49
|
+
$ bin/rake db:migrate
|
50
50
|
|
51
51
|
Create a class for each type of job you want to run:
|
52
52
|
|
53
|
-
# app/jobs/charge_credit_card.rb
|
54
|
-
class ChargeCreditCard < Que::Job
|
55
|
-
# Default settings for this job. These are optional - without them, jobs
|
56
|
-
# will default to priority 100 and run immediately.
|
57
|
-
@priority = 10
|
58
|
-
@run_at = proc { 1.minute.from_now }
|
59
|
-
|
60
|
-
def run(user_id, options)
|
61
|
-
# Do stuff.
|
62
|
-
user = User[user_id]
|
63
|
-
card = CreditCard[options[:credit_card_id]]
|
64
|
-
|
65
|
-
ActiveRecord::Base.transaction do
|
66
|
-
# Write any changes you'd like to the database.
|
67
|
-
user.update_attributes :charged_at => Time.now
|
68
|
-
|
69
|
-
# It's best to destroy the job in the same transaction as any other
|
70
|
-
# changes you make. Que will destroy the job for you after the run
|
71
|
-
# method if you don't do it yourself, but if your job writes to the
|
72
|
-
# DB but doesn't destroy the job in the same transaction, it's
|
73
|
-
# possible that the job could be repeated in the event of a crash.
|
74
|
-
destroy
|
75
|
-
end
|
76
|
-
end
|
77
|
-
end
|
78
53
|
|
79
|
-
|
54
|
+
``` ruby
|
55
|
+
# app/jobs/charge_credit_card.rb
|
56
|
+
class ChargeCreditCard < Que::Job
|
57
|
+
# Default settings for this job. These are optional - without them, jobs
|
58
|
+
# will default to priority 100 and run immediately.
|
59
|
+
@priority = 10
|
60
|
+
@run_at = proc { 1.minute.from_now }
|
61
|
+
|
62
|
+
def run(user_id, options)
|
63
|
+
# Do stuff.
|
64
|
+
user = User[user_id]
|
65
|
+
card = CreditCard[options[:credit_card_id]]
|
80
66
|
|
81
67
|
ActiveRecord::Base.transaction do
|
82
|
-
#
|
83
|
-
|
84
|
-
|
68
|
+
# Write any changes you'd like to the database.
|
69
|
+
user.update_attributes :charged_at => Time.now
|
70
|
+
|
71
|
+
# It's best to destroy the job in the same transaction as any other
|
72
|
+
# changes you make. Que will destroy the job for you after the run
|
73
|
+
# method if you don't do it yourself, but if your job writes to the
|
74
|
+
# DB but doesn't destroy the job in the same transaction, it's
|
75
|
+
# possible that the job could be repeated in the event of a crash.
|
76
|
+
destroy
|
85
77
|
end
|
78
|
+
end
|
79
|
+
end
|
80
|
+
```
|
81
|
+
|
82
|
+
Queue your job. Again, it's best to do this in a transaction with other changes you're making. Also note that any arguments you pass will be serialized to JSON and back again, so stick to simple types (strings, integers, floats, hashes, and arrays).
|
83
|
+
|
84
|
+
``` ruby
|
85
|
+
ActiveRecord::Base.transaction do
|
86
|
+
# Persist credit card information
|
87
|
+
card = CreditCard.create(params[:credit_card])
|
88
|
+
ChargeCreditCard.enqueue(current_user.id, :credit_card_id => card.id)
|
89
|
+
end
|
90
|
+
```
|
86
91
|
|
87
92
|
You can also add options to run the job after a specific time, or with a specific priority:
|
88
93
|
|
89
|
-
|
90
|
-
|
94
|
+
``` ruby
|
95
|
+
# The default priority is 100, and a lower number means a higher priority. 5 would be very important.
|
96
|
+
ChargeCreditCard.enqueue current_user.id, :credit_card_id => card.id, :run_at => 1.day.from_now, :priority => 5
|
97
|
+
```
|
91
98
|
|
92
99
|
To determine what happens when a job is queued, you can set Que's mode in your application configuration. There are a few options for the mode:
|
93
100
|
|
94
|
-
* `config.que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `rails console`.
|
95
|
-
* `config.que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread. This is the default when running `rails server`. See the docs for [more information on managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md).
|
96
|
-
* `config.que.mode = :sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.enqueue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
|
101
|
+
* `config.que.mode = :off` - In this mode, queueing a job will simply insert it into the database - the current process will make no effort to run it. You should use this if you want to use a dedicated process to work tasks (there's a rake task to do this, see below). This is the default when running `bin/rails console`.
|
102
|
+
* `config.que.mode = :async` - In this mode, a pool of background workers is spun up, each running in their own thread. This is the default when running `bin/rails server`. See the docs for [more information on managing workers](https://github.com/chanks/que/blob/master/docs/managing_workers.md).
|
103
|
+
* `config.que.mode = :sync` - In this mode, any jobs you queue will be run in the same thread, synchronously (that is, `MyJob.enqueue` runs the job and won't return until it's completed). This makes your application's behavior easier to test, so it's the default in the test environment.
|
97
104
|
|
98
105
|
## Contributing
|
99
106
|
|
100
|
-
1. Fork it
|
101
|
-
2. Create your feature branch (`git checkout -b my-new-feature`)
|
102
|
-
3. Commit your changes (`git commit -am 'Add some feature'`)
|
103
|
-
4. Push to the branch (`git push origin my-new-feature`)
|
104
|
-
5. Create new Pull Request
|
105
|
-
|
107
|
+
1. Fork it
|
108
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
109
|
+
3. Commit your changes (`git commit -am 'Add some feature'`)
|
110
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
111
|
+
5. Create new Pull Request
|
112
|
+
|
106
113
|
A note on running specs - Que's worker system is multithreaded and therefore prone to race conditions (especially on Rubinius). As such, if you've touched that code, a single spec run passing isn't a guarantee that any changes you've made haven't introduced bugs. One thing I like to do before pushing changes is rerun the specs many times and watching for hangs. You can do this from the command line with something like:
|
107
114
|
|
108
115
|
for i in {1..1000}; do rspec -b --seed $i; done
|
data/docs/using_sequel.md
CHANGED
data/lib/que/version.rb
CHANGED
data/lib/que/worker.rb
CHANGED
@@ -110,12 +110,6 @@ module Que
|
|
110
110
|
# a worker, and make sure to wake up the wrangler when @wake_interval is
|
111
111
|
# changed in Que.wake_interval= below.
|
112
112
|
@wake_interval = 5
|
113
|
-
@wrangler = Thread.new do
|
114
|
-
loop do
|
115
|
-
sleep *@wake_interval
|
116
|
-
wake! if @wake_interval
|
117
|
-
end
|
118
|
-
end
|
119
113
|
|
120
114
|
class << self
|
121
115
|
attr_reader :mode, :wake_interval
|
@@ -136,6 +130,7 @@ module Que
|
|
136
130
|
def worker_count=(count)
|
137
131
|
set_mode(count > 0 ? :async : :off)
|
138
132
|
set_worker_count(count)
|
133
|
+
wrangler # Make sure the wrangler thread has been instantiated.
|
139
134
|
end
|
140
135
|
|
141
136
|
def worker_count
|
@@ -144,7 +139,7 @@ module Que
|
|
144
139
|
|
145
140
|
def wake_interval=(interval)
|
146
141
|
@wake_interval = interval
|
147
|
-
|
142
|
+
wrangler.wakeup
|
148
143
|
end
|
149
144
|
|
150
145
|
def wake!
|
@@ -157,6 +152,15 @@ module Que
|
|
157
152
|
|
158
153
|
private
|
159
154
|
|
155
|
+
def wrangler
|
156
|
+
@wrangler ||= Thread.new do
|
157
|
+
loop do
|
158
|
+
sleep *@wake_interval
|
159
|
+
wake! if @wake_interval
|
160
|
+
end
|
161
|
+
end
|
162
|
+
end
|
163
|
+
|
160
164
|
def set_mode(mode)
|
161
165
|
if mode != @mode
|
162
166
|
Que.log :event => 'mode_change', :value => mode.to_s
|
@@ -39,6 +39,8 @@ unless defined?(RUBY_ENGINE) && RUBY_ENGINE == 'jruby'
|
|
39
39
|
|
40
40
|
it "should recreate the prepared statements" do
|
41
41
|
expect { Que::Job.enqueue }.not_to raise_error
|
42
|
+
|
43
|
+
DB[:que_jobs].count.should == 2
|
42
44
|
end
|
43
45
|
|
44
46
|
it "should work properly even in a transaction" do
|
metadata
CHANGED
@@ -1,36 +1,36 @@
|
|
1
|
-
--- !ruby/object:Gem::Specification
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
2
|
name: que
|
3
|
-
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.7.
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.7.2
|
5
5
|
platform: ruby
|
6
|
-
authors:
|
6
|
+
authors:
|
7
7
|
- Chris Hanks
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
|
12
|
-
|
13
|
-
|
11
|
+
|
12
|
+
date: 2014-05-18 00:00:00 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
14
15
|
name: bundler
|
15
|
-
requirement: !ruby/object:Gem::Requirement
|
16
|
-
requirements:
|
17
|
-
- - ~>
|
18
|
-
- !ruby/object:Gem::Version
|
19
|
-
version: '1.3'
|
20
|
-
type: :development
|
21
16
|
prerelease: false
|
22
|
-
|
23
|
-
requirements:
|
17
|
+
requirement: &id001 !ruby/object:Gem::Requirement
|
18
|
+
requirements:
|
24
19
|
- - ~>
|
25
|
-
- !ruby/object:Gem::Version
|
26
|
-
version:
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: "1.3"
|
22
|
+
type: :development
|
23
|
+
version_requirements: *id001
|
27
24
|
description: A job queue that uses PostgreSQL's advisory locks for speed and reliability.
|
28
|
-
email:
|
25
|
+
email:
|
29
26
|
- christopher.m.hanks@gmail.com
|
30
27
|
executables: []
|
28
|
+
|
31
29
|
extensions: []
|
30
|
+
|
32
31
|
extra_rdoc_files: []
|
33
|
-
|
32
|
+
|
33
|
+
files:
|
34
34
|
- .gitignore
|
35
35
|
- .rspec
|
36
36
|
- .travis.yml
|
@@ -103,30 +103,32 @@ files:
|
|
103
103
|
- tasks/rspec.rb
|
104
104
|
- tasks/safe_shutdown.rb
|
105
105
|
homepage: https://github.com/chanks/que
|
106
|
-
licenses:
|
106
|
+
licenses:
|
107
107
|
- MIT
|
108
108
|
metadata: {}
|
109
|
+
|
109
110
|
post_install_message:
|
110
111
|
rdoc_options: []
|
111
|
-
|
112
|
+
|
113
|
+
require_paths:
|
112
114
|
- lib
|
113
|
-
required_ruby_version: !ruby/object:Gem::Requirement
|
114
|
-
requirements:
|
115
|
-
-
|
116
|
-
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
|
121
|
-
|
122
|
-
version: '0'
|
115
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
116
|
+
requirements:
|
117
|
+
- &id002
|
118
|
+
- ">="
|
119
|
+
- !ruby/object:Gem::Version
|
120
|
+
version: "0"
|
121
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
122
|
+
requirements:
|
123
|
+
- *id002
|
123
124
|
requirements: []
|
125
|
+
|
124
126
|
rubyforge_project:
|
125
127
|
rubygems_version: 2.2.2
|
126
128
|
signing_key:
|
127
129
|
specification_version: 4
|
128
130
|
summary: A PostgreSQL-based Job Queue
|
129
|
-
test_files:
|
131
|
+
test_files:
|
130
132
|
- spec/adapters/active_record_spec.rb
|
131
133
|
- spec/adapters/connection_pool_spec.rb
|
132
134
|
- spec/adapters/pg_spec.rb
|