resque-brokered 0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/LICENSE ADDED
@@ -0,0 +1,23 @@
1
+ Redistribution and use in source and binary forms, with or without
2
+ modification, are permitted provided that the following conditions are
3
+ met:
4
+
5
+ Redistributions of source code must retain the above copyright notice,
6
+ this list of conditions and the following disclaimer.
7
+
8
+ Redistributions in binary form must reproduce the above copyright
9
+ notice, this list of conditions and the following disclaimer in the
10
+ documentation and/or other materials provided with the distribution.
11
+
12
+
13
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
14
+ IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
15
+ TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
16
+ PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
17
+ HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
18
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
19
+ TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
20
+ PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
21
+ LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
22
+ NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
23
+ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
data/README.md ADDED
@@ -0,0 +1,44 @@
1
+ # resque-brokered
2
+
3
+ ## Introduction
4
+ Resque-brokered is an effort at allowing you to inject different
5
+ strategies for picking resque jobs off queues. It was initial created to
6
+ meet the requirement of having workers reading from multiple queues, but
7
+ only having one job from each queue being worked on at any one time.
8
+
9
+ ## Installation
10
+ If you're using bundler, then something like this will be required in
11
+ your Gemfile:
12
+
13
+ gem 'resque-brokered'
14
+
15
+ If you're not using bundler, then you should take a good hard look at
16
+ yourself, then if you're still not using bundler, then you'll need to do
17
+ something like this:
18
+
19
+ $ gem install resque-brokered
20
+
21
+ Then somewhere in your code (before you do any queueing stuff):
22
+
23
+ require 'resque'
24
+ require 'resque-brokered'
25
+
26
+ ## Using resque-brokered
27
+ Brokered queues are defined by having a two-part queue name, the
28
+ group-name, then the queue name. Thus you could have a queue system with
29
+ queues `big_process:user1`, `big_process:user2` ... `big_process:usern` then you
30
+ can have as many `big_process` workers as you need to cater to your user
31
+ base, but only handle one `big_process` per user at any one time.
32
+
33
+ To start a worker to consume from all the `big_process` queues, simply
34
+ specify the first part of the queue name like so:
35
+
36
+ QUEUES=big_process: rake resque:work
37
+
38
+ Note the colon. It's the special sauce.
39
+
40
+ Normally named queues and queues without colons work as per usual. It's
41
+ suggested that you don't use the catch-all `QUEUES=*` in conjunction
42
+ with resque-brokered as that will ignore all the limiting and
43
+ consistency work which resque-brokered brings.
44
+
@@ -0,0 +1,40 @@
1
+ require 'thread'
2
+
3
+ module Resque::Plugins::Brokered
4
+ class Broker
5
+ def initialize redis, queues
6
+ @redis = redis
7
+ @queues = queues
8
+ end
9
+
10
+ def available_queues
11
+ @redis.sdiff :queues, :active_queues
12
+ end
13
+
14
+ def filter_queues queues
15
+ queues.select {|q| queues_regex.match q}
16
+ end
17
+
18
+ def get_queue
19
+ queues = filter_queues(available_queues)
20
+ queues.shuffle.detect {|name| @redis.llen("queue:#{name}") > 0 }
21
+ end
22
+
23
+ def queues_regex
24
+ /^(?:#{@queues.join('|')}).*/
25
+ end
26
+
27
+ def pop
28
+ @redis.watch "#{@redis.namespace}:active_queues"
29
+ return nil unless queue_name = get_queue
30
+ @redis.multi
31
+ @redis.sadd :active_queues, queue_name
32
+ @redis.lpop "queue:#{queue_name}"
33
+ add, value = @redis.exec
34
+
35
+ [queue_name, Resque.decode(value)]
36
+ ensure
37
+ @redis.unwatch
38
+ end
39
+ end
40
+ end
@@ -0,0 +1,46 @@
1
+ require 'thread'
2
+
3
+ module Resque
4
+ module Plugins
5
+ module Brokered
6
+
7
+ def reserve_with_strategy
8
+ if @queues.any? {|q| q.include? ":"}
9
+ reserve_with_broker
10
+ else
11
+ default_reserve
12
+ end
13
+ end
14
+
15
+ def reserve_with_broker
16
+ interval = interval.to_i
17
+
18
+ broker = Broker.new(redis, @queues)
19
+ begin
20
+ queue, job = broker.pop
21
+ rescue ThreadError
22
+ queue, job = nil
23
+ end
24
+
25
+ log! "Found job on #{queue}"
26
+ Resque::Job.new(queue, job) if queue && job
27
+ end
28
+
29
+ def done_working_release_queue
30
+ queue = job['queue']
31
+ redis.srem :active_queues, queue
32
+ default_done_working
33
+ end
34
+
35
+ def self.included(klass)
36
+ klass.instance_eval do
37
+ alias_method :default_reserve, :reserve
38
+ alias_method :reserve, :reserve_with_strategy
39
+ alias_method :default_done_working, :done_working
40
+ alias_method :done_working, :done_working_release_queue
41
+ end
42
+ end
43
+ end
44
+ Resque::Worker.send(:include, Brokered)
45
+ end
46
+ end
@@ -0,0 +1,3 @@
1
+ require 'resque'
2
+ require 'resque/plugins/brokered'
3
+ require 'resque/plugins/brokered/broker'
metadata ADDED
@@ -0,0 +1,131 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: resque-brokered
3
+ version: !ruby/object:Gem::Version
4
+ hash: 15
5
+ prerelease:
6
+ segments:
7
+ - 0
8
+ - 2
9
+ version: "0.2"
10
+ platform: ruby
11
+ authors:
12
+ - Andrew Harvey
13
+ - James Sadler
14
+ autorequire:
15
+ bindir: bin
16
+ cert_chain: []
17
+
18
+ date: 2012-05-15 00:00:00 Z
19
+ dependencies:
20
+ - !ruby/object:Gem::Dependency
21
+ name: resque
22
+ prerelease: false
23
+ requirement: &id001 !ruby/object:Gem::Requirement
24
+ none: false
25
+ requirements:
26
+ - - ~>
27
+ - !ruby/object:Gem::Version
28
+ hash: 71
29
+ segments:
30
+ - 1
31
+ - 20
32
+ - 0
33
+ version: 1.20.0
34
+ type: :runtime
35
+ version_requirements: *id001
36
+ - !ruby/object:Gem::Dependency
37
+ name: redis
38
+ prerelease: false
39
+ requirement: &id002 !ruby/object:Gem::Requirement
40
+ none: false
41
+ requirements:
42
+ - - ">="
43
+ - !ruby/object:Gem::Version
44
+ hash: 3
45
+ segments:
46
+ - 0
47
+ version: "0"
48
+ type: :runtime
49
+ version_requirements: *id002
50
+ - !ruby/object:Gem::Dependency
51
+ name: rake
52
+ prerelease: false
53
+ requirement: &id003 !ruby/object:Gem::Requirement
54
+ none: false
55
+ requirements:
56
+ - - ">="
57
+ - !ruby/object:Gem::Version
58
+ hash: 3
59
+ segments:
60
+ - 0
61
+ version: "0"
62
+ type: :development
63
+ version_requirements: *id003
64
+ - !ruby/object:Gem::Dependency
65
+ name: rspec
66
+ prerelease: false
67
+ requirement: &id004 !ruby/object:Gem::Requirement
68
+ none: false
69
+ requirements:
70
+ - - ~>
71
+ - !ruby/object:Gem::Version
72
+ hash: 3
73
+ segments:
74
+ - 2
75
+ - 0
76
+ version: "2.0"
77
+ type: :development
78
+ version_requirements: *id004
79
+ description: You're definitely going to want to replace a lot of this
80
+ email:
81
+ - andrew@mootpointer.com
82
+ - freshtonic@gmail.com
83
+ executables: []
84
+
85
+ extensions: []
86
+
87
+ extra_rdoc_files: []
88
+
89
+ files:
90
+ - lib/resque/plugins/brokered/broker.rb
91
+ - lib/resque/plugins/brokered.rb
92
+ - lib/resque-brokered.rb
93
+ - LICENSE
94
+ - README.md
95
+ homepage: http://github.com/mootpointer/resque-brokered
96
+ licenses:
97
+ - BSD
98
+ post_install_message:
99
+ rdoc_options: []
100
+
101
+ require_paths:
102
+ - lib
103
+ required_ruby_version: !ruby/object:Gem::Requirement
104
+ none: false
105
+ requirements:
106
+ - - ">="
107
+ - !ruby/object:Gem::Version
108
+ hash: 3
109
+ segments:
110
+ - 0
111
+ version: "0"
112
+ required_rubygems_version: !ruby/object:Gem::Requirement
113
+ none: false
114
+ requirements:
115
+ - - ">="
116
+ - !ruby/object:Gem::Version
117
+ hash: 23
118
+ segments:
119
+ - 1
120
+ - 3
121
+ - 6
122
+ version: 1.3.6
123
+ requirements: []
124
+
125
+ rubyforge_project:
126
+ rubygems_version: 1.8.17
127
+ signing_key:
128
+ specification_version: 3
129
+ summary: Resque plugin to add some broker logic to picking jobs up off queues
130
+ test_files: []
131
+