woodhouse 0.1.5 → 1.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.travis.yml +3 -2
- data/README.markdown +4 -146
- data/lib/generators/woodhouse_generator.rb +13 -0
- data/lib/woodhouse/dispatchers.rb +1 -0
- data/lib/woodhouse/dispatchers/bunny_dispatcher.rb +4 -2
- data/lib/woodhouse/dispatchers/common_amqp_dispatcher.rb +1 -2
- data/lib/woodhouse/dispatchers/file_dispatcher.rb +56 -0
- data/lib/woodhouse/extensions/progress.rb +2 -2
- data/lib/woodhouse/node_configuration.rb +28 -0
- data/lib/woodhouse/process.rb +19 -9
- data/lib/woodhouse/rails.rb +13 -11
- data/lib/woodhouse/runners.rb +1 -0
- data/lib/woodhouse/runners/file_runner.rb +80 -0
- data/lib/woodhouse/scheduler.rb +1 -1
- data/lib/woodhouse/version.rb +1 -1
- data/woodhouse.gemspec +6 -4
- metadata +61 -56
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 82eac8a615e21e61744337db2453b756cfa2168c
|
4
|
+
data.tar.gz: b538599fd8757f81113dafa997ddb9e930427c2b
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 391c9243d9d1b81794272ebeccecf4a33dadd56c589cbe588aa7046da27e8fcc4681fe5b326166533d8024a85c4c86fab5139490ca508240c9c6e828eb888e92
|
7
|
+
data.tar.gz: bc0568ef140f9911673201eadf1a0c8601e69e81e6358f63a0b2880d46519a76b379a9af1e2c1c1510871007efb7a36e4b29896ad19e5a53ebe01219400d0af7
|
data/.travis.yml
CHANGED
data/README.markdown
CHANGED
@@ -2,121 +2,15 @@
|
|
2
2
|
|
3
3
|
[<img src="https://secure.travis-ci.org/mboeh/woodhouse.png?branch=master" alt="Build Status" />](http://travis-ci.org/mboeh/woodhouse)
|
4
4
|
|
5
|
-
|
5
|
+
A RabbitMQ-based background worker system for Ruby designed to make managing heterogenous tasks relatively easy.
|
6
6
|
|
7
7
|
The use case for Woodhouse is for reliable and sane performance in situations where jobs on a single queue may vary significantly
|
8
8
|
in length. The goal is to permit large numbers of quick jobs to be serviced even when many slow jobs are in the queue. A secondary
|
9
9
|
goal is to provide a sane way for jobs on a given queue to be given special priority or dispatched to a server more suited to them.
|
10
10
|
|
11
|
-
Woodhouse
|
11
|
+
Woodhouse 1.0, located in the 1-0-stable branch, is production-ready and stable for Ruby 1.9.
|
12
12
|
|
13
|
-
|
14
|
-
|
15
|
-
### Rails
|
16
|
-
|
17
|
-
Add
|
18
|
-
|
19
|
-
gem 'woodhouse', github: 'mboeh/woodhouse'
|
20
|
-
|
21
|
-
to your Gemfile.
|
22
|
-
|
23
|
-
Run
|
24
|
-
|
25
|
-
% rails generate woodhouse
|
26
|
-
|
27
|
-
to create script/woodhouse and config/initializers/woodhouse.rb.
|
28
|
-
|
29
|
-
### Basic Usage
|
30
|
-
|
31
|
-
The simplest way to set up a worker class is to include Woodhouse::Worker and define public methods.
|
32
|
-
|
33
|
-
class IsisWorker
|
34
|
-
include Woodhouse::Worker
|
35
|
-
|
36
|
-
def pam_gossip(job)
|
37
|
-
puts "Pam gossips about #{job[:who]}."
|
38
|
-
end
|
39
|
-
|
40
|
-
def sterling_insult(job)
|
41
|
-
puts "Sterling insults #{job[:who]}."
|
42
|
-
end
|
43
|
-
end
|
44
|
-
|
45
|
-
Jobs are dispatched asynchronously to a worker by adding `async_` to the method name:
|
46
|
-
|
47
|
-
IsisWorker.async_pam_gossip :who => "Cyril"
|
48
|
-
|
49
|
-
Woodhouse jobs always take a hash of arguments. The worker receives a Woodhouse::Job, which acts like a hash
|
50
|
-
but also supplies additional functionality.
|
51
|
-
|
52
|
-
### Dispatchers
|
53
|
-
|
54
|
-
The dispatcher used for sending out jobs can be set in the Woodhouse config block:
|
55
|
-
|
56
|
-
Woodhouse.configure do |woodhouse|
|
57
|
-
woodhouse.dispatcher_type = :local # :local_pool | :amqp | :test
|
58
|
-
end
|
59
|
-
|
60
|
-
Calling the `async` version of a job method sends it to the currently configured dispatcher. The default dispatcher
|
61
|
-
type is `:local`, which simply executes the job synchronously (although still passing it through middleware; see below).
|
62
|
-
|
63
|
-
If you are running tests and you want to be able to test that your code is dispatching Woodhouse jobs (without running
|
64
|
-
them), use the `:test` dispatcher and the dispatcher will simply accumulate jobs (of class Woodhouse::Job):
|
65
|
-
|
66
|
-
IsisWorker.async_pam_gossip :who => "Cyril"
|
67
|
-
that_job = Woodhouse.dispatcher.jobs.last
|
68
|
-
that_job.worker_class_name # ==> "IsisWorker"
|
69
|
-
that_job.job_method # ==> "pam_gossip"
|
70
|
-
that_job.arguments[:who] # ==> "Cyril"
|
71
|
-
|
72
|
-
If you want `girl_friday` style in-process threaded backgrounding, you can get that by selecting the `:local_pool`
|
73
|
-
dispatcher.
|
74
|
-
|
75
|
-
Finally, if you want to run your jobs in a background process, you'll need to set up the `:amqp` dispatcher. This will
|
76
|
-
use either the Hot Bunnies library (on JRuby) or the Bunny library (on all other Ruby engines). Bunny is suitable for
|
77
|
-
dispatch but can be a little bit CPU-hungry in the background process. Hot Bunnies works great for both. You don't have
|
78
|
-
to use the same Ruby version for your background process as for the dispatching application -- we use Woodhouse in production
|
79
|
-
with a JRuby background process and MRI frontend processes.
|
80
|
-
|
81
|
-
You'll also need to have RabbitMQ running. If it's running with the default (open) permissions on the local server, you don't
|
82
|
-
need to configure it at all. Otherwise, you'll have to set the server connection info. You have two options for this. On Rails,
|
83
|
-
you can create a config/woodhouse.yml file, formatted similar to config/database.yml:
|
84
|
-
|
85
|
-
production:
|
86
|
-
host: myrabbitmq.server.local
|
87
|
-
vhost: /some-vhost
|
88
|
-
|
89
|
-
(The parameters accepted here are the same used for Bunny.connect; I promise to document them here soon.)
|
90
|
-
|
91
|
-
Otherwise, you can do it in the Woodhouse config block:
|
92
|
-
|
93
|
-
Woodhouse.configure do |woodhouse|
|
94
|
-
woodhouse.server_info = { :host => "myrabbitmq.server.local" }
|
95
|
-
end
|
96
|
-
|
97
|
-
### Running The Background Process
|
98
|
-
|
99
|
-
All you have to do is run `script/woodhouse`. It'll load your Rails environment and start the server process. It responds to QUIT
|
100
|
-
and INT signals correctly; I'm working on seeing if I can get it to restart worker processes with HUP and to dump/load the current
|
101
|
-
layout with USR1/USR2.
|
102
|
-
|
103
|
-
`script/woodhouse` logs job execution and results to `log/woodhouse.log`.
|
104
|
-
|
105
|
-
### Performance Errata
|
106
|
-
|
107
|
-
If you're using JRuby with a large application, I've found that the JVM's permanent generation can get exhausted. If you have
|
108
|
-
plenty of heap but still get GC overhead errors, try bumping up the PermGen by including this on the JRuby command line:
|
109
|
-
|
110
|
-
-J-XX:MaxPermSize=128m
|
111
|
-
|
112
|
-
Performance will generally be better with the `--server` flag:
|
113
|
-
|
114
|
-
--server
|
115
|
-
|
116
|
-
A lot of the jobs I run tend to allocate and dispose a lot of memory very quickly, and Woodhouse will be a long-running process.
|
117
|
-
I've gotten good results from enabling aggressive heap tuning:
|
118
|
-
|
119
|
-
-J-XX:+AggressiveHeap
|
13
|
+
Please look at the [wiki](https://github.com/mboeh/woodhouse/wiki) for documentation.
|
120
14
|
|
121
15
|
## Features
|
122
16
|
|
@@ -129,49 +23,13 @@ I've gotten good results from enabling aggressive heap tuning:
|
|
129
23
|
* Live status reporting with the `status` extension
|
130
24
|
* Job dispatch and execution middleware stacks
|
131
25
|
|
132
|
-
## Available Extensions
|
133
|
-
|
134
|
-
Extensions are loaded in the `Woodhouse.configure` block. Some extensions take arguments.
|
135
|
-
|
136
|
-
Woodhouse.configure do |woodhouse|
|
137
|
-
woodhouse.extension :new_relic
|
138
|
-
woodhouse.extension :status, host: "127.0.0.1", port: "10786"
|
139
|
-
end
|
140
|
-
|
141
|
-
### Built-In
|
142
|
-
|
143
|
-
* *progress*: Live status reporting on the progress of jobs.
|
144
|
-
* *new_relic*: New Relic background job monitoring.
|
145
|
-
|
146
|
-
### Packaged Separately
|
147
|
-
|
148
|
-
* [*status*][https://github.com/mboeh/woodhouse-status]: HTTP server embedded in Woodhouse to provide current status and liveness information via JSON.
|
149
|
-
|
150
26
|
## Upcoming
|
151
27
|
|
152
28
|
* Live reconfiguration of workers -- add or remove workers across one or more nodes without restarting
|
153
29
|
* Persistent configuration changes -- configuration changes saved to a data store and kept across deploys
|
154
30
|
* Web interface
|
155
31
|
|
156
|
-
##
|
157
|
-
|
158
|
-
* Examples and guides
|
159
|
-
* More documentation
|
160
|
-
|
161
|
-
## Supported Versions
|
162
|
-
|
163
|
-
### woodhouse 0.1.x
|
164
|
-
|
165
|
-
* bunny 0.9.x, RabbitMQ 2.x or later
|
166
|
-
* ruby 1.9
|
167
|
-
* MRI, JRuby, Rubinius 2
|
168
|
-
|
169
|
-
### woodhouse 0.0.x
|
170
|
-
|
171
|
-
* ruby 1.8
|
172
|
-
* MRI, JRuby, Rubinius
|
173
|
-
|
174
|
-
### Acknowledgements
|
32
|
+
## Acknowledgements
|
175
33
|
|
176
34
|
Woodhouse originated in a substantially modified version of the Workling background worker system, although all code has since
|
177
35
|
been replaced.
|
@@ -35,4 +35,17 @@ Woodhouse::Process.new.execute
|
|
35
35
|
EOF
|
36
36
|
end
|
37
37
|
|
38
|
+
def create_config
|
39
|
+
create_file "config/woodhouse.yml", <<-EOF
|
40
|
+
development:
|
41
|
+
dispatcher_type: local
|
42
|
+
test:
|
43
|
+
dispatcher_type: local
|
44
|
+
production:
|
45
|
+
dispatcher_type: amqp
|
46
|
+
server_info:
|
47
|
+
host: localhost
|
48
|
+
EOF
|
49
|
+
end
|
50
|
+
|
38
51
|
end
|
@@ -16,5 +16,6 @@ require 'woodhouse/dispatchers/bunny_dispatcher'
|
|
16
16
|
require 'woodhouse/dispatchers/hot_bunnies_dispatcher'
|
17
17
|
require 'woodhouse/dispatchers/local_pool_dispatcher'
|
18
18
|
require 'woodhouse/dispatchers/test_dispatcher'
|
19
|
+
require 'woodhouse/dispatchers/file_dispatcher'
|
19
20
|
|
20
21
|
Woodhouse::Dispatchers::AmqpDispatcher = Woodhouse::Dispatchers.default_amqp_dispatcher
|
@@ -20,9 +20,9 @@ class Woodhouse::Dispatchers::BunnyDispatcher < Woodhouse::Dispatchers::CommonAm
|
|
20
20
|
@pool.with do |conn|
|
21
21
|
yield conn
|
22
22
|
end
|
23
|
-
rescue Bunny::ClientTimeout
|
23
|
+
rescue Bunny::ClientTimeout => err
|
24
24
|
if retried
|
25
|
-
raise
|
25
|
+
raise Woodhouse::ConnectionError, "timed out while contacting AMQP server: #{err.message}"
|
26
26
|
else
|
27
27
|
new_pool!
|
28
28
|
retried = true
|
@@ -43,6 +43,8 @@ class Woodhouse::Dispatchers::BunnyDispatcher < Woodhouse::Dispatchers::CommonAm
|
|
43
43
|
@bunny.start
|
44
44
|
|
45
45
|
ConnectionPool.new { bunny.create_channel }
|
46
|
+
rescue Bunny::TCPConnectionFailed => err
|
47
|
+
raise Woodhouse::ConnectionError, "unable to connect to AMQP server: #{err.message}"
|
46
48
|
end
|
47
49
|
|
48
50
|
end
|
@@ -19,8 +19,7 @@ class Woodhouse::Dispatchers::CommonAmqpDispatcher < Woodhouse::Dispatcher
|
|
19
19
|
def deliver_job_update(job, data)
|
20
20
|
run do |client|
|
21
21
|
exchange = client.exchange("woodhouse.progress", :type => :direct)
|
22
|
-
|
23
|
-
client.queue(job.job_id, :durable => true).bind(exchange, :routing_key => job.job_id)
|
22
|
+
client.queue(job.job_id, :arguments => {"x-expires" => 5*60*1000}).bind(exchange, :routing_key => job.job_id)
|
24
23
|
exchange.publish(data.to_json, :routing_key => job.job_id)
|
25
24
|
end
|
26
25
|
end
|
@@ -0,0 +1,56 @@
|
|
1
|
+
require 'fileutils'
|
2
|
+
|
3
|
+
class Woodhouse::Dispatchers::FileDispatcher < Woodhouse::Dispatcher
|
4
|
+
attr_accessor :jobs_dir, :queue_dir
|
5
|
+
|
6
|
+
DEFAULT_QUEUE_DIR = '/tmp/woodhouse/queue'
|
7
|
+
|
8
|
+
def initialize(config, opts = {}, &blk)
|
9
|
+
super
|
10
|
+
|
11
|
+
server_info = @config.server_info || {}
|
12
|
+
self.queue_dir = server_info[:path] || DEFAULT_QUEUE_DIR
|
13
|
+
self.jobs_dir = "#{queue_dir}/jobs"
|
14
|
+
|
15
|
+
unless File.directory?(jobs_dir) # subdirectory of queue_dir
|
16
|
+
@config.logger.debug "[Woodhouse initialize] Creating queue directory '#{queue_dir}'"
|
17
|
+
FileUtils.mkdir_p jobs_dir
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
|
22
|
+
private
|
23
|
+
|
24
|
+
def deliver_job(job)
|
25
|
+
filename = "#{jobs_dir}/#{job.job_id}"
|
26
|
+
payload = YAML.dump(job)
|
27
|
+
|
28
|
+
@config.logger.debug "[Woodhouse] Writing job #{job.exchange_name} to #{filename}"
|
29
|
+
File.open(filename, 'w') {|f| f.write(YAML.dump(job)) }
|
30
|
+
|
31
|
+
enqueue(filename)
|
32
|
+
end
|
33
|
+
|
34
|
+
def deliver_job_update(job, data)
|
35
|
+
@config.logger.info "[Woodhouse job update] #{job.job_id} -- #{data.inspect}"
|
36
|
+
end
|
37
|
+
|
38
|
+
def enqueue(job_filename)
|
39
|
+
enqueued_filename = Dir["#{queue_dir}/j-*"].max || "#{queue_dir}/j-00000000"
|
40
|
+
10.times do
|
41
|
+
begin
|
42
|
+
enqueued_filename.succ!
|
43
|
+
File.symlink(job_filename, enqueued_filename)
|
44
|
+
break
|
45
|
+
rescue Errno::EEXIST
|
46
|
+
# Another dispatcher beat us to this position, try again
|
47
|
+
end
|
48
|
+
|
49
|
+
raise "Woodhouse FileDispatcher is not designed for high load scenarios. " +
|
50
|
+
"Maybe you should be using the AMQP dispatcher instead?"
|
51
|
+
end
|
52
|
+
|
53
|
+
enqueued_filename
|
54
|
+
end
|
55
|
+
|
56
|
+
end
|
@@ -64,7 +64,7 @@ module Woodhouse::Progress
|
|
64
64
|
begin
|
65
65
|
channel = bunny.create_channel
|
66
66
|
exchange = channel.direct("woodhouse.progress")
|
67
|
-
queue = channel.queue(job_id, :
|
67
|
+
queue = channel.queue(job_id, :arguments => {"x-expires" => 5*60*1000})
|
68
68
|
queue.bind(exchange, :routing_key => job_id)
|
69
69
|
payload = nil
|
70
70
|
queue.message_count.times do
|
@@ -143,7 +143,7 @@ module Woodhouse::Progress
|
|
143
143
|
def update_progress(data)
|
144
144
|
job = self
|
145
145
|
sink = progress_sink
|
146
|
-
Celluloid
|
146
|
+
Celluloid.internal_pool.get { sink.update_job(job, data) }
|
147
147
|
nil
|
148
148
|
end
|
149
149
|
|
@@ -46,6 +46,34 @@ class Woodhouse::NodeConfiguration
|
|
46
46
|
Woodhouse::Extension.install_extension(name, self, opts, &blk)
|
47
47
|
end
|
48
48
|
|
49
|
+
def load_yaml(path, keyw = {})
|
50
|
+
return unless File.exist?(path)
|
51
|
+
|
52
|
+
section = keyw[:section]
|
53
|
+
environment = keyw[:environment]
|
54
|
+
|
55
|
+
config_info = YAML.load(File.read(path))
|
56
|
+
|
57
|
+
if environment
|
58
|
+
config_info = config_info[environment]
|
59
|
+
end
|
60
|
+
if section
|
61
|
+
config_info = { section => config_info }
|
62
|
+
end
|
63
|
+
|
64
|
+
set config_info
|
65
|
+
end
|
66
|
+
|
67
|
+
def set(hash)
|
68
|
+
return unless hash
|
69
|
+
|
70
|
+
hash.each do |key, val|
|
71
|
+
if respond_to?("#{key}=")
|
72
|
+
send("#{key}=", val)
|
73
|
+
end
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
49
77
|
private
|
50
78
|
|
51
79
|
def lookup_key(key, namespace)
|
data/lib/woodhouse/process.rb
CHANGED
@@ -3,8 +3,18 @@ class Woodhouse::Process
|
|
3
3
|
|
4
4
|
def initialize(keyw = {})
|
5
5
|
@server = keyw[:server] || build_default_server(keyw)
|
6
|
+
self.class.register_instance self
|
6
7
|
end
|
7
|
-
|
8
|
+
|
9
|
+
def self.register_instance(instance)
|
10
|
+
@instance = instance
|
11
|
+
end
|
12
|
+
|
13
|
+
# Returns the current global Woodhouse process instance, if it is running.
|
14
|
+
def self.instance
|
15
|
+
@instance
|
16
|
+
end
|
17
|
+
|
8
18
|
def execute
|
9
19
|
# Borrowed this from sidekiq. https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb
|
10
20
|
trap "INT" do
|
@@ -16,25 +26,25 @@ class Woodhouse::Process
|
|
16
26
|
end
|
17
27
|
|
18
28
|
Woodhouse::Watchdog.start
|
19
|
-
Woodhouse::Watchdog.listen do |id, transition|
|
20
|
-
Woodhouse.global_configuration.logger.info "[##{id}] #{transition}"
|
21
|
-
end
|
22
29
|
|
23
30
|
begin
|
24
|
-
@server.start
|
31
|
+
@server.async.start
|
25
32
|
puts "Woodhouse serving as of #{Time.now}. Ctrl-C to stop."
|
26
33
|
@server.wait(:shutdown)
|
27
34
|
rescue Interrupt
|
28
|
-
|
29
|
-
@server.shutdown!
|
30
|
-
@server.wait(:shutdown)
|
35
|
+
shutdown
|
31
36
|
ensure
|
32
37
|
@server.terminate
|
33
38
|
Woodhouse::Watchdog.stop
|
34
|
-
exit
|
35
39
|
end
|
36
40
|
end
|
37
41
|
|
42
|
+
def shutdown
|
43
|
+
puts "Shutting down."
|
44
|
+
@server.async.shutdown
|
45
|
+
@server.wait(:shutdown)
|
46
|
+
end
|
47
|
+
|
38
48
|
private
|
39
49
|
|
40
50
|
def build_default_server(keyw)
|
data/lib/woodhouse/rails.rb
CHANGED
@@ -19,27 +19,29 @@ if defined?(Rails::Railtie)
|
|
19
19
|
Woodhouse.extend Woodhouse::RailsExtensions
|
20
20
|
|
21
21
|
class Woodhouse::Rails < Rails::Engine
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
22
|
+
initializer 'woodhouse-defaults', before: :load_config_initializers do
|
23
|
+
# Legacy config file just containing AMQP information.
|
24
|
+
legacy_config_path = Rails.root.join("config/workling.yml")
|
25
|
+
# New config file containing any configuration options.
|
26
|
+
config_path = Rails.root.join("config/woodhouse.yml")
|
27
|
+
|
28
28
|
# Preload everything in app/workers so default layout includes them
|
29
29
|
Rails.root.join("app/workers").tap do |workers|
|
30
30
|
Pathname.glob(workers.join("**/*.rb")).each do |worker_path|
|
31
31
|
worker_path.relative_path_from(workers).basename(".rb").to_s.camelize.constantize
|
32
32
|
end
|
33
33
|
end
|
34
|
+
|
34
35
|
# Set up reasonable defaults
|
35
36
|
Woodhouse.configure do |config|
|
36
37
|
config.logger = ::Rails.logger
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
end
|
41
|
-
end
|
38
|
+
|
39
|
+
config.load_yaml legacy_config_path, section: "server_info", environment: ::Rails.env
|
40
|
+
config.load_yaml config_path, environment: ::Rails.env
|
42
41
|
end
|
42
|
+
end
|
43
|
+
|
44
|
+
initializer "woodhouse-layout" do
|
43
45
|
Woodhouse.finish_loading_layout!
|
44
46
|
end
|
45
47
|
end
|
data/lib/woodhouse/runners.rb
CHANGED
@@ -0,0 +1,80 @@
|
|
1
|
+
require 'fileutils'
|
2
|
+
|
3
|
+
class Woodhouse::Runners::FileRunner < Woodhouse::Runner
|
4
|
+
attr_accessor :jobs_dir, :queue_dir
|
5
|
+
|
6
|
+
DEFAULT_QUEUE_DIR = '/tmp/woodhouse/queue'
|
7
|
+
|
8
|
+
def initialize(worker, config)
|
9
|
+
super
|
10
|
+
|
11
|
+
server_info = config.server_info || {}
|
12
|
+
self.queue_dir = server_info[:path] || DEFAULT_QUEUE_DIR
|
13
|
+
self.jobs_dir = "#{queue_dir}/jobs"
|
14
|
+
|
15
|
+
unless File.directory?(jobs_dir) # subdirectory of queue_dir
|
16
|
+
config.logger.debug "[Woodhouse initialize] Creating queue directory '#{queue_dir}'"
|
17
|
+
FileUtils.mkdir_p jobs_dir
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def subscribe
|
22
|
+
until @shutdown do
|
23
|
+
service_jobs
|
24
|
+
sleep 5
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
def spin_down
|
29
|
+
@shutdown = true
|
30
|
+
signal :spin_down
|
31
|
+
end
|
32
|
+
|
33
|
+
def service_jobs
|
34
|
+
each_job do |job,queue_id|
|
35
|
+
if can_service_job?(job)
|
36
|
+
reserve_job(queue_id) { service_job(job) }
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
def each_job(&block)
|
42
|
+
queue = Dir["#{queue_dir}/j-*"].sort
|
43
|
+
|
44
|
+
queue.each do |job_path|
|
45
|
+
job = YAML.load(File.read(job_path))
|
46
|
+
queue_id = File.basename(job_path)[2..-1]
|
47
|
+
|
48
|
+
yield(job, queue_id)
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
def reserve_job(queue_id, &block)
|
53
|
+
enqueued = "#{queue_dir}/j-#{queue_id}"
|
54
|
+
processing = "#{queue_dir}/p-#{queue_id}"
|
55
|
+
failed = "#{queue_dir}/f-#{queue_id}"
|
56
|
+
|
57
|
+
begin
|
58
|
+
FileUtils.mv(enqueued, processing)
|
59
|
+
|
60
|
+
if yield
|
61
|
+
# Success, clean up
|
62
|
+
File.unlink(processing)
|
63
|
+
end
|
64
|
+
|
65
|
+
rescue Errno::ENOENT
|
66
|
+
# Another worker beat us to the job
|
67
|
+
false
|
68
|
+
|
69
|
+
rescue => err
|
70
|
+
# Woodhouse internal error occurred during processing
|
71
|
+
File.open(processing, 'a') {|f| f.write YAML.dump(err) }
|
72
|
+
raise
|
73
|
+
|
74
|
+
ensure
|
75
|
+
# If file still hanging around then it failed
|
76
|
+
FileUtils.mv(processing, failed) if File.exists?(processing)
|
77
|
+
end
|
78
|
+
end
|
79
|
+
|
80
|
+
end
|
data/lib/woodhouse/scheduler.rb
CHANGED
@@ -52,7 +52,7 @@ class Woodhouse::Scheduler
|
|
52
52
|
@config.logger.debug "Spinning up thread #{idx} for worker #{@worker_def.describe}"
|
53
53
|
worker = @config.runner_type.new_link(@worker_def, @config)
|
54
54
|
@threads << worker
|
55
|
-
worker.subscribe
|
55
|
+
worker.async.subscribe
|
56
56
|
end
|
57
57
|
end
|
58
58
|
|
data/lib/woodhouse/version.rb
CHANGED
data/woodhouse.gemspec
CHANGED
@@ -8,6 +8,7 @@ Gem::Specification.new do |s|
|
|
8
8
|
s.platform = Gem::Platform::RUBY
|
9
9
|
s.authors = ["Matthew Boeh"]
|
10
10
|
s.email = ["matt@crowdcompass.com", "matthew.boeh@gmail.com"]
|
11
|
+
s.licenses = ["MIT"]
|
11
12
|
s.homepage = "http://github.com/mboeh/woodhouse"
|
12
13
|
s.summary = %q{An AMQP-based background worker system for Ruby}
|
13
14
|
s.description = %q{An AMQP-based background worker system for Ruby designed to make managing heterogenous tasks relatively easy.
|
@@ -18,10 +19,11 @@ Gem::Specification.new do |s|
|
|
18
19
|
|
19
20
|
s.rubyforge_project = "woodhouse"
|
20
21
|
|
21
|
-
s.add_dependency 'celluloid', '~> 0.
|
22
|
-
s.add_dependency 'bunny', "~> 0.9.
|
23
|
-
s.add_dependency 'connection_pool'
|
24
|
-
s.add_dependency 'json'
|
22
|
+
s.add_dependency 'celluloid', '~> 0.15'
|
23
|
+
s.add_dependency 'bunny', "~> 0.9.8"
|
24
|
+
s.add_dependency 'connection_pool', '~> 2.0'
|
25
|
+
s.add_dependency 'json', '~> 1.8'
|
26
|
+
s.add_dependency 'cause', '~> 0.1'
|
25
27
|
|
26
28
|
s.add_development_dependency 'rspec', '~> 1.3.1'
|
27
29
|
s.add_development_dependency 'rake'
|
metadata
CHANGED
@@ -1,84 +1,88 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: woodhouse
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
5
|
-
prerelease:
|
4
|
+
version: 1.0.0
|
6
5
|
platform: ruby
|
7
6
|
authors:
|
8
7
|
- Matthew Boeh
|
9
8
|
autorequire:
|
10
9
|
bindir: bin
|
11
10
|
cert_chain: []
|
12
|
-
date:
|
11
|
+
date: 2014-08-26 00:00:00.000000000 Z
|
13
12
|
dependencies:
|
14
13
|
- !ruby/object:Gem::Dependency
|
15
14
|
name: celluloid
|
16
15
|
requirement: !ruby/object:Gem::Requirement
|
17
|
-
none: false
|
18
16
|
requirements:
|
19
17
|
- - ~>
|
20
18
|
- !ruby/object:Gem::Version
|
21
|
-
version: 0.
|
19
|
+
version: '0.15'
|
22
20
|
type: :runtime
|
23
21
|
prerelease: false
|
24
22
|
version_requirements: !ruby/object:Gem::Requirement
|
25
|
-
none: false
|
26
23
|
requirements:
|
27
24
|
- - ~>
|
28
25
|
- !ruby/object:Gem::Version
|
29
|
-
version: 0.
|
26
|
+
version: '0.15'
|
30
27
|
- !ruby/object:Gem::Dependency
|
31
28
|
name: bunny
|
32
29
|
requirement: !ruby/object:Gem::Requirement
|
33
|
-
none: false
|
34
30
|
requirements:
|
35
31
|
- - ~>
|
36
32
|
- !ruby/object:Gem::Version
|
37
|
-
version: 0.9.
|
33
|
+
version: 0.9.8
|
38
34
|
type: :runtime
|
39
35
|
prerelease: false
|
40
36
|
version_requirements: !ruby/object:Gem::Requirement
|
41
|
-
none: false
|
42
37
|
requirements:
|
43
38
|
- - ~>
|
44
39
|
- !ruby/object:Gem::Version
|
45
|
-
version: 0.9.
|
40
|
+
version: 0.9.8
|
46
41
|
- !ruby/object:Gem::Dependency
|
47
42
|
name: connection_pool
|
48
43
|
requirement: !ruby/object:Gem::Requirement
|
49
|
-
none: false
|
50
44
|
requirements:
|
51
|
-
- -
|
45
|
+
- - ~>
|
52
46
|
- !ruby/object:Gem::Version
|
53
|
-
version: '0'
|
47
|
+
version: '2.0'
|
54
48
|
type: :runtime
|
55
49
|
prerelease: false
|
56
50
|
version_requirements: !ruby/object:Gem::Requirement
|
57
|
-
none: false
|
58
51
|
requirements:
|
59
|
-
- -
|
52
|
+
- - ~>
|
60
53
|
- !ruby/object:Gem::Version
|
61
|
-
version: '0'
|
54
|
+
version: '2.0'
|
62
55
|
- !ruby/object:Gem::Dependency
|
63
56
|
name: json
|
64
57
|
requirement: !ruby/object:Gem::Requirement
|
65
|
-
none: false
|
66
58
|
requirements:
|
67
|
-
- -
|
59
|
+
- - ~>
|
68
60
|
- !ruby/object:Gem::Version
|
69
|
-
version: '
|
61
|
+
version: '1.8'
|
70
62
|
type: :runtime
|
71
63
|
prerelease: false
|
72
64
|
version_requirements: !ruby/object:Gem::Requirement
|
73
|
-
none: false
|
74
65
|
requirements:
|
75
|
-
- -
|
66
|
+
- - ~>
|
76
67
|
- !ruby/object:Gem::Version
|
77
|
-
version: '
|
68
|
+
version: '1.8'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: cause
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ~>
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0.1'
|
76
|
+
type: :runtime
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ~>
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0.1'
|
78
83
|
- !ruby/object:Gem::Dependency
|
79
84
|
name: rspec
|
80
85
|
requirement: !ruby/object:Gem::Requirement
|
81
|
-
none: false
|
82
86
|
requirements:
|
83
87
|
- - ~>
|
84
88
|
- !ruby/object:Gem::Version
|
@@ -86,7 +90,6 @@ dependencies:
|
|
86
90
|
type: :development
|
87
91
|
prerelease: false
|
88
92
|
version_requirements: !ruby/object:Gem::Requirement
|
89
|
-
none: false
|
90
93
|
requirements:
|
91
94
|
- - ~>
|
92
95
|
- !ruby/object:Gem::Version
|
@@ -94,68 +97,60 @@ dependencies:
|
|
94
97
|
- !ruby/object:Gem::Dependency
|
95
98
|
name: rake
|
96
99
|
requirement: !ruby/object:Gem::Requirement
|
97
|
-
none: false
|
98
100
|
requirements:
|
99
|
-
- -
|
101
|
+
- - '>='
|
100
102
|
- !ruby/object:Gem::Version
|
101
103
|
version: '0'
|
102
104
|
type: :development
|
103
105
|
prerelease: false
|
104
106
|
version_requirements: !ruby/object:Gem::Requirement
|
105
|
-
none: false
|
106
107
|
requirements:
|
107
|
-
- -
|
108
|
+
- - '>='
|
108
109
|
- !ruby/object:Gem::Version
|
109
110
|
version: '0'
|
110
111
|
- !ruby/object:Gem::Dependency
|
111
112
|
name: guard
|
112
113
|
requirement: !ruby/object:Gem::Requirement
|
113
|
-
none: false
|
114
114
|
requirements:
|
115
|
-
- -
|
115
|
+
- - '>='
|
116
116
|
- !ruby/object:Gem::Version
|
117
117
|
version: '0'
|
118
118
|
type: :development
|
119
119
|
prerelease: false
|
120
120
|
version_requirements: !ruby/object:Gem::Requirement
|
121
|
-
none: false
|
122
121
|
requirements:
|
123
|
-
- -
|
122
|
+
- - '>='
|
124
123
|
- !ruby/object:Gem::Version
|
125
124
|
version: '0'
|
126
125
|
- !ruby/object:Gem::Dependency
|
127
126
|
name: guard-rspec
|
128
127
|
requirement: !ruby/object:Gem::Requirement
|
129
|
-
none: false
|
130
128
|
requirements:
|
131
|
-
- -
|
129
|
+
- - '>='
|
132
130
|
- !ruby/object:Gem::Version
|
133
131
|
version: '0'
|
134
132
|
type: :development
|
135
133
|
prerelease: false
|
136
134
|
version_requirements: !ruby/object:Gem::Requirement
|
137
|
-
none: false
|
138
135
|
requirements:
|
139
|
-
- -
|
136
|
+
- - '>='
|
140
137
|
- !ruby/object:Gem::Version
|
141
138
|
version: '0'
|
142
139
|
- !ruby/object:Gem::Dependency
|
143
140
|
name: mocha
|
144
141
|
requirement: !ruby/object:Gem::Requirement
|
145
|
-
none: false
|
146
142
|
requirements:
|
147
|
-
- -
|
143
|
+
- - '>='
|
148
144
|
- !ruby/object:Gem::Version
|
149
145
|
version: '0'
|
150
146
|
type: :development
|
151
147
|
prerelease: false
|
152
148
|
version_requirements: !ruby/object:Gem::Requirement
|
153
|
-
none: false
|
154
149
|
requirements:
|
155
|
-
- -
|
150
|
+
- - '>='
|
156
151
|
- !ruby/object:Gem::Version
|
157
152
|
version: '0'
|
158
|
-
description:
|
153
|
+
description: "An AMQP-based background worker system for Ruby designed to make managing
|
159
154
|
heterogenous tasks relatively easy.\n \n The use case for Woodhouse is for reliable
|
160
155
|
and sane performance in situations where jobs on a single queue may vary significantly
|
161
156
|
in length. The goal is to permit large numbers of quick jobs to be serviced even
|
@@ -187,6 +182,7 @@ files:
|
|
187
182
|
- lib/woodhouse/dispatchers.rb
|
188
183
|
- lib/woodhouse/dispatchers/bunny_dispatcher.rb
|
189
184
|
- lib/woodhouse/dispatchers/common_amqp_dispatcher.rb
|
185
|
+
- lib/woodhouse/dispatchers/file_dispatcher.rb
|
190
186
|
- lib/woodhouse/dispatchers/hot_bunnies_dispatcher.rb
|
191
187
|
- lib/woodhouse/dispatchers/local_dispatcher.rb
|
192
188
|
- lib/woodhouse/dispatchers/local_pool_dispatcher.rb
|
@@ -217,6 +213,7 @@ files:
|
|
217
213
|
- lib/woodhouse/runners.rb
|
218
214
|
- lib/woodhouse/runners/bunny_runner.rb
|
219
215
|
- lib/woodhouse/runners/dummy_runner.rb
|
216
|
+
- lib/woodhouse/runners/file_runner.rb
|
220
217
|
- lib/woodhouse/runners/hot_bunnies_runner.rb
|
221
218
|
- lib/woodhouse/scheduler.rb
|
222
219
|
- lib/woodhouse/server.rb
|
@@ -239,33 +236,41 @@ files:
|
|
239
236
|
- spec/worker_spec.rb
|
240
237
|
- woodhouse.gemspec
|
241
238
|
homepage: http://github.com/mboeh/woodhouse
|
242
|
-
licenses:
|
239
|
+
licenses:
|
240
|
+
- MIT
|
241
|
+
metadata: {}
|
243
242
|
post_install_message:
|
244
243
|
rdoc_options: []
|
245
244
|
require_paths:
|
246
245
|
- lib
|
247
246
|
required_ruby_version: !ruby/object:Gem::Requirement
|
248
|
-
none: false
|
249
247
|
requirements:
|
250
|
-
- -
|
248
|
+
- - '>='
|
251
249
|
- !ruby/object:Gem::Version
|
252
250
|
version: '0'
|
253
|
-
segments:
|
254
|
-
- 0
|
255
|
-
hash: -1311770161800686581
|
256
251
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
257
|
-
none: false
|
258
252
|
requirements:
|
259
|
-
- -
|
253
|
+
- - '>='
|
260
254
|
- !ruby/object:Gem::Version
|
261
255
|
version: '0'
|
262
|
-
segments:
|
263
|
-
- 0
|
264
|
-
hash: -1311770161800686581
|
265
256
|
requirements: []
|
266
257
|
rubyforge_project: woodhouse
|
267
|
-
rubygems_version:
|
258
|
+
rubygems_version: 2.2.2
|
268
259
|
signing_key:
|
269
|
-
specification_version:
|
260
|
+
specification_version: 4
|
270
261
|
summary: An AMQP-based background worker system for Ruby
|
271
|
-
test_files:
|
262
|
+
test_files:
|
263
|
+
- spec/integration/bunny_worker_process_spec.rb
|
264
|
+
- spec/layout_builder_spec.rb
|
265
|
+
- spec/layout_spec.rb
|
266
|
+
- spec/middleware_stack_spec.rb
|
267
|
+
- spec/mixin_registry_spec.rb
|
268
|
+
- spec/node_configuration_spec.rb
|
269
|
+
- spec/progress_spec.rb
|
270
|
+
- spec/queue_criteria_spec.rb
|
271
|
+
- spec/scheduler_spec.rb
|
272
|
+
- spec/server_spec.rb
|
273
|
+
- spec/shared_contexts.rb
|
274
|
+
- spec/test_dispatcher_spec.rb
|
275
|
+
- spec/worker_spec.rb
|
276
|
+
has_rdoc:
|