woodhouse 0.1.5 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/.travis.yml +3 -2
- data/README.markdown +4 -146
- data/lib/generators/woodhouse_generator.rb +13 -0
- data/lib/woodhouse/dispatchers.rb +1 -0
- data/lib/woodhouse/dispatchers/bunny_dispatcher.rb +4 -2
- data/lib/woodhouse/dispatchers/common_amqp_dispatcher.rb +1 -2
- data/lib/woodhouse/dispatchers/file_dispatcher.rb +56 -0
- data/lib/woodhouse/extensions/progress.rb +2 -2
- data/lib/woodhouse/node_configuration.rb +28 -0
- data/lib/woodhouse/process.rb +19 -9
- data/lib/woodhouse/rails.rb +13 -11
- data/lib/woodhouse/runners.rb +1 -0
- data/lib/woodhouse/runners/file_runner.rb +80 -0
- data/lib/woodhouse/scheduler.rb +1 -1
- data/lib/woodhouse/version.rb +1 -1
- data/woodhouse.gemspec +6 -4
- metadata +61 -56
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 82eac8a615e21e61744337db2453b756cfa2168c
|
4
|
+
data.tar.gz: b538599fd8757f81113dafa997ddb9e930427c2b
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 391c9243d9d1b81794272ebeccecf4a33dadd56c589cbe588aa7046da27e8fcc4681fe5b326166533d8024a85c4c86fab5139490ca508240c9c6e828eb888e92
|
7
|
+
data.tar.gz: bc0568ef140f9911673201eadf1a0c8601e69e81e6358f63a0b2880d46519a76b379a9af1e2c1c1510871007efb7a36e4b29896ad19e5a53ebe01219400d0af7
|
data/.travis.yml
CHANGED
data/README.markdown
CHANGED
@@ -2,121 +2,15 @@
|
|
2
2
|
|
3
3
|
[<img src="https://secure.travis-ci.org/mboeh/woodhouse.png?branch=master" alt="Build Status" />](http://travis-ci.org/mboeh/woodhouse)
|
4
4
|
|
5
|
-
|
5
|
+
A RabbitMQ-based background worker system for Ruby designed to make managing heterogenous tasks relatively easy.
|
6
6
|
|
7
7
|
The use case for Woodhouse is for reliable and sane performance in situations where jobs on a single queue may vary significantly
|
8
8
|
in length. The goal is to permit large numbers of quick jobs to be serviced even when many slow jobs are in the queue. A secondary
|
9
9
|
goal is to provide a sane way for jobs on a given queue to be given special priority or dispatched to a server more suited to them.
|
10
10
|
|
11
|
-
Woodhouse
|
11
|
+
Woodhouse 1.0, located in the 1-0-stable branch, is production-ready and stable for Ruby 1.9.
|
12
12
|
|
13
|
-
|
14
|
-
|
15
|
-
### Rails
|
16
|
-
|
17
|
-
Add
|
18
|
-
|
19
|
-
gem 'woodhouse', github: 'mboeh/woodhouse'
|
20
|
-
|
21
|
-
to your Gemfile.
|
22
|
-
|
23
|
-
Run
|
24
|
-
|
25
|
-
% rails generate woodhouse
|
26
|
-
|
27
|
-
to create script/woodhouse and config/initializers/woodhouse.rb.
|
28
|
-
|
29
|
-
### Basic Usage
|
30
|
-
|
31
|
-
The simplest way to set up a worker class is to include Woodhouse::Worker and define public methods.
|
32
|
-
|
33
|
-
class IsisWorker
|
34
|
-
include Woodhouse::Worker
|
35
|
-
|
36
|
-
def pam_gossip(job)
|
37
|
-
puts "Pam gossips about #{job[:who]}."
|
38
|
-
end
|
39
|
-
|
40
|
-
def sterling_insult(job)
|
41
|
-
puts "Sterling insults #{job[:who]}."
|
42
|
-
end
|
43
|
-
end
|
44
|
-
|
45
|
-
Jobs are dispatched asynchronously to a worker by adding `async_` to the method name:
|
46
|
-
|
47
|
-
IsisWorker.async_pam_gossip :who => "Cyril"
|
48
|
-
|
49
|
-
Woodhouse jobs always take a hash of arguments. The worker receives a Woodhouse::Job, which acts like a hash
|
50
|
-
but also supplies additional functionality.
|
51
|
-
|
52
|
-
### Dispatchers
|
53
|
-
|
54
|
-
The dispatcher used for sending out jobs can be set in the Woodhouse config block:
|
55
|
-
|
56
|
-
Woodhouse.configure do |woodhouse|
|
57
|
-
woodhouse.dispatcher_type = :local # :local_pool | :amqp | :test
|
58
|
-
end
|
59
|
-
|
60
|
-
Calling the `async` version of a job method sends it to the currently configured dispatcher. The default dispatcher
|
61
|
-
type is `:local`, which simply executes the job synchronously (although still passing it through middleware; see below).
|
62
|
-
|
63
|
-
If you are running tests and you want to be able to test that your code is dispatching Woodhouse jobs (without running
|
64
|
-
them), use the `:test` dispatcher and the dispatcher will simply accumulate jobs (of class Woodhouse::Job):
|
65
|
-
|
66
|
-
IsisWorker.async_pam_gossip :who => "Cyril"
|
67
|
-
that_job = Woodhouse.dispatcher.jobs.last
|
68
|
-
that_job.worker_class_name # ==> "IsisWorker"
|
69
|
-
that_job.job_method # ==> "pam_gossip"
|
70
|
-
that_job.arguments[:who] # ==> "Cyril"
|
71
|
-
|
72
|
-
If you want `girl_friday` style in-process threaded backgrounding, you can get that by selecting the `:local_pool`
|
73
|
-
dispatcher.
|
74
|
-
|
75
|
-
Finally, if you want to run your jobs in a background process, you'll need to set up the `:amqp` dispatcher. This will
|
76
|
-
use either the Hot Bunnies library (on JRuby) or the Bunny library (on all other Ruby engines). Bunny is suitable for
|
77
|
-
dispatch but can be a little bit CPU-hungry in the background process. Hot Bunnies works great for both. You don't have
|
78
|
-
to use the same Ruby version for your background process as for the dispatching application -- we use Woodhouse in production
|
79
|
-
with a JRuby background process and MRI frontend processes.
|
80
|
-
|
81
|
-
You'll also need to have RabbitMQ running. If it's running with the default (open) permissions on the local server, you don't
|
82
|
-
need to configure it at all. Otherwise, you'll have to set the server connection info. You have two options for this. On Rails,
|
83
|
-
you can create a config/woodhouse.yml file, formatted similar to config/database.yml:
|
84
|
-
|
85
|
-
production:
|
86
|
-
host: myrabbitmq.server.local
|
87
|
-
vhost: /some-vhost
|
88
|
-
|
89
|
-
(The parameters accepted here are the same used for Bunny.connect; I promise to document them here soon.)
|
90
|
-
|
91
|
-
Otherwise, you can do it in the Woodhouse config block:
|
92
|
-
|
93
|
-
Woodhouse.configure do |woodhouse|
|
94
|
-
woodhouse.server_info = { :host => "myrabbitmq.server.local" }
|
95
|
-
end
|
96
|
-
|
97
|
-
### Running The Background Process
|
98
|
-
|
99
|
-
All you have to do is run `script/woodhouse`. It'll load your Rails environment and start the server process. It responds to QUIT
|
100
|
-
and INT signals correctly; I'm working on seeing if I can get it to restart worker processes with HUP and to dump/load the current
|
101
|
-
layout with USR1/USR2.
|
102
|
-
|
103
|
-
`script/woodhouse` logs job execution and results to `log/woodhouse.log`.
|
104
|
-
|
105
|
-
### Performance Errata
|
106
|
-
|
107
|
-
If you're using JRuby with a large application, I've found that the JVM's permanent generation can get exhausted. If you have
|
108
|
-
plenty of heap but still get GC overhead errors, try bumping up the PermGen by including this on the JRuby command line:
|
109
|
-
|
110
|
-
-J-XX:MaxPermSize=128m
|
111
|
-
|
112
|
-
Performance will generally be better with the `--server` flag:
|
113
|
-
|
114
|
-
--server
|
115
|
-
|
116
|
-
A lot of the jobs I run tend to allocate and dispose a lot of memory very quickly, and Woodhouse will be a long-running process.
|
117
|
-
I've gotten good results from enabling aggressive heap tuning:
|
118
|
-
|
119
|
-
-J-XX:+AggressiveHeap
|
13
|
+
Please look at the [wiki](https://github.com/mboeh/woodhouse/wiki) for documentation.
|
120
14
|
|
121
15
|
## Features
|
122
16
|
|
@@ -129,49 +23,13 @@ I've gotten good results from enabling aggressive heap tuning:
|
|
129
23
|
* Live status reporting with the `status` extension
|
130
24
|
* Job dispatch and execution middleware stacks
|
131
25
|
|
132
|
-
## Available Extensions
|
133
|
-
|
134
|
-
Extensions are loaded in the `Woodhouse.configure` block. Some extensions take arguments.
|
135
|
-
|
136
|
-
Woodhouse.configure do |woodhouse|
|
137
|
-
woodhouse.extension :new_relic
|
138
|
-
woodhouse.extension :status, host: "127.0.0.1", port: "10786"
|
139
|
-
end
|
140
|
-
|
141
|
-
### Built-In
|
142
|
-
|
143
|
-
* *progress*: Live status reporting on the progress of jobs.
|
144
|
-
* *new_relic*: New Relic background job monitoring.
|
145
|
-
|
146
|
-
### Packaged Separately
|
147
|
-
|
148
|
-
* [*status*][https://github.com/mboeh/woodhouse-status]: HTTP server embedded in Woodhouse to provide current status and liveness information via JSON.
|
149
|
-
|
150
26
|
## Upcoming
|
151
27
|
|
152
28
|
* Live reconfiguration of workers -- add or remove workers across one or more nodes without restarting
|
153
29
|
* Persistent configuration changes -- configuration changes saved to a data store and kept across deploys
|
154
30
|
* Web interface
|
155
31
|
|
156
|
-
##
|
157
|
-
|
158
|
-
* Examples and guides
|
159
|
-
* More documentation
|
160
|
-
|
161
|
-
## Supported Versions
|
162
|
-
|
163
|
-
### woodhouse 0.1.x
|
164
|
-
|
165
|
-
* bunny 0.9.x, RabbitMQ 2.x or later
|
166
|
-
* ruby 1.9
|
167
|
-
* MRI, JRuby, Rubinius 2
|
168
|
-
|
169
|
-
### woodhouse 0.0.x
|
170
|
-
|
171
|
-
* ruby 1.8
|
172
|
-
* MRI, JRuby, Rubinius
|
173
|
-
|
174
|
-
### Acknowledgements
|
32
|
+
## Acknowledgements
|
175
33
|
|
176
34
|
Woodhouse originated in a substantially modified version of the Workling background worker system, although all code has since
|
177
35
|
been replaced.
|
@@ -35,4 +35,17 @@ Woodhouse::Process.new.execute
|
|
35
35
|
EOF
|
36
36
|
end
|
37
37
|
|
38
|
+
def create_config
|
39
|
+
create_file "config/woodhouse.yml", <<-EOF
|
40
|
+
development:
|
41
|
+
dispatcher_type: local
|
42
|
+
test:
|
43
|
+
dispatcher_type: local
|
44
|
+
production:
|
45
|
+
dispatcher_type: amqp
|
46
|
+
server_info:
|
47
|
+
host: localhost
|
48
|
+
EOF
|
49
|
+
end
|
50
|
+
|
38
51
|
end
|
@@ -16,5 +16,6 @@ require 'woodhouse/dispatchers/bunny_dispatcher'
|
|
16
16
|
require 'woodhouse/dispatchers/hot_bunnies_dispatcher'
|
17
17
|
require 'woodhouse/dispatchers/local_pool_dispatcher'
|
18
18
|
require 'woodhouse/dispatchers/test_dispatcher'
|
19
|
+
require 'woodhouse/dispatchers/file_dispatcher'
|
19
20
|
|
20
21
|
Woodhouse::Dispatchers::AmqpDispatcher = Woodhouse::Dispatchers.default_amqp_dispatcher
|
@@ -20,9 +20,9 @@ class Woodhouse::Dispatchers::BunnyDispatcher < Woodhouse::Dispatchers::CommonAm
|
|
20
20
|
@pool.with do |conn|
|
21
21
|
yield conn
|
22
22
|
end
|
23
|
-
rescue Bunny::ClientTimeout
|
23
|
+
rescue Bunny::ClientTimeout => err
|
24
24
|
if retried
|
25
|
-
raise
|
25
|
+
raise Woodhouse::ConnectionError, "timed out while contacting AMQP server: #{err.message}"
|
26
26
|
else
|
27
27
|
new_pool!
|
28
28
|
retried = true
|
@@ -43,6 +43,8 @@ class Woodhouse::Dispatchers::BunnyDispatcher < Woodhouse::Dispatchers::CommonAm
|
|
43
43
|
@bunny.start
|
44
44
|
|
45
45
|
ConnectionPool.new { bunny.create_channel }
|
46
|
+
rescue Bunny::TCPConnectionFailed => err
|
47
|
+
raise Woodhouse::ConnectionError, "unable to connect to AMQP server: #{err.message}"
|
46
48
|
end
|
47
49
|
|
48
50
|
end
|
@@ -19,8 +19,7 @@ class Woodhouse::Dispatchers::CommonAmqpDispatcher < Woodhouse::Dispatcher
|
|
19
19
|
def deliver_job_update(job, data)
|
20
20
|
run do |client|
|
21
21
|
exchange = client.exchange("woodhouse.progress", :type => :direct)
|
22
|
-
|
23
|
-
client.queue(job.job_id, :durable => true).bind(exchange, :routing_key => job.job_id)
|
22
|
+
client.queue(job.job_id, :arguments => {"x-expires" => 5*60*1000}).bind(exchange, :routing_key => job.job_id)
|
24
23
|
exchange.publish(data.to_json, :routing_key => job.job_id)
|
25
24
|
end
|
26
25
|
end
|
@@ -0,0 +1,56 @@
|
|
1
|
+
require 'fileutils'
|
2
|
+
|
3
|
+
class Woodhouse::Dispatchers::FileDispatcher < Woodhouse::Dispatcher
|
4
|
+
attr_accessor :jobs_dir, :queue_dir
|
5
|
+
|
6
|
+
DEFAULT_QUEUE_DIR = '/tmp/woodhouse/queue'
|
7
|
+
|
8
|
+
def initialize(config, opts = {}, &blk)
|
9
|
+
super
|
10
|
+
|
11
|
+
server_info = @config.server_info || {}
|
12
|
+
self.queue_dir = server_info[:path] || DEFAULT_QUEUE_DIR
|
13
|
+
self.jobs_dir = "#{queue_dir}/jobs"
|
14
|
+
|
15
|
+
unless File.directory?(jobs_dir) # subdirectory of queue_dir
|
16
|
+
@config.logger.debug "[Woodhouse initialize] Creating queue directory '#{queue_dir}'"
|
17
|
+
FileUtils.mkdir_p jobs_dir
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
|
22
|
+
private
|
23
|
+
|
24
|
+
def deliver_job(job)
|
25
|
+
filename = "#{jobs_dir}/#{job.job_id}"
|
26
|
+
payload = YAML.dump(job)
|
27
|
+
|
28
|
+
@config.logger.debug "[Woodhouse] Writing job #{job.exchange_name} to #{filename}"
|
29
|
+
File.open(filename, 'w') {|f| f.write(YAML.dump(job)) }
|
30
|
+
|
31
|
+
enqueue(filename)
|
32
|
+
end
|
33
|
+
|
34
|
+
def deliver_job_update(job, data)
|
35
|
+
@config.logger.info "[Woodhouse job update] #{job.job_id} -- #{data.inspect}"
|
36
|
+
end
|
37
|
+
|
38
|
+
def enqueue(job_filename)
|
39
|
+
enqueued_filename = Dir["#{queue_dir}/j-*"].max || "#{queue_dir}/j-00000000"
|
40
|
+
10.times do
|
41
|
+
begin
|
42
|
+
enqueued_filename.succ!
|
43
|
+
File.symlink(job_filename, enqueued_filename)
|
44
|
+
break
|
45
|
+
rescue Errno::EEXIST
|
46
|
+
# Another dispatcher beat us to this position, try again
|
47
|
+
end
|
48
|
+
|
49
|
+
raise "Woodhouse FileDispatcher is not designed for high load scenarios. " +
|
50
|
+
"Maybe you should be using the AMQP dispatcher instead?"
|
51
|
+
end
|
52
|
+
|
53
|
+
enqueued_filename
|
54
|
+
end
|
55
|
+
|
56
|
+
end
|
@@ -64,7 +64,7 @@ module Woodhouse::Progress
|
|
64
64
|
begin
|
65
65
|
channel = bunny.create_channel
|
66
66
|
exchange = channel.direct("woodhouse.progress")
|
67
|
-
queue = channel.queue(job_id, :
|
67
|
+
queue = channel.queue(job_id, :arguments => {"x-expires" => 5*60*1000})
|
68
68
|
queue.bind(exchange, :routing_key => job_id)
|
69
69
|
payload = nil
|
70
70
|
queue.message_count.times do
|
@@ -143,7 +143,7 @@ module Woodhouse::Progress
|
|
143
143
|
def update_progress(data)
|
144
144
|
job = self
|
145
145
|
sink = progress_sink
|
146
|
-
Celluloid
|
146
|
+
Celluloid.internal_pool.get { sink.update_job(job, data) }
|
147
147
|
nil
|
148
148
|
end
|
149
149
|
|
@@ -46,6 +46,34 @@ class Woodhouse::NodeConfiguration
|
|
46
46
|
Woodhouse::Extension.install_extension(name, self, opts, &blk)
|
47
47
|
end
|
48
48
|
|
49
|
+
def load_yaml(path, keyw = {})
|
50
|
+
return unless File.exist?(path)
|
51
|
+
|
52
|
+
section = keyw[:section]
|
53
|
+
environment = keyw[:environment]
|
54
|
+
|
55
|
+
config_info = YAML.load(File.read(path))
|
56
|
+
|
57
|
+
if environment
|
58
|
+
config_info = config_info[environment]
|
59
|
+
end
|
60
|
+
if section
|
61
|
+
config_info = { section => config_info }
|
62
|
+
end
|
63
|
+
|
64
|
+
set config_info
|
65
|
+
end
|
66
|
+
|
67
|
+
def set(hash)
|
68
|
+
return unless hash
|
69
|
+
|
70
|
+
hash.each do |key, val|
|
71
|
+
if respond_to?("#{key}=")
|
72
|
+
send("#{key}=", val)
|
73
|
+
end
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
49
77
|
private
|
50
78
|
|
51
79
|
def lookup_key(key, namespace)
|
data/lib/woodhouse/process.rb
CHANGED
@@ -3,8 +3,18 @@ class Woodhouse::Process
|
|
3
3
|
|
4
4
|
def initialize(keyw = {})
|
5
5
|
@server = keyw[:server] || build_default_server(keyw)
|
6
|
+
self.class.register_instance self
|
6
7
|
end
|
7
|
-
|
8
|
+
|
9
|
+
def self.register_instance(instance)
|
10
|
+
@instance = instance
|
11
|
+
end
|
12
|
+
|
13
|
+
# Returns the current global Woodhouse process instance, if it is running.
|
14
|
+
def self.instance
|
15
|
+
@instance
|
16
|
+
end
|
17
|
+
|
8
18
|
def execute
|
9
19
|
# Borrowed this from sidekiq. https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb
|
10
20
|
trap "INT" do
|
@@ -16,25 +26,25 @@ class Woodhouse::Process
|
|
16
26
|
end
|
17
27
|
|
18
28
|
Woodhouse::Watchdog.start
|
19
|
-
Woodhouse::Watchdog.listen do |id, transition|
|
20
|
-
Woodhouse.global_configuration.logger.info "[##{id}] #{transition}"
|
21
|
-
end
|
22
29
|
|
23
30
|
begin
|
24
|
-
@server.start
|
31
|
+
@server.async.start
|
25
32
|
puts "Woodhouse serving as of #{Time.now}. Ctrl-C to stop."
|
26
33
|
@server.wait(:shutdown)
|
27
34
|
rescue Interrupt
|
28
|
-
|
29
|
-
@server.shutdown!
|
30
|
-
@server.wait(:shutdown)
|
35
|
+
shutdown
|
31
36
|
ensure
|
32
37
|
@server.terminate
|
33
38
|
Woodhouse::Watchdog.stop
|
34
|
-
exit
|
35
39
|
end
|
36
40
|
end
|
37
41
|
|
42
|
+
def shutdown
|
43
|
+
puts "Shutting down."
|
44
|
+
@server.async.shutdown
|
45
|
+
@server.wait(:shutdown)
|
46
|
+
end
|
47
|
+
|
38
48
|
private
|
39
49
|
|
40
50
|
def build_default_server(keyw)
|
data/lib/woodhouse/rails.rb
CHANGED
@@ -19,27 +19,29 @@ if defined?(Rails::Railtie)
|
|
19
19
|
Woodhouse.extend Woodhouse::RailsExtensions
|
20
20
|
|
21
21
|
class Woodhouse::Rails < Rails::Engine
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
22
|
+
initializer 'woodhouse-defaults', before: :load_config_initializers do
|
23
|
+
# Legacy config file just containing AMQP information.
|
24
|
+
legacy_config_path = Rails.root.join("config/workling.yml")
|
25
|
+
# New config file containing any configuration options.
|
26
|
+
config_path = Rails.root.join("config/woodhouse.yml")
|
27
|
+
|
28
28
|
# Preload everything in app/workers so default layout includes them
|
29
29
|
Rails.root.join("app/workers").tap do |workers|
|
30
30
|
Pathname.glob(workers.join("**/*.rb")).each do |worker_path|
|
31
31
|
worker_path.relative_path_from(workers).basename(".rb").to_s.camelize.constantize
|
32
32
|
end
|
33
33
|
end
|
34
|
+
|
34
35
|
# Set up reasonable defaults
|
35
36
|
Woodhouse.configure do |config|
|
36
37
|
config.logger = ::Rails.logger
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
end
|
41
|
-
end
|
38
|
+
|
39
|
+
config.load_yaml legacy_config_path, section: "server_info", environment: ::Rails.env
|
40
|
+
config.load_yaml config_path, environment: ::Rails.env
|
42
41
|
end
|
42
|
+
end
|
43
|
+
|
44
|
+
initializer "woodhouse-layout" do
|
43
45
|
Woodhouse.finish_loading_layout!
|
44
46
|
end
|
45
47
|
end
|
data/lib/woodhouse/runners.rb
CHANGED
@@ -0,0 +1,80 @@
|
|
1
|
+
require 'fileutils'
|
2
|
+
|
3
|
+
class Woodhouse::Runners::FileRunner < Woodhouse::Runner
|
4
|
+
attr_accessor :jobs_dir, :queue_dir
|
5
|
+
|
6
|
+
DEFAULT_QUEUE_DIR = '/tmp/woodhouse/queue'
|
7
|
+
|
8
|
+
def initialize(worker, config)
|
9
|
+
super
|
10
|
+
|
11
|
+
server_info = config.server_info || {}
|
12
|
+
self.queue_dir = server_info[:path] || DEFAULT_QUEUE_DIR
|
13
|
+
self.jobs_dir = "#{queue_dir}/jobs"
|
14
|
+
|
15
|
+
unless File.directory?(jobs_dir) # subdirectory of queue_dir
|
16
|
+
config.logger.debug "[Woodhouse initialize] Creating queue directory '#{queue_dir}'"
|
17
|
+
FileUtils.mkdir_p jobs_dir
|
18
|
+
end
|
19
|
+
end
|
20
|
+
|
21
|
+
def subscribe
|
22
|
+
until @shutdown do
|
23
|
+
service_jobs
|
24
|
+
sleep 5
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
def spin_down
|
29
|
+
@shutdown = true
|
30
|
+
signal :spin_down
|
31
|
+
end
|
32
|
+
|
33
|
+
def service_jobs
|
34
|
+
each_job do |job,queue_id|
|
35
|
+
if can_service_job?(job)
|
36
|
+
reserve_job(queue_id) { service_job(job) }
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
def each_job(&block)
|
42
|
+
queue = Dir["#{queue_dir}/j-*"].sort
|
43
|
+
|
44
|
+
queue.each do |job_path|
|
45
|
+
job = YAML.load(File.read(job_path))
|
46
|
+
queue_id = File.basename(job_path)[2..-1]
|
47
|
+
|
48
|
+
yield(job, queue_id)
|
49
|
+
end
|
50
|
+
end
|
51
|
+
|
52
|
+
def reserve_job(queue_id, &block)
|
53
|
+
enqueued = "#{queue_dir}/j-#{queue_id}"
|
54
|
+
processing = "#{queue_dir}/p-#{queue_id}"
|
55
|
+
failed = "#{queue_dir}/f-#{queue_id}"
|
56
|
+
|
57
|
+
begin
|
58
|
+
FileUtils.mv(enqueued, processing)
|
59
|
+
|
60
|
+
if yield
|
61
|
+
# Success, clean up
|
62
|
+
File.unlink(processing)
|
63
|
+
end
|
64
|
+
|
65
|
+
rescue Errno::ENOENT
|
66
|
+
# Another worker beat us to the job
|
67
|
+
false
|
68
|
+
|
69
|
+
rescue => err
|
70
|
+
# Woodhouse internal error occurred during processing
|
71
|
+
File.open(processing, 'a') {|f| f.write YAML.dump(err) }
|
72
|
+
raise
|
73
|
+
|
74
|
+
ensure
|
75
|
+
# If file still hanging around then it failed
|
76
|
+
FileUtils.mv(processing, failed) if File.exists?(processing)
|
77
|
+
end
|
78
|
+
end
|
79
|
+
|
80
|
+
end
|
data/lib/woodhouse/scheduler.rb
CHANGED
@@ -52,7 +52,7 @@ class Woodhouse::Scheduler
|
|
52
52
|
@config.logger.debug "Spinning up thread #{idx} for worker #{@worker_def.describe}"
|
53
53
|
worker = @config.runner_type.new_link(@worker_def, @config)
|
54
54
|
@threads << worker
|
55
|
-
worker.subscribe
|
55
|
+
worker.async.subscribe
|
56
56
|
end
|
57
57
|
end
|
58
58
|
|
data/lib/woodhouse/version.rb
CHANGED
data/woodhouse.gemspec
CHANGED
@@ -8,6 +8,7 @@ Gem::Specification.new do |s|
|
|
8
8
|
s.platform = Gem::Platform::RUBY
|
9
9
|
s.authors = ["Matthew Boeh"]
|
10
10
|
s.email = ["matt@crowdcompass.com", "matthew.boeh@gmail.com"]
|
11
|
+
s.licenses = ["MIT"]
|
11
12
|
s.homepage = "http://github.com/mboeh/woodhouse"
|
12
13
|
s.summary = %q{An AMQP-based background worker system for Ruby}
|
13
14
|
s.description = %q{An AMQP-based background worker system for Ruby designed to make managing heterogenous tasks relatively easy.
|
@@ -18,10 +19,11 @@ Gem::Specification.new do |s|
|
|
18
19
|
|
19
20
|
s.rubyforge_project = "woodhouse"
|
20
21
|
|
21
|
-
s.add_dependency 'celluloid', '~> 0.
|
22
|
-
s.add_dependency 'bunny', "~> 0.9.
|
23
|
-
s.add_dependency 'connection_pool'
|
24
|
-
s.add_dependency 'json'
|
22
|
+
s.add_dependency 'celluloid', '~> 0.15'
|
23
|
+
s.add_dependency 'bunny', "~> 0.9.8"
|
24
|
+
s.add_dependency 'connection_pool', '~> 2.0'
|
25
|
+
s.add_dependency 'json', '~> 1.8'
|
26
|
+
s.add_dependency 'cause', '~> 0.1'
|
25
27
|
|
26
28
|
s.add_development_dependency 'rspec', '~> 1.3.1'
|
27
29
|
s.add_development_dependency 'rake'
|
metadata
CHANGED
@@ -1,84 +1,88 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: woodhouse
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
5
|
-
prerelease:
|
4
|
+
version: 1.0.0
|
6
5
|
platform: ruby
|
7
6
|
authors:
|
8
7
|
- Matthew Boeh
|
9
8
|
autorequire:
|
10
9
|
bindir: bin
|
11
10
|
cert_chain: []
|
12
|
-
date:
|
11
|
+
date: 2014-08-26 00:00:00.000000000 Z
|
13
12
|
dependencies:
|
14
13
|
- !ruby/object:Gem::Dependency
|
15
14
|
name: celluloid
|
16
15
|
requirement: !ruby/object:Gem::Requirement
|
17
|
-
none: false
|
18
16
|
requirements:
|
19
17
|
- - ~>
|
20
18
|
- !ruby/object:Gem::Version
|
21
|
-
version: 0.
|
19
|
+
version: '0.15'
|
22
20
|
type: :runtime
|
23
21
|
prerelease: false
|
24
22
|
version_requirements: !ruby/object:Gem::Requirement
|
25
|
-
none: false
|
26
23
|
requirements:
|
27
24
|
- - ~>
|
28
25
|
- !ruby/object:Gem::Version
|
29
|
-
version: 0.
|
26
|
+
version: '0.15'
|
30
27
|
- !ruby/object:Gem::Dependency
|
31
28
|
name: bunny
|
32
29
|
requirement: !ruby/object:Gem::Requirement
|
33
|
-
none: false
|
34
30
|
requirements:
|
35
31
|
- - ~>
|
36
32
|
- !ruby/object:Gem::Version
|
37
|
-
version: 0.9.
|
33
|
+
version: 0.9.8
|
38
34
|
type: :runtime
|
39
35
|
prerelease: false
|
40
36
|
version_requirements: !ruby/object:Gem::Requirement
|
41
|
-
none: false
|
42
37
|
requirements:
|
43
38
|
- - ~>
|
44
39
|
- !ruby/object:Gem::Version
|
45
|
-
version: 0.9.
|
40
|
+
version: 0.9.8
|
46
41
|
- !ruby/object:Gem::Dependency
|
47
42
|
name: connection_pool
|
48
43
|
requirement: !ruby/object:Gem::Requirement
|
49
|
-
none: false
|
50
44
|
requirements:
|
51
|
-
- -
|
45
|
+
- - ~>
|
52
46
|
- !ruby/object:Gem::Version
|
53
|
-
version: '0'
|
47
|
+
version: '2.0'
|
54
48
|
type: :runtime
|
55
49
|
prerelease: false
|
56
50
|
version_requirements: !ruby/object:Gem::Requirement
|
57
|
-
none: false
|
58
51
|
requirements:
|
59
|
-
- -
|
52
|
+
- - ~>
|
60
53
|
- !ruby/object:Gem::Version
|
61
|
-
version: '0'
|
54
|
+
version: '2.0'
|
62
55
|
- !ruby/object:Gem::Dependency
|
63
56
|
name: json
|
64
57
|
requirement: !ruby/object:Gem::Requirement
|
65
|
-
none: false
|
66
58
|
requirements:
|
67
|
-
- -
|
59
|
+
- - ~>
|
68
60
|
- !ruby/object:Gem::Version
|
69
|
-
version: '
|
61
|
+
version: '1.8'
|
70
62
|
type: :runtime
|
71
63
|
prerelease: false
|
72
64
|
version_requirements: !ruby/object:Gem::Requirement
|
73
|
-
none: false
|
74
65
|
requirements:
|
75
|
-
- -
|
66
|
+
- - ~>
|
76
67
|
- !ruby/object:Gem::Version
|
77
|
-
version: '
|
68
|
+
version: '1.8'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: cause
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ~>
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0.1'
|
76
|
+
type: :runtime
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ~>
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0.1'
|
78
83
|
- !ruby/object:Gem::Dependency
|
79
84
|
name: rspec
|
80
85
|
requirement: !ruby/object:Gem::Requirement
|
81
|
-
none: false
|
82
86
|
requirements:
|
83
87
|
- - ~>
|
84
88
|
- !ruby/object:Gem::Version
|
@@ -86,7 +90,6 @@ dependencies:
|
|
86
90
|
type: :development
|
87
91
|
prerelease: false
|
88
92
|
version_requirements: !ruby/object:Gem::Requirement
|
89
|
-
none: false
|
90
93
|
requirements:
|
91
94
|
- - ~>
|
92
95
|
- !ruby/object:Gem::Version
|
@@ -94,68 +97,60 @@ dependencies:
|
|
94
97
|
- !ruby/object:Gem::Dependency
|
95
98
|
name: rake
|
96
99
|
requirement: !ruby/object:Gem::Requirement
|
97
|
-
none: false
|
98
100
|
requirements:
|
99
|
-
- -
|
101
|
+
- - '>='
|
100
102
|
- !ruby/object:Gem::Version
|
101
103
|
version: '0'
|
102
104
|
type: :development
|
103
105
|
prerelease: false
|
104
106
|
version_requirements: !ruby/object:Gem::Requirement
|
105
|
-
none: false
|
106
107
|
requirements:
|
107
|
-
- -
|
108
|
+
- - '>='
|
108
109
|
- !ruby/object:Gem::Version
|
109
110
|
version: '0'
|
110
111
|
- !ruby/object:Gem::Dependency
|
111
112
|
name: guard
|
112
113
|
requirement: !ruby/object:Gem::Requirement
|
113
|
-
none: false
|
114
114
|
requirements:
|
115
|
-
- -
|
115
|
+
- - '>='
|
116
116
|
- !ruby/object:Gem::Version
|
117
117
|
version: '0'
|
118
118
|
type: :development
|
119
119
|
prerelease: false
|
120
120
|
version_requirements: !ruby/object:Gem::Requirement
|
121
|
-
none: false
|
122
121
|
requirements:
|
123
|
-
- -
|
122
|
+
- - '>='
|
124
123
|
- !ruby/object:Gem::Version
|
125
124
|
version: '0'
|
126
125
|
- !ruby/object:Gem::Dependency
|
127
126
|
name: guard-rspec
|
128
127
|
requirement: !ruby/object:Gem::Requirement
|
129
|
-
none: false
|
130
128
|
requirements:
|
131
|
-
- -
|
129
|
+
- - '>='
|
132
130
|
- !ruby/object:Gem::Version
|
133
131
|
version: '0'
|
134
132
|
type: :development
|
135
133
|
prerelease: false
|
136
134
|
version_requirements: !ruby/object:Gem::Requirement
|
137
|
-
none: false
|
138
135
|
requirements:
|
139
|
-
- -
|
136
|
+
- - '>='
|
140
137
|
- !ruby/object:Gem::Version
|
141
138
|
version: '0'
|
142
139
|
- !ruby/object:Gem::Dependency
|
143
140
|
name: mocha
|
144
141
|
requirement: !ruby/object:Gem::Requirement
|
145
|
-
none: false
|
146
142
|
requirements:
|
147
|
-
- -
|
143
|
+
- - '>='
|
148
144
|
- !ruby/object:Gem::Version
|
149
145
|
version: '0'
|
150
146
|
type: :development
|
151
147
|
prerelease: false
|
152
148
|
version_requirements: !ruby/object:Gem::Requirement
|
153
|
-
none: false
|
154
149
|
requirements:
|
155
|
-
- -
|
150
|
+
- - '>='
|
156
151
|
- !ruby/object:Gem::Version
|
157
152
|
version: '0'
|
158
|
-
description:
|
153
|
+
description: "An AMQP-based background worker system for Ruby designed to make managing
|
159
154
|
heterogenous tasks relatively easy.\n \n The use case for Woodhouse is for reliable
|
160
155
|
and sane performance in situations where jobs on a single queue may vary significantly
|
161
156
|
in length. The goal is to permit large numbers of quick jobs to be serviced even
|
@@ -187,6 +182,7 @@ files:
|
|
187
182
|
- lib/woodhouse/dispatchers.rb
|
188
183
|
- lib/woodhouse/dispatchers/bunny_dispatcher.rb
|
189
184
|
- lib/woodhouse/dispatchers/common_amqp_dispatcher.rb
|
185
|
+
- lib/woodhouse/dispatchers/file_dispatcher.rb
|
190
186
|
- lib/woodhouse/dispatchers/hot_bunnies_dispatcher.rb
|
191
187
|
- lib/woodhouse/dispatchers/local_dispatcher.rb
|
192
188
|
- lib/woodhouse/dispatchers/local_pool_dispatcher.rb
|
@@ -217,6 +213,7 @@ files:
|
|
217
213
|
- lib/woodhouse/runners.rb
|
218
214
|
- lib/woodhouse/runners/bunny_runner.rb
|
219
215
|
- lib/woodhouse/runners/dummy_runner.rb
|
216
|
+
- lib/woodhouse/runners/file_runner.rb
|
220
217
|
- lib/woodhouse/runners/hot_bunnies_runner.rb
|
221
218
|
- lib/woodhouse/scheduler.rb
|
222
219
|
- lib/woodhouse/server.rb
|
@@ -239,33 +236,41 @@ files:
|
|
239
236
|
- spec/worker_spec.rb
|
240
237
|
- woodhouse.gemspec
|
241
238
|
homepage: http://github.com/mboeh/woodhouse
|
242
|
-
licenses:
|
239
|
+
licenses:
|
240
|
+
- MIT
|
241
|
+
metadata: {}
|
243
242
|
post_install_message:
|
244
243
|
rdoc_options: []
|
245
244
|
require_paths:
|
246
245
|
- lib
|
247
246
|
required_ruby_version: !ruby/object:Gem::Requirement
|
248
|
-
none: false
|
249
247
|
requirements:
|
250
|
-
- -
|
248
|
+
- - '>='
|
251
249
|
- !ruby/object:Gem::Version
|
252
250
|
version: '0'
|
253
|
-
segments:
|
254
|
-
- 0
|
255
|
-
hash: -1311770161800686581
|
256
251
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
257
|
-
none: false
|
258
252
|
requirements:
|
259
|
-
- -
|
253
|
+
- - '>='
|
260
254
|
- !ruby/object:Gem::Version
|
261
255
|
version: '0'
|
262
|
-
segments:
|
263
|
-
- 0
|
264
|
-
hash: -1311770161800686581
|
265
256
|
requirements: []
|
266
257
|
rubyforge_project: woodhouse
|
267
|
-
rubygems_version:
|
258
|
+
rubygems_version: 2.2.2
|
268
259
|
signing_key:
|
269
|
-
specification_version:
|
260
|
+
specification_version: 4
|
270
261
|
summary: An AMQP-based background worker system for Ruby
|
271
|
-
test_files:
|
262
|
+
test_files:
|
263
|
+
- spec/integration/bunny_worker_process_spec.rb
|
264
|
+
- spec/layout_builder_spec.rb
|
265
|
+
- spec/layout_spec.rb
|
266
|
+
- spec/middleware_stack_spec.rb
|
267
|
+
- spec/mixin_registry_spec.rb
|
268
|
+
- spec/node_configuration_spec.rb
|
269
|
+
- spec/progress_spec.rb
|
270
|
+
- spec/queue_criteria_spec.rb
|
271
|
+
- spec/scheduler_spec.rb
|
272
|
+
- spec/server_spec.rb
|
273
|
+
- spec/shared_contexts.rb
|
274
|
+
- spec/test_dispatcher_spec.rb
|
275
|
+
- spec/worker_spec.rb
|
276
|
+
has_rdoc:
|