workers 0.2.2 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/README.md +74 -6
- data/lib/workers.rb +21 -9
- data/lib/workers/bucket_scheduler.rb +46 -0
- data/lib/workers/scheduler.rb +2 -2
- data/lib/workers/version.rb +1 -1
- metadata +9 -10
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 4b66f1edc747b5df57e671c9efe7abbd4f36a863
|
4
|
+
data.tar.gz: 16b9fcb187bfda5a629160de3b0c742c66f5f1b4
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 1c4db47993b135c9dfbb6d8feb703ee82946bd67f98a953ef00673e5b83d432426764ab84e2604650b46628c55e11172b76940960024b26a1a6b9cf5a924cb06
|
7
|
+
data.tar.gz: 4d1a0306500a6e26741d0516ae0906b7ad9c9f5fc2e6ab76561174ab73c8dada2a88d683a4d3c2108d90b62b79c0a96e72f7f95716023c1238cea1cf36fd75bd
|
data/README.md
CHANGED
@@ -104,6 +104,8 @@ This method uses a mutex so you can serialize portions of your tasks that aren't
|
|
104
104
|
|
105
105
|
The main purpose of the Worker class is to add an event system on top of Ruby's built-in Thread class.
|
106
106
|
This greatly simplifies inter-thread communication.
|
107
|
+
Workers are fairly low level and don't handle exceptions for you.
|
108
|
+
Failing to handle exceptions will result in dead workers so you must rescue them in your application code.
|
107
109
|
|
108
110
|
# Initialize a worker pool.
|
109
111
|
pool = Workers::Pool.new
|
@@ -111,8 +113,12 @@ This greatly simplifies inter-thread communication.
|
|
111
113
|
# Perform some work in the background.
|
112
114
|
100.times do
|
113
115
|
pool.perform do
|
114
|
-
|
115
|
-
|
116
|
+
begin
|
117
|
+
sleep(rand(3))
|
118
|
+
puts "Hello world from thread #{Thread.current.object_id}"
|
119
|
+
rescue Exception => e
|
120
|
+
puts "Oh no, my hello world failed!"
|
121
|
+
end
|
116
122
|
end
|
117
123
|
end
|
118
124
|
|
@@ -134,8 +140,12 @@ The Worker class is designed to be customized through inheritence and its event
|
|
134
140
|
def process_event(event)
|
135
141
|
case event.command
|
136
142
|
when :my_custom
|
137
|
-
|
138
|
-
|
143
|
+
begin
|
144
|
+
puts "Worker received custom event: #{event.data}"
|
145
|
+
sleep(1)
|
146
|
+
rescue Exception => e
|
147
|
+
puts "This is a very sad program."
|
148
|
+
end
|
139
149
|
end
|
140
150
|
end
|
141
151
|
end
|
@@ -156,8 +166,30 @@ The Worker class is designed to be customized through inheritence and its event
|
|
156
166
|
# Wait for the workers to shutdown.
|
157
167
|
pool.join
|
158
168
|
|
159
|
-
|
169
|
+
#### Without pools
|
170
|
+
|
171
|
+
In most cases you will be using a group of workers (a pool) as demonstrated above.
|
172
|
+
In certain cases, you may want to use a worker directly without the pool.
|
160
173
|
This effectively gives you direct access to a single event-driven thread.
|
174
|
+
Note that you must handle exceptions yourself since you are working directly with the worker class:
|
175
|
+
|
176
|
+
# Create a single worker.
|
177
|
+
worker = Workers::Worker.new
|
178
|
+
|
179
|
+
# Perform some work in the background.
|
180
|
+
25.times do |i|
|
181
|
+
worker.perform do
|
182
|
+
begin
|
183
|
+
sleep(0.1)
|
184
|
+
puts "Hello world from thread #{Thread.current.object_id}"
|
185
|
+
rescue Exception => e
|
186
|
+
puts "Oh no, my hello world failed!"
|
187
|
+
end
|
188
|
+
end
|
189
|
+
end
|
190
|
+
|
191
|
+
# Tell the worker to shutdown.
|
192
|
+
worker.shutdown
|
161
193
|
|
162
194
|
#### Options
|
163
195
|
|
@@ -197,7 +229,8 @@ Pools can be adjusted using the below methods:
|
|
197
229
|
|
198
230
|
## Timers
|
199
231
|
|
200
|
-
Timers provide a way to execute code in the future
|
232
|
+
Timers provide a way to execute code in the future.
|
233
|
+
You can easily use them to tell a Worker or it's higher level classes (Task, TaskGroup, etc) to perform work in the future.
|
201
234
|
|
202
235
|
# Create a one shot timer that executes in 1.5 seconds.
|
203
236
|
timer = Workers::Timer.new(1.5) do
|
@@ -261,6 +294,41 @@ You can create additional schedulers as necessary:
|
|
261
294
|
:pool => Workers::Pool.new # The workers pool used to execute timer callbacks.
|
262
295
|
)
|
263
296
|
|
297
|
+
## Concurrency and performance
|
298
|
+
|
299
|
+
Workers is tested with both JRuby and MRI (C Ruby).
|
300
|
+
Below are some notes specific to each Ruby implementation.
|
301
|
+
In summary, JRuby is the recommended Ruby to use with Workers since it provides the highest performance with multiple CPU cores.
|
302
|
+
|
303
|
+
#### JRuby (recommended)
|
304
|
+
|
305
|
+
JRuby is designed for multi-threaded apps running on multiple cores.
|
306
|
+
When used with Workers, you will be able to saturate all of your CPU cores with little to no tuning.
|
307
|
+
It is highly recommended you increase the number of workers in your pool if you have a large number of cores.
|
308
|
+
A good starting point is 1x - 2x the number of cores for CPU bound apps.
|
309
|
+
For IO bound apps you will need to do some benchmarking to figure out what is best for you.
|
310
|
+
A good starting point is 4x - 10x the number of cores for IO bound apps.
|
311
|
+
|
312
|
+
#### MRI 1.9.x or newer (supported)
|
313
|
+
|
314
|
+
MRI 1.9 and above use real operating system threads with a global interpreter lock (GIL).
|
315
|
+
The bad news is that due to the GIL, only one thread can execute Ruby code at a given point in time.
|
316
|
+
This means your app will be CPU bound to a single core.
|
317
|
+
The good news is that IO bound applications will still see huge benefits from Workers.
|
318
|
+
Examples of such IO are web service requests, web servers, web crawlers, database queries, writing to disk, etc.
|
319
|
+
Threads performing such IO will temporarily release the GIL and thus let another thread execute Ruby code.
|
320
|
+
|
321
|
+
#### MRI 1.8.x or older (not supported)
|
322
|
+
|
323
|
+
These old versions of Ruby used green threads (application layer threads) instead of operating system level threads.
|
324
|
+
I recommend you upgrade to a newer version as I haven't tested Workers with them.
|
325
|
+
They also aren't officially supported by the Ruby community at this point.
|
326
|
+
|
327
|
+
#### Rubinius
|
328
|
+
|
329
|
+
I haven't tested Workers with Rubinius, but in theory it should just work.
|
330
|
+
The above JRuby notes should apply.
|
331
|
+
|
264
332
|
## Contributing
|
265
333
|
|
266
334
|
1. Fork it
|
data/lib/workers.rb
CHANGED
@@ -9,6 +9,7 @@ require 'workers/pool'
|
|
9
9
|
require 'workers/event'
|
10
10
|
require 'workers/log_proxy'
|
11
11
|
require 'workers/scheduler'
|
12
|
+
require 'workers/bucket_scheduler'
|
12
13
|
require 'workers/timer'
|
13
14
|
require 'workers/periodic_timer'
|
14
15
|
require 'workers/task'
|
@@ -16,30 +17,41 @@ require 'workers/task_group'
|
|
16
17
|
|
17
18
|
module Workers
|
18
19
|
def self.pool
|
19
|
-
|
20
|
+
lock do
|
21
|
+
return @pool ||= Workers::Pool.new
|
22
|
+
end
|
20
23
|
end
|
21
24
|
|
22
25
|
def self.pool=(val)
|
23
|
-
|
24
|
-
|
26
|
+
lock do
|
27
|
+
@pool.dispose if @pool
|
28
|
+
@pool = val
|
29
|
+
end
|
25
30
|
end
|
26
31
|
|
27
32
|
def self.scheduler
|
28
|
-
|
33
|
+
lock do
|
34
|
+
return @scheduler ||= Workers::Scheduler.new
|
35
|
+
end
|
29
36
|
end
|
30
37
|
|
31
38
|
def self.scheduler=(val)
|
32
|
-
|
33
|
-
|
39
|
+
lock do
|
40
|
+
@scheduler.dispose if @scheduler
|
41
|
+
@scheduler = val
|
42
|
+
end
|
34
43
|
end
|
35
44
|
|
36
45
|
def self.map(inputs, options = {}, &block)
|
37
46
|
return Workers::TaskGroup.new.map(inputs, options) do |i|
|
38
|
-
|
47
|
+
yield(i)
|
39
48
|
end
|
40
49
|
end
|
50
|
+
|
51
|
+
def self.lock(&block)
|
52
|
+
(@lock ||= Monitor.new).synchronize { yield if block_given? }
|
53
|
+
end
|
41
54
|
end
|
42
55
|
|
43
56
|
# Force initialization of defaults.
|
44
|
-
Workers.
|
45
|
-
Workers.scheduler
|
57
|
+
Workers.lock
|
@@ -0,0 +1,46 @@
|
|
1
|
+
module Workers
|
2
|
+
class BucketScheduler
|
3
|
+
DEFAULT_BUCKET_SIZE = 100
|
4
|
+
DEFAULT_POOL_SIZE = 1
|
5
|
+
|
6
|
+
def initialize(options = {})
|
7
|
+
options[:bucket_size] ||= DEFAULT_BUCKET_SIZE
|
8
|
+
options[:pool_size] ||= DEFAULT_POOL_SIZE
|
9
|
+
|
10
|
+
@logger = Workers::LogProxy.new(options[:logger])
|
11
|
+
@options = options
|
12
|
+
|
13
|
+
@schedulers = (0...(options[:bucket_size])).map {
|
14
|
+
Workers::Scheduler.new(:pool => Workers::Pool.new(:logger => @logger, :size => options[:pool_size]))
|
15
|
+
}
|
16
|
+
end
|
17
|
+
|
18
|
+
def schedule(timer)
|
19
|
+
@schedulers[timer.hash % @options[:bucket_size]].schedule(timer)
|
20
|
+
|
21
|
+
return nil
|
22
|
+
end
|
23
|
+
|
24
|
+
def unschedule(timer)
|
25
|
+
@schedulers[timer.hash % @options[:bucket_size]].unschedule(timer)
|
26
|
+
|
27
|
+
return nil
|
28
|
+
end
|
29
|
+
|
30
|
+
def wakeup
|
31
|
+
@schedulers.each { |s| s.wakeup }
|
32
|
+
|
33
|
+
return nil
|
34
|
+
end
|
35
|
+
|
36
|
+
def dispose
|
37
|
+
@schedulers.each { |s| s.dispose }
|
38
|
+
|
39
|
+
return nil
|
40
|
+
end
|
41
|
+
|
42
|
+
def alive?
|
43
|
+
return @schedulers.all? { |s| s.alive? }
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
data/lib/workers/scheduler.rb
CHANGED
@@ -4,7 +4,7 @@ module Workers
|
|
4
4
|
|
5
5
|
def initialize(options = {})
|
6
6
|
@logger = Workers::LogProxy.new(options[:logger])
|
7
|
-
@pool = options[:pool] || Workers.
|
7
|
+
@pool = options[:pool] || Workers::Pool.new
|
8
8
|
@schedule = SortedSet.new
|
9
9
|
@mutex = Mutex.new
|
10
10
|
@thread = Thread.new { start_loop }
|
@@ -27,7 +27,7 @@ module Workers
|
|
27
27
|
@schedule.delete(timer)
|
28
28
|
end
|
29
29
|
|
30
|
-
return
|
30
|
+
return nil
|
31
31
|
end
|
32
32
|
|
33
33
|
def wakeup
|
data/lib/workers/version.rb
CHANGED
metadata
CHANGED
@@ -1,15 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: workers
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
5
|
-
prerelease:
|
4
|
+
version: 0.3.0
|
6
5
|
platform: ruby
|
7
6
|
authors:
|
8
7
|
- Chad Remesch
|
9
8
|
autorequire:
|
10
9
|
bindir: bin
|
11
10
|
cert_chain: []
|
12
|
-
date:
|
11
|
+
date: 2015-02-23 00:00:00.000000000 Z
|
13
12
|
dependencies: []
|
14
13
|
description: Workers is a Ruby gem for performing work in background threads.
|
15
14
|
email:
|
@@ -18,12 +17,13 @@ executables: []
|
|
18
17
|
extensions: []
|
19
18
|
extra_rdoc_files: []
|
20
19
|
files:
|
21
|
-
- .gitignore
|
20
|
+
- ".gitignore"
|
22
21
|
- Gemfile
|
23
22
|
- LICENSE.txt
|
24
23
|
- README.md
|
25
24
|
- Rakefile
|
26
25
|
- lib/workers.rb
|
26
|
+
- lib/workers/bucket_scheduler.rb
|
27
27
|
- lib/workers/event.rb
|
28
28
|
- lib/workers/helpers.rb
|
29
29
|
- lib/workers/log_proxy.rb
|
@@ -38,27 +38,26 @@ files:
|
|
38
38
|
- workers.gemspec
|
39
39
|
homepage: https://github.com/chadrem/workers
|
40
40
|
licenses: []
|
41
|
+
metadata: {}
|
41
42
|
post_install_message:
|
42
43
|
rdoc_options: []
|
43
44
|
require_paths:
|
44
45
|
- lib
|
45
46
|
required_ruby_version: !ruby/object:Gem::Requirement
|
46
|
-
none: false
|
47
47
|
requirements:
|
48
|
-
- -
|
48
|
+
- - ">="
|
49
49
|
- !ruby/object:Gem::Version
|
50
50
|
version: '0'
|
51
51
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
52
|
-
none: false
|
53
52
|
requirements:
|
54
|
-
- -
|
53
|
+
- - ">="
|
55
54
|
- !ruby/object:Gem::Version
|
56
55
|
version: '0'
|
57
56
|
requirements: []
|
58
57
|
rubyforge_project:
|
59
|
-
rubygems_version:
|
58
|
+
rubygems_version: 2.4.5
|
60
59
|
signing_key:
|
61
|
-
specification_version:
|
60
|
+
specification_version: 4
|
62
61
|
summary: Workers is a Ruby gem for performing work in background threads. Design goals
|
63
62
|
include high performance, low latency, simple API, customizability, and multi-layered
|
64
63
|
architecture. It provides a number of simple to use classes that solve a wide range
|