workers 0.2.2 → 0.3.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 4b66f1edc747b5df57e671c9efe7abbd4f36a863
4
+ data.tar.gz: 16b9fcb187bfda5a629160de3b0c742c66f5f1b4
5
+ SHA512:
6
+ metadata.gz: 1c4db47993b135c9dfbb6d8feb703ee82946bd67f98a953ef00673e5b83d432426764ab84e2604650b46628c55e11172b76940960024b26a1a6b9cf5a924cb06
7
+ data.tar.gz: 4d1a0306500a6e26741d0516ae0906b7ad9c9f5fc2e6ab76561174ab73c8dada2a88d683a4d3c2108d90b62b79c0a96e72f7f95716023c1238cea1cf36fd75bd
data/README.md CHANGED
@@ -104,6 +104,8 @@ This method uses a mutex so you can serialize portions of your tasks that aren't
104
104
 
105
105
  The main purpose of the Worker class is to add an event system on top of Ruby's built-in Thread class.
106
106
  This greatly simplifies inter-thread communication.
107
+ Workers are fairly low level and don't handle exceptions for you.
108
+ Failing to handle exceptions will result in dead workers so you must rescue them in your application code.
107
109
 
108
110
  # Initialize a worker pool.
109
111
  pool = Workers::Pool.new
@@ -111,8 +113,12 @@ This greatly simplifies inter-thread communication.
111
113
  # Perform some work in the background.
112
114
  100.times do
113
115
  pool.perform do
114
- sleep(rand(3))
115
- puts "Hello world from thread #{Thread.current.object_id}"
116
+ begin
117
+ sleep(rand(3))
118
+ puts "Hello world from thread #{Thread.current.object_id}"
119
+ rescue Exception => e
120
+ puts "Oh no, my hello world failed!"
121
+ end
116
122
  end
117
123
  end
118
124
 
@@ -134,8 +140,12 @@ The Worker class is designed to be customized through inheritence and its event
134
140
  def process_event(event)
135
141
  case event.command
136
142
  when :my_custom
137
- puts "Worker received custom event: #{event.data}"
138
- sleep(1)
143
+ begin
144
+ puts "Worker received custom event: #{event.data}"
145
+ sleep(1)
146
+ rescue Exception => e
147
+ puts "This is a very sad program."
148
+ end
139
149
  end
140
150
  end
141
151
  end
@@ -156,8 +166,30 @@ The Worker class is designed to be customized through inheritence and its event
156
166
  # Wait for the workers to shutdown.
157
167
  pool.join
158
168
 
159
- Note that you can use custom workers without a pool.
169
+ #### Without pools
170
+
171
+ In most cases you will be using a group of workers (a pool) as demonstrated above.
172
+ In certain cases, you may want to use a worker directly without the pool.
160
173
  This effectively gives you direct access to a single event-driven thread.
174
+ Note that you must handle exceptions yourself since you are working directly with the worker class:
175
+
176
+ # Create a single worker.
177
+ worker = Workers::Worker.new
178
+
179
+ # Perform some work in the background.
180
+ 25.times do |i|
181
+ worker.perform do
182
+ begin
183
+ sleep(0.1)
184
+ puts "Hello world from thread #{Thread.current.object_id}"
185
+ rescue Exception => e
186
+ puts "Oh no, my hello world failed!"
187
+ end
188
+ end
189
+ end
190
+
191
+ # Tell the worker to shutdown.
192
+ worker.shutdown
161
193
 
162
194
  #### Options
163
195
 
@@ -197,7 +229,8 @@ Pools can be adjusted using the below methods:
197
229
 
198
230
  ## Timers
199
231
 
200
- Timers provide a way to execute code in the future:
232
+ Timers provide a way to execute code in the future.
233
+ You can easily use them to tell a Worker or it's higher level classes (Task, TaskGroup, etc) to perform work in the future.
201
234
 
202
235
  # Create a one shot timer that executes in 1.5 seconds.
203
236
  timer = Workers::Timer.new(1.5) do
@@ -261,6 +294,41 @@ You can create additional schedulers as necessary:
261
294
  :pool => Workers::Pool.new # The workers pool used to execute timer callbacks.
262
295
  )
263
296
 
297
+ ## Concurrency and performance
298
+
299
+ Workers is tested with both JRuby and MRI (C Ruby).
300
+ Below are some notes specific to each Ruby implementation.
301
+ In summary, JRuby is the recommended Ruby to use with Workers since it provides the highest performance with multiple CPU cores.
302
+
303
+ #### JRuby (recommended)
304
+
305
+ JRuby is designed for multi-threaded apps running on multiple cores.
306
+ When used with Workers, you will be able to saturate all of your CPU cores with little to no tuning.
307
+ It is highly recommended you increase the number of workers in your pool if you have a large number of cores.
308
+ A good starting point is 1x - 2x the number of cores for CPU bound apps.
309
+ For IO bound apps you will need to do some benchmarking to figure out what is best for you.
310
+ A good starting point is 4x - 10x the number of cores for IO bound apps.
311
+
312
+ #### MRI 1.9.x or newer (supported)
313
+
314
+ MRI 1.9 and above use real operating system threads with a global interpreter lock (GIL).
315
+ The bad news is that due to the GIL, only one thread can execute Ruby code at a given point in time.
316
+ This means your app will be CPU bound to a single core.
317
+ The good news is that IO bound applications will still see huge benefits from Workers.
318
+ Examples of such IO are web service requests, web servers, web crawlers, database queries, writing to disk, etc.
319
+ Threads performing such IO will temporarily release the GIL and thus let another thread execute Ruby code.
320
+
321
+ #### MRI 1.8.x or older (not supported)
322
+
323
+ These old versions of Ruby used green threads (application layer threads) instead of operating system level threads.
324
+ I recommend you upgrade to a newer version as I haven't tested Workers with them.
325
+ They also aren't officially supported by the Ruby community at this point.
326
+
327
+ #### Rubinius
328
+
329
+ I haven't tested Workers with Rubinius, but in theory it should just work.
330
+ The above JRuby notes should apply.
331
+
264
332
  ## Contributing
265
333
 
266
334
  1. Fork it
data/lib/workers.rb CHANGED
@@ -9,6 +9,7 @@ require 'workers/pool'
9
9
  require 'workers/event'
10
10
  require 'workers/log_proxy'
11
11
  require 'workers/scheduler'
12
+ require 'workers/bucket_scheduler'
12
13
  require 'workers/timer'
13
14
  require 'workers/periodic_timer'
14
15
  require 'workers/task'
@@ -16,30 +17,41 @@ require 'workers/task_group'
16
17
 
17
18
  module Workers
18
19
  def self.pool
19
- return @pool ||= Workers::Pool.new
20
+ lock do
21
+ return @pool ||= Workers::Pool.new
22
+ end
20
23
  end
21
24
 
22
25
  def self.pool=(val)
23
- @pool.dispose if @pool
24
- @pool = val
26
+ lock do
27
+ @pool.dispose if @pool
28
+ @pool = val
29
+ end
25
30
  end
26
31
 
27
32
  def self.scheduler
28
- return @scheduler ||= Workers::Scheduler.new
33
+ lock do
34
+ return @scheduler ||= Workers::Scheduler.new
35
+ end
29
36
  end
30
37
 
31
38
  def self.scheduler=(val)
32
- @scheduler.dispose if @scheduler
33
- @scheduler = val
39
+ lock do
40
+ @scheduler.dispose if @scheduler
41
+ @scheduler = val
42
+ end
34
43
  end
35
44
 
36
45
  def self.map(inputs, options = {}, &block)
37
46
  return Workers::TaskGroup.new.map(inputs, options) do |i|
38
- block.call(i)
47
+ yield(i)
39
48
  end
40
49
  end
50
+
51
+ def self.lock(&block)
52
+ (@lock ||= Monitor.new).synchronize { yield if block_given? }
53
+ end
41
54
  end
42
55
 
43
56
  # Force initialization of defaults.
44
- Workers.pool
45
- Workers.scheduler
57
+ Workers.lock
@@ -0,0 +1,46 @@
1
+ module Workers
2
+ class BucketScheduler
3
+ DEFAULT_BUCKET_SIZE = 100
4
+ DEFAULT_POOL_SIZE = 1
5
+
6
+ def initialize(options = {})
7
+ options[:bucket_size] ||= DEFAULT_BUCKET_SIZE
8
+ options[:pool_size] ||= DEFAULT_POOL_SIZE
9
+
10
+ @logger = Workers::LogProxy.new(options[:logger])
11
+ @options = options
12
+
13
+ @schedulers = (0...(options[:bucket_size])).map {
14
+ Workers::Scheduler.new(:pool => Workers::Pool.new(:logger => @logger, :size => options[:pool_size]))
15
+ }
16
+ end
17
+
18
+ def schedule(timer)
19
+ @schedulers[timer.hash % @options[:bucket_size]].schedule(timer)
20
+
21
+ return nil
22
+ end
23
+
24
+ def unschedule(timer)
25
+ @schedulers[timer.hash % @options[:bucket_size]].unschedule(timer)
26
+
27
+ return nil
28
+ end
29
+
30
+ def wakeup
31
+ @schedulers.each { |s| s.wakeup }
32
+
33
+ return nil
34
+ end
35
+
36
+ def dispose
37
+ @schedulers.each { |s| s.dispose }
38
+
39
+ return nil
40
+ end
41
+
42
+ def alive?
43
+ return @schedulers.all? { |s| s.alive? }
44
+ end
45
+ end
46
+ end
@@ -4,7 +4,7 @@ module Workers
4
4
 
5
5
  def initialize(options = {})
6
6
  @logger = Workers::LogProxy.new(options[:logger])
7
- @pool = options[:pool] || Workers.pool
7
+ @pool = options[:pool] || Workers::Pool.new
8
8
  @schedule = SortedSet.new
9
9
  @mutex = Mutex.new
10
10
  @thread = Thread.new { start_loop }
@@ -27,7 +27,7 @@ module Workers
27
27
  @schedule.delete(timer)
28
28
  end
29
29
 
30
- return true
30
+ return nil
31
31
  end
32
32
 
33
33
  def wakeup
@@ -1,3 +1,3 @@
1
1
  module Workers
2
- VERSION = '0.2.2'
2
+ VERSION = '0.3.0'
3
3
  end
metadata CHANGED
@@ -1,15 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: workers
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.2
5
- prerelease:
4
+ version: 0.3.0
6
5
  platform: ruby
7
6
  authors:
8
7
  - Chad Remesch
9
8
  autorequire:
10
9
  bindir: bin
11
10
  cert_chain: []
12
- date: 2013-06-03 00:00:00.000000000 Z
11
+ date: 2015-02-23 00:00:00.000000000 Z
13
12
  dependencies: []
14
13
  description: Workers is a Ruby gem for performing work in background threads.
15
14
  email:
@@ -18,12 +17,13 @@ executables: []
18
17
  extensions: []
19
18
  extra_rdoc_files: []
20
19
  files:
21
- - .gitignore
20
+ - ".gitignore"
22
21
  - Gemfile
23
22
  - LICENSE.txt
24
23
  - README.md
25
24
  - Rakefile
26
25
  - lib/workers.rb
26
+ - lib/workers/bucket_scheduler.rb
27
27
  - lib/workers/event.rb
28
28
  - lib/workers/helpers.rb
29
29
  - lib/workers/log_proxy.rb
@@ -38,27 +38,26 @@ files:
38
38
  - workers.gemspec
39
39
  homepage: https://github.com/chadrem/workers
40
40
  licenses: []
41
+ metadata: {}
41
42
  post_install_message:
42
43
  rdoc_options: []
43
44
  require_paths:
44
45
  - lib
45
46
  required_ruby_version: !ruby/object:Gem::Requirement
46
- none: false
47
47
  requirements:
48
- - - ! '>='
48
+ - - ">="
49
49
  - !ruby/object:Gem::Version
50
50
  version: '0'
51
51
  required_rubygems_version: !ruby/object:Gem::Requirement
52
- none: false
53
52
  requirements:
54
- - - ! '>='
53
+ - - ">="
55
54
  - !ruby/object:Gem::Version
56
55
  version: '0'
57
56
  requirements: []
58
57
  rubyforge_project:
59
- rubygems_version: 1.8.24
58
+ rubygems_version: 2.4.5
60
59
  signing_key:
61
- specification_version: 3
60
+ specification_version: 4
62
61
  summary: Workers is a Ruby gem for performing work in background threads. Design goals
63
62
  include high performance, low latency, simple API, customizability, and multi-layered
64
63
  architecture. It provides a number of simple to use classes that solve a wide range