hash_queue 0.1.0

Sign up to get free protection for your applications and to get access to all the features.
data/.gitignore ADDED
@@ -0,0 +1,17 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ Gemfile.lock
7
+ InstalledFiles
8
+ _yardoc
9
+ coverage
10
+ doc/
11
+ lib/bundler/man
12
+ pkg
13
+ rdoc
14
+ spec/reports
15
+ test/tmp
16
+ test/version_tmp
17
+ tmp
data/.rvmrc ADDED
@@ -0,0 +1 @@
1
+ rvm use 1.9.3@hash-queue --create
data/CHANGELOG.md ADDED
@@ -0,0 +1,5 @@
1
+ # Changelog
2
+
3
+ ## 0.1
4
+
5
+ + Initial release
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source 'https://rubygems.org'
2
+
3
+ gemspec
data/LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2012 Michal Krejčí
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,285 @@
1
+ # HashQueue
2
+
3
+ Simple namespaced Queue for highly concurrent environments. Maintains separate queues for different keys accessible from numerous threads without a trouble. Think of it as an extension to stdlib's Queue class. It could be even used as its replacement. Features nice locking capabilities, read on.
4
+
5
+ ## Installation
6
+
7
+ Add this line to your application's Gemfile:
8
+
9
+ gem 'hash_queue', '~> 0.1'
10
+
11
+ And then execute:
12
+
13
+ $ bundle
14
+
15
+ Or install it yourself as:
16
+
17
+ $ gem install hash_queue
18
+
19
+ For those living on the edge:
20
+
21
+ gem 'hash_queue', git: 'git@github.com:mikekreeki/hash_queue.git'
22
+
23
+ ..with Bundler 1.1 or newer:
24
+
25
+ gem 'mimi', github: 'mikekreeki/hash_queue'
26
+
27
+ ## Usage
28
+
29
+ To initialize new HashQueue just simply do:
30
+
31
+ ```ruby
32
+ require 'hash_queue'
33
+
34
+ hash_queue = HashQueue.new
35
+ ```
36
+
37
+ ### Queueing
38
+
39
+ To queue stuff it's just as easy as:
40
+
41
+ ```ruby
42
+ hash_queue[:my_queue].queue Stuff.new
43
+
44
+ # or
45
+
46
+ hash_queue.queue :my_queue, Stuff.new
47
+ ```
48
+
49
+ Keys (or namespaces if you prefer) can be anything you want. Usually those will be symbols or strings but don't need to be. Objects, classes, numbers or even `true` or `nil` will work. Same applies for queued items.
50
+
51
+ `#queue` is aliased as `#enqueue` and `#push` for convenience on both hash_queue and individual queues.
52
+
53
+ ### Working with hash_queue as a whole
54
+
55
+ HashQueue instances offer some handy methods.
56
+
57
+ ##### size
58
+
59
+ Returns combined sizes of all queues managed.
60
+
61
+ ```ruby
62
+ hash_queue = HashQueue.new
63
+ hash_queue.queue :animals, "tenderlove's cat"
64
+ hash_queue.queue :favourite_rubies, 'rubinius'
65
+ hash_queue.size # => 2
66
+ ```
67
+
68
+ ##### empty?
69
+
70
+ Returns `true` or `false` whether hash_queue is empty or not.
71
+
72
+ ```ruby
73
+ hash_queue = HashQueue.new
74
+ hash_queue.empty? # => true
75
+ hash_queue.queue :animals, "tenderlove's cat"
76
+ hash_queue.empty? # => false
77
+ ```
78
+
79
+ ##### clear
80
+
81
+ Removes all managed queues.
82
+
83
+ ```ruby
84
+ hash_queue = HashQueue.new
85
+ hash_queue.queue :animals, "tenderlove's cat"
86
+ hash_queue.empty? # => false
87
+ hash_queue.clear
88
+ hash_queue.empty? # => true
89
+ ```
90
+
91
+ ##### clean
92
+
93
+ When using lots of different keys (e.g. hostnames) your HashQueue instance will eventually grow in size. `#clean` removes empty queues to save precious RAM and speed things up. It's just a maintenance method, it won't break anything, you can use individual queues later the same way you're doing now, they will be automatically redeclared when you touch them.
94
+
95
+ ##### keys
96
+
97
+ Returns maintained keys.
98
+
99
+ ```ruby
100
+ hash_queue = HashQueue.new
101
+ hash_queue.queue :foo, 'Now or never.'
102
+ hash_queue.keys # => [:foo]
103
+ ```
104
+
105
+ Not very interesting but comes handy, for interesting stuff check Popping and Locking section down below.
106
+
107
+ ### Working with individual queues
108
+
109
+ To get a specific queue you want to put your hands on, just do:
110
+
111
+ ```ruby
112
+ hash_queue[:my_queue]
113
+ ```
114
+
115
+ It also features some not that fancy methods like `#size`, `#empty?` and `#clear`.
116
+
117
+ It's also possible to bypass namespacing capabilities and use HashQueue's individual queue directly as a replacement for stdlib's Queue. Just initialize new queue:
118
+
119
+ ```ruby
120
+ queue = HashQueue::Queue.new
121
+ queue.push :foo
122
+ queue.size # => 1
123
+ queue.pop # => :foo
124
+ queue.empty? # => true
125
+ ```
126
+
127
+ ### Popping stuff out of the queues
128
+
129
+ There are two way to pop stuff from HashQueue instance. One way is to pop from all the queues at once. When you call `pop` on HashQueue instance it will pop one item from each individual queue it manages (unless it's locked). Always returns an Array.
130
+
131
+ ```ruby
132
+ hash_queue = HashQueue.new
133
+ hash_queue.queue :animals, :cat
134
+ hash_queue.queue :animals, :dog
135
+ hash_queue.queue :rubyists, :tenderlove
136
+ hash_queue.queue :rubyists, :yehuda
137
+ hash_queue.queue :rubyists, :matz
138
+
139
+ hash_queue.pop # => [:cat, :tenderlove]
140
+ hash_queue.pop # => [:dog, :yehuda]
141
+ hash_queue.pop # => [:matz]
142
+ ```
143
+
144
+ `#pop` method takes optional `:size` option with which you can specify how many items you want to pop out of each queue.
145
+
146
+ ```ruby
147
+ hash_queue = HashQueue.new
148
+
149
+ 10.times { |i| hash_queue.queue :queue_1, i }
150
+ 10.times { |i| hash_queue.queue :queue_2, i + 100 }
151
+
152
+ hash_queue.pop(size: 2) # => [0, 1, 100, 101]
153
+ ```
154
+
155
+ Or you can provide `:blocking` option. When called with `blocking: true` it won't return until there's something to return. When there's nothing to be returned (either all individual queues are empty, locked or there aren't any) the call holds until something is queued from a background thread or some queue is unlocked.
156
+
157
+ ```ruby
158
+ hash_queue = HashQueue.new
159
+
160
+ Thread.new {
161
+ sleep 1
162
+ @hash_queue[:foo].queue :bar
163
+ }
164
+
165
+ @hash_queue.pop(blocking: true) # waits until a background thread queues something and then returns, [:bar] in this case
166
+ ```
167
+
168
+ You can also pop items from individual queues:
169
+
170
+ ```ruby
171
+ hash_queue = HashQueue.new
172
+ 3.times { hash_queue[:foo].queue Object.new }
173
+
174
+ hash_queue[:foo].pop # => #<Object:0x000001008d4658>
175
+ hash_queue[:foo].pop(size: 2) # => [#<Object:0x000001008ae228>, #<Object:0x000001008ae200>]
176
+ ```
177
+
178
+ You can use same options as mentioned above.
179
+
180
+ ### Locking capabilities
181
+
182
+ HashQueue was designed for a specific use case and for that it provides a flexible locking facility.
183
+
184
+ You can lock a specific queue so you won't be able to pop until you unlock it.
185
+
186
+ ```ruby
187
+ hash_queue = HashQueue.new
188
+ hash_queue[:foo].queue Object.new
189
+ hash_queue[:foo].lock
190
+ hash_queue[:foo].pop # => nil
191
+ hash_queue[:foo].unlock
192
+ hash_queue[:foo].pop # => #<Object:0x000001008a5bf0>
193
+ ```
194
+
195
+ And you can place multiple locks on a queue and then you'll need to unlock them all to be able to pop stuff out again.
196
+
197
+ ```ruby
198
+ hash_queue = HashQueue.new
199
+ hash_queue[:foo].queue Object.new
200
+ hash_queue[:foo].lock(3)
201
+ hash_queue[:foo].pop # => nil
202
+ hash_queue[:foo].unlock
203
+ hash_queue[:foo].pop # => nil
204
+ hash_queue[:foo].unlock
205
+ hash_queue[:foo].unlock
206
+ hash_queue[:foo].pop # => #<Object:0x000001008a5bf0>
207
+ ```
208
+
209
+ Both `#lock` and `#unlock` take as an argument number of locks you want to put or remove from the queue. Defaults to `1`. There's even a convenient `#unlock_all` method and `#count_locks` that returns current number of locks placed on the queue. You can always check whether a specific queue is locked with `#locked?`.
210
+
211
+ ```ruby
212
+ hash_queue[:foo].lock
213
+ hash_queue[:foo].locked? # => true
214
+ hash_queue[:foo].unlock
215
+ hash_queue[:foo].locked? # => false
216
+ ```
217
+
218
+ Here it gets a little bit tricky. When popping stuff out of a queue you can specify `lock: true` option. That means while popping it will place the same number of locks on the queue as the number of items it pops out.
219
+
220
+ ```ruby
221
+ hash_queue[:foo].locked? # => false
222
+ hash_queue[:foo].pop(size: 2, lock: true)
223
+ hash_queue[:foo].locked? # => true
224
+ hash_queue[:foo].count_locks # => 2
225
+ ```
226
+
227
+ While it seems complicated the use case is fairly simple. Locking facility ensures that only a certain number of items could be out of a specific queue at one time. Imagine a crawler, you have opened 200 parallel connections but want to make sure you won't open more than 10 connections to one server at the time. You can use HashQueue as a local queue for URLs to be crawled, hostnames using as a key to keep separate queues for each hostname. You pop out 10 URLs placing 10 locks on a hostname, when a page is processed then unlock one lock.
228
+
229
+ You may see a flaw in this schema, because I lied a little. The issue with this example is that you can't pop new URLs to process until all previous 10 have been processed, right? You would have to wait until all locks have been unlocked. Well, HashQueue got this covered - You can pop from a queue while the queue is locked!
230
+
231
+ Imagine a queue with a lot of items in it. When you tell a queue to pop X items but the queue has Y locks on it, it just pops `X-Y` items. When you look in specs you'll find something like this:
232
+
233
+ ```ruby
234
+ it 'should allow you to pop items even if locked when you want more items then there are locks but taking existing locks into account' do
235
+ @hash_queue[:foo].lock(2)
236
+ @hash_queue[:foo].pop(size: 10).size.must_equal 8
237
+ end
238
+ ```
239
+
240
+ ## Acknowledgements
241
+
242
+ Requires Ruby 1.9. Tested under MRI 1.9.3 and JRuby.
243
+
244
+ ## TODO
245
+
246
+ + Someone please check and correct my grammar in readme. English is not my native language. I will send you my love and hug over Twitter.
247
+
248
+ ## Specs
249
+
250
+ To run the specs just run:
251
+
252
+ rake
253
+
254
+ ## Contributing
255
+
256
+ 1. Fork it
257
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
258
+ 3. Commit your changes (`git commit -am 'Added some feature'`)
259
+ 4. Push to the branch (`git push origin my-new-feature`)
260
+ 5. Create new Pull Request
261
+
262
+ ## License
263
+
264
+ This library is released under the MIT license.
265
+
266
+ ```
267
+ Permission is hereby granted, free of charge, to any person obtaining
268
+ a copy of this software and associated documentation files (the
269
+ ‘Software’), to deal in the Software without restriction, including
270
+ without limitation the rights to use, copy, modify, merge, publish,
271
+ distribute, sublicense, and/or sell copies of the Software, and to
272
+ permit persons to whom the Software is furnished to do so, subject to
273
+ the following conditions:
274
+
275
+ The above copyright notice and this permission notice shall be
276
+ included in all copies or substantial portions of the Software.
277
+
278
+ THE SOFTWARE IS PROVIDED ‘AS IS’, WITHOUT WARRANTY OF ANY KIND,
279
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
280
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
281
+ IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
282
+ CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
283
+ TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
284
+ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
285
+ ```
data/Rakefile ADDED
@@ -0,0 +1,15 @@
1
+ #!/usr/bin/env rake
2
+
3
+ $:.unshift File.expand_path(File.dirname(__FILE__) + '/lib')
4
+
5
+ require "bundler/gem_tasks"
6
+ require 'rake/testtask'
7
+
8
+ Rake::TestTask.new do |t|
9
+ t.libs << 'spec'
10
+ t.pattern = 'spec/**/*_spec.rb'
11
+ t.verbose = true
12
+ end
13
+
14
+ desc "Run tests"
15
+ task :default => :test
@@ -0,0 +1,21 @@
1
+ # -*- encoding: utf-8 -*-
2
+ require File.expand_path('../lib/hash_queue/version', __FILE__)
3
+
4
+ Gem::Specification.new do |gem|
5
+ gem.authors = ["Michal Krejčí"]
6
+ gem.email = ["mikekreeki@gmail.com"]
7
+ gem.description = %q{Threadsafe namespaced Queue with locking capabilities.}
8
+ gem.summary = %q{Threadsafe namespaced Queue with locking capabilities.}
9
+ gem.homepage = "http://github.com/mikekreeki/hash_queue"
10
+
11
+ gem.add_development_dependency('minitest')
12
+ gem.add_development_dependency('turn')
13
+ gem.add_development_dependency('term-ansicolor')
14
+
15
+ gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
16
+ gem.files = `git ls-files`.split("\n")
17
+ gem.test_files = `git ls-files -- {test,spec,features}/*`.split("\n")
18
+ gem.name = "hash_queue"
19
+ gem.require_paths = ["lib"]
20
+ gem.version = HashQueue::VERSION
21
+ end
@@ -0,0 +1,109 @@
1
+ module HashQueue
2
+ class Hash
3
+
4
+ def initialize
5
+ @hash_queue = {}
6
+ @mutex = Mutex.new
7
+ end
8
+
9
+ def [](key)
10
+ QueueProxy.new(self, key)
11
+ end
12
+
13
+ def queue(key, obj)
14
+ get_queue(key).queue obj
15
+ end
16
+ alias_method :enqueue, :queue
17
+ alias_method :push, :queue
18
+
19
+ def pop(options = {})
20
+ loop do
21
+ results = pop_from_queues(options)
22
+
23
+ if options[:blocking]
24
+ return results unless results.empty?
25
+ sleep 0.01
26
+ else
27
+ return results
28
+ end
29
+ end
30
+ end
31
+ alias_method :shift, :pop
32
+
33
+ def size
34
+ @mutex.synchronize do
35
+ @hash_queue.each_value.inject(0) do |sum, queue|
36
+ sum += queue.size
37
+ end
38
+ end
39
+ end
40
+ alias_method :count, :size
41
+ alias_method :length, :size
42
+
43
+ def empty?
44
+ size.zero?
45
+ end
46
+
47
+ def keys
48
+ @mutex.synchronize do
49
+ @hash_queue.keys
50
+ end
51
+ end
52
+
53
+ def clean
54
+ @mutex.synchronize do
55
+ @hash_queue.reject! do |key, queue|
56
+ queue.empty?
57
+ end
58
+ end
59
+ end
60
+
61
+ def clear
62
+ @mutex.synchronize do
63
+ @hash_queue.clear
64
+ end
65
+ end
66
+
67
+ def get_queue(key)
68
+ @mutex.synchronize do
69
+ @hash_queue.fetch(key) do
70
+ @hash_queue[key] = Queue.new
71
+ end
72
+ end
73
+ end
74
+
75
+ private
76
+
77
+ def pop_from_queues(options)
78
+ options = options.dup
79
+ options.delete(:blocking)
80
+
81
+ @mutex.synchronize do
82
+ @hash_queue.each_value.each_with_object([]) do |queue, results|
83
+ queue.pop(options,results)
84
+ end
85
+ end
86
+ end
87
+
88
+ end
89
+
90
+ class QueueProxy
91
+
92
+ [ :queue, :enqueue, :push, :<<, :pop, :shift, :size, :count, :length, :empty?, :clear, :lock, :unlock,
93
+ :locked?, :unlock_all, :count_locks, :locks_count].each do |m|
94
+ define_method m do |*args|
95
+ subject.send m, *args
96
+ end
97
+ end
98
+
99
+ def initialize(parent, key)
100
+ @parent = parent
101
+ @key = key
102
+ end
103
+
104
+ def subject
105
+ @parent.get_queue(@key)
106
+ end
107
+
108
+ end
109
+ end
@@ -0,0 +1,50 @@
1
+ module HashQueue
2
+ module Lockable
3
+
4
+ def lock(n = 1)
5
+ @mutex.synchronize do
6
+ _lock(n)
7
+ end
8
+ end
9
+
10
+ def unlock(n = 1)
11
+ @mutex.synchronize do
12
+ @locks.shift(n)
13
+ end
14
+ end
15
+
16
+ def locked?
17
+ @mutex.synchronize do
18
+ _locked?
19
+ end
20
+ end
21
+
22
+ def unlock_all
23
+ @mutex.synchronize do
24
+ @locks.clear
25
+ end
26
+ end
27
+
28
+ def count_locks
29
+ @mutex.synchronize do
30
+ _count_locks
31
+ end
32
+ end
33
+ alias_method :locks_count, :count_locks
34
+
35
+ private
36
+
37
+ def _lock(n = 1)
38
+ n.times { @locks.push true }
39
+ end
40
+
41
+ def _locked?
42
+ not @locks.empty?
43
+ end
44
+
45
+ def _count_locks
46
+ @locks.count
47
+ end
48
+
49
+ end
50
+ end
@@ -0,0 +1,89 @@
1
+ require 'hash_queue/queue/lockable'
2
+
3
+ module HashQueue
4
+ class Queue
5
+ include Lockable
6
+
7
+ def initialize
8
+ @mutex = Mutex.new
9
+ @queue = []
10
+ @locks = []
11
+ end
12
+
13
+ def queue(obj)
14
+ @mutex.synchronize do
15
+ @queue.push obj
16
+ end
17
+ end
18
+ alias_method :enqueue, :queue
19
+ alias_method :push, :queue
20
+ alias_method :<<, :queue
21
+
22
+ def pop(options = {}, results = [])
23
+ if options[:blocking]
24
+ loop do
25
+ result = _pop(options, results)
26
+
27
+ return result unless result.nil? or result == []
28
+ sleep 0.01
29
+ end
30
+ else
31
+ _pop(options,results)
32
+ end
33
+ end
34
+ alias_method :shift, :pop
35
+
36
+ def size
37
+ @mutex.synchronize do
38
+ @queue.size
39
+ end
40
+ end
41
+ alias_method :count, :size
42
+ alias_method :length, :size
43
+
44
+ def empty?
45
+ @mutex.synchronize do
46
+ _empty?
47
+ end
48
+ end
49
+
50
+ def clear
51
+ @mutex.synchronize do
52
+ @queue.clear
53
+ end
54
+ end
55
+
56
+ private
57
+
58
+ def _pop(options,results)
59
+ @mutex.synchronize do
60
+ size = options.fetch(:size, 1)
61
+
62
+ if _locked? and _count_locks >= size
63
+ if options.key? :size
64
+ return []
65
+ else
66
+ return nil
67
+ end
68
+ end
69
+
70
+ (size - _count_locks).times do
71
+ break if _empty?
72
+ results.push @queue.shift
73
+ _lock if options[:lock]
74
+ end
75
+
76
+ if options.key? :size
77
+ results
78
+ else
79
+ results[0]
80
+ end
81
+ end
82
+ end
83
+
84
+ def _empty?
85
+ @queue.empty?
86
+ end
87
+
88
+ end
89
+ end
@@ -0,0 +1,3 @@
1
+ module HashQueue
2
+ VERSION = "0.1.0"
3
+ end
data/lib/hash_queue.rb ADDED
@@ -0,0 +1,10 @@
1
+ require 'hash_queue/hash'
2
+ require 'hash_queue/queue'
3
+
4
+ module HashQueue
5
+ extend self
6
+
7
+ def new
8
+ Hash.new
9
+ end
10
+ end
@@ -0,0 +1,167 @@
1
+ require 'spec_helper'
2
+ #require 'timeout'
3
+
4
+ describe HashQueue do
5
+ it 'should be able to create new HashQueue instance' do
6
+ HashQueue.new.must_be_instance_of HashQueue::Hash
7
+ end
8
+ end
9
+
10
+ describe HashQueue::Hash do
11
+
12
+ describe 'when initialized' do
13
+ before do
14
+ @hash_queue = HashQueue::Hash.new
15
+ end
16
+
17
+ it 'should be empty' do
18
+ @hash_queue.empty?.must_equal true
19
+ end
20
+
21
+ it 'should return correct size' do
22
+ @hash_queue.size.must_equal 0
23
+ end
24
+
25
+ it 'should create Queues on the fly' do
26
+ @hash_queue[:foo].queue 1
27
+ @hash_queue.keys.size.must_equal 1
28
+ end
29
+
30
+ it 'should be able to queue stuff' do
31
+ @hash_queue.queue :foo, 1
32
+ @hash_queue.size.must_equal 1
33
+ end
34
+
35
+ it 'should be able to get Queue object when asked with a key' do
36
+ @hash_queue.get_queue(:foo).must_be_instance_of HashQueue::Queue
37
+ @hash_queue.get_queue(:non_existent).must_be_instance_of HashQueue::Queue
38
+ end
39
+ end
40
+
41
+
42
+ describe 'when we queue stuff' do
43
+ before do
44
+ @hash_queue = HashQueue::Hash.new
45
+ @hash_queue.queue :foo, true
46
+ @hash_queue.queue :bar, false
47
+ end
48
+
49
+ it 'shouldnt be empty' do
50
+ @hash_queue.empty?.must_equal false
51
+ end
52
+
53
+ it 'should return correct size' do
54
+ @hash_queue.size.must_equal 2
55
+ end
56
+
57
+ it 'should return maintained keys' do
58
+ @hash_queue.keys.sort.must_equal [:foo, :bar].sort
59
+ end
60
+
61
+ it 'should be able to clear itself' do
62
+ @hash_queue.clear
63
+ @hash_queue.size.must_equal 0
64
+ @hash_queue.keys.must_equal []
65
+ end
66
+
67
+ it 'should be able to clean itself' do
68
+ @hash_queue[:foo].pop
69
+ @hash_queue.clean
70
+ @hash_queue.size.must_equal 1
71
+ @hash_queue.keys.must_equal [:bar]
72
+ end
73
+ end
74
+
75
+
76
+ describe 'when we queue an array' do
77
+ before do
78
+ @hash_queue = HashQueue::Hash.new
79
+ @hash_queue.queue :foo, [1]
80
+ @hash_queue.queue :bar, [2]
81
+ end
82
+
83
+ it 'should return array values' do
84
+ @hash_queue.pop.sort.must_equal [[1],[2]].sort
85
+ end
86
+ end
87
+
88
+ describe 'when we queue weird stuff' do
89
+ before do
90
+ @hash_queue = HashQueue::Hash.new
91
+ end
92
+
93
+ describe 'like nil' do
94
+ before do
95
+ @hash_queue.queue :foo, nil
96
+ end
97
+
98
+ it 'should return nil when popping' do
99
+ @hash_queue.pop.must_equal [nil]
100
+ end
101
+ end
102
+
103
+ describe 'like an empty array' do
104
+ before do
105
+ @hash_queue.queue :foo, []
106
+ end
107
+
108
+ it 'should return an empty array when popping' do
109
+ @hash_queue.pop.must_equal [[]]
110
+ end
111
+ end
112
+
113
+ end
114
+
115
+ describe 'when popping' do
116
+
117
+ describe 'from empty HashQueue instance' do
118
+ before do
119
+ @hash_queue = HashQueue::Hash.new
120
+ end
121
+
122
+ it 'should return empty array' do
123
+ @hash_queue.pop.must_equal []
124
+ end
125
+ end
126
+
127
+
128
+ describe 'from non-empty HashQueue instance' do
129
+ before do
130
+ @hash_queue = HashQueue::Hash.new
131
+ @hash_queue[:foo].queue 1
132
+ @hash_queue[:foo].queue 2
133
+ @hash_queue[:bar].queue 3
134
+ @hash_queue[:bar].queue 4
135
+ end
136
+
137
+ it 'should return array of items, one from each queue' do
138
+ @hash_queue.pop.sort.must_equal [1,3].sort
139
+ end
140
+
141
+ it 'should return appropriate number of items from each queue when size option specified' do
142
+ @hash_queue.pop(size: 2).sort.must_equal [1,2,3,4]
143
+ end
144
+
145
+ end
146
+
147
+
148
+ describe 'with blocking option' do
149
+ before do
150
+ @hash_queue = HashQueue::Hash.new
151
+ end
152
+
153
+ it 'should wait until some results are available' do
154
+ Thread.new {
155
+ sleep 0.5
156
+ @hash_queue[:foo].queue :bar
157
+ }
158
+
159
+ @hash_queue.pop(blocking: true).wont_be_empty
160
+ end
161
+
162
+ end
163
+
164
+ end
165
+
166
+ end
167
+
@@ -0,0 +1,20 @@
1
+ require 'spec_helper'
2
+
3
+ describe HashQueue::QueueProxy do
4
+ before do
5
+ hash_queue = HashQueue.new
6
+ @proxy = HashQueue::QueueProxy.new(hash_queue, :foo)
7
+ end
8
+
9
+ it 'should proxy public methods of Queue and modules included to Queue object itself in sane matter' do
10
+ methods = [HashQueue::Queue, HashQueue::Lockable].map{ |k| k.instance_methods(false) }.flatten
11
+ methods.each do |m|
12
+ @proxy.must_respond_to m
13
+ end
14
+ end
15
+
16
+ it 'should return the subject, Queue object itself (but really never use it directly, misbehaves in edge cases)' do
17
+ @proxy.subject.must_be_instance_of HashQueue::Queue
18
+ end
19
+ end
20
+
@@ -0,0 +1,152 @@
1
+ require 'spec_helper'
2
+
3
+ describe HashQueue::Queue do
4
+ before do
5
+ @queue = HashQueue::Queue.new
6
+ end
7
+
8
+ describe 'when initialized' do
9
+
10
+ it 'should be empty' do
11
+ @queue.empty?.must_equal true
12
+ @queue.size.must_equal 0
13
+ end
14
+
15
+ end
16
+
17
+ describe 'when queued stuff' do
18
+ before do
19
+ @queue.queue 1
20
+ @queue.queue 2
21
+ end
22
+
23
+ it 'shouldnt be empty' do
24
+ @queue.empty?.must_equal false
25
+ @queue.size.must_equal 2
26
+ end
27
+
28
+ it 'should be able to clear itself' do
29
+ @queue.clear
30
+ @queue.empty?.must_equal true
31
+ end
32
+
33
+ it 'should be able to pop the stuff in right order' do
34
+ @queue.pop.must_equal 1
35
+ @queue.pop.must_equal 2
36
+ end
37
+
38
+ it 'should be empty when you pop all the stuff' do
39
+ @queue.pop
40
+ @queue.empty?.must_equal false
41
+ @queue.pop
42
+ @queue.empty?.must_equal true
43
+ end
44
+
45
+ it 'should be able to pop more stuff if you tell it how much stuff do you want' do
46
+ @queue.pop(size: 2).must_equal [1, 2]
47
+ end
48
+
49
+ it 'should return an array every time when size option specified' do
50
+ @queue.pop(size: 1).must_equal [1]
51
+
52
+ @queue.clear
53
+ @queue.pop(size: 1).must_equal []
54
+ end
55
+ end
56
+
57
+ describe 'when we queue weird stuff' do
58
+ describe 'like nil' do
59
+ before do
60
+ 3.times { @queue.queue nil }
61
+ end
62
+
63
+ it 'should return nil when popping' do
64
+ @queue.pop.must_equal nil
65
+ @queue.pop(size: 2).must_equal [nil, nil]
66
+ end
67
+ end
68
+
69
+ describe 'like an empty array' do
70
+ before do
71
+ 3.times { @queue.queue [] }
72
+ end
73
+
74
+ it 'should return an empty array when popping' do
75
+ @queue.pop.must_equal []
76
+ @queue.pop(size: 2).must_equal [[], []]
77
+ end
78
+ end
79
+
80
+ end
81
+
82
+ describe 'when popping with blocking option' do
83
+ it 'should wait until some results are available' do
84
+ Thread.new {
85
+ sleep 0.5
86
+ @queue.queue 1
87
+ }
88
+
89
+ @queue.pop(blocking: true, size: 1).wont_be_empty
90
+ end
91
+
92
+ end
93
+
94
+ describe 'locking' do
95
+
96
+ it 'should be unlocked when initialized' do
97
+ @queue.locked?.must_equal false
98
+ end
99
+
100
+ it 'should be able to be locked' do
101
+ @queue.lock
102
+ @queue.locked?.must_equal true
103
+ @queue.count_locks.must_equal 1
104
+ end
105
+
106
+ it 'should be able to be unloacked when locked' do
107
+ @queue.lock
108
+ @queue.unlock
109
+ @queue.locked?.must_equal false
110
+ @queue.count_locks.must_equal 0
111
+ end
112
+
113
+
114
+ it 'should be able to specify how much locks you want to apply' do
115
+ @queue.lock(2)
116
+ @queue.count_locks.must_equal 2
117
+ @queue.unlock(2)
118
+ @queue.count_locks.must_equal 0
119
+ end
120
+
121
+ it 'should be able to unlock all locks' do
122
+ @queue.lock(2)
123
+ @queue.unlock_all
124
+ @queue.locked?.must_equal false
125
+ @queue.count_locks.must_equal 0
126
+ end
127
+
128
+ describe 'when popping with lock option' do
129
+ before do
130
+ 10.times { @queue.queue :foo }
131
+ end
132
+
133
+ it 'should place lock on the queue' do
134
+ @queue.pop(lock: true)
135
+ @queue.locked?.must_equal true
136
+ @queue.count_locks.must_equal 1
137
+ end
138
+
139
+ it 'should put appropriate amount of locks on the queue when custom pop size is specified' do
140
+ @queue.pop(size: 2, lock: true)
141
+ @queue.locked?.must_equal true
142
+ @queue.count_locks.must_equal 2
143
+ end
144
+
145
+ it 'should allow you to pop items even if locked when you want more items then there are locks but taking existing locks into account' do
146
+ @queue.lock(2)
147
+ @queue.pop(size: 10).size.must_equal 8
148
+ end
149
+ end
150
+
151
+ end
152
+ end
@@ -0,0 +1,7 @@
1
+ require 'bundler/setup'
2
+
3
+ require 'minitest/autorun'
4
+ require 'turn/autorun'
5
+
6
+ require 'hash_queue'
7
+
metadata ADDED
@@ -0,0 +1,114 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: hash_queue
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ prerelease:
6
+ platform: ruby
7
+ authors:
8
+ - Michal Krejčí
9
+ autorequire:
10
+ bindir: bin
11
+ cert_chain: []
12
+ date: 2012-09-09 00:00:00.000000000 Z
13
+ dependencies:
14
+ - !ruby/object:Gem::Dependency
15
+ name: minitest
16
+ requirement: !ruby/object:Gem::Requirement
17
+ none: false
18
+ requirements:
19
+ - - ! '>='
20
+ - !ruby/object:Gem::Version
21
+ version: '0'
22
+ type: :development
23
+ prerelease: false
24
+ version_requirements: !ruby/object:Gem::Requirement
25
+ none: false
26
+ requirements:
27
+ - - ! '>='
28
+ - !ruby/object:Gem::Version
29
+ version: '0'
30
+ - !ruby/object:Gem::Dependency
31
+ name: turn
32
+ requirement: !ruby/object:Gem::Requirement
33
+ none: false
34
+ requirements:
35
+ - - ! '>='
36
+ - !ruby/object:Gem::Version
37
+ version: '0'
38
+ type: :development
39
+ prerelease: false
40
+ version_requirements: !ruby/object:Gem::Requirement
41
+ none: false
42
+ requirements:
43
+ - - ! '>='
44
+ - !ruby/object:Gem::Version
45
+ version: '0'
46
+ - !ruby/object:Gem::Dependency
47
+ name: term-ansicolor
48
+ requirement: !ruby/object:Gem::Requirement
49
+ none: false
50
+ requirements:
51
+ - - ! '>='
52
+ - !ruby/object:Gem::Version
53
+ version: '0'
54
+ type: :development
55
+ prerelease: false
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ none: false
58
+ requirements:
59
+ - - ! '>='
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ description: Threadsafe namespaced Queue with locking capabilities.
63
+ email:
64
+ - mikekreeki@gmail.com
65
+ executables: []
66
+ extensions: []
67
+ extra_rdoc_files: []
68
+ files:
69
+ - .gitignore
70
+ - .rvmrc
71
+ - CHANGELOG.md
72
+ - Gemfile
73
+ - LICENSE
74
+ - README.md
75
+ - Rakefile
76
+ - hash_queue.gemspec
77
+ - lib/hash_queue.rb
78
+ - lib/hash_queue/hash.rb
79
+ - lib/hash_queue/queue.rb
80
+ - lib/hash_queue/queue/lockable.rb
81
+ - lib/hash_queue/version.rb
82
+ - spec/hash_queue/hash_queue_spec.rb
83
+ - spec/hash_queue/queue_proxy_spec.rb
84
+ - spec/hash_queue/queue_spec.rb
85
+ - spec/spec_helper.rb
86
+ homepage: http://github.com/mikekreeki/hash_queue
87
+ licenses: []
88
+ post_install_message:
89
+ rdoc_options: []
90
+ require_paths:
91
+ - lib
92
+ required_ruby_version: !ruby/object:Gem::Requirement
93
+ none: false
94
+ requirements:
95
+ - - ! '>='
96
+ - !ruby/object:Gem::Version
97
+ version: '0'
98
+ required_rubygems_version: !ruby/object:Gem::Requirement
99
+ none: false
100
+ requirements:
101
+ - - ! '>='
102
+ - !ruby/object:Gem::Version
103
+ version: '0'
104
+ requirements: []
105
+ rubyforge_project:
106
+ rubygems_version: 1.8.24
107
+ signing_key:
108
+ specification_version: 3
109
+ summary: Threadsafe namespaced Queue with locking capabilities.
110
+ test_files:
111
+ - spec/hash_queue/hash_queue_spec.rb
112
+ - spec/hash_queue/queue_proxy_spec.rb
113
+ - spec/hash_queue/queue_spec.rb
114
+ - spec/spec_helper.rb