smq 0.1.1

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1 @@
1
+ pkg
data/LICENSE ADDED
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2010 Tim Blair <tim@bla.ir>
2
+
3
+ Permission is hereby granted, free of charge, to any person
4
+ obtaining a copy of this software and associated documentation
5
+ files (the "Software"), to deal in the Software without
6
+ restriction, including without limitation the rights to use,
7
+ copy, modify, merge, publish, distribute, sublicense, and/or sell
8
+ copies of the Software, and to permit persons to whom the
9
+ Software is furnished to do so, subject to the following
10
+ conditions:
11
+
12
+ The above copyright notice and this permission notice shall be
13
+ included in all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
16
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
17
+ OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
18
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
19
+ HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
20
+ WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
21
+ FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
22
+ OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,88 @@
1
+ # Simple Message Queue (SMQ)
2
+
3
+ SMQ is a database-backed, JSON-based message queue and worker platform.
4
+
5
+ ## Description
6
+
7
+ SMQ uses ActiveRecord to provide a database-agnositc, JSON-based [message queue](http://en.wikipedia.org/wiki/Message_queue) and worker platform; this is not to be confused with [job queues](http://en.wikipedia.org/wiki/Job_queue) such as [Resque](http://github.com/defunkt/resque) or [Delayed::Job](http://github.com/tobi/delayed_job).
8
+
9
+ Other simple message queue systems exist, but these generally either use Marshal'd ruby objects which are not transferable to other platforms, or are run as a separate server daemon. SMQ uses a simple database table for persisting message queues and JSON encoding for message payloads to enable use cross-platform.
10
+
11
+ ## Installation
12
+
13
+ SMQ is provided as a gem courtesy of Gemcutter:
14
+
15
+ gem install smq
16
+
17
+ Once you've established your ActiveRecord connection, you'll need to initialise the (single) database table:
18
+
19
+ SMQ.load_schema!
20
+
21
+ ## Queues
22
+
23
+ Creating a queue is as simple as:
24
+
25
+ queue = SMQ::Queue.new("queue_name")
26
+
27
+ A "queue" doesn't exist as a persisted entity: it's effectively just a tag applied to a message; pushing a message onto a queue simply saves the Message with the queue name assigned.
28
+
29
+ ## Adding Messages to a Queue
30
+
31
+ A message's payload can be any data structure that can be serialised to JSON. There are three ways that a message can be added to a queue (here we're assuming that `msg_data` already exists):
32
+
33
+ queue.enqueue(msg_data)
34
+
35
+ # or
36
+
37
+ queue.push(SMQ::Message.build(msg_data))
38
+
39
+ # or
40
+
41
+ SMQ::Message.build(msg_data, "queue_name").save
42
+
43
+ The first two methods are effectively simplified abstractions of the third. Note that a single message can only be added to one queue; to push to multiple queues, additional instances of the message must be created.
44
+
45
+ ### Non-Ruby Messages
46
+
47
+ Adding to a queue from outside a ruby environment is as simple as `INSERT`ing a JSON-encoded packet into the queue table:
48
+
49
+ INSERT INTO smq_messages (queue, payload, created_at) VALUES (
50
+ 'queue_name',
51
+ '{"json":"encoded stuff"}',
52
+ NOW()
53
+ )
54
+
55
+ ## Workers
56
+
57
+ A worker is an instance of `SMQ::Worker` bound to a single named queue. The `Worker#work` method takes care of reserving `Message`s and then passing them back to the given block:
58
+
59
+ SMQ::Worker.new("queue_name").work do |msg|
60
+ puts msg.data.inspect
61
+ msg.ack!
62
+ end
63
+
64
+ When a worker has finished with a `Message`, it should either call `Message#ack!` to acknowledge receipt of that message, or `Message#fail` followed by `Message#save` to mark the message as failed. In addition, if a message should fail for a transient reason, it can be pushed back into the queue by calling `Message#retry!`. A message will be retried up to 5 times before automatically being marked as failed.
65
+
66
+ `Worker#work` takes a single optional argument (in addition to the callback block): `until_empty`. If `true`, the worker will stop when the queue is empty (but see detail on locking below); if `false` the worker will continue to look for new jobs indefinitely (or until `Worker#stop!` is called), waiting 1 second between queue polls.
67
+
68
+ ### Locking Strategy
69
+
70
+ To facilitate the locking ("delivery") of individual messages, the following strategy is used:
71
+
72
+ 1. Find the next 5 messages in the queue;
73
+ 2. Randomise these messages;
74
+ 3. Attempt to `UPDATE` each message based on the message ID, lock status and last updated stamp;
75
+ 4. When the `UPDATE` returns a row change count of 1, that message has been locked and can be passed off to be processed;
76
+ 5. If a lock isn't aquired for any of the 5 messages, either wait for 1 second and then start the process again or, if `until_empty` is `true`, end the processing loop and stop the worker.
77
+
78
+ The effect of this is that, although multiple workers can be employed on a single queue, the increase in throughput is not linear as may be expected. In fact, if you increase the number of workers on queue past 5, throughput may actually diminish due to the extra time spent attempting to aquire a message lock.
79
+
80
+ This process, although heavy on `SELECT`s, results in a minimum of table locking which would actually slow the lock process down. For example, by using an `UPDATE` with a `LIMIT` of `1` when the queue size is more than a couple of hundred results in a noticible slow down in lock aquisition.
81
+
82
+ These limitations are deemed acceptable due to the simple nature of this queueing system.
83
+
84
+ ## Licensing and Attribution
85
+
86
+ SMQ is released under the MIT license as detailed in the LICENSE file that should be distributed with this library; the source code is [freely available](http://github.com/timblair/smq).
87
+
88
+ SMQ was developed by [Tim Blair](http://tim.bla.ir/) during work on [White Label Dating](http://www.whitelabeldating.com/), while employed by [Global Personals Ltd](http://www.globalpersonals.co.uk). Global Personals Ltd have kindly agreed to the extraction and release of this software under the license terms above.
@@ -0,0 +1,53 @@
1
+ require 'rubygems'
2
+ require 'rake'
3
+
4
+ begin
5
+ require 'jeweler'
6
+ Jeweler::Tasks.new do |gem|
7
+ gem.name = "smq"
8
+ gem.summary = %Q{Simple Message Queue}
9
+ gem.description = %Q{A simple database-backed, JSON-based message queue and worker base}
10
+ gem.email = "tim@bla.ir"
11
+ gem.homepage = "http://github.com/timblair/smq"
12
+ gem.authors = ["Tim Blair"]
13
+ gem.add_dependency 'activerecord'
14
+ gem.add_dependency 'yajl-ruby'
15
+ gem.add_development_dependency 'sqlite3-ruby'
16
+ end
17
+ Jeweler::GemcutterTasks.new
18
+ task :test => :check_dependencies
19
+ rescue LoadError
20
+ puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
21
+ end
22
+
23
+ require 'rake/testtask'
24
+ Rake::TestTask.new(:test) do |test|
25
+ test.libs << 'lib' << 'test'
26
+ test.pattern = 'test/**/test_*.rb'
27
+ test.verbose = true
28
+ end
29
+
30
+ begin
31
+ require 'rcov/rcovtask'
32
+ Rcov::RcovTask.new do |test|
33
+ test.libs << 'test'
34
+ test.pattern = 'test/**/test_*.rb'
35
+ test.verbose = true
36
+ end
37
+ rescue LoadError
38
+ task :rcov do
39
+ abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
40
+ end
41
+ end
42
+
43
+ task :default => :test
44
+
45
+ require 'rake/rdoctask'
46
+ Rake::RDocTask.new do |rdoc|
47
+ version = File.exist?('VERSION') ? File.read('VERSION') : ""
48
+
49
+ rdoc.rdoc_dir = 'rdoc'
50
+ rdoc.title = "smq #{version}"
51
+ rdoc.rdoc_files.include('README*')
52
+ rdoc.rdoc_files.include('lib/**/*.rb')
53
+ end
data/VERSION ADDED
@@ -0,0 +1 @@
1
+ 0.1.1
@@ -0,0 +1,3 @@
1
+ development:
2
+ adapter: sqlite3
3
+ database: /tmp/smq-dev.sqlite
@@ -0,0 +1,25 @@
1
+ require 'rubygems'
2
+
3
+ begin
4
+ require 'smq'
5
+ rescue LoadError
6
+ $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
7
+ require 'smq'
8
+ end
9
+
10
+ ActiveRecord::Base.configurations = YAML.load_file(File.dirname(__FILE__) + '/database.yml')
11
+ ActiveRecord::Base.establish_connection 'development'
12
+
13
+ SMQ::load_schema!
14
+
15
+ worker = SMQ::Worker.new("queue_name")
16
+ 100.times { |i| worker.queue.enqueue(i+1) }
17
+
18
+ count = 0
19
+ worker.work(true) do |msg|
20
+ count += 1
21
+ puts msg.inspect
22
+ msg.ack!
23
+ end
24
+
25
+ puts "Handled: #{count}"
@@ -0,0 +1,51 @@
1
+ require 'rubygems'
2
+ require 'active_record'
3
+ require 'yajl'
4
+
5
+ require File.dirname(__FILE__) + '/smq/worker'
6
+ require File.dirname(__FILE__) + '/smq/message'
7
+ require File.dirname(__FILE__) + '/smq/queue'
8
+
9
+ module SMQ
10
+
11
+ def self.now
12
+ (ActiveRecord::Base.default_timezone == :utc) ? Time.now.utc : Time.now
13
+ end
14
+
15
+ def self.encode(object)
16
+ Yajl::Encoder.encode(object)
17
+ end
18
+
19
+ def self.decode(object)
20
+ return unless object
21
+ Yajl::Parser.parse(object, :check_utf8 => false)
22
+ end
23
+
24
+ def self.load_schema!(force = false)
25
+ return if !force && ActiveRecord::Base.connection.tables.include?(SMQ::Message.table_name)
26
+ ActiveRecord::Schema.define do
27
+ create_table :smq_messages, :force => force do |t|
28
+ t.string :queue, :limit => 30, :null => false
29
+ t.text :payload
30
+ t.datetime :locked_at
31
+ t.string :locked_by, :limit => 50
32
+ t.integer :attempts, :limit => 2, :default => 0
33
+ t.datetime :failed_at
34
+ t.datetime :completed_at
35
+ t.timestamps
36
+ end
37
+ if ActiveRecord::Base.connection.instance_variable_get(:@config)[:adapter] == 'mysql'
38
+ execute "ALTER TABLE `smq_messages` MODIFY COLUMN `updated_at` TIMESTAMP"
39
+ end
40
+ add_index "smq_messages", ["queue", "completed_at", "locked_by"], :name => "idx_smq_available"
41
+ add_index "smq_messages", ["id", "updated_at", "locked_by", "completed_at"], :name => "idx_smq_unlocked"
42
+ end
43
+ end
44
+
45
+ def self.flush_all_queues!
46
+ # yes, not the most efficient, but *should* only be called during testing
47
+ # so it's not that much of a concern
48
+ SMQ::Message.delete_all
49
+ end
50
+
51
+ end
@@ -0,0 +1,66 @@
1
+ module SMQ
2
+
3
+ class Message < ActiveRecord::Base
4
+ set_table_name :smq_messages
5
+ validates_presence_of :queue
6
+ MAX_ATTEMPTS = 5
7
+
8
+ def self.build(payload, queue = nil)
9
+ msg = self.new
10
+ msg.payload = SMQ.encode(payload)
11
+ msg.queue = queue
12
+ msg
13
+ end
14
+
15
+ def data
16
+ SMQ.decode(payload)
17
+ end
18
+ def data=(object)
19
+ self.payload = SMQ.encode(object)
20
+ end
21
+
22
+ def retry!
23
+ self.locked_by = nil
24
+ self.locked_at = nil
25
+ fail if self.attempts >= MAX_ATTEMPTS
26
+ save!
27
+ end
28
+
29
+ def lock!(worker)
30
+ rows = self.class.update_all(
31
+ ["updated_at = ?, locked_at = ?, locked_by = ?, attempts = (attempts+1)", SMQ.now, SMQ.now, worker.name],
32
+ ["id = ? AND updated_at = ? AND locked_by IS NULL AND completed_at IS NULL", self.id, self.updated_at]
33
+ )
34
+ if rows == 1
35
+ self.reload
36
+ return self
37
+ end
38
+ nil
39
+ end
40
+
41
+ def locked?
42
+ !self.locked_by.nil?
43
+ end
44
+ def failed?
45
+ !self.failed_at.nil?
46
+ end
47
+ def completed?
48
+ !self.completed_at.nil?
49
+ end
50
+
51
+ def complete
52
+ self.completed_at = SMQ.now
53
+ end
54
+ def fail
55
+ self.failed_at = SMQ.now
56
+ complete
57
+ end
58
+
59
+ def ack!
60
+ complete
61
+ save!
62
+ end
63
+
64
+ end
65
+
66
+ end
@@ -0,0 +1,65 @@
1
+ module SMQ
2
+
3
+ class Queue
4
+
5
+ attr_accessor :name
6
+ attr_accessor :batch_size
7
+
8
+ def initialize(name, batch_size = 5)
9
+ @name = name
10
+ @batch_size = batch_size
11
+ end
12
+
13
+ def to_s
14
+ @name
15
+ end
16
+
17
+ def push(msg)
18
+ msg.queue = @name
19
+ msg.save!
20
+ msg
21
+ end
22
+
23
+ def enqueue(payload)
24
+ push SMQ::Message.build(payload)
25
+ end
26
+
27
+ def reserve(worker)
28
+ find_available.each do |msg|
29
+ m = msg.lock!(worker)
30
+ return m unless m == nil
31
+ end
32
+ nil
33
+ end
34
+
35
+ def find_available
36
+ SMQ::Message.find(:all, :select => 'id, updated_at', :conditions => ["queue = ? AND completed_at IS NULL AND locked_by IS NULL", @name], :order => "id ASC", :limit => @batch_size).sort_by { rand() }
37
+ end
38
+
39
+ def length
40
+ SMQ::Message.count(:conditions => ["queue = ? AND completed_at IS NULL", @name])
41
+ end
42
+
43
+ def flush!
44
+ delete_queue_items
45
+ end
46
+
47
+ def clear_completed!
48
+ delete_queue_items "completed_at IS NOT NULL"
49
+ end
50
+ def clear_successful!
51
+ delete_queue_items "completed_at IS NOT NULL AND failed_at IS NULL"
52
+ end
53
+ def clear_failed!
54
+ delete_queue_items "completed_at IS NOT NULL AND failed_at IS NOT NULL"
55
+ end
56
+
57
+ private
58
+
59
+ def delete_queue_items(where = nil)
60
+ SMQ::Message.delete_all(["queue = ? #{where ? 'AND ' + where : ''}", @name])
61
+ end
62
+
63
+ end
64
+
65
+ end
@@ -0,0 +1,58 @@
1
+ require "socket"
2
+
3
+ module SMQ
4
+
5
+ class Worker
6
+
7
+ attr_accessor :name
8
+ attr_accessor :queue
9
+
10
+ @working = false
11
+ @stopping = false
12
+
13
+ def initialize(queue)
14
+ self.name = "#{Socket.gethostname}:#{Process.pid}" rescue "pid:#{Process.pid}"
15
+ self.queue = queue.instance_of?(SMQ::Queue) ? queue : SMQ::Queue.new(queue)
16
+ end
17
+
18
+ def to_s
19
+ self.name
20
+ end
21
+
22
+ def work(until_empty = false, &block)
23
+ @working = true
24
+ empty = false
25
+ empty_for = 0
26
+ while(!empty && !@stopping) do
27
+ if work_one_message(&block).nil?
28
+ empty = true if until_empty
29
+ sleep 1 unless (empty && (empty_for += 1) >= 10)
30
+ else
31
+ empty_for = 0
32
+ end
33
+ end
34
+ @stopping = false
35
+ @working = false
36
+ end
37
+
38
+ def work_one_message
39
+ msg = self.queue.reserve(self)
40
+ yield msg if !msg.nil? && block_given?
41
+ msg
42
+ end
43
+
44
+ def stop!
45
+ @stopping = true
46
+ end
47
+
48
+ def is_working?
49
+ @working
50
+ end
51
+
52
+ def is_stopping?
53
+ @stopping
54
+ end
55
+
56
+ end
57
+
58
+ end
@@ -0,0 +1,3 @@
1
+ test:
2
+ adapter: sqlite3
3
+ database: /tmp/smq-test.sqlite
@@ -0,0 +1,21 @@
1
+ require 'rubygems'
2
+ require 'test/unit'
3
+ begin; require 'redgreen'; rescue LoadError; end
4
+
5
+ $LOAD_PATH.unshift(File.dirname(__FILE__))
6
+ $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
7
+ require 'smq'
8
+
9
+ ActiveRecord::Base.configurations = YAML.load_file(File.dirname(__FILE__) + '/database.yml')
10
+ ActiveRecord::Base.establish_connection 'test'
11
+
12
+ SMQ::load_schema! # will check if table already exists
13
+ SMQ::flush_all_queues! # clear out old test data
14
+
15
+ class Test::Unit::TestCase
16
+
17
+ def populate_queue(queue, msgs = 5)
18
+ msgs.times { |i| SMQ::Message.build(i, queue.to_s).save! }
19
+ end
20
+
21
+ end
@@ -0,0 +1,112 @@
1
+ require 'helper'
2
+
3
+ class MessageTest < Test::Unit::TestCase
4
+
5
+ def setup
6
+ @simple_value = "string"
7
+ @complex_value = SMQ::decode(SMQ::encode({ :a => 1, :b => 2, :c => [1,2,3] })) # JSON-ifying muddles things around
8
+ @msg = SMQ::Message.build(@simple_value, "test_queue")
9
+ @worker = SMQ::Worker.new("test_queue")
10
+ end
11
+
12
+ def teardown
13
+ SMQ.flush_all_queues!
14
+ end
15
+
16
+ def test_builing_message_returns_instance_of_message
17
+ assert_instance_of SMQ::Message, @msg
18
+ end
19
+
20
+ def test_building_message_correctly_encodes_payload_for_simple_value
21
+ assert_equal @msg.payload, SMQ::encode(@simple_value)
22
+ end
23
+
24
+ def test_building_message_correctly_encodes_payload_for_complex_value
25
+ msg = SMQ::Message.build(@complex_value)
26
+ assert_equal msg.payload, SMQ::encode(@complex_value)
27
+ end
28
+
29
+ def test_data_is_decoded_payload_for_simple_value
30
+ assert_equal @msg.data, @simple_value
31
+ end
32
+
33
+ def test_data_is_decoded_payload_for_complex_value
34
+ msg = SMQ::Message.build(@complex_value)
35
+ assert_equal msg.data, @complex_value
36
+ end
37
+
38
+ def test_setting_data_correctly_serialises_to_payload
39
+ @msg.data = @complex_value
40
+ assert_equal @msg.data, @complex_value
41
+ assert_equal @msg.payload, SMQ::encode(@complex_value)
42
+ end
43
+
44
+ def test_complete_sets_completed_at_timestamp
45
+ @msg.complete
46
+ assert_not_nil @msg.completed_at
47
+ end
48
+
49
+ def test_fail_sets_completed_and_fail_at_timestamps
50
+ @msg.fail
51
+ assert_not_nil @msg.completed_at
52
+ assert_not_nil @msg.failed_at
53
+ end
54
+
55
+ def test_message_should_not_save_without_a_queue_name
56
+ @msg.queue = nil
57
+ assert_raise ActiveRecord::RecordInvalid do
58
+ @msg.save!
59
+ end
60
+ end
61
+
62
+ def test_ack_marks_message_as_complete_and_saves
63
+ @msg.ack!
64
+ assert_not_nil @msg.completed_at
65
+ assert_not_nil @msg.id
66
+ end
67
+
68
+ def test_should_lock_an_unlocked_message
69
+ @msg.save!
70
+ msg = @msg.lock!(@worker)
71
+ assert_equal msg.id, @msg.id
72
+ assert_not_nil @msg.locked_at
73
+ assert_equal @msg.locked_by, @worker.name
74
+ end
75
+
76
+ def test_should_return_nil_when_locking_a_completed_message
77
+ @msg.ack!
78
+ assert_nil @msg.lock!(@worker)
79
+ end
80
+
81
+ def test_should_return_nil_when_locking_a_locked_message_locked_by_someone_else
82
+ @msg.locked_by = 'random_worker'
83
+ @msg.save!
84
+ assert_nil @msg.lock!(@worker)
85
+ end
86
+
87
+ def test_locking_a_message_increments_the_attempts_count
88
+ @msg.save!
89
+ assert_equal @msg.attempts + 1, @msg.lock!(SMQ::Worker.new("test_queue")).attempts
90
+ end
91
+
92
+ def test_message_should_report_as_locked
93
+ @msg.save!
94
+ assert !@msg.locked?, "Message locked"
95
+ @msg.lock!(@worker)
96
+ assert @msg.locked?, "Message not locked"
97
+ end
98
+
99
+ def test_message_should_report_as_failed
100
+ assert !@msg.failed?, "Message failed"
101
+ @msg.fail
102
+ @msg.save!
103
+ assert @msg.failed?, "Message not failed"
104
+ end
105
+
106
+ def test_message_should_report_as_completed
107
+ assert !@msg.completed?, "Message completed"
108
+ @msg.ack!
109
+ assert @msg.completed?, "Message not completed"
110
+ end
111
+
112
+ end
@@ -0,0 +1,89 @@
1
+ require 'helper'
2
+
3
+ class QueueTest < Test::Unit::TestCase
4
+
5
+ def setup
6
+ @queue = SMQ::Queue.new("test_queue")
7
+ @worker = SMQ::Worker.new(@queue.name)
8
+ end
9
+
10
+ def teardown
11
+ SMQ.flush_all_queues!
12
+ end
13
+
14
+ def test_pushing_a_message_should_add_it_to_the_queue
15
+ msg = SMQ::Message.build("payload")
16
+ @queue.push(msg)
17
+ assert_equal 1, @queue.length
18
+ end
19
+
20
+ def test_enqueueing_a_payload_should_create_a_message_and_add_it_to_the_queue
21
+ msg = @queue.enqueue("payload")
22
+ assert_instance_of SMQ::Message, msg
23
+ assert_equal 1, @queue.length
24
+ end
25
+
26
+ def test_should_reserve_a_message_when_one_is_available
27
+ populate_queue(@queue, 1)
28
+ assert_instance_of SMQ::Message, @queue.reserve(@worker)
29
+ end
30
+
31
+ def test_reserve_should_return_nil_when_no_message_to_reserve
32
+ assert_nil @queue.reserve(@worker)
33
+ end
34
+
35
+ def test_find_available_should_return_up_to_batch_size_queued_messages
36
+ populate_queue(@queue, 10)
37
+ @queue.batch_size = 3
38
+ assert_equal @queue.batch_size, @queue.find_available.length
39
+ end
40
+
41
+ def test_flush_should_clear_all_messages_from_the_queue
42
+ populate_queue(@queue, 5)
43
+ @queue.flush!
44
+ assert_equal 0, @queue.length
45
+ end
46
+
47
+ def test_clearing_completed_should_not_touch_incomplete_messages
48
+ populate_queue(@queue, 5)
49
+ SMQ::Message.find(:first).ack!
50
+ @queue.clear_completed!
51
+ assert_equal 4, @queue.length
52
+ end
53
+
54
+ def test_clearing_successful_should_not_touch_failed_messages
55
+ populate_and_ack_all_but_fail_one
56
+ @queue.clear_successful!
57
+ assert_equal 1, SMQ::Message.count(:conditions => ["queue = ?", @queue.name])
58
+ end
59
+
60
+ def test_clearing_failures_should_not_touch_successful_messages
61
+ populate_and_ack_all_but_fail_one
62
+ @queue.clear_failed!
63
+ assert_equal 4, SMQ::Message.count(:conditions => ["queue = ?", @queue.name])
64
+ end
65
+
66
+ def test_clearing_one_queue_should_not_affect_another
67
+ @other_queue = SMQ::Queue.new("other_queue")
68
+ populate_queue(@queue, 5)
69
+ populate_queue(@other_queue, 5)
70
+ @queue.flush!
71
+ assert_equal 5, @other_queue.length
72
+ end
73
+
74
+ def test_should_return_correct_queue_size
75
+ populate_queue(@queue, 5)
76
+ assert_equal 5, @queue.length
77
+ end
78
+
79
+ private
80
+
81
+ def populate_and_ack_all_but_fail_one
82
+ populate_queue(@queue, 5)
83
+ msg = SMQ::Message.find(:first)
84
+ msg.fail
85
+ msg.save!
86
+ @worker.work(true) { |m| m.ack! }
87
+ end
88
+
89
+ end
@@ -0,0 +1,60 @@
1
+ require 'helper'
2
+
3
+ class WorkerTest < Test::Unit::TestCase
4
+
5
+ def setup
6
+ @worker = SMQ::Worker.new("test_queue")
7
+ end
8
+
9
+ def teardown
10
+ SMQ.flush_all_queues!
11
+ end
12
+
13
+ def test_new_worker_should_have_correctly_named_queue
14
+ assert_equal "test_queue", @worker.queue.name
15
+ end
16
+
17
+ def test_can_init_worker_with_queue_instance
18
+ worker = SMQ::Worker.new(@worker.queue)
19
+ assert_instance_of SMQ::Queue, worker.queue
20
+ assert_equal @worker.queue.name, worker.queue.name
21
+ end
22
+
23
+ def test_working_should_yield_a_message_when_one_is_available
24
+ SMQ::Message.build("string", @worker.queue.name).save!
25
+ yielded_msg = nil
26
+ @worker.work_one_message { |m| yielded_msg = m }
27
+ assert_instance_of SMQ::Message, yielded_msg
28
+ end
29
+
30
+ def test_working_should_return_message_locked_to_correct_worker
31
+ SMQ::Message.build("string", @worker.queue.name).save!
32
+ assert_equal @worker.work_one_message.locked_by, @worker.name
33
+ end
34
+
35
+ def test_working_should_return_nil_when_none_are_available
36
+ assert_nil @worker.work_one_message
37
+ end
38
+
39
+ def test_working_an_empty_queue_until_empty_should_do_nothing
40
+ block_calls = 0
41
+ @worker.work(true) { block_calls += 1 }
42
+ assert_equal 0, block_calls
43
+ end
44
+
45
+ def test_working_a_populated_queue_until_empty_should_work_all_jobs
46
+ populate_queue(@worker.queue.name, 5)
47
+ block_calls = 0
48
+ @worker.work(true) { block_calls += 1 }
49
+ assert_equal 5, block_calls
50
+ end
51
+
52
+ def test_working_continues_until_told_to_stop
53
+ t = Thread.new { @worker.work }
54
+ assert @worker.is_working?, "Not working"
55
+ @worker.stop!
56
+ sleep 2
57
+ assert !@worker.is_working?, "Working and should have stopped"
58
+ end
59
+
60
+ end
metadata ADDED
@@ -0,0 +1,117 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: smq
3
+ version: !ruby/object:Gem::Version
4
+ prerelease: false
5
+ segments:
6
+ - 0
7
+ - 1
8
+ - 1
9
+ version: 0.1.1
10
+ platform: ruby
11
+ authors:
12
+ - Tim Blair
13
+ autorequire:
14
+ bindir: bin
15
+ cert_chain: []
16
+
17
+ date: 2010-03-10 00:00:00 +00:00
18
+ default_executable:
19
+ dependencies:
20
+ - !ruby/object:Gem::Dependency
21
+ name: activerecord
22
+ prerelease: false
23
+ requirement: &id001 !ruby/object:Gem::Requirement
24
+ requirements:
25
+ - - ">="
26
+ - !ruby/object:Gem::Version
27
+ segments:
28
+ - 0
29
+ version: "0"
30
+ type: :runtime
31
+ version_requirements: *id001
32
+ - !ruby/object:Gem::Dependency
33
+ name: yajl-ruby
34
+ prerelease: false
35
+ requirement: &id002 !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - ">="
38
+ - !ruby/object:Gem::Version
39
+ segments:
40
+ - 0
41
+ version: "0"
42
+ type: :runtime
43
+ version_requirements: *id002
44
+ - !ruby/object:Gem::Dependency
45
+ name: sqlite3-ruby
46
+ prerelease: false
47
+ requirement: &id003 !ruby/object:Gem::Requirement
48
+ requirements:
49
+ - - ">="
50
+ - !ruby/object:Gem::Version
51
+ segments:
52
+ - 0
53
+ version: "0"
54
+ type: :development
55
+ version_requirements: *id003
56
+ description: A simple database-backed, JSON-based message queue and worker base
57
+ email: tim@bla.ir
58
+ executables: []
59
+
60
+ extensions: []
61
+
62
+ extra_rdoc_files:
63
+ - LICENSE
64
+ - README.markdown
65
+ files:
66
+ - .gitignore
67
+ - LICENSE
68
+ - README.markdown
69
+ - Rakefile
70
+ - VERSION
71
+ - examples/database.yml
72
+ - examples/worker.rb
73
+ - lib/smq.rb
74
+ - lib/smq/message.rb
75
+ - lib/smq/queue.rb
76
+ - lib/smq/worker.rb
77
+ - test/database.yml
78
+ - test/helper.rb
79
+ - test/test_message.rb
80
+ - test/test_queue.rb
81
+ - test/test_worker.rb
82
+ has_rdoc: true
83
+ homepage: http://github.com/timblair/smq
84
+ licenses: []
85
+
86
+ post_install_message:
87
+ rdoc_options:
88
+ - --charset=UTF-8
89
+ require_paths:
90
+ - lib
91
+ required_ruby_version: !ruby/object:Gem::Requirement
92
+ requirements:
93
+ - - ">="
94
+ - !ruby/object:Gem::Version
95
+ segments:
96
+ - 0
97
+ version: "0"
98
+ required_rubygems_version: !ruby/object:Gem::Requirement
99
+ requirements:
100
+ - - ">="
101
+ - !ruby/object:Gem::Version
102
+ segments:
103
+ - 0
104
+ version: "0"
105
+ requirements: []
106
+
107
+ rubyforge_project:
108
+ rubygems_version: 1.3.6
109
+ signing_key:
110
+ specification_version: 3
111
+ summary: Simple Message Queue
112
+ test_files:
113
+ - test/helper.rb
114
+ - test/test_message.rb
115
+ - test/test_queue.rb
116
+ - test/test_worker.rb
117
+ - examples/worker.rb