smq 0.1.1
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +1 -0
- data/LICENSE +22 -0
- data/README.markdown +88 -0
- data/Rakefile +53 -0
- data/VERSION +1 -0
- data/examples/database.yml +3 -0
- data/examples/worker.rb +25 -0
- data/lib/smq.rb +51 -0
- data/lib/smq/message.rb +66 -0
- data/lib/smq/queue.rb +65 -0
- data/lib/smq/worker.rb +58 -0
- data/test/database.yml +3 -0
- data/test/helper.rb +21 -0
- data/test/test_message.rb +112 -0
- data/test/test_queue.rb +89 -0
- data/test/test_worker.rb +60 -0
- metadata +117 -0
data/.gitignore
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
pkg
|
data/LICENSE
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2010 Tim Blair <tim@bla.ir>
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person
|
4
|
+
obtaining a copy of this software and associated documentation
|
5
|
+
files (the "Software"), to deal in the Software without
|
6
|
+
restriction, including without limitation the rights to use,
|
7
|
+
copy, modify, merge, publish, distribute, sublicense, and/or sell
|
8
|
+
copies of the Software, and to permit persons to whom the
|
9
|
+
Software is furnished to do so, subject to the following
|
10
|
+
conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be
|
13
|
+
included in all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
16
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
|
17
|
+
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
18
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
|
19
|
+
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
20
|
+
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
21
|
+
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
|
22
|
+
OTHER DEALINGS IN THE SOFTWARE.
|
data/README.markdown
ADDED
@@ -0,0 +1,88 @@
|
|
1
|
+
# Simple Message Queue (SMQ)
|
2
|
+
|
3
|
+
SMQ is a database-backed, JSON-based message queue and worker platform.
|
4
|
+
|
5
|
+
## Description
|
6
|
+
|
7
|
+
SMQ uses ActiveRecord to provide a database-agnositc, JSON-based [message queue](http://en.wikipedia.org/wiki/Message_queue) and worker platform; this is not to be confused with [job queues](http://en.wikipedia.org/wiki/Job_queue) such as [Resque](http://github.com/defunkt/resque) or [Delayed::Job](http://github.com/tobi/delayed_job).
|
8
|
+
|
9
|
+
Other simple message queue systems exist, but these generally either use Marshal'd ruby objects which are not transferable to other platforms, or are run as a separate server daemon. SMQ uses a simple database table for persisting message queues and JSON encoding for message payloads to enable use cross-platform.
|
10
|
+
|
11
|
+
## Installation
|
12
|
+
|
13
|
+
SMQ is provided as a gem courtesy of Gemcutter:
|
14
|
+
|
15
|
+
gem install smq
|
16
|
+
|
17
|
+
Once you've established your ActiveRecord connection, you'll need to initialise the (single) database table:
|
18
|
+
|
19
|
+
SMQ.load_schema!
|
20
|
+
|
21
|
+
## Queues
|
22
|
+
|
23
|
+
Creating a queue is as simple as:
|
24
|
+
|
25
|
+
queue = SMQ::Queue.new("queue_name")
|
26
|
+
|
27
|
+
A "queue" doesn't exist as a persisted entity: it's effectively just a tag applied to a message; pushing a message onto a queue simply saves the Message with the queue name assigned.
|
28
|
+
|
29
|
+
## Adding Messages to a Queue
|
30
|
+
|
31
|
+
A message's payload can be any data structure that can be serialised to JSON. There are three ways that a message can be added to a queue (here we're assuming that `msg_data` already exists):
|
32
|
+
|
33
|
+
queue.enqueue(msg_data)
|
34
|
+
|
35
|
+
# or
|
36
|
+
|
37
|
+
queue.push(SMQ::Message.build(msg_data))
|
38
|
+
|
39
|
+
# or
|
40
|
+
|
41
|
+
SMQ::Message.build(msg_data, "queue_name").save
|
42
|
+
|
43
|
+
The first two methods are effectively simplified abstractions of the third. Note that a single message can only be added to one queue; to push to multiple queues, additional instances of the message must be created.
|
44
|
+
|
45
|
+
### Non-Ruby Messages
|
46
|
+
|
47
|
+
Adding to a queue from outside a ruby environment is as simple as `INSERT`ing a JSON-encoded packet into the queue table:
|
48
|
+
|
49
|
+
INSERT INTO smq_messages (queue, payload, created_at) VALUES (
|
50
|
+
'queue_name',
|
51
|
+
'{"json":"encoded stuff"}',
|
52
|
+
NOW()
|
53
|
+
)
|
54
|
+
|
55
|
+
## Workers
|
56
|
+
|
57
|
+
A worker is an instance of `SMQ::Worker` bound to a single named queue. The `Worker#work` method takes care of reserving `Message`s and then passing them back to the given block:
|
58
|
+
|
59
|
+
SMQ::Worker.new("queue_name").work do |msg|
|
60
|
+
puts msg.data.inspect
|
61
|
+
msg.ack!
|
62
|
+
end
|
63
|
+
|
64
|
+
When a worker has finished with a `Message`, it should either call `Message#ack!` to acknowledge receipt of that message, or `Message#fail` followed by `Message#save` to mark the message as failed. In addition, if a message should fail for a transient reason, it can be pushed back into the queue by calling `Message#retry!`. A message will be retried up to 5 times before automatically being marked as failed.
|
65
|
+
|
66
|
+
`Worker#work` takes a single optional argument (in addition to the callback block): `until_empty`. If `true`, the worker will stop when the queue is empty (but see detail on locking below); if `false` the worker will continue to look for new jobs indefinitely (or until `Worker#stop!` is called), waiting 1 second between queue polls.
|
67
|
+
|
68
|
+
### Locking Strategy
|
69
|
+
|
70
|
+
To facilitate the locking ("delivery") of individual messages, the following strategy is used:
|
71
|
+
|
72
|
+
1. Find the next 5 messages in the queue;
|
73
|
+
2. Randomise these messages;
|
74
|
+
3. Attempt to `UPDATE` each message based on the message ID, lock status and last updated stamp;
|
75
|
+
4. When the `UPDATE` returns a row change count of 1, that message has been locked and can be passed off to be processed;
|
76
|
+
5. If a lock isn't aquired for any of the 5 messages, either wait for 1 second and then start the process again or, if `until_empty` is `true`, end the processing loop and stop the worker.
|
77
|
+
|
78
|
+
The effect of this is that, although multiple workers can be employed on a single queue, the increase in throughput is not linear as may be expected. In fact, if you increase the number of workers on queue past 5, throughput may actually diminish due to the extra time spent attempting to aquire a message lock.
|
79
|
+
|
80
|
+
This process, although heavy on `SELECT`s, results in a minimum of table locking which would actually slow the lock process down. For example, by using an `UPDATE` with a `LIMIT` of `1` when the queue size is more than a couple of hundred results in a noticible slow down in lock aquisition.
|
81
|
+
|
82
|
+
These limitations are deemed acceptable due to the simple nature of this queueing system.
|
83
|
+
|
84
|
+
## Licensing and Attribution
|
85
|
+
|
86
|
+
SMQ is released under the MIT license as detailed in the LICENSE file that should be distributed with this library; the source code is [freely available](http://github.com/timblair/smq).
|
87
|
+
|
88
|
+
SMQ was developed by [Tim Blair](http://tim.bla.ir/) during work on [White Label Dating](http://www.whitelabeldating.com/), while employed by [Global Personals Ltd](http://www.globalpersonals.co.uk). Global Personals Ltd have kindly agreed to the extraction and release of this software under the license terms above.
|
data/Rakefile
ADDED
@@ -0,0 +1,53 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'rake'
|
3
|
+
|
4
|
+
begin
|
5
|
+
require 'jeweler'
|
6
|
+
Jeweler::Tasks.new do |gem|
|
7
|
+
gem.name = "smq"
|
8
|
+
gem.summary = %Q{Simple Message Queue}
|
9
|
+
gem.description = %Q{A simple database-backed, JSON-based message queue and worker base}
|
10
|
+
gem.email = "tim@bla.ir"
|
11
|
+
gem.homepage = "http://github.com/timblair/smq"
|
12
|
+
gem.authors = ["Tim Blair"]
|
13
|
+
gem.add_dependency 'activerecord'
|
14
|
+
gem.add_dependency 'yajl-ruby'
|
15
|
+
gem.add_development_dependency 'sqlite3-ruby'
|
16
|
+
end
|
17
|
+
Jeweler::GemcutterTasks.new
|
18
|
+
task :test => :check_dependencies
|
19
|
+
rescue LoadError
|
20
|
+
puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
|
21
|
+
end
|
22
|
+
|
23
|
+
require 'rake/testtask'
|
24
|
+
Rake::TestTask.new(:test) do |test|
|
25
|
+
test.libs << 'lib' << 'test'
|
26
|
+
test.pattern = 'test/**/test_*.rb'
|
27
|
+
test.verbose = true
|
28
|
+
end
|
29
|
+
|
30
|
+
begin
|
31
|
+
require 'rcov/rcovtask'
|
32
|
+
Rcov::RcovTask.new do |test|
|
33
|
+
test.libs << 'test'
|
34
|
+
test.pattern = 'test/**/test_*.rb'
|
35
|
+
test.verbose = true
|
36
|
+
end
|
37
|
+
rescue LoadError
|
38
|
+
task :rcov do
|
39
|
+
abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
task :default => :test
|
44
|
+
|
45
|
+
require 'rake/rdoctask'
|
46
|
+
Rake::RDocTask.new do |rdoc|
|
47
|
+
version = File.exist?('VERSION') ? File.read('VERSION') : ""
|
48
|
+
|
49
|
+
rdoc.rdoc_dir = 'rdoc'
|
50
|
+
rdoc.title = "smq #{version}"
|
51
|
+
rdoc.rdoc_files.include('README*')
|
52
|
+
rdoc.rdoc_files.include('lib/**/*.rb')
|
53
|
+
end
|
data/VERSION
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
0.1.1
|
data/examples/worker.rb
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
|
3
|
+
begin
|
4
|
+
require 'smq'
|
5
|
+
rescue LoadError
|
6
|
+
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
|
7
|
+
require 'smq'
|
8
|
+
end
|
9
|
+
|
10
|
+
ActiveRecord::Base.configurations = YAML.load_file(File.dirname(__FILE__) + '/database.yml')
|
11
|
+
ActiveRecord::Base.establish_connection 'development'
|
12
|
+
|
13
|
+
SMQ::load_schema!
|
14
|
+
|
15
|
+
worker = SMQ::Worker.new("queue_name")
|
16
|
+
100.times { |i| worker.queue.enqueue(i+1) }
|
17
|
+
|
18
|
+
count = 0
|
19
|
+
worker.work(true) do |msg|
|
20
|
+
count += 1
|
21
|
+
puts msg.inspect
|
22
|
+
msg.ack!
|
23
|
+
end
|
24
|
+
|
25
|
+
puts "Handled: #{count}"
|
data/lib/smq.rb
ADDED
@@ -0,0 +1,51 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'active_record'
|
3
|
+
require 'yajl'
|
4
|
+
|
5
|
+
require File.dirname(__FILE__) + '/smq/worker'
|
6
|
+
require File.dirname(__FILE__) + '/smq/message'
|
7
|
+
require File.dirname(__FILE__) + '/smq/queue'
|
8
|
+
|
9
|
+
module SMQ
|
10
|
+
|
11
|
+
def self.now
|
12
|
+
(ActiveRecord::Base.default_timezone == :utc) ? Time.now.utc : Time.now
|
13
|
+
end
|
14
|
+
|
15
|
+
def self.encode(object)
|
16
|
+
Yajl::Encoder.encode(object)
|
17
|
+
end
|
18
|
+
|
19
|
+
def self.decode(object)
|
20
|
+
return unless object
|
21
|
+
Yajl::Parser.parse(object, :check_utf8 => false)
|
22
|
+
end
|
23
|
+
|
24
|
+
def self.load_schema!(force = false)
|
25
|
+
return if !force && ActiveRecord::Base.connection.tables.include?(SMQ::Message.table_name)
|
26
|
+
ActiveRecord::Schema.define do
|
27
|
+
create_table :smq_messages, :force => force do |t|
|
28
|
+
t.string :queue, :limit => 30, :null => false
|
29
|
+
t.text :payload
|
30
|
+
t.datetime :locked_at
|
31
|
+
t.string :locked_by, :limit => 50
|
32
|
+
t.integer :attempts, :limit => 2, :default => 0
|
33
|
+
t.datetime :failed_at
|
34
|
+
t.datetime :completed_at
|
35
|
+
t.timestamps
|
36
|
+
end
|
37
|
+
if ActiveRecord::Base.connection.instance_variable_get(:@config)[:adapter] == 'mysql'
|
38
|
+
execute "ALTER TABLE `smq_messages` MODIFY COLUMN `updated_at` TIMESTAMP"
|
39
|
+
end
|
40
|
+
add_index "smq_messages", ["queue", "completed_at", "locked_by"], :name => "idx_smq_available"
|
41
|
+
add_index "smq_messages", ["id", "updated_at", "locked_by", "completed_at"], :name => "idx_smq_unlocked"
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def self.flush_all_queues!
|
46
|
+
# yes, not the most efficient, but *should* only be called during testing
|
47
|
+
# so it's not that much of a concern
|
48
|
+
SMQ::Message.delete_all
|
49
|
+
end
|
50
|
+
|
51
|
+
end
|
data/lib/smq/message.rb
ADDED
@@ -0,0 +1,66 @@
|
|
1
|
+
module SMQ
|
2
|
+
|
3
|
+
class Message < ActiveRecord::Base
|
4
|
+
set_table_name :smq_messages
|
5
|
+
validates_presence_of :queue
|
6
|
+
MAX_ATTEMPTS = 5
|
7
|
+
|
8
|
+
def self.build(payload, queue = nil)
|
9
|
+
msg = self.new
|
10
|
+
msg.payload = SMQ.encode(payload)
|
11
|
+
msg.queue = queue
|
12
|
+
msg
|
13
|
+
end
|
14
|
+
|
15
|
+
def data
|
16
|
+
SMQ.decode(payload)
|
17
|
+
end
|
18
|
+
def data=(object)
|
19
|
+
self.payload = SMQ.encode(object)
|
20
|
+
end
|
21
|
+
|
22
|
+
def retry!
|
23
|
+
self.locked_by = nil
|
24
|
+
self.locked_at = nil
|
25
|
+
fail if self.attempts >= MAX_ATTEMPTS
|
26
|
+
save!
|
27
|
+
end
|
28
|
+
|
29
|
+
def lock!(worker)
|
30
|
+
rows = self.class.update_all(
|
31
|
+
["updated_at = ?, locked_at = ?, locked_by = ?, attempts = (attempts+1)", SMQ.now, SMQ.now, worker.name],
|
32
|
+
["id = ? AND updated_at = ? AND locked_by IS NULL AND completed_at IS NULL", self.id, self.updated_at]
|
33
|
+
)
|
34
|
+
if rows == 1
|
35
|
+
self.reload
|
36
|
+
return self
|
37
|
+
end
|
38
|
+
nil
|
39
|
+
end
|
40
|
+
|
41
|
+
def locked?
|
42
|
+
!self.locked_by.nil?
|
43
|
+
end
|
44
|
+
def failed?
|
45
|
+
!self.failed_at.nil?
|
46
|
+
end
|
47
|
+
def completed?
|
48
|
+
!self.completed_at.nil?
|
49
|
+
end
|
50
|
+
|
51
|
+
def complete
|
52
|
+
self.completed_at = SMQ.now
|
53
|
+
end
|
54
|
+
def fail
|
55
|
+
self.failed_at = SMQ.now
|
56
|
+
complete
|
57
|
+
end
|
58
|
+
|
59
|
+
def ack!
|
60
|
+
complete
|
61
|
+
save!
|
62
|
+
end
|
63
|
+
|
64
|
+
end
|
65
|
+
|
66
|
+
end
|
data/lib/smq/queue.rb
ADDED
@@ -0,0 +1,65 @@
|
|
1
|
+
module SMQ
|
2
|
+
|
3
|
+
class Queue
|
4
|
+
|
5
|
+
attr_accessor :name
|
6
|
+
attr_accessor :batch_size
|
7
|
+
|
8
|
+
def initialize(name, batch_size = 5)
|
9
|
+
@name = name
|
10
|
+
@batch_size = batch_size
|
11
|
+
end
|
12
|
+
|
13
|
+
def to_s
|
14
|
+
@name
|
15
|
+
end
|
16
|
+
|
17
|
+
def push(msg)
|
18
|
+
msg.queue = @name
|
19
|
+
msg.save!
|
20
|
+
msg
|
21
|
+
end
|
22
|
+
|
23
|
+
def enqueue(payload)
|
24
|
+
push SMQ::Message.build(payload)
|
25
|
+
end
|
26
|
+
|
27
|
+
def reserve(worker)
|
28
|
+
find_available.each do |msg|
|
29
|
+
m = msg.lock!(worker)
|
30
|
+
return m unless m == nil
|
31
|
+
end
|
32
|
+
nil
|
33
|
+
end
|
34
|
+
|
35
|
+
def find_available
|
36
|
+
SMQ::Message.find(:all, :select => 'id, updated_at', :conditions => ["queue = ? AND completed_at IS NULL AND locked_by IS NULL", @name], :order => "id ASC", :limit => @batch_size).sort_by { rand() }
|
37
|
+
end
|
38
|
+
|
39
|
+
def length
|
40
|
+
SMQ::Message.count(:conditions => ["queue = ? AND completed_at IS NULL", @name])
|
41
|
+
end
|
42
|
+
|
43
|
+
def flush!
|
44
|
+
delete_queue_items
|
45
|
+
end
|
46
|
+
|
47
|
+
def clear_completed!
|
48
|
+
delete_queue_items "completed_at IS NOT NULL"
|
49
|
+
end
|
50
|
+
def clear_successful!
|
51
|
+
delete_queue_items "completed_at IS NOT NULL AND failed_at IS NULL"
|
52
|
+
end
|
53
|
+
def clear_failed!
|
54
|
+
delete_queue_items "completed_at IS NOT NULL AND failed_at IS NOT NULL"
|
55
|
+
end
|
56
|
+
|
57
|
+
private
|
58
|
+
|
59
|
+
def delete_queue_items(where = nil)
|
60
|
+
SMQ::Message.delete_all(["queue = ? #{where ? 'AND ' + where : ''}", @name])
|
61
|
+
end
|
62
|
+
|
63
|
+
end
|
64
|
+
|
65
|
+
end
|
data/lib/smq/worker.rb
ADDED
@@ -0,0 +1,58 @@
|
|
1
|
+
require "socket"
|
2
|
+
|
3
|
+
module SMQ
|
4
|
+
|
5
|
+
class Worker
|
6
|
+
|
7
|
+
attr_accessor :name
|
8
|
+
attr_accessor :queue
|
9
|
+
|
10
|
+
@working = false
|
11
|
+
@stopping = false
|
12
|
+
|
13
|
+
def initialize(queue)
|
14
|
+
self.name = "#{Socket.gethostname}:#{Process.pid}" rescue "pid:#{Process.pid}"
|
15
|
+
self.queue = queue.instance_of?(SMQ::Queue) ? queue : SMQ::Queue.new(queue)
|
16
|
+
end
|
17
|
+
|
18
|
+
def to_s
|
19
|
+
self.name
|
20
|
+
end
|
21
|
+
|
22
|
+
def work(until_empty = false, &block)
|
23
|
+
@working = true
|
24
|
+
empty = false
|
25
|
+
empty_for = 0
|
26
|
+
while(!empty && !@stopping) do
|
27
|
+
if work_one_message(&block).nil?
|
28
|
+
empty = true if until_empty
|
29
|
+
sleep 1 unless (empty && (empty_for += 1) >= 10)
|
30
|
+
else
|
31
|
+
empty_for = 0
|
32
|
+
end
|
33
|
+
end
|
34
|
+
@stopping = false
|
35
|
+
@working = false
|
36
|
+
end
|
37
|
+
|
38
|
+
def work_one_message
|
39
|
+
msg = self.queue.reserve(self)
|
40
|
+
yield msg if !msg.nil? && block_given?
|
41
|
+
msg
|
42
|
+
end
|
43
|
+
|
44
|
+
def stop!
|
45
|
+
@stopping = true
|
46
|
+
end
|
47
|
+
|
48
|
+
def is_working?
|
49
|
+
@working
|
50
|
+
end
|
51
|
+
|
52
|
+
def is_stopping?
|
53
|
+
@stopping
|
54
|
+
end
|
55
|
+
|
56
|
+
end
|
57
|
+
|
58
|
+
end
|
data/test/database.yml
ADDED
data/test/helper.rb
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'test/unit'
|
3
|
+
begin; require 'redgreen'; rescue LoadError; end
|
4
|
+
|
5
|
+
$LOAD_PATH.unshift(File.dirname(__FILE__))
|
6
|
+
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
|
7
|
+
require 'smq'
|
8
|
+
|
9
|
+
ActiveRecord::Base.configurations = YAML.load_file(File.dirname(__FILE__) + '/database.yml')
|
10
|
+
ActiveRecord::Base.establish_connection 'test'
|
11
|
+
|
12
|
+
SMQ::load_schema! # will check if table already exists
|
13
|
+
SMQ::flush_all_queues! # clear out old test data
|
14
|
+
|
15
|
+
class Test::Unit::TestCase
|
16
|
+
|
17
|
+
def populate_queue(queue, msgs = 5)
|
18
|
+
msgs.times { |i| SMQ::Message.build(i, queue.to_s).save! }
|
19
|
+
end
|
20
|
+
|
21
|
+
end
|
@@ -0,0 +1,112 @@
|
|
1
|
+
require 'helper'
|
2
|
+
|
3
|
+
class MessageTest < Test::Unit::TestCase
|
4
|
+
|
5
|
+
def setup
|
6
|
+
@simple_value = "string"
|
7
|
+
@complex_value = SMQ::decode(SMQ::encode({ :a => 1, :b => 2, :c => [1,2,3] })) # JSON-ifying muddles things around
|
8
|
+
@msg = SMQ::Message.build(@simple_value, "test_queue")
|
9
|
+
@worker = SMQ::Worker.new("test_queue")
|
10
|
+
end
|
11
|
+
|
12
|
+
def teardown
|
13
|
+
SMQ.flush_all_queues!
|
14
|
+
end
|
15
|
+
|
16
|
+
def test_builing_message_returns_instance_of_message
|
17
|
+
assert_instance_of SMQ::Message, @msg
|
18
|
+
end
|
19
|
+
|
20
|
+
def test_building_message_correctly_encodes_payload_for_simple_value
|
21
|
+
assert_equal @msg.payload, SMQ::encode(@simple_value)
|
22
|
+
end
|
23
|
+
|
24
|
+
def test_building_message_correctly_encodes_payload_for_complex_value
|
25
|
+
msg = SMQ::Message.build(@complex_value)
|
26
|
+
assert_equal msg.payload, SMQ::encode(@complex_value)
|
27
|
+
end
|
28
|
+
|
29
|
+
def test_data_is_decoded_payload_for_simple_value
|
30
|
+
assert_equal @msg.data, @simple_value
|
31
|
+
end
|
32
|
+
|
33
|
+
def test_data_is_decoded_payload_for_complex_value
|
34
|
+
msg = SMQ::Message.build(@complex_value)
|
35
|
+
assert_equal msg.data, @complex_value
|
36
|
+
end
|
37
|
+
|
38
|
+
def test_setting_data_correctly_serialises_to_payload
|
39
|
+
@msg.data = @complex_value
|
40
|
+
assert_equal @msg.data, @complex_value
|
41
|
+
assert_equal @msg.payload, SMQ::encode(@complex_value)
|
42
|
+
end
|
43
|
+
|
44
|
+
def test_complete_sets_completed_at_timestamp
|
45
|
+
@msg.complete
|
46
|
+
assert_not_nil @msg.completed_at
|
47
|
+
end
|
48
|
+
|
49
|
+
def test_fail_sets_completed_and_fail_at_timestamps
|
50
|
+
@msg.fail
|
51
|
+
assert_not_nil @msg.completed_at
|
52
|
+
assert_not_nil @msg.failed_at
|
53
|
+
end
|
54
|
+
|
55
|
+
def test_message_should_not_save_without_a_queue_name
|
56
|
+
@msg.queue = nil
|
57
|
+
assert_raise ActiveRecord::RecordInvalid do
|
58
|
+
@msg.save!
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
def test_ack_marks_message_as_complete_and_saves
|
63
|
+
@msg.ack!
|
64
|
+
assert_not_nil @msg.completed_at
|
65
|
+
assert_not_nil @msg.id
|
66
|
+
end
|
67
|
+
|
68
|
+
def test_should_lock_an_unlocked_message
|
69
|
+
@msg.save!
|
70
|
+
msg = @msg.lock!(@worker)
|
71
|
+
assert_equal msg.id, @msg.id
|
72
|
+
assert_not_nil @msg.locked_at
|
73
|
+
assert_equal @msg.locked_by, @worker.name
|
74
|
+
end
|
75
|
+
|
76
|
+
def test_should_return_nil_when_locking_a_completed_message
|
77
|
+
@msg.ack!
|
78
|
+
assert_nil @msg.lock!(@worker)
|
79
|
+
end
|
80
|
+
|
81
|
+
def test_should_return_nil_when_locking_a_locked_message_locked_by_someone_else
|
82
|
+
@msg.locked_by = 'random_worker'
|
83
|
+
@msg.save!
|
84
|
+
assert_nil @msg.lock!(@worker)
|
85
|
+
end
|
86
|
+
|
87
|
+
def test_locking_a_message_increments_the_attempts_count
|
88
|
+
@msg.save!
|
89
|
+
assert_equal @msg.attempts + 1, @msg.lock!(SMQ::Worker.new("test_queue")).attempts
|
90
|
+
end
|
91
|
+
|
92
|
+
def test_message_should_report_as_locked
|
93
|
+
@msg.save!
|
94
|
+
assert !@msg.locked?, "Message locked"
|
95
|
+
@msg.lock!(@worker)
|
96
|
+
assert @msg.locked?, "Message not locked"
|
97
|
+
end
|
98
|
+
|
99
|
+
def test_message_should_report_as_failed
|
100
|
+
assert !@msg.failed?, "Message failed"
|
101
|
+
@msg.fail
|
102
|
+
@msg.save!
|
103
|
+
assert @msg.failed?, "Message not failed"
|
104
|
+
end
|
105
|
+
|
106
|
+
def test_message_should_report_as_completed
|
107
|
+
assert !@msg.completed?, "Message completed"
|
108
|
+
@msg.ack!
|
109
|
+
assert @msg.completed?, "Message not completed"
|
110
|
+
end
|
111
|
+
|
112
|
+
end
|
data/test/test_queue.rb
ADDED
@@ -0,0 +1,89 @@
|
|
1
|
+
require 'helper'
|
2
|
+
|
3
|
+
class QueueTest < Test::Unit::TestCase
|
4
|
+
|
5
|
+
def setup
|
6
|
+
@queue = SMQ::Queue.new("test_queue")
|
7
|
+
@worker = SMQ::Worker.new(@queue.name)
|
8
|
+
end
|
9
|
+
|
10
|
+
def teardown
|
11
|
+
SMQ.flush_all_queues!
|
12
|
+
end
|
13
|
+
|
14
|
+
def test_pushing_a_message_should_add_it_to_the_queue
|
15
|
+
msg = SMQ::Message.build("payload")
|
16
|
+
@queue.push(msg)
|
17
|
+
assert_equal 1, @queue.length
|
18
|
+
end
|
19
|
+
|
20
|
+
def test_enqueueing_a_payload_should_create_a_message_and_add_it_to_the_queue
|
21
|
+
msg = @queue.enqueue("payload")
|
22
|
+
assert_instance_of SMQ::Message, msg
|
23
|
+
assert_equal 1, @queue.length
|
24
|
+
end
|
25
|
+
|
26
|
+
def test_should_reserve_a_message_when_one_is_available
|
27
|
+
populate_queue(@queue, 1)
|
28
|
+
assert_instance_of SMQ::Message, @queue.reserve(@worker)
|
29
|
+
end
|
30
|
+
|
31
|
+
def test_reserve_should_return_nil_when_no_message_to_reserve
|
32
|
+
assert_nil @queue.reserve(@worker)
|
33
|
+
end
|
34
|
+
|
35
|
+
def test_find_available_should_return_up_to_batch_size_queued_messages
|
36
|
+
populate_queue(@queue, 10)
|
37
|
+
@queue.batch_size = 3
|
38
|
+
assert_equal @queue.batch_size, @queue.find_available.length
|
39
|
+
end
|
40
|
+
|
41
|
+
def test_flush_should_clear_all_messages_from_the_queue
|
42
|
+
populate_queue(@queue, 5)
|
43
|
+
@queue.flush!
|
44
|
+
assert_equal 0, @queue.length
|
45
|
+
end
|
46
|
+
|
47
|
+
def test_clearing_completed_should_not_touch_incomplete_messages
|
48
|
+
populate_queue(@queue, 5)
|
49
|
+
SMQ::Message.find(:first).ack!
|
50
|
+
@queue.clear_completed!
|
51
|
+
assert_equal 4, @queue.length
|
52
|
+
end
|
53
|
+
|
54
|
+
def test_clearing_successful_should_not_touch_failed_messages
|
55
|
+
populate_and_ack_all_but_fail_one
|
56
|
+
@queue.clear_successful!
|
57
|
+
assert_equal 1, SMQ::Message.count(:conditions => ["queue = ?", @queue.name])
|
58
|
+
end
|
59
|
+
|
60
|
+
def test_clearing_failures_should_not_touch_successful_messages
|
61
|
+
populate_and_ack_all_but_fail_one
|
62
|
+
@queue.clear_failed!
|
63
|
+
assert_equal 4, SMQ::Message.count(:conditions => ["queue = ?", @queue.name])
|
64
|
+
end
|
65
|
+
|
66
|
+
def test_clearing_one_queue_should_not_affect_another
|
67
|
+
@other_queue = SMQ::Queue.new("other_queue")
|
68
|
+
populate_queue(@queue, 5)
|
69
|
+
populate_queue(@other_queue, 5)
|
70
|
+
@queue.flush!
|
71
|
+
assert_equal 5, @other_queue.length
|
72
|
+
end
|
73
|
+
|
74
|
+
def test_should_return_correct_queue_size
|
75
|
+
populate_queue(@queue, 5)
|
76
|
+
assert_equal 5, @queue.length
|
77
|
+
end
|
78
|
+
|
79
|
+
private
|
80
|
+
|
81
|
+
def populate_and_ack_all_but_fail_one
|
82
|
+
populate_queue(@queue, 5)
|
83
|
+
msg = SMQ::Message.find(:first)
|
84
|
+
msg.fail
|
85
|
+
msg.save!
|
86
|
+
@worker.work(true) { |m| m.ack! }
|
87
|
+
end
|
88
|
+
|
89
|
+
end
|
data/test/test_worker.rb
ADDED
@@ -0,0 +1,60 @@
|
|
1
|
+
require 'helper'
|
2
|
+
|
3
|
+
class WorkerTest < Test::Unit::TestCase
|
4
|
+
|
5
|
+
def setup
|
6
|
+
@worker = SMQ::Worker.new("test_queue")
|
7
|
+
end
|
8
|
+
|
9
|
+
def teardown
|
10
|
+
SMQ.flush_all_queues!
|
11
|
+
end
|
12
|
+
|
13
|
+
def test_new_worker_should_have_correctly_named_queue
|
14
|
+
assert_equal "test_queue", @worker.queue.name
|
15
|
+
end
|
16
|
+
|
17
|
+
def test_can_init_worker_with_queue_instance
|
18
|
+
worker = SMQ::Worker.new(@worker.queue)
|
19
|
+
assert_instance_of SMQ::Queue, worker.queue
|
20
|
+
assert_equal @worker.queue.name, worker.queue.name
|
21
|
+
end
|
22
|
+
|
23
|
+
def test_working_should_yield_a_message_when_one_is_available
|
24
|
+
SMQ::Message.build("string", @worker.queue.name).save!
|
25
|
+
yielded_msg = nil
|
26
|
+
@worker.work_one_message { |m| yielded_msg = m }
|
27
|
+
assert_instance_of SMQ::Message, yielded_msg
|
28
|
+
end
|
29
|
+
|
30
|
+
def test_working_should_return_message_locked_to_correct_worker
|
31
|
+
SMQ::Message.build("string", @worker.queue.name).save!
|
32
|
+
assert_equal @worker.work_one_message.locked_by, @worker.name
|
33
|
+
end
|
34
|
+
|
35
|
+
def test_working_should_return_nil_when_none_are_available
|
36
|
+
assert_nil @worker.work_one_message
|
37
|
+
end
|
38
|
+
|
39
|
+
def test_working_an_empty_queue_until_empty_should_do_nothing
|
40
|
+
block_calls = 0
|
41
|
+
@worker.work(true) { block_calls += 1 }
|
42
|
+
assert_equal 0, block_calls
|
43
|
+
end
|
44
|
+
|
45
|
+
def test_working_a_populated_queue_until_empty_should_work_all_jobs
|
46
|
+
populate_queue(@worker.queue.name, 5)
|
47
|
+
block_calls = 0
|
48
|
+
@worker.work(true) { block_calls += 1 }
|
49
|
+
assert_equal 5, block_calls
|
50
|
+
end
|
51
|
+
|
52
|
+
def test_working_continues_until_told_to_stop
|
53
|
+
t = Thread.new { @worker.work }
|
54
|
+
assert @worker.is_working?, "Not working"
|
55
|
+
@worker.stop!
|
56
|
+
sleep 2
|
57
|
+
assert !@worker.is_working?, "Working and should have stopped"
|
58
|
+
end
|
59
|
+
|
60
|
+
end
|
metadata
ADDED
@@ -0,0 +1,117 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: smq
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
prerelease: false
|
5
|
+
segments:
|
6
|
+
- 0
|
7
|
+
- 1
|
8
|
+
- 1
|
9
|
+
version: 0.1.1
|
10
|
+
platform: ruby
|
11
|
+
authors:
|
12
|
+
- Tim Blair
|
13
|
+
autorequire:
|
14
|
+
bindir: bin
|
15
|
+
cert_chain: []
|
16
|
+
|
17
|
+
date: 2010-03-10 00:00:00 +00:00
|
18
|
+
default_executable:
|
19
|
+
dependencies:
|
20
|
+
- !ruby/object:Gem::Dependency
|
21
|
+
name: activerecord
|
22
|
+
prerelease: false
|
23
|
+
requirement: &id001 !ruby/object:Gem::Requirement
|
24
|
+
requirements:
|
25
|
+
- - ">="
|
26
|
+
- !ruby/object:Gem::Version
|
27
|
+
segments:
|
28
|
+
- 0
|
29
|
+
version: "0"
|
30
|
+
type: :runtime
|
31
|
+
version_requirements: *id001
|
32
|
+
- !ruby/object:Gem::Dependency
|
33
|
+
name: yajl-ruby
|
34
|
+
prerelease: false
|
35
|
+
requirement: &id002 !ruby/object:Gem::Requirement
|
36
|
+
requirements:
|
37
|
+
- - ">="
|
38
|
+
- !ruby/object:Gem::Version
|
39
|
+
segments:
|
40
|
+
- 0
|
41
|
+
version: "0"
|
42
|
+
type: :runtime
|
43
|
+
version_requirements: *id002
|
44
|
+
- !ruby/object:Gem::Dependency
|
45
|
+
name: sqlite3-ruby
|
46
|
+
prerelease: false
|
47
|
+
requirement: &id003 !ruby/object:Gem::Requirement
|
48
|
+
requirements:
|
49
|
+
- - ">="
|
50
|
+
- !ruby/object:Gem::Version
|
51
|
+
segments:
|
52
|
+
- 0
|
53
|
+
version: "0"
|
54
|
+
type: :development
|
55
|
+
version_requirements: *id003
|
56
|
+
description: A simple database-backed, JSON-based message queue and worker base
|
57
|
+
email: tim@bla.ir
|
58
|
+
executables: []
|
59
|
+
|
60
|
+
extensions: []
|
61
|
+
|
62
|
+
extra_rdoc_files:
|
63
|
+
- LICENSE
|
64
|
+
- README.markdown
|
65
|
+
files:
|
66
|
+
- .gitignore
|
67
|
+
- LICENSE
|
68
|
+
- README.markdown
|
69
|
+
- Rakefile
|
70
|
+
- VERSION
|
71
|
+
- examples/database.yml
|
72
|
+
- examples/worker.rb
|
73
|
+
- lib/smq.rb
|
74
|
+
- lib/smq/message.rb
|
75
|
+
- lib/smq/queue.rb
|
76
|
+
- lib/smq/worker.rb
|
77
|
+
- test/database.yml
|
78
|
+
- test/helper.rb
|
79
|
+
- test/test_message.rb
|
80
|
+
- test/test_queue.rb
|
81
|
+
- test/test_worker.rb
|
82
|
+
has_rdoc: true
|
83
|
+
homepage: http://github.com/timblair/smq
|
84
|
+
licenses: []
|
85
|
+
|
86
|
+
post_install_message:
|
87
|
+
rdoc_options:
|
88
|
+
- --charset=UTF-8
|
89
|
+
require_paths:
|
90
|
+
- lib
|
91
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
92
|
+
requirements:
|
93
|
+
- - ">="
|
94
|
+
- !ruby/object:Gem::Version
|
95
|
+
segments:
|
96
|
+
- 0
|
97
|
+
version: "0"
|
98
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
99
|
+
requirements:
|
100
|
+
- - ">="
|
101
|
+
- !ruby/object:Gem::Version
|
102
|
+
segments:
|
103
|
+
- 0
|
104
|
+
version: "0"
|
105
|
+
requirements: []
|
106
|
+
|
107
|
+
rubyforge_project:
|
108
|
+
rubygems_version: 1.3.6
|
109
|
+
signing_key:
|
110
|
+
specification_version: 3
|
111
|
+
summary: Simple Message Queue
|
112
|
+
test_files:
|
113
|
+
- test/helper.rb
|
114
|
+
- test/test_message.rb
|
115
|
+
- test/test_queue.rb
|
116
|
+
- test/test_worker.rb
|
117
|
+
- examples/worker.rb
|