simple_message_queue 0.0.1
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +134 -0
- data/lib/config.yml +3 -0
- data/lib/config.yml.sample +4 -0
- data/lib/simple_message_queue.rb +73 -0
- data/lib/simple_message_queue/configuration.rb +13 -0
- data/lib/simple_message_queue/core_ext/string.rb +11 -0
- data/lib/simple_message_queue/errors.rb +17 -0
- data/lib/simple_message_queue/version.rb +3 -0
- metadata +75 -0
data/README.md
ADDED
@@ -0,0 +1,134 @@
|
|
1
|
+
# Simple Message Queue
|
2
|
+
|
3
|
+
Simple Message Queue is a gem to create and receive messages from AWS's SQS. You will need an AWS account with SQS enabled in order to use Simple Message Queue.
|
4
|
+
|
5
|
+
Simple Message Queue can be used by a single application for background job processing (e.g. sending emails in the background), or by multiple applications to communicate with each other (e.g. sending data between applications).
|
6
|
+
|
7
|
+
### Setting up Simple Message Queue
|
8
|
+
|
9
|
+
In order to use Simple Message Queue in your application, you must first configure it. In order to configure Simple Message Queue, you must call a configure block and pass at least your AWS access_key_id and secret_access_key:
|
10
|
+
|
11
|
+
```ruby
|
12
|
+
SimpleMessageQueue.configure do |config|
|
13
|
+
config.access_key_id = 'your_access_key_id'
|
14
|
+
config.secret_access_key= 'your_secret_access_key'
|
15
|
+
end
|
16
|
+
```
|
17
|
+
|
18
|
+
In a rails application this is best done in an initializer. Create a file in config/initializers called simple_message_queue.rb and add the configure block.
|
19
|
+
|
20
|
+
You can also pass in a few other variables to the configure block such as an idle_timeout and wait_time_seconds (both used only when receiving messages) as well as a logger to replace the default logger (STDOUT).
|
21
|
+
|
22
|
+
```ruby
|
23
|
+
SimpleMessageQueue.configure do |config|
|
24
|
+
config.access_key_id = 'your_access_key_id'
|
25
|
+
config.secret_access_key= 'your_secret_access_key'
|
26
|
+
config.idle_timeout = 10 # optional
|
27
|
+
config.wait_time_seconds = 20 # optional
|
28
|
+
config.logger = Logger.new('simple_message_queue.log') # optional
|
29
|
+
config.delay_between_polls = 60*30 # optional
|
30
|
+
end
|
31
|
+
```
|
32
|
+
|
33
|
+
### Using Simple Message Queue
|
34
|
+
|
35
|
+
To use Simple Message Queue, you will need to start by creating a model for your queue such as TestQueue. This model can be created anywhere, although you will need to make sure that it is in a location that is loaded by your application. For rails applications it is best practice to place your queues in app/queues.
|
36
|
+
|
37
|
+
Next you will need to extend SimpleMessageQueue in your model:
|
38
|
+
|
39
|
+
```ruby
|
40
|
+
class TestQueue
|
41
|
+
extend SimpleMessageQueue
|
42
|
+
end
|
43
|
+
```
|
44
|
+
|
45
|
+
#### Queue Naming
|
46
|
+
|
47
|
+
By default, your SQS queue will be named after the model you created (e.g. TestQueue will have a queue named test_queue). You can overwrite this in your model by adding a queue_name method to the class:
|
48
|
+
|
49
|
+
```ruby
|
50
|
+
class TestQueue
|
51
|
+
extend SimpleMessageQueue
|
52
|
+
|
53
|
+
def self.queue_name
|
54
|
+
'my_new_queue_name'
|
55
|
+
end
|
56
|
+
end
|
57
|
+
```
|
58
|
+
|
59
|
+
#### Sending Messages
|
60
|
+
|
61
|
+
In order to send a message with Simple Message Queue, simply call the following:
|
62
|
+
|
63
|
+
```ruby
|
64
|
+
TestQueue.send('my_string')
|
65
|
+
```
|
66
|
+
|
67
|
+
You can send anything you would like to the queue, but it must be a string. You can send a simple message, or you can send the json representation of an object. What you send is completely up to you.
|
68
|
+
|
69
|
+
#### Receiving Messages
|
70
|
+
|
71
|
+
In order to receive messages, you will need to define a process_message method for your new queue model. Since every queue will process messages differently, you will need to tell your queue how to process these messages.
|
72
|
+
|
73
|
+
```ruby
|
74
|
+
class TestQueue
|
75
|
+
extend SimpleMessageQueue
|
76
|
+
|
77
|
+
def self.process_message(message)
|
78
|
+
logger.info message.body
|
79
|
+
end
|
80
|
+
end
|
81
|
+
```
|
82
|
+
|
83
|
+
Once you have defined the process_message method, you can call TestQueue.receive to receive messages once. This will poll your queue with the idle_timeout and wait_time_seconds (either the defaults or what was defined in the configure block).
|
84
|
+
|
85
|
+
If you want to continuously receive messages (as opposed to once), you will need to set up something that will keep calling TestQueue.receive. You could create a rake task which is then called by a cron job, or you could set up a daemon.
|
86
|
+
|
87
|
+
##### Using daemons-rails gem to continuously receive messages
|
88
|
+
|
89
|
+
One of the easiest ways to continuously receive messages is to set up a daemon that will keep calling TestQueue.receive. If you are using Simple Message Queue with rails, you can use the 'daemons-rails' gem (https://github.com/mirasrael/daemons-rails). Place this in your Gemfile and bundle. Next, generate a daemon with 'rails g daemon [name]'. 2 files will be created in lib/daemons, [name]_daemon_ctl (a setup file for your daemon) and [name]_daemon.rb (which is where we will place the code to receive messages). In [name]_daemon.rb, within the while($running) block, we want to call TestQueue.receive and then have it sleep. The length we have it sleep will be the time between polling for messages.
|
90
|
+
|
91
|
+
test_queue_daemon.rb
|
92
|
+
```ruby
|
93
|
+
#!/usr/bin/env ruby
|
94
|
+
|
95
|
+
# You might want to change this
|
96
|
+
ENV["RAILS_ENV"] ||= "production"
|
97
|
+
|
98
|
+
root = File.expand_path(File.dirname(__FILE__))
|
99
|
+
root = File.dirname(root) until File.exists?(File.join(root, 'config'))
|
100
|
+
Dir.chdir(root)
|
101
|
+
|
102
|
+
require File.join(root, "config", "environment")
|
103
|
+
|
104
|
+
$running = true
|
105
|
+
Signal.trap("TERM") do
|
106
|
+
$running = false
|
107
|
+
end
|
108
|
+
|
109
|
+
while($running) do
|
110
|
+
|
111
|
+
TestQueue.run
|
112
|
+
sleep 60*5
|
113
|
+
|
114
|
+
end
|
115
|
+
```
|
116
|
+
|
117
|
+
This is the default [name]_daemon.rb file generated, with out code placed in the while($running) block. You can now access all of the rake tasks associated with the daemon (refer to the daemon-rails github page for these rake tasks).
|
118
|
+
|
119
|
+
**NOTE:** You will want to monitor these daemons and have something to restart them if they stop. Something like God (http://godrb.com/) or Monit (http://mmonit.com/monit/) will work nicely.
|
120
|
+
|
121
|
+
### Single Site Communication
|
122
|
+
|
123
|
+
If you are using Simple Message Queue for background processing on a single site, all you need to do is create your model and extend SimpleMessageQueue. You will then be able to call the send and receive methods on that model.
|
124
|
+
|
125
|
+
### Multiple Site Communication
|
126
|
+
|
127
|
+
If you are using Simple Message Queue for background processing between sites you will need to take a few additional steps. You will need to create a model in each application that extends SimpleMessageQueue. Next, both of these models will need to have the same queue_name. You can do this with careful naming, or define the queue_name method as described above and give them the same name (recommended).
|
128
|
+
|
129
|
+
**Note:** If you have multiple models receiving messages from the same queue, there is no guarantee which model will receive which message. This is why it is best to only have a single model (on a single site) receiving messages from a specific queue. If you need multiple queues, simply create multiple models with different queue_names.
|
130
|
+
|
131
|
+
## Contributing
|
132
|
+
|
133
|
+
In order to contribute to this gem, please submit a pull request with passing tests.
|
134
|
+
In order to test locally, you will need to rename lib/config.yml.sample to lib/config.yml and add in your AWS access_key_id and secret_access_key.
|
data/lib/config.yml
ADDED
@@ -0,0 +1,73 @@
|
|
1
|
+
require 'simple_message_queue/configuration'
|
2
|
+
require 'simple_message_queue/core_ext/string'
|
3
|
+
require 'simple_message_queue/errors'
|
4
|
+
require 'aws'
|
5
|
+
|
6
|
+
module SimpleMessageQueue
|
7
|
+
class << self
|
8
|
+
# Class Methods only available to SimpleMessageQueue
|
9
|
+
attr_accessor :configuration
|
10
|
+
|
11
|
+
def configure
|
12
|
+
self.configuration ||= Configuration.new
|
13
|
+
yield(configuration)
|
14
|
+
end
|
15
|
+
|
16
|
+
end
|
17
|
+
|
18
|
+
def sqs
|
19
|
+
raise SimpleMessageQueue::ConfigurationError unless SimpleMessageQueue.configuration
|
20
|
+
@@sqs ||= AWS::SQS.new(:access_key_id => SimpleMessageQueue.configuration.access_key_id, :secret_access_key => SimpleMessageQueue.configuration.secret_access_key)
|
21
|
+
end
|
22
|
+
|
23
|
+
def queue_name
|
24
|
+
name.underscore.gsub('/', '_')
|
25
|
+
end
|
26
|
+
|
27
|
+
def queue
|
28
|
+
@queue ||= sqs.queues.create(queue_name)
|
29
|
+
end
|
30
|
+
|
31
|
+
def count
|
32
|
+
queue.approximate_number_of_messages
|
33
|
+
end
|
34
|
+
|
35
|
+
def delete_queue
|
36
|
+
queue.delete
|
37
|
+
end
|
38
|
+
|
39
|
+
def exists?
|
40
|
+
# Although there is a queue.exists? method, that is only relevant if you already have the queue stored in a variable and then delete it
|
41
|
+
# Trying to look it up by name will either return the queue object or throw an error, hence the rescue
|
42
|
+
true if sqs.queues.named(queue_name)
|
43
|
+
rescue
|
44
|
+
false
|
45
|
+
end
|
46
|
+
|
47
|
+
def send(message)
|
48
|
+
queue.send_message(message)
|
49
|
+
end
|
50
|
+
|
51
|
+
def logger
|
52
|
+
if SimpleMessageQueue.configuration.logger
|
53
|
+
@logger ||= SimpleMessageQueue.configuration.logger
|
54
|
+
else
|
55
|
+
@logger ||= Logger.new(STDOUT)
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
def receive
|
60
|
+
@count = 0
|
61
|
+
logger.info "Receiving messages for #{queue_name} at #{DateTime.now}"
|
62
|
+
queue.poll(:idle_timeout => SimpleMessageQueue.configuration.idle_timeout, :wait_time_seconds => SimpleMessageQueue.configuration.wait_time_seconds) do |message|
|
63
|
+
@count += 1
|
64
|
+
process_message(message)
|
65
|
+
end
|
66
|
+
@count
|
67
|
+
end
|
68
|
+
|
69
|
+
def process_message(message)
|
70
|
+
raise SimpleMessageQueue::NotImplementedError.new(name)
|
71
|
+
end
|
72
|
+
|
73
|
+
end
|
@@ -0,0 +1,13 @@
|
|
1
|
+
module SimpleMessageQueue
|
2
|
+
class Configuration
|
3
|
+
attr_accessor :access_key_id, :secret_access_key, :logger, :idle_timeout, :wait_time_seconds
|
4
|
+
|
5
|
+
def initialize
|
6
|
+
@access_key_id = nil
|
7
|
+
@secret_access_key = nil
|
8
|
+
@logger = nil
|
9
|
+
@idle_timeout = 10
|
10
|
+
@wait_time_seconds = 20
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
@@ -0,0 +1,17 @@
|
|
1
|
+
module SimpleMessageQueue
|
2
|
+
|
3
|
+
class ConfigurationError < StandardError
|
4
|
+
def initialize
|
5
|
+
message = "SimpleMessageQueue has not been configured. Create an initializer with a SimpleMessageQueue.configure block and set the access_key_id and secret_access_key."
|
6
|
+
super(message)
|
7
|
+
end
|
8
|
+
end
|
9
|
+
|
10
|
+
class NotImplementedError < StandardError
|
11
|
+
def initialize(name)
|
12
|
+
message = "You must define the process_message method for #{name}"
|
13
|
+
super(message)
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
end
|
metadata
ADDED
@@ -0,0 +1,75 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: simple_message_queue
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.0.1
|
5
|
+
prerelease:
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Jim Smith
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2013-08-01 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: aws-sdk
|
16
|
+
requirement: &70108391216460 !ruby/object:Gem::Requirement
|
17
|
+
none: false
|
18
|
+
requirements:
|
19
|
+
- - ! '>='
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: '0'
|
22
|
+
type: :runtime
|
23
|
+
prerelease: false
|
24
|
+
version_requirements: *70108391216460
|
25
|
+
- !ruby/object:Gem::Dependency
|
26
|
+
name: minitest
|
27
|
+
requirement: &70108391216040 !ruby/object:Gem::Requirement
|
28
|
+
none: false
|
29
|
+
requirements:
|
30
|
+
- - ! '>='
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: '0'
|
33
|
+
type: :runtime
|
34
|
+
prerelease: false
|
35
|
+
version_requirements: *70108391216040
|
36
|
+
description: SimpleMessageQueue is a simple interface for Amazon Web Service's SQS.
|
37
|
+
email:
|
38
|
+
- jim@jimsmithdesign.com
|
39
|
+
executables: []
|
40
|
+
extensions: []
|
41
|
+
extra_rdoc_files: []
|
42
|
+
files:
|
43
|
+
- lib/config.yml
|
44
|
+
- lib/config.yml.sample
|
45
|
+
- lib/simple_message_queue/configuration.rb
|
46
|
+
- lib/simple_message_queue/core_ext/string.rb
|
47
|
+
- lib/simple_message_queue/errors.rb
|
48
|
+
- lib/simple_message_queue/version.rb
|
49
|
+
- lib/simple_message_queue.rb
|
50
|
+
- README.md
|
51
|
+
homepage: https://github.com/cbi/simple_message_queue
|
52
|
+
licenses: []
|
53
|
+
post_install_message:
|
54
|
+
rdoc_options: []
|
55
|
+
require_paths:
|
56
|
+
- lib
|
57
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
58
|
+
none: false
|
59
|
+
requirements:
|
60
|
+
- - ! '>='
|
61
|
+
- !ruby/object:Gem::Version
|
62
|
+
version: '0'
|
63
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
64
|
+
none: false
|
65
|
+
requirements:
|
66
|
+
- - ! '>='
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
69
|
+
requirements: []
|
70
|
+
rubyforge_project:
|
71
|
+
rubygems_version: 1.8.15
|
72
|
+
signing_key:
|
73
|
+
specification_version: 3
|
74
|
+
summary: SimpleMessageQueue
|
75
|
+
test_files: []
|