songkick_queue 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: e8a5f871097112612e6bc9bc581afe8f7a1dcced
4
+ data.tar.gz: 983110a1e0eb96ea33c88584ba9bdeb198365acf
5
+ SHA512:
6
+ metadata.gz: 0a78a279786c3ea6b30cb35e4b667a8dfa3d4c661bd930e2802c8ec0f2666e447ec38c00e567358cf45e960e377f8a1b5dd0d007f9bbd119db92e4443d6a6d04
7
+ data.tar.gz: 1bd3ae7c10e1c9913817aeaa06e3816c53911b1dd296549c234ac2668613e08ff298272fe844f0fbda99d720d2f3996b4ee4b85729ba923b44d77cd30d678f09
@@ -0,0 +1,15 @@
1
+ /.bundle/
2
+ /.yardoc
3
+ /Gemfile.lock
4
+ /_yardoc/
5
+ /coverage/
6
+ /doc/
7
+ /pkg/
8
+ /spec/reports/
9
+ /tmp/
10
+ *.bundle
11
+ *.so
12
+ *.o
13
+ *.a
14
+ mkmf.log
15
+ vendor/bundle
data/.rspec ADDED
@@ -0,0 +1,3 @@
1
+ --color
2
+ --require spec_helper
3
+ --format documentation
@@ -0,0 +1,6 @@
1
+ language: ruby
2
+ script: bundle exec rspec
3
+ rvm:
4
+ - 2.2.0
5
+ - 2.1.0
6
+ - 2.0.0
@@ -0,0 +1 @@
1
+ -M redcarpet bin/* lib/**/*.rb - README.md
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in songkick_queue.gemspec
4
+ gemspec
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2015 Songkick.com Ltd
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,154 @@
1
+ # SongkickQueue
2
+
3
+ A gem for processing tasks asynchronously, powered by RabbitMQ.
4
+
5
+ [![Build status](https://travis-ci.org/songkick/queue.svg?branch=master)](https://travis-ci.org/songkick/queue)
6
+
7
+ ## Dependencies
8
+
9
+ * Ruby 2.0+
10
+ * RabbitMQ ~v2.8
11
+
12
+ ## Installation
13
+
14
+ Add this line to your application's Gemfile:
15
+
16
+ ```ruby
17
+ gem 'songkick_queue'
18
+ ```
19
+
20
+ And then execute:
21
+
22
+ $ bundle
23
+
24
+ Or install it yourself as:
25
+
26
+ $ gem install songkick_queue
27
+
28
+ ## Usage
29
+
30
+ ### Setup
31
+
32
+ You must define both the AMQP URL and a logger instance:
33
+
34
+ ```ruby
35
+ SongkickQueue.configure do |config|
36
+ config.amqp = 'amqp://localhost:5672'
37
+ config.logger = Logger.new(STDOUT)
38
+ end
39
+ ```
40
+
41
+ ### Creating consumers
42
+
43
+ To create a consumer simply construct a new class and include the `SongkickQueue::Consumer`
44
+ module.
45
+
46
+ Consumers must declare a queue name to consume from (by calling `consume_from_queue`) and
47
+ and define a `#process` method which receives a message.
48
+
49
+ For example:
50
+
51
+ ```ruby
52
+ class TweetConsumer
53
+ include SongkickQueue::Consumer
54
+
55
+ consume_from_queue 'notifications-service.tweets'
56
+
57
+ def process(message)
58
+ logger.info "Received message: #{message.inspect}"
59
+
60
+ TwitterClient.send_tweet(message[:text], message[:user_id])
61
+ rescue TwitterClient::HttpError => e
62
+ logger.warn(e)
63
+ rescue StandardError => e
64
+ logger.error(e)
65
+ end
66
+ end
67
+ ```
68
+
69
+ Consumers have the logger you declared in the configuration available to them.
70
+
71
+ ### Running consumers
72
+
73
+ Run the built in binary:
74
+
75
+ ```sh
76
+ $ songkick_queue --help
77
+ Usage: songkick_queue [options]
78
+ -r, --require LIBRARY Path to require LIBRARY. Usually this will be a file that
79
+ requires some consumers
80
+ -c, --consumer CLASS_NAME Register consumer with CLASS_NAME
81
+ -n, --name NAME Set the process name to NAME
82
+ -h, --help Show this message
83
+ ```
84
+
85
+ Both the `--require` and `--consumer` arguments can be passed multiple times, enabling you to run
86
+ multiple consumers in one process.
87
+
88
+ Example usage:
89
+
90
+ ```sh
91
+ $ songkick_queue -r ./lib/environment.rb -c TweetConsumer -n notifications_worker
92
+ ```
93
+
94
+ ```sh
95
+ $ ps aux | grep 'notifications_worker'
96
+ 22320 0.0 0.3 2486900 25368 s001 S+ 4:59pm 0:00.84 notifications_worker[idle]
97
+ ```
98
+
99
+ NB. The `songkick_queue` process does not daemonize. We recommend running it using something like
100
+ [supervisor](http://supervisord.org/) or [god](http://godrb.com/).
101
+
102
+ ### Publishing messages
103
+
104
+ To publish messages for consumers, call the `#publish` method on `SongkickQueue`, passing in the
105
+ name of the queue to publish to and the message to send.
106
+
107
+ The queue name must match one declared in a consumer by calling `consume_from_queue`.
108
+
109
+ The message can be any primitive Ruby object that can be serialized into JSON. Messages are
110
+ serialized whilst enqueued and deserialized for being passed to the `#process` method in your
111
+ consumer.
112
+
113
+ ```ruby
114
+ SongkickQueue.publish('notifications-service.tweets', { text: 'Hello world', user_id: 57237722 })
115
+ ```
116
+
117
+ ## Tests
118
+
119
+ See the current build status on Travis CI: https://travis-ci.org/songkick/queue
120
+
121
+ The tests are written in RSpec. Run them by calling:
122
+
123
+ ```sh
124
+ $ rspec
125
+ ```
126
+
127
+ ## Documentation
128
+
129
+ Up to date docs are available on RubyDoc: http://www.rubydoc.info/github/songkick/queue
130
+
131
+ The documentation is written inline in the source code and processed using YARD. To generate and
132
+ view the documentation locally, run:
133
+
134
+ ```sh
135
+ $ yardoc
136
+ $ yard server --reload
137
+
138
+ $ open http://localhost:8808/
139
+ ```
140
+
141
+ ## TODO
142
+
143
+ * Add a message UUID when publishing (add to process name when processing)
144
+ * Look at adding #requeue and #reject methods in consumer mixin
145
+
146
+ ## Contributing
147
+
148
+ Pull requests are welcome!
149
+
150
+ 1. Fork it ( https://github.com/songkick/queue/fork )
151
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
152
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
153
+ 4. Push to the branch (`git push origin my-new-feature`)
154
+ 5. Create a new Pull Request
@@ -0,0 +1 @@
1
+ require "bundler/gem_tasks"
@@ -0,0 +1,5 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require_relative '../lib/songkick_queue'
4
+
5
+ SongkickQueue::CLI.new(ARGV).run
@@ -0,0 +1,28 @@
1
+ # Example environment file
2
+ # Require this file when running `songkick_queue` like so:
3
+ #
4
+ # $ bin/songkick_queue --require ./examples/environment.rb --consumer TweetConsumer
5
+ #
6
+ require_relative '../lib/songkick_queue'
7
+
8
+ SongkickQueue.configure do |config|
9
+ config.amqp = 'amqp://localhost:5672'
10
+ config.logger = Logger.new(STDOUT)
11
+ end
12
+
13
+ class TweetConsumer
14
+ include SongkickQueue::Consumer
15
+
16
+ consume_from_queue 'notifications-service.tweets'
17
+
18
+ def process(payload)
19
+ puts "TweetConsumer#process(#{payload})"
20
+
21
+ 10.times do
22
+ sleep 1
23
+ puts "Processing..."
24
+ end
25
+
26
+ puts "Done processing!"
27
+ end
28
+ end
@@ -0,0 +1,10 @@
1
+ require_relative '../lib/songkick_queue'
2
+
3
+ SongkickQueue.configure do |config|
4
+ config.amqp = 'amqp://localhost:5672'
5
+ config.logger = Logger.new(STDOUT)
6
+ end
7
+
8
+ 3.times do
9
+ SongkickQueue.publish('notifications-service.tweets', { text: 'Hello world', user_id: 57237722 })
10
+ end
@@ -0,0 +1,42 @@
1
+ require 'json'
2
+ require 'bunny'
3
+
4
+ require 'songkick_queue/version'
5
+ require 'songkick_queue/client'
6
+ require 'songkick_queue/consumer'
7
+ require 'songkick_queue/producer'
8
+ require 'songkick_queue/worker'
9
+ require 'songkick_queue/cli'
10
+
11
+ module SongkickQueue
12
+ Configuration = Struct.new(:amqp, :logger)
13
+ ConfigurationError = Class.new(StandardError)
14
+
15
+ # Retrieve configuration for SongkickQueue
16
+ #
17
+ # @return [Configuration]
18
+ def self.configuration
19
+ @configuration ||= Configuration.new
20
+ end
21
+
22
+ # Yields a block, passing the memoized configuration instance
23
+ #
24
+ # @yield [Configuration]
25
+ def self.configure
26
+ yield(configuration)
27
+ end
28
+
29
+ # Publishes the given message to the given queue
30
+ #
31
+ # @param queue_name [String] to publish to
32
+ # @param message [#to_json] to serialize and enqueue
33
+ def self.publish(queue_name, message)
34
+ producer.publish(queue_name, message)
35
+ end
36
+
37
+ private
38
+
39
+ def self.producer
40
+ @producer ||= Producer.new
41
+ end
42
+ end
@@ -0,0 +1,71 @@
1
+ require 'optparse'
2
+ require 'ostruct'
3
+
4
+ module SongkickQueue
5
+ class CLI
6
+ attr_reader :options
7
+
8
+ # @param argv [Array<String>] of command line arguments
9
+ def initialize(argv)
10
+ @options = OpenStruct.new(
11
+ libraries: [],
12
+ consumers: [],
13
+ process_name: 'songkick_queue',
14
+ )
15
+
16
+ parse_options(argv)
17
+ end
18
+
19
+ # Parse the command line arguments using OptionParser
20
+ #
21
+ # @param argv [Array<String>] of command line arguments
22
+ def parse_options(argv)
23
+ option_parser = OptionParser.new do |opts|
24
+ opts.banner = 'Usage: songkick_queue [options]'
25
+
26
+ opts.on('-r', '--require LIBRARY',
27
+ 'Path to require LIBRARY. Usually this will be a file that ',
28
+ 'requires some consumers') do |lib|
29
+ options.libraries << lib
30
+ end
31
+
32
+ opts.on('-c', '--consumer CLASS_NAME',
33
+ 'Register consumer with CLASS_NAME') do |class_name|
34
+ options.consumers << class_name
35
+ end
36
+
37
+ opts.on('-n', '--name NAME',
38
+ 'Set the process name to NAME') do |name|
39
+ options.process_name = name
40
+ end
41
+
42
+ opts.on_tail("-h", "--help", "Show this message") do
43
+ puts opts
44
+ exit
45
+ end
46
+ end
47
+
48
+ option_parser.parse!(argv)
49
+ end
50
+
51
+ # Instantiates and runs a new Worker for the parsed options. Calling this
52
+ # method blocks the main Thread. See Worker#run for more info
53
+ #
54
+ def run
55
+ options.libraries.each do |lib|
56
+ require lib
57
+ end
58
+
59
+ if options.consumers.empty?
60
+ puts 'No consumers provided, exiting. Run `songkick_queue --help` for more info.'
61
+ exit 1
62
+ end
63
+
64
+ consumers = options.consumers.map do |class_name|
65
+ Object.const_get(class_name)
66
+ end
67
+
68
+ Worker.new(options.process_name, consumers).run
69
+ end
70
+ end
71
+ end
@@ -0,0 +1,44 @@
1
+ module SongkickQueue
2
+ class Client
3
+ def default_exchange
4
+ channel.default_exchange
5
+ end
6
+
7
+ # Creates a memoized channel for issuing RabbitMQ commands
8
+ #
9
+ # @return [Bunny::Channel]
10
+ def channel
11
+ @channel ||= begin
12
+ channel = connection.create_channel
13
+ channel.prefetch(1)
14
+
15
+ channel
16
+ end
17
+ end
18
+
19
+ # Creates a memoized connection to RabbitMQ
20
+ #
21
+ # @return [Bunny::Session]
22
+ def connection
23
+ @connection ||= begin
24
+ connection = Bunny.new(config_amqp)
25
+ connection.start
26
+
27
+ connection
28
+ end
29
+ end
30
+
31
+ private
32
+
33
+ # Retrieve the AMQP URL from the configuration
34
+ #
35
+ # @raise [ConfigurationError] if not defined
36
+ def config_amqp
37
+ config.amqp || fail(ConfigurationError, 'missing AMQP URL from config')
38
+ end
39
+
40
+ def config
41
+ SongkickQueue.configuration
42
+ end
43
+ end
44
+ end
@@ -0,0 +1,44 @@
1
+ module SongkickQueue
2
+ module Consumer
3
+ attr_reader :delivery_info, :logger
4
+
5
+ module ClassMethods
6
+ # Define the name of the queue this consumer with process messages from
7
+ #
8
+ # @param queue_name [String]
9
+ def consume_from_queue(queue_name)
10
+ @queue_name = queue_name
11
+ end
12
+
13
+ # Return the quene name set by #consume_from_queue
14
+ #
15
+ # @raise [NotImplementedError] if queue name was not already defined
16
+ def queue_name
17
+ @queue_name || fail(NotImplementedError, 'you must declare a queue name to consume from ' +
18
+ 'by calling #consume_from_queue in your consumer class. See README for more info.')
19
+ end
20
+ end
21
+
22
+ def self.included(base)
23
+ base.extend(ClassMethods)
24
+ end
25
+
26
+ # @param delivery_info [Bunny::DeliveryInfo#delivery_tag] to use for
27
+ # acknowledgement and requeues, rejects etc...
28
+ # @param logger [Logger] to expose to the client consumer for logging
29
+ def initialize(delivery_info, logger)
30
+ @delivery_info = delivery_info
31
+ @logger = logger
32
+ end
33
+
34
+ # Placeholder method to ensure each client consumer defines their own
35
+ # process message
36
+ #
37
+ # @param message [Object] to process
38
+ # @raise [NotImplementedError]
39
+ def process(message)
40
+ fail NotImplementedError, 'you must define a #process method in your ' +
41
+ 'consumer class, see the README for more info.'
42
+ end
43
+ end
44
+ end
@@ -0,0 +1,24 @@
1
+ module SongkickQueue
2
+ class Producer
3
+ def initialize
4
+ @client = Client.new
5
+ end
6
+
7
+ # Serializes the given message and publishes it to the default RabbitMQ
8
+ # exchange
9
+ #
10
+ # @param queue_name [String] to publish to
11
+ # @param message [#to_json] to serialize and enqueue
12
+ def publish(queue_name, message)
13
+ payload = JSON.generate(message)
14
+
15
+ client
16
+ .default_exchange
17
+ .publish(payload, routing_key: String(queue_name))
18
+ end
19
+
20
+ private
21
+
22
+ attr_reader :client
23
+ end
24
+ end
@@ -0,0 +1,3 @@
1
+ module SongkickQueue
2
+ VERSION = "0.1.0"
3
+ end
@@ -0,0 +1,130 @@
1
+ module SongkickQueue
2
+ class Worker
3
+ attr_reader :process_name, :consumer_classes
4
+
5
+ # @param process_name [String] of the custom process name to use
6
+ # @param consumer_classes [Array<Class>, Class] of consumer class names
7
+ def initialize(process_name, consumer_classes = [])
8
+ @process_name = process_name
9
+ @consumer_classes = Array(consumer_classes)
10
+
11
+ if @consumer_classes.empty?
12
+ fail ArgumentError, 'no consumer classes given to Worker'
13
+ end
14
+
15
+ @client = Client.new
16
+ end
17
+
18
+ # Subscribes the consumers classes to their defined message queues and
19
+ # blocks until all the work pool consumers have finished. Also sets up
20
+ # signal catching for graceful exits no interrupt
21
+ def run
22
+ set_process_name
23
+
24
+ consumer_classes.each do |consumer_class|
25
+ subscribe_to_queue(consumer_class)
26
+ end
27
+
28
+ setup_signal_catching
29
+ stop_if_signal_caught
30
+
31
+ channel.work_pool.join
32
+ end
33
+
34
+ private
35
+
36
+ attr_reader :client
37
+
38
+ def setup_signal_catching
39
+ trap('INT') { @shutdown = 'INT' }
40
+ trap('TERM') { @shutdown = 'TERM' }
41
+ end
42
+
43
+ # Checks for presence of @shutdown every 1 second and if found instructs
44
+ # all the channel's work pool consumers to shutdown. Each work pool thread
45
+ # will finish its current task and then join the main thread. Once all the
46
+ # threads have joined then `channel.work_pool.join` will cease blocking and
47
+ # return, causing the process to terminate.
48
+ def stop_if_signal_caught
49
+ Thread.new do
50
+ loop do
51
+ sleep 1
52
+
53
+ if @shutdown
54
+ logger.info "Recevied SIG#{@shutdown}, shutting down consumers"
55
+
56
+ @client.channel.work_pool.shutdown
57
+ @shutdown = nil
58
+ end
59
+ end
60
+ end
61
+ end
62
+
63
+ # Declare a queue and subscribe to it
64
+ #
65
+ # @param consumer_class [Class] to subscribe to
66
+ def subscribe_to_queue(consumer_class)
67
+ queue = channel.queue(consumer_class.queue_name, durable: true,
68
+ arguments: {'x-ha-policy' => 'all'})
69
+
70
+ queue.subscribe(manual_ack: true) do |delivery_info, properties, payload|
71
+ process_message(consumer_class, delivery_info, properties, payload)
72
+ end
73
+
74
+ logger.info "Subscribed #{consumer_class} to #{consumer_class.queue_name}"
75
+ end
76
+
77
+ # Handle receipt of a subscribed message
78
+ #
79
+ # @param consumer_class [Class] that was subscribed to
80
+ # @param delivery_info [Bunny::DeliveryInfo]
81
+ # @param properties [Bunny::MessageProperties]
82
+ # @param payload [String] to deserialize
83
+ def process_message(consumer_class, delivery_info, properties, payload)
84
+ logger.info "Processing message via #{consumer_class}..."
85
+
86
+ set_process_name(consumer_class)
87
+
88
+ message = JSON.parse(payload, symbolize_names: true)
89
+
90
+ consumer = consumer_class.new(delivery_info, logger)
91
+ consumer.process(message)
92
+ rescue Object => exception
93
+ logger.error(exception)
94
+ ensure
95
+ set_process_name
96
+ channel.ack(delivery_info.delivery_tag, false)
97
+ end
98
+
99
+ def channel
100
+ client.channel
101
+ end
102
+
103
+ # Retrieve the logger defined in the configuration
104
+ #
105
+ # @raise [ConfigurationError] if not defined
106
+ def logger
107
+ config.logger || fail(ConfigurationError, 'No logger configured, see README for more details')
108
+ end
109
+
110
+ def config
111
+ SongkickQueue.configuration
112
+ end
113
+
114
+ # Update the name of this process, as viewed in `ps` or `top`
115
+ #
116
+ # @example idle
117
+ # set_process_name #=> "songkick_queue[idle]"
118
+ # @example consumer running
119
+ # set_process_name(TweetConsumer) #=> "songkick_queue[tweet_consumer]"
120
+ # @param status [String] of the program
121
+ def set_process_name(status = 'idle')
122
+ formatted_status = String(status)
123
+ .gsub('::', '')
124
+ .gsub(/([A-Z]+)/) { "_#{$1.downcase}" }
125
+ .sub(/^_(\w)/) { $1 }
126
+
127
+ $PROGRAM_NAME = "#{process_name}[#{formatted_status}]"
128
+ end
129
+ end
130
+ end
@@ -0,0 +1,29 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'songkick_queue/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = "songkick_queue"
8
+ spec.version = SongkickQueue::VERSION
9
+ spec.authors = ["Dan Lucraft", "Paul Springett"]
10
+ spec.email = ["dan.lucraft@songkick.com", "paul.springett@songkick.com"]
11
+ spec.summary = %q{A gem for processing tasks asynchronously, powered by RabbitMQ.}
12
+ spec.description = %q{A gem for processing tasks asynchronously, powered by RabbitMQ.}
13
+ spec.homepage = "https://github.com/songkick/songkick_queue"
14
+ spec.license = "MIT"
15
+
16
+ spec.files = `git ls-files -z`.split("\x0")
17
+ spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
18
+ spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
19
+ spec.require_paths = ["lib"]
20
+
21
+ spec.add_development_dependency "rake", "~> 10.0"
22
+ spec.add_development_dependency "rspec", "~> 3.2"
23
+ spec.add_development_dependency "yard"
24
+
25
+ # Used by yardoc for processing README.md code snippets
26
+ spec.add_development_dependency "redcarpet"
27
+
28
+ spec.add_dependency "bunny", "~> 1.2.1"
29
+ end
@@ -0,0 +1,69 @@
1
+ require 'songkick_queue/cli'
2
+
3
+ module SongkickQueue
4
+ RSpec.describe CLI do
5
+ describe "#initialize" do
6
+ it "should build an options object with defaults" do
7
+ cli = CLI.new([])
8
+ options = cli.options
9
+
10
+ expect(options.libraries).to eq []
11
+ expect(options.consumers).to eq []
12
+ expect(options.process_name).to eq 'songkick_queue'
13
+ end
14
+ end
15
+
16
+ describe "#parse_options" do
17
+ it "should parse required libraries" do
18
+ cli = CLI.new(%w[--require foo -r bar])
19
+ options = cli.options
20
+
21
+ expect(options.libraries).to eq ['foo', 'bar']
22
+ end
23
+
24
+ it "should parse consumers" do
25
+ cli = CLI.new(%w[--consumer FooConsumer -c BarConsumer])
26
+ options = cli.options
27
+
28
+ expect(options.consumers).to eq ['FooConsumer', 'BarConsumer']
29
+ end
30
+
31
+ it "should parse consumers" do
32
+ cli = CLI.new(%w[--name example_worker])
33
+ options = cli.options
34
+
35
+ expect(options.process_name).to eq 'example_worker'
36
+ end
37
+ end
38
+
39
+ describe "#run" do
40
+ it "should try and require given paths" do
41
+ cli = CLI.new(%w[--require path/to/app])
42
+
43
+ expect { cli.run }.to raise_error(LoadError, 'cannot load such file -- path/to/app')
44
+ end
45
+
46
+ it "should exit with useful message if no consumers given" do
47
+ cli = CLI.new([])
48
+
49
+ expect {
50
+ expect(STDOUT).to receive(:puts)
51
+ .with('No consumers provided, exiting. Run `songkick_queue --help` for more info.')
52
+
53
+ cli.run
54
+ }.to raise_error(SystemExit)
55
+ end
56
+
57
+ it "should build and run a Worker" do
58
+ ::ExampleConsumer = Class.new
59
+
60
+ worker = instance_double(Worker, run: :running)
61
+ cli = CLI.new(%w[--consumer ExampleConsumer --name example_worker])
62
+
63
+ expect(Worker).to receive(:new).with('example_worker', [ExampleConsumer]) { worker }
64
+
65
+ cli.run
66
+ end
67
+ end
68
+ end
69
+ end
@@ -0,0 +1,49 @@
1
+ require 'songkick_queue/consumer'
2
+
3
+ module SongkickQueue
4
+ RSpec.describe Consumer do
5
+ describe ".from_queue" do
6
+ it "should fail if .consume_from_queue has not been called" do
7
+ class ExampleConsumer
8
+ include SongkickQueue::Consumer
9
+ end
10
+
11
+ expect { ExampleConsumer.queue_name }.to raise_error(NotImplementedError)
12
+ end
13
+
14
+ it "should return the queue name set by .consume_from_queue" do
15
+ class ExampleConsumer
16
+ include SongkickQueue::Consumer
17
+
18
+ consume_from_queue 'app.examples'
19
+ end
20
+
21
+ expect(ExampleConsumer.queue_name).to eq 'app.examples'
22
+ end
23
+ end
24
+
25
+ describe "#initialize" do
26
+ it "should pass a logger" do
27
+ class ExampleConsumer
28
+ include SongkickQueue::Consumer
29
+ end
30
+
31
+ consumer = ExampleConsumer.new(:delivery_info, :logger)
32
+
33
+ expect(consumer.logger).to eq :logger
34
+ end
35
+ end
36
+
37
+ describe "#process" do
38
+ it "should fail if not overridden" do
39
+ class ExampleConsumer
40
+ include SongkickQueue::Consumer
41
+ end
42
+
43
+ consumer = ExampleConsumer.new(:delivery_info, :logger)
44
+
45
+ expect { consumer.process(:message) }.to raise_error(NotImplementedError)
46
+ end
47
+ end
48
+ end
49
+ end
@@ -0,0 +1,20 @@
1
+ require 'songkick_queue/producer'
2
+
3
+ module SongkickQueue
4
+ RSpec.describe Producer do
5
+ describe "#publish" do
6
+ it "should publish the message to the exchange as JSON, with the correct routing key" do
7
+ producer = Producer.new
8
+
9
+ exchange = double(:exchange, publish: :published)
10
+ client = instance_double(Client, default_exchange: exchange)
11
+ allow(producer).to receive(:client) { client }
12
+
13
+ expect(exchange).to receive(:publish)
14
+ .with('{"example":"message","value":true}', routing_key: 'queue_name')
15
+
16
+ producer.publish(:queue_name, { example: 'message', value: true })
17
+ end
18
+ end
19
+ end
20
+ end
@@ -0,0 +1,98 @@
1
+ require 'songkick_queue/worker'
2
+
3
+ module SongkickQueue
4
+ RSpec.describe Worker do
5
+ describe "#initialize" do
6
+ it "should set process_name" do
7
+ worker = Worker.new(:process_name, [:foo_consumer])
8
+
9
+ expect(worker.process_name).to eq :process_name
10
+ end
11
+
12
+ it "should set consumer_classes" do
13
+ worker = Worker.new(:process_name, [:foo_consumer, :bar_consumer])
14
+
15
+ expect(worker.consumer_classes).to eq [:foo_consumer, :bar_consumer]
16
+ end
17
+
18
+ it "should convert single consumer into array of consumers" do
19
+ worker = Worker.new(:process_name, :bar_consumer)
20
+
21
+ expect(worker.consumer_classes).to eq [:bar_consumer]
22
+ end
23
+
24
+ it "should fail if no consumer passed" do
25
+ expect { Worker.new(:process_name) }.to raise_error(ArgumentError,
26
+ 'no consumer classes given to Worker')
27
+ end
28
+ end
29
+
30
+ describe "#run" do
31
+ it "should subscribe each given consumer" do
32
+ worker = Worker.new(:process_name, [:foo_consumer, :bar_consumer])
33
+
34
+ allow(worker).to receive(:set_process_name)
35
+ allow(worker).to receive(:setup_signal_catching)
36
+ allow(worker).to receive(:stop_if_signal_caught)
37
+
38
+ allow(worker).to receive_message_chain(:channel, :work_pool, :join)
39
+
40
+ expect(worker).to receive(:subscribe_to_queue).with(:foo_consumer)
41
+ expect(worker).to receive(:subscribe_to_queue).with(:bar_consumer)
42
+
43
+ worker.run
44
+ end
45
+ end
46
+
47
+ describe "#subscribe_to_queue" do
48
+ it "should declare the queue and subscribe" do
49
+ consumer_class = double(:consumer_class, queue_name: 'app.examples')
50
+ worker = Worker.new(:process_name, consumer_class)
51
+
52
+ queue = double(:queue, subscribe: :null)
53
+ channel = double(:channel, queue: queue)
54
+
55
+ allow(worker).to receive(:channel) { channel }
56
+ allow(worker).to receive(:logger) { double(:logger, info: :null) }
57
+
58
+ expect(channel).to receive(:queue).with('app.examples', durable: true,
59
+ arguments: {'x-ha-policy' => 'all'})
60
+
61
+ expect(queue).to receive(:subscribe)
62
+
63
+ worker.send(:subscribe_to_queue, consumer_class)
64
+ end
65
+ end
66
+
67
+ describe "#process_message" do
68
+ it "should instantiate the consumer and call #process" do
69
+ ::FooConsumer = Class.new
70
+ worker = Worker.new(:process_name, FooConsumer)
71
+
72
+ logger = double(:logger, info: :null)
73
+ allow(worker).to receive(:logger) { logger }
74
+
75
+ channel = double(:channel, ack: :null)
76
+ allow(worker).to receive(:channel) { channel }
77
+
78
+ delivery_info = double(:delivery_info, delivery_tag: 'tag')
79
+
80
+ consumer = double(FooConsumer, process: :null)
81
+
82
+ expect(FooConsumer).to receive(:new)
83
+ .with(delivery_info, logger) { consumer }
84
+
85
+ expect(consumer).to receive(:process)
86
+ .with({ example: 'message', value: true})
87
+
88
+ worker.send(:process_message, FooConsumer, delivery_info,
89
+ :properties, '{"example":"message","value":true}')
90
+
91
+ expect(logger).to have_received(:info)
92
+ .with('Processing message via FooConsumer...')
93
+
94
+ expect(channel).to have_received(:ack).with('tag', false)
95
+ end
96
+ end
97
+ end
98
+ end
@@ -0,0 +1,22 @@
1
+ require 'songkick_queue'
2
+
3
+ RSpec.describe SongkickQueue do
4
+ describe "#configure" do
5
+ it "should yield instance of Configuration" do
6
+ expect { |b|
7
+ SongkickQueue.configure(&b)
8
+ }.to yield_with_args instance_of(SongkickQueue::Configuration)
9
+ end
10
+ end
11
+
12
+ describe "#publish" do
13
+ it "should call #publish on instance Producer" do
14
+ producer = instance_double(SongkickQueue::Producer)
15
+ allow(SongkickQueue).to receive(:producer) { producer }
16
+
17
+ expect(producer).to receive(:publish).with(:queue_name, :message)
18
+
19
+ SongkickQueue.publish(:queue_name, :message)
20
+ end
21
+ end
22
+ end
@@ -0,0 +1,91 @@
1
+ # This file was generated by the `rspec --init` command. Conventionally, all
2
+ # specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`.
3
+ # The generated `.rspec` file contains `--require spec_helper` which will cause
4
+ # this file to always be loaded, without a need to explicitly require it in any
5
+ # files.
6
+ #
7
+ # Given that it is always loaded, you are encouraged to keep this file as
8
+ # light-weight as possible. Requiring heavyweight dependencies from this file
9
+ # will add to the boot time of your test suite on EVERY test run, even for an
10
+ # individual file that may not need all of that loaded. Instead, consider making
11
+ # a separate helper file that requires the additional dependencies and performs
12
+ # the additional setup, and require it from the spec files that actually need
13
+ # it.
14
+ #
15
+ # The `.rspec` file also contains a few flags that are not defaults but that
16
+ # users commonly want.
17
+ #
18
+ # See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
19
+ RSpec.configure do |config|
20
+ # rspec-expectations config goes here. You can use an alternate
21
+ # assertion/expectation library such as wrong or the stdlib/minitest
22
+ # assertions if you prefer.
23
+ config.expect_with :rspec do |expectations|
24
+ # This option will default to `true` in RSpec 4. It makes the `description`
25
+ # and `failure_message` of custom matchers include text for helper methods
26
+ # defined using `chain`, e.g.:
27
+ # be_bigger_than(2).and_smaller_than(4).description
28
+ # # => "be bigger than 2 and smaller than 4"
29
+ # ...rather than:
30
+ # # => "be bigger than 2"
31
+ expectations.include_chain_clauses_in_custom_matcher_descriptions = true
32
+ end
33
+
34
+ # rspec-mocks config goes here. You can use an alternate test double
35
+ # library (such as bogus or mocha) by changing the `mock_with` option here.
36
+ config.mock_with :rspec do |mocks|
37
+ # Prevents you from mocking or stubbing a method that does not exist on
38
+ # a real object. This is generally recommended, and will default to
39
+ # `true` in RSpec 4.
40
+ mocks.verify_partial_doubles = true
41
+ end
42
+
43
+ # The settings below are suggested to provide a good initial experience
44
+ # with RSpec, but feel free to customize to your heart's content.
45
+ =begin
46
+ # These two settings work together to allow you to limit a spec run
47
+ # to individual examples or groups you care about by tagging them with
48
+ # `:focus` metadata. When nothing is tagged with `:focus`, all examples
49
+ # get run.
50
+ config.filter_run :focus
51
+ config.run_all_when_everything_filtered = true
52
+
53
+ # Limits the available syntax to the non-monkey patched syntax that is
54
+ # recommended. For more details, see:
55
+ # - http://myronmars.to/n/dev-blog/2012/06/rspecs-new-expectation-syntax
56
+ # - http://teaisaweso.me/blog/2013/05/27/rspecs-new-message-expectation-syntax/
57
+ # - http://myronmars.to/n/dev-blog/2014/05/notable-changes-in-rspec-3#new__config_option_to_disable_rspeccore_monkey_patching
58
+ config.disable_monkey_patching!
59
+
60
+ # This setting enables warnings. It's recommended, but in some cases may
61
+ # be too noisy due to issues in dependencies.
62
+ config.warnings = true
63
+
64
+ # Many RSpec users commonly either run the entire suite or an individual
65
+ # file, and it's useful to allow more verbose output when running an
66
+ # individual spec file.
67
+ if config.files_to_run.one?
68
+ # Use the documentation formatter for detailed output,
69
+ # unless a formatter has already been configured
70
+ # (e.g. via a command-line flag).
71
+ config.default_formatter = 'doc'
72
+ end
73
+
74
+ # Print the 10 slowest examples and example groups at the
75
+ # end of the spec run, to help surface which specs are running
76
+ # particularly slow.
77
+ config.profile_examples = 10
78
+
79
+ # Run specs in random order to surface order dependencies. If you find an
80
+ # order dependency and want to debug it, you can fix the order by providing
81
+ # the seed, which is printed after each run.
82
+ # --seed 1234
83
+ config.order = :random
84
+
85
+ # Seed global randomization in this process using the `--seed` CLI option.
86
+ # Setting this allows you to use `--seed` to deterministically reproduce
87
+ # test failures related to randomization by passing the same `--seed` value
88
+ # as the one that triggered the failure.
89
+ Kernel.srand config.seed
90
+ =end
91
+ end
metadata ADDED
@@ -0,0 +1,149 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: songkick_queue
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Dan Lucraft
8
+ - Paul Springett
9
+ autorequire:
10
+ bindir: bin
11
+ cert_chain: []
12
+ date: 2015-03-25 00:00:00.000000000 Z
13
+ dependencies:
14
+ - !ruby/object:Gem::Dependency
15
+ name: rake
16
+ requirement: !ruby/object:Gem::Requirement
17
+ requirements:
18
+ - - ~>
19
+ - !ruby/object:Gem::Version
20
+ version: '10.0'
21
+ type: :development
22
+ prerelease: false
23
+ version_requirements: !ruby/object:Gem::Requirement
24
+ requirements:
25
+ - - ~>
26
+ - !ruby/object:Gem::Version
27
+ version: '10.0'
28
+ - !ruby/object:Gem::Dependency
29
+ name: rspec
30
+ requirement: !ruby/object:Gem::Requirement
31
+ requirements:
32
+ - - ~>
33
+ - !ruby/object:Gem::Version
34
+ version: '3.2'
35
+ type: :development
36
+ prerelease: false
37
+ version_requirements: !ruby/object:Gem::Requirement
38
+ requirements:
39
+ - - ~>
40
+ - !ruby/object:Gem::Version
41
+ version: '3.2'
42
+ - !ruby/object:Gem::Dependency
43
+ name: yard
44
+ requirement: !ruby/object:Gem::Requirement
45
+ requirements:
46
+ - - '>='
47
+ - !ruby/object:Gem::Version
48
+ version: '0'
49
+ type: :development
50
+ prerelease: false
51
+ version_requirements: !ruby/object:Gem::Requirement
52
+ requirements:
53
+ - - '>='
54
+ - !ruby/object:Gem::Version
55
+ version: '0'
56
+ - !ruby/object:Gem::Dependency
57
+ name: redcarpet
58
+ requirement: !ruby/object:Gem::Requirement
59
+ requirements:
60
+ - - '>='
61
+ - !ruby/object:Gem::Version
62
+ version: '0'
63
+ type: :development
64
+ prerelease: false
65
+ version_requirements: !ruby/object:Gem::Requirement
66
+ requirements:
67
+ - - '>='
68
+ - !ruby/object:Gem::Version
69
+ version: '0'
70
+ - !ruby/object:Gem::Dependency
71
+ name: bunny
72
+ requirement: !ruby/object:Gem::Requirement
73
+ requirements:
74
+ - - ~>
75
+ - !ruby/object:Gem::Version
76
+ version: 1.2.1
77
+ type: :runtime
78
+ prerelease: false
79
+ version_requirements: !ruby/object:Gem::Requirement
80
+ requirements:
81
+ - - ~>
82
+ - !ruby/object:Gem::Version
83
+ version: 1.2.1
84
+ description: A gem for processing tasks asynchronously, powered by RabbitMQ.
85
+ email:
86
+ - dan.lucraft@songkick.com
87
+ - paul.springett@songkick.com
88
+ executables:
89
+ - songkick_queue
90
+ extensions: []
91
+ extra_rdoc_files: []
92
+ files:
93
+ - .gitignore
94
+ - .rspec
95
+ - .travis.yml
96
+ - .yardopts
97
+ - Gemfile
98
+ - LICENSE.txt
99
+ - README.md
100
+ - Rakefile
101
+ - bin/songkick_queue
102
+ - examples/environment.rb
103
+ - examples/producer.rb
104
+ - lib/songkick_queue.rb
105
+ - lib/songkick_queue/cli.rb
106
+ - lib/songkick_queue/client.rb
107
+ - lib/songkick_queue/consumer.rb
108
+ - lib/songkick_queue/producer.rb
109
+ - lib/songkick_queue/version.rb
110
+ - lib/songkick_queue/worker.rb
111
+ - songkick_queue.gemspec
112
+ - spec/songkick_queue/cli_spec.rb
113
+ - spec/songkick_queue/consumer_spec.rb
114
+ - spec/songkick_queue/producer_spec.rb
115
+ - spec/songkick_queue/worker_spec.rb
116
+ - spec/songkick_queue_spec.rb
117
+ - spec/spec_helper.rb
118
+ homepage: https://github.com/songkick/songkick_queue
119
+ licenses:
120
+ - MIT
121
+ metadata: {}
122
+ post_install_message:
123
+ rdoc_options: []
124
+ require_paths:
125
+ - lib
126
+ required_ruby_version: !ruby/object:Gem::Requirement
127
+ requirements:
128
+ - - '>='
129
+ - !ruby/object:Gem::Version
130
+ version: '0'
131
+ required_rubygems_version: !ruby/object:Gem::Requirement
132
+ requirements:
133
+ - - '>='
134
+ - !ruby/object:Gem::Version
135
+ version: '0'
136
+ requirements: []
137
+ rubyforge_project:
138
+ rubygems_version: 2.4.6
139
+ signing_key:
140
+ specification_version: 4
141
+ summary: A gem for processing tasks asynchronously, powered by RabbitMQ.
142
+ test_files:
143
+ - spec/songkick_queue/cli_spec.rb
144
+ - spec/songkick_queue/consumer_spec.rb
145
+ - spec/songkick_queue/producer_spec.rb
146
+ - spec/songkick_queue/worker_spec.rb
147
+ - spec/songkick_queue_spec.rb
148
+ - spec/spec_helper.rb
149
+ has_rdoc: