leveret 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 78c17ba2d29034d05262a5c3454ddf1e423b6cdd
+  data.tar.gz: a5346c1ba41a7647f997b2599730c682bc0e885a
+SHA512:
+  metadata.gz: 95b4f65d3fac2a06b6a2c06def7bd9408d35d1bf31aead3828030f8d001f5792fbe6c8c2c63e2d1116c3e983e323ec3a1e83bcc0677253805657696cfb9a9872
+  data.tar.gz: 6c46a540248cc82b981ec5e5ff869fa578e5cbefc6b27ae1c7daed891e6178c62031cd2fc374cfc3dc14901886fbcc8ab0d56f9222c9d8f02e933ba6e2303c5e

data/.rspec

@@ -0,0 +1,2 @@
+--format documentation
+--color

data/.travis.yml

@@ -0,0 +1,4 @@
+language: ruby
+rvm:
+  - 2.2.3
+before_install: gem install bundler -v 1.10.6

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in leveret.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2016 Dan Wentworth
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,204 @@
+# Leveret
+
+Leveret is an easy to use RabbitMQ backed job runner.
+
+It's designed specifically to execute long running jobs (multiple hours) while allowing the applicatioin to be
+restarted with no adverse effects on the currently running jobs.
+
+Leveret has been tested with Ruby 2.2.3+ and RabbitMQ 3.5.0+.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'leveret'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install leveret
+
+To use Leveret you need a running RabbitMQ installation. If you don't have it installed yet, you can do so via
+[Homebrew](https://www.rabbitmq.com/install-homebrew.html) (MacOS), [APT](https://www.rabbitmq.com/install-debian.html)
+(Debian/Ubuntu) or [RPM](https://www.rabbitmq.com/install-rpm.html) (Redhat/Fedora).
+
+RabbitMQ version **3.5.0** or higher is recommended as this is the first version to support message priorities.
+
+## Usage
+
+To create a job include `Leveret::Job` in a new class, and define a `perform` method that will do the work in
+your job. Call `enqueue` on your new class with any parameters you want passed to the job at execution time.
+
+```ruby
+class MyJob
+  include Leveret::Job
+
+  def perform
+    File.open('/tmp/leveret-test-file.txt', 'a+') do |f|
+      f.puts params[:test_text]
+    end
+
+    sleep 5 # Job takes a long time
+  end
+end
+
+MyJob.enqueue(test_text: "Hi there! Please write me to the test file.")
+```
+
+Now start a worker to execute the job:
+
+```bash
+bundle exec leveret_worker
+```
+
+### Queues
+
+By default all are defined on a single standard queue (see Configuration for details). However, it's possible to use
+multiple queues for different jobs. To do this set the `queue_name` in your job class. You'll also need to tell the
+worker about your new queue when starting that.
+
+```ruby
+class MyOtherQueueJob
+  include Leveret::Job
+
+  queue_name 'other'
+
+  def perform
+    # ...
+  end
+end
+
+MyOtherQueueJob.enqueue(test_text: "Hi there! Please write me to the test file.")
+```
+
+If you don't always want to place the job on your other queue, you can specify the queue name when enqueuing it. Pass
+the `queue_name` option when enqueuing the job.
+
+```ruby
+MyJob.enqueue(test_text: "Hi there! Please write me to the test file.", queue_name: 'other')
+```
+
+### Priorities
+
+Leveret supports 3 levels of job priority, `:low`, `:normal` and `:high`. To set the priority you can define it in your
+job class, or specify it at enqueue time by passing the `priority` option.
+
+```ruby
+class MyHighPriorityMyJob
+  include Leveret::Job
+
+  priority :high
+
+  def perform
+    # very important work...
+  end
+end
+
+MyHighPriorityJob.enqueue
+```
+
+To specify priority at enqueue time:
+
+```ruby
+MyJob.enqueue(test_text: "Hi there! Please write me to the test file.", priority: :high)
+```
+
+### Workers
+
+To start a leveret worker, simply run the `leveret_worker` executable included in the gem. Started with no arguments it
+will create a worker monitoring the default queue and process one job at a time.
+
+Changing the queues that a worker monitors requires passing a comma separated list of queue names in the environment
+variable `QUEUES`. The example below watches for jobs on the queues `standard` and `other`.
+
+```bash
+bundle exec leveret_worker QUEUES=standard,other
+```
+
+By default, workers will only process one job at a time. For each job that is executed, a child process is forked, and
+the job run in the new process. When the job completes, the fork exits. We can process more jobs simultaniously simply
+by allowing more forks to run. To increase this limit set the `PROCESSES` environment variable. There is no limit to
+this variable in Leveret, but you should be aware of your own OS and resource limits.
+
+```bash
+bundle exec leveret_worker PROCESSES=5
+```
+
+## Configuration
+
+Configuration in Leveret is done via a configure block. In a Rails application it is recommended you place your
+configuration in `config/initializers/leveret.rb`. Leveret comes configured with sane defaults for development, but you
+may wish to change some for production use.
+
+```ruby
+Leveret.configure do |config|
+  # Location of your RabbitMQ server
+  config.amqp = "amqp://guest:guest@localhost:5672/"
+  # Name of the exchange Levert will create on RabbitMQ
+  config.exchange_name = 'leveret_exch'
+  # Path to send log output to
+  config.log_file = STDOUT
+  # Verbosity of log output
+  config.log_level = Logger::DEBUG
+  # String that is prepended to all queues created in RabbitMQ
+  config.queue_name_prefix = 'leveret_queue'
+  # Name of the queue to use if none other is specified
+  config.default_queue_name = 'standard'
+  # A block that should be called every time a child fork is created to process a job
+  config.after_fork = proc {}
+  # A block that is called whenever an exception occurs in a job
+  config.error_handler = proc { |ex| ex }
+  # The default number of jobs to process simultaniously, this can be overridden by the PROCESSES
+  # environment variable when starting a worker
+  config.concurrent_fork_count = 1
+end
+```
+
+Most of these are pretty self-explanatory and can be left to their default values, however `after_fork` and
+`error_handler` could use a little more explaining.
+
+`after_fork` Is called immediately after a child process is forked to run a job. Any connections that need to be
+reinitialized on fork should be done so here. For example, if you're using Rails you'll probably want to reconnect
+to ActiveRecord here:
+
+```ruby
+Leveret.configure do |config|
+  config.after_fork = proc do
+    ActiveRecord::Base.establish_connection
+  end
+end
+```
+
+`error_handler` is called whenever an exception is raised in your job. These exceptions are caught and logged, but not
+raised afterwards. `error_handler` is your chance to decide what to do with these exceptions. You may wish to log them
+using a service such as [Airbrake](https://airbrake.io/) or [Sentry](https://getsentry.com/welcome/). To configure an
+error handler to log to Sentry the following would be necessary:
+
+```ruby
+Leveret.configure do |config|
+  config.error_handler = proc do |exception|
+    Raven.capture_exception(exception, tags: {component: 'leveret'})
+  end
+end
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/darkphnx/leveret.
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
+

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "leveret"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start

data/bin/setup

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -euo pipefail
+IFS=$'\n\t'
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/exe/leveret_worker

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "leveret"
+
+if File.exist?("./config/environment.rb")
+  require File.expand_path("./config/environment.rb")
+end
+
+queues = ENV["QUEUES"].to_s.split(',')
+queues << Leveret.configuration.default_queue_name if queues.empty?
+
+concurrent_fork_count = ENV["PROCESSES"] || Leveret.configuration.concurrent_fork_count
+
+worker = Leveret::Worker.new(queues: queues, concurrent_fork_count: concurrent_fork_count)
+worker.do_work

data/leveret.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'leveret/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "leveret"
+  spec.version       = Leveret::VERSION
+  spec.authors       = ["Dan Wentworth"]
+  spec.email         = ["dan@atechmedia.com"]
+
+  spec.summary       = "Simple RabbitMQ backed backround worker"
+  spec.homepage      = "https://github.com/darkphnx/leveret"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "bunny", '~> 2.3'
+  spec.add_dependency "json", '~> 1.8'
+  spec.add_development_dependency "bundler", "~> 1.10"
+  spec.add_development_dependency "rake", "~> 10.0"
+  spec.add_development_dependency "rspec"
+end

data/lib/leveret/configuration.rb

@@ -0,0 +1,48 @@
+module Leveret
+  # Contains everything needed to configure leveret for work. Sensible defaults are included
+  # and will be initialized with the class.
+  #
+  # @!attribute amqp
+  #   @return [String] Location of your RabbitMQ server. Default: +"amqp://guest:guest@localhost:5672/"+
+  # @!attribute exchange_name
+  #   @return [String] Name of the exchange for Leveret to publish messages to. Default: +"leveret_exch"+
+  # @!attribute queue_name_prefix
+  #   @return [String] This value will be prefixed to all queues created on your RabbitMQ instance.
+  #     Default: +"leveret_queue"+
+  # @!attribute log_file
+  #   @return [String] The path where logfiles should be written to. Default: +STDOUT+
+  # @!attribute log_level
+  #   @return [Integer] The log severity which should be output to the log. Default: +Logger::DEBUG+
+  # @!attribute default_queue_name
+  #   @return [String] The name of the queue that will be use unless explicitly specified in a job. Default:
+  #     +"standard"+
+  # @!attribute after_fork
+  #   @return [Proc] A proc which will be executed in a child after forking to process a message. Default: +proc {}+
+  # @!attribute error_handler
+  #   @return [Proc] A proc which will be called if a job raises an exception. Default: +proc {|ex| ex }+
+  # @!attribute concurrent_fork_count
+  #   @return [Integer] The number of jobs that can be processes simultanously. Default: +1+
+  class Configuration
+    attr_accessor :amqp, :exchange_name, :queue_name_prefix, :log_file, :log_level, :default_queue_name, :after_fork,
+      :error_handler, :concurrent_fork_count
+
+    # Create a new instance of Configuration with a set of sane defaults.
+    def initialize
+      assign_defaults
+    end
+
+    private
+
+    def assign_defaults
+      self.amqp = "amqp://guest:guest@localhost:5672/"
+      self.exchange_name = 'leveret_exch'
+      self.log_file = STDOUT
+      self.log_level = Logger::DEBUG
+      self.queue_name_prefix = 'leveret_queue'
+      self.default_queue_name = 'standard'
+      self.after_fork = proc {}
+      self.error_handler = proc { |ex| ex }
+      self.concurrent_fork_count = 1
+    end
+  end
+end

data/lib/leveret/job.rb

@@ -0,0 +1,159 @@
+module Leveret
+  # Include this module in your job to create a leveret compatible job.
+  # Once included, simply override #perform to do your jobs action.
+  #
+  # To set a different queue name call #queue_name in your class, to set
+  # the default priority call #priority in your class.
+  #
+  # To queue a job simply call #enqueue on the class with the parameters
+  # to be passed. These params will be serialized as JSON in the interim,
+  # so ensure that your params are json-safe.
+  #
+  # @example Job Class
+  #   class MyJob
+  #     include Leveret::Job
+  #
+  #     queue_name 'my_custom_queue' # omit for default
+  #     priority :high # omit for default
+  #
+  #     def perform
+  #       File.open('/tmp/leveret-test-file.txt', 'a+') do |f|
+  #         f.puts params[:test_text]
+  #       end
+  #
+  #       sleep 5 # Job takes a long time
+  #     end
+  #   end
+  #
+  # @example Queueing a Job
+  #   # With options defined in class
+  #   MyJob.enqueue(test_text: "Hi there, please write this text to the file")
+  #
+  #   # Set the job priority at queue time
+  #   MyJob.enqueue(test_text: "Hi there, please write this important text to the file", priority: :high)
+  #
+  #   # Place in a different queue to the one defined in the class
+  #   MyJob.enqueue(test_text: "Hi there, please write this different text to the file", queue_name: 'other_queue')
+  #
+  module Job
+    # Raise this when your job has failed, but try again when a worker is
+    # available again.
+    class RequeueJob < StandardError; end
+
+    # Raise this when your job has failed, but you don't want to requeue it
+    # and try again.
+    class RejectJob < StandardError; end
+
+    # Instance methods to mixin with your job
+    module InstanceMethods
+      # @!attribute params
+      #   @return [Paramters] The parameters required for task execution
+      attr_accessor :params
+
+      # Create a new job ready for execution
+      #
+      # @param [Parameters] params Parameters to be consumed by {#perform} when performing the job
+      def initialize(params = Parameters.new)
+        self.params = params
+      end
+
+      # Runs the job and captures any exceptions to turn them into symbols which represent the status of the job
+      #
+      # @return [Symbol] :success, :requeue, :reject depending on job success
+      def run
+        Leveret.log.info "Running #{self.class.name} with #{params}"
+        perform
+        :success
+      rescue Leveret::Job::RequeueJob
+        Leveret.log.warn "Requeueing job #{self.class.name} with #{params}"
+        :requeue
+      rescue Leveret::Job::RejectJob
+        Leveret.log.warn "Rejecting job #{self.class.name} with #{params}"
+        :reject
+      rescue StandardError => e
+        Leveret.log.error "#{e.message} when processing #{self.class.name} with #{params}"
+        Leveret.configuration.error_handler.call(e)
+        :reject
+      end
+
+      # Run the job with no error handling. Generally you should call {#run} to execute the job since that'll write
+      # and log output and call any error handlers if the job goes sideways.
+      #
+      # @note Your class should override this method to contain the work to be done in this job.
+      #
+      # @raise [RequeueJob] Reject this job and put it back on the queue for future execution
+      # @raise [RejectJob] Reject this job and do not requeue it.
+      def perform
+        raise NotImplementedError
+      end
+    end
+
+    # Class methods to mixin with your job
+    module ClassMethods
+      # Shorthand to intialize a new job and run it with error handling
+      #
+      # @param [Parameters] params Parameters to pass to the job for execution
+      #
+      # @return [Symbol] :success, :requeue or :reject depending on job execution
+      def perform(params = Parameters.new)
+        new(params).run
+      end
+
+      # Set a custom queue for this job
+      #
+      # @param [String] name Name of the queue to assign this job to
+      def queue_name(name)
+        job_options[:queue_name] = name
+      end
+
+      # Set a custom priority for this job
+      #
+      # @param [Symbol] priority Priority for this job, see {Queue::PRIORITY_MAP} for details
+      def priority(priority)
+        job_options[:priority] = priority
+      end
+
+      # @return [Hash] The current set of options for this job, the +queue_name+ and +priority+.
+      def job_options
+        @job_options ||= {
+          queue_name: Leveret.configuration.default_queue_name,
+          priority: :normal
+        }
+      end
+
+      # Place a job onto the queue for processing by a worker.
+      #
+      # @param [Hash] params The parameters to be included with the work request. These can be anything, however
+      #   the keys +:priority+ and +:queue_name+ are reserved for customising those aspects of the job's execution
+      #
+      # @option params [Symbol] :priority (job_options[:priority]) Override the class-level priority for this job only
+      # @option params [String] :queue_name (job_options[:queue_name]) Override the class-level queue name for this
+      #   job only.
+      def enqueue(params = {})
+        priority = params.delete(:priority) || job_options[:priority]
+        q_name = params.delete(:queue_name) || job_options[:queue_name]
+
+        Leveret.log.info "Queuing #{name} to #{q_name} (#{priority}) with #{params}"
+
+        payload = { job: self.name, params: params }
+        queue(q_name).publish(payload, priority: priority)
+      end
+
+      # @private Cache the queue for this job
+      #
+      # @param [q_name] The name of the queue we want to cache. If nil it'll use the name defined in
+      #   +job_options[:queue_name]+
+      # @return [Queue] Cached Queue object for publishing jobs
+      def queue(q_name = nil)
+        q_name ||= job_options[:queue_name]
+        @queue ||= {}
+        @queue[q_name] ||= Leveret::Queue.new(q_name)
+      end
+    end
+
+    def self.included(base)
+      base.extend ClassMethods
+      base.include InstanceMethods
+    end
+  end
+end

data/lib/leveret/log_formatter.rb

@@ -0,0 +1,24 @@
+module Leveret
+  # Prettier logging than the default
+  class LogFormatter < Logger::Formatter
+    # ANSI colour codes for different message types
+    SEVERITY_TO_COLOR_MAP = { 'DEBUG' => '0;37', 'INFO' => '32', 'WARN' => '33', 'ERROR' => '31', 'FATAL' => '31',
+                              'UNKNOWN' => '37' }.freeze
+
+    # Build a pretty formatted log line
+    #
+    # @param [String] severity Log level, one of debug, info, warn, error, fatal  or unknown
+    # @param [Time] datetime Timestamp of log event
+    # @param [String] _progname (Unused) the name of the progname set in the logger
+    # @param [String] msg Body of the log message
+    #
+    # @return [String] Formatted and coloured log message in the format:
+    #   "YYYY-MM-DD HH:MM:SS:USEC [SEVERITY] MESSAGE (pid:Process ID)"
+    def call(severity, datetime, _progname, msg)
+      formatted_time = datetime.strftime("%Y-%m-%d %H:%M:%S") << datetime.usec.to_s[0..2].rjust(3)
+      color = SEVERITY_TO_COLOR_MAP[severity]
+
+      "\033[0;37m#{formatted_time}\033[0m [\033[#{color}m#{severity}\033[0m] #{msg2str(msg)} (pid:#{Process.pid})\n"
+    end
+  end
+end

data/lib/leveret/parameters.rb

@@ -0,0 +1,59 @@
+module Leveret
+  # Provides basic indifferent hash access for jobs, allows strings to be used to access symbol keyed values,
+  # or symbols to be used to access string keyed values.
+  #
+  # Overrides the [] method of the hash, all other calls (except {#serialize}) are delegated to the {#params} object
+  #
+  # Beware of using both strings and symbols with the same value in the same hash e.g. +{:one => 1, 'one' => 1}+
+  # only one of these values will ever be returned.
+  class Parameters
+    extend Forwardable
+
+    # The parameters hash wrapped up by this object
+    attr_accessor :params
+
+    def_delegators :params, :==, :inspect, :to_s
+
+    # @param [Hash] params Hash you wish to access indifferently
+    def initialize(params = {})
+      self.params = params || {}
+    end
+
+    # Access {#params} indifferently. Tries the passed key directly first, then tries it as a symbol, then tries
+    # it as a string.
+    #
+    # @param [Object] key Key of the item we're trying to access in {#params}
+    #
+    # @return [Object, nil] Value related to key, or nil if object is not found
+    def [](key)
+      params[key] || (key.respond_to?(:to_sym) && params[key.to_sym]) || (key.respond_to?(:to_s) && params[key.to_s])
+    end
+
+    # Delegate any unknown methods to the {#params} hash
+    def method_missing(method_name, *arguments, &block)
+      params.send(method_name, *arguments, &block)
+    end
+
+    # Check the {#params} hash as well as this class for a method's existence
+    def respond_to?(method_name, include_private = false)
+      params.respond_to?(method_name, include_private) || super
+    end
+
+    # Serialize the current value of {#params}. Outputs JSON.
+    #
+    # @return [String] JSON encoded representation of the params
+    def serialize
+      JSON.dump(params)
+    end
+
+    # Create a new instance of this class from a a serialized JSON object.
+    #
+    # @param [String] json JSON representation of the parameters we want to access
+    #
+    # @return [Parameters] New instance based on the passed JSON
+    def self.from_json(json)
+      params = JSON.load(json)
+      new(params)
+    end
+  end
+end

data/lib/leveret/queue.rb

@@ -0,0 +1,105 @@
+module Leveret
+  # Facilitates the publishing or subscribing of messages to the message queue.
+  #
+  # @!attribute [r] name
+  #   @return [String] Name of the queue. This will have Leveret.queue_name_prefix prepended to it when creating a
+  #     corresponding queue in RabbitMQ.
+  # @!attribute [r] queue
+  #   @return [Bunny::Queue] The backend RabbitMQ queue
+  #   @see http://reference.rubybunny.info/Bunny/Queue.html Bunny::Queue Documentation
+  class Queue
+    extend Forwardable
+
+    # Map the symbol names for priorities to the integers that RabbitMQ requires.
+    PRIORITY_MAP = { low: 0, normal: 1, high: 2 }.freeze
+    attr_reader :name, :queue
+
+    def_delegators :Leveret, :exchange, :channel, :log
+    def_delegators :queue, :pop, :purge
+
+    # Create a new queue with the name given in the params, if no name is given it will default to
+    # {Configuration#default_queue_name}. On instantiation constructor will immedaitely connect to
+    # RabbitMQ backend and create a queue with the appropriate name, or join an existing one.
+    #
+    # @param [String] name Name of the queue to connect to.
+    def initialize(name = nil)
+      @name = name || Leveret.configuration.default_queue_name
+      @queue = connect_to_queue
+    end
+
+    # Publish a mesage onto the queue. Fire and forget, this method is non-blocking and will not wait until
+    # the message is definitely on the queue.
+    #
+    # @param [Hash] payload The data we wish to send onto the queue, this will be serialized and automatically
+    #   deserialized when received by a {#subscribe} block.
+    # @option options [Symbol] :priority (:normal) The priority this message should be treated with on the queue
+    #   see {PRIORITY_MAP} for available options.
+    #
+    # @return [void]
+    def publish(payload, options = {})
+      priority_id = PRIORITY_MAP[options[:priority]] || PRIORITY_MAP[:normal]
+      payload = serialize_payload(payload)
+
+      log.debug "Publishing #{payload.inspect} for queue #{name} (Priority: #{priority_id})"
+      queue.publish(payload, persistent: true, routing_key: name, priority: priority_id)
+    end
+
+    # Subscribe to this queue and yield a block for every message received. This method does not block, receiving and
+    # dispatching of messages will be handled in a separate thread.
+    #
+    # The receiving block is responsible for acknowledging or rejecting the message. This must be done using the
+    # same channel the message was received # on, {#Leveret.channel}. {Worker#ack_message} provides an example
+    # implementation of this acknowledgement.
+    #
+    # @note The receiving block is responsible for acking/rejecting the message. Please see the note for more details.
+    #
+    # @yieldparam delivery_tag [String] The identifier for this message that must be used do ack/reject the message
+    # @yieldparam payload [Parameters] A deserialized version of the payload contained in the message
+    #
+    # @return [void]
+    def subscribe
+      log.info "Subscribing to #{name}"
+      queue.subscribe(manual_ack: true) do |delivery_info, _properties, msg|
+        log.debug "Received #{msg} from #{name}"
+        yield(delivery_info.delivery_tag, deserialize_payload(msg))
+      end
+    end
+
+    private
+
+    # Convert a set of parameters passed into a serialized form suitable for transport on the message queue
+    #
+    # @param [Hash] Paramets to be serialized
+    #
+    # @return [String] Encoded params ready to be sent onto the queue
+    def serialize_payload(params)
+      Leveret::Parameters.new(params).serialize
+    end
+
+    # Convert a set of serialized parameters into a {Parameters} object
+    #
+    # @param [String] JSON representation of the parameters
+    #
+    # @return [Parameters] Useful object representation of the parameters
+    def deserialize_payload(json)
+      Leveret::Parameters.from_json(json)
+    end
+
+    # Create or return a representation of the queue on the RabbitMQ backend
+    #
+    # @return [Bunny::Queue] RabbitMQ queue
+    def connect_to_queue
+      queue = channel.queue(mq_name, durable: true, auto_delete: false, arguments: { 'x-max-priority' => 2 })
+      queue.bind(exchange, routing_key: name)
+      log.debug "Connected to #{mq_name}, bound on #{name}"
+      queue
+    end
+
+    # Calculate the name of the queue that should be used on the RabbitMQ backend
+    #
+    # @return [String] Backend queue name
+    def mq_name
+      @mq_name ||= [Leveret.configuration.queue_name_prefix, name].join('_')
+    end
+  end
+end

data/lib/leveret/version.rb

@@ -0,0 +1,3 @@
+module Leveret
+  VERSION = "0.1.0"
+end

data/lib/leveret/worker.rb

@@ -0,0 +1,147 @@
+module Leveret
+  # Subscribes to one or more queues and forks workers to perform jobs as they arrive
+  #
+  # Call {#do_work} to subscribe to all queues and block the main thread.
+  class Worker
+    extend Forwardable
+
+    # @!attribute queues
+    #   @return [Array<Queue>] All of the queues this worker is going to subscribe to
+    # @!attribute consumers
+    #   @return [Array<Bunny::Consumer>] All of the actively subscribed queues
+    attr_accessor :queues, :consumers
+
+    def_delegators :Leveret, :log, :channel, :configuration
+
+    # Create a new worker to process jobs from the list of queue names passed
+    #
+    # @option options [Array<String>] queues ([Leveret.configuration.default_queue_name]) A list of queue names for
+    #   this worker to subscribe to and process
+    # @option options [Integer] concurrent_fork_count (Leveret.configuration.concurrent_fork_count) How many messages
+    #   at a time should this worker process?
+    def initialize(options = {})
+      options = {
+        queues: [configuration.default_queue_name],
+        concurrent_fork_count: [configuration.concurrent_fork_count]
+      }.merge(options)
+
+      Leveret.configuration.concurrent_fork_count = options[:concurrent_fork_count]
+
+      self.queues = options[:queues].map { |name| Leveret::Queue.new(name) }
+      self.consumers = []
+      @time_to_die = false
+    end
+
+    # Subscribe to all of the {#queues} and begin processing jobs from them. This will block the main
+    # thread until an interrupt is received.
+    def do_work
+      log.info "Starting master process for #{queues.map(&:name).join(', ')}"
+      prepare_for_work
+
+      loop do
+        if @time_to_die
+          cancel_subscriptions
+          break
+        end
+        sleep 1
+      end
+      log.info "Exiting master process"
+    end
+
+    private
+
+    # Steps that need to be prepared before we can begin processing jobs
+    def prepare_for_work
+      setup_traps
+      self.process_name = 'leveret-worker-parent'
+      start_subscriptions
+    end
+
+    # Catch INT and TERM signals and set an instance variable to signal the main loop to quit when possible
+    def setup_traps
+      trap('INT') do
+        @time_to_die = true
+      end
+      trap('TERM') do
+        @time_to_die = true
+      end
+    end
+
+    # Set the title of this process so it's easier on the eye in top
+    def process_name=(name)
+      Process.setproctitle(name)
+    end
+
+    # Subscribe to each queue defined in {#queues} and add the returned consumer to {#consumers}. This will
+    # allow us to gracefully cancel these subscriptions when we need to quit.
+    def start_subscriptions
+      queues.map do |queue|
+        consumers << queue.subscribe do |delivery_tag, payload|
+          fork_and_run(delivery_tag, payload)
+        end
+      end
+    end
+
+    # Send cancel to each consumer in the {#consumers} list. This will end the current subscription.
+    def cancel_subscriptions
+      log.info "Interrupt received, preparing to exit"
+      consumers.each do |consumer|
+        log.debug "Cancelling consumer on #{consumer.queue.name}"
+        consumer.cancel
+      end
+    end
+
+    # Fork the current process and run the job described by #payload in the newly created child process.
+    # Detach the main process from the child so we can return to the main loop without waiting for it to finish
+    # processing the job.
+    #
+    # @param [String] delivery_tag The identifier that RabbitMQ uses to track the message. This will be used to ack
+    #   or reject the message after processing.
+    # @param [Parameters] payload The job name and parameters the job requires
+    def fork_and_run(delivery_tag, payload)
+      pid = fork do
+        self.process_name = 'leveret-worker-child'
+        log.info "[#{delivery_tag}] Forked to child process #{pid} to run #{payload[:job]}"
+
+        Leveret.configuration.after_fork.call
+
+        result = perform_job(payload)
+        log.info "[#{delivery_tag}] Job returned #{result}"
+        ack_message(delivery_tag, result)
+
+        log.info "[#{delivery_tag}] Exiting child process #{pid}"
+        exit!(0)
+      end
+
+      # Master doesn't need to know how it all went down, the worker will report it's own status back to the queue
+      Process.detach(pid)
+    end
+
+    # Constantize the class name in the payload and execute the job with parameters
+    #
+    # @param [Parameters] payload The job name and parameters the job requires
+    # @return [Symbol] :success, :reject or :requeue depending on how job execution went
+    def perform_job(payload)
+      job_klass = Object.const_get(payload[:job])
+      job_klass.perform(Leveret::Parameters.new(payload[:params]))
+    end
+
+    # Sends a message back to RabbitMQ confirming the completed execution of the message
+    #
+    # @param [String] delivery_tag The identifier that RabbitMQ uses to track the message. This will be used to ack
+    #   or reject the message after processing.
+    # @param [Symbol] result :success, :reject or :requeue depending on how we want to acknowledge the message
+    def ack_message(delivery_tag, result)
+      if result == :reject
+        log.debug "[#{delivery_tag}] Rejecting message"
+        channel.reject(delivery_tag)
+      elsif result == :requeue
+        log.debug "[#{delivery_tag}] Requeueing message"
+        channel.reject(delivery_tag, true)
+      else
+        log.debug "[#{delivery_tag}] Acknowledging message"
+        channel.acknowledge(delivery_tag)
+      end
+    end
+  end
+end

data/lib/leveret.rb

@@ -0,0 +1,78 @@
+require 'bunny'
+require 'json'
+require 'logger'
+
+require 'leveret/configuration'
+require 'leveret/job'
+require 'leveret/log_formatter'
+require 'leveret/parameters'
+require 'leveret/queue'
+require 'leveret/worker'
+require "leveret/version"
+
+# Top level module, contains things that are required globally by Leveret, such as configuration,
+# the RabbitMQ channel and the logger.
+module Leveret
+  class << self
+    # @!attribute [w] configuration
+    #   @return [Configuration] Set a totally new configuration object
+    attr_writer :configuration
+
+    # @return [Configuration] The current configuration of Leveret
+    def configuration
+      @configuration ||= Configuration.new
+    end
+
+    # Allows leveret to be configured via a block
+    #
+    # @see Configuration Attributes that can be configured
+    # @yield [config] The current configuration object
+    def configure
+      yield(configuration) if block_given?
+    end
+
+    # Connect to the RabbitMQ exchange that Leveret uses, used by the {Queue} for publishing and subscribing, not
+    # recommended for general use.
+    #
+    # @see http://reference.rubybunny.info/Bunny/Exchange.html Bunny documentation
+    # @return [Bunny::Exchange] RabbitMQ exchange
+    def exchange
+      @exchange ||= channel.exchange(Leveret.configuration.exchange_name, type: :direct, durable: true,
+        auto_delete: false)
+    end
+
+    # Connect to the RabbitMQ channel that {Queue} and {Worker} both use. This channel is not thread safe, so should
+    # be reinitialized if necessary. Not recommended for general use.
+    #
+    # @see http://reference.rubybunny.info/Bunny/Channel.html Bunny documentation
+    # @return [Bunny::Channel] RabbitMQ chanel
+    def channel
+      @channel ||= begin
+        chan = mq_connection.create_channel
+        chan.prefetch(configuration.concurrent_fork_count)
+        chan
+      end
+    end
+
+    # Logger used throughout Leveret, see {Configuration} for config options.
+    #
+    # @return [Logger] Standard ruby logger
+    def log
+      @log ||= Logger.new(configuration.log_file).tap do |log|
+        log.level = configuration.log_level
+        log.progname = 'Leveret'
+        log.formatter = Leveret::LogFormatter.new
+      end
+    end
+
+    private
+
+    def mq_connection
+      @mq_connection ||= begin
+        conn = Bunny.new(amqp: configuration.amqp)
+        conn.start
+        conn
+      end
+    end
+  end
+end

metadata

@@ -0,0 +1,133 @@
+--- !ruby/object:Gem::Specification
+name: leveret
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Dan Wentworth
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2016-07-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: bunny
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.3'
+- !ruby/object:Gem::Dependency
+  name: json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.8'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.8'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.10'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- dan@atechmedia.com
+executables:
+- leveret_worker
+extensions: []
+extra_rdoc_files: []
+files:
+- ".rspec"
+- ".travis.yml"
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- bin/console
+- bin/setup
+- exe/leveret_worker
+- leveret.gemspec
+- lib/leveret.rb
+- lib/leveret/configuration.rb
+- lib/leveret/job.rb
+- lib/leveret/log_formatter.rb
+- lib/leveret/parameters.rb
+- lib/leveret/queue.rb
+- lib/leveret/version.rb
+- lib/leveret/worker.rb
+homepage: https://github.com/darkphnx/leveret
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.1
+signing_key: 
+specification_version: 4
+summary: Simple RabbitMQ backed backround worker
+test_files: []