osbourne 0.1.3

data/README.md

@@ -0,0 +1,106 @@
+[![Maintainability](https://api.codeclimate.com/v1/badges/295897ee565c04ad1aa5/maintainability)](https://codeclimate.com/github/stevenallen05/osbourne/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/295897ee565c04ad1aa5/test_coverage)](https://codeclimate.com/github/stevenallen05/osbourne/test_coverage)
+
+# Osbourne
+
+A fan-out pubsub message implementation for Rails 5. Named after the world's most famous plumber, Ozzy Osbourne.
+
+### Features
+
+* Publish messages via `Osbourne.publish("topic_name", message)`. Messages are expected to either be a `String` or respond to `#to_json`
+* Message processor via the `osbourne` shell command
+* Worker generator via `rails g osbourne:worker worker_name topic`
+* Auto-provisioning of SQS queues, SNS topics, and subscriptions between them
+* Built-in support for locking to prevent accidental duplicate message delivery 
+
+Inspired heavily by the excellent Shoryuken & Circuitry gems
+
+## Installation
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'osbourne'
+```
+
+And then execute:
+```bash
+$ bundle install
+$ bundle exec rails g osbourne:install
+```
+
+Installation creates `config/initializers/osbourne.rb` and `config/osbourne.yml`
+
+### AWS credentials
+
+There are a few ways to configure the AWS client:
+
+* Ensure the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` env vars are set.
+* Create a `~/.aws/credentials` file.
+* Set `Aws.config[:credentials]` or `Osbourne.sqs_client = Aws::SQS::Client.new(...)` and `Osbourne.sns_client = Aws::SNS::Client.new(...)` from Ruby code
+* Use the Instance Profiles feature. The IAM role of the targeted machine must have an adequate SQS Policy.
+
+You can read about these in more detail [here](http://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html).
+
+### Production AWS
+
+The default `config/osbourne.yml` production configuration is blank. This is not by accident. Hitting real AWS resources requires no additional configuration.
+
+### Mock AWS for local devlopment
+
+This gem has been tested successfully with [localstack](https://github.com/localstack/localstack). The `localstack` tools can succesfully emulate SQS & SNS, as well as subscription marshalling.
+
+The following settings in `config/osbourne.yml` will configure Osbourne to use a mocked SQS & SNS on `localhost`:
+
+```yaml
+development:
+  publisher:
+    endpoint: http://localhost:4575
+  subscriber:
+    endpoint: http://localhost:4576
+    verify_checksums: false
+```
+
+Please note `verify_checksums: false` is required for the `subscriber` configuration.
+
+It is recommended to use the `dotenv` gem and add these dummy credentials to your `.env`:
+
+```
+AWS_ACCESS_KEY_ID=AKXXXXXXXXXXXXXXXXXXX
+AWS_SECRET_ACCESS_KEY=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
+AWS_REGION=us-east-1
+```
+
+There must be a region set. Use whichever is appropriate for your situation.
+
+## Usage
+
+### Publishing a message
+
+Publishing a message is simple: `Osbourne.publish("topic_name", message)`
+
+Osbourne will automatically provision the SNS topic if it does not already exist. The `message` object is expected to either be a `String` object, or respond to `#to_json`.
+
+### Generating a worker
+
+```bash
+$ bundle exec rails g osbourne:worker worker_name topic1 topic2
+```
+
+This will generate a `WorkerNameWorker` in `app/workers/worker_name_worker.rb`, subscribed to `topic1` and `topic2`.
+
+There is some configuration options available within the generated worker. See comments in the worker for options.
+
+SNS messages broadcast through an SQS queue will have some layers of envelop wrappings around them. The `message` object passed into the worker's `#perform` method contains some helpers to make parsing this easier. `#parsed_body` is the most important one, as it contains the actual string of the message that was originally broadcast.
+
+### Running workers
+
+To run all workers in `app/workers`:
+
+```bash
+$ bundle exec osbourne
+```
+
+This can be easily added to a [Procman](https://github.com/adamcooke/procman) configuration to run alongside your ActiveJob processor (eg: sidekiq)
+
+
+## License
+The gem is available as open source under the terms of the [GPL-3.0 License](https://opensource.org/licenses/GPL-3.0).

data/Rakefile

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+begin
+  require "bundler/setup"
+rescue LoadError
+  puts "You must `gem install bundler` and `bundle install` to run rake tasks"
+end
+
+require "rdoc/task"
+
+RDoc::Task.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = "rdoc"
+  rdoc.title    = "Osbourne"
+  rdoc.options << "--line-numbers"
+  rdoc.rdoc_files.include("README.md")
+  rdoc.rdoc_files.include("lib/**/*.rb")
+end
+
+require "bundler/gem_tasks"
+
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "lib"
+  t.libs << "test"
+  t.pattern = "test/**/*_test.rb"
+  t.verbose = false
+end
+
+task default: :test

data/bin/cli/base.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+module Osbourne
+  module CLI
+    class Base < Thor
+      no_commands do
+        def print_table(entries)
+          column_sizes = print_columns_size(entries)
+
+          entries.map do |entry|
+            puts entry.map.with_index {|e, i| print_format_column(e, column_sizes[i]) }.join
+          end
+        end
+
+        def print_columns_size(entries)
+          column_sizes = Hash.new(0)
+
+          entries.each do |entry|
+            entry.each_with_index do |e, i|
+              e = e.to_s
+              column_sizes[i] = e.size if column_sizes[i] < e.size
+            end
+          end
+
+          column_sizes
+        end
+
+        def print_format_column(column, size)
+          size_with_padding = size + 4
+          column = column.to_s.ljust(size_with_padding)
+          column
+        end
+
+        def fail_task(msg, quit=true)
+          say "[FAIL] #{msg}", :red
+          exit(1) if quit
+        end
+      end
+    end
+  end
+end

data/bin/osbourne

@@ -0,0 +1,33 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+
+require 'thor'
+
+require 'rails'
+require File.expand_path('config/environment.rb')
+
+require_relative 'cli/base'
+require_relative '../lib/osbourne/runner'
+
+module Osbourne
+  module CLI
+    class Runner < Thor
+      default_task :start
+      desc 'start', 'Starts Osbourne'
+
+      def start
+        puts "Launching Osbourne workers"
+        Osbourne::Runner.instance.run
+      end
+
+      desc 'version', 'Prints version'
+      def version
+        say "Osbourne #{Osbourne::VERSION}"
+      end
+
+    end
+  end
+end
+
+Osbourne::CLI::Runner.start

data/lib/generators/osbourne/install/install_generator.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+
+module Osbourne
+  class InstallGenerator < Rails::Generators::Base
+    source_root File.expand_path("templates", __dir__)
+
+    def generate_install
+      template "osbourne_yaml_template.template", "config/osbourne.yml"
+      template "osbourne_initializer_template.template", "config/initializers/osbourne.rb"
+    end
+  end
+end

data/lib/generators/osbourne/install/templates/osbourne_initializer_template.template

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+Dir[File.expand_path("app/workers/**/*.rb")].each {|f| require f }
+
+Osbourne.configure do |config|
+  # Defaults to a null cache. Uncomment to use the Rails cache,
+  # or substitute any rails-cache compatible adapter of your choice
+  # config.cache = Rails.cache
+
+  # Dead letter queues are used to store messages that fail processing for some reason
+  # They are enabled by default
+  # config.dead_letter = true
+
+  # A message will be attempted `max_retry_count` times before being
+  # sent to the dead letter queue
+  # config.max_retry_count = 5 
+
+  # Amount of time each worker will wait between attempting to fetch messages
+  # config.sleep_time = 15.seconds
+
+  config.logger = Rails.logger
+
+  # The lock strategy to be used
+  # Supported lock strategies:
+  #  * Osbourne::Locks::Redis - requires a redis client or redis config
+  #  * Osbourne::Locks::NOOP - the default. No locking. Bad for production
+  #  * Osbourne::Locks::Memory - uses pure memory 
+  # config.lock = Osbourne::Locks::Redis.new(url: 'redis://localhost:6379')
+
+end

data/lib/generators/osbourne/install/templates/osbourne_yaml_template.template

@@ -0,0 +1,15 @@
+development:
+  publisher:
+    endpoint: http://localhost:4575
+  subscriber:
+    endpoint: http://localhost:4576
+    verify_checksums: false
+
+test:
+  publisher:
+    endpoint: http://localhost:4575
+  subscriber:
+    endpoint: http://localhost:4576
+    verify_checksums: false
+
+production:

data/lib/generators/osbourne/worker/USAGE

@@ -0,0 +1,11 @@
+Description:
+    Generates a fan-out worker that ensures the given queues, topics, and subscriptions are instantiated
+
+Example:
+    rails generate queue_worker WorkerName topic_1 <other_topics>
+
+    This will create:
+        app/workers/<name>_worker.rb
+
+    `topic_1` will be the message topic that the queue will listen to. Multiple topics can listed 
+    

data/lib/generators/osbourne/worker/templates/worker_template.template

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+class <%= name.singularize.camelcase %>Worker < Osbourne::WorkerBase
+
+  worker_config topics: %w[<%= topic.join(" ") %>] #, max_batch_size: 10, max_wait: 10
+
+  def process(message)
+    puts message.raw_body    # This will always be a string, representing the entirety of the message
+    puts message.parsed_body # If the message came from a SNS broadcast,
+                             #   as opposed to a direct SQS message, it will come from here 
+    puts message.topic       # The topic this message was published to.
+                             #   Useful if this worker subscribes to more than one topic
+    puts message.id          # The UUID for this message. Useful for validating if this is
+                             #   the first time it has been processed 
+  end
+
+  # class << self
+  #   # override this to set how many times a message will be retried before
+  #   # being redirected to the dead letter queue (if enabled)
+  #   def max_retry_count
+  #   end
+
+  #   # override this to set the worker's dead letter queue name
+  #   def dead_letter_queue_name
+  #   end
+
+  #   # override this to `false` to disable the dead letter queu for this worker
+  #   def dead_letter_queue
+  #   end
+
+  #   # override this to set the worker queue name
+  #   def queue_name
+  #   end
+  # end
+end

data/lib/generators/osbourne/worker/worker_generator.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+
+module Osbourne
+  class WorkerGenerator < Rails::Generators::NamedBase
+    source_root File.expand_path("templates", __dir__)
+    argument :topic, type: :array
+
+    def generate_worker
+      template "worker_template.template", "app/workers/#{name.underscore}_worker.rb"
+    end
+
+    private
+
+    def config_file
+      Rails.root.join("config", "queue", "queue_workers.yml")
+    end
+  end
+end

data/lib/osbourne.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+require "osbourne/railtie"
+require "osbourne/services/sns"
+require "osbourne/services/sqs"
+require "osbourne/topic"
+require "osbourne/queue"
+require "osbourne/subscription"
+require "osbourne/config/shared_configs"
+require "osbourne/worker_base"
+require "osbourne/services/queue_provisioner"
+require "osbourne/launcher"
+require "osbourne/message"
+require "osbourne/existing_subscriptions"
+require "osbourne/locks/base"
+require "osbourne/locks/noop"
+require "osbourne/locks/memory"
+require "osbourne/locks/redis"
+
+module Osbourne
+  class << self
+    include Osbourne::Config::SharedConfigs
+    include Osbourne::Services::QueueProvisioner
+    include Osbourne::ExistingSubscriptions
+    attr_writer :sns_client, :sqs_client
+
+    def sns_client
+      @sns_client ||= Aws::SNS::Client.new(Osbourne.config.sns_config)
+    end
+
+    def sqs_client
+      @sqs_client ||= Aws::SQS::Client.new(Osbourne.config.sqs_config)
+    end
+
+    def publish(topic, message)
+      Topic.new(topic).publish(message)
+    end
+
+    def configure
+      yield config
+    end
+  end
+end

data/lib/osbourne/config/file_loader.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "yaml"
+require "erb"
+require "fileutils"
+
+module Osbourne
+  module Config
+    module FileLoader
+      def self.load(cfile, environment="development")
+        return nil unless File.exist?(cfile)
+
+        base_opts = YAML.safe_load(ERB.new(IO.read(cfile)).result) || {}
+        env_opts = base_opts[environment] || {}
+
+        Osbourne.config.sns_config = env_opts["publisher"].symbolize_keys || {}
+        Osbourne.config.sqs_config = env_opts["subscriber"].symbolize_keys || {}
+        true
+      end
+    end
+  end
+end

data/lib/osbourne/config/shared_configs.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module Osbourne
+  module Config
+    module SharedConfigs
+      attr_writer :sqs_config, :sns_config, :aws_credentials, :config
+
+      def config
+        @config ||= ActiveSupport::OrderedOptions.new
+      end
+
+      def cache
+        config.cache ||= ActiveSupport::Cache::NullStore.new
+      end
+
+      def dead_letter
+        config.dead_letter ||= true
+      end
+
+      def max_retry_count
+        @max_retry_count ||= (config.max_retry_count.presence || 5).to_s
+      end
+
+      def logger
+        config.logger ||= Logger.new(STDOUT)
+      end
+
+      def lock
+        config.lock || Osbourne::Locks::NOOP.new
+      end
+
+      def sleep_time
+        config.sleep_time || 15
+      end
+    end
+  end
+end