piscina 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7be7bd0b99e9978bb6758a13a6a8b8f7e775f871
+  data.tar.gz: 42aeb2c118c89db8abb5d76bc1f0287dda408b6b
+SHA512:
+  metadata.gz: 9b195cea45cb009df944c2898a7bec584e8c655b69ad0409db9339142e04a52a8767ee538aab0b17833f51607a9254cea8ff6c961dad9f79bffc2bfe3099ce75
+  data.tar.gz: 893c4025b9046ac206ec0c7419da930188cbaf64b8094cc1a5814865c336855eceddc14bf0318f09ab891ed298b054738db1d71ad19a223a17967b397a2f88b1

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 Joseph Feeney
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,104 @@
+Piscina
+=======
+
+Spanish for "Pool"
+
+## Blurb
+JRuby SQS job processor/dispatcher and multi-threaded logger.
+
+## Dependencies
+- Some flavor of JRuby
+- Ruby Standard Library
+- AWS SDK
+- FakeSQS for tests
+
+## Piscina
+Each instance of Piscina creates N + 2 OS threads. One thread to pull SQS messages off the queue
+and place them into a buffer, 1 thread for logging, and N threads in a pool that process those jobs.
+If the number of jobs that are placed into the pool are greater than N, the pool will just buffer
+those jobs until there is a thread ready to process them. This is handled internally by the Java
+class Excecutors.
+
+Piscina instances handle SIGTERM's and other shutdowns gracefully by allowing the buffered jobs
+to finish before the thread pool is terminated.
+
+Piscina makes the assumption that each SQS queue can be handled by one class. The class itself
+can have more complicated logic to process different types of messages based on the content of the
+message itself but that is left up to you (the developer).
+
+Piscina takes two arguments
+SQS_URL  : the specific URL for your SQS queue
+Klass    : Klass must know Klass#perform(msg) and must handle the deletion or the immediate
+           requeue-ing of the message if perform is unsuccessful. Otherwise default SQS
+           visibility rules will apply. Exceptions are caught without exception (**ahem**). Dead Letter
+           Queue (DLQ) policies are handled by the SQS queue itself. E.g. three attempts are made
+           to deliver the message but all three attempts result in an exception the SQS queue will
+           place the message in the DLQ for further inspection.
+
+options[:pool_threads] : How many active threads there should be in each Piscina pool.
+
+### In use
+If, for example, you'd like to use a local SQS API in development and in test, but hit the real
+end points in production you could
+
+```ruby
+# Development.rb
+sqs_url = "https://localhost:4658/QUEUE_NAME"
+Piscina.new(sqs_url, HttpJob)
+
+# Production.rb
+sqs_url = "https://sqs.AWS_REGION.amazonaws.com/YOUR_ACCOUNT_NUM/QUEUE_NAME"
+Piscina.new(sqs_url, HttpJob)
+```
+
+Otherwise you can initialize all of your queues at startup
+
+## Piscina Logger
+PiscinaLogger is created to handle logging events for Piscina. However, it is suitable for any
+logging where many threads need to write to the same file at once, and the order in which they
+are logged is unimportant.
+
+PiscinaLogger uses a fixed thread pool of size one to write to the log file. Writes are queued
+up until the thread is ready to process again.
+
+### Config Options
+Configuration Defaults:
+- log_directory = RailsRoot/log
+- logging_level = Logger:INFO
+
+```ruby
+PiscinaLogger.configure do |config|
+  config.log_directory = "/User/home/derp_kid/derp_code/log"
+  config.logging_level = Logger::DEBUG
+end
+```
+
+The default options are pretty sane. Check them out in configuration.rb.
+
+## HTTPJob
+An example job (that works!). This is just an arbitrary job that can be performed with any
+queue that implements the format below in the message body.
+
+REMEMBER: SQS does NOT guarantee that messages will be delivered once. Make sure any calls
+to your endpoint can handle this (idempotency).
+
+Format:
+```json
+{
+  "headers": {
+               "user_token":"That's a nice token"     // Any number of Key:Value pairs
+             },
+  "http_method": "GET",                               // or POST PUT DELETE
+  "uri":"https://SOMEHOST.WHATEVER/your/endpoint",    // can be HTTP or HTTPS.
+                                                      // we'll dynamically catch that
+
+  "body":"THIS IS SOME TEXT OR A BODY"                // Body will only be processed in Post's or
+                                                      // Put's
+}
+```
+
+# Build and Install without gem
+`gem build piscina.gemspec && gem install piscina-*.gem`
+
+# Test
+`jruby test/simple_test.rb`

data/lib/piscina.rb

@@ -0,0 +1,3 @@
+require 'piscina/piscina'
+require 'piscina/piscina_logger'
+require 'piscina/http_job'

data/lib/piscina/http_job.rb

@@ -0,0 +1,73 @@
+require 'net/http'
+require 'json'
+
+class HttpJob
+  HTTP_DELETE = "DELETE"
+  HTTP_GET    = "GET"
+  HTTP_POST   = "POST"
+  HTTP_PUT    = "PUT"
+
+  HTTP_METHODS = [
+                    HTTP_DELETE,
+                    HTTP_GET,
+                    HTTP_POST,
+                    HTTP_PUT
+                 ]
+
+  def self.perform(msg)
+    params  = JSON.parse(msg.body)
+
+    method  = params["http_method"]
+    uri     = params["uri"]
+    headers = params["headers"]
+    body    = params["body"]
+
+    req = HttpJob.create_request(method, uri, headers, body)
+    res = HttpJob.get_response(req, uri)
+
+    # Net::HTTPSuccess covers all 2XX responses
+    unless res.is_a?(Net::HTTPSuccess)
+      # Messages not explicitly deleted will be placed back into the queue. DLQ policies will take
+      # effect if a message cannot be processed a number of times.
+      raise "Could not perform request to:#{uri.to_s}"
+    end
+
+    msg.delete
+  end
+
+  def self.create_request(method, uri, headers, body)
+    raise "http_method or uri not defined" unless method && uri
+    raise "#{method} is not a valid HTTP method" unless HTTP_METHODS.include?(method)
+
+    begin
+      uri_obj = URI.parse(uri)
+    rescue => e
+      raise "There was an error parsing #{uri}. ErrorMessage: #{e.message}"
+    end
+
+    req = "Net::HTTP::#{method.capitalize}".constantize.new(uri_obj.request_uri)
+
+    if headers
+      headers.each do |header, val|
+        req[header] = val
+      end
+    end
+
+    if [HTTP_PUT, HTTP_POST].include?(method) && body
+      req.body = body
+    end
+
+    req
+  end
+
+  def self.get_response(req, uri)
+    uri = URI.parse(uri)
+
+    Net::HTTP.start(uri.hostname,
+                    uri.port,
+                    :use_ssl => uri.scheme == 'https') do |http|
+
+      http.request(req)
+    end
+  end
+end

data/lib/piscina/piscina.rb

@@ -0,0 +1,99 @@
+require_relative 'piscina_logger'
+
+java_import java.util.concurrent.Executors
+
+class Piscina
+  DEFAULT_THREADS_IN_POOL    = 10    # threads
+  DEFAULT_MESSAGE_TIMEOUT    = 15    # seconds
+  MESSAGE_VISIBILITY_TIMEOUT = 12*60 # seconds
+  MESSAGE_RETRY_VISIBILITY   = 5     # seconds
+
+  AWS_REGION      = 'us-east-1'
+  AWS_ACCOUNT_NUM = '682315851866'
+
+  def initialize(sqs_url, klass, options={})
+    @queue = create_or_initialize_queue(sqs_url)
+    @klass = klass
+
+    pool_threads = options[:pool_threads] || DEFAULT_THREADS_IN_POOL
+    @thread_pool = Executors.newFixedThreadPool(pool_threads)
+
+    # Use SQS queue name for log name.
+    queue_name = sqs_url.split("/")[-1]
+    @logger = PiscinaLogger.new(queue_name)
+
+    # Listen for SIGTERM's and other shutdown messages
+    at_exit do
+      shutdown_instance
+    end
+
+    self.poll
+  end
+
+  # Unfortunately, we can't use the AWS SDK's Queue#poll with a block as it uses
+  # Queue#call_message_block which will ALWAYS delete a received message.
+  def poll
+    # Creates a real OS thread through JRuby
+    Thread.new do
+      poll_sqs
+    end
+  end
+
+  def poll_sqs
+    loop do
+      msg = @queue.receive_messages(visibility_timeout: MESSAGE_VISIBILITY_TIMEOUT,
+                                    wait_time_seconds: DEFAULT_MESSAGE_TIMEOUT)
+
+      # receive_message can time out and return nil
+      next  if msg.nil?
+      break if @thread_pool.nil? || @thread_pool.isShutdown
+
+      process_message(msg)
+    end
+  end
+
+  def process_message(msg)
+    @thread_pool.execute do
+      begin
+        @klass.perform(msg)
+
+        body = msg.body.delete("\n").delete(" ")
+
+        @logger.info("Successfully processed message:#{msg.id};body:#{body}")
+      rescue => e
+        @logger.error("Could not process message:#{msg.id};body:#{body};error:#{e.message}")
+
+        # DLQ policy -> Messages are attempted N times and then benched. Policy is defined by the
+        # queue itself.
+        msg.visibility_timeout = MESSAGE_RETRY_VISIBILITY
+      end
+    end
+  end
+
+  def shutdown_instance
+    # Shutting down thread pools does not happen immediately;
+    # all jobs are allowed to finished before thread is closed.
+    #TODO make sure to wait for shutdown
+    @thread_pool.shutdown
+    @logger.shutdown
+  end
+
+  def send_message(text)
+    @queue.send_message(text)
+  end
+
+  private
+  def create_or_initialize_queue(sqs_url)
+    queue = AWS::SQS::Queue.new(sqs_url)
+
+    # TODO should probably not create queues dynamically in prod
+    # make this configurable
+    unless queue.exists?
+      # raise "Queue does not exist" unless queue.exists?
+      queue_name = sqs_url.split("/")[-1]
+      AWS::SQS.new.queues.create(queue_name)
+    end
+
+    queue
+  end
+end

data/lib/piscina/piscina_logger.rb

@@ -0,0 +1,75 @@
+java_import java.util.concurrent.Executors
+
+class PiscinaLogger
+  class << self
+    attr_accessor :configuration
+  end
+
+  def initialize(log_name)
+    # Create a buffered pool of size one that will handle writing to the logs
+    @thread_pool = Executors.newFixedThreadPool(1)
+
+    # Creates a standard Ruby Logger
+    @logger = PiscinaLogger.create_logger(log_name)
+  end
+
+  def self.create_logger(log_name)
+    log_path = construct_path_to_log(log_name)
+    logger = Logger.new(log_path, 'daily')
+
+    logger.formatter = proc do |severity, datetime, progname, msg|
+       "#{severity} [#{datetime.strftime('%Y-%m-%d %H:%M:%S.%L')}]: #{msg}\n"
+    end
+
+    logger.level = PiscinaLogger.configuration.logging_level
+
+    logger
+  end
+
+  def self.construct_path_to_log(log_name)
+    raise "No logging directory defined" unless PiscinaLogger.configuration.log_directory
+
+    log_dir = PiscinaLogger.configuration.log_directory
+    File.join(log_dir, "#{log_name}_piscina.log")
+  end
+
+  def write(message, level)
+    @thread_pool.execute do
+      @logger.send(level, message)
+    end
+  end
+
+  [:debug, :info, :warn, :error, :fatal, :unknown].each do |level|
+    define_method("#{level.to_s}") do |message|
+      write(message, level)
+    end
+  end
+
+  def shutdown
+    # TODO wait for pool to close down before closing logger
+    @thread_pool.shutdown
+    @logger.close
+  end
+
+  # Configure!
+  def self.configure
+    self.configuration ||= Configuration.new
+    yield(configuration)
+  end
+
+  class Configuration
+    attr_accessor :log_directory
+    attr_accessor :logging_level
+
+    def initialize
+      @logging_level = Logger::INFO
+
+      # Assume a rails app
+      @log_directory = if defined?(Rails)
+                         File.join(Rails.root.to_s, "log")
+                       else
+                         nil
+                       end
+    end
+  end
+end

metadata

@@ -0,0 +1,80 @@
+--- !ruby/object:Gem::Specification
+name: piscina
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Joe Feeney
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-06-14 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.44'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.44'
+- !ruby/object:Gem::Dependency
+  name: fake_sqs
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+description: 'Piscina is a threaded SQS consumer for JRuby. Each Piscina instance
+  is associated with a Klass that knows how to #perform(msg). Piscina can also handle
+  multi threaded logging to a single file through PiscinaLogger.'
+email:
+- joe@glasswaves.co
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- lib/piscina.rb
+- lib/piscina/http_job.rb
+- lib/piscina/piscina.rb
+- lib/piscina/piscina_logger.rb
+homepage: https://github.com/iamatypeofwalrus/Piscina
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.3.0
+signing_key: 
+specification_version: 4
+summary: JRuby SQS job processor/dispatcher and multi-threaded logger
+test_files: []