fifo 1.0.0

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2012 Siddharth Batra
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.textile

@@ -0,0 +1,234 @@
+h1. FIFO
+
+FIFO is a ruby queueing library built on top of Amazon SQS (Simple Queue Service). Like DelayedJob it encapsulates the common pattern of executing time consuming tasks in the background but unlike DelayedJob it doesn't rely on a database.
+
+h3. Features
+* Built on Amazon's reliable and scalable queue service.
+* Connection to SQS are opened lazily negating any initial load time.
+* Use multiple queues with ease.
+* Doesn't poll the database.
+* Rails ActiveRecord objects maintain state through the queue.
+* Built-in retry mechanism.
+
+h3. Uses
+
+FIFO is extracted from the Mine (http://getmine.com) codebase. Here are some of the things we use it for:
+
+* Sending emails
+* Processing images
+* Indexing 
+* Sharing to social networks
+* Cache management
+* Launching cron jobs
+* Communicate between disjoint parts of a complex application (cron, web, processing servers)
+
+h2. Installation
+
+<pre>
+Pending
+</pre>
+
+
+h2. Usage
+
+
+h3. Credentials
+
+Setup an aws access id and aws secret key. They can be found here: https://portal.aws.amazon.com/gp/aws/securityCredentials
+
+<pre>
+FIFO::QueueManager.setup <aws_access_id>, <aws_secret_key>
+</pre>
+
+
+h3. Initialization
+
+Create a queue object. If an queue by this name doesn't exist on the Amazon SQS account this will create one.
+
+For Rails, this can be added to a file in "config/initializers/".  Additionally, queue names can include the current Rails environment name for modularity.
+
+There is an additional parameter for message visibility. For more details see: http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/Welcome.html
+
+<pre>
+queue = FIFO::Queue.new <queue_name>
+</pre>
+
+To prevent delays at the start of the application SQS connections are opened lazily. This saves the overheard when initialzing multiple queues:
+
+<pre>
+queues = []
+(1..10).each do |i|
+  queues << FIFO::Queue.new <queue_name_i>
+end
+</pre>
+
+h3. Insertion
+
+The most basic case takes a Ruby object, a method to be called and a list of method parameters. Example of a ruby command and how to run it in the background:
+
+<pre>
+"World".insert 0,"Hello "
+
+queue.push "World",:insert,0,"Hello "
+</pre>
+
+
+FIFO is designed to work with ActiveRecord objects. Freshness of ActiveRecord objects is ensured by reloading them from the database when they're popped from the queue. Consider an instance method to process images:
+
+<pre>
+user.process_profile_image {:max_width => 200,:max_height => 150}
+
+queue.push user,:process_profile_image,{:max_width => 200,:max_height => 150}
+</pre>
+
+
+FIFO also works with ActiveRecord class methods. For example, sending newsletters:
+
+<pre>
+UserMailer.deliver_newsletter emails
+
+queue.push UserMailer,:deliver_newsletter,emails
+</pre>
+
+
+h3. Processing
+
+Every item pushed into the queue is converted into a FIFO::Payload object which executes the passed method.
+
+Basic flow of processing an item from the queue:
+
+<pre>
+queue.push "Hello World",:length
+payload = queue.pop
+payload.process
+=> 11
+</pre>
+
+Retry functionality takes a failed payload and adds it back into the queue. Payload objects maintain total number of processing attempts.
+
+<pre>
+queue.push "Hello World",:length
+
+begin
+  payload = queue.pop
+  payload.process
+rescue => ex
+  payload.retry if payload.attempts < 3
+end
+</pre>
+
+
+Process all items in the queue:
+
+<pre>
+while(true)
+  payload = queue.pop
+  break unless payload
+  
+  begin
+    payload.process
+  rescue => ex
+    payload.retry if payload.attempts < 3
+  end
+end
+</pre>
+
+Multiple queues can be used to simulate priority:
+
+<pre>
+while(true)
+  payload = queue.pop
+  payload = queue1.pop unless payload
+  payload = queue2.pop unless payload
+  
+  begin
+    payload.process
+  rescue => ex
+    payload.retry if payload.attempts < 3
+  end
+
+  sleep 10
+end
+</pre>
+
+
+The Payload flow can be entirely skipped by popping the raw response from the queue:
+
+<pre>
+queue.pop :raw => true
+</pre>
+
+
+h4. Daemon
+
+h5. Installation
+
+<pre>
+sudo gem install daemons
+Install daemon_generator from: https://github.com/dougal/daemon_generator
+
+./script/generate daemon <name>
+</pre>
+
+h5. Source
+
+<pre>
+require File.dirname(__FILE__) + "/../../config/environment"
+
+$running = true
+Signal.trap("TERM") do 
+  $running = false
+end
+
+FIFO::QueueManager.setup <aws_access_id>, <aws_secret_key>
+
+queue = FIFO::Queue.new <queue_name> 
+logger = Logger.new(File.join(RAILS_ROOT,"log/<name>.rb.log"))
+
+
+while($running) do
+  payload = queue.pop
+
+  if payload
+
+    begin
+      start_time = Time.now
+      payload.process
+      end_time = Time.now
+
+      logger.info "Finished #{payload.to_s} #{end_time - start_time}"
+
+    rescue => ex
+      if payload.attempts < 3
+        logger.info "Recovering #{payload.to_s}"
+        payload.retry
+      else
+        #Log Exception
+      end
+    end
+
+  end 
+  
+  sleep 5
+end
+</pre>
+
+To control the state of the daemon:
+
+<pre>
+lib/daemon/<name>_ctl <start|stop|run>
+</pre>
+
+h2. SQS
+
+Things to know about Amazon's SQS:
+
+* The order of items isn't always maintained. It's not 100% fifo but close enough.
+* There are multiple connection modes to SQS: per_request, per_thread, single
+* http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/Welcome.html
+
+
+h2. Credit
+
+FIFO is built by the wonderful people who make Mine: http://getmine.com
+

data/Rakefile

@@ -0,0 +1,70 @@
+# encoding: utf-8
+
+require 'rubygems'
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "fifo"
+  gem.homepage = "http://github.com/denwen/fifo"
+  gem.license = "MIT"
+  gem.summary = %Q{Queueing library in Ruby built on top of Amazon SQS (Simple Queue Service).}
+  gem.description = %Q{Queueing library in Ruby built on top of Amazon SQS(Simple Queue Service) for processing background tasks and enabling communication between parts of a complex application.}
+  gem.email = "siddharthabatra@gmail.com"
+  gem.authors = ["Siddharth Batra"]
+  gem.add_runtime_dependency 'aws', '>= 2.5.6'
+  # Include your dependencies below. Runtime dependencies are required when using your gem,
+  # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
+  #  gem.add_runtime_dependency 'jabber4r', '> 0.1'
+  #  gem.add_development_dependency 'rspec', '> 1.2.3'
+  gem.add_development_dependency "shoulda", ">= 0"
+  # Include your dependencies below. Runtime dependencies are required when using your gem,
+  # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
+  #  gem.add_runtime_dependency 'jabber4r', '> 0.1'
+  #  gem.add_development_dependency 'rspec', '> 1.2.3'
+  gem.add_development_dependency "rdoc", "~> 3.12"
+  # Include your dependencies below. Runtime dependencies are required when using your gem,
+  # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
+  #  gem.add_runtime_dependency 'jabber4r', '> 0.1'
+  #  gem.add_development_dependency 'rspec', '> 1.2.3'
+  gem.add_development_dependency "bundler", "~> 1.0.0"
+  # Include your dependencies below. Runtime dependencies are required when using your gem,
+  # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
+  #  gem.add_runtime_dependency 'jabber4r', '> 0.1'
+  #  gem.add_development_dependency 'rspec', '> 1.2.3'
+  gem.add_development_dependency "jeweler", "~> 1.8.3"
+  # Include your dependencies below. Runtime dependencies are required when using your gem,
+  # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
+  #  gem.add_runtime_dependency 'jabber4r', '> 0.1'
+  #  gem.add_development_dependency 'rspec', '> 1.2.3'
+  gem.add_development_dependency "rcov", ">= 0"
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+require 'rcov/rcovtask'
+Rcov::RcovTask.new do |test|
+  test.libs << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+  test.rcov_opts << '--exclude "gems/*"'
+end
+
+task :default => :test
+
+require 'rdoc/task'
+Rake::RDocTask.new do |rdoc|
+  version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "fifo #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION

@@ -0,0 +1 @@
+1.0.0

data/lib/fifo.rb

@@ -0,0 +1,8 @@
+require 'yaml'
+require 'aws'
+require File.dirname(__FILE__) + '/fifo/payload'
+require File.dirname(__FILE__) + '/fifo/queue'
+require File.dirname(__FILE__) + '/fifo/queue_manager'
+
+module FIFO
+end

data/lib/fifo/payload.rb

@@ -0,0 +1,91 @@
+module FIFO
+
+  class Payload
+    attr_accessor :queue
+    attr_reader :attempts
+
+    # Public. Create a payload with an object a method of that
+    # object and optional arguments for that method.
+    #
+    # object - The Object the payload carries.
+    # method - The Symbol method name for the object.
+    # args - The optional array of arguments for the method.
+    #
+    # Returns nothing.
+    # Raises ArgumentError if object or method aren't present.
+    def initialize(object,method,*args)
+      raise ArgumentError, "object and method required" unless object && method
+
+      @attempts = 1
+      @object = object.is_a?(Class) ? 
+                  object.to_s.split("::").last :
+                  object.to_yaml 
+      @method = method
+      @arguments = args
+    end
+
+    # Public. Call the method on the carried object with
+    # the carried arguments. If the carried object is an
+    # ActiveRecord::Base object, reload it from the db
+    # to get a fresh state.
+    #
+    # Returns the result of the method call on the object.
+    def process
+      if is_object_yaml?
+        object = YAML.load(@object)
+
+        if defined?(ActiveRecord) && object.is_a?(ActiveRecord::Base)
+          object = object.class.find object.id 
+        end
+      else
+        object = Object.const_get(@object)
+      end
+
+      object.send(@method.to_sym,*@arguments)
+    end
+
+    # Public. Mark the instance as failed by increaseing
+    # the number of attempts to process it.
+    #
+    # Returns the Integer number of attempts used.
+    def failed
+      @attempts += 1
+    end
+
+    # Public. Add the payload back onto the queue.
+    #
+    def retry
+      failed
+      queue.push self
+    end
+
+    # Public. Override to_s for a loggable output.
+    #
+    # Returns the String form of the Payload.
+    def to_s
+      output = ""
+
+      if is_object_yaml?
+        object = YAML.load(@object)
+        output << object.class.name 
+      else
+        output << Object.const_get(@object).name
+      end
+
+      output << " :#{@method} #{@arguments.join("|")}"
+      output
+    end
+
+
+    private 
+
+    # Private. Test if the object is a ruby object in YAML.
+    #
+    # Returns the Boolean value of the test.
+    def is_object_yaml?
+      @object.start_with? "--- "
+    end
+
+  end
+
+end

data/lib/fifo/queue.rb

@@ -0,0 +1,79 @@
+module FIFO
+
+  class Queue
+
+    # Public. Constructor logic.
+    #
+    # name - The String name of the queue.
+    # visibility - The Integer visibility of an item in the queue.
+    #
+    # Returns nothing.
+    def initialize(name,visibility=90)
+      @name = name
+      @visibility = visibility
+    end
+
+    # Public. Pop the first entry in the queue and create a payload
+    # object from it.
+    #
+    # options - The Hash of options.
+    #   :raw - Return raw queue value instead of a payload.
+    #
+    # Returns the Payload object if found or nil.
+    def pop(options={})
+      message = queue.pop
+
+      if message
+        if options[:raw]
+          payload = message.body
+        else
+          payload = YAML.load(message.body) 
+          payload.queue = self
+        end
+      end
+
+      payload
+    end
+
+    # Push a new entry onto the queue. 
+    #
+    # args - This can either be a Payload object or a Class or any ruby object
+    # including ActiveRecord::Base objects followed by a symbol for a method and optional
+    # method arguments. During processing ActiveRecord::Base
+    # objects are fetched again from the DB to avoid staleness.
+    #
+    # Returns nothing.
+    def push(*args)
+      payload = nil
+
+      if args.first.class == Payload
+        payload = args.first
+        payload.queue = nil
+      else
+        payload = Payload.new(*args)
+      end
+
+      queue.push(payload.to_yaml)
+    end
+
+    # Public: Clear all entries in the queue.
+    #
+    # Returns nothing.
+    def clear
+      queue.clear
+    end
+
+
+    private
+    
+    # Private. Getter method for the queue. Enables opening
+    # lazy connections.
+    #
+    # Returns the Aws::Sqs::Queue object the QueueManager creates.
+    def queue
+      @queue ||= QueueManager.fetch(@name,@visibility)
+    end
+
+  end
+
+end

data/lib/fifo/queue_manager.rb

@@ -0,0 +1,42 @@
+module FIFO
+
+  class QueueManager
+
+    # Public. Setup AWS credentials and connection_mode for the service.
+    # 
+    # aws_access_id - The String AWS access id.
+    # aws_secret_key - The String AWS secret key.
+    # connection_mode - The Symbol for threading options - 
+    #                   :single,:per_thread,:per_request. 
+    #                   See: https://github.com/appoxy/aws/ (default: :single)
+    #
+    def self.setup(aws_access_id,aws_secret_key,connection_mode=:single)
+      @aws_access_id = aws_access_id
+      @aws_secret_key = aws_secret_key
+      @connection_mode = connection_mode
+    end
+      
+    # Public. Open a connection and fetch an Aws::Sqs::Queue.
+    #
+    # Returns the Aws::Sqs::Queue.
+    def self.fetch(name,visibility)
+      Aws::Sqs::Queue.create(sqs,name,true,visibility)
+    end
+
+
+    private
+
+    # Private: Getter method for Aws::Sqs object. Enables
+    # lazily creating an SqsInterface.
+    #
+    # Returns the Aws::Sqs.
+    def self.sqs
+      @sqs ||= Aws::Sqs.new(
+                @aws_access_id,
+                @aws_secret_key,
+                {:connection_mode => @connection_mode})
+    end
+
+  end
+
+end

data/test/helper.rb

@@ -0,0 +1,18 @@
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'test/unit'
+require 'shoulda'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'fifo'
+
+class Test::Unit::TestCase
+end

data/test/test_fifo.rb

@@ -0,0 +1,7 @@
+require 'helper'
+
+class TestFifo < Test::Unit::TestCase
+  should "probably rename this file and start testing for real" do
+    flunk "hey buddy, you should probably rename this file and start testing for real"
+  end
+end

metadata

@@ -0,0 +1,168 @@
+--- !ruby/object:Gem::Specification 
+name: fifo
+version: !ruby/object:Gem::Version 
+  hash: 23
+  prerelease: 
+  segments: 
+  - 1
+  - 0
+  - 0
+  version: 1.0.0
+platform: ruby
+authors: 
+- Siddharth Batra
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2012-10-17 00:00:00 +00:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: aws
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 23
+        segments: 
+        - 2
+        - 5
+        - 6
+        version: 2.5.6
+  type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: shoulda
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: rdoc
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 31
+        segments: 
+        - 3
+        - 12
+        version: "3.12"
+  type: :development
+  version_requirements: *id003
+- !ruby/object:Gem::Dependency 
+  name: bundler
+  prerelease: false
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 23
+        segments: 
+        - 1
+        - 0
+        - 0
+        version: 1.0.0
+  type: :development
+  version_requirements: *id004
+- !ruby/object:Gem::Dependency 
+  name: jeweler
+  prerelease: false
+  requirement: &id005 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 49
+        segments: 
+        - 1
+        - 8
+        - 3
+        version: 1.8.3
+  type: :development
+  version_requirements: *id005
+- !ruby/object:Gem::Dependency 
+  name: rcov
+  prerelease: false
+  requirement: &id006 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :development
+  version_requirements: *id006
+description: Queueing library in Ruby built on top of Amazon SQS(Simple Queue Service) for processing background tasks and enabling communication between parts of a complex application.
+email: siddharthabatra@gmail.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE.txt
+- README.textile
+files: 
+- .document
+- LICENSE.txt
+- README.textile
+- Rakefile
+- VERSION
+- lib/fifo.rb
+- lib/fifo/payload.rb
+- lib/fifo/queue.rb
+- lib/fifo/queue_manager.rb
+- test/helper.rb
+- test/test_fifo.rb
+has_rdoc: true
+homepage: http://github.com/denwen/fifo
+licenses: 
+- MIT
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.5.2
+signing_key: 
+specification_version: 3
+summary: Queueing library in Ruby built on top of Amazon SQS (Simple Queue Service).
+test_files: []
+