kiqit 0.0.1 → 1.3.0

data/.gitignore

@@ -1,4 +1,8 @@
+.rvmrc
 *.gem
 .bundle
 Gemfile.lock
 pkg/*
+.tags
+.tags_sorted_by_file
+coverage/*

data/.rspec

@@ -0,0 +1 @@
+--colour

data/.travis.yml

@@ -0,0 +1,3 @@
+language: ruby
+rvm:
+  - 1.9.3

data/README.md

@@ -0,0 +1,193 @@
+[![Build Status](https://secure.travis-ci.org/KensoDev/kiqit.png)](https://secure.travis-ci.org/KensoDev/kiqit)
+
+## Overview
+Kiqit is a gem meant to work with the [Sidekiq](http://github.com/mperham/sidekiq) queue system. It was adapted from the Perform Later gem.
+
+Usually, when working with Sidekiq, you need separate "Worker" classes and you also need to do `Sidekiq.enqueue` whenever you want to add a task to the queue.
+
+That can be a real hassle if you are adding Sidekiq to an existing project, it can also add quite a bit of code to your system.
+
+`kiqit` fills this need, it offers a suite to handle all of your queuing needs, both for Objects and for ActiveRecord models.
+
+## Why?
+*Why* should you queue something for later?
+
+You should queue something whenever the method handles some heavy lifting, some timely actions like API, 3rd party HTTP requests and more.
+
+The basic logic is that whatever you don't need to do NOW, you should do later, this will make your site faster and the users will feel it.
+
+## Real life use case
+At [Gogobot](http://gogobot.com) whenever you post a review, there's major score calculation going on. This can sometimes take up to a minute, depending on the user graph.
+
+The user should not wait for this on submit, it can be queued into later execution.
+
+## Installation
+gem install kiqit
+
+If you are using bundler, simply add
+`gem "kiqit"` to your Gemfile
+
+
+## Configuration
+In an initializer, all you need to say is whether you want perform later to be enabled or not, typically, it will be something like this
+
+```ruby
+unless Rails.env.test?
+  Kiqit.config.enabled = true # this will default to false if unset
+end
+```
+
+## Usage
+
+### ActiveRecord
+
+`kiqit` comes with a special method you can use on ActiveRecord models.
+
+
+```ruby
+
+	class User < ActiveRecord::Base
+	  def long_running_method
+	    # Your code here
+	  end
+	  later :long_running_method
+	
+	  def long_running_method_2
+	    # Your code here
+	  end
+	  later :long_running_method_2, queue: :some_queue_name
+	
+	  def lonely_long_running_method
+	    # Your code here
+	  end
+	  later :lonely_long_running_method, :loner => true, queue: :some_queue_name
+
+    def delayed_long_running_method
+      # Your code here
+    end
+    later :delayed_long_running_method, :delay => 30, queue: :some_queue_name
+	end
+	
+```
+
+```ruby
+	u = User.find(some_user_id)
+	u.long_running_method # Method will be queued into the :generic queue
+	u.long_running_method_2 # Method will be queued into :some_queue_name queue
+	u.lonely_long_running_method # Method will be queued into the :some_queue_name queue, only a single instance of this method can exist in the queue.
+  u.delayed_long_running_method # Method will be queued into :some_queue_name queue only after 30 seconds have passed.
+```
+
+You can of course choose to run the method off the queue, just prepend `now_` to the method name and it will be executed in sync.
+
+```ruby
+	u = User.find(some_user_id)
+	u.now_long_running_method
+```
+
+### Objects/Classes
+
+If you want class methods to be queued, you will have to use the `kiqit` special method.
+
+```ruby
+	class SomeClass
+		def self.some_heavy_lifting_method
+	  	  # Your code here
+	  	end
+	  	
+		def self.some_more_heavy_lifting(user_id)
+	  	  # Your code here
+	  	end  	
+	end
+	
+	SomeClass.kiqit(:queue_name, :some_heavy_lifting_method)
+	SomeClass.kiqit(:queue_name, :some_more_heavy_lifting, user_id)
+	
+
+```
+
+If you want the method to be a loner (only a single instance in the queue), you will need to use the `kiqit!` method.
+
+```ruby
+	SomeClass.kiqit!(:queue_name, :some_more_heavy_lifting, user_id)
+```
+
+## The params parser
+`kiqit` has a special class called `ArgsParser`, this class is in charge of *translating* the args you are passing into params that can actually be serialized to JSON cleanly.
+
+Examples:
+
+```ruby
+	user = User.find(1)
+	Kiqit::ArgsParser.params_to_sidekiq(user) => 'AR:#User:1'
+	
+	hotel = Hotel.find(1)
+	Kiqit::ArgsParser.params_to_sidekiq(hotel) => 'AR:#Hotel:1'
+	
+	hash = { name: "something", other: "something else" }
+	Kiqit::ArgsParser.params_to_sidekiq(hash) 
+	=> ---
+		:name: something
+		:other: something else
+	# Hashes are translated into YAML
+```
+
+Basically, the `ArgsParser` class allows you to keep passing any args you want to your methods without worrying about whether they serialize cleanly or not.
+
+`ArgsParser` also patched `sidekiq-mailer` so you can pass in AR objects to mailers as well.
+
+## The custom finder
+I found the need to add a custom finder to the args parser.
+
+### Why?
+At Gogobot for example, we use slave databases, those sometimes have lag, so when the finder is executed it returns nil, even though the record is actually on the master.
+
+So, I added support for custom finders.
+
+#### Example:
+
+```ruby
+	class CustomFinder
+		def self.find(klass, id)
+			Octopus.using(:master) {
+				klass.where(id: id).first
+			} unless klass.where(id: id).first
+		end
+	end
+```
+
+Then in an initializer
+
+```ruby
+	Kiqit::Plugins.add_finder(CustomFinder)
+```
+
+You can also remove the finder in runtime
+
+```ruby
+	Kiqit::Plugins.clear_finder!
+```
+
+So, at Gogobot for example, we will fall back to master if the record was not found on the slave DB.
+
+ 
+## Contribute / Bug reports
+
+If you have an issue with this gem, please open an issue in the main repo, it will help tons if you could supply a failing spec with that, so I can better track where the bug is coming from, if not, no worries, just report I will do my best to address it as fast and efficient as I can.
+
+If you want to contribute (awesome), open a feature branch, base it on master.
+
+Be as descriptive as you can in the pull request description, just to be clear what problem you are solving or what feature are you adding.
+
+## Authors
+
+Avi Tzurel ([@kensodev](http://twitter.com/kensodev)) http://www.kensodev.com
+Tom Caspy
+
+## Contributors
+
+* Felipe Lima ([@felipecsl](http://twitter.com/felipecsl)) 
+http://blog.felipel.com/
+
+Felipe did awesome work on making sure `kiqit` can work with any args and any number of args passed into the methods.
+Felipe now has commit rights to the repo.

data/Rakefile

@@ -1 +1,9 @@
-require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+desc 'Default: run specs.'
+task :default => :spec
+
+desc "Run specs"
+RSpec::Core::RakeTask.new do |t|
+  
+end

data/init.rb

@@ -0,0 +1,2 @@
+# init
+require 'kiqit'

data/kiqit.gemspec

@@ -5,11 +5,11 @@ require "kiqit/version"
 Gem::Specification.new do |s|
   s.name        = "kiqit"
   s.version     = Kiqit::VERSION
-  s.authors     = ["Avi Tzurel"]
-  s.email       = ["avi@kensodev.com"]
-  s.homepage    = "http://www.kensodev.com"
-  s.summary     = %q{your SideKiq helper}
-  s.description = %q{Kiqit is a set of helpers to use with the SideKiq gem, like an async method and more}
+  s.authors     = ["Avi Tzurel", "Tom Caspy"]
+  s.email       = ["avi@kensodev.com", "tom@kensodev.com"]
+  s.homepage    = "http://www.github.com/kensodev/kiqit"
+  s.summary     = %q{Queue any method in any class or instance with no need for additional Worker class and no extra code}
+  s.description = %q{Queue any method in any class or instance with no need for additional Worker class and no extra code}
 
   s.rubyforge_project = "kiqit"
 
@@ -18,11 +18,12 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
 
-  s.add_development_dependency 'rails', '~> 3.0'
+  s.add_dependency 'sidekiq'
+  s.add_dependency 'activerecord'
 
-  s.add_development_dependency 'sidekiq', '~> 2.2.0'
-  s.add_development_dependency 'rspec'
+  s.add_development_dependency 'rake'
   s.add_development_dependency 'rspec-rails'
-  s.add_development_dependency 'simplecov'
-  s.add_development_dependency 'redis', '~> 2.2.0'
-end
+  s.add_development_dependency 'rspec'
+  s.add_development_dependency 'sqlite3'
+  s.add_development_dependency 'fakeredis'
+end

data/lib/kiqit/args_parser.rb

@@ -0,0 +1,56 @@
+require 'json'
+
+module Kiqit
+  class ArgsParser
+    # inspired by DelayedJob
+    CLASS_STRING_FORMAT = /^CLASS\:([A-Z][\w\:]+)$/
+    AR_STRING_FORMAT    = /^AR\:([A-Z][\w\:]+)\:(\d+)$/
+    YAML_STRING_FORMAT  = /\A---/
+
+    def self.args_to_sidekiq(args)
+      return nil unless args
+      return arg_to_sidekiq(args) unless args.is_a?(Array)
+      return args.map { |o| arg_to_sidekiq o }
+    end
+    
+    def self.args_from_sidekiq(args)
+      args = args.map { |o|
+        if o
+          o = args_from_sidekiq(o) if o.is_a?(Array)
+          case o
+            when CLASS_STRING_FORMAT
+              $1.constantize
+            when AR_STRING_FORMAT
+              runner_class = $1.constantize
+              id = $2
+
+              if Kiqit::Plugins.finder_class
+                Kiqit::Plugins.finder_class.find(runner_class, id)
+              else
+                runner_class.where(id: id).first  
+              end
+            when YAML_STRING_FORMAT
+              YAML.load(o)
+            else 
+              o
+          end
+        end
+      } if args
+    end
+
+  private
+
+    def self.arg_to_sidekiq(arg)
+      case arg
+      when ActiveRecord::Base
+        "AR:#{arg.class.name}:#{arg.id}"
+      when Class, Module
+        "CLASS:#{arg.name}"
+      when Hash
+        arg.to_yaml
+      else
+        arg
+      end
+    end
+  end
+end

data/lib/kiqit/config.rb

@@ -0,0 +1,11 @@
+module Kiqit
+  class Config
+    def self.enabled=(value)
+      @enabled = value
+    end
+
+    def self.enabled?
+      !!@enabled
+    end
+  end
+end

data/lib/kiqit/job_creator.rb

@@ -0,0 +1,28 @@
+module Kiqit
+  class JobCreator
+    
+    attr_reader :queue, :worker, :klass_name, :id, :method
+    attr_accessor :args
+
+    def initialize(queue, worker, klass_name, id, method, *args)
+      @queue       = queue
+      @worker      = worker
+      @klass_name  = klass_name
+      @id          = id
+      @method      = method
+      @args        = args
+    end
+
+    def enqueue(delay=nil)
+      params = {}
+      if delay
+        delay = delay.is_a?(Hash) ? delay[:delay] : delay
+        params["at"] = (Time.now + delay).to_i
+      end
+
+      params.merge!({"queue" => queue, "class" => worker, "args" => [klass_name, id, method, *args]})
+
+      Sidekiq::Client.push(params)
+    end
+  end
+end

data/lib/kiqit/payload_helper.rb

@@ -0,0 +1,20 @@
+module Kiqit
+  class PayloadHelper
+    # Public: Get Digest for the loner key.
+    #
+    # klass - Class name.
+    # method - Method name.
+    # Args - Args to send on the method
+    #
+    # Examples => 
+    #   PayloadHelper.get_digest("SomeClass", "some_method", "arg1", "arg2")
+    # 
+    def self.get_digest(klass, method, *args)
+      args    = args.flatten
+      payload = { :class => klass, :method => method.to_s, :args => args }.to_s
+      digest  = Digest::MD5.hexdigest("#{payload}")
+
+      "loner:#{digest}"
+    end
+  end
+end

data/lib/kiqit/plugins.rb

@@ -0,0 +1,17 @@
+module Kiqit
+  class Plugins
+    def self.finder_class
+      @@finder_class ||= nil
+    end
+
+    def self.add_finder(klass)
+      if klass.respond_to?(:find)
+        @@finder_class = klass
+      end
+    end
+
+    def self.clear_finder!
+      @@finder_class = nil
+    end
+  end
+end

data/lib/kiqit/railtie.rb

@@ -0,0 +1,5 @@
+module Kiqit
+  class Railtie < ::Rails::Railtie
+    config.perf_later = Kiqit::Config
+  end
+end

data/lib/kiqit/version.rb

@@ -1,3 +1,3 @@
 module Kiqit
-  VERSION = "0.0.1"
+  VERSION = "1.3.0"
 end

data/lib/kiqit/workers/active_record/lone_worker.rb

@@ -0,0 +1,19 @@
+module Kiqit
+  module Workers
+    module ActiveRecord
+      class LoneWorker < Kiqit::Workers::Base
+        def perform(klass, id, method, *args)
+          # Remove the loner flag from redis
+          digest       = Kiqit::PayloadHelper.get_digest(klass, method, args)
+          Sidekiq.redis.del(digest)
+          
+          args         = Kiqit::ArgsParser.args_from_sidekiq(args)
+          runner_klass = klass.constantize
+          record       = runner_klass.find(id)
+
+          perform_job(record, method, args)
+        end
+      end
+    end
+  end
+end

data/lib/kiqit/workers/active_record/worker.rb

@@ -0,0 +1,15 @@
+module Kiqit
+  module Workers
+    module ActiveRecord
+      class Worker < Kiqit::Workers::Base
+        def perform(klass, id, method, *args)
+          args         = Kiqit::ArgsParser.args_from_sidekiq(args)
+          runner_klass = klass.constantize
+          record       = runner_klass.find(id)
+          
+          perform_job(record, method, args)
+        end
+      end
+    end
+  end
+end

data/lib/kiqit/workers/base.rb

@@ -0,0 +1,21 @@
+module Kiqit
+  module Workers
+		class Base
+
+      include Sidekiq::Worker
+      
+  		protected
+  			def perform_job(object, method, arguments)
+  				unless arguments.empty?
+            if arguments.size == 1
+              object.send(method, arguments.first)
+            else
+              object.send(method, *arguments)
+            end
+          else
+            object.send(method)
+          end
+  			end
+  		end
+	end
+end

data/lib/kiqit/workers/objects/lone_worker.rb

@@ -0,0 +1,16 @@
+module Kiqit
+  module Workers
+    module Objects
+      class LoneWorker < Kiqit::Workers::Base
+        def perform(klass_name, method, *args)
+          digest = Kiqit::PayloadHelper.get_digest(klass_name, method, args)
+          Sidekiq.redis.del(digest)
+
+          arguments = Kiqit::ArgsParser.args_from_sidekiq(args)
+          
+          perform_job(klass_name.constantize, method, arguments)
+        end
+      end
+    end
+  end
+end

data/lib/kiqit/workers/objects/worker.rb

@@ -0,0 +1,13 @@
+module Kiqit
+  module Workers
+    module Objects
+      class Worker < Kiqit::Workers::Base
+        def perform(klass_name, method, *args)
+          arguments = Kiqit::ArgsParser.args_from_sidekiq(args)
+
+          perform_job(klass_name.constantize, method, arguments)
+        end
+      end
+    end
+  end
+end

data/lib/kiqit.rb

@@ -1,5 +1,33 @@
-require "kiqit/version"
+require 'sidekiq'
+require 'active_support/dependencies'
+require 'kiqit/version'
+require 'kiqit/config'
+require 'kiqit/payload_helper'
+require 'kiqit/args_parser'
+require 'kiqit/plugins'
+require 'kiqit/job_creator'
+require 'active_record'
+require 'object_perform_later'
+require 'kiqit/workers/base'
+require 'kiqit/workers/active_record/worker'
+require 'kiqit/workers/active_record/lone_worker'
+require 'kiqit/workers/objects/worker'
+require 'kiqit/workers/objects/lone_worker'
 
 module Kiqit
-  # Your code goes here...
+  def self.config
+    Kiqit::Config
+  end
+end
+
+module Sidekiq
+  module Plugins
+    module Later
+      autoload :Method, 'sidekiq/plugins/later/method'
+    end
+  end
+end
+
+ActiveSupport.on_load(:active_record) do
+  include Sidekiq::Plugins::Later::Method
 end

data/lib/object_perform_later.rb

@@ -0,0 +1,36 @@
+module ObjectKiqit
+  def kiqit(queue, method, *args)
+    return perform_now(method, args) unless Kiqit.config.enabled?
+
+    worker = Kiqit::Workers::Objects::Worker
+    kiqit_enqueue(worker, queue, method, args)
+  end
+
+  def kiqit!(queue, method, *args)
+    return perform_now(method, args) unless Kiqit.config.enabled?
+
+    return "EXISTS!" if loner_exists(method, args)
+
+    worker = Kiqit::Workers::Objects::LoneWorker
+    kiqit_enqueue(worker, queue, method, args)
+  end
+
+  private 
+    def loner_exists(method, *args)
+      digest = Kiqit::PayloadHelper.get_digest(self.name, method, args)
+
+      !Sidekiq.redis{|i| i.setnx(digest, 'EXISTS')}
+    end
+
+    def kiqit_enqueue(worker, queue, method, args)
+      args = Kiqit::ArgsParser.args_to_sidekiq(args)
+      params = {"queue" => queue, "class" => worker, "args" => [self.name, method, *args]}
+      Sidekiq::Client.push(params)
+    end
+
+    def perform_now(method, args)
+      args.size == 1 ? send(method, args.first) : send(method, *args)
+    end
+end
+
+Object.send(:include, ObjectKiqit)

data/lib/resque_mailer_patch.rb

@@ -0,0 +1,17 @@
+module Sidekiq
+  module Mailer
+    module ClassMethods
+      def perform(action, *args)
+        args = Kiqit::ArgsParser.args_from_sidekiq(args)
+        self.send(:new, action, *args).message.deliver
+      end
+    end
+
+    class MessageDecoy
+      def deliver
+        args = Kiqit::ArgsParser.args_to_sidekiq(@args)
+        sidekiq.enqueue(@mailer_class, @method_name, *args)
+      end
+    end
+  end
+end

data/lib/sidekiq/plugins/later/method.rb

@@ -0,0 +1,88 @@
+module Sidekiq::Plugins::Later::Method
+  extend ActiveSupport::Concern
+
+  module ClassMethods
+    def later(method_name, opts={})
+      alias_method "now_#{method_name}", method_name
+      return unless Kiqit.config.enabled?
+
+      define_method "#{method_name}" do |*args|
+        loner          = opts.fetch(:loner, false)
+        queue          = opts.fetch(:queue, :generic)
+        delay          = opts.fetch(:delay, false)
+        klass          = Kiqit::Workers::ActiveRecord::Worker
+        klass          = Kiqit::Workers::ActiveRecord::LoneWorker if loner
+        args           = Kiqit::ArgsParser.args_to_sidekiq(args)
+        digest         = Kiqit::PayloadHelper.get_digest(klass, method_name, args)
+
+        if loner
+          return "AR EXISTS!" if Sidekiq.redis{|i| i.get(digest).present?}
+          Sidekiq.redis{|i| i.set(digest, 'EXISTS')}
+        end
+
+        job = Kiqit::JobCreator.new(queue, klass, send(:class).name, send(:id), "now_#{method_name}", *args)
+        job.enqueue(delay)
+      end
+    end
+
+  end
+
+  def kiqit(queue, method, *args)
+    return perform_now(method, args) if plugin_disabled?
+
+    worker  = Kiqit::Workers::ActiveRecord::Worker
+    job     = Kiqit::JobCreator.new(queue, worker, self.class.name, self.id, method, *args) 
+    enqueue_in_sidekiq_or_send(job)
+  end
+
+  def kiqit!(queue, method, *args)
+    return perform_now(method, args) if plugin_disabled?
+    return "AR EXISTS!" if loner_exists(method, args)
+    
+    worker  = Kiqit::Workers::ActiveRecord::LoneWorker
+    job     = Kiqit::JobCreator.new(queue, worker, self.class.name, self.id, method, *args) 
+    enqueue_in_sidekiq_or_send(job)
+  end
+
+  def kiqit_in(delay, queue, method, *args)
+    return perform_now(method, args) if plugin_disabled?
+
+    worker  = Kiqit::Workers::ActiveRecord::Worker
+    job     = Kiqit::JobCreator.new(queue, worker, self.class.name, self.id, method, *args) 
+    enqueue_in_sidekiq_or_send(job, delay)
+  end
+  
+  def kiqit_in!(delay, queue, method, *args)
+    return  perform_now(method, args) if plugin_disabled?
+
+    worker  = Kiqit::Workers::ActiveRecord::LoneWorker
+    job     = Kiqit::JobCreator.new(queue, worker, self.class.name, self.id, method, *args) 
+    enqueue_in_sidekiq_or_send(job, delay)
+  end
+
+
+
+  private 
+    def loner_exists(method, args)
+      args = Kiqit::ArgsParser.args_to_sidekiq(args)
+      digest = Kiqit::PayloadHelper.get_digest(self.class.name, method, args)
+
+      return true unless Sidekiq.redis{ |i| i.get(digest).blank?}
+      Sidekiq.redis{|i| i.set(digest, 'EXISTS')}
+
+      return false
+    end
+
+    def enqueue_in_sidekiq_or_send(job, delay=nil)
+      job.args = Kiqit::ArgsParser.args_to_sidekiq(job.args)
+      job.enqueue(delay)
+    end
+        
+    def plugin_disabled?
+      !Kiqit.config.enabled?
+    end
+
+    def perform_now(method, args)
+      return self.send(method, *args)
+    end
+end