beanpicker 0.1.0

data/.rspec

@@ -0,0 +1,2 @@
+-c
+-f d

data/README.markdown

@@ -0,0 +1,186 @@
+# Beanpicker
+
+## What is it?
+
+Beanpicker is a job queueing DSL for Beanstalk
+
+## What? Beanstalk? It make coffe?
+
+[beanstalk(d)][beanstalk] is a fast, lightweight queueing backend inspired by memcached. The Ruby Beanstalk client is a bit raw, however, so Beanpicker provides a thin wrapper to make job queueing from your Ruby app easy and fun.
+
+## Is this similar to Stalker?
+
+Yes, it is inspired in [stalker][stalker] and in [Minion][minion]
+
+### Why should I use Beanpicker instead of Stalker?
+
+Beanpicker work with subprocess. It create a fork for every request and destroy it in the end.
+
+The vantages of Beanpicker are:
+
+*  Your job can use a large amount of RAM, it will be discarted when it end
+*  You can change vars in jobs, they will not be changed to other jobs nor the principal process
+*  If all your jobs need a large lib to be loaded(Rails?), you load it once in principal process and it will be available to all jobs without ocupping extra RAM
+*  It use YAML instead of JSON, so you can pass certain Ruby Objects(Time, Date, etc) //Please, don't try to pass a Rails Model or anything similar, pass only the ID so the job can avoid a possible outdated object
+
+### How is his performance?
+
+My machine:
+
+*  Notebook LG 590 5100
+*  Processor Intel Core i3 330M
+*  4GB RAM DDR3
+*  Arch Linux x86-64
+
+The speed with 10000 requests given 'with fork' is :fork => :every and 'without fork' is :fork => :master/false
+
+\# time / requests per second / cpu load
+
+*  MRI 1.9.2
+   *  With fork: 117.85 / 84.85 / 100%
+   *  Without fork: 4.14 / 2415 / 30%
+*  MRI 1.8.7
+   *  With fork: 122.27 / 81.78 / 58%
+   *  Without fork: 6.34 / 1577 / 30~40%
+*  REE 1.8.7
+   *  With fork: 121.92 / 82.02 / 20~60%
+   *  Without fork: 4.77 / 2096 / 30%
+* JRuby 1.5.6 + OpenJDK 6.2b0\_1.9.3 VM 1.6.0\_20
+   *  With fork: don't accept fork?
+   *  Without fork: 10.99 / 909.91 / 97%
+* Rubinius 1.2.0
+   *  With fork: don't try, too much errors
+   *  Without fork: 11.24 / 889 / 36~52%
+
+Fork is activated by default, it should slow down your application but keep safe from memory leaks.
+
+You can easy active or desactive the fork for a job with:
+
+    job "job.without.fork", :fork => false do |args|
+      debug "Running on a thread in main process"
+      warn "This process will grow because of any job running on main process"
+    end
+
+    job "job.with.fork.every.time", :fork => :every do |args|
+      debug "Running on a fork of main process"
+      debug "This process will be killed on end of this job"
+      debug "This decrease the peformance but save from memory leaks"
+      debug "All extra memory used by this process will vanish in end"
+    end
+
+    job "job.with.fork.once", :fork => :master do |args|
+      debug "Running on a fork of main process"
+      debug "This process will not be killed on end of this job"
+      debug "This increase the performance but don't save from memory leaks"
+      debug "This process will only grow in memory because of code executed in 'job.with.fork.once'"
+    end
+
+You can pass :fork\_every => true(default)/false and :fork\_master => true/false(default)
+
+The :fork argument overwrite :fork\_every and :fork\_master
+
+The default :fork\_every and :fork\_master are setted on Beanpicker::default\_fork\_[master|every]
+
+Beanpicker::fork\_every and Beanpicker::fork\_master overwrite the job options, so, if you set they false the jobs will run in the main thread even if they specify the :fork, :fork\_every and/or :fork\_master
+
+## Queueing jobs
+
+From anywhere in your app:
+
+    require 'beanpicker'
+
+    Beanpicker.enqueue('email.send', :to => 'joe@example.com')
+    Beanpicker.enqueue('post.cleanup.all')
+    Beanpicker.enqueue('post.cleanup', :id => post.id)
+
+### Chain jobs
+
+If you have a task that requires more than one step just pass an array of queues when you enqueue.
+
+    require 'beanpicker/job_server'
+
+    Beanpicker::Worker.new do
+      # this is a slow job, so we'll spawn 10 forks of it :)
+      job "email.fetch_attachments", :childs => 10 do |args|
+        attachment_ids = Email.fetch_attachments_for args[:email_id]
+        { :attachment_ids => attachment_ids }
+      end
+      
+      # by default :childs is 1
+      job "email.send" do |args|
+        Email.send({
+          :id => args[:email_id],
+          :attachments => args[:attachment_ids].map { |a| Attachment.find(a) }
+        })
+      end
+
+    end
+
+    Beanpicker.enqueue(["email.fetch_attachments", "email.send"], :email_id => 10)
+
+
+## Output Messages
+
+Inside of a job you can use debug, info, warn, error and fatal. It will be redirected to logger(STDOUT by default)
+
+## Options
+
+### Global options
+
+All options are inside of module Beanpicker
+
+*   Used for jobs:
+   *   Global:
+      *   default\_fork\_every: If should fork a job and destroy every time It will run. This options is overwrited by specified job options. Default true
+      *   default\_fork\_master: If should fork the child process. This options is overwrited by specified job options. Default false
+      *   fork\_every: Like default\_fork\_every, but overwrite job options. Default nil
+      *   fork\_master: Like default\_fork\_master, but overwrite job options. Default nil
+      *   default\_childs\_number: How much childs should be started for every job? Default 1
+   *   In job file(not function):
+      *   log\_file : Use a own log file, this file should be used to all jobs, except the ones who specify a :log\_file
+   *   In 'job' function:
+      *   :fork\_every : Overwrite default\_fork\_every and is overwrited by fork\_every
+      *   :fork\_master : Overwrite default\_fork\_master and is overwrited by fork\_master
+      *   :fork : Overwrite :fork\_every and :fork\_master, expect :every to every=true and master=false, :master to every=false and master=true or other value(any) to every=false and master=false. The result can be overwrited by fork\_master and fork\_every.
+      *   :log\_file : Use a own log file
+*   Used for enqueue
+   *   Global
+      *   default\_pri: The priority of job. Default 65536
+      *   default\_delay: The delay to start the job. Default 0
+      *   default\_ttr: The time to run the job. Default is 120
+   *   In 'enqueue' function
+      *   :pri
+      *   :delay
+      *   :ttr
+
+
+## Using combine
+
+Beanpicker ships with "combine", "A Beanpicker server"
+
+Try combine --help to see all options
+
+e.g. command:
+
+    combine -r config.rb -l log/jobs.log sandwich_jobs.rb email_jobs.rb
+
+
+## Multiple Beanstalk servers
+
+Beanpicker look in ENV variables BEANSTALK\_URL and BEANSTALK\_URLS.
+
+In BEANSTALK\_URL it expect a url like "server[:port]" or "beanstalk://server[:port]".
+
+In BEANSTALK\_URLS it expect a list of urls separed by comma. e.g. "localhost,localhost:11301,10.1.1.9,10.1.1.10:3000"
+
+## Credits
+
+Created by [Renan Fernandes][renan-website]
+
+Released under the [MIT License][license]
+
+[beanstalk]: http://kr.github.com/beanstalkd/ "Beanstalk" 
+[stalker]: http://github.com/adamwiggins/stalker "Stalker"
+[minion]: http://github.com/orionz/minion "Minion"
+[license]: http://www.opensource.org/licenses/mit-license.php "MIT License"
+[renan-website]: http://renanfernandes.com.br "Author's Website"

data/Rakefile

@@ -0,0 +1,47 @@
+require 'rubygems'
+require 'rake'
+
+require File.expand_path("../lib/beanpicker/version", __FILE__)
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name        = "beanpicker"
+    gem.summary     = "DSL for beanstalkd, similar to Stalker/Minion"
+    gem.description = "DSL for beanstalkd, similar to Stalker/Minion but uses subprocesses"
+    gem.email       = "renan@kauamanga.com.br"
+    gem.homepage    = "http://github.com/ShadowBelmolve/beanpicker"
+    gem.authors     = ["Renan Fernandes"]
+    gem.license     = "MIT"
+    gem.version     = Beanpicker::VERSION_STRING
+    gem.add_dependency "beanstalk-client"
+    gem.add_development_dependency "rspec", ">= 2.0"
+    gem.executables = ["combine"]
+
+
+  end
+  Jeweler::GemcutterTasks.new
+  Jeweler::RubygemsDotOrgTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+
+desc "Run the tests with RSpec"
+task :test do
+  require 'rspec/autorun'
+  ARGV.clear
+  ARGV << "spec"
+end
+
+task :default => :test
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+
+  rdoc.rdoc_dir = 'doc'
+  rdoc.title = "Beanpicker #{Beanpicker::VERSION_STRING}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+

data/beanpicker.gemspec

@@ -0,0 +1,61 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{beanpicker}
+  s.version = "0.1.0"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Renan Fernandes"]
+  s.date = %q{2010-12-28}
+  s.default_executable = %q{combine}
+  s.description = %q{DSL for beanstalkd, similar to Stalker/Minion but uses subprocesses}
+  s.email = %q{renan@kauamanga.com.br}
+  s.executables = ["combine"]
+  s.extra_rdoc_files = [
+    "README.markdown"
+  ]
+  s.files = [
+    ".rspec",
+    "README.markdown",
+    "Rakefile",
+    "beanpicker.gemspec",
+    "bin/combine",
+    "examples/sandwich.rb",
+    "lib/beanpicker.rb",
+    "lib/beanpicker/job_server.rb",
+    "lib/beanpicker/process.rb",
+    "lib/beanpicker/version.rb",
+    "spec/beanpicker_spec.rb",
+    "spec/spec_helper.rb"
+  ]
+  s.homepage = %q{http://github.com/ShadowBelmolve/beanpicker}
+  s.licenses = ["MIT"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.7}
+  s.summary = %q{DSL for beanstalkd, similar to Stalker/Minion}
+  s.test_files = [
+    "examples/sandwich.rb",
+    "spec/beanpicker_spec.rb",
+    "spec/spec_helper.rb"
+  ]
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<beanstalk-client>, [">= 0"])
+      s.add_development_dependency(%q<rspec>, [">= 2.0"])
+    else
+      s.add_dependency(%q<beanstalk-client>, [">= 0"])
+      s.add_dependency(%q<rspec>, [">= 2.0"])
+    end
+  else
+    s.add_dependency(%q<beanstalk-client>, [">= 0"])
+    s.add_dependency(%q<rspec>, [">= 2.0"])
+  end
+end
+

data/bin/combine

@@ -0,0 +1,88 @@
+#!/usr/bin/env ruby
+
+require 'optparse'
+require File.expand_path( File.join(File.dirname(__FILE__), "..", "lib", "beanpicker", "job_server") )
+
+_opts = nil
+
+parser = OptionParser.new do |opts|
+  _opts = opts
+  
+  opts.banner = 'combine is a beanpicker server'
+  opts.separator ''
+  opts.separator "e.g. #{$0} -r default.rb -em -l log/out.log sandwhich_jobs.rb email_jobs.rb"
+  opts.separator ''
+
+  opts.on('-r', '--require FILE', "Require a file to be loaded before the workers") do |f|
+    require File.expand_path(f)
+  end
+
+  opts.on('-e', '--[no-]fork-every', "Run every job in their own process?",
+          "This option overwrite the options of jobs") do |e|
+    Beanpicker::fork_every = e
+  end
+
+  opts.on('-m', '--[no-]fork-master', "Run every child in their own process?",
+          "This option overwrite the options of jobs") do |m|
+    Beanpicker::fork_master = m
+  end
+
+  opts.on('-y', '--[no-]fork-every-default', "Same of -e, but don't overwrite jobs options",
+         "Default is true") do |y|
+    Beanpicker::default_fork_every = y
+  end
+
+  opts.on('-a', '--[no-]fork-master-default', "Same of -m, but don't overwrite jobs options",
+         "Default is false") do |a|
+    Beanpicker::default_fork_master = a
+  end
+
+  opts.on('-c', '--childs-number-default N', Integer, "The number of childs every job should have",
+         "Default is 1") do |n|
+    Beanpicker::default_childs_number = n > 1 ? n : 1
+  end
+
+  opts.on('-l', '--logger [FILE]', "Redirect messages to file", "Default is STDOUT") do |f|
+    Beanpicker::log_handler = f || STDOUT
+  end
+
+  opts.on_tail('-h', '--help', "Show this message and exit") do
+    puts opts
+    exit
+  end
+
+  opts.on_tail('-v', '--version', "Show the version and exit") do
+    puts "beanpicker version: #{Beanpicker::VERSION_STRING}"
+    exit
+  end
+
+end
+
+begin
+  parser.parse!(ARGV)
+rescue => e
+  STDERR.puts "Error when parsing options.\n#{e.message}\n\n"
+  puts _opts
+  exit
+end
+
+if ARGV.empty?
+  puts "You should specify at least a file with jobs to run"
+  puts _opts
+  exit
+end
+
+for file in ARGV
+  if File.exists?(file)
+    if not File.readable?(file)
+      puts "File #{file} exits but isn't readable!"
+      exit
+    end
+  else
+    puts "File #{file} don't exists!"
+    exit
+  end
+end
+
+server = Beanpicker::Server.new(ARGV)
+server.run

data/examples/sandwich.rb

@@ -0,0 +1,46 @@
+#example of jobs
+
+
+# It's a slow job, so we'll spawn 10 forks
+# It'll be forked every time
+job "sandwich.make", :childs => 10, :fork => :every do |args|
+  debug "Making a sandwich to #{args[:for]}"
+  # very slow job
+  id = 10
+  { :id => id }
+end
+
+# This is as fast job and don't make memory leaks, so We'll fork once
+job "sandwich.sell", :fork => :master do |args|
+  debug "Selling the sandwich #{args[:id]} to #{args[:for]}"
+  #client = Client.find_by_name(args[:for])
+  #sandwich = Sandwich.find(args[:id])
+  #sale = client.buy_sandwich sandwich
+  sale = args[:for] == "Renan"
+  if sale
+    { :sale_id => 20 }
+  else
+    warn "Can't sell the sandwich to #{args[:for]} :("
+    false
+  end
+end
+
+# fast job but make memory leaks, so We'll fork every time
+job "sandwich.ingredients.recalcule", :childs => 3, :fork => :every do |args|
+  debug "Recalculating the ingredients of sandwich #{args[:id]} sold to #{args[:for]} in sale #{args[:sale_id]}"
+  #Ingredients.recalcule_based_on_sale(args[:sale_id])
+end
+
+# Example of Normal Jobs
+#   def make_sandwich_for(client_name)
+#     Beanpicker.enqueue("sandwich.make", :for => client_name)
+#   end
+#   This will call the "sandwich.make" job and pass a hash with client_name on :for key
+
+# Example of Chain Jobs
+#   Beanpicker.enqueue(["sandwich.make", "sandwich.sell", "sandwich.ingredients.recalcule"], :for => "Renan")
+#   This will call all the three jobs
+#
+#   Beanpicker.enqueue(["sandwich.make", "sandwich.sell", "sandwich.ingredients.recalcule"], :for => "Raphael")
+#   This will call the two first jobs, but not the third. Raphael don't have money to buy a sandwich, so the
+#     second job will return false and Beanpicker will break the chain

data/lib/beanpicker.rb

@@ -0,0 +1,268 @@
+require 'rubygems'
+require 'beanstalk-client'
+require 'uri'
+require 'logger'
+
+$:.unshift( File.expand_path(File.dirname(__FILE__)) )
+
+require 'beanpicker/version'
+
+# The fucking master job DSL to beanstalkd
+#
+# Just use it and go to beach ;)
+module Beanpicker
+
+  extend self
+
+  # Abstract logger methods
+  module MsgLogger
+
+    # call .debug of logger
+    def debug(m)
+      log_handler.debug msg(m)
+    end
+
+    # call .info of logger
+    def info(m)
+      log_handler.info msg(m)
+    end
+
+    # call .warn of logger
+    def warn(m)
+      log_handler.warn msg(m)
+    end
+
+    # call .error of logger
+    def error(m)
+      log_handler.error msg(m)
+    end
+
+    # call .fatal of logger
+    def fatal(m)
+      log_handler.fatal msg(m)
+    end
+       
+    # prepare the message for logger
+    def msg(msg)
+      if @name
+        "[#{name}] #{msg}"
+      else
+        msg
+      end
+    end
+
+    # return the current logger os create a new
+    def log_handler
+      @log_handler ||= ::Logger.new(STDOUT)
+    end
+
+    # set a new logger
+    # if the argument is a String/IO it will create a new instance of Logger using it argument
+    # else it will see if the argument respond to debug, info, warn, error and fatal
+    def log_handler=(v)
+      if [String, IO].include?(v.class)
+        @log_handler = ::Logger.new(v)
+      else
+        for m in [:debug, :info, :warn, :error, :fatal]
+          unless v.respond_to?(m)
+            error "Logger #{v} don't respond to #{m}. Aborting!"
+            return
+          end
+        end
+        @log_handler = v
+      end
+    end
+
+  end
+
+  extend MsgLogger
+
+  # Send a new queue to beanstalkd
+  #
+  # The first argument is a String with the name of job or a Array of Strings to do job chains
+  # 
+  # The second argument should be any object that will be passed in a YAML format to the job
+  #
+  # The third argument should be a hash containing :pri(priority) => Integer, :delay => Integer and :ttr(time-to-work) => Integer
+  #
+  # If beanstalk raise a Beanstalk::NotConnected, enqueue will create a new instance of beanstalk connection and retry.
+  # If it raise again, enqueue will raise the error
+  def enqueue(jobs, args={}, o={})
+    opts = [
+      o[:pri]   || default_pri,
+      o[:delay] || default_delay,
+      o[:ttr]   || default_ttr
+    ]
+
+    jobs = [jobs.to_s] unless jobs.is_a?(Array)
+    jobs.compact!
+    raise ArgumentError, "you need at least 1 job" if jobs.empty?
+    job = jobs.first
+
+    beanstalk.use(job)
+    beanstalk.yput({ :args => args, :next_jobs => jobs[1..-1]}, *opts)
+  rescue Beanstalk::NotConnected => e
+    raise e if defined?(r)
+    r = true
+    error exception_message(e, "You have a problem with beanstalkd.\nIs it running?")
+    @@beanstalk = new_beanstalk
+    retry
+  end
+
+  # Return the default beanstalk connection os create a new one with new_beanstalk
+  def beanstalk
+    @@beanstalk ||= new_beanstalk
+  end
+
+  # Create a new beanstalk connection using the urls from beanstalk_urls
+  def new_beanstalk
+    Beanstalk::Pool.new(beanstalk_urls)
+  end
+
+  # Look in ENV['BEANSTALK_URL'] and ENV['BEANSTALK_URLS'] for beanstalk urls and process returning a array.
+  #
+  # If don't find a good url it will return a array with just "localhost:11300"(default beanstalk port)
+  def beanstalk_urls
+    urls = [ENV['BEANSTALK_URL'], ENV['BEANSTALK_URLS']].compact.join(",").split(",").map do |url|
+      if url =~ /^beanstalk:\/\//
+        uri = URI.parse(url)
+        url = "#{uri.host}:#{uri.port}"
+      else
+        url = url.gsub(/^([^:\/]+)(:(\d+)).*$/) { "#{$1}:#{$3 || 11300}" }
+      end
+      url =~ /^[^:\/]+:\d+$/ ? url : nil
+    end.compact
+    urls.empty? ? ["localhost:11300"] : urls
+  end
+
+  # Helper to should a exception message
+  def exception_message(e, msg=nil)
+    m = []
+    m << msg if msg
+    m << e.message
+    m += e.backtrace
+    m.join("\n")
+  end
+
+  # Return the default priority(65536 by default).
+  #
+  # This is used by enqueue
+  def default_pri
+    @@default_pri ||= 65536
+  end
+
+  # Set the default priority
+  def default_pri=(v)
+    @@default_pri = v
+  end
+
+  # Return the default delay(0 by default)
+  #
+  # This is used by enqueue
+  def default_delay
+    @@default_delay ||= 0
+  end
+
+  # Set the default delay
+  def default_delay=(v)
+    @@default_delay = v
+  end
+
+  # Set the default time-to-work(120 by default)
+  #
+  # This is used by enqueue
+  def default_ttr
+    @@default_ttr ||= 120
+  end
+
+  # Set the default time-to-work
+  def default_ttr=(v)
+    @@default_ttr = v
+  end
+
+  # Return the default number of childs that a Worker should create(1 by default)
+  #
+  # This is used by Worker::Child::process
+  def default_childs_number
+    @@default_childs_number ||= 1
+  end
+
+  # Set the default childs number
+  def default_childs_number=(v)
+    @@default_childs_number = v
+  end
+
+  # Return if a child should fork every time that a job will process.
+  # This option is overwrited by job options and fork_every
+  #
+  # This is used by Worker::Child
+  def default_fork_every
+    defined?(@@default_fork_every) ? @@default_fork_every : true
+  end
+
+  # Set the default_fork_every
+  def default_fork_every=(v)
+    @@default_fork_every = !!v
+  end
+
+  # Return if a child should fork itself on intialize.
+  # This should be used when default_fork_every is false.
+  # This option is overwrited by job options and fork_master
+  #
+  # Use it only if the jobs need high speed and are "memory leak"-safe
+  #
+  # This is used by Worker::Child
+  def default_fork_master
+    defined?(@@default_fork_master) ? @@default_fork_master : false
+  end
+
+  # Set the default_fork_master
+  def default_fork_master=(v)
+    @@default_fork_master = !!v
+  end
+
+  # See default_fork_every
+  #
+  # This option overwrite all others
+  def fork_every
+    defined?(@@fork_every) ? @@fork_every : nil
+  end
+
+  # Set the fork_every
+  def fork_every=(v)
+    @@fork_every = v.nil? ? nil : !!v
+  end
+
+  # See default_fork_master
+  #
+  # This option overwrite all others
+  def fork_master
+    defined?(@@fork_master) ? @@fork_master : nil
+  end
+
+  # Set the fork_master
+  def fork_master=(v)
+    @@fork_master = v.nil? ? nil : !!v
+  end
+
+  # Return a Array with the workers registered
+  def workers
+    @@workers ||= []
+  end
+
+  # Add a worker to the list of workers
+  def add_worker(worker)
+    workers << worker
+  end
+
+  # Call die! for all childs of every worker and clear the list
+  # See workers
+  def stop_workers
+    for worker in workers
+      for child in worker.childs
+        child.die!
+      end
+    end
+    workers.clear
+  end
+end