gaspar 0.1.1

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,10 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in gaspar.gemspec
+gemspec
+
+gem 'colorize'
+gem 'rspec'
+gem 'redis'
+gem 'active_support'
+gem 'timecop'

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Mashable, Inc
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,91 @@
+# Gaspar
+<img src="http://cdn.wikimg.net/strategywiki/images/0/04/Chrono_Trigger_Sprites_Gaspar.png" align="right" style="margin: 0 0 20px 20px" />
+
+Gaspar is a recurring job ("cron") manager for Ruby daemons. It's primarily intended to be used with Rails + Sidekiq/Resque and friends.
+
+Gaspar runs in-process, meaning there is no additional daemon to configure or maintain. Just define your jobs and they'll get fired by *something*,
+whether that's your Rails processes, Sidekiq workers, or whatnot. Of course, you can always run it in a separate daemon if you wanted, too.
+
+By default, Gaspar does not run if you are running under Rails and `Rails.env.test?`. Pass `:permit_test_mode => true` to `Gaspar.schedule` if you want Gaspar to run in test mode.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'gaspar'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install gaspar
+
+## Usage
+
+Usage is straightforward. Jobs are defined in a DSL. At a minimum, you need to pass a Redis instance to Gaspar, which is used for synchronizing job locks. The redis instance should be threadsafe; if you haven't turned it off explicitly, thread safety should already be present.
+
+    Gaspar.configure(:logger => Rails.logger) do
+      every "5s", "Ping" do
+        Rails.logger.debug "ping"
+      end
+    end
+
+Job definitions take one of several formats:
+
+    every [time], [job name] do
+      # Code to run
+    end
+
+If you have Resque or Sidekiq installed, you can pass a worker class name as string or symbol (plus any args), and it'll be automatically enqueued when the job fires:
+
+    every [time], :DoStuffWorker, 1, 2, 3
+    # If you are using Resque, this will fire `Resque.enqueue(DoStuffWorker, 1, 2, 3)`
+    # If you are using Sidekiq, this will fire `DoStuffWorker.perform_async(1, 2, 3)`
+
+Jobs also accept cron formats:
+
+    cron "15,30 * * * *" do
+      # Run stuff at 15 and 30 past the hour
+    end
+
+Jobs are each run in their own individual thread, but you should keep your jobs as lightweight as possible, so best practices will generally mean firing off background workers. Jobs should never exceed 15 seconds runtime, as on process exit, Gaspar will delay for up to 15 seconds to allow currently-running jobs to terminate before they are abandoned. Additionally, jobs should be threadsafe.
+
+    Gaspar.configure(:logger => Rails.logger) do
+      every "10m",         :UpdateStuffWorker
+      cron  "0 * * * * ",  :DailyUpdateWorker, "with", :some_options => true
+      every("1h")          { HourlyUpdateWorker.perform_async }
+    end
+
+Once you have Gaspar configured, you'll need to choose when to start it, and you'll pass a Redis connection for Gaspar to use. This is done separately from the configuration with the expectation that you won't want to run Gaspar for everything that boots your app, and you'll need to take care to close Gaspar (using `Gaspar#retire`) pre-forking, and to start it post-forking (using `Gaspar#start!`)
+
+    Gaspar.start!(Redis.new)
+
+Since Gaspar uses a Redis connection, you should initialize it after your daemon forks. For example, to use it with Unicorn:
+
+    before_fork do |server, worker|
+      Gaspar.retire
+    end
+
+    after_fork do |server, worker|
+      Gaspar.start! Redis.new
+    end
+
+Or with Passenger:
+
+    PhusionPassenger.on_event(:starting_worker_process) do |forked|
+      if forked
+        Gaspar.start! Redis.new
+      end
+    end
+
+Finally, Gaspar will refuse to initialize if the process has a controlling TTY. This prevents it from running for, say, rake tasks and the like.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,3 @@
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new('spec')

data/gaspar.gemspec

@@ -0,0 +1,26 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'gaspar/version'
+
+Gem::Specification.new do |gem|
+  gem.name          = "gaspar"
+  gem.version       = Gaspar::VERSION
+  gem.authors       = ["Chris Heald"]
+  gem.email         = ["cheald@mashable.com"]
+  gem.description   = %q{Gaspar is an in-process recurring job manager. It is intended to be used in place of cron when you don't want a separate daemon.}
+  gem.summary       = %q{Gaspar is an in-process recurring job manager.}
+  gem.homepage      = "http://github.com/mashable/gaspar"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+
+  gem.add_dependency('rufus-scheduler')
+  gem.add_dependency('redis', '>= 2.2.0')
+  gem.add_dependency('colorize')
+  gem.add_dependency('activesupport')
+  gem.add_development_dependency('rspec')
+  gem.add_development_dependency('timecop')
+end

data/lib/gaspar/version.rb

@@ -0,0 +1,3 @@
+class Gaspar
+  VERSION = "0.1.1"
+end

data/lib/gaspar.rb

@@ -0,0 +1,200 @@
+require 'thread'
+require "gaspar/version"
+require "rufus/scheduler"
+require 'colorize'
+require 'active_support/core_ext/array/extract_options'
+require 'active_support/inflector'
+require 'active_support/concern'
+require 'active_support/callbacks'
+
+class Gaspar
+  attr_reader :drift, :scheduler
+  include ActiveSupport::Callbacks
+  define_callbacks :run
+
+  class << self
+    attr_reader :singleton
+
+    # Public: Configure Gaspar
+    #
+    # options - an options hash
+    def configure(options = {}, &block)
+      @singleton ||= new(options, &block)
+      self
+    end
+
+    # Public: Get whether Gaspar has been configured yet or not
+    #
+    # Returns: [Boolean]
+    def configured?
+      !@singleton.nil?
+    end
+
+    # Public: Stop processing jobs and destroy the singleton. Returns Gaspar to an unconfigured state.
+    def destruct!
+      retire
+      @singleton = nil
+    end
+
+    # Public: Execute the configuration and start processing jobs.
+    #
+    # redis - The redis instance to use for synchronization
+    def start!(redis)
+      raise "Gaspar#configure has not been called, or did not succeed" if @singleton.nil?
+      @singleton.send(:start!, redis) if @singleton
+    end
+
+    def retire
+      return unless @singleton
+      @singleton.send(:shutdown!)
+    end
+
+    def log(logger, message)
+      logger.debug "[%s] %s" % ["Gaspar".yellow, message] if logger
+    end
+  end
+
+  def before_each(&block)
+    self.class.set_callback :run, :before, &block
+  end
+
+  def after_each(&block)
+    self.class.set_callback :run, :after, &block
+  end
+
+  def around_each(&block)
+    self.class.set_callback :run, :around, &block
+  end
+
+  def every(timing, *args, &block)
+    options = args.extract_options!
+
+    # In order to make sure that jobs are executed at the same time regardless of who runs them
+    # we quantitize the start time to the next-nearest time slice. This more closely emulates
+    # cron-style behavior.
+    if timing.is_a? String
+      seconds = Rufus.parse_duration_string(timing)
+    else
+      seconds = timing.to_i
+    end
+    now = Time.now.to_i - drift
+    start_at = Time.at( now + (seconds - (now % seconds)) )
+
+    options = options.merge(:first_at => start_at)
+    options[:period] = seconds
+
+    schedule :every, timing, args, options, &block
+  end
+
+  def cron(timing, *args, &block)
+    options = args.extract_options!
+    next_fire = Rufus::CronLine.new(timing).next_time
+
+    options[:period] = next_fire.to_i - Time.now.to_i
+    schedule :cron, timing, args, options, &block
+  end
+
+  private
+
+  def lock
+    @lock.synchronize { yield }
+  end
+
+  def schedule(method, timing, args = [], options = {}, &block)
+    if block_given?
+      options[:name] ||= args.first
+    else
+      klass, worker_args = *args
+      options[:name] ||= "%s(%s)" % [klass, args.join(", ")]
+      klass = klass.to_s
+      case @options[:worker]
+      when :resque
+        block = Proc.new { Resque.enqueue klass.constantize, *worker_args }
+      when :sidekiq
+        block = Proc.new { klass.constantize.perform_async *worker_args }
+      end
+    end
+
+    name = options.delete :name
+    if name.nil?
+      if Object.const_defined? :Sourcify
+        name = Digest::SHA1.hexdigest(block.to_source)
+      else
+        raise "No :name specified and sourcify is not available. Specify a name, or add sourcify to your bundle."
+      end
+    end
+    key = "#{@options[:namespace]}:%s-%s" % [timing, name]
+    period = options.delete :period
+    expiry = period - 5
+    expiry = 1 if expiry < 1
+
+    scheduler.send method, timing, options do
+      # If we can acquire a lock...
+      if @redis.setnx key, Process.pid
+        log "#{Process.pid} running #{name}"
+        # ...set the lock to expire, which makes sure that staggered workers with out-of-sync clocks don't
+        lock { @running_jobs += 1 }
+        @redis.expire key, expiry.to_i
+        # ...and then run the job
+        run_callbacks(:run) { block.call }
+        lock { @running_jobs -= 1 }
+      end
+    end
+  end
+
+  def log(message)
+    self.class.log @logger, message
+  end
+
+  def initialize(options = {}, &block)
+    @logger       = options[:logger]
+    @options      = options
+    @block        = block
+    @lock = Mutex.new
+    lock { @running_jobs = 0 }
+
+    @options[:namespace] ||= "gaspar"
+    @options[:worker]    ||= :sidekiq if Object.const_defined? :Sidekiq
+    @options[:worker]    ||= :resque  if Object.const_defined? :Resque
+  end
+
+  def start!(redis)
+    return log "Running under a controlling TTY. Refusing to start. Try starting from a daemonized process." if STDOUT.tty? or STDERR.tty?
+    if Object.const_defined? :Rails and Rails.env.test?
+      return unless @options[:permit_test_mode]
+    end
+
+    @redis = redis
+
+    return if @started
+
+    @started = true
+    sync_watches
+    @scheduler = Rufus::Scheduler.start_new
+    @scheduler.every("1h") { sync_watches }
+    instance_eval &@block
+
+    at_exit do
+      @scheduler.stop if @scheduler
+      force_shutdown_at = Time.now.to_i + 15
+      sleep(0.1) while lock { @running_jobs } > 0 and Time.now.to_i < force_shutdown_at
+    end
+  end
+
+  def shutdown!
+    @scheduler.stop if @scheduler
+    @started = false
+  end
+
+  # Abuse Redis key TTLs to synchronize our watches
+  def sync_watches
+    if @redis.setnx "#{@options[:namespace]}:timesync", Time.now.to_i
+      @redis.expire "#{@options[:namespace]}:timesync", 3.2e8.to_i  # Set to expire in ~100 years
+    end
+    epoch  = @redis.get("#{@options[:namespace]}:timesync").to_i
+    ttl    = @redis.ttl "#{@options[:namespace]}:timesync"
+    offset = (3.2e8 - ttl)
+    @drift = Time.now.to_i - (epoch + offset)
+    log "Resynced - Drift is #{@drift}"
+  end
+end

data/spec/gaspar/gaspar_spec.rb

@@ -0,0 +1,190 @@
+require 'spec_helper'
+
+describe Gaspar do
+  let(:redis) { Redis.new }
+  after(:each) { Gaspar.destruct! }
+  context "when running in a non-daemon" do
+    it "should refuse to start if under a controlling TTY" do
+      STDOUT.stub(:tty?).and_return(true)
+      Gaspar.should_receive(:log).with(nil, "Running under a controlling TTY. Refusing to start. Try starting from a daemonized process.")
+      Gaspar.configure do
+        every "5m", :Foo
+      end.start!(redis)
+    end
+  end
+
+  context "when running in a daemon" do
+    before(:each) {
+      STDOUT.stub(:tty?).and_return(false)
+      STDERR.stub(:tty?).and_return(false)
+    }
+    after(:each) { Gaspar.reset_callbacks(:run) }
+
+    context "configuration" do
+      it "should accept #every during configuration" do
+        Gaspar.any_instance.should_receive(:schedule).with(:every, "5m", [], instance_of(Hash))
+        Gaspar.configure do
+          every("5m") { puts "Doing stuff" }
+        end.start!(redis)
+      end
+
+      it "should require a name if a block is passed to a job" do
+        expect {
+          Gaspar.configure do
+            every("5m") { puts "Doing stuff" }
+          end.start!(redis)
+        }.to raise_error("No :name specified and sourcify is not available. Specify a name, or add sourcify to your bundle.")
+      end
+
+      it "should run callbacks" do
+        callbacks = []
+        Gaspar.configure do
+          before_each { callbacks.push "before" }
+          after_each  { callbacks.push "after" }
+          around_each {|&block| callbacks.push "around"; block.call }
+
+          every("0.35s", "run callbacks") { callbacks.push "inside" }
+        end.start!(redis)
+        sleep(0.4)
+
+        callbacks.should == %w(before around inside after)
+      end
+
+      it "should not require a name if a symbol is passed to a job" do
+        expect {
+          Gaspar.configure do
+            every "5m", :Foobar
+          end.start!(redis)
+        }.to_not raise_error
+      end
+
+      it "should enqueue a job" do
+        Time.stub!(:now).and_return Time.at(1351061802)
+        Gaspar.any_instance.stub(:drift).and_return(0)
+        Gaspar.configure do
+          every "5m", :Foobar
+        end
+        scheduler = double(:scheduler)
+        Gaspar.singleton.stub(:scheduler).and_return scheduler
+        scheduler.should_receive(:every).with("5m", :first_at => Time.at(1351062000))
+        Gaspar.start!(redis)
+      end
+
+      it "should accept #cron during configuration" do
+        Gaspar.any_instance.should_receive(:schedule).with(:cron, "* * * * *", [], instance_of(Hash))
+        Gaspar.configure do
+          cron("* * * * *") { puts "Doing stuff" }
+        end.start!(redis)
+      end
+
+      it "should quantitize #every to the next timeslice" do
+        Time.stub!(:now).and_return Time.at(1351061802)
+        Gaspar.any_instance.stub(:drift).and_return(0)
+        Gaspar.any_instance.should_receive(:schedule).with(:every, "5m", [], {:first_at => Time.at(1351062000), :period => 300.0})
+        Gaspar.configure do
+          every("5m") { puts "Doing stuff" }
+        end.start!(redis)
+      end
+
+      it "should detect Resque" do
+        Resque = 1
+        Gaspar.configure do
+          every "5m", :Foo
+        end.start!(redis)
+        Gaspar.singleton.instance_variable_get("@options")[:worker].should == :resque
+        Object.send :remove_const, :Resque
+      end
+
+      it "should detect Sidekiq" do
+        Sidekiq = 1
+        Gaspar.configure do
+          every "5m", :Foo
+        end.start!(redis)
+        Gaspar.singleton.instance_variable_get("@options")[:worker].should == :sidekiq
+        Object.send :remove_const, :Sidekiq
+      end
+
+      it "should process jobs" do
+        value = 0
+        sleep 0.4
+        Gaspar.configure do
+          every "0.35s", "update variable" do
+            value += 1
+          end
+        end.start!(redis)
+        value.should == 0
+        sleep(0.4)
+        value.should > 0
+      end
+
+      it "should prevent jobs from running multiple times for the same time period" do
+        value = 0
+        sleep 0.4
+        Gaspar.configure do
+          every("0.35s", "update variable with lock") { value += 1 }
+          every("0.35s", "update variable with lock") { value += 1 }
+          every("0.35s", "update variable with lock") { value += 1 }
+        end.start!(redis)
+        value.should == 0
+        sleep(0.4)
+        value.should == 1
+      end
+
+    end
+
+    context "class methods" do
+      describe "#log" do
+        it "should silently eat logging messages when no logger is specified" do
+          expect { Gaspar.log(nil, "foobar") }.to_not raise_error
+        end
+      end
+
+      describe "#log" do
+        it "should log when a logger is passed" do
+          logger = double(:logger, :debug => true)
+          expect { Gaspar.log(logger, "foobar") }.to_not raise_error
+        end
+      end
+
+      describe "#destruct!" do
+        it "should not fail when the singleton has not been initialized" do
+          expect { Gaspar.destruct! }.to_not raise_error
+        end
+
+        it "should kill the singleton if it has been previous initialized" do
+          Gaspar.configure do
+            every("5m") { puts "Doing stuff" }
+          end
+          Gaspar.singleton.should_receive(:shutdown!)
+          Gaspar.destruct!
+        end
+      end
+
+      describe "#start!" do
+        it "should fail if the singleton has not been initialized" do
+          expect { Gaspar.start!(redis) }.to raise_error "Gaspar#configure has not been called, or did not succeed"
+        end
+
+        it "succeeds if the singleton has been initialized" do
+          Gaspar.configure do
+            every("5m", :name => "do stuff") { puts "Doing stuff" }
+          end.start!(redis)
+          Gaspar.singleton.instance_variable_get("@started").should == true
+        end
+      end
+    end
+
+    context "instance methods" do
+      describe "#sync_watches" do
+        it "should compute drift" do
+          Time.stub!(:now).and_return(150)
+          redis = double :redis, :setnx => false, :get => "100", :ttl => (3.2e8.to_i - 25)
+          instance = Gaspar.send(:new)
+          instance.instance_variable_set(:@redis, redis) 
+          instance.send :sync_watches
+          instance.drift.should == 25
+        end
+      end
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,7 @@
+require 'rubygems'
+require 'bundler/setup'
+require 'gaspar'
+require 'redis'
+
+RSpec.configure do |config|
+end

metadata

@@ -0,0 +1,155 @@
+--- !ruby/object:Gem::Specification
+name: gaspar
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+  prerelease: 
+platform: ruby
+authors:
+- Chris Heald
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-03-12 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rufus-scheduler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.2.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.2.0
+- !ruby/object:Gem::Dependency
+  name: colorize
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: timecop
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Gaspar is an in-process recurring job manager. It is intended to be used
+  in place of cron when you don't want a separate daemon.
+email:
+- cheald@mashable.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- gaspar.gemspec
+- lib/gaspar.rb
+- lib/gaspar/version.rb
+- spec/gaspar/gaspar_spec.rb
+- spec/spec_helper.rb
+homepage: http://github.com/mashable/gaspar
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Gaspar is an in-process recurring job manager.
+test_files:
+- spec/gaspar/gaspar_spec.rb
+- spec/spec_helper.rb
+has_rdoc: