sidekiq-cron 0.1.0

data/.document

@@ -0,0 +1,5 @@
+lib/**/*.rb
+bin/*
+- 
+features/**/*.feature
+LICENSE.txt

data/.travis.yml

@@ -0,0 +1,18 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - jruby-19mode
+  - rbx-19mode
+  - 2.0.0
+branches:
+  only:
+    - master
+notifications:
+  email:
+    recipients:
+      - ondrej@bartas.cz
+matrix:
+  allow_failures:
+    - rvm: jruby-19mode
+    - rvm: rbx-19mode
+    - rvm: 2.0.0

data/Changes.md

@@ -0,0 +1,8 @@
+
+
+v 0.1.1
+-------
+
+- add Web fontend with enabled/disable job, unqueue now, delete job
+- add cron poller - enqueu cro jobs
+- add cron job - save all needed data to redis

data/Gemfile

@@ -0,0 +1,30 @@
+source 'https://rubygems.org'
+
+gem "sidekiq", ">= 2.13.1"
+gem 'parse-cron', '>= 0.1.2'
+
+group :development do
+  gem "bundler"
+  gem "simplecov"
+  
+  gem 'shoulda-context'
+  gem "turn"
+
+  gem 'rack'
+  gem 'rack-test'
+
+  gem "jeweler", "~> 1.8.3"
+  
+  gem "sdoc" # sdoc -N .
+  
+  gem "slim"
+  gem "sinatra"
+  
+  gem 'mocha'
+  gem 'coveralls'
+
+  gem "shotgun"
+
+  # gem 'guard'
+  # gem 'guard-minitest'
+end

data/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2013 Ondrej Bartas
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,152 @@
+Sidekiq-Cron
+============
+
+[![Build Status](https://travis-ci.org/ondrejbartas/sidekiq-cron.png?branch=master)](https://travis-ci.org/ondrejbartas/sidekiq-cron) [![Coverage Status](https://coveralls.io/repos/ondrejbartas/sidekiq-cron/badge.png?branch=master)](https://coveralls.io/r/ondrejbartas/sidekiq-cron?branch=master)
+
+Add-on for [Sidekiq](http://sidekiq.org)
+
+Allows you to schedule recurring jobs for sidekiq workers using cron notation _* * * * *_.
+
+Requirements
+-----------------
+
+- Redis 2.4 or greater is required.
+- Sidekiq 2.13.1 or grater is required.
+
+
+Installation
+------------
+
+    $ gem install sidekiq-cron
+
+or add to your Gemfile
+
+    gem "sidekiq-cron", "~> 0.1.0"
+
+
+Getting Started
+-----------------
+
+
+If you are not using Rails you need to add `require 'sidekiq-cron'` somewhere after `require 'sidekiq'`.
+
+_Job properties_:
+
+```ruby
+{
+ 'name'  => 'name_of_job', #must be uniq!
+ 'cron'  => '1 * * * *',
+ 'klass' => 'MyClass',
+ #OPTIONAL
+ 'queue' => 'name of queue',
+ 'args'  => '[Array or Hash] of arguments hich will be passed to perform method'
+}
+```
+
+#### Adding Cron job:
+```ruby
+
+class HardWorker
+  include Sidekiq::Worker
+  def perform(name, count)
+    # do something
+  end
+end
+
+Sidekiq::Cron::Job.create( name: 'Hard worker - every 5min', cron: '*/5 * * * *', klass: 'HardWorker')
+# => true
+```
+
+`create` method will return only true/false if job was saved or not.
+
+```ruby
+job = Sidekiq::Cron::Job.new( name: 'Hard worker - every 5min', cron: '*/5 * * * *', klass: 'HardWorker')
+
+if job.valid?
+  job.save
+else
+  puts job.errors
+end
+
+#or simple
+
+unless job.save
+  puts job.errors #will return array of errors
+end
+```
+
+#### Finding jobs
+```ruby
+#return array of all jobs
+Sidekiq::Cron::Job.all
+
+#return one job by its uniq name - case sensitive
+Sidekiq::Cron::Job.find "Job Name"
+
+#return one job by its uniq name - you can use hash with 'name' key
+Sidekiq::Cron::Job.find name: "Job Name"
+
+#if job can't be found nil is returned
+```
+
+#### Destroy jobs:
+```ruby
+#destroys all jobs
+Sidekiq::Cron::Job.destroy_all!
+
+#destroy job by its name
+Sidekiq::Cron::Job.destroy "Job Name"
+
+#destroy founded job
+Sidekiq::Cron::Job.find('Job name').destroy
+```
+
+#### Work with job:
+```ruby
+job = Sidekiq::Cron::Job.find('Job name')
+
+#disable cron scheduling
+job.disable!
+
+#enable cron scheduling
+job.enable!
+
+#get status of job:
+job.status
+# => enabled/disabled
+
+#enqueue job right now!
+job.enque!
+```
+
+How to start scheduling?
+Just start sidekiq workers by:
+
+    sidekiq
+
+### Web Ui for Cron Jobs
+
+If you are using sidekiq web ui and you would like to add cron josb to web too,
+add `require 'sidekiq-cron'` after `require 'sidekiq/web'`.
+By this you will get:
+![Web UI](https://github.com/ondrejbartas/sidekiq-cron/raw/master/examples/web-cron-ui.png)
+
+
+
+## Contributing to sidekiq-cron
+
+
+* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet.
+* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it.
+* Fork the project.
+* Start a feature/bugfix branch.
+* Commit and push until you are happy with your contribution.
+* Make sure to add tests for it. This is important so I don't break it in a future version unintentionally.
+* Please try not to mess with the Rakefile, version, or history. If you want to have your own version, or is otherwise necessary, that is fine, but please isolate to its own commit so I can cherry-pick around it.
+
+
+## Copyright
+
+Copyright (c) 2013 Ondrej Bartas. See LICENSE.txt for
+further details.
+

data/Rakefile

@@ -0,0 +1,61 @@
+# encoding: utf-8
+
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'jeweler'
+Jeweler::Tasks.new do |gem|
+  # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
+  gem.name = "sidekiq-cron"
+  gem.homepage = "http://github.com/ondrejbartas/sidekiq-cron"
+  gem.license = "MIT"
+  gem.summary = %Q{Sidekiq Cron helps to add repeated scheduled jobs}
+  gem.description = %Q{Enables to set jobs to be run in specified time (using CRON notation)}
+  gem.email = "ondrej@bartas.cz"
+  gem.authors = ["Ondrej Bartas"]
+  # dependencies defined in Gemfile
+end
+Jeweler::RubygemsDotOrgTasks.new
+
+#TESTING
+
+task :doc do
+  system 'sdoc -N .'
+end
+
+require 'rake/testtask'
+task :default => :test
+
+Rake::TestTask.new(:test) do |t|
+  t.test_files = FileList['test/functional/**/*_test.rb', 'test/unit/**/*_test.rb','test/integration/**/*_test.rb']
+  t.warning = false
+  t.verbose = false
+end
+
+namespace :test do
+  Rake::TestTask.new(:unit) do |t|
+    t.test_files = FileList['test/unit/**/*_test.rb']
+    t.warning = false
+    t.verbose = false
+  end
+
+  Rake::TestTask.new(:functional) do |t|
+    t.test_files = FileList['test/functional/**/*_test.rb']
+    t.warning = false
+    t.verbose = false
+  end
+
+  Rake::TestTask.new(:integration) do |t|
+    t.test_files = FileList['test/integration/**/*_test.rb']
+    t.warning = false
+    t.verbose = false
+  end
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/config.ru

@@ -0,0 +1,14 @@
+require 'sidekiq'
+
+Sidekiq.configure_client do |config|
+  config.redis = { :size => 1 }
+end
+
+require 'sidekiq/web'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+
+require 'sidekiq-cron'
+
+run Sidekiq::Web

data/lib/sidekiq-cron.rb

@@ -0,0 +1,4 @@
+
+require "sidekiq"
+
+require "sidekiq/cron"

data/lib/sidekiq/cron.rb

@@ -0,0 +1,29 @@
+begin
+  require "sidekiq/web"
+rescue LoadError
+  # client-only usage
+end
+
+require "sidekiq/cron/job"
+require "sidekiq/cron/web_extension"
+
+#require poller only if celluloid is defined
+if defined?(Celluloid)
+  require "sidekiq/cron/poller"
+end
+
+module Sidekiq
+  module Cron
+  end
+end
+
+if defined?(Sidekiq::Web)
+  Sidekiq::Web.register Sidekiq::Cron::WebExtension
+
+  if Sidekiq::Web.tabs.is_a?(Array)
+    # For sidekiq < 2.5
+    Sidekiq::Web.tabs << "cron"
+  else
+    Sidekiq::Web.tabs["Cron"] = "cron"
+  end
+end

data/lib/sidekiq/cron/job.rb

@@ -0,0 +1,397 @@
+require 'sidekiq'
+require 'sidekiq/util'
+require 'sidekiq/actor'
+require 'parse-cron'
+
+module Sidekiq
+  module Cron
+
+    class Job
+      include Util
+      extend Util
+
+      #how long we would like to store informations about previous enqueues
+      REMEMBER_THRESHOLD = 24 * 60 * 60 
+
+      #crucial part of whole enquing job
+      def should_enque? time
+        out = false
+        Sidekiq.redis do |conn|
+          out = (
+            status == "enabled" && 
+            @last_run_time < last_time(time) && 
+            conn.zadd(job_enqueued_key, time.to_f.to_s, formated_last_time(time) )
+          )
+        end
+        out
+      end
+
+      # remove previous informations about run times 
+      # this will clear redis and make sure that redis will
+      # not overflow with memory
+      def remove_previous_enques time
+        Sidekiq.redis do |conn|
+          conn.zremrangebyscore(job_enqueued_key, 0, "(#{(time.to_f - REMEMBER_THRESHOLD).to_s}")
+        end
+      end
+
+      #test if job should be enqued If yes add it to queue
+      def test_and_enque_for_time! time
+        #should this job be enqued?
+        if should_enque?(time)
+          enque!
+
+          remove_previous_enques(time)
+        end
+      end
+
+      #enque cron job to queue
+      def enque! time = Time.now
+        @last_run_time = time
+
+        Sidekiq::Client.push(@message.is_a?(String) ? Sidekiq.load_json(@message) : @message)
+
+        save
+        logger.debug { "enqueued #{@name}: #{@message}" }
+      end
+
+      # load cron jobs from Hash
+      # input structure should look like:
+      # {
+      #   'name_of_job' => {
+      #     'class' => 'MyClass',
+      #     'cron'  => '1 * * * *',
+      #     'args'  => '(OPTIONAL) [Array or Hash]'
+      #   },
+      #   'My super iber cool job' => {
+      #     'class' => 'SecondClass',
+      #     'cron'  => '*/5 * * * *'
+      #   }
+      # }
+      #
+      def self.load_from_hash hash
+        array = hash.inject([]) do |out,(key, job)|
+          job['name'] = key
+          out << job
+        end
+        load_from_array array
+      end
+
+
+      # load cron jobs from Array
+      # input structure should look like:
+      # [
+      #   {
+      #     'name'  => 'name_of_job',
+      #     'class' => 'MyClass',
+      #     'cron'  => '1 * * * *',
+      #     'args'  => '(OPTIONAL) [Array or Hash]'
+      #   },
+      #   {
+      #     'name'  => 'Cool Job for Second Class',
+      #     'class' => 'SecondClass',
+      #     'cron'  => '*/5 * * * *'
+      #   }
+      # ]
+      #
+      def self.load_from_array array
+        errors = {}
+        array.each do |job_data|
+          job = new(job_data)
+          errors[job.name] = job.errors unless job.save
+        end
+        errors
+      end
+
+
+      # get all cron jobs
+      def self.all
+        out = []
+        Sidekiq.redis do |conn|
+          out = conn.smembers(jobs_key).collect do |key|
+            if conn.exists key
+              Job.new conn.hgetall(key)
+            else
+              nil
+            end
+          end
+        end
+        out.select{|j| !j.nil? }
+      end
+
+      def self.count
+        out = 0
+        Sidekiq.redis do |conn|
+          out = conn.scard(jobs_key)
+        end
+        out
+      end
+
+      def self.find name
+        #if name is hash try to get name from it
+        name = name[:name] || name['name'] if name.is_a?(Hash)
+
+        output = nil
+        Sidekiq.redis do |conn|
+          if exists? name
+            output = Job.new conn.hgetall( redis_key(name) )
+          end
+        end
+        output
+      end
+
+      # create new instance of cron job
+      def self.create hash
+        new(hash).save
+      end
+
+      #destroy job by name
+      def self.destroy name
+        #if name is hash try to get name from it
+        name = name[:name] || name['name'] if name.is_a?(Hash)
+
+        if job = find(name)
+          job.destroy
+        else
+          false
+        end
+      end
+
+      attr_accessor :name, :cron, :klass, :args, :message
+      attr_reader   :last_run_time
+
+      def initialize input_args = {}
+        args = input_args.stringify_keys
+
+        @name = args["name"]
+        @cron = args["cron"]
+
+        #get class from klass or class
+        @klass = args["klass"] || args["class"]
+
+        #set status of job
+        @status = args['status'] || status_from_redis
+
+        #set last run time
+        @last_run_time = Time.parse(args['last_run_time'].to_s) rescue Time.now
+
+        #get right arguments for job
+        @args = args["args"].nil? ? [] : (args["args"].is_a?(Array) ? args["args"] : [ args["args"] ])
+
+        if args["message"]
+          @message = args["message"]
+        elsif @klass
+          message_data = {
+            "class" => @klass.to_s,
+            "args"  => @args,
+          }
+
+          #get right data for message
+          #only if message wasn't specified before
+          message_data = case @klass
+            when Class
+              @klass.get_sidekiq_options.merge(message_data)
+            when String
+              begin
+                @klass.constantize.get_sidekiq_options.merge(message_data)
+              rescue 
+                #Unknown class
+                message_data.merge("queue"=>"default")
+              end
+
+          end
+
+          #override queue if setted in config
+          #only if message is hash - can be string (dumped JSON)
+          message_data['queue'] = args['queue'] if args['queue']
+
+          #dump message as json
+          @message = message_data
+        end
+
+      end
+
+      def status 
+        @status
+      end
+
+      def disable!
+        @status = "disabled"
+        save
+      end
+
+      def enable!
+        @status = "enabled"
+        save
+      end
+
+      def status_from_redis
+        if exists?
+          out = "enabled"
+          Sidekiq.redis do |conn|
+            out = conn.hget redis_key, "status"
+          end
+          out
+        else
+          "enabled"
+        end
+      end
+
+      #export job data to hash
+      def to_hash
+        {
+          name: @name,
+          klass: @klass,
+          cron: @cron,
+          args: @args.is_a?(String) ? @args : Sidekiq.dump_json(@args || []),
+          message: @message.is_a?(String) ? @message : Sidekiq.dump_json(@message || {}),
+          status: @status,
+          last_run_time: @last_run_time,
+        }
+      end
+
+      def errors 
+        @errors ||= []
+      end
+
+      def valid?
+        #clear previos errors
+        @errors = []
+
+        errors << "'name' must be set" if @name.nil? || @name.size == 0
+        if @cron.nil? || @cron.size == 0
+          errors << "'cron' must be set" 
+        else
+          begin 
+            cron = CronParser.new(@cron)
+            cron.next(Time.now)
+          rescue Exception => e
+            errors << "'cron' -> #{@cron}: #{e.message}"
+          end
+        end
+
+        errors << "'klass' (or class) must be set" if @klass.nil? || @klass.size == 0
+
+        !errors.any?
+      end
+
+      # add job to cron jobs
+      # input:
+      #   name: (string) - name of job
+      #   cron: (string: '* * * * *' - cron specification when to run job
+      #   class: (string|class) - which class to perform
+      # optional input:
+      #   queue: (string) - which queue to use for enquing (will override class queue)
+      #   args: (array|hash|nil) - arguments for permorm method
+
+      def save
+        #if job is invalid return false
+        return false unless valid?
+
+        Sidekiq.redis do |conn|
+
+          #add to set of all jobs
+          conn.sadd self.class.jobs_key, redis_key
+
+          #add informations for this job!
+          conn.hmset redis_key, *hash_to_redis(to_hash)
+
+          #add information about last time! - don't enque right after scheduler poller starts!
+          time = Time.now
+          conn.zadd(job_enqueued_key, time.to_f.to_s, formated_last_time(time).to_s)
+        end
+        logger.info { "Cron Jobs - add job with name: #{@name}" }
+      end
+      
+      # remove job from cron jobs by name
+      # input:
+      #   first arg: name (string) - name of job (must be same - case sensitive)
+      def destroy
+        Sidekiq.redis do |conn|
+          #delete from set
+          conn.srem self.class.jobs_key, redis_key
+          
+          #delete runned timestamps
+          conn.del job_enqueued_key
+
+          #delete main job
+          conn.del redis_key
+        end
+        logger.info { "Cron Jobs - deleted job with name: #{@name}" }
+      end
+
+      # remove all job from cron
+      def self.destroy_all!
+        all.each do |job|
+          job.destroy
+        end
+        logger.info { "Cron Jobs - deleted all jobs" }
+      end
+
+      # Parse cron specification '* * * * *' and returns
+      # time when last run should be performed
+      def last_time now = Time.now
+        # add 1 minute to Time now - Cron parser return last time after minute ends,
+        # so by adding 60 second we will get last time after the right time happens 
+        # without any delay!
+        CronParser.new(@cron).last(now + 60)
+      end
+
+      def formated_last_time now = Time.now
+        last_time(now).getutc
+      end
+
+      def self.exists? name
+        out = false
+        Sidekiq.redis do |conn|
+          out = conn.exists redis_key name
+        end
+        out
+      end
+
+      def exists?
+        self.class.exists? @name
+      end
+
+      def sort_name
+        "#{status == "enabled" ? 0 : 1}_#{name}".downcase
+      end
+      
+      private
+
+      # Redis key for set of all cron jobs
+      def self.jobs_key
+        "cron_jobs"
+      end
+
+      # Redis key for storing one cron job
+      def self.redis_key name
+        "cron_job:#{name}"
+      end
+
+      # Redis key for storing one cron job
+      def redis_key
+        self.class.redis_key @name
+      end
+
+      # Redis key for storing one cron job run times
+      # (when poller added job to queue)
+      def self.job_enqueued_key name
+        "cron_job:#{name}:enqueued"
+      end
+
+      # Redis key for storing one cron job run times
+      # (when poller added job to queue)
+      def job_enqueued_key
+        self.class.job_enqueued_key @name
+      end
+
+      # Give Hash
+      # returns array for using it for redis.hmset
+      def hash_to_redis hash
+        hash.inject([]){ |arr,kv| arr + [kv[0], kv[1]] }
+      end
+
+    end
+  end
+end