perfectsched 0.7.19 → 0.8.0

data/.gitignore

@@ -0,0 +1,6 @@
+.bundle
+Gemfile.lock
+vendor/*
+coverage/*
+pkg/*
+spec/test.db

data/ChangeLog

@@ -1,39 +1,3 @@
-== 2016-08-02 version 0.7.19
-
-* re-package
-
-== 2016-08-02 version 0.7.17
-
-* Upgraded perfectqueue v0.8.x
-* Upgraded sequel v3.48.0
-* Upgraded tzinfo v1.2.2
-
-
-== 2014-10-08 version 0.7.16
-
-* Upgraded perfectqueue v0.7.31
-* Upgraded sequel v3.48.0
-
-
-== 2014-06-10 version 0.7.15
-
-* RDBBackend: fixed modify_checked to update timeout and next_time
-
-
-== 2014-04-25 version 0.7.14
-
-* RDBBackend: fixed undefined local variable: 'url' to '@uri'
-
-
-== 2014-04-25 version 0.7.13
-
-* RDBBackend: fixed NoMethodError for @url
-
-
-== 2012-02-21 version 0.7.12
-
-* RDBBackend: added MySQL's SSL support
-
 
 == 2012-02-21 version 0.7.10
 

data/Gemfile

@@ -0,0 +1,3 @@
+source :rubygems
+
+gemspec

data/README.md

@@ -0,0 +1,219 @@
+# PerfectSched
+
+PerfectSched is a highly available distributed cron built on top of RDBMS.
+
+It provides at-least-once semantics; Even if a worker node fails during process a task, the task is retried by another worker.
+
+PerfectSched also guarantees that only one worker server processes a task if the server is alive.
+
+All you have to consider is implementing idempotent worker programs. It's recommended to use [PerfectQueue](https://github.com/treasure-data/perfectqueue) with PerfectSched.
+
+
+## API overview
+
+```
+# open a schedule collection
+PerfectSched.open(config, &block)  #=> #<ScheduleCollection>
+
+# add a schedule
+ScheduleCollection#add(task_id, type, options)
+
+# poll a scheduled task
+# (you don't have to use this method directly. see following sections)
+ScheduleCollection#poll  #=> #<Task>
+
+# get data associated with a task
+Task#data  #=> #<Hash>
+
+# finish a task
+Task#finish!
+
+# retry a task
+Task#retry!
+
+# create a schedule reference
+ScheduleCollection#[](key)  #=> #<Schedule>
+
+# chack the existance of the schedule
+Schedule#exists?
+
+# delete a schedule
+Schedule#delete!
+```
+
+### Error classes
+
+```
+ScheduleError < StandardError
+
+##
+# Workers may get these errors:
+#
+
+AlreadyFinishedError < ScheduleError
+
+NotFoundError < ScheduleError
+
+PreemptedError < ScheduleError
+
+ProcessStopError < RuntimeError
+
+##
+# Client or other situation:
+#
+
+ConfigError < RuntimeError
+
+AlreadyExistsError < ScheduleError
+
+NotSupportedError < ScheduleError
+```
+
+
+### Example
+
+```ruby
+# submit a task
+PerfectSched.open(config) {|sc|
+  data = {'key'=>"value"}
+  options = {
+    :cron => '0 * * * *',
+    :delay => 30,
+    :timezone => 'Asia/Tokyo',
+    :next_time => Time.parse('2013-01-01 00:00:00 +0900').to_i,
+    :data => data,
+  }
+  sc.submit("sched-id", "type1", options)
+}
+```
+
+
+## Writing a worker application
+
+### 1. Implement PerfectSched::Application::Base
+
+```ruby
+class TestHandler < PerfectSched::Application::Base
+  # implement run method
+  def run
+    # do something ...
+    puts "acquired task: #{task.inspect}"
+
+    # call task.finish!, task.retry! or task.release!
+    task.finish!
+  end
+end
+```
+
+### 2. Implement PerfectSched::Application::Dispatch
+
+```ruby
+class Dispatch < PerfectSched::Application::Dispatch
+  # describe routing
+  route "type1" => TestHandler
+  route /^regexp-.*$/ => :TestHandler  # String or Regexp => Class or Symbol
+end
+```
+
+### 3. Run the worker
+
+In a launcher script or rake file:
+
+```ruby
+system('perfectsched run -I. -rapp/schedules/dispatch Dispatch')
+```
+
+or:
+
+```ruby
+require 'perfectsched'
+require 'app/schedules/dispatch'
+
+PerfectSched::Worker.run(Dispatch) {
+  # this method is called when the worker process is restarted
+  raw = File.read('config/perfectsched.yml')
+  yml = YAJL.load(raw)
+  yml[ENV['RAILS_ENV'] || 'development']
+}
+```
+
+### Signal handlers
+
+- **TERM**,**INT**,**QUIT:** shutdown
+- **USR1**,**HUP:** restart
+- **USR2:** reopen log files
+
+## Configuration
+
+- **type:** backend type (required; see following sections)
+- **log:** log file path (default: use stderr)
+- **poll\_interval:** interval to poll tasks in seconds (default: 1.0 sec)
+- **timezone:** default timezone (default: 'UTC')
+- **alive\_time:** duration to continue a heartbeat request (default: 300 sec)
+- **retry\_wait:** duration to retry a retried task (default: 300 sec)
+
+## Backend types
+
+### rdb\_compat
+
+additional configuration:
+
+- **url:** URL to the RDBMS (example: 'mysql://user:password@host:port/database')
+- **table:** name of the table to use
+
+### rdb
+
+Not implemented yet.
+
+
+## Command line management tool
+
+```
+Usage: perfectsched [options] <command>
+
+commands:
+    list                             Show list of registered schedules
+    add <key> <type> <cron> <data>   Register a new schedule
+    delete <key>                     Delete a registered schedule
+    run <class>                      Run a worker process
+    init                             Initialize a backend database
+
+options:
+    -e, --environment ENV            Framework environment (default: development)
+    -c, --config PATH.yml            Path to a configuration file (default: config/perfectsched.yml)
+
+options for add:
+    -d, --delay SEC                  Delay time before running a schedule (default: 0)
+    -t, --timezone NAME              Set timezone (default: UTC)
+    -s, --start UNIXTIME             Set the first schedule time (default: now)
+    -a, --at UNIXTIME                Set the first run time (default: start+delay)
+
+options for run:
+    -I, --include PATH               Add $LOAD_PATH directory
+    -r, --require PATH               Require files before starting
+```
+
+### initializing a database
+
+    # assume that the config/perfectsched.yml exists
+    $ perfectsched init
+
+### submitting a task
+
+    $ perfectsched add s1 user_task '* * * * *' '{}'
+
+### listing tasks
+
+    $ perfectsched list
+                               key            type               cron   delay    timezone                    next_time                next_run_time  data
+                                s1       user_task          * * * * *       0         UTC      2012-05-18 22:04:00 UTC      2012-05-18 22:04:00 UTC  {}
+    1 entries.
+
+### delete a schedule
+
+    $ perfectsched delete s1
+
+### running a worker
+
+    $ perfectsched run -I. -Ilib -rconfig/boot.rb -rapps/schedules/schedule_dispatch.rb ScheduleDispatch
+

data/Rakefile

@@ -0,0 +1,19 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = ["-c", "-f progress", "-r ./spec/spec_helper.rb"]
+  t.pattern = 'spec/**/*_spec.rb'
+  t.verbose = true
+end
+
+task :coverage do |t|
+  ENV['SIMPLE_COV'] = '1'
+  Rake::Task["spec"].invoke
+end
+
+task :default => :build
+

data/lib/perfectsched/application.rb

@@ -0,0 +1,25 @@
+#
+# PerfectSched
+#
+# Copyright (C) 2012 FURUHASHI Sadayuki
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+
+module PerfectSched
+
+  require 'perfectqueue/application'
+  Application = PerfectQueue::Application
+
+end
+

data/lib/perfectsched/backend/rdb_compat.rb

@@ -0,0 +1,254 @@
+#
+# PerfectSched
+#
+# Copyright (C) 2012 FURUHASHI Sadayuki
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+
+module PerfectSched
+  module Backend
+    class RDBCompatBackend
+      include BackendHelper
+
+      class Token < Struct.new(:row_id, :scheduled_time, :cron, :delay, :timezone)
+      end
+
+      def initialize(client, config)
+        super
+
+        require 'sequel'
+        url = config[:url]
+        unless url
+          raise ConfigError, "url option is required for the rdb_compat backend"
+        end
+
+        @table = config[:table]
+        unless @table
+          raise ConfigError, "table option is required for the rdb_compat backend"
+        end
+
+        #password = config[:password]
+        #user = config[:user]
+        @db = Sequel.connect(url, :max_connections=>1)
+        @mutex = Mutex.new
+
+        connect {
+          # connection test
+        }
+      end
+
+      MAX_SELECT_ROW = 4
+
+      attr_reader :db
+
+      def init_database(options)
+        sql = %[
+          CREATE TABLE IF NOT EXISTS `test_scheds` (
+            id VARCHAR(256) NOT NULL,
+            timeout INT NOT NULL,
+            next_time INT NOT NULL,
+            cron VARCHAR(128) NOT NULL,
+            delay INT NOT NULL,
+            data BLOB NOT NULL,
+            timezone VARCHAR(256) NULL,
+            PRIMARY KEY (id)
+          );]
+        connect {
+          @db.run sql
+        }
+      end
+
+      def get_schedule_metadata(key, options={})
+        connect {
+          row = @db.fetch("SELECT id, timeout, next_time, cron, delay, data, timezone FROM `#{@table}` LIMIT 1").first
+          unless row
+            raise NotFoundError, "schedule key=#{key} does not exist"
+          end
+          attributes = create_attributes(row)
+          return ScheduleMetadata.new(@client, key, attributes)
+        }
+      end
+
+      def list(options, &block)
+        connect {
+          @db.fetch("SELECT id, timeout, next_time, cron, delay, data, timezone FROM `#{@table}` ORDER BY timeout ASC") {|row|
+            attributes = create_attributes(row)
+            sched = ScheduleWithMetadata.new(@client, row[:id], attributes)
+            yield sched
+          }
+        }
+      end
+
+      def add(key, type, cron, delay, timezone, data, next_time, next_run_time, options)
+        data = data.dup
+        data[:type] = type
+        connect {
+          begin
+            n = @db["INSERT INTO `#{@table}` (id, timeout, next_time, cron, delay, data, timezone) VALUES (?, ?, ?, ?, ?, ?, ?);", key, next_run_time, next_time, cron, delay, data.to_json, timezone].insert
+            return Schedule.new(@client, key)
+          rescue Sequel::DatabaseError
+            raise AlreadyExistsError, "schedule key=#{key} already exists"
+          end
+        }
+      end
+
+      def delete(key, options)
+        connect {
+          n = @db["DELETE FROM `#{@table}` WHERE id=?;", key].delete
+          if n <= 0
+            raise NotFoundError, "schedule key=#{key} does no exist"
+          end
+        }
+      end
+
+      def modify(key, options)
+        ks = []
+        vs = []
+        [:cron, :delay, :timezone].each {|k|
+          # TODO type and data are not supported
+          if v = options[k]
+            ks << k
+            vs << v
+          end
+        }
+        return nil if ks.empty?
+
+        sql = "UPDATE `#{@table}` SET "
+        sql << ks.map {|k| "#{k}=?" }.join(', ')
+        sql << " WHERE id=?"
+
+        args = [sql].concat(vs)
+        args << key
+
+        connect {
+          n = @db[*args].update
+          if n <= 0
+            raise NotFoundError, "schedule key=#{key} does not exist"
+          end
+        }
+      end
+
+      def acquire(alive_time, max_acquire, options)
+        now = (options[:now] || Time.now).to_i
+        next_timeout = now + alive_time
+
+        connect {
+          while true
+            rows = 0
+            @db.fetch("SELECT id, timeout, next_time, cron, delay, data, timezone FROM `#{@table}` WHERE timeout <= ? ORDER BY timeout ASC LIMIT #{MAX_SELECT_ROW};", now) {|row|
+
+              n = @db["UPDATE `#{@table}` SET timeout=? WHERE id=? AND timeout=?;", next_timeout, row[:id], row[:timeout]].update
+              if n > 0
+                scheduled_time = Time.at(row[:next_time]).utc
+                attributes = create_attributes(row)
+                task_token = Token.new(row[:id], row[:next_time], attributes[:cron], attributes[:delay], attributes[:timezone])
+                task = Task.new(@client, row[:id], attributes, scheduled_time, task_token)
+                return [task]
+              end
+
+              rows += 1
+            }
+            if rows < MAX_SELECT_ROW
+              return nil
+            end
+          end
+        }
+      end
+
+      def heartbeat(task_token, alive_time, options)
+        now = (options[:now] || Time.now).to_i
+        row_id = task_token.row_id
+        scheduled_time = task_token.scheduled_time
+        next_run_time = now + alive_time
+
+        connect {
+          n = @db["UPDATE `#{@table}` SET timeout=? WHERE id=? AND next_time=?;", next_run_time, row_id, scheduled_time].update
+          if n < 0
+            raise AlreadyFinishedError, "task time=#{Time.at(scheduled_time).utc} is already finished"
+          end
+        }
+      end
+
+      def finish(task_token, options)
+        row_id = task_token.row_id
+        scheduled_time = task_token.scheduled_time
+        next_time = PerfectSched.next_time(task_token.cron, scheduled_time, task_token.timezone)
+        next_run_time = next_time + task_token.delay
+
+        connect {
+          n = @db["UPDATE `#{@table}` SET timeout=?, next_time=? WHERE id=? AND next_time=?;", next_run_time, next_time, row_id, scheduled_time].update
+          if n < 0
+            raise AlreadyFinishedError, "task time=#{Time.at(scheduled_time).utc} is already finished"
+          end
+        }
+      end
+
+      protected
+      def connect(&block)
+        @mutex.synchronize do
+          retry_count = 0
+          begin
+            block.call
+          rescue
+            # workaround for "Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction" error
+            if $!.to_s.include?('try restarting transaction')
+              err = ([$!] + $!.backtrace.map {|bt| "  #{bt}" }).join("\n")
+              retry_count += 1
+              if retry_count < MAX_RETRY
+                STDERR.puts err + "\n  retrying."
+                sleep 0.5
+                retry
+              else
+                STDERR.puts err + "\n  abort."
+              end
+            end
+            raise
+          ensure
+            @db.disconnect
+          end
+        end
+      end
+
+      def create_attributes(row)
+        timezone = row[:timezone] || 'UTC'
+        delay = row[:delay] || 0
+        cron = row[:cron]
+        next_time = Time.at(row[:next_time]).utc
+        next_run_time = Time.at(row[:timeout]).utc
+
+        begin
+          data = JSON.parse(row[:data] || '{}')
+        rescue
+          data = {}
+        end
+
+        type = data.delete('type') || ''
+
+        attributes = {
+          :timezone => timezone,
+          :delay => delay,
+          :cron => cron,
+          :data => data,
+          :next_time => next_time,
+          :next_run_time => next_run_time,
+          :type => type,
+          :message => nil,  # not supported
+          :node => nil,  # not supported
+        }
+      end
+
+    end
+  end
+end
+