perfectsched 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/ChangeLog ADDED
@@ -0,0 +1,5 @@
1
+
2
+ == 2011-09-03 version 0.7.0
3
+
4
+ * Initial release
5
+
data/README.rdoc ADDED
@@ -0,0 +1,137 @@
1
+ = PerfectSched
2
+
3
+ Highly available distributed cron works with PerfectQueue.
4
+
5
+ It provides exactly-once semantics unless backend database fails. Registered schedules are surely pushed to a queue provided by PerfectQueue every time in order.
6
+
7
+ You can register, modify and delete schedules using the command line utility or library API.
8
+
9
+ Backend database is pluggable. PerfectSched supports RDBMS and Amazon SimpleDB for now.
10
+
11
+
12
+ == Architecture
13
+
14
+ PerfectSched uses following database schema:
15
+
16
+ (
17
+ id:string -- unique identifier of the schedule
18
+ data:blob -- additional attributes to be pushed to PerfectQueue
19
+ next_time:int -- unix time of the next schedule
20
+ cron:string -- description of the schedule
21
+ delay:int -- delay time before running a schedule
22
+ timeout:int
23
+ )
24
+
25
+ 1. list: lists tasks whose timeout column is old enough.
26
+ 2. lock: updates timeout column of the first task
27
+ 3. push: push a message to the PerfectQueue
28
+ 4. update: if it succeeded, updates the next_time and timeout columns
29
+ 5. or leave: if it failed, leave the row and expect to be retried.
30
+
31
+
32
+ === Cooperation with PerfectQueue
33
+
34
+ PerfectSched pushes a task to PerfectQueue every time on schedule. The ID of the task becomes "<id of the scuedule>.<unix time of the schedule>". For example, the identifier of the schedule is "my-sched", and a schedule runs at "2011-08-30 00:00:00 UTC" (1314662400 in UNIX TIME), the ID of the task is "my-sched.1314662400". The data of the task is same as the schedule.
35
+
36
+
37
+ == Library usage
38
+
39
+ === Adding a schedule
40
+
41
+ require 'perfectsched'
42
+
43
+ # RDBMS
44
+ require 'perfectsched/backend/rdb'
45
+ sched = PerfectSched::Backend::RDBBackend.new(
46
+ 'mysql://user:password@localhost/mydb', table='perfectsched')
47
+
48
+ # SimpleDB
49
+ require 'perfectsched/backend/simpledb'
50
+ sched = PerfectSched::Backend::SimpleDBBackend.new(
51
+ 'AWS_KEY_ID', 'AWS_SECRET_KEY', 'your-simpledb-domain-name')
52
+
53
+ id = 'unique-key-id'
54
+ cron = "* * * * *"
55
+ delay = 0
56
+ data = '{"any":"data"}'
57
+ start = Time.now.to_i
58
+ sched.add(id, cron, delay, data, start)
59
+
60
+ === Deleting a schedule
61
+
62
+ sched.delete(id)
63
+
64
+ === Modifying a schedule
65
+
66
+ cron = "* * * * 0"
67
+ delay = 10
68
+ sched.modify_sched(id, cron, delay)
69
+
70
+ data = '{"user":1}'
71
+ sched.modify_data(id, data)
72
+
73
+ sched.modify(id, cron, delay, data)
74
+
75
+
76
+
77
+ == Command line usage
78
+
79
+ Usage: perfectsched [options]
80
+ --setup PATH.yaml Write example configuration file
81
+ -f, --file PATH.yaml Set path to the configuration file
82
+
83
+ --list Show registered schedule
84
+ --delete ID Delete a registered schedule
85
+
86
+ --add <ID> <CRON> <DATA> Register a schedule
87
+ -d, --delay SEC Delay time before running a schedule (default: 0)
88
+ -s, --start UNIXTIME Start time to run a schedule (default: now)
89
+
90
+ -S, --modify-sched <ID> <CRON> Modify schedule of a registered schedule
91
+ -D, --modify-delay <ID> <DELAY> Modify delay of a registered schedule
92
+ -J, --modify-data <ID> <DATA> Modify data of a registered schedule
93
+
94
+ -b, --daemon PIDFILE Daemonize (default: foreground)
95
+ -o, --log PATH log file path
96
+ -v, --verbose verbose mode
97
+
98
+
99
+ === Configuration
100
+
101
+ First of all, create a configuration file:
102
+
103
+ $ perfectsched --setup config.yaml
104
+ $ edit config.yaml
105
+
106
+
107
+ === Adding a schedule
108
+
109
+ $ perfectsched -f config.yaml --add unique-key-id "* * * * *" '{"any":"data"}'
110
+
111
+
112
+ === Deleting a schedule
113
+
114
+ $ perfectsched -f config.yaml --delete unique-key-id
115
+
116
+
117
+ === Modifying a schedule
118
+
119
+ $ perfectsched -f config.yaml --modify-sched unique-key-id "* * * * 0"
120
+ $ perfectsched -f config.yaml --modify-delay unique-key-id 10
121
+ $ perfectsched -f config.yaml --modify-data unique-key-id '{"user":1}'
122
+
123
+
124
+ === Listing registered schedules
125
+
126
+ $ perfectsched -f config.yaml --list
127
+ id schedule delay next time next run data
128
+ test1 * * * * * 0 2011-08-30 01:29:42 +0900 2011-08-30 01:29:42 +0900 {"attr1":"val1","attr":"val2"}
129
+ 1 entries.
130
+
131
+
132
+ === Running a scheduler
133
+
134
+ $ perfectsched -f config.yaml
135
+
136
+ It's recommended to run the scheduler on several servers for availability.
137
+
data/bin/perfectsched ADDED
@@ -0,0 +1,6 @@
1
+ #!/usr/bin/env ruby
2
+ # -*- coding: utf-8 -*-
3
+ require 'rubygems' unless defined?(gem)
4
+ here = File.dirname(__FILE__)
5
+ $LOAD_PATH << File.expand_path(File.join(here, '..', 'lib'))
6
+ require 'perfectsched/command/perfectsched'
@@ -0,0 +1,4 @@
1
+ require 'perfectsched/engine'
2
+ require 'perfectsched/croncalc'
3
+ require 'perfectsched/backend'
4
+ require 'perfectsched/version'
@@ -0,0 +1,84 @@
1
+
2
+ module PerfectSched
3
+
4
+
5
+ class Task
6
+ def initialize(id, time, cron, delay, data)
7
+ @id = id
8
+ @time = time
9
+ @cron = cron
10
+ @delay = delay
11
+ @data = data
12
+ end
13
+
14
+ attr_reader :id, :time, :cron, :delay, :data
15
+ end
16
+
17
+
18
+ class Backend
19
+ def initialize
20
+ @croncalc = CronCalc.new
21
+ end
22
+
23
+ # => list {|id,cron,delay,data,next_time,timeout| ... }
24
+ def list(&block)
25
+ end
26
+
27
+ # => token, task
28
+ def acquire(timeout, now=Time.now.to_i)
29
+ end
30
+
31
+ # => true (success) or false (canceled)
32
+ def finish(token, next_time)
33
+ end
34
+
35
+ # => true (success) or nil (already exists)
36
+ def add(id, cron, delay, data, start_time)
37
+ first_time = @croncalc.next_time(cron, start_time.to_i)
38
+ timeout = first_time + delay
39
+ add_checked(id, cron, delay, data, first_time, timeout)
40
+ end
41
+
42
+ # => true (success) or nil (already exists)
43
+ def add_checked(id, cron, delay, data, next_time, timeout)
44
+ end
45
+
46
+ # => true (success) or false (not found, canceled or finished)
47
+ def delete(id)
48
+ end
49
+
50
+ # => true (success) or false (not found)
51
+ def modify(id, cron, delay, data)
52
+ cron = cron.strip
53
+ @croncalc.next_time(cron, 0)
54
+ modify_checked(id, cron, delay, data)
55
+ end
56
+
57
+ def modify_checked(id, cron, delay, data)
58
+ end
59
+
60
+ # => true (success) or false (not found)
61
+ def modify_sched(id, cron, delay)
62
+ cron = cron.strip
63
+ @croncalc.next_time(cron, 0)
64
+ modify_sched_checked(id, cron, delay)
65
+ end
66
+
67
+ def modify_sched_checked(id, cron, delay)
68
+ end
69
+
70
+ # => true (success) or false (not found)
71
+ def modify_data(id, data)
72
+ modify_data_checked(id, data)
73
+ end
74
+
75
+ def modify_data_checked(id, data)
76
+ end
77
+
78
+ def close
79
+ end
80
+ end
81
+
82
+
83
+ end
84
+
@@ -0,0 +1,144 @@
1
+
2
+ module PerfectSched
3
+
4
+
5
+ class RDBBackend < Backend
6
+ def initialize(uri, table)
7
+ super()
8
+ require 'sequel'
9
+ @uri = uri
10
+ @table = table
11
+ @db = Sequel.connect(@uri)
12
+ init_db(@uri.split(':',2)[0])
13
+ end
14
+
15
+ private
16
+ def init_db(type)
17
+ sql = ''
18
+ case type
19
+ when /mysql/i
20
+ sql << "CREATE TABLE IF NOT EXISTS `#{@table}` ("
21
+ sql << " id VARCHAR(256) NOT NULL,"
22
+ sql << " timeout INT NOT NULL,"
23
+ sql << " next_time INT NOT NULL,"
24
+ sql << " cron VARCHAR(128) NOT NULL,"
25
+ sql << " delay INT NOT NULL,"
26
+ sql << " data BLOB NOT NULL,"
27
+ sql << " PRIMARY KEY (id)"
28
+ sql << ") ENGINE=INNODB;"
29
+ else
30
+ sql << "CREATE TABLE IF NOT EXISTS `#{@table}` ("
31
+ sql << " id VARCHAR(256) NOT NULL,"
32
+ sql << " timeout INT NOT NULL,"
33
+ sql << " next_time INT NOT NULL,"
34
+ sql << " cron VARCHAR(128) NOT NULL,"
35
+ sql << " delay INT NOT NULL,"
36
+ sql << " data BLOB NOT NULL,"
37
+ sql << " PRIMARY KEY (id)"
38
+ sql << ");"
39
+ end
40
+ # TODO index
41
+ connect {
42
+ @db.run sql
43
+ }
44
+ end
45
+
46
+ def connect(&block)
47
+ begin
48
+ block.call
49
+ ensure
50
+ @db.disconnect
51
+ end
52
+ end
53
+
54
+ public
55
+ def list(&block)
56
+ @db.fetch("SELECT id, timeout, next_time, cron, delay, data FROM `#{@table}` ORDER BY timeout ASC") {|row|
57
+ yield row[:id], row[:cron], row[:delay], row[:data], row[:next_time], row[:timeout]
58
+ }
59
+ end
60
+
61
+ MAX_SELECT_ROW = 32
62
+
63
+ def acquire(timeout, now=Time.now.to_i)
64
+ connect {
65
+ while true
66
+ rows = 0
67
+ @db.fetch("SELECT id, timeout, next_time, cron, delay, data FROM `#{@table}` WHERE timeout <= ? ORDER BY timeout ASC LIMIT #{MAX_SELECT_ROW};", now) {|row|
68
+
69
+ n = @db["UPDATE `#{@table}` SET timeout=? WHERE id=? AND timeout=?;", timeout, row[:id], row[:timeout]].update
70
+ salt = timeout
71
+ if n > 0
72
+ return [row[:id],salt], Task.new(row[:id], row[:next_time], row[:cron], row[:delay], row[:data])
73
+ end
74
+
75
+ rows += 1
76
+ }
77
+ if rows < MAX_SELECT_ROW
78
+ return nil
79
+ end
80
+ end
81
+ }
82
+ end
83
+
84
+ def finish(token, next_time, timeout)
85
+ connect {
86
+ id, salt = *token
87
+ n = @db["UPDATE `#{@table}` SET timeout=?, next_time=? WHERE id=? AND timeout=?;", timeout, next_time, id, salt].update
88
+ return n > 0
89
+ }
90
+ end
91
+
92
+ def add_checked(id, cron, delay, data, next_time, timeout)
93
+ connect {
94
+ begin
95
+ n = @db["INSERT INTO `#{@table}` (id, timeout, next_time, cron, delay, data) VALUES (?, ?, ?, ?, ?, ?);", id, timeout, next_time, cron, delay, data].insert
96
+ return true
97
+ rescue Sequel::DatabaseError
98
+ return nil
99
+ end
100
+ }
101
+ end
102
+
103
+ def delete(id)
104
+ connect {
105
+ n = @db["DELETE FROM `#{@table}` WHERE id=?;", id].delete
106
+ return n > 0
107
+ }
108
+ end
109
+
110
+ def get(id)
111
+ connect {
112
+ @db.fetch("SELECT id, timeout, next_time, cron, delay, data FROM `#{@table}` WHERE id=?;", id) {|row|
113
+ return row[:cron], row[:delay], row[:data]
114
+ }
115
+ return nil
116
+ }
117
+ end
118
+
119
+ def modify_checked(id, cron, delay, data)
120
+ connect {
121
+ n = @db["UPDATE `#{@table}` SET cron=?, delay=?, data=? WHERE id=?;", cron, delay, data, id].update
122
+ return n > 0
123
+ }
124
+ end
125
+
126
+ def modify_sched_checked(id, cron, delay)
127
+ connect {
128
+ n = @db["UPDATE `#{@table}` SET cron=?, delay=? WHERE id=?;", cron, delay, id].update
129
+ return n > 0
130
+ }
131
+ end
132
+
133
+ def modify_data_checked(id, data)
134
+ connect {
135
+ n = @db["UPDATE `#{@table}` SET data=? WHERE id=?;", data, id].update
136
+ return n > 0
137
+ }
138
+ end
139
+
140
+ end
141
+
142
+
143
+ end
144
+
@@ -0,0 +1,165 @@
1
+
2
+ module PerfectSched
3
+
4
+
5
+ class SimpleDBBackend < Backend
6
+ def initialize(key_id, secret_key, domain)
7
+ super()
8
+ gem "aws-sdk"
9
+ require 'aws'
10
+ @consistent_read = false
11
+
12
+ @db = AWS::SimpleDB.new(
13
+ :access_key_id => key_id,
14
+ :secret_access_key => secret_key)
15
+
16
+ @domain_name = domain
17
+ @domain = @db.domains[@domain_name]
18
+ unless @domain.exists?
19
+ @domain = @db.domains.create(@domain_name)
20
+ end
21
+ end
22
+
23
+ attr_accessor :consistent_read
24
+
25
+ def use_consistent_read(b=true)
26
+ @consistent_read = b
27
+ self
28
+ end
29
+
30
+ def list(&block)
31
+ rows = 0
32
+ @domain.items.select('timeout', 'next_time', 'cron', 'delay', 'data',
33
+ :where => "timeout > '#{int_encode(0)}'", # required by SimpleDB
34
+ :order => [:timeout, :asc],
35
+ :consistent_read => @consistent_read,
36
+ :limit => MAX_SELECT_ROW) {|itemdata|
37
+ id = itemdata.name
38
+ attrs = itemdata.attributes
39
+
40
+ next_time = int_decode(attrs['next_time'].first)
41
+ cron = attrs['cron'].first
42
+ delay = int_decode(attrs['delay'].first)
43
+ data = attrs['data'].first
44
+ timeout = int_decode(attrs['timeout'].first)
45
+
46
+ yield id, cron, delay, data, next_time, timeout
47
+ }
48
+ end
49
+
50
+ MAX_SELECT_ROW = 32
51
+
52
+ def acquire(timeout, now=Time.now.to_i)
53
+ while true
54
+ rows = 0
55
+ @domain.items.select('timeout', 'next_time', 'cron', 'delay', 'data',
56
+ :where => "timeout <= '#{int_encode(now)}'",
57
+ :order => [:timeout, :asc],
58
+ :consistent_read => @consistent_read,
59
+ :limit => MAX_SELECT_ROW) {|itemdata|
60
+ begin
61
+ id = itemdata.name
62
+ attrs = itemdata.attributes
63
+
64
+ @domain.items[id].attributes.replace('timeout'=>int_encode(timeout),
65
+ :if=>{'timeout'=>attrs['timeout'].first})
66
+
67
+ next_time = int_decode(attrs['next_time'].first)
68
+ cron = attrs['cron'].first
69
+ delay = int_decode(attrs['delay'].first)
70
+ data = attrs['data'].first
71
+ salt = int_encode(timeout)
72
+
73
+ return [id,salt], Task.new(id, next_time, cron, delay, data)
74
+
75
+ rescue AWS::SimpleDB::Errors::ConditionalCheckFailed, AWS::SimpleDB::Errors::AttributeDoesNotExist
76
+ end
77
+
78
+ rows += 1
79
+ }
80
+ if rows < MAX_SELECT_ROW
81
+ return nil
82
+ end
83
+ end
84
+ end
85
+
86
+ def finish(token, next_time, timeout)
87
+ begin
88
+ id, salt = *token
89
+ @domain.items[id].attributes.replace('timeout'=>int_encode(timeout), 'next_time'=>int_encode(next_time),
90
+ :if=>{'timeout'=>salt})
91
+ return true
92
+ rescue AWS::SimpleDB::Errors::ConditionalCheckFailed, AWS::SimpleDB::Errors::AttributeDoesNotExist
93
+ return false
94
+ end
95
+ end
96
+
97
+ def add_checked(id, cron, delay, data, next_time, timeout)
98
+ begin
99
+ @domain.items[id].attributes.replace('timeout'=>int_encode(timeout), 'next_time'=>int_encode(next_time),
100
+ 'cron'=>cron, 'delay'=>int_encode(delay), 'data'=>data,
101
+ :unless=>'timeout')
102
+ return true
103
+ rescue AWS::SimpleDB::Errors::ConditionalCheckFailed, AWS::SimpleDB::Errors::ExistsAndExpectedValue
104
+ return nil
105
+ end
106
+ end
107
+
108
+ def delete(id)
109
+ # TODO return value
110
+ begin
111
+ @domain.items[id].delete
112
+ return true
113
+ rescue AWS::SimpleDB::Errors::ConditionalCheckFailed, AWS::SimpleDB::Errors::AttributeDoesNotExist
114
+ return false
115
+ end
116
+ end
117
+
118
+ def get(id)
119
+ attrs = @domain.items[id].data.attributes
120
+ cron = attrs['cron'].first
121
+ unless cron
122
+ return nil
123
+ end
124
+ delay = int_decode(attrs['delay'].first)
125
+ data = attrs['data'].first
126
+ return cron, delay, data
127
+ end
128
+
129
+ def modify_checked(id, cron, delay, data)
130
+ unless get(id)
131
+ return false
132
+ end
133
+ @domain.items[id].attributes.replace('cron'=>cron, 'delay'=>int_encode(delay), 'data'=>data)
134
+ return true
135
+ end
136
+
137
+ def modify_sched_checked(id, cron, delay)
138
+ unless get(id)
139
+ return false
140
+ end
141
+ @domain.items[id].attributes.replace('cron'=>cron, 'delay'=>int_encode(delay))
142
+ return true
143
+ end
144
+
145
+ def modify_data_checked(id, data)
146
+ unless get(id)
147
+ return false
148
+ end
149
+ @domain.items[id].attributes.replace('data'=>data)
150
+ return true
151
+ end
152
+
153
+ private
154
+ def int_encode(num)
155
+ "%08x" % num
156
+ end
157
+
158
+ def int_decode(str)
159
+ str.to_i(16)
160
+ end
161
+ end
162
+
163
+
164
+ end
165
+
@@ -0,0 +1,347 @@
1
+ require 'optparse'
2
+ require 'perfectsched/version'
3
+
4
+ op = OptionParser.new
5
+
6
+ op.banner += ""
7
+ op.version = PerfectSched::VERSION
8
+
9
+ type = nil
10
+ id = nil
11
+ confout = nil
12
+
13
+ conf = {
14
+ :timeout => 600,
15
+ :poll_interval => 1,
16
+ #:expire => 345600,
17
+ }
18
+
19
+ add_conf = {
20
+ :delay => 0,
21
+ }
22
+
23
+
24
+ op.on('--setup PATH.yaml', 'Write example configuration file') {|s|
25
+ type = :conf
26
+ confout = s
27
+ }
28
+
29
+ op.on('-f', '--file PATH.yaml', 'Set path to the configuration file') {|s|
30
+ (conf[:files] ||= []) << s
31
+ }
32
+
33
+ op.separator("")
34
+
35
+ op.on('--list', 'Show registered schedule', TrueClass) {|b|
36
+ type = :list
37
+ }
38
+
39
+ op.on('--delete ID', 'Delete a registered schedule') {|s|
40
+ type = :delete
41
+ id = s
42
+ }
43
+
44
+ op.separator("")
45
+
46
+ op.on('--add <ID> <CRON> <DATA>', 'Register a schedule') {|s|
47
+ type = :add
48
+ id = s
49
+ }
50
+
51
+ op.on('-d', '--delay SEC', 'Delay time before running a schedule (default: 0)', Integer) {|i|
52
+ add_conf[:delay] = i
53
+ }
54
+
55
+ op.on('-s', '--start UNIXTIME', 'Start time to run a schedule (default: now)', Integer) {|i|
56
+ add_conf[:start] = i
57
+ }
58
+
59
+ op.separator("")
60
+
61
+ op.on('-S', '--modify-sched <ID> <CRON>', 'Modify schedule of a registered schedule') {|s|
62
+ type = :modify_sched
63
+ id = s
64
+ }
65
+
66
+ op.on('-D', '--modify-delay <ID> <DELAY>', 'Modify delay of a registered schedule') {|s|
67
+ type = :modify_delay
68
+ id = s
69
+ }
70
+
71
+ op.on('-J', '--modify-data <ID> <DATA>', 'Modify data of a registered schedule') {|s|
72
+ type = :modify_data
73
+ id = s
74
+ }
75
+
76
+ op.separator("")
77
+
78
+ op.on('-b', '--daemon PIDFILE', 'Daemonize (default: foreground)') {|s|
79
+ conf[:daemon] = s
80
+ }
81
+
82
+ op.on('-o', '--log PATH', "log file path") {|s|
83
+ conf[:log] = s
84
+ }
85
+
86
+ op.on('-v', '--verbose', "verbose mode", TrueClass) {|b|
87
+ conf[:verbose] = true
88
+ }
89
+
90
+
91
+ (class<<self;self;end).module_eval do
92
+ define_method(:usage) do |msg|
93
+ puts op.to_s
94
+ puts "error: #{msg}" if msg
95
+ exit 1
96
+ end
97
+ end
98
+
99
+
100
+ begin
101
+ op.parse!(ARGV)
102
+
103
+ type ||= :run
104
+
105
+ case type
106
+ case :add
107
+ if ARGV.length != 2
108
+ usage nil
109
+ end
110
+ add_conf[:cron] = ARGV[0]
111
+ add_conf[:data] = ARGV[1]
112
+
113
+ when :modify_sched
114
+ if ARGV.length != 1
115
+ usage nil
116
+ end
117
+ add_conf[:cron] = ARGV[0]
118
+
119
+ when :modify_data
120
+ if ARGV.length != 1
121
+ usage nil
122
+ end
123
+ add_conf[:data] = ARGV[0]
124
+
125
+ when :modify_delay
126
+ if ARGV.length != 1 || ARGV[0].to_i.to_s != ARGV[0]
127
+ usage nil
128
+ end
129
+ add_conf[:delay] = ARGV[0].to_i
130
+
131
+ else
132
+ if ARGV.length != 0
133
+ usage nil
134
+ end
135
+ end
136
+
137
+ if confout
138
+ require 'yaml'
139
+
140
+ File.open(confout, "w") {|f|
141
+ f.write <<EOF
142
+ ---
143
+ timeout: 300
144
+ poll_interval: 1
145
+ backend:
146
+ database: "mysql://user:password@localhost/mydb"
147
+ table: "perfectsched"
148
+ #simpledb: your-simpledb-domain-name-for-scheduler
149
+ #aws_key_id: "AWS_ACCESS_KEY_ID"
150
+ #aws_secret_key: "AWS_SECRET_ACCESS_KEY"
151
+ queue:
152
+ database: "mysql://user:password@localhost/mydb"
153
+ table: "perfectqueue"
154
+ #simpledb: your-simpledb-domain-name-for-queue
155
+ #aws_key_id: "AWS_ACCESS_KEY_ID"
156
+ #aws_secret_key: "AWS_SECRET_ACCESS_KEY"
157
+ EOF
158
+ }
159
+ exit 0
160
+ end
161
+
162
+ unless conf[:files]
163
+ raise "-f, --file PATH.yaml option is required"
164
+ end
165
+
166
+ rescue
167
+ usage $!.to_s
168
+ end
169
+
170
+
171
+ require 'perfectsched'
172
+ require 'perfectqueue'
173
+
174
+ require 'yaml'
175
+ docs = ''
176
+ conf[:files].each {|file|
177
+ docs << File.read(file)
178
+ }
179
+ YAML.load_documents(docs) {|yaml|
180
+ yaml.each_pair {|k,v| conf[k.to_sym] = v }
181
+ }
182
+
183
+ conf[:timeout] ||= 60
184
+ conf[:poll_interval] ||= 1
185
+
186
+ # backend
187
+ bconf = yaml['backend']
188
+ if domain = bconf['simpledb']
189
+ require 'perfectsched/backend/simpledb'
190
+ key_id = bconf['aws_key_id'] || ENV['AWS_ACCESS_KEY_ID']
191
+ secret_key = bconf['aws_secret_key'] || ENV['AWS_SECRET_ACCESS_KEY']
192
+ backend = PerfectSched::SimpleDBBackend.new(key_id, secret_key, domain)
193
+ if type != :run
194
+ backend.use_consistent_read
195
+ end
196
+
197
+ elsif uri = bconf['database']
198
+ require 'perfectsched/backend/rdb'
199
+ table = bconf['table'] || "perfectsched"
200
+ backend = PerfectSched::RDBBackend.new(uri, table)
201
+
202
+ else
203
+ $stderr.puts "Invalid configuration file: backend section is required"
204
+ exit 1
205
+ end
206
+
207
+ # queue
208
+ bconf = yaml['queue']
209
+ if domain = bconf['simpledb']
210
+ require 'perfectqueue/backend/simpledb'
211
+ key_id = bconf['aws_key_id'] || ENV['AWS_ACCESS_KEY_ID']
212
+ secret_key = bconf['aws_secret_key'] || ENV['AWS_SECRET_ACCESS_KEY']
213
+ queue = PerfectQueue::SimpleDBBackend.new(key_id, secret_key, domain)
214
+
215
+ elsif uri = bconf['database']
216
+ require 'perfectqueue/backend/rdb'
217
+ table = bconf['table'] || "perfectqueue"
218
+ queue = PerfectQueue::RDBBackend.new(uri, table)
219
+
220
+ else
221
+ $stderr.puts "Invalid configuration file: queue section is required"
222
+ exit 1
223
+ end
224
+
225
+ require 'logger'
226
+
227
+ case type
228
+ when :list
229
+ format = "%26s %20s %8s %26s %26s %s"
230
+ puts format % ["id", "schedule", "delay", "next time", "next run", "data"]
231
+ n = 0
232
+ backend.list {|id,cron,delay,data,next_time,timeout|
233
+ puts format % [id, cron, delay, Time.at(next_time), Time.at(timeout), data]
234
+ n += 1
235
+ }
236
+ puts "#{n} entries."
237
+
238
+ when :delete
239
+ deleted = backend.delete(id)
240
+ if deleted
241
+ puts "Schedule id=#{id} is deleted."
242
+ else
243
+ puts "Schedule id=#{id} does not exist."
244
+ exit 1
245
+ end
246
+
247
+ when :add
248
+ cron = add_conf[:cron]
249
+ data = add_conf[:data]
250
+ delay = add_conf[:delay]
251
+ start = add_conf[:start] || Time.now.to_i
252
+
253
+ added = backend.add(id, cron, delay, data, start)
254
+ if added
255
+ puts "Schedule id=#{id} is added."
256
+ else
257
+ puts "Schedule id=#{id} already exists."
258
+ exit 1
259
+ end
260
+
261
+ when :modify_sched, :modify_delay, :modify_data
262
+ cron, delay, data = backend.get(id)
263
+ unless cron
264
+ puts "Schedule id=#{id} does not exist."
265
+ exit 1
266
+ end
267
+
268
+ case type
269
+ when :modify_sched
270
+ cron = add_conf[:cron]
271
+ modified = backend.modify_sched(id, cron, delay)
272
+
273
+ when :modify_delay
274
+ delay = add_conf[:delay]
275
+ modified = backend.modify_sched(id, cron, delay)
276
+
277
+ when :modify_data
278
+ data = add_conf[:data]
279
+ modified = backend.modify_data(id, data)
280
+ end
281
+
282
+ if modified
283
+ puts "Schedule id=#{id} is modified."
284
+ else
285
+ puts "Schedule id=#{id} does not exist."
286
+ exit 1
287
+ end
288
+
289
+ when :run
290
+ if conf[:daemon]
291
+ exit!(0) if fork
292
+ Process.setsid
293
+ exit!(0) if fork
294
+ File.umask(0)
295
+ STDIN.reopen("/dev/null")
296
+ STDOUT.reopen("/dev/null", "w")
297
+ STDERR.reopen("/dev/null", "w")
298
+ File.open(conf[:daemon], "w") {|f|
299
+ f.write Process.pid.to_s
300
+ }
301
+ end
302
+
303
+ if log_file = conf[:log]
304
+ log_out = File.open(conf[:log], "a")
305
+ else
306
+ log_out = STDOUT
307
+ end
308
+
309
+ log = Logger.new(log_out)
310
+ if conf[:verbose]
311
+ log.level = Logger::DEBUG
312
+ else
313
+ log.level = Logger::INFO
314
+ end
315
+
316
+ engine = PerfectSched::Engine.new(backend, queue, log, conf)
317
+
318
+ trap :INT do
319
+ log.info "shutting down..."
320
+ engine.stop
321
+ end
322
+
323
+ trap :TERM do
324
+ log.info "shutting down..."
325
+ engine.stop
326
+ end
327
+
328
+ trap :HUP do
329
+ if log_file
330
+ log_out.reopen(log_file, "a")
331
+ end
332
+ end
333
+
334
+ log.info "PerfectSched-#{PerfectSched::VERSION}"
335
+
336
+ begin
337
+ engine.run
338
+ engine.shutdown
339
+ rescue
340
+ log.error $!.to_s
341
+ $!.backtrace.each {|x|
342
+ log.error " #{x}"
343
+ }
344
+ exit 1
345
+ end
346
+ end
347
+
@@ -0,0 +1,26 @@
1
+
2
+ module PerfectSched
3
+
4
+
5
+ class CronCalc
6
+ def initialize
7
+ require 'cron-spec'
8
+ # TODO optimize
9
+ end
10
+
11
+ def next_time(cron, time)
12
+ t = Time.at(time)
13
+ tab = CronSpec::CronSpecification.new(cron)
14
+ while true
15
+ t += 60
16
+ if tab.is_specification_in_effect?(t)
17
+ return t.to_i
18
+ end
19
+ # FIXME break
20
+ end
21
+ end
22
+ end
23
+
24
+
25
+ end
26
+
@@ -0,0 +1,71 @@
1
+
2
+ module PerfectSched
3
+
4
+
5
+ class Engine
6
+ def initialize(backend, queue, log, conf)
7
+ require 'time'
8
+ @backend = backend
9
+ @queue = queue
10
+ @log = log
11
+
12
+ @timeout = conf[:timeout]
13
+ @poll_interval = conf[:poll_interval] || 1
14
+
15
+ @croncalc = CronCalc.new
16
+ @finished = false
17
+ end
18
+
19
+ def finished?
20
+ @finished
21
+ end
22
+
23
+ def run
24
+ until finished?
25
+ @log.debug "polling... #{@timeout} #{@poll_interval}"
26
+ token, task = @backend.acquire(Time.now.to_i+@timeout)
27
+
28
+ unless token
29
+ sleep @poll_interval
30
+ next
31
+ end
32
+
33
+ process(token, task)
34
+ end
35
+ end
36
+
37
+ def process(token, task)
38
+ @log.info "processing schedule id=#{task.id} time=#{Time.at(task.time).iso8601} at #{Time.now.iso8601}"
39
+
40
+ begin
41
+ id = gen_id(task)
42
+ @queue.submit(id, task.data)
43
+ # ignore already exists error
44
+
45
+ next_time = @croncalc.next_time(task.cron, task.time)
46
+ next_run = next_time + task.delay
47
+ @backend.finish(token, next_time, next_run)
48
+
49
+ @log.info "submitted schedule id=#{task.id}"
50
+
51
+ rescue
52
+ @log.info "failed schedule id=#{task.id}: #{$!}"
53
+ end
54
+ end
55
+
56
+ def stop
57
+ @finished = true
58
+ end
59
+
60
+ def shutdown
61
+ end
62
+
63
+ private
64
+ def gen_id(task)
65
+ "#{task.id}.#{task.time}"
66
+ end
67
+ end
68
+
69
+
70
+ end
71
+
@@ -0,0 +1,5 @@
1
+ module PerfectSched
2
+
3
+ VERSION = '0.7.0'
4
+
5
+ end
@@ -0,0 +1,188 @@
1
+ require File.dirname(__FILE__)+'/test_helper'
2
+
3
+ class BackendTest < Test::Unit::TestCase
4
+ SCHED = 120
5
+ TIMEOUT = 60
6
+ DB_PATH = File.dirname(__FILE__)+'/test.db'
7
+ DB_URI = "sqlite://#{DB_PATH}"
8
+
9
+ def clean_backend
10
+ @key_prefix = "test-#{"%08x"%rand(2**32)}-"
11
+ db = open_backend
12
+ db.list {|id,cron,delay,data,next_time,timeout|
13
+ db.delete(id)
14
+ }
15
+ FileUtils.rm_f DB_PATH
16
+ end
17
+
18
+ def open_backend
19
+ #PerfectSched::SimpleDBBackend.new(ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'], 'perfectsched-test-1').use_consistent_read
20
+ PerfectSched::RDBBackend.new(DB_URI, "perfectdb_test")
21
+ end
22
+
23
+ it 'acquire' do
24
+ clean_backend
25
+
26
+ db1 = open_backend
27
+ db2 = open_backend
28
+ db3 = open_backend
29
+
30
+ time = 0
31
+
32
+ ok = db1.add(@key_prefix+'test1', "* * * * *", 0, 'data1', time)
33
+ assert_equal true, ok
34
+
35
+ token, task = db2.acquire(180, 60)
36
+ assert_not_equal nil, task
37
+ assert_equal @key_prefix+'test1', task.id
38
+ assert_equal "* * * * *", task.cron
39
+ assert_equal 0, task.delay
40
+ assert_equal 'data1', task.data
41
+ assert_equal 60, task.time
42
+
43
+ token_, task_ = db3.acquire(180, 60)
44
+ assert_equal nil, token_
45
+
46
+ token_, task_ = db3.acquire(240, 120)
47
+ assert_equal nil, token_
48
+
49
+ ok = db2.finish(token, 120, 120)
50
+ assert_equal true, ok
51
+
52
+ token, task = db3.acquire(240, 120)
53
+ assert_not_equal nil, task
54
+ assert_equal @key_prefix+'test1', task.id
55
+ assert_equal "* * * * *", task.cron
56
+ assert_equal 0, task.delay
57
+ assert_equal 'data1', task.data
58
+ assert_equal 120, task.time
59
+ end
60
+
61
+ it 'timeout' do
62
+ clean_backend
63
+
64
+ db1 = open_backend
65
+ db2 = open_backend
66
+ db3 = open_backend
67
+
68
+ time = 0
69
+
70
+ ok = db1.add(@key_prefix+'test1', "* * * * *", 0, 'data1', time)
71
+ assert_equal true, ok
72
+
73
+ token, task = db2.acquire(180, 60)
74
+ assert_not_equal nil, task
75
+ assert_equal @key_prefix+'test1', task.id
76
+ assert_equal "* * * * *", task.cron
77
+ assert_equal 0, task.delay
78
+ assert_equal 'data1', task.data
79
+ assert_equal 60, task.time
80
+
81
+ token, task = db3.acquire(240, 180)
82
+ assert_not_equal nil, task
83
+ assert_equal @key_prefix+'test1', task.id
84
+ assert_equal "* * * * *", task.cron
85
+ assert_equal 0, task.delay
86
+ assert_equal 'data1', task.data
87
+ assert_equal 60, task.time
88
+ end
89
+
90
+ it 'delay' do
91
+ clean_backend
92
+
93
+ db1 = open_backend
94
+ db2 = open_backend
95
+ db3 = open_backend
96
+
97
+ time = 0
98
+
99
+ ok = db1.add(@key_prefix+'test1', "* * * * *", 30, 'data1', time)
100
+ assert_equal true, ok
101
+
102
+ token_, task_ = db2.acquire(180, 60)
103
+ assert_equal nil, token_
104
+
105
+ token, task = db2.acquire(210, 90)
106
+ assert_not_equal nil, task
107
+ assert_equal @key_prefix+'test1', task.id
108
+ assert_equal "* * * * *", task.cron
109
+ assert_equal 30, task.delay
110
+ assert_equal 'data1', task.data
111
+ assert_equal 60, task.time
112
+ end
113
+
114
+ it 'invalid format' do
115
+ clean_backend
116
+
117
+ db1 = open_backend
118
+
119
+ assert_raise(RuntimeError) do
120
+ db1.add('k', '???', 0, 'data1', 0)
121
+ end
122
+
123
+ assert_raise(RuntimeError) do
124
+ db1.add('k', '* * * * * *', 0, 'data1', 0)
125
+ end
126
+ end
127
+
128
+ it 'unique id' do
129
+ clean_backend
130
+
131
+ db1 = open_backend
132
+ time = 0
133
+ key = @key_prefix+'test1'
134
+
135
+ ok = db1.add(key, "* * * * *", 0, 'data1', time)
136
+ assert_equal true, ok
137
+
138
+ ok = db1.add(key, "* * * * *", 0, 'data1', time)
139
+ assert_not_equal true, ok
140
+
141
+ ok = db1.delete(key)
142
+ assert_equal true, ok
143
+
144
+ ok = db1.add(key, "* * * * *", 0, 'data1', time)
145
+ assert_equal true, ok
146
+ end
147
+
148
+ it 'modify' do
149
+ clean_backend
150
+
151
+ db1 = open_backend
152
+ time = 0
153
+ key = @key_prefix+'test1'
154
+
155
+ ok = db1.add(key, "* * * * *", 0, 'data1', time)
156
+ assert_equal true, ok
157
+
158
+ cron, delay, data = db1.get(key)
159
+ assert_equal "* * * * *", cron
160
+ assert_equal 0, delay
161
+ assert_equal 'data1', data
162
+
163
+ ok = db1.modify_sched(key, "* * * * 1", 10)
164
+ assert_equal true, ok
165
+
166
+ cron, delay, data = db1.get(key)
167
+ assert_equal "* * * * 1", cron
168
+ assert_equal 10, delay
169
+ assert_equal 'data1', data
170
+
171
+ ok = db1.modify_data(key, "data2")
172
+ assert_equal true, ok
173
+
174
+ cron, delay, data = db1.get(key)
175
+ assert_equal "* * * * 1", cron
176
+ assert_equal 10, delay
177
+ assert_equal 'data2', data
178
+
179
+ ok = db1.modify(key, "* * * * 2", 20, "data3")
180
+ assert_equal true, ok
181
+
182
+ cron, delay, data = db1.get(key)
183
+ assert_equal "* * * * 2", cron
184
+ assert_equal 20, delay
185
+ assert_equal 'data3', data
186
+ end
187
+ end
188
+
@@ -0,0 +1,17 @@
1
+ require 'test/unit'
2
+ $LOAD_PATH << File.dirname(__FILE__)+"/../lib"
3
+ require 'perfectsched'
4
+ require 'shellwords'
5
+ require 'perfectsched/backend/rdb'
6
+ require 'perfectsched/backend/simpledb'
7
+ require 'fileutils'
8
+
9
+ class Test::Unit::TestCase
10
+ #class << self
11
+ # alias_method :it, :test
12
+ #end
13
+ def self.it(name, &block)
14
+ define_method("test_#{name}", &block)
15
+ end
16
+ end
17
+
metadata ADDED
@@ -0,0 +1,144 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: perfectsched
3
+ version: !ruby/object:Gem::Version
4
+ hash: 3
5
+ prerelease: false
6
+ segments:
7
+ - 0
8
+ - 7
9
+ - 0
10
+ version: 0.7.0
11
+ platform: ruby
12
+ authors:
13
+ - Sadayuki Furuhashi
14
+ autorequire:
15
+ bindir: bin
16
+ cert_chain: []
17
+
18
+ date: 2011-09-03 00:00:00 +09:00
19
+ default_executable: perfectsched
20
+ dependencies:
21
+ - !ruby/object:Gem::Dependency
22
+ name: cron-spec
23
+ prerelease: false
24
+ requirement: &id001 !ruby/object:Gem::Requirement
25
+ none: false
26
+ requirements:
27
+ - - "="
28
+ - !ruby/object:Gem::Version
29
+ hash: 31
30
+ segments:
31
+ - 0
32
+ - 1
33
+ - 2
34
+ version: 0.1.2
35
+ type: :runtime
36
+ version_requirements: *id001
37
+ - !ruby/object:Gem::Dependency
38
+ name: sequel
39
+ prerelease: false
40
+ requirement: &id002 !ruby/object:Gem::Requirement
41
+ none: false
42
+ requirements:
43
+ - - ~>
44
+ - !ruby/object:Gem::Version
45
+ hash: 111
46
+ segments:
47
+ - 3
48
+ - 26
49
+ - 0
50
+ version: 3.26.0
51
+ type: :runtime
52
+ version_requirements: *id002
53
+ - !ruby/object:Gem::Dependency
54
+ name: aws-sdk
55
+ prerelease: false
56
+ requirement: &id003 !ruby/object:Gem::Requirement
57
+ none: false
58
+ requirements:
59
+ - - ~>
60
+ - !ruby/object:Gem::Version
61
+ hash: 17
62
+ segments:
63
+ - 1
64
+ - 1
65
+ - 1
66
+ version: 1.1.1
67
+ type: :runtime
68
+ version_requirements: *id003
69
+ - !ruby/object:Gem::Dependency
70
+ name: perfectqueue
71
+ prerelease: false
72
+ requirement: &id004 !ruby/object:Gem::Requirement
73
+ none: false
74
+ requirements:
75
+ - - ~>
76
+ - !ruby/object:Gem::Version
77
+ hash: 3
78
+ segments:
79
+ - 0
80
+ - 7
81
+ - 0
82
+ version: 0.7.0
83
+ type: :runtime
84
+ version_requirements: *id004
85
+ description:
86
+ email: frsyuki@gmail.com
87
+ executables:
88
+ - perfectsched
89
+ extensions: []
90
+
91
+ extra_rdoc_files:
92
+ - ChangeLog
93
+ - README.rdoc
94
+ files:
95
+ - bin/perfectsched
96
+ - lib/perfectsched.rb
97
+ - lib/perfectsched/backend.rb
98
+ - lib/perfectsched/backend/rdb.rb
99
+ - lib/perfectsched/backend/simpledb.rb
100
+ - lib/perfectsched/command/perfectsched.rb
101
+ - lib/perfectsched/croncalc.rb
102
+ - lib/perfectsched/engine.rb
103
+ - lib/perfectsched/version.rb
104
+ - ChangeLog
105
+ - README.rdoc
106
+ - test/backend_test.rb
107
+ - test/test_helper.rb
108
+ has_rdoc: true
109
+ homepage: https://github.com/treasure-data/perfectsched
110
+ licenses: []
111
+
112
+ post_install_message:
113
+ rdoc_options:
114
+ - --charset=UTF-8
115
+ require_paths:
116
+ - lib
117
+ required_ruby_version: !ruby/object:Gem::Requirement
118
+ none: false
119
+ requirements:
120
+ - - ">="
121
+ - !ruby/object:Gem::Version
122
+ hash: 3
123
+ segments:
124
+ - 0
125
+ version: "0"
126
+ required_rubygems_version: !ruby/object:Gem::Requirement
127
+ none: false
128
+ requirements:
129
+ - - ">="
130
+ - !ruby/object:Gem::Version
131
+ hash: 3
132
+ segments:
133
+ - 0
134
+ version: "0"
135
+ requirements: []
136
+
137
+ rubyforge_project:
138
+ rubygems_version: 1.3.7
139
+ signing_key:
140
+ specification_version: 3
141
+ summary: Highly available distributed cron built on RDBMS or SimpleDB
142
+ test_files:
143
+ - test/backend_test.rb
144
+ - test/test_helper.rb