manband 0.4.0
Sign up to get free protection for your applications and to get access to all the features.
- data/LICENSE +10 -0
- data/README +108 -0
- data/bin/jobhandler.rb +135 -0
- data/bin/manband.rb +108 -0
- data/bin/remoteband.rb +76 -0
- data/bin/userband.rb +75 -0
- data/bin/workflowhandler.rb +182 -0
- data/lib/manband/bandmanager.rb +176 -0
- data/lib/manband/flowconfig.rb +87 -0
- data/lib/manband/job.rb +282 -0
- data/lib/manband/store.rb +77 -0
- data/lib/manband/user.rb +47 -0
- data/lib/manband/workflow.rb +274 -0
- metadata +175 -0
data/LICENSE
ADDED
data/README
ADDED
@@ -0,0 +1,108 @@
|
|
1
|
+
# Requirements
|
2
|
+
|
3
|
+
RabbitMQ, MYSQL with one user and one database
|
4
|
+
|
5
|
+
gem install amqp
|
6
|
+
gem install json
|
7
|
+
gem install data_mapper
|
8
|
+
gem install dm-myql-adapter
|
9
|
+
gem install dm-migrations
|
10
|
+
gem install zip
|
11
|
+
gem install mb-minion
|
12
|
+
gem install mb-aws-s3
|
13
|
+
|
14
|
+
For debug, one can use dm-sqlite-adapter. SQLite does not fit for production and concurrency issues may occur. To avoid this, only 1 workflow handler and 1 job handler should be used.
|
15
|
+
MySQL handles correctly the concurrency with lock mechanisms.
|
16
|
+
|
17
|
+
# Installation
|
18
|
+
|
19
|
+
Export or clone repository git@github.com:osallou/aws-s3.git
|
20
|
+
Copy lib/aws directory to manband lib directory (lib/manband/aws)
|
21
|
+
S3 host information can be set via jobhandler command-line
|
22
|
+
|
23
|
+
# Configuration
|
24
|
+
|
25
|
+
Some configuration can be set via a config file with the --conf option.
|
26
|
+
Program will also search for a file ".manband" in user home directory.
|
27
|
+
If none is used, program will use defaults.
|
28
|
+
An example conf file is available in test directory (conf.yaml).
|
29
|
+
|
30
|
+
For storage in S3, user credentials must be stored in database.
|
31
|
+
If user has access to the database (non shared install), the userband script will help setting up the user (see User Management).
|
32
|
+
In a shared install, one MUST use the web interface to setup credentials and launch workflows as there is no control on user id.
|
33
|
+
|
34
|
+
# Security
|
35
|
+
|
36
|
+
Each workflow is ran in a specific set of queues in a virtual host in RabbitMQ. This access is controlled via AMQP_URL credentials, restricting access only to a user or a set of users.
|
37
|
+
Scripts using database access (manband, userband) must not be used in a shared install. One should use remoteband to launch workflows (warning: workflow file must be accessible by nodes).
|
38
|
+
Access to user credentials (s3,..) should be allowed only via the web interface or a custom interface.
|
39
|
+
|
40
|
+
# Execution
|
41
|
+
|
42
|
+
Before executing the programs, one must set in environment:
|
43
|
+
|
44
|
+
export AMQP_URL="amqp://johndoe:abc123@localhost/my_vhost"
|
45
|
+
export MYSQL_URL="mysql://USER:PASSWORD@localhost/MYDB"
|
46
|
+
|
47
|
+
All programs, including the web interface need access to the database and RabbitMQ.
|
48
|
+
In a cloud environment, where each user or group of user have to launch his own instances, two solutions:
|
49
|
+
1) A master instance holds the web interface and the database. The master instance could then launch new instances (workflowhandler or jobhandler) with MYSQL_URL refering to master instance
|
50
|
+
2) For each user, provide with the AMQP queue a dedicated MySQL database. Then user can refer to this database for his handlers.
|
51
|
+
|
52
|
+
In all cases, cloud instances would need to get at boot time those 2 variables before executing the handlers or the web interface.
|
53
|
+
|
54
|
+
## Workflow Handler
|
55
|
+
|
56
|
+
It is possible to run multiple workflow handler to handler the message load
|
57
|
+
|
58
|
+
ruby -rubygems workflowhandler.rb
|
59
|
+
|
60
|
+
Use -d for message tracking.
|
61
|
+
|
62
|
+
## Job Handler
|
63
|
+
|
64
|
+
It is possible to run multiple job handlers to handler job commands
|
65
|
+
|
66
|
+
ruby -rubygems jobhandler.rb -i myhandlername
|
67
|
+
|
68
|
+
## Main program
|
69
|
+
|
70
|
+
For usage:
|
71
|
+
|
72
|
+
ruby -rubygems manband -h
|
73
|
+
|
74
|
+
|
75
|
+
To start a new workflow:
|
76
|
+
|
77
|
+
ruby manband.rb -w test/samplew.yaml
|
78
|
+
|
79
|
+
If user do not have MYSQL credentials, user can use:
|
80
|
+
|
81
|
+
ruby remoteband.rb -w test/samplew.yaml
|
82
|
+
|
83
|
+
remoteband initiate a "new" message to start a workflow without MYSQL credentials need.
|
84
|
+
|
85
|
+
|
86
|
+
## Web interface
|
87
|
+
|
88
|
+
For production mode, on port 4567:
|
89
|
+
|
90
|
+
rackup -E production -p 4567
|
91
|
+
|
92
|
+
## User management
|
93
|
+
|
94
|
+
By default, admin account has password "admin". This should be modified after installation.
|
95
|
+
|
96
|
+
userband.rb can help to add or update users without the web interface
|
97
|
+
|
98
|
+
## S3 storage
|
99
|
+
|
100
|
+
By default, bucket used is "manband". It can be specified in command-line.
|
101
|
+
|
102
|
+
## Gem creation
|
103
|
+
|
104
|
+
Execute:
|
105
|
+
rack package
|
106
|
+
|
107
|
+
Gem is created in pkg directory
|
108
|
+
Webband may use this gem, or use current files with same hierarchy
|
data/bin/jobhandler.rb
ADDED
@@ -0,0 +1,135 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'yaml'
|
3
|
+
require 'optparse'
|
4
|
+
require 'amqp'
|
5
|
+
require 'json'
|
6
|
+
require 'mb-minion'
|
7
|
+
|
8
|
+
include Minion
|
9
|
+
|
10
|
+
if ENV["MYSQL_URL"]==nil
|
11
|
+
puts "MYSQL_URL environment variable is not set exiting..."
|
12
|
+
exit 1
|
13
|
+
end
|
14
|
+
if ENV["AMQP_URL"]==nil
|
15
|
+
puts "AMQP_URL environment variable is not set, using defaults."
|
16
|
+
end
|
17
|
+
|
18
|
+
$:.push File.expand_path("../lib")
|
19
|
+
|
20
|
+
require 'manband/job.rb'
|
21
|
+
require 'manband/store.rb'
|
22
|
+
|
23
|
+
handler = Time.now.to_s
|
24
|
+
|
25
|
+
log = Logger.new(STDOUT)
|
26
|
+
log.level = Logger::INFO
|
27
|
+
|
28
|
+
options = {}
|
29
|
+
|
30
|
+
optparse = OptionParser.new do|opts|
|
31
|
+
|
32
|
+
# This displays the help screen, all programs are
|
33
|
+
# assumed to have this option.
|
34
|
+
opts.on( '-h', '--help', 'Display this screen' ) do
|
35
|
+
puts opts
|
36
|
+
exit
|
37
|
+
end
|
38
|
+
|
39
|
+
opts.on( '-i', '--id ID', 'Handler id' ) do |id|
|
40
|
+
handler = id.to_s
|
41
|
+
end
|
42
|
+
|
43
|
+
options[:queue] = ""
|
44
|
+
opts.on( '-q', '--queue QUEUE', 'Queue registration filter' ) do |queue|
|
45
|
+
options[:queue] = "."+queue
|
46
|
+
end
|
47
|
+
|
48
|
+
options[:store] = nil
|
49
|
+
opts.on( '-s', '--store S3HOST', 'S3 host information host:port:path, e.g. localhost:8773:/services/Walrus' ) do |store|
|
50
|
+
options[:store] = store
|
51
|
+
end
|
52
|
+
|
53
|
+
options[:conf] = nil
|
54
|
+
opts.on( '-c', '--conf CONFIG', 'Use config file' ) do |config|
|
55
|
+
options[:conf] = config
|
56
|
+
end
|
57
|
+
|
58
|
+
end
|
59
|
+
|
60
|
+
optparse.parse!
|
61
|
+
|
62
|
+
userconf = File.expand_path("~")+"/.manband"
|
63
|
+
if options[:conf]==nil && File.exists?(userconf)
|
64
|
+
options[:conf]=userconf
|
65
|
+
end
|
66
|
+
|
67
|
+
if options[:conf]!=nil
|
68
|
+
if !File.exists?(options[:conf])
|
69
|
+
puts "Config file does not exists"
|
70
|
+
exit 1
|
71
|
+
end
|
72
|
+
conf = YAML.load_file(options[:conf])
|
73
|
+
puts "Using configuration file "+options[:conf]
|
74
|
+
FlowConfig.sets3(conf["s3"]["host"],conf["s3"]["port"],conf["s3"]["path"])
|
75
|
+
FlowConfig.setworkdir(conf["workdir"])
|
76
|
+
end
|
77
|
+
|
78
|
+
if options[:store]!=nil
|
79
|
+
elts = options[:store].split(':')
|
80
|
+
FlowConfig.sets3(elts[0],elts[1],elts[2])
|
81
|
+
end
|
82
|
+
|
83
|
+
log.info "Start handler "+handler
|
84
|
+
|
85
|
+
job "manband.node"+options[:queue] do |args|
|
86
|
+
msg = JSON.parse(args["msg"])
|
87
|
+
if args["operation"] == OP_DESTROY
|
88
|
+
myworkflow = WorkFlow.get(msg["id"])
|
89
|
+
if myworkflow == nil
|
90
|
+
log.error "Error: workflow does not exists "+msg["id"]
|
91
|
+
else
|
92
|
+
myworkflow.clean
|
93
|
+
jobs = Job.all(:wid => msg[:id])
|
94
|
+
if jobs!=nil
|
95
|
+
jobs.destroy
|
96
|
+
end
|
97
|
+
links = JobLink.all(:wid => msg[:id])
|
98
|
+
if links!=nil
|
99
|
+
links.destroy
|
100
|
+
end
|
101
|
+
myworkflow.destroy
|
102
|
+
messages = BandMessage.all(:wid => msg[:id])
|
103
|
+
if messages != nil
|
104
|
+
messages.destroy
|
105
|
+
end
|
106
|
+
end
|
107
|
+
elsif args["operation"] == OP_CLEAN
|
108
|
+
myworkflow = WorkFlow.get(msg["id"])
|
109
|
+
if myworkflow == nil
|
110
|
+
log.error "Error: workflow does not exists "+msg["id"]
|
111
|
+
else
|
112
|
+
myworkflow.clean
|
113
|
+
end
|
114
|
+
else
|
115
|
+
myjob = Job.get(msg["id"])
|
116
|
+
if myjob == nil
|
117
|
+
log.error "Error: job does not exists "+msg["id"]
|
118
|
+
else
|
119
|
+
if args["operation"] == OP_RUN
|
120
|
+
if msg["instance"]!=nil
|
121
|
+
myjob.run(handler, msg["instance"].to_i)
|
122
|
+
else
|
123
|
+
myjob.run(handler)
|
124
|
+
end
|
125
|
+
elsif args["operation"] == OP_SKIP
|
126
|
+
myjob.skip(handler)
|
127
|
+
elsif args["operation"] == OP_STORE
|
128
|
+
storage = Storeband.new
|
129
|
+
uid = WorkFlow.get(myjob.wid).uid
|
130
|
+
storage.store(myjob,uid,msg["bucket"])
|
131
|
+
end
|
132
|
+
end
|
133
|
+
end
|
134
|
+
end
|
135
|
+
|
data/bin/manband.rb
ADDED
@@ -0,0 +1,108 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'yaml'
|
3
|
+
require 'optparse'
|
4
|
+
require 'amqp'
|
5
|
+
require 'json'
|
6
|
+
require 'mb-minion'
|
7
|
+
|
8
|
+
include Minion
|
9
|
+
|
10
|
+
$:.push File.expand_path("../lib")
|
11
|
+
|
12
|
+
require 'manband/workflow.rb'
|
13
|
+
require 'manband/flowconfig.rb'
|
14
|
+
require 'manband/job.rb'
|
15
|
+
require 'manband/bandmanager.rb'
|
16
|
+
|
17
|
+
log = Logger.new(STDOUT)
|
18
|
+
log.level = Logger::INFO
|
19
|
+
|
20
|
+
options = {}
|
21
|
+
|
22
|
+
optparse = OptionParser.new do|opts|
|
23
|
+
|
24
|
+
# This displays the help screen, all programs are
|
25
|
+
# assumed to have this option.
|
26
|
+
opts.on( '-h', '--help', 'Display this screen' ) do
|
27
|
+
puts opts
|
28
|
+
exit
|
29
|
+
end
|
30
|
+
|
31
|
+
opts.on( '-s', '--show WORKFLOW', 'show workflow with id WORKFLOW') do |workflow|
|
32
|
+
workflow = WorkFlow.get(workflow)
|
33
|
+
puts "Workflow "+workflow.id.to_s
|
34
|
+
puts " -- start: "+workflow.created_at.to_s
|
35
|
+
puts " -- end : "+workflow.terminated_at.to_s
|
36
|
+
jobs = Job.all(:wid => workflow.id)
|
37
|
+
jobs.each do |job|
|
38
|
+
if job.node != "root"
|
39
|
+
puts " ---- "+job.id.to_s+","+job.node+","+job.status.to_s
|
40
|
+
puts " ------ Workdir: "+job.workdir unless job.workdir == nil
|
41
|
+
puts " ------ Handler: "+job.handler unless job.handler == nil
|
42
|
+
end
|
43
|
+
end
|
44
|
+
end
|
45
|
+
|
46
|
+
|
47
|
+
options[:workflow] = nil
|
48
|
+
opts.on( '-w', '--workflow FILE', 'Use input worflow') do |file|
|
49
|
+
puts "using "+file
|
50
|
+
options[:workflow] = file
|
51
|
+
end
|
52
|
+
|
53
|
+
options[:conf] = nil
|
54
|
+
opts.on( '-c', '--conf CONFIG', 'Use config file' ) do |config|
|
55
|
+
options[:conf] = config
|
56
|
+
end
|
57
|
+
|
58
|
+
|
59
|
+
options[:uid] = 'admin'
|
60
|
+
opts.on( '-u', '--user UID', 'User id') do |uid|
|
61
|
+
options[:uid] = uid
|
62
|
+
end
|
63
|
+
|
64
|
+
options[:bucket] = 'manband'
|
65
|
+
opts.on( '-b', '--bucket BUCKET', 'S3 bucket if storage is used') do |bucket|
|
66
|
+
options[:bucket] = bucket
|
67
|
+
end
|
68
|
+
|
69
|
+
options[:var] = Array.new
|
70
|
+
opts.on( '-v', '--var KEY1=VAL1,KEY2=VAL2,KEY3=VAL3', Array, 'Runtime variables') do |runtimevars|
|
71
|
+
options[:var] = runtimevars
|
72
|
+
end
|
73
|
+
|
74
|
+
options[:debug] = false
|
75
|
+
opts.on( '-d', '--debug', 'Debug mode, do not execute commands') do
|
76
|
+
options[:debug] = true
|
77
|
+
end
|
78
|
+
end
|
79
|
+
|
80
|
+
optparse.parse!
|
81
|
+
|
82
|
+
if options[:workflow]==nil
|
83
|
+
exit
|
84
|
+
end
|
85
|
+
|
86
|
+
userconf = File.expand_path("~")+"/.manband"
|
87
|
+
if options[:conf]==nil && File.exists?(userconf)
|
88
|
+
options[:conf]=userconf
|
89
|
+
end
|
90
|
+
|
91
|
+
if options[:conf]!=nil
|
92
|
+
if !File.exists?(options[:conf])
|
93
|
+
puts "Config file does not exists"
|
94
|
+
exit 1
|
95
|
+
end
|
96
|
+
conf = YAML.load_file(options[:conf])
|
97
|
+
puts "Using configuration file "+options[:conf]
|
98
|
+
FlowConfig.setworkdir(conf["workdir"])
|
99
|
+
end
|
100
|
+
|
101
|
+
|
102
|
+
wid = BandManager.launch(options[:workflow],options[:var],options[:uid],options[:bucket],options[:debug])
|
103
|
+
|
104
|
+
if wid == nil
|
105
|
+
log.error "Could not launch the workflow"
|
106
|
+
else
|
107
|
+
log.info "Workflow "+wid.to_s+" starting..."
|
108
|
+
end
|
data/bin/remoteband.rb
ADDED
@@ -0,0 +1,76 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'yaml'
|
3
|
+
require 'optparse'
|
4
|
+
require 'amqp'
|
5
|
+
require 'json'
|
6
|
+
require 'mb-minion'
|
7
|
+
|
8
|
+
include Minion
|
9
|
+
|
10
|
+
|
11
|
+
log = Logger.new(STDOUT)
|
12
|
+
log.level = Logger::INFO
|
13
|
+
|
14
|
+
options = {}
|
15
|
+
|
16
|
+
optparse = OptionParser.new do|opts|
|
17
|
+
|
18
|
+
# This displays the help screen, all programs are
|
19
|
+
# assumed to have this option.
|
20
|
+
opts.on( '-h', '--help', 'Display this screen' ) do
|
21
|
+
puts opts
|
22
|
+
exit
|
23
|
+
end
|
24
|
+
|
25
|
+
opts.on( '-s', '--show WORKFLOW', 'show workflow with id WORKFLOW') do |workflow|
|
26
|
+
workflow = WorkFlow.get(workflow)
|
27
|
+
puts "Workflow "+workflow.id.to_s
|
28
|
+
puts " -- start: "+workflow.created_at.to_s
|
29
|
+
puts " -- end : "+workflow.terminated_at.to_s
|
30
|
+
jobs = Job.all(:wid => workflow.id)
|
31
|
+
jobs.each do |job|
|
32
|
+
if job.node != "root"
|
33
|
+
puts " ---- "+job.id.to_s+","+job.node+","+job.status.to_s
|
34
|
+
puts " ------ Workdir: "+job.workdir unless job.workdir == nil
|
35
|
+
puts " ------ Handler: "+job.handler unless job.handler == nil
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
|
40
|
+
|
41
|
+
options[:workflow] = nil
|
42
|
+
opts.on( '-w', '--workflow FILE', 'Use input worflow') do |file|
|
43
|
+
puts "using "+file
|
44
|
+
options[:workflow] = file
|
45
|
+
end
|
46
|
+
|
47
|
+
options[:uid] = 'admin'
|
48
|
+
opts.on( '-u', '--user UID', 'User id') do |uid|
|
49
|
+
options[:uid] = uid
|
50
|
+
end
|
51
|
+
|
52
|
+
options[:var] = Array.new
|
53
|
+
opts.on( '-v', '--var KEY1=VAL1,KEY2=VAL2,KEY3=VAL3', Array, 'Runtime variables') do |runtimevars|
|
54
|
+
options[:var] = runtimevars
|
55
|
+
end
|
56
|
+
|
57
|
+
options[:bucket] = 'manband'
|
58
|
+
opts.on( '-b', '--bucket BUCKET', 'S3 bucket if storage is used') do |bucket|
|
59
|
+
options[:bucket] = bucket
|
60
|
+
end
|
61
|
+
|
62
|
+
|
63
|
+
end
|
64
|
+
|
65
|
+
optparse.parse!
|
66
|
+
|
67
|
+
if options[:workflow]==nil
|
68
|
+
exit
|
69
|
+
end
|
70
|
+
|
71
|
+
msg = Hash.new
|
72
|
+
msg["wfile"]=options[:workflow]
|
73
|
+
msg["var"]=options[:var]
|
74
|
+
msg["uid"]=options[:uid]
|
75
|
+
msg["bucket"]=options[:bucket]
|
76
|
+
Minion.enqueue("manband.master", { "operation" => "new", "msg" => msg.to_json })
|
data/bin/userband.rb
ADDED
@@ -0,0 +1,75 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'yaml'
|
3
|
+
require 'optparse'
|
4
|
+
require 'json'
|
5
|
+
require 'logger'
|
6
|
+
|
7
|
+
$:.push File.expand_path("../lib")
|
8
|
+
|
9
|
+
require 'manband/flowconfig.rb'
|
10
|
+
|
11
|
+
log = Logger.new(STDOUT)
|
12
|
+
log.level = Logger::INFO
|
13
|
+
|
14
|
+
user = nil
|
15
|
+
|
16
|
+
options = {}
|
17
|
+
|
18
|
+
optparse = OptionParser.new do|opts|
|
19
|
+
|
20
|
+
# This displays the help screen, all programs are
|
21
|
+
# assumed to have this option.
|
22
|
+
opts.on( '-h', '--help', 'Display this screen' ) do
|
23
|
+
puts opts
|
24
|
+
exit
|
25
|
+
end
|
26
|
+
|
27
|
+
options[:login] = nil
|
28
|
+
opts.on( '-u', '--user UID', 'User id') do |uid|
|
29
|
+
options[:login] = uid
|
30
|
+
end
|
31
|
+
|
32
|
+
options[:password] = nil
|
33
|
+
opts.on( '-p', '--password PASSWORD', 'User password') do |passwd|
|
34
|
+
options[:password] = passwd
|
35
|
+
end
|
36
|
+
|
37
|
+
options[:access] = nil
|
38
|
+
opts.on( '-a', '--access S3_ACCESS', 'S3 Access key') do |access|
|
39
|
+
options[:access] = access
|
40
|
+
end
|
41
|
+
|
42
|
+
options[:secret] = nil
|
43
|
+
opts.on( '-s', '--secret S3_SECRET', 'S3 secret key') do |secret|
|
44
|
+
options[:secret] = secret
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
optparse.parse!
|
49
|
+
|
50
|
+
if options[:login]==nil
|
51
|
+
puts "User id is mandatory"
|
52
|
+
exit
|
53
|
+
end
|
54
|
+
|
55
|
+
user = User.get(options[:login])
|
56
|
+
|
57
|
+
if user == nil
|
58
|
+
user = User.new(:login => options[:login], :password => Digest::SHA1.hexdigest(options[:password]), :s3_access => options[:access], :s3_secret => options[:secret] )
|
59
|
+
user.save
|
60
|
+
log.info "User "+user.id+" created"
|
61
|
+
else
|
62
|
+
if options[:password] == nil
|
63
|
+
options[:password] = user.password
|
64
|
+
end
|
65
|
+
if options[:access] == nil
|
66
|
+
options[:access] = user.s3_access
|
67
|
+
end
|
68
|
+
if options[:secret] == nil
|
69
|
+
options[:secret] = user.s3_secret
|
70
|
+
end
|
71
|
+
user.update(:password => Digest::SHA1.hexdigest(options[:password]), :s3_access => options[:access], :s3_secret => options[:secret])
|
72
|
+
log.info "User "+user.id.to_s+" updated"
|
73
|
+
end
|
74
|
+
|
75
|
+
|
@@ -0,0 +1,182 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'yaml'
|
3
|
+
require 'optparse'
|
4
|
+
require 'amqp'
|
5
|
+
require 'json'
|
6
|
+
require 'mb-minion'
|
7
|
+
|
8
|
+
include Minion
|
9
|
+
|
10
|
+
if ENV["MYSQL_URL"]==nil
|
11
|
+
puts "MYSQL_URL environment variable is not set exiting..."
|
12
|
+
exit 1
|
13
|
+
end
|
14
|
+
if ENV["AMQP_URL"]==nil
|
15
|
+
puts "AMQP_URL environment variable is not set, using defaults."
|
16
|
+
end
|
17
|
+
|
18
|
+
$:.push File.expand_path("../lib")
|
19
|
+
|
20
|
+
require 'manband/workflow.rb'
|
21
|
+
require 'manband/job.rb'
|
22
|
+
require 'manband/bandmanager.rb'
|
23
|
+
|
24
|
+
log = Logger.new(STDOUT)
|
25
|
+
log.level = Logger::INFO
|
26
|
+
|
27
|
+
@@options = {}
|
28
|
+
|
29
|
+
optparse = OptionParser.new do|opts|
|
30
|
+
|
31
|
+
# This displays the help screen, all programs are
|
32
|
+
# assumed to have this option.
|
33
|
+
opts.on( '-h', '--help', 'Display this screen' ) do
|
34
|
+
puts opts
|
35
|
+
exit
|
36
|
+
end
|
37
|
+
|
38
|
+
@@options[:conf] = nil
|
39
|
+
opts.on( '-c', '--conf CONFIG', 'Use config file' ) do |config|
|
40
|
+
@@options[:conf] = config
|
41
|
+
end
|
42
|
+
|
43
|
+
|
44
|
+
@@options[:debug]=false
|
45
|
+
opts.on( '-d', '--debug', 'Debug mode, store all messages in database') do
|
46
|
+
@@options[:debug]=true
|
47
|
+
Job.debug(true)
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
optparse.parse!
|
52
|
+
|
53
|
+
userconf = File.expand_path("~")+"/.manband"
|
54
|
+
if @@options[:conf]==nil && File.exists?(userconf)
|
55
|
+
options[:conf]=userconf
|
56
|
+
end
|
57
|
+
|
58
|
+
if @@options[:conf]!=nil
|
59
|
+
if !File.exists?(@@options[:conf])
|
60
|
+
puts "Config file does not exists"
|
61
|
+
exit 1
|
62
|
+
end
|
63
|
+
conf = YAML.load_file(@@options[:conf])
|
64
|
+
puts "Using configuration file "+@@options[:conf]
|
65
|
+
FlowConfig.sets3(conf["s3"]["host"],conf["s3"]["port"],conf["s3"]["path"])
|
66
|
+
FlowConfig.setworkdir(conf["workdir"])
|
67
|
+
end
|
68
|
+
|
69
|
+
job "manband.master" do |args|
|
70
|
+
msg = JSON.parse(args["msg"])
|
71
|
+
if args["operation"] == OP_NEW
|
72
|
+
BandManager.launch(msg["wfile"],msg["var"],msg["uid"],msg["bucket"])
|
73
|
+
end
|
74
|
+
if args["operation"] == OP_START
|
75
|
+
savemessage(msg["id"],args["operation"],args["msg"])
|
76
|
+
workflow = WorkFlow.get(msg["id"])
|
77
|
+
if workflow == nil
|
78
|
+
log.error "Error: workflow does not exists "+msg["id"]
|
79
|
+
else
|
80
|
+
workflow.update(:status => STATUS_RUNNING)
|
81
|
+
myjob = Job.get(msg["root"])
|
82
|
+
myjob.runnext
|
83
|
+
end
|
84
|
+
end
|
85
|
+
if args["operation"] == OP_SKIP
|
86
|
+
savemessage(msg["id"],args["operation"],args["msg"])
|
87
|
+
workflow = WorkFlow.get(msg["id"])
|
88
|
+
if workflow == nil
|
89
|
+
log.error "Error: workflow does not exists "+msg["id"]
|
90
|
+
else
|
91
|
+
workflow.update(:status => STATUS_RUNNING)
|
92
|
+
myjob = Job.get(msg["root"])
|
93
|
+
myjob.runnext(true)
|
94
|
+
end
|
95
|
+
end
|
96
|
+
if args["operation"] == OP_ERROR
|
97
|
+
savemessage(msg["workflow"],args["operation"],args["msg"])
|
98
|
+
myjob = Job.get(msg["id"])
|
99
|
+
if myjob == nil
|
100
|
+
log.error "Error: job does not exists "+msg["id"]
|
101
|
+
else
|
102
|
+
log.debug "Job "+msg["id"]+" in error"
|
103
|
+
instance = 0
|
104
|
+
if msg["instance"]!=nil
|
105
|
+
instance = msg["instance"]
|
106
|
+
end
|
107
|
+
myjob.error!(instance)
|
108
|
+
end
|
109
|
+
end
|
110
|
+
if args["operation"] == OP_FINISH
|
111
|
+
savemessage(msg["workflow"],args["operation"],args["msg"])
|
112
|
+
myjob = Job.get(msg["id"])
|
113
|
+
if myjob == nil
|
114
|
+
log.error "Error: job does not exists "+msg["id"]
|
115
|
+
else
|
116
|
+
if msg["handler"]!=nil && myjob.handler != msg["handler"]
|
117
|
+
log.debug "Finish received from a different handler "+myjob.handler+" vs "+msg["handler"]+", skipping message"
|
118
|
+
else
|
119
|
+
myjob.finish
|
120
|
+
if myjob.isover?
|
121
|
+
myjob.runnext
|
122
|
+
end
|
123
|
+
end
|
124
|
+
end
|
125
|
+
end
|
126
|
+
if args["operation"] == OP_WSUSPEND
|
127
|
+
savemessage(msg["id"],args["operation"],args["msg"])
|
128
|
+
workflow = WorkFlow.get(msg["id"])
|
129
|
+
if workflow == nil
|
130
|
+
log.error "Error: workflow does not exists "+msg["id"]
|
131
|
+
else
|
132
|
+
workflow.update(:status => STATUS_SUSPEND)
|
133
|
+
end
|
134
|
+
end
|
135
|
+
if args["operation"] == OP_WRESUME
|
136
|
+
savemessage(msg["id"],args["operation"],args["msg"])
|
137
|
+
workflow = WorkFlow.get(msg["id"])
|
138
|
+
if workflow == nil
|
139
|
+
log.error "Error: workflow does not exists "+msg["id"]
|
140
|
+
else
|
141
|
+
log.info "Resuming workflow "+msg["id"]
|
142
|
+
workflow.update(:status => STATUS_RUNNING)
|
143
|
+
suspendedjobs = Job.all(:status => STATUS_SUSPEND, :wid => workflow.id)
|
144
|
+
suspendedjobs.each do |job|
|
145
|
+
job.resume
|
146
|
+
end
|
147
|
+
errorjobs = Job.all(:status => STATUS_ERROR, :wid => workflow.id)
|
148
|
+
errorjobs.each do |job|
|
149
|
+
job.resume
|
150
|
+
end
|
151
|
+
end
|
152
|
+
end
|
153
|
+
if args["operation"] == OP_JSUSPEND
|
154
|
+
savemessage(msg["workflow"],args["operation"],args["msg"])
|
155
|
+
curjob = Job.get(msg["id"])
|
156
|
+
if curjob == nil
|
157
|
+
log.error "Error: Job does not exists "+msg["id"]
|
158
|
+
else
|
159
|
+
if curjob.status != STATUS_SKIP
|
160
|
+
curjob.update(:status => STATUS_SUSPEND)
|
161
|
+
end
|
162
|
+
end
|
163
|
+
end
|
164
|
+
if args["operation"] == OP_JRESUME
|
165
|
+
savemessage(msg["workflow"],args["operation"],args["msg"])
|
166
|
+
curjob = Job.get(msg["id"])
|
167
|
+
if curjob == nil
|
168
|
+
log.error "Error: Job does not exists "+msg["id"]
|
169
|
+
else
|
170
|
+
log.info "Resuming job "+msg["id"]
|
171
|
+
curjob.resume
|
172
|
+
end
|
173
|
+
end
|
174
|
+
end
|
175
|
+
|
176
|
+
def savemessage(id,operation,msg)
|
177
|
+
if @@options[:debug] == false
|
178
|
+
return
|
179
|
+
end
|
180
|
+
bmsg = BandMessage.new(:wid => id, :message => '{ "operation" => '+operation+', "msg" => '+msg+' }' )
|
181
|
+
bmsg.save
|
182
|
+
end
|