mobilize-ssh 1.3 → 1.21
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +8 -65
- data/lib/mobilize-ssh.rb +1 -1
- data/lib/mobilize-ssh/extensions/string.rb +5 -0
- data/lib/mobilize-ssh/handlers/ssh.rb +75 -43
- data/lib/mobilize-ssh/version.rb +1 -1
- data/mobilize-ssh.gemspec +1 -1
- data/test/code2.sh +1 -1
- data/test/mobilize-ssh_test.rb +2 -5
- data/test/ssh_job_rows.yml +6 -13
- metadata +11 -6
- data/lib/mobilize-ssh/handlers/git.rb +0 -111
- data/lib/mobilize-ssh/helpers/ssh_helper.rb +0 -37
data/README.md
CHANGED
@@ -13,7 +13,6 @@ Table Of Contents
|
|
13
13
|
* [Install Dirs and Files](#section_Install_Dirs_and_Files)
|
14
14
|
* [Configure](#section_Configure)
|
15
15
|
* [Ssh](#section_Configure_Ssh)
|
16
|
-
* [Git](#section_Configure_Git)
|
17
16
|
* [Start](#section_Start)
|
18
17
|
* [Create Job](#section_Start_Create_Job)
|
19
18
|
* [Run Test](#section_Start_Run_Test)
|
@@ -80,6 +79,9 @@ Configure
|
|
80
79
|
### Configure Ssh
|
81
80
|
|
82
81
|
The Ssh configuration consists of:
|
82
|
+
* tmp_file_dir, which is where files will be stored before being scp'd
|
83
|
+
over to the nodes. They will be deleted afterwards, unless the job
|
84
|
+
fails in mid-copy. By default this is tmp/file/.
|
83
85
|
* nodes, identified by aliases, such as `test_node`. This alias is what you should
|
84
86
|
pass into the "node" param over in the ssh.run task.
|
85
87
|
* if no node is specified, commands will default to the first node listed.
|
@@ -107,6 +109,7 @@ Sample ssh.yml:
|
|
107
109
|
``` yml
|
108
110
|
---
|
109
111
|
development:
|
112
|
+
tmp_file_dir: tmp/file/
|
110
113
|
nodes:
|
111
114
|
dev_node:
|
112
115
|
sudoers:
|
@@ -123,6 +126,7 @@ development:
|
|
123
126
|
port: 22
|
124
127
|
user: gateway_user
|
125
128
|
test:
|
129
|
+
tmp_file_dir: tmp/file/
|
126
130
|
nodes:
|
127
131
|
test_node:
|
128
132
|
sudoers:
|
@@ -139,6 +143,7 @@ test:
|
|
139
143
|
port: 22
|
140
144
|
user: gateway_user
|
141
145
|
production:
|
146
|
+
tmp_file_dir: tmp/file/
|
142
147
|
nodes:
|
143
148
|
prod_node:
|
144
149
|
sudoers:
|
@@ -156,57 +161,6 @@ production:
|
|
156
161
|
user: gateway_user
|
157
162
|
```
|
158
163
|
|
159
|
-
<a name='section_Configure_Git'></a>
|
160
|
-
### Configure Git
|
161
|
-
|
162
|
-
Git configuration is not required but recommended, as it allows you to
|
163
|
-
pull files directly from public or private Git repositories.
|
164
|
-
|
165
|
-
The Git configuration consists of:
|
166
|
-
* domains, identified by aliases, such as `private` and `public`.
|
167
|
-
* domains are passed into the source parameters in the git.run task.
|
168
|
-
* if no domain is specified, commands will default to the first domain listed.
|
169
|
-
|
170
|
-
Each domain has:
|
171
|
-
* a host;
|
172
|
-
* a key (optional); If you don't need an ssh key to access the repo, remove that row from the configuration file.
|
173
|
-
* this is the relative path of the ssh key required to access the repository.
|
174
|
-
* a user, which is the user used for the git clone command.
|
175
|
-
|
176
|
-
Sample git.yml:
|
177
|
-
|
178
|
-
``` yml
|
179
|
-
---
|
180
|
-
development:
|
181
|
-
domains:
|
182
|
-
private:
|
183
|
-
host: github.<domain>.com
|
184
|
-
key: config/mobilize/ssh_private.key
|
185
|
-
user: git
|
186
|
-
public:
|
187
|
-
host: github.com
|
188
|
-
user: git
|
189
|
-
test:
|
190
|
-
domains:
|
191
|
-
private:
|
192
|
-
host: github.<domain>.com
|
193
|
-
key: config/mobilize/ssh_private.key
|
194
|
-
user: git
|
195
|
-
public:
|
196
|
-
host: github.com
|
197
|
-
user: git
|
198
|
-
production:
|
199
|
-
domains:
|
200
|
-
private:
|
201
|
-
host: github.<domain>.com
|
202
|
-
key: config/mobilize/ssh_private.key
|
203
|
-
user: git
|
204
|
-
public:
|
205
|
-
host: github.com
|
206
|
-
user: git
|
207
|
-
```
|
208
|
-
|
209
|
-
|
210
164
|
<a name='section_Start'></a>
|
211
165
|
Start
|
212
166
|
-----
|
@@ -215,21 +169,10 @@ Start
|
|
215
169
|
### Create Job
|
216
170
|
|
217
171
|
* For mobilize-ssh, the following task is available:
|
218
|
-
* ssh.run `node: <node_alias>, cmd: <command>, user: user, sources:[*<source_paths>]
|
219
|
-
* user, sources,
|
172
|
+
* ssh.run `node: <node_alias>, cmd: <command>, user: user, sources:[*<source_paths>]`, which reads sources, copies them to a temporary folder on the selected node, and runs the command inside that folder.
|
173
|
+
* user, sources, and node are optional; cmd is required.
|
220
174
|
* specifying user will cause the command to be prefixed with sudo su <user> -c.
|
221
175
|
* non-google sources will also be read as the specified user.
|
222
|
-
* git sources can be specified with syntax `git://<domain>/<repo_owner>/<repo_name>/<file_path>`.
|
223
|
-
* Accessing private repos requires that you add the Mobilize public key to the repository as a deploy key.
|
224
|
-
* there is no user-level access control for git repositories at this time.
|
225
|
-
* domain defaults to the first one listed, if not included.
|
226
|
-
* params are also optional for all of the below. They replace tokens in sources and the command.
|
227
|
-
* params are passed as a YML or JSON, as in:
|
228
|
-
* `ssh.run source:<source_path>, params:{'date':'2013-03-01', 'unit':'widgets'}`
|
229
|
-
* this example replaces all the keys, preceded by '@' in all source hqls with the value.
|
230
|
-
* The preceding '@' is used to keep from replacing instances
|
231
|
-
of "date" and "unit" in the command/source file; you should have `@date` and `@unit` in your actual HQL
|
232
|
-
if you'd like to replace those tokens.
|
233
176
|
* not specifying node will cause the command to be run on the default node.
|
234
177
|
* ssh sources can be specified with syntax
|
235
178
|
`ssh://<node><file_full_path>`. If node is omitted, default node will be used.
|
data/lib/mobilize-ssh.rb
CHANGED
@@ -6,10 +6,10 @@ require "net/scp"
|
|
6
6
|
require "mobilize-ssh/extensions/net-ssh-connection-session"
|
7
7
|
require "mobilize-ssh/extensions/net-ssh-gateway"
|
8
8
|
require "mobilize-ssh/extensions/socket"
|
9
|
+
require "mobilize-ssh/extensions/string"
|
9
10
|
|
10
11
|
module Mobilize
|
11
12
|
module Ssh
|
12
13
|
end
|
13
14
|
end
|
14
15
|
require "mobilize-ssh/handlers/ssh"
|
15
|
-
require "mobilize-ssh/handlers/git"
|
@@ -1,8 +1,45 @@
|
|
1
1
|
module Mobilize
|
2
2
|
module Ssh
|
3
|
-
|
4
|
-
|
3
|
+
def Ssh.config
|
4
|
+
Base.config('ssh')
|
5
|
+
end
|
6
|
+
|
7
|
+
def Ssh.tmp_file_dir
|
8
|
+
Ssh.config['tmp_file_dir']
|
9
|
+
end
|
10
|
+
|
11
|
+
def Ssh.host(node)
|
12
|
+
Ssh.config['nodes'][node]['host']
|
13
|
+
end
|
14
|
+
|
15
|
+
def Ssh.gateway(node)
|
16
|
+
Ssh.config['nodes'][node]['gateway']
|
17
|
+
end
|
18
|
+
|
19
|
+
def Ssh.sudoers(node)
|
20
|
+
Ssh.config['nodes'][node]['sudoers']
|
21
|
+
end
|
22
|
+
|
23
|
+
def Ssh.su_all_users(node)
|
24
|
+
Ssh.config['nodes'][node]['su_all_users']
|
25
|
+
end
|
26
|
+
|
27
|
+
def Ssh.nodes
|
28
|
+
Ssh.config['nodes'].keys
|
29
|
+
end
|
30
|
+
|
31
|
+
def Ssh.default_node
|
32
|
+
Ssh.nodes.first
|
33
|
+
end
|
34
|
+
|
35
|
+
#determine if current machine is on host domain, needs gateway if one is provided and it is not
|
36
|
+
def Ssh.needs_gateway?(node)
|
37
|
+
host_domain_name = Ssh.host(node)['name'].split(".")[-2..-1].join(".")
|
38
|
+
return true if Ssh.gateway(node) and Socket.domain_name != host_domain_name
|
39
|
+
end
|
40
|
+
|
5
41
|
def Ssh.pop_comm_dir(comm_dir,file_hash)
|
42
|
+
FileUtils.rm_r comm_dir, :force=>true
|
6
43
|
file_hash.each do |fname,fdata|
|
7
44
|
fpath = "#{comm_dir}/#{fname}"
|
8
45
|
#for now, only gz is binary
|
@@ -13,8 +50,15 @@ module Mobilize
|
|
13
50
|
return true if file_hash.keys.length>0
|
14
51
|
end
|
15
52
|
|
53
|
+
def Ssh.set_key_permissions(key_path)
|
54
|
+
#makes sure permissions are set as appropriate for ssh key
|
55
|
+
raise "could not find ssh key at #{key_path}" unless File.exists?(key_path)
|
56
|
+
File.chmod(0600,key_path) unless File.stat(key_path).mode.to_s(8)[3..5] == "600"
|
57
|
+
return true
|
58
|
+
end
|
59
|
+
|
16
60
|
# converts a source path or target path to a dst in the context of handler and stage
|
17
|
-
def Ssh.path_to_dst(path,stage_path
|
61
|
+
def Ssh.path_to_dst(path,stage_path)
|
18
62
|
has_handler = true if path.index("://")
|
19
63
|
red_path = path.split("://").last
|
20
64
|
#is user has a handler, their first path node is a node name,
|
@@ -25,7 +69,7 @@ module Mobilize
|
|
25
69
|
return Dataset.find_or_create_by_url(ssh_url)
|
26
70
|
end
|
27
71
|
#otherwise, use Gsheet
|
28
|
-
return Gsheet.path_to_dst(red_path,stage_path
|
72
|
+
return Gsheet.path_to_dst(red_path,stage_path)
|
29
73
|
end
|
30
74
|
|
31
75
|
def Ssh.url_by_path(path,user_name)
|
@@ -53,6 +97,7 @@ module Mobilize
|
|
53
97
|
def Ssh.scp(node,from_path,to_path)
|
54
98
|
name,key,port,user = Ssh.host(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
|
55
99
|
key_path = "#{Base.root}/#{key}"
|
100
|
+
Ssh.set_key_permissions(key_path)
|
56
101
|
opts = {:port=>(port || 22),:keys=>key_path}
|
57
102
|
if Ssh.needs_gateway?(node)
|
58
103
|
gname,gkey,gport,guser = Ssh.gateway(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
|
@@ -67,30 +112,25 @@ module Mobilize
|
|
67
112
|
return true
|
68
113
|
end
|
69
114
|
|
70
|
-
def Ssh.run(node,command,
|
71
|
-
|
115
|
+
def Ssh.run(node,command,user,file_hash={})
|
116
|
+
key,default_user = Ssh.host(node).ie{|h| ['key','user'].map{|k| h[k]}}
|
117
|
+
key_path = "#{Base.root}/#{key}"
|
118
|
+
Ssh.set_key_permissions(key_path)
|
72
119
|
file_hash ||= {}
|
73
|
-
run_params ||={}
|
74
120
|
#make sure the dir for this command is clear
|
75
|
-
comm_md5 = [
|
76
|
-
comm_dir =
|
77
|
-
#
|
78
|
-
run_params.each do |k,v|
|
79
|
-
command.gsub!("@#{k}",v)
|
80
|
-
file_hash.each do |name,data|
|
81
|
-
data.gsub!("@#{k}",v)
|
82
|
-
end
|
83
|
-
end
|
84
|
-
#populate comm dir with any files
|
121
|
+
comm_md5 = [user,node,command,file_hash.keys.to_s].join.to_md5
|
122
|
+
comm_dir = "#{Ssh.tmp_file_dir}#{comm_md5}"
|
123
|
+
#populate comm dir with any files
|
85
124
|
Ssh.pop_comm_dir(comm_dir,file_hash)
|
125
|
+
#move any files up to the node
|
126
|
+
rem_dir = nil
|
86
127
|
#make sure user starts in rem_dir
|
87
128
|
rem_dir = "#{comm_md5}/"
|
88
129
|
#make sure the rem_dir is gone
|
89
130
|
Ssh.fire!(node,"sudo rm -rf #{rem_dir}")
|
90
131
|
if File.exists?(comm_dir)
|
91
132
|
Ssh.scp(node,comm_dir,rem_dir)
|
92
|
-
|
93
|
-
FileUtils.rm_r(comm_dir,:force=>true)
|
133
|
+
FileUtils.rm_r comm_dir, :force=>true
|
94
134
|
else
|
95
135
|
#create folder
|
96
136
|
mkdir_command = "mkdir #{rem_dir}"
|
@@ -102,9 +142,9 @@ module Mobilize
|
|
102
142
|
Ssh.write(node,command,cmd_path)
|
103
143
|
full_cmd = "(cd #{rem_dir} && sh #{cmd_file})"
|
104
144
|
#fire_cmd runs sh on cmd_path, optionally with sudo su
|
105
|
-
if
|
145
|
+
if user != default_user
|
106
146
|
#make sure user owns the folder and all files
|
107
|
-
fire_cmd = %{sudo chown -R #{
|
147
|
+
fire_cmd = %{sudo chown -R #{user} #{rem_dir}; sudo su #{user} -c "#{full_cmd}"}
|
108
148
|
rm_cmd = %{sudo rm -rf #{rem_dir}}
|
109
149
|
else
|
110
150
|
fire_cmd = full_cmd
|
@@ -118,6 +158,7 @@ module Mobilize
|
|
118
158
|
def Ssh.fire!(node,cmd)
|
119
159
|
name,key,port,user = Ssh.host(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
|
120
160
|
key_path = "#{Base.root}/#{key}"
|
161
|
+
Ssh.set_key_permissions(key_path)
|
121
162
|
opts = {:port=>(port || 22),:keys=>key_path}
|
122
163
|
response = if Ssh.needs_gateway?(node)
|
123
164
|
gname,gkey,gport,guser = Ssh.gateway(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
|
@@ -147,15 +188,17 @@ module Mobilize
|
|
147
188
|
def Ssh.write(node,fdata,to_path,binary=false)
|
148
189
|
from_path = Ssh.tmp_file(fdata,binary)
|
149
190
|
Ssh.scp(node,from_path,to_path)
|
150
|
-
|
151
|
-
FileUtils.rm_r(from_path,:force=>true)
|
191
|
+
FileUtils.rm from_path
|
152
192
|
return true
|
153
193
|
end
|
154
194
|
|
155
195
|
def Ssh.tmp_file(fdata,binary=false,fpath=nil)
|
156
196
|
#creates a file under tmp/files with an md5 from the data
|
157
|
-
tmp_file_path = fpath || "#{
|
197
|
+
tmp_file_path = fpath || "#{Ssh.tmp_file_dir}#{(fdata + Time.now.utc.to_f.to_s).to_md5}"
|
158
198
|
write_mode = binary ? "wb" : "w"
|
199
|
+
#make sure folder is created
|
200
|
+
tmp_file_dir = tmp_file_path.split("/")[0..-2].join("/")
|
201
|
+
FileUtils.mkdir_p(tmp_file_dir)
|
159
202
|
#write data to path
|
160
203
|
File.open(tmp_file_path,write_mode) {|f| f.print(fdata)}
|
161
204
|
return tmp_file_path
|
@@ -165,7 +208,7 @@ module Mobilize
|
|
165
208
|
s = Stage.where(:path=>stage_path).first
|
166
209
|
u = s.job.runner.user
|
167
210
|
user_name = s.params['user']
|
168
|
-
node = s.params['node']
|
211
|
+
node = s.params['node']
|
169
212
|
node = Ssh.default_node unless Ssh.nodes.include?(node)
|
170
213
|
if user_name and !Ssh.sudoers(node).include?(u.name)
|
171
214
|
raise "#{u.name} does not have su permissions for this node"
|
@@ -175,27 +218,23 @@ module Mobilize
|
|
175
218
|
return user_name
|
176
219
|
end
|
177
220
|
|
178
|
-
def Ssh.file_hash_by_stage_path(stage_path
|
221
|
+
def Ssh.file_hash_by_stage_path(stage_path)
|
179
222
|
file_hash = {}
|
180
223
|
s = Stage.where(:path=>stage_path).first
|
181
224
|
u = s.job.runner.user
|
182
225
|
user_name = Ssh.user_name_by_stage_path(stage_path)
|
183
|
-
s.sources
|
226
|
+
s.sources.each do |sdst|
|
184
227
|
split_path = sdst.path.split("/")
|
185
228
|
#if path is to stage output, name with stage name
|
186
|
-
file_name = if
|
187
|
-
|
229
|
+
file_name = if split_path.last == "out" and
|
230
|
+
(1..5).to_a.map{|n| "stage#{n.to_s}"}.include?(split_path[-2].to_s)
|
188
231
|
"#{split_path[-2]}.out"
|
189
|
-
elsif (1..5).to_a.map{|n| "stage#{n.to_s}"}.include?(split_path.last[-6..-1])
|
190
|
-
#runner<jobname>stage1
|
191
|
-
"#{split_path.last[-6..-1]}.out"
|
192
232
|
else
|
193
233
|
split_path.last
|
194
234
|
end
|
195
235
|
if ["gsheet","gfile"].include?(sdst.handler)
|
196
236
|
#google drive sources are always read as the user
|
197
|
-
|
198
|
-
file_hash[file_name] = sdst.read(u.name,gdrive_slot)
|
237
|
+
file_hash[file_name] = sdst.read(u.name)
|
199
238
|
else
|
200
239
|
#other sources should be read by su-user
|
201
240
|
file_hash[file_name] = sdst.read(user_name)
|
@@ -205,24 +244,17 @@ module Mobilize
|
|
205
244
|
end
|
206
245
|
|
207
246
|
def Ssh.run_by_stage_path(stage_path)
|
208
|
-
gdrive_slot = Gdrive.slot_worker_by_path(stage_path)
|
209
|
-
#return blank response if there are no slots available
|
210
|
-
return nil unless gdrive_slot
|
211
247
|
s = Stage.where(:path=>stage_path).first
|
212
248
|
params = s.params
|
213
249
|
node, command = [params['node'],params['cmd']]
|
214
250
|
node ||= Ssh.default_node
|
215
251
|
user_name = Ssh.user_name_by_stage_path(stage_path)
|
216
|
-
file_hash = Ssh.file_hash_by_stage_path(stage_path
|
217
|
-
|
218
|
-
run_params = params['params']
|
219
|
-
result = Ssh.run(node,command,user_name,file_hash,run_params)
|
252
|
+
file_hash = Ssh.file_hash_by_stage_path(stage_path)
|
253
|
+
result = Ssh.run(node,command,user_name,file_hash)
|
220
254
|
#use Gridfs to cache result
|
221
255
|
response = {}
|
222
256
|
response['out_url'] = Dataset.write_by_url("gridfs://#{s.path}/out",result['stdout'].to_s,Gdrive.owner_name)
|
223
257
|
response['err_url'] = Dataset.write_by_url("gridfs://#{s.path}/err",result['stderr'].to_s,Gdrive.owner_name) if result['stderr'].to_s.length>0
|
224
|
-
#is an error if there is no out and there is an err, regardless of signal
|
225
|
-
result['exit_code'] = 500 if result['stdout'].to_s.strip.length==0 and result['stderr'].to_s.strip.length>0
|
226
258
|
response['signal'] = result['exit_code']
|
227
259
|
response
|
228
260
|
end
|
data/lib/mobilize-ssh/version.rb
CHANGED
data/mobilize-ssh.gemspec
CHANGED
@@ -16,7 +16,7 @@ Gem::Specification.new do |gem|
|
|
16
16
|
gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
|
17
17
|
gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
|
18
18
|
gem.require_paths = ["lib"]
|
19
|
-
gem.add_runtime_dependency "mobilize-base","1.
|
19
|
+
gem.add_runtime_dependency "mobilize-base","1.21"
|
20
20
|
gem.add_runtime_dependency "net-ssh"
|
21
21
|
gem.add_runtime_dependency "net-scp"
|
22
22
|
gem.add_runtime_dependency "net-ssh-gateway"
|
data/test/code2.sh
CHANGED
@@ -1 +1 @@
|
|
1
|
-
tail
|
1
|
+
tail syslog
|
data/test/mobilize-ssh_test.rb
CHANGED
@@ -14,8 +14,8 @@ describe "Mobilize" do
|
|
14
14
|
|
15
15
|
gdrive_slot = Mobilize::Gdrive.owner_email
|
16
16
|
puts "create user 'mobilize'"
|
17
|
-
|
18
|
-
u = Mobilize::User.where(:name=>
|
17
|
+
user = gdrive_slot.split("@").first
|
18
|
+
u = Mobilize::User.where(:name=>user).first
|
19
19
|
r = u.runner
|
20
20
|
|
21
21
|
rb_code_sheet = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/code.rb",gdrive_slot)
|
@@ -42,7 +42,6 @@ describe "Mobilize" do
|
|
42
42
|
ssh_target_sheet_1 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_1.out",gdrive_slot)
|
43
43
|
ssh_target_sheet_2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_2.out",gdrive_slot)
|
44
44
|
ssh_target_sheet_3 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_3.out",gdrive_slot)
|
45
|
-
ssh_target_sheet_4 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_4.out",gdrive_slot)
|
46
45
|
[ssh_target_sheet_1,ssh_target_sheet_2,ssh_target_sheet_3].each {|s| s.delete if s}
|
47
46
|
|
48
47
|
ssh_job_rows = ::YAML.load_file("#{Mobilize::Base.root}/test/ssh_job_rows.yml")
|
@@ -60,12 +59,10 @@ describe "Mobilize" do
|
|
60
59
|
ssh_target_sheet_1 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_1.out",gdrive_slot)
|
61
60
|
ssh_target_sheet_2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_2.out",gdrive_slot)
|
62
61
|
ssh_target_sheet_3 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_3.out",gdrive_slot)
|
63
|
-
ssh_target_sheet_4 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_4.out",gdrive_slot)
|
64
62
|
|
65
63
|
assert ssh_target_sheet_1.to_tsv.length > 100
|
66
64
|
assert ssh_target_sheet_2.to_tsv.length > 100
|
67
65
|
assert ssh_target_sheet_3.to_tsv.length > 3
|
68
|
-
assert ssh_target_sheet_4.to_tsv.length > 100
|
69
66
|
|
70
67
|
end
|
71
68
|
|
data/test/ssh_job_rows.yml
CHANGED
@@ -1,25 +1,18 @@
|
|
1
|
-
---
|
2
1
|
- name: test_ssh_1
|
3
2
|
active: true
|
4
3
|
trigger: once
|
5
4
|
status: ""
|
6
|
-
stage1: ssh.run node:"test_node", cmd:"ruby code.rb", user:"root", sources:["code.rb", "code.sh"]
|
7
|
-
stage2: gsheet.write source:"stage1", target:"test_ssh_1.out"
|
5
|
+
stage1: 'ssh.run node:"test_node", cmd:"ruby code.rb", user:"root", sources:["code.rb", "code.sh"]'
|
6
|
+
stage2: 'gsheet.write source:"stage1", target:"test_ssh_1.out"'
|
8
7
|
- name: test_ssh_2
|
9
8
|
active: true
|
10
9
|
trigger: "after test_ssh_1"
|
11
10
|
status: ""
|
12
|
-
stage1: ssh.run cmd:"sh code2.sh", user:"root", sources:["code2.sh","test_node/var/log/syslog"]
|
13
|
-
stage2: gsheet.write source:"stage1", target:"test_ssh_2.out"
|
11
|
+
stage1: 'ssh.run cmd:"sh code2.sh", user:"root", sources:["code2.sh","test_node/var/log/syslog"]'
|
12
|
+
stage2: 'gsheet.write source:"stage1", target:"test_ssh_2.out"'
|
14
13
|
- name: test_ssh_3
|
15
14
|
active: true
|
16
15
|
trigger: "after test_ssh_2"
|
17
16
|
status: ""
|
18
|
-
stage1: ssh.run cmd:"
|
19
|
-
stage2: gsheet.write source:"stage1", target:"test_ssh_3.out"
|
20
|
-
- name: test_ssh_4
|
21
|
-
active: true
|
22
|
-
trigger: "after test_ssh_3"
|
23
|
-
status: ""
|
24
|
-
stage1: ssh.run node:"test_node", user:root, sources:["git://DeNA/mobilize-ssh/test/code.rb","git://DeNA/mobilize-ssh/test/code.sh"], cmd:"ruby code.rb"
|
25
|
-
stage2: gsheet.write source:stage1, target:"test_ssh_4.out"
|
17
|
+
stage1: 'ssh.run cmd:"whoami"'
|
18
|
+
stage2: 'gsheet.write source:"stage1", target:"test_ssh_3.out"'
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: mobilize-ssh
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: '1.
|
4
|
+
version: '1.21'
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2013-
|
12
|
+
date: 2013-03-22 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: mobilize-base
|
@@ -18,7 +18,7 @@ dependencies:
|
|
18
18
|
requirements:
|
19
19
|
- - '='
|
20
20
|
- !ruby/object:Gem::Version
|
21
|
-
version: '1.
|
21
|
+
version: '1.21'
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
24
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -26,7 +26,7 @@ dependencies:
|
|
26
26
|
requirements:
|
27
27
|
- - '='
|
28
28
|
- !ruby/object:Gem::Version
|
29
|
-
version: '1.
|
29
|
+
version: '1.21'
|
30
30
|
- !ruby/object:Gem::Dependency
|
31
31
|
name: net-ssh
|
32
32
|
requirement: !ruby/object:Gem::Requirement
|
@@ -91,9 +91,8 @@ files:
|
|
91
91
|
- lib/mobilize-ssh/extensions/net-ssh-connection-session.rb
|
92
92
|
- lib/mobilize-ssh/extensions/net-ssh-gateway.rb
|
93
93
|
- lib/mobilize-ssh/extensions/socket.rb
|
94
|
-
- lib/mobilize-ssh/
|
94
|
+
- lib/mobilize-ssh/extensions/string.rb
|
95
95
|
- lib/mobilize-ssh/handlers/ssh.rb
|
96
|
-
- lib/mobilize-ssh/helpers/ssh_helper.rb
|
97
96
|
- lib/mobilize-ssh/tasks.rb
|
98
97
|
- lib/mobilize-ssh/version.rb
|
99
98
|
- lib/samples/ssh.yml
|
@@ -117,12 +116,18 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
117
116
|
- - ! '>='
|
118
117
|
- !ruby/object:Gem::Version
|
119
118
|
version: '0'
|
119
|
+
segments:
|
120
|
+
- 0
|
121
|
+
hash: 2839265652338346626
|
120
122
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
121
123
|
none: false
|
122
124
|
requirements:
|
123
125
|
- - ! '>='
|
124
126
|
- !ruby/object:Gem::Version
|
125
127
|
version: '0'
|
128
|
+
segments:
|
129
|
+
- 0
|
130
|
+
hash: 2839265652338346626
|
126
131
|
requirements: []
|
127
132
|
rubyforge_project:
|
128
133
|
rubygems_version: 1.8.25
|
@@ -1,111 +0,0 @@
|
|
1
|
-
module Mobilize
|
2
|
-
module Git
|
3
|
-
def Git.config
|
4
|
-
Base.config('git')
|
5
|
-
end
|
6
|
-
|
7
|
-
def Git.host(domain)
|
8
|
-
Git.config['domains'][domain]['host']
|
9
|
-
end
|
10
|
-
|
11
|
-
def Git.domains
|
12
|
-
Git.config['domains'].keys
|
13
|
-
end
|
14
|
-
|
15
|
-
def Git.default_domain
|
16
|
-
Git.domains.first
|
17
|
-
end
|
18
|
-
|
19
|
-
# converts a source path or target path to a dst in the context of handler and stage
|
20
|
-
def Git.path_to_dst(path,stage_path,gdrive_slot)
|
21
|
-
red_path = path.split("://").last
|
22
|
-
git_url = Git.url_by_path(red_path)
|
23
|
-
return Dataset.find_or_create_by_url(git_url)
|
24
|
-
end
|
25
|
-
|
26
|
-
def Git.url_by_path(path)
|
27
|
-
path_nodes = path.split("/")
|
28
|
-
domain = path_nodes.first.to_s
|
29
|
-
revision = "HEAD"
|
30
|
-
if Git.domains.include?(domain)
|
31
|
-
repo = path_nodes[1..2].join("/")
|
32
|
-
file_path = path_nodes[3..-1].join("/")
|
33
|
-
else
|
34
|
-
domain = Git.default_domain
|
35
|
-
repo = path_nodes[0..1].join("/")
|
36
|
-
file_path = path_nodes[2..-1].join("/")
|
37
|
-
end
|
38
|
-
url = "git://#{domain}/#{repo}/#{revision}/#{file_path}"
|
39
|
-
return url
|
40
|
-
end
|
41
|
-
|
42
|
-
#return path to tar.gz of git repo
|
43
|
-
def Git.pack(domain,repo,revision="HEAD")
|
44
|
-
repo_dir = Git.pull(domain,repo,revision)
|
45
|
-
repo_name = repo.split("/").last
|
46
|
-
tar_gz_path = "#{repo_dir}/../#{repo_name}.tar.gz"
|
47
|
-
pack_cmd = "cd #{repo_dir} && git archive #{revision} --format=tar.gz > #{tar_gz_path}"
|
48
|
-
pack_cmd.bash(true)
|
49
|
-
FileUtils.rm_r(repo_dir,:force=>true)
|
50
|
-
return tar_gz_path
|
51
|
-
end
|
52
|
-
|
53
|
-
#confirm that git file exists
|
54
|
-
def Git.exists?(url)
|
55
|
-
domain,repo,revision,file_path=[]
|
56
|
-
url.split("/").ie do |url_nodes|
|
57
|
-
domain = url_nodes[2]
|
58
|
-
repo = url_nodes[3..4].join("/")
|
59
|
-
revision = url_nodes[5]
|
60
|
-
file_path = url_nodes[6..-1].join("/")
|
61
|
-
end
|
62
|
-
repo_dir = Git.pull(domain,repo,revision)
|
63
|
-
full_path = "#{repo_dir}/#{file_path}"
|
64
|
-
exists = File.exists?(full_path)
|
65
|
-
if exists
|
66
|
-
FileUtils.rm_r(repo_dir,:force=>true)
|
67
|
-
return exists
|
68
|
-
else
|
69
|
-
raise "Unable to find #{full_path}"
|
70
|
-
end
|
71
|
-
end
|
72
|
-
|
73
|
-
#pulls a git repo and sets it to the specified revision in the
|
74
|
-
#specified folder
|
75
|
-
def Git.pull(domain,repo,revision,run_dir=Dir.mktmpdir)
|
76
|
-
domain_properties = Git.config['domains'][domain]
|
77
|
-
user,host,key = ['user','host','key'].map{|k| domain_properties[k]}
|
78
|
-
#create folder for repo and command
|
79
|
-
run_file_path = run_dir + "/cmd.sh"
|
80
|
-
#put together command
|
81
|
-
git_prefix = key ? "ssh-add #{Base.root}/#{key};" : ""
|
82
|
-
git_suffix = (revision=="HEAD" ? " --depth=1" : "; git checkout -q #{revision}")
|
83
|
-
#add keys, clone repo, go to specific revision, execute command
|
84
|
-
full_cmd = "cd #{run_dir};#{git_prefix}git clone -q #{user}@#{host}:#{repo}.git#{git_suffix}"
|
85
|
-
#put command in file, run ssh-agent bash on it
|
86
|
-
File.open(run_file_path,"w") {|f| f.print(full_cmd)}
|
87
|
-
run_cmd = "ssh-agent bash #{run_file_path}"
|
88
|
-
#run the command, it will return an exception if there are issues
|
89
|
-
run_cmd.bash(true)
|
90
|
-
repo_name = repo.split("/").last
|
91
|
-
repo_dir = "#{run_dir}/#{repo_name}"
|
92
|
-
return repo_dir
|
93
|
-
end
|
94
|
-
|
95
|
-
def Git.read_by_dataset_path(dst_path,user_name,*args)
|
96
|
-
domain,repo,revision,file_path = []
|
97
|
-
dst_path.split("/").ie do |path_nodes|
|
98
|
-
domain = path_nodes[0]
|
99
|
-
repo = path_nodes[1..2].join("/")
|
100
|
-
revision = path_nodes[3]
|
101
|
-
file_path = path_nodes[4..-1].join("/")
|
102
|
-
end
|
103
|
-
#slash in front of path
|
104
|
-
repo_dir = Git.pull(domain,repo,revision)
|
105
|
-
full_path = "#{repo_dir}/#{file_path}"
|
106
|
-
result = "cat #{full_path}".bash(true)
|
107
|
-
FileUtils.rm_r(repo_dir,:force=>true)
|
108
|
-
return result
|
109
|
-
end
|
110
|
-
end
|
111
|
-
end
|
@@ -1,37 +0,0 @@
|
|
1
|
-
module Mobilize
|
2
|
-
module Ssh
|
3
|
-
def self.config
|
4
|
-
Base.config('ssh')
|
5
|
-
end
|
6
|
-
|
7
|
-
def self.host(node)
|
8
|
-
self.config['nodes'][node]['host']
|
9
|
-
end
|
10
|
-
|
11
|
-
def self.gateway(node)
|
12
|
-
self.config['nodes'][node]['gateway']
|
13
|
-
end
|
14
|
-
|
15
|
-
def self.sudoers(node)
|
16
|
-
self.config['nodes'][node]['sudoers']
|
17
|
-
end
|
18
|
-
|
19
|
-
def self.su_all_users(node)
|
20
|
-
self.config['nodes'][node]['su_all_users']
|
21
|
-
end
|
22
|
-
|
23
|
-
def self.nodes
|
24
|
-
self.config['nodes'].keys
|
25
|
-
end
|
26
|
-
|
27
|
-
def self.default_node
|
28
|
-
self.nodes.first
|
29
|
-
end
|
30
|
-
|
31
|
-
#determine if current machine is on host domain, needs gateway if one is provided and it is not
|
32
|
-
def self.needs_gateway?(node)
|
33
|
-
host_domain_name = self.host(node)['name'].split(".")[-2..-1].join(".")
|
34
|
-
return true if self.gateway(node) and Socket.domain_name != host_domain_name
|
35
|
-
end
|
36
|
-
end
|
37
|
-
end
|