mobilize-ssh 1.3 → 1.21

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -13,7 +13,6 @@ Table Of Contents
13
13
  * [Install Dirs and Files](#section_Install_Dirs_and_Files)
14
14
  * [Configure](#section_Configure)
15
15
  * [Ssh](#section_Configure_Ssh)
16
- * [Git](#section_Configure_Git)
17
16
  * [Start](#section_Start)
18
17
  * [Create Job](#section_Start_Create_Job)
19
18
  * [Run Test](#section_Start_Run_Test)
@@ -80,6 +79,9 @@ Configure
80
79
  ### Configure Ssh
81
80
 
82
81
  The Ssh configuration consists of:
82
+ * tmp_file_dir, which is where files will be stored before being scp'd
83
+ over to the nodes. They will be deleted afterwards, unless the job
84
+ fails in mid-copy. By default this is tmp/file/.
83
85
  * nodes, identified by aliases, such as `test_node`. This alias is what you should
84
86
  pass into the "node" param over in the ssh.run task.
85
87
  * if no node is specified, commands will default to the first node listed.
@@ -107,6 +109,7 @@ Sample ssh.yml:
107
109
  ``` yml
108
110
  ---
109
111
  development:
112
+ tmp_file_dir: tmp/file/
110
113
  nodes:
111
114
  dev_node:
112
115
  sudoers:
@@ -123,6 +126,7 @@ development:
123
126
  port: 22
124
127
  user: gateway_user
125
128
  test:
129
+ tmp_file_dir: tmp/file/
126
130
  nodes:
127
131
  test_node:
128
132
  sudoers:
@@ -139,6 +143,7 @@ test:
139
143
  port: 22
140
144
  user: gateway_user
141
145
  production:
146
+ tmp_file_dir: tmp/file/
142
147
  nodes:
143
148
  prod_node:
144
149
  sudoers:
@@ -156,57 +161,6 @@ production:
156
161
  user: gateway_user
157
162
  ```
158
163
 
159
- <a name='section_Configure_Git'></a>
160
- ### Configure Git
161
-
162
- Git configuration is not required but recommended, as it allows you to
163
- pull files directly from public or private Git repositories.
164
-
165
- The Git configuration consists of:
166
- * domains, identified by aliases, such as `private` and `public`.
167
- * domains are passed into the source parameters in the git.run task.
168
- * if no domain is specified, commands will default to the first domain listed.
169
-
170
- Each domain has:
171
- * a host;
172
- * a key (optional); If you don't need an ssh key to access the repo, remove that row from the configuration file.
173
- * this is the relative path of the ssh key required to access the repository.
174
- * a user, which is the user used for the git clone command.
175
-
176
- Sample git.yml:
177
-
178
- ``` yml
179
- ---
180
- development:
181
- domains:
182
- private:
183
- host: github.<domain>.com
184
- key: config/mobilize/ssh_private.key
185
- user: git
186
- public:
187
- host: github.com
188
- user: git
189
- test:
190
- domains:
191
- private:
192
- host: github.<domain>.com
193
- key: config/mobilize/ssh_private.key
194
- user: git
195
- public:
196
- host: github.com
197
- user: git
198
- production:
199
- domains:
200
- private:
201
- host: github.<domain>.com
202
- key: config/mobilize/ssh_private.key
203
- user: git
204
- public:
205
- host: github.com
206
- user: git
207
- ```
208
-
209
-
210
164
  <a name='section_Start'></a>
211
165
  Start
212
166
  -----
@@ -215,21 +169,10 @@ Start
215
169
  ### Create Job
216
170
 
217
171
  * For mobilize-ssh, the following task is available:
218
- * ssh.run `node: <node_alias>, cmd: <command>, user: user, sources:[*<source_paths>], params:[{<key,value pairs>}]`, which reads sources, copies them to a temporary folder on the selected node, and runs the command inside that folder.
219
- * user, sources, node, and params are optional; cmd is required.
172
+ * ssh.run `node: <node_alias>, cmd: <command>, user: user, sources:[*<source_paths>]`, which reads sources, copies them to a temporary folder on the selected node, and runs the command inside that folder.
173
+ * user, sources, and node are optional; cmd is required.
220
174
  * specifying user will cause the command to be prefixed with sudo su <user> -c.
221
175
  * non-google sources will also be read as the specified user.
222
- * git sources can be specified with syntax `git://<domain>/<repo_owner>/<repo_name>/<file_path>`.
223
- * Accessing private repos requires that you add the Mobilize public key to the repository as a deploy key.
224
- * there is no user-level access control for git repositories at this time.
225
- * domain defaults to the first one listed, if not included.
226
- * params are also optional for all of the below. They replace tokens in sources and the command.
227
- * params are passed as a YML or JSON, as in:
228
- * `ssh.run source:<source_path>, params:{'date':'2013-03-01', 'unit':'widgets'}`
229
- * this example replaces all the keys, preceded by '@' in all source hqls with the value.
230
- * The preceding '@' is used to keep from replacing instances
231
- of "date" and "unit" in the command/source file; you should have `@date` and `@unit` in your actual HQL
232
- if you'd like to replace those tokens.
233
176
  * not specifying node will cause the command to be run on the default node.
234
177
  * ssh sources can be specified with syntax
235
178
  `ssh://<node><file_full_path>`. If node is omitted, default node will be used.
data/lib/mobilize-ssh.rb CHANGED
@@ -6,10 +6,10 @@ require "net/scp"
6
6
  require "mobilize-ssh/extensions/net-ssh-connection-session"
7
7
  require "mobilize-ssh/extensions/net-ssh-gateway"
8
8
  require "mobilize-ssh/extensions/socket"
9
+ require "mobilize-ssh/extensions/string"
9
10
 
10
11
  module Mobilize
11
12
  module Ssh
12
13
  end
13
14
  end
14
15
  require "mobilize-ssh/handlers/ssh"
15
- require "mobilize-ssh/handlers/git"
@@ -0,0 +1,5 @@
1
+ class String
2
+ def to_md5
3
+ Digest::MD5.hexdigest(self)
4
+ end
5
+ end
@@ -1,8 +1,45 @@
1
1
  module Mobilize
2
2
  module Ssh
3
- #adds convenience methods
4
- require "#{File.dirname(__FILE__)}/../helpers/ssh_helper"
3
+ def Ssh.config
4
+ Base.config('ssh')
5
+ end
6
+
7
+ def Ssh.tmp_file_dir
8
+ Ssh.config['tmp_file_dir']
9
+ end
10
+
11
+ def Ssh.host(node)
12
+ Ssh.config['nodes'][node]['host']
13
+ end
14
+
15
+ def Ssh.gateway(node)
16
+ Ssh.config['nodes'][node]['gateway']
17
+ end
18
+
19
+ def Ssh.sudoers(node)
20
+ Ssh.config['nodes'][node]['sudoers']
21
+ end
22
+
23
+ def Ssh.su_all_users(node)
24
+ Ssh.config['nodes'][node]['su_all_users']
25
+ end
26
+
27
+ def Ssh.nodes
28
+ Ssh.config['nodes'].keys
29
+ end
30
+
31
+ def Ssh.default_node
32
+ Ssh.nodes.first
33
+ end
34
+
35
+ #determine if current machine is on host domain, needs gateway if one is provided and it is not
36
+ def Ssh.needs_gateway?(node)
37
+ host_domain_name = Ssh.host(node)['name'].split(".")[-2..-1].join(".")
38
+ return true if Ssh.gateway(node) and Socket.domain_name != host_domain_name
39
+ end
40
+
5
41
  def Ssh.pop_comm_dir(comm_dir,file_hash)
42
+ FileUtils.rm_r comm_dir, :force=>true
6
43
  file_hash.each do |fname,fdata|
7
44
  fpath = "#{comm_dir}/#{fname}"
8
45
  #for now, only gz is binary
@@ -13,8 +50,15 @@ module Mobilize
13
50
  return true if file_hash.keys.length>0
14
51
  end
15
52
 
53
+ def Ssh.set_key_permissions(key_path)
54
+ #makes sure permissions are set as appropriate for ssh key
55
+ raise "could not find ssh key at #{key_path}" unless File.exists?(key_path)
56
+ File.chmod(0600,key_path) unless File.stat(key_path).mode.to_s(8)[3..5] == "600"
57
+ return true
58
+ end
59
+
16
60
  # converts a source path or target path to a dst in the context of handler and stage
17
- def Ssh.path_to_dst(path,stage_path,gdrive_slot)
61
+ def Ssh.path_to_dst(path,stage_path)
18
62
  has_handler = true if path.index("://")
19
63
  red_path = path.split("://").last
20
64
  #is user has a handler, their first path node is a node name,
@@ -25,7 +69,7 @@ module Mobilize
25
69
  return Dataset.find_or_create_by_url(ssh_url)
26
70
  end
27
71
  #otherwise, use Gsheet
28
- return Gsheet.path_to_dst(red_path,stage_path,gdrive_slot)
72
+ return Gsheet.path_to_dst(red_path,stage_path)
29
73
  end
30
74
 
31
75
  def Ssh.url_by_path(path,user_name)
@@ -53,6 +97,7 @@ module Mobilize
53
97
  def Ssh.scp(node,from_path,to_path)
54
98
  name,key,port,user = Ssh.host(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
55
99
  key_path = "#{Base.root}/#{key}"
100
+ Ssh.set_key_permissions(key_path)
56
101
  opts = {:port=>(port || 22),:keys=>key_path}
57
102
  if Ssh.needs_gateway?(node)
58
103
  gname,gkey,gport,guser = Ssh.gateway(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
@@ -67,30 +112,25 @@ module Mobilize
67
112
  return true
68
113
  end
69
114
 
70
- def Ssh.run(node,command,user_name,file_hash={},run_params=nil)
71
- default_user_name = Ssh.host(node)['user']
115
+ def Ssh.run(node,command,user,file_hash={})
116
+ key,default_user = Ssh.host(node).ie{|h| ['key','user'].map{|k| h[k]}}
117
+ key_path = "#{Base.root}/#{key}"
118
+ Ssh.set_key_permissions(key_path)
72
119
  file_hash ||= {}
73
- run_params ||={}
74
120
  #make sure the dir for this command is clear
75
- comm_md5 = [user_name,node,command,file_hash.keys.to_s,Time.now.to_f.to_s].join.to_md5
76
- comm_dir = Dir.mktmpdir
77
- #replace any params in the file_hash and command
78
- run_params.each do |k,v|
79
- command.gsub!("@#{k}",v)
80
- file_hash.each do |name,data|
81
- data.gsub!("@#{k}",v)
82
- end
83
- end
84
- #populate comm dir with any files
121
+ comm_md5 = [user,node,command,file_hash.keys.to_s].join.to_md5
122
+ comm_dir = "#{Ssh.tmp_file_dir}#{comm_md5}"
123
+ #populate comm dir with any files
85
124
  Ssh.pop_comm_dir(comm_dir,file_hash)
125
+ #move any files up to the node
126
+ rem_dir = nil
86
127
  #make sure user starts in rem_dir
87
128
  rem_dir = "#{comm_md5}/"
88
129
  #make sure the rem_dir is gone
89
130
  Ssh.fire!(node,"sudo rm -rf #{rem_dir}")
90
131
  if File.exists?(comm_dir)
91
132
  Ssh.scp(node,comm_dir,rem_dir)
92
- #make sure comm_dir is removed
93
- FileUtils.rm_r(comm_dir,:force=>true)
133
+ FileUtils.rm_r comm_dir, :force=>true
94
134
  else
95
135
  #create folder
96
136
  mkdir_command = "mkdir #{rem_dir}"
@@ -102,9 +142,9 @@ module Mobilize
102
142
  Ssh.write(node,command,cmd_path)
103
143
  full_cmd = "(cd #{rem_dir} && sh #{cmd_file})"
104
144
  #fire_cmd runs sh on cmd_path, optionally with sudo su
105
- if user_name != default_user_name
145
+ if user != default_user
106
146
  #make sure user owns the folder and all files
107
- fire_cmd = %{sudo chown -R #{user_name} #{rem_dir}; sudo su #{user_name} -c "#{full_cmd}"}
147
+ fire_cmd = %{sudo chown -R #{user} #{rem_dir}; sudo su #{user} -c "#{full_cmd}"}
108
148
  rm_cmd = %{sudo rm -rf #{rem_dir}}
109
149
  else
110
150
  fire_cmd = full_cmd
@@ -118,6 +158,7 @@ module Mobilize
118
158
  def Ssh.fire!(node,cmd)
119
159
  name,key,port,user = Ssh.host(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
120
160
  key_path = "#{Base.root}/#{key}"
161
+ Ssh.set_key_permissions(key_path)
121
162
  opts = {:port=>(port || 22),:keys=>key_path}
122
163
  response = if Ssh.needs_gateway?(node)
123
164
  gname,gkey,gport,guser = Ssh.gateway(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
@@ -147,15 +188,17 @@ module Mobilize
147
188
  def Ssh.write(node,fdata,to_path,binary=false)
148
189
  from_path = Ssh.tmp_file(fdata,binary)
149
190
  Ssh.scp(node,from_path,to_path)
150
- #make sure local is removed
151
- FileUtils.rm_r(from_path,:force=>true)
191
+ FileUtils.rm from_path
152
192
  return true
153
193
  end
154
194
 
155
195
  def Ssh.tmp_file(fdata,binary=false,fpath=nil)
156
196
  #creates a file under tmp/files with an md5 from the data
157
- tmp_file_path = fpath || "#{Dir.mktmpdir}/#{(fdata + Time.now.utc.to_f.to_s).to_md5}"
197
+ tmp_file_path = fpath || "#{Ssh.tmp_file_dir}#{(fdata + Time.now.utc.to_f.to_s).to_md5}"
158
198
  write_mode = binary ? "wb" : "w"
199
+ #make sure folder is created
200
+ tmp_file_dir = tmp_file_path.split("/")[0..-2].join("/")
201
+ FileUtils.mkdir_p(tmp_file_dir)
159
202
  #write data to path
160
203
  File.open(tmp_file_path,write_mode) {|f| f.print(fdata)}
161
204
  return tmp_file_path
@@ -165,7 +208,7 @@ module Mobilize
165
208
  s = Stage.where(:path=>stage_path).first
166
209
  u = s.job.runner.user
167
210
  user_name = s.params['user']
168
- node = s.params['node']
211
+ node = s.params['node']
169
212
  node = Ssh.default_node unless Ssh.nodes.include?(node)
170
213
  if user_name and !Ssh.sudoers(node).include?(u.name)
171
214
  raise "#{u.name} does not have su permissions for this node"
@@ -175,27 +218,23 @@ module Mobilize
175
218
  return user_name
176
219
  end
177
220
 
178
- def Ssh.file_hash_by_stage_path(stage_path,gdrive_slot)
221
+ def Ssh.file_hash_by_stage_path(stage_path)
179
222
  file_hash = {}
180
223
  s = Stage.where(:path=>stage_path).first
181
224
  u = s.job.runner.user
182
225
  user_name = Ssh.user_name_by_stage_path(stage_path)
183
- s.sources(gdrive_slot).each do |sdst|
226
+ s.sources.each do |sdst|
184
227
  split_path = sdst.path.split("/")
185
228
  #if path is to stage output, name with stage name
186
- file_name = if (split_path.last == "out" and (1..5).to_a.map{|n| "stage#{n.to_s}"}.include?(split_path[-2].to_s))
187
- #<jobname>/stage1/out
229
+ file_name = if split_path.last == "out" and
230
+ (1..5).to_a.map{|n| "stage#{n.to_s}"}.include?(split_path[-2].to_s)
188
231
  "#{split_path[-2]}.out"
189
- elsif (1..5).to_a.map{|n| "stage#{n.to_s}"}.include?(split_path.last[-6..-1])
190
- #runner<jobname>stage1
191
- "#{split_path.last[-6..-1]}.out"
192
232
  else
193
233
  split_path.last
194
234
  end
195
235
  if ["gsheet","gfile"].include?(sdst.handler)
196
236
  #google drive sources are always read as the user
197
- #with the apportioned slot
198
- file_hash[file_name] = sdst.read(u.name,gdrive_slot)
237
+ file_hash[file_name] = sdst.read(u.name)
199
238
  else
200
239
  #other sources should be read by su-user
201
240
  file_hash[file_name] = sdst.read(user_name)
@@ -205,24 +244,17 @@ module Mobilize
205
244
  end
206
245
 
207
246
  def Ssh.run_by_stage_path(stage_path)
208
- gdrive_slot = Gdrive.slot_worker_by_path(stage_path)
209
- #return blank response if there are no slots available
210
- return nil unless gdrive_slot
211
247
  s = Stage.where(:path=>stage_path).first
212
248
  params = s.params
213
249
  node, command = [params['node'],params['cmd']]
214
250
  node ||= Ssh.default_node
215
251
  user_name = Ssh.user_name_by_stage_path(stage_path)
216
- file_hash = Ssh.file_hash_by_stage_path(stage_path,gdrive_slot)
217
- Gdrive.unslot_worker_by_path(stage_path)
218
- run_params = params['params']
219
- result = Ssh.run(node,command,user_name,file_hash,run_params)
252
+ file_hash = Ssh.file_hash_by_stage_path(stage_path)
253
+ result = Ssh.run(node,command,user_name,file_hash)
220
254
  #use Gridfs to cache result
221
255
  response = {}
222
256
  response['out_url'] = Dataset.write_by_url("gridfs://#{s.path}/out",result['stdout'].to_s,Gdrive.owner_name)
223
257
  response['err_url'] = Dataset.write_by_url("gridfs://#{s.path}/err",result['stderr'].to_s,Gdrive.owner_name) if result['stderr'].to_s.length>0
224
- #is an error if there is no out and there is an err, regardless of signal
225
- result['exit_code'] = 500 if result['stdout'].to_s.strip.length==0 and result['stderr'].to_s.strip.length>0
226
258
  response['signal'] = result['exit_code']
227
259
  response
228
260
  end
@@ -1,5 +1,5 @@
1
1
  module Mobilize
2
2
  module Ssh
3
- VERSION = "1.3"
3
+ VERSION = "1.21"
4
4
  end
5
5
  end
data/mobilize-ssh.gemspec CHANGED
@@ -16,7 +16,7 @@ Gem::Specification.new do |gem|
16
16
  gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
17
17
  gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
18
18
  gem.require_paths = ["lib"]
19
- gem.add_runtime_dependency "mobilize-base","1.3"
19
+ gem.add_runtime_dependency "mobilize-base","1.21"
20
20
  gem.add_runtime_dependency "net-ssh"
21
21
  gem.add_runtime_dependency "net-scp"
22
22
  gem.add_runtime_dependency "net-ssh-gateway"
data/test/code2.sh CHANGED
@@ -1 +1 @@
1
- tail @file
1
+ tail syslog
@@ -14,8 +14,8 @@ describe "Mobilize" do
14
14
 
15
15
  gdrive_slot = Mobilize::Gdrive.owner_email
16
16
  puts "create user 'mobilize'"
17
- user_name = gdrive_slot.split("@").first
18
- u = Mobilize::User.where(:name=>user_name).first
17
+ user = gdrive_slot.split("@").first
18
+ u = Mobilize::User.where(:name=>user).first
19
19
  r = u.runner
20
20
 
21
21
  rb_code_sheet = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/code.rb",gdrive_slot)
@@ -42,7 +42,6 @@ describe "Mobilize" do
42
42
  ssh_target_sheet_1 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_1.out",gdrive_slot)
43
43
  ssh_target_sheet_2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_2.out",gdrive_slot)
44
44
  ssh_target_sheet_3 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_3.out",gdrive_slot)
45
- ssh_target_sheet_4 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_4.out",gdrive_slot)
46
45
  [ssh_target_sheet_1,ssh_target_sheet_2,ssh_target_sheet_3].each {|s| s.delete if s}
47
46
 
48
47
  ssh_job_rows = ::YAML.load_file("#{Mobilize::Base.root}/test/ssh_job_rows.yml")
@@ -60,12 +59,10 @@ describe "Mobilize" do
60
59
  ssh_target_sheet_1 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_1.out",gdrive_slot)
61
60
  ssh_target_sheet_2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_2.out",gdrive_slot)
62
61
  ssh_target_sheet_3 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_3.out",gdrive_slot)
63
- ssh_target_sheet_4 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_4.out",gdrive_slot)
64
62
 
65
63
  assert ssh_target_sheet_1.to_tsv.length > 100
66
64
  assert ssh_target_sheet_2.to_tsv.length > 100
67
65
  assert ssh_target_sheet_3.to_tsv.length > 3
68
- assert ssh_target_sheet_4.to_tsv.length > 100
69
66
 
70
67
  end
71
68
 
@@ -1,25 +1,18 @@
1
- ---
2
1
  - name: test_ssh_1
3
2
  active: true
4
3
  trigger: once
5
4
  status: ""
6
- stage1: ssh.run node:"test_node", cmd:"ruby code.rb", user:"root", sources:["code.rb", "code.sh"]
7
- stage2: gsheet.write source:"stage1", target:"test_ssh_1.out"
5
+ stage1: 'ssh.run node:"test_node", cmd:"ruby code.rb", user:"root", sources:["code.rb", "code.sh"]'
6
+ stage2: 'gsheet.write source:"stage1", target:"test_ssh_1.out"'
8
7
  - name: test_ssh_2
9
8
  active: true
10
9
  trigger: "after test_ssh_1"
11
10
  status: ""
12
- stage1: ssh.run cmd:"sh code2.sh", user:"root", sources:["code2.sh","test_node/var/log/syslog"], params:{file:"syslog"}
13
- stage2: gsheet.write source:"stage1", target:"test_ssh_2.out"
11
+ stage1: 'ssh.run cmd:"sh code2.sh", user:"root", sources:["code2.sh","test_node/var/log/syslog"]'
12
+ stage2: 'gsheet.write source:"stage1", target:"test_ssh_2.out"'
14
13
  - name: test_ssh_3
15
14
  active: true
16
15
  trigger: "after test_ssh_2"
17
16
  status: ""
18
- stage1: ssh.run cmd:"echo '@test_param'", params:{test_param:"test param successful"}
19
- stage2: gsheet.write source:"stage1", target:"test_ssh_3.out"
20
- - name: test_ssh_4
21
- active: true
22
- trigger: "after test_ssh_3"
23
- status: ""
24
- stage1: ssh.run node:"test_node", user:root, sources:["git://DeNA/mobilize-ssh/test/code.rb","git://DeNA/mobilize-ssh/test/code.sh"], cmd:"ruby code.rb"
25
- stage2: gsheet.write source:stage1, target:"test_ssh_4.out"
17
+ stage1: 'ssh.run cmd:"whoami"'
18
+ stage2: 'gsheet.write source:"stage1", target:"test_ssh_3.out"'
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: mobilize-ssh
3
3
  version: !ruby/object:Gem::Version
4
- version: '1.3'
4
+ version: '1.21'
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2013-04-18 00:00:00.000000000 Z
12
+ date: 2013-03-22 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: mobilize-base
@@ -18,7 +18,7 @@ dependencies:
18
18
  requirements:
19
19
  - - '='
20
20
  - !ruby/object:Gem::Version
21
- version: '1.3'
21
+ version: '1.21'
22
22
  type: :runtime
23
23
  prerelease: false
24
24
  version_requirements: !ruby/object:Gem::Requirement
@@ -26,7 +26,7 @@ dependencies:
26
26
  requirements:
27
27
  - - '='
28
28
  - !ruby/object:Gem::Version
29
- version: '1.3'
29
+ version: '1.21'
30
30
  - !ruby/object:Gem::Dependency
31
31
  name: net-ssh
32
32
  requirement: !ruby/object:Gem::Requirement
@@ -91,9 +91,8 @@ files:
91
91
  - lib/mobilize-ssh/extensions/net-ssh-connection-session.rb
92
92
  - lib/mobilize-ssh/extensions/net-ssh-gateway.rb
93
93
  - lib/mobilize-ssh/extensions/socket.rb
94
- - lib/mobilize-ssh/handlers/git.rb
94
+ - lib/mobilize-ssh/extensions/string.rb
95
95
  - lib/mobilize-ssh/handlers/ssh.rb
96
- - lib/mobilize-ssh/helpers/ssh_helper.rb
97
96
  - lib/mobilize-ssh/tasks.rb
98
97
  - lib/mobilize-ssh/version.rb
99
98
  - lib/samples/ssh.yml
@@ -117,12 +116,18 @@ required_ruby_version: !ruby/object:Gem::Requirement
117
116
  - - ! '>='
118
117
  - !ruby/object:Gem::Version
119
118
  version: '0'
119
+ segments:
120
+ - 0
121
+ hash: 2839265652338346626
120
122
  required_rubygems_version: !ruby/object:Gem::Requirement
121
123
  none: false
122
124
  requirements:
123
125
  - - ! '>='
124
126
  - !ruby/object:Gem::Version
125
127
  version: '0'
128
+ segments:
129
+ - 0
130
+ hash: 2839265652338346626
126
131
  requirements: []
127
132
  rubyforge_project:
128
133
  rubygems_version: 1.8.25
@@ -1,111 +0,0 @@
1
- module Mobilize
2
- module Git
3
- def Git.config
4
- Base.config('git')
5
- end
6
-
7
- def Git.host(domain)
8
- Git.config['domains'][domain]['host']
9
- end
10
-
11
- def Git.domains
12
- Git.config['domains'].keys
13
- end
14
-
15
- def Git.default_domain
16
- Git.domains.first
17
- end
18
-
19
- # converts a source path or target path to a dst in the context of handler and stage
20
- def Git.path_to_dst(path,stage_path,gdrive_slot)
21
- red_path = path.split("://").last
22
- git_url = Git.url_by_path(red_path)
23
- return Dataset.find_or_create_by_url(git_url)
24
- end
25
-
26
- def Git.url_by_path(path)
27
- path_nodes = path.split("/")
28
- domain = path_nodes.first.to_s
29
- revision = "HEAD"
30
- if Git.domains.include?(domain)
31
- repo = path_nodes[1..2].join("/")
32
- file_path = path_nodes[3..-1].join("/")
33
- else
34
- domain = Git.default_domain
35
- repo = path_nodes[0..1].join("/")
36
- file_path = path_nodes[2..-1].join("/")
37
- end
38
- url = "git://#{domain}/#{repo}/#{revision}/#{file_path}"
39
- return url
40
- end
41
-
42
- #return path to tar.gz of git repo
43
- def Git.pack(domain,repo,revision="HEAD")
44
- repo_dir = Git.pull(domain,repo,revision)
45
- repo_name = repo.split("/").last
46
- tar_gz_path = "#{repo_dir}/../#{repo_name}.tar.gz"
47
- pack_cmd = "cd #{repo_dir} && git archive #{revision} --format=tar.gz > #{tar_gz_path}"
48
- pack_cmd.bash(true)
49
- FileUtils.rm_r(repo_dir,:force=>true)
50
- return tar_gz_path
51
- end
52
-
53
- #confirm that git file exists
54
- def Git.exists?(url)
55
- domain,repo,revision,file_path=[]
56
- url.split("/").ie do |url_nodes|
57
- domain = url_nodes[2]
58
- repo = url_nodes[3..4].join("/")
59
- revision = url_nodes[5]
60
- file_path = url_nodes[6..-1].join("/")
61
- end
62
- repo_dir = Git.pull(domain,repo,revision)
63
- full_path = "#{repo_dir}/#{file_path}"
64
- exists = File.exists?(full_path)
65
- if exists
66
- FileUtils.rm_r(repo_dir,:force=>true)
67
- return exists
68
- else
69
- raise "Unable to find #{full_path}"
70
- end
71
- end
72
-
73
- #pulls a git repo and sets it to the specified revision in the
74
- #specified folder
75
- def Git.pull(domain,repo,revision,run_dir=Dir.mktmpdir)
76
- domain_properties = Git.config['domains'][domain]
77
- user,host,key = ['user','host','key'].map{|k| domain_properties[k]}
78
- #create folder for repo and command
79
- run_file_path = run_dir + "/cmd.sh"
80
- #put together command
81
- git_prefix = key ? "ssh-add #{Base.root}/#{key};" : ""
82
- git_suffix = (revision=="HEAD" ? " --depth=1" : "; git checkout -q #{revision}")
83
- #add keys, clone repo, go to specific revision, execute command
84
- full_cmd = "cd #{run_dir};#{git_prefix}git clone -q #{user}@#{host}:#{repo}.git#{git_suffix}"
85
- #put command in file, run ssh-agent bash on it
86
- File.open(run_file_path,"w") {|f| f.print(full_cmd)}
87
- run_cmd = "ssh-agent bash #{run_file_path}"
88
- #run the command, it will return an exception if there are issues
89
- run_cmd.bash(true)
90
- repo_name = repo.split("/").last
91
- repo_dir = "#{run_dir}/#{repo_name}"
92
- return repo_dir
93
- end
94
-
95
- def Git.read_by_dataset_path(dst_path,user_name,*args)
96
- domain,repo,revision,file_path = []
97
- dst_path.split("/").ie do |path_nodes|
98
- domain = path_nodes[0]
99
- repo = path_nodes[1..2].join("/")
100
- revision = path_nodes[3]
101
- file_path = path_nodes[4..-1].join("/")
102
- end
103
- #slash in front of path
104
- repo_dir = Git.pull(domain,repo,revision)
105
- full_path = "#{repo_dir}/#{file_path}"
106
- result = "cat #{full_path}".bash(true)
107
- FileUtils.rm_r(repo_dir,:force=>true)
108
- return result
109
- end
110
- end
111
- end
@@ -1,37 +0,0 @@
1
- module Mobilize
2
- module Ssh
3
- def self.config
4
- Base.config('ssh')
5
- end
6
-
7
- def self.host(node)
8
- self.config['nodes'][node]['host']
9
- end
10
-
11
- def self.gateway(node)
12
- self.config['nodes'][node]['gateway']
13
- end
14
-
15
- def self.sudoers(node)
16
- self.config['nodes'][node]['sudoers']
17
- end
18
-
19
- def self.su_all_users(node)
20
- self.config['nodes'][node]['su_all_users']
21
- end
22
-
23
- def self.nodes
24
- self.config['nodes'].keys
25
- end
26
-
27
- def self.default_node
28
- self.nodes.first
29
- end
30
-
31
- #determine if current machine is on host domain, needs gateway if one is provided and it is not
32
- def self.needs_gateway?(node)
33
- host_domain_name = self.host(node)['name'].split(".")[-2..-1].join(".")
34
- return true if self.gateway(node) and Socket.domain_name != host_domain_name
35
- end
36
- end
37
- end