mobilize-ssh 1.36 → 1.291

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7ed1a7197b5d22a82360be98d54de38cc907f491
+  data.tar.gz: 9f13be4303ff61696a5e44087f0a88954585b4ae
+SHA512:
+  metadata.gz: eb9b1f01968105c9d79886008e653cc04ba50788437b571373a20e2588c9c1025ab91cea4cccf2224517b479b4dc4aa3e9c0924c860294574f03db19d90350e4
+  data.tar.gz: 39035b250da3e23cb6a8330ca51433c59628bd8d8724c259d3701f57953b835927625d8b007fca23654d45de8b1802278f9d8c3a9ff2a35f968757e2ea4b26be

data/README.md

@@ -1,4 +1,224 @@
-Mobilize
-========
+Mobilize-Ssh
+============
 
-Please refer to the mobilize-server wiki: https://github.com/DeNA/mobilize-server/wiki
+Mobilize-Ssh adds the power of ssh to [mobilize-base][mobilize-base].
+* move files, execute scripts, and output logs and datasets, all through
+Google Spreadsheets.
+
+Table Of Contents
+-----------------
+* [Overview](#section_Overview)
+* [Install](#section_Install)
+  * [Mobilize-Ssh](#section_Install_Mobilize-Ssh)
+  * [Install Dirs and Files](#section_Install_Dirs_and_Files)
+* [Configure](#section_Configure)
+  * [Ssh](#section_Configure_Ssh)
+* [Start](#section_Start)
+  * [Create Job](#section_Start_Create_Job)
+  * [Run Test](#section_Start_Run_Test)
+* [Meta](#section_Meta)
+* [Author](#section_Author)
+
+<a name='section_Overview'></a>
+Overview
+-----------
+
+* Mobilize-ssh adds script deployment to mobilize-base.
+
+<a name='section_Install'></a>
+Install
+------------
+
+Make sure you go through all the steps in the [mobilize-base][mobilize-base]
+install section first.
+
+<a name='section_Install_Mobilize-Ssh'></a>
+### Mobilize-Ssh
+
+add this to your Gemfile:
+
+``` ruby
+gem "mobilize-ssh"
+```
+
+or do
+
+  $ gem install mobilize-ssh
+
+for a ruby-wide install.
+
+<a name='section_Install_Dirs_and_Files'></a>
+### Dirs and Files
+
+### Rakefile
+
+Inside the Rakefile in your project's root dir, make sure you have:
+
+``` ruby
+require 'mobilize-base/tasks'
+require 'mobilize-ssh/tasks'
+```
+
+This defines rake tasks essential to run the environment.
+
+### Config Dir
+
+run 
+
+  $ rake mobilize_ssh:setup
+
+This will copy over a sample ssh.yml to your config dir.
+
+It will also add mobilize-ssh to the extensions in jobtracker.yml.
+
+<a name='section_Configure'></a>
+Configure
+------------
+
+<a name='section_Configure_Ssh'></a>
+### Configure Ssh
+
+The Ssh configuration consists of:
+* tmp_file_dir, which is where files will be stored before being scp'd
+over to the nodes. They will be deleted afterwards, unless the job
+fails in mid-copy. By default this is tmp/file/.
+* nodes, identified by aliases, such as `test_node`. This alias is what you should
+pass into the "node" param over in the ssh.run task.
+  * if no node is specified, commands will default to the first node listed.
+
+Each node has: 
+* a host;
+* a gateway (optional); If you don't need a gateway, remove that row from the configuration file.
+* sudoers; these are user names that are allowed to pass user params
+to the run call. This requires passwordless sudo for the host user.
+* su_all_users true/false option, which ensures that commands are executed by the
+user on the Runner. It prefixes all commands with sudo su <user> before executing the
+command. This is strongly recommended if possible as it ensures users do
+not overstep their permissions. This requires passwordless sudo for the
+host user and accounts on the host machine for each user.
+
+Each host and gateway has a series of ssh params:
+* name - the ip address or name of the host
+* key - the relative path of the ssh key file. Default is
+"config/mobilize/ssh_private.key"
+* port - the port to connect on
+* user - the user you are connecting as
+
+Sample ssh.yml:
+
+``` yml
+---
+development:
+  tmp_file_dir: tmp/file/
+  nodes:
+    dev_node:
+      sudoers: 
+      - sudo_user
+      su_all_users: true
+      host: 
+        name: dev-host.com 
+        key: config/mobilize/ssh_private.key
+        port: 22
+        user: host_user
+      gateway: 
+        name: dev-gateway.com 
+        key: config/mobilize/ssh_private.key 
+        port: 22 
+        user: gateway_user
+test:
+  tmp_file_dir: tmp/file/
+  nodes:
+    test_node:
+      sudoers: 
+      - sudo_user
+      su_all_users: true
+      host: 
+        name: test-host.com 
+        key: config/mobilize/ssh_private.key 
+        port: 22 
+        user: host_user
+      gateway: 
+        name: test-gateway.com 
+        key: config/mobilize/ssh_private.key 
+        port: 22 
+        user: gateway_user
+production:
+  tmp_file_dir: tmp/file/
+  nodes:
+    prod_node:
+      sudoers: 
+      - sudo_user
+      su_all_users: true
+      host:
+        name: prod-host.com 
+        key: config/mobilize/ssh_private.key 
+        port: 22 
+        user: host_user
+      gateway: 
+        name: prod-gateway.com 
+        key: config/mobilize/ssh_private.key 
+        port: 22 
+        user: gateway_user
+```
+
+<a name='section_Start'></a>
+Start
+-----
+
+<a name='section_Start_Create_Job'></a>
+### Create Job
+
+* For mobilize-ssh, the following task is available:
+  * ssh.run `node: <node_alias>, cmd: <command>, user: user, sources:[*<source_paths>]`, which reads sources, copies them to a temporary folder on the selected node, and runs the command inside that folder. 
+  * user, sources, and node are optional; cmd is required. 
+  * specifying user will cause the command to be prefixed with sudo su <user> -c. 
+    * non-google sources will also be read as the specified user.
+  * not specifying node will cause the command to be run on the default node.
+  * ssh sources can be specified with syntax
+`ssh://<node><file_full_path>`. If node is omitted, default node will be used.
+  * `<node><file_full_path>` and `<file_full_path>` can be used in the context of ssh.run, but if the path has only 1 slash, or none, it will try to find a google sheet or file instead.
+  * The test uses `ssh.run node:"test_node", cmd:"ruby code.rb", user: "root", sources:["code.rb","code.sh"]`
+
+<a name='section_Start_Run_Test'></a>
+### Run Test
+
+To run tests, you will need to 
+
+1) go through the [mobilize-base][mobilize-base] test first
+
+2) clone the mobilize-ssh repository 
+
+From the project folder, run
+
+3) $ rake mobilize_ssh:setup
+
+Copy over the config files from the mobilize-base project into the
+config dir, and populate the values in the ssh.yml file, esp. the
+test_node item.
+
+You should also copy the ssh private key you wish to use into your
+desired path (by default: config/mobilize/ssh_private.key), and make sure it is referenced in ssh.yml
+
+3) $ rake test
+
+This will populate your test Runner from mobilize-base with a sample ssh job.
+
+The purpose of the test will be to deploy two code files, have the first
+execute the second, which is a "tail /var/log/syslog" command, and write the resulting output to a gsheet.
+
+<a name='section_Meta'></a>
+Meta
+----
+
+* Code: `git clone git://github.com/dena/mobilize-ssh.git`
+* Home: <https://github.com/dena/mobilize-ssh>
+* Bugs: <https://github.com/dena/mobilize-ssh/issues>
+* Gems: <http://rubygems.org/gems/mobilize-ssh>
+
+<a name='section_Author'></a>
+Author
+------
+
+Cassio Paes-Leme :: cpaesleme@dena.com :: @cpaesleme
+
+[mobilize-base]: https://github.com/dena/mobilize-base

data/lib/mobilize-ssh/extensions/string.rb

@@ -0,0 +1,5 @@
+class String
+  def to_md5
+    Digest::MD5.hexdigest(self)
+  end
+end

data/lib/mobilize-ssh/handlers/ssh.rb

@@ -1,8 +1,45 @@
 module Mobilize
   module Ssh
-    #adds convenience methods
-    require "#{File.dirname(__FILE__)}/../helpers/ssh_helper"
+    def Ssh.config
+      Base.config('ssh')
+    end
+
+    def Ssh.tmp_file_dir
+      Ssh.config['tmp_file_dir']
+    end
+
+    def Ssh.host(node)
+      Ssh.config['nodes'][node]['host']
+    end
+
+    def Ssh.gateway(node)
+      Ssh.config['nodes'][node]['gateway']
+    end
+
+    def Ssh.sudoers(node)
+      Ssh.config['nodes'][node]['sudoers']
+    end
+
+    def Ssh.su_all_users(node)
+      Ssh.config['nodes'][node]['su_all_users']
+    end
+
+    def Ssh.nodes
+      Ssh.config['nodes'].keys
+    end
+
+    def Ssh.default_node
+      Ssh.nodes.first
+    end
+
+    #determine if current machine is on host domain, needs gateway if one is provided and it is not
+    def Ssh.needs_gateway?(node)
+      host_domain_name = Ssh.host(node)['name'].split(".")[-2..-1].join(".")
+      return true if Ssh.gateway(node) and Socket.domain_name != host_domain_name
+    end
+
     def Ssh.pop_comm_dir(comm_dir,file_hash)
+      FileUtils.rm_r comm_dir, :force=>true
       file_hash.each do |fname,fdata|
         fpath = "#{comm_dir}/#{fname}"
         #for now, only gz is binary
@@ -13,6 +50,13 @@ module Mobilize
       return true if file_hash.keys.length>0
     end
 
+    def Ssh.set_key_permissions(key_path)
+      #makes sure permissions are set as appropriate for ssh key
+      raise "could not find ssh key at #{key_path}" unless File.exists?(key_path)
+      File.chmod(0600,key_path) unless File.stat(key_path).mode.to_s(8)[3..5] == "600"
+      return true
+    end
+
     # converts a source path or target path to a dst in the context of handler and stage
     def Ssh.path_to_dst(path,stage_path,gdrive_slot)
       has_handler = true if path.index("://")
@@ -53,6 +97,7 @@ module Mobilize
     def Ssh.scp(node,from_path,to_path)
       name,key,port,user = Ssh.host(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
       key_path = "#{Base.root}/#{key}"
+      Ssh.set_key_permissions(key_path)
       opts = {:port=>(port || 22),:keys=>key_path}
       if Ssh.needs_gateway?(node)
         gname,gkey,gport,guser = Ssh.gateway(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
@@ -67,30 +112,25 @@ module Mobilize
       return true
     end
 
-    def Ssh.run(node,command,user_name,file_hash={},run_params=nil)
-      default_user_name = Ssh.host(node)['user']
+    def Ssh.run(node,command,user,file_hash={})
+      key,default_user = Ssh.host(node).ie{|h| ['key','user'].map{|k| h[k]}}
+      key_path = "#{Base.root}/#{key}"
+      Ssh.set_key_permissions(key_path)
       file_hash ||= {}
-      run_params ||={}
       #make sure the dir for this command is clear
-      comm_md5 = [user_name,node,command,file_hash.keys.to_s,Time.now.to_f.to_s].join.to_md5
-      comm_dir = Dir.mktmpdir
-      #replace any params in the file_hash and command
-      run_params.each do |k,v|
-        command.gsub!("@#{k}",v)
-        file_hash.each do |name,data|
-          data.gsub!("@#{k}",v)
-        end
-      end
-     #populate comm dir with any files
+      comm_md5 = [user,node,command,file_hash.keys.to_s,Time.now.to_f.to_s].join.to_md5
+      comm_dir = "#{Ssh.tmp_file_dir}#{comm_md5}"
+      #populate comm dir with any files
       Ssh.pop_comm_dir(comm_dir,file_hash)
+      #move any files up to the node
+      rem_dir = nil
       #make sure user starts in rem_dir
       rem_dir = "#{comm_md5}/"
       #make sure the rem_dir is gone
       Ssh.fire!(node,"sudo rm -rf #{rem_dir}")
       if File.exists?(comm_dir)
         Ssh.scp(node,comm_dir,rem_dir)
-        #make sure comm_dir is removed
-        FileUtils.rm_r(comm_dir,:force=>true)
+        FileUtils.rm_r comm_dir, :force=>true
       else
         #create folder
         mkdir_command = "mkdir #{rem_dir}"
@@ -102,9 +142,9 @@ module Mobilize
       Ssh.write(node,command,cmd_path)
       full_cmd = "(cd #{rem_dir} && sh #{cmd_file})"
       #fire_cmd runs sh on cmd_path, optionally with sudo su
-      if user_name != default_user_name
+      if user != default_user
         #make sure user owns the folder and all files
-        fire_cmd = %{sudo chown -R #{user_name} #{rem_dir}; sudo su #{user_name} -c "#{full_cmd}"}
+        fire_cmd = %{sudo chown -R #{user} #{rem_dir}; sudo su #{user} -c "#{full_cmd}"}
         rm_cmd = %{sudo rm -rf #{rem_dir}}
       else
         fire_cmd = full_cmd
@@ -118,6 +158,7 @@ module Mobilize
     def Ssh.fire!(node,cmd)
       name,key,port,user = Ssh.host(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
       key_path = "#{Base.root}/#{key}"
+      Ssh.set_key_permissions(key_path)
       opts = {:port=>(port || 22),:keys=>key_path}
       response = if Ssh.needs_gateway?(node)
                    gname,gkey,gport,guser = Ssh.gateway(node).ie{|h| ['name','key','port','user'].map{|k| h[k]}}
@@ -147,15 +188,17 @@ module Mobilize
     def Ssh.write(node,fdata,to_path,binary=false)
       from_path = Ssh.tmp_file(fdata,binary)
       Ssh.scp(node,from_path,to_path)
-      #make sure local is removed
-      FileUtils.rm_r(from_path,:force=>true)
+      FileUtils.rm from_path
       return true
     end
 
     def Ssh.tmp_file(fdata,binary=false,fpath=nil)
       #creates a file under tmp/files with an md5 from the data
-      tmp_file_path = fpath || "#{Dir.mktmpdir}/#{(fdata + Time.now.utc.to_f.to_s).to_md5}"
+      tmp_file_path = fpath || "#{Ssh.tmp_file_dir}#{(fdata + Time.now.utc.to_f.to_s).to_md5}"
       write_mode = binary ? "wb" : "w"
+      #make sure folder is created
+      tmp_file_dir = tmp_file_path.split("/")[0..-2].join("/")
+      FileUtils.mkdir_p(tmp_file_dir)
       #write data to path
       File.open(tmp_file_path,write_mode) {|f| f.print(fdata)}
       return tmp_file_path
@@ -165,7 +208,7 @@ module Mobilize
       s = Stage.where(:path=>stage_path).first
       u = s.job.runner.user
       user_name = s.params['user']
-      node = s.params['node']
+      node = s.params['node'] 
       node = Ssh.default_node unless Ssh.nodes.include?(node)
       if user_name and !Ssh.sudoers(node).include?(u.name)
         raise "#{u.name} does not have su permissions for this node"
@@ -183,12 +226,9 @@ module Mobilize
       s.sources(gdrive_slot).each do |sdst|
                        split_path = sdst.path.split("/")
                        #if path is to stage output, name with stage name
-                       file_name = if (split_path.last == "out" and (1..5).to_a.map{|n| "stage#{n.to_s}"}.include?(split_path[-2].to_s))
-                                     #<jobname>/stage1/out
+                       file_name = if split_path.last == "out" and
+                                     (1..5).to_a.map{|n| "stage#{n.to_s}"}.include?(split_path[-2].to_s)
                                      "#{split_path[-2]}.out"
-                                   elsif (1..5).to_a.map{|n| "stage#{n.to_s}"}.include?(split_path.last[-6..-1])
-                                     #runner<jobname>stage1
-                                   "#{split_path.last[-6..-1]}.out"
                                    else
                                      split_path.last
                                    end
@@ -215,14 +255,11 @@ module Mobilize
       user_name = Ssh.user_name_by_stage_path(stage_path)
       file_hash = Ssh.file_hash_by_stage_path(stage_path,gdrive_slot)
       Gdrive.unslot_worker_by_path(stage_path)
-      run_params = params['params']
-      result = Ssh.run(node,command,user_name,file_hash,run_params)
+      result = Ssh.run(node,command,user_name,file_hash)
       #use Gridfs to cache result
       response = {}
       response['out_url'] = Dataset.write_by_url("gridfs://#{s.path}/out",result['stdout'].to_s,Gdrive.owner_name)
       response['err_url'] = Dataset.write_by_url("gridfs://#{s.path}/err",result['stderr'].to_s,Gdrive.owner_name) if result['stderr'].to_s.length>0
-      #is an error if there is no out and there is an err, regardless of signal
-      result['exit_code'] = 500 if result['stdout'].to_s.strip.length==0 and result['stderr'].to_s.strip.length>0
       response['signal'] = result['exit_code']
       response
     end

data/lib/mobilize-ssh/tasks.rb

@@ -1,7 +1,7 @@
-require 'yaml'
-namespace :mobilize do
+namespace :mobilize_ssh do
   desc "Set up config and log folders and files"
-  task :setup_ssh do
+  task :setup do
+    require 'yaml'
     sample_dir = File.dirname(__FILE__) + '/../samples/'
     sample_files = Dir.entries(sample_dir)
     config_dir = (ENV['MOBILIZE_CONFIG_DIR'] ||= "config/mobilize/")

data/lib/mobilize-ssh/version.rb

@@ -1,5 +1,5 @@
 module Mobilize
   module Ssh
-    VERSION = "1.36"
+    VERSION = "1.291"
   end
 end

data/lib/mobilize-ssh.rb

@@ -6,13 +6,10 @@ require "net/scp"
 require "mobilize-ssh/extensions/net-ssh-connection-session"
 require "mobilize-ssh/extensions/net-ssh-gateway"
 require "mobilize-ssh/extensions/socket"
+require "mobilize-ssh/extensions/string"
 
 module Mobilize
   module Ssh
-    def Ssh.home_dir
-      File.expand_path('..',File.dirname(__FILE__))
-    end
   end
 end
 require "mobilize-ssh/handlers/ssh"
-require "mobilize-ssh/handlers/git"

data/lib/samples/ssh.yml

@@ -1,5 +1,6 @@
 ---
 development:
+  tmp_file_dir: tmp/file/
   nodes:
     dev_node:
       sudoers: 
@@ -16,6 +17,7 @@ development:
         port: 22 
         user: gateway_user
 test:
+  tmp_file_dir: tmp/file/
   nodes:
     test_node:
       sudoers: 
@@ -32,6 +34,7 @@ test:
         port: 22 
         user: gateway_user
 production:
+  tmp_file_dir: tmp/file/
   nodes:
     prod_node:
       sudoers: 

data/mobilize-ssh.gemspec

@@ -16,7 +16,7 @@ Gem::Specification.new do |gem|
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
-  gem.add_runtime_dependency "mobilize-base","1.36"
+  gem.add_runtime_dependency "mobilize-base","1.29"
   gem.add_runtime_dependency "net-ssh"
   gem.add_runtime_dependency "net-scp"
   gem.add_runtime_dependency "net-ssh-gateway"

data/test/code2.sh

@@ -0,0 +1 @@
+tail syslog

data/test/mobilize-ssh_test.rb

@@ -0,0 +1,92 @@
+require 'test_helper'
+
+describe "Mobilize" do
+
+  def before
+    puts 'nothing before'
+  end
+
+  # enqueues 4 workers on Resque
+  it "runs integration test" do
+
+    puts "restart workers"
+    Mobilize::Jobtracker.restart_workers!
+
+    gdrive_slot = Mobilize::Gdrive.owner_email
+    puts "create user 'mobilize'"
+    user = gdrive_slot.split("@").first
+    u = Mobilize::User.where(:name=>user).first
+    r = u.runner
+
+    rb_code_sheet = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/code.rb",gdrive_slot)
+    sh_code_sheet = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/code.sh",gdrive_slot)
+    sh_code_sheet2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/code2.sh",gdrive_slot)
+    [rb_code_sheet,sh_code_sheet].each {|s| s.delete if s}
+
+    puts "add test code"
+    rb_code_sheet = Mobilize::Gsheet.find_or_create_by_path("#{r.path.split("/")[0..-2].join("/")}/code.rb",gdrive_slot)
+    rb_code_tsv = File.open("#{Mobilize::Base.root}/test/code.rb").read
+    rb_code_sheet.write(rb_code_tsv,Mobilize::Gdrive.owner_name)
+
+    sh_code_sheet = Mobilize::Gsheet.find_or_create_by_path("#{r.path.split("/")[0..-2].join("/")}/code.sh",gdrive_slot)
+    sh_code_tsv = File.open("#{Mobilize::Base.root}/test/code.sh").read
+    sh_code_sheet.write(sh_code_tsv,Mobilize::Gdrive.owner_name)
+
+    sh_code_sheet2 = Mobilize::Gsheet.find_or_create_by_path("#{r.path.split("/")[0..-2].join("/")}/code2.sh",gdrive_slot)
+    sh_code_tsv2 = File.open("#{Mobilize::Base.root}/test/code2.sh").read
+    sh_code_sheet2.write(sh_code_tsv2,Mobilize::Gdrive.owner_name)
+
+    jobs_sheet = r.gsheet(gdrive_slot)
+
+    #delete target sheets if they exist
+    ssh_target_sheet_1 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_1.out",gdrive_slot)
+    ssh_target_sheet_2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_2.out",gdrive_slot)
+    ssh_target_sheet_3 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_3.out",gdrive_slot)
+    [ssh_target_sheet_1,ssh_target_sheet_2,ssh_target_sheet_3].each {|s| s.delete if s}
+
+    ssh_job_rows = ::YAML.load_file("#{Mobilize::Base.root}/test/ssh_job_rows.yml")
+    ssh_job_rows.map{|j| r.jobs(j['name'])}.each{|j| j.delete if j}
+    jobs_sheet.add_or_update_rows(ssh_job_rows)
+
+    puts "job row added, force enqueue runner, wait for stages"
+    r.enqueue!
+    wait_for_stages(900)
+
+    puts "update job status and activity"
+    r.update_gsheet(gdrive_slot)
+
+    puts "jobtracker posted data to test sheet"
+    ssh_target_sheet_1 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_1.out",gdrive_slot)
+    ssh_target_sheet_2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_2.out",gdrive_slot)
+    ssh_target_sheet_3 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_3.out",gdrive_slot)
+
+    assert ssh_target_sheet_1.to_tsv.length > 100
+    assert ssh_target_sheet_2.to_tsv.length > 100
+    assert ssh_target_sheet_3.to_tsv.length > 3
+
+  end
+
+  def wait_for_stages(time_limit=600,stage_limit=120,wait_length=10)
+    time = 0
+    time_since_stage = 0
+    #check for 10 min
+    while time < time_limit and time_since_stage < stage_limit
+      sleep wait_length
+      job_classes = Mobilize::Resque.jobs.map{|j| j['class']}
+      if job_classes.include?("Mobilize::Stage")
+        time_since_stage = 0
+        puts "saw stage at #{time.to_s} seconds"
+      else
+        time_since_stage += wait_length
+        puts "#{time_since_stage.to_s} seconds since stage seen"
+      end
+      time += wait_length
+      puts "total wait time #{time.to_s} seconds"
+    end
+
+    if time >= time_limit
+      raise "Timed out before stage completion"
+    end
+  end
+
+end

data/test/ssh_job_rows.yml

@@ -0,0 +1,18 @@
+- name: test_ssh_1
+  active: true
+  trigger: once
+  status: ""
+  stage1: 'ssh.run node:"test_node", cmd:"ruby code.rb", user:"root", sources:["code.rb", "code.sh"]'
+  stage2: 'gsheet.write source:"stage1", target:"test_ssh_1.out"'
+- name: test_ssh_2
+  active: true
+  trigger: "after test_ssh_1"
+  status: ""
+  stage1: 'ssh.run cmd:"sh code2.sh", user:"root", sources:["code2.sh","test_node/var/log/syslog"]'
+  stage2: 'gsheet.write source:"stage1", target:"test_ssh_2.out"'
+- name: test_ssh_3
+  active: true
+  trigger: "after test_ssh_2"
+  status: ""
+  stage1: 'ssh.run cmd:"whoami"'
+  stage2: 'gsheet.write source:"stage1", target:"test_ssh_3.out"'

data/test/test_helper.rb

@@ -8,4 +8,3 @@ $dir = File.dirname(File.expand_path(__FILE__))
 ENV['MOBILIZE_ENV'] = 'test'
 require 'mobilize-ssh'
 $TESTING = true
-require "#{Mobilize::Base.home_dir}/test/test_helper"

metadata

@@ -1,78 +1,69 @@
 --- !ruby/object:Gem::Specification
 name: mobilize-ssh
 version: !ruby/object:Gem::Version
-  version: '1.36'
-  prerelease: 
+  version: '1.291'
 platform: ruby
 authors:
 - Cassio Paes-Leme
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-05-21 00:00:00.000000000 Z
+date: 2013-03-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mobilize-base
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: '1.36'
+        version: '1.29'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: '1.36'
+        version: '1.29'
 - !ruby/object:Gem::Dependency
   name: net-ssh
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: net-scp
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: net-ssh-gateway
   requirement: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
-    none: false
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 description: mobilize-ssh allows you to automate ssh commands and files across hosts
@@ -91,58 +82,47 @@ files:
 - lib/mobilize-ssh/extensions/net-ssh-connection-session.rb
 - lib/mobilize-ssh/extensions/net-ssh-gateway.rb
 - lib/mobilize-ssh/extensions/socket.rb
-- lib/mobilize-ssh/handlers/git.rb
+- lib/mobilize-ssh/extensions/string.rb
 - lib/mobilize-ssh/handlers/ssh.rb
-- lib/mobilize-ssh/helpers/ssh_helper.rb
 - lib/mobilize-ssh/tasks.rb
 - lib/mobilize-ssh/version.rb
-- lib/samples/git.yml
 - lib/samples/ssh.yml
 - mobilize-ssh.gemspec
-- test/fixtures/code.rb
-- test/fixtures/code.sh
-- test/fixtures/code2.sh
-- test/fixtures/integration_expected.yml
-- test/fixtures/integration_jobs.yml
-- test/integration/mobilize-ssh_test.rb
+- test/code.rb
+- test/code.sh
+- test/code2.sh
+- test/mobilize-ssh_test.rb
 - test/redis-test.conf
+- test/ssh_job_rows.yml
 - test/test_helper.rb
 homepage: http://github.com/dena/mobilize-ssh
 licenses: []
+metadata: {}
 post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: 3534560230256585865
 required_rubygems_version: !ruby/object:Gem::Requirement
-  none: false
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
-      segments:
-      - 0
-      hash: 3534560230256585865
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.25
+rubygems_version: 2.0.3
 signing_key: 
-specification_version: 3
+specification_version: 4
 summary: extend mobilize-base with the ability to run files across hosts
 test_files:
-- test/fixtures/code.rb
-- test/fixtures/code.sh
-- test/fixtures/code2.sh
-- test/fixtures/integration_expected.yml
-- test/fixtures/integration_jobs.yml
-- test/integration/mobilize-ssh_test.rb
+- test/code.rb
+- test/code.sh
+- test/code2.sh
+- test/mobilize-ssh_test.rb
 - test/redis-test.conf
+- test/ssh_job_rows.yml
 - test/test_helper.rb

data/lib/mobilize-ssh/handlers/git.rb

@@ -1,120 +0,0 @@
-module Mobilize
-  module Git
-    def Git.config
-      Base.config('git')
-    end
-
-    def Git.host(domain)
-      Git.config['domains'][domain]['host']
-    end
-
-    def Git.domains
-      Git.config['domains'].keys
-    end
-
-    def Git.default_domain
-      Git.domains.first
-    end
-
-    def Git.repo_key(domain,repo)
-      begin
-        Git.config['domains'][domain]['repo_keys'][repo]
-      rescue
-        nil #no key for public repos
-      end
-    end
-
-    # converts a source path or target path to a dst in the context of handler and stage
-    def Git.path_to_dst(path,stage_path,gdrive_slot)
-      red_path = path.split("://").last
-      git_url = Git.url_by_path(red_path)
-      return Dataset.find_or_create_by_url(git_url)
-    end
-
-    def Git.url_by_path(path)
-      path_nodes = path.split("/")
-      domain = path_nodes.first.to_s
-      revision = "HEAD"
-      if Git.domains.include?(domain)
-        repo = path_nodes[1..2].join("/")
-        file_path = path_nodes[3..-1].join("/")
-      else
-        domain = Git.default_domain
-        repo = path_nodes[0..1].join("/")
-        file_path = path_nodes[2..-1].join("/")
-      end
-      url = "git://#{domain}/#{repo}/#{revision}/#{file_path}"
-      return url
-    end
-
-    #return path to tar.gz of git repo
-    def Git.pack(domain,repo,revision="HEAD")
-      repo_dir = Git.pull(domain,repo,revision)
-      repo_name = repo.split("/").last
-      tar_gz_path = "#{repo_dir}/../#{repo_name}.tar.gz"
-      pack_cmd = "cd #{repo_dir} && git archive #{revision} --format=tar.gz > #{tar_gz_path}"
-      pack_cmd.bash(true)
-      FileUtils.rm_r(repo_dir,:force=>true)
-      return tar_gz_path
-    end
-
-    #confirm that git file exists
-    def Git.exists?(url)
-      domain,repo,revision,file_path=[]
-      url.split("/").ie do |url_nodes|
-        domain    = url_nodes[2]
-        repo      = url_nodes[3..4].join("/")
-        revision  = url_nodes[5]
-        file_path = url_nodes[6..-1].join("/")
-      end
-      repo_dir = Git.pull(domain,repo,revision)
-      full_path = "#{repo_dir}/#{file_path}"
-      exists = File.exists?(full_path)
-      if exists
-        FileUtils.rm_r(repo_dir,:force=>true)
-        return exists
-      else
-        raise "Unable to find #{full_path}"
-      end
-    end
-
-    #pulls a git repo and sets it to the specified revision in the
-    #specified folder
-    def Git.pull(domain,repo,revision,run_dir=Dir.mktmpdir)
-      domain_properties = Git.config['domains'][domain]
-      user,host= ['user','host'].map{|k| domain_properties[k]}
-      key = Git.repo_key(domain,repo)
-      #create folder for repo and command
-      run_file_path = run_dir + "/cmd.sh"
-      #put together command
-      git_prefix = key ? "ssh-add #{Base.root}/#{key};" : ""
-      git_suffix = (revision=="HEAD" ? " --depth=1" : "; git checkout -q #{revision}")
-      #add keys, clone repo, go to specific revision, execute command
-      full_cmd = "cd #{run_dir};#{git_prefix}git clone -q #{user}@#{host}:#{repo}.git#{git_suffix}"
-      #put command in file, run ssh-agent bash on it
-      File.open(run_file_path,"w") {|f| f.print(full_cmd)}
-      run_cmd = "ssh-agent bash #{run_file_path}"
-      #run the command, it will return an exception if there are issues
-      run_cmd.bash(true)
-      repo_name = repo.split("/").last
-      repo_dir = "#{run_dir}/#{repo_name}"
-      return repo_dir
-    end
-
-    def Git.read_by_dataset_path(dst_path,user_name,*args)
-      domain,repo,revision,file_path = []
-      dst_path.split("/").ie do |path_nodes|
-        domain    = path_nodes[0]
-        repo      = path_nodes[1..2].join("/")
-        revision  = path_nodes[3]
-        file_path = path_nodes[4..-1].join("/")
-      end
-      #slash in front of path
-      repo_dir = Git.pull(domain,repo,revision)
-      full_path = "#{repo_dir}/#{file_path}"
-      result = "cat #{full_path}".bash(true)
-      FileUtils.rm_r(repo_dir,:force=>true)
-      return result
-    end
-  end
-end

data/lib/mobilize-ssh/helpers/ssh_helper.rb

@@ -1,37 +0,0 @@
-module Mobilize
-  module Ssh
-    def self.config
-      Base.config('ssh')
-    end
-
-    def self.host(node)
-      self.config['nodes'][node]['host']
-    end
-
-    def self.gateway(node)
-      self.config['nodes'][node]['gateway']
-    end
-
-    def self.sudoers(node)
-      self.config['nodes'][node]['sudoers']
-    end
-
-    def self.su_all_users(node)
-      self.config['nodes'][node]['su_all_users']
-    end
-
-    def self.nodes
-      self.config['nodes'].keys
-    end
-
-    def self.default_node
-      self.nodes.first
-    end
-
-    #determine if current machine is on host domain, needs gateway if one is provided and it is not
-    def self.needs_gateway?(node)
-      host_domain_name = self.host(node)['name'].split(".")[-2..-1].join(".")
-      return true if self.gateway(node) and Socket.domain_name != host_domain_name
-    end
-  end
-end

data/lib/samples/git.yml

@@ -1,34 +0,0 @@
----
-development:
-  domains:
-    private:
-      host: github.private.com
-      user: git
-      repo_keys:
-        "repo_path_1": 'local/path/to/key1'
-        "repo_path_2": 'local/path/to/key1'
-    public:
-      host: github.com
-      user: git
-test:
-  domains:
-    public:
-      host: github.com
-      user: git
-    private:
-      host: github.private.com
-      user: git
-      repo_keys:
-        "repo_path_1": 'local/path/to/key1'
-        "repo_path_2": 'local/path/to/key1'
-production:
-  domains:
-    private:
-      host: github.private.com
-      user: git
-      repo_keys:
-        "repo_path_1": 'local/path/to/key1'
-        "repo_path_2": 'local/path/to/key1'
-    public:
-      host: github.com
-      user: git

data/test/fixtures/code2.sh

@@ -1 +0,0 @@
-tail @file

data/test/fixtures/integration_expected.yml

@@ -1,37 +0,0 @@
----
-- path: "Runner_mobilize(test)/jobs"
-  state: working
-  count: 1
-  confirmed_ats: []
-- path: "Runner_mobilize(test)/jobs/ssh1/stage1"
-  state: working
-  count: 1
-  confirmed_ats: []
-- path: "Runner_mobilize(test)/jobs/ssh1/stage2"
-  state: working
-  count: 1
-  confirmed_ats: []
-- path: "Runner_mobilize(test)/jobs/ssh2/stage1"
-  state: working
-  count: 1
-  confirmed_ats: []
-- path: "Runner_mobilize(test)/jobs/ssh2/stage2"
-  state: working
-  count: 1
-  confirmed_ats: []
-- path: "Runner_mobilize(test)/jobs/ssh3/stage1"
-  state: working
-  count: 1
-  confirmed_ats: []
-- path: "Runner_mobilize(test)/jobs/ssh3/stage2"
-  state: working
-  count: 1
-  confirmed_ats: []
-- path: "Runner_mobilize(test)/jobs/ssh4/stage1"
-  state: working
-  count: 1
-  confirmed_ats: []
-- path: "Runner_mobilize(test)/jobs/ssh4/stage2"
-  state: working
-  count: 1
-  confirmed_ats: []

data/test/fixtures/integration_jobs.yml

@@ -1,25 +0,0 @@
----
-- name: ssh1
-  active: true
-  trigger: once
-  status: ""
-  stage1: ssh.run node:"test_node", cmd:"ruby code.rb", user:"root", sources:["code.rb", "code.sh"]
-  stage2: gsheet.write source:"stage1", target:"ssh1.out"
-- name: ssh2
-  active: true
-  trigger: "after ssh1"
-  status: ""
-  stage1: ssh.run cmd:"sh code2.sh", user:"root", sources:["code2.sh","test_node/var/log/syslog"], params:{file:"syslog"}
-  stage2: gsheet.write source:"stage1", target:"ssh2.out"
-- name: ssh3
-  active: true
-  trigger: "after ssh2"
-  status: ""
-  stage1: ssh.run cmd:"echo '@test_param'", params:{test_param:"test param successful"}
-  stage2: gsheet.write source:"stage1", target:"ssh3.out"
-- name: ssh4
-  active: true
-  trigger: "after ssh3"
-  status: ""
-  stage1: ssh.run node:"test_node", user:root, sources:["git://DeNA/mobilize-ssh/test/code.rb","git://DeNA/mobilize-ssh/test/code.sh"], cmd:"ruby code.rb"
-  stage2: gsheet.write source:stage1, target:"ssh4.out"

data/test/integration/mobilize-ssh_test.rb

@@ -1,46 +0,0 @@
-require 'test_helper'
-describe "Mobilize" do
-  # enqueues 4 workers on Resque
-  it "runs integration test" do
-
-    puts "restart workers"
-    Mobilize::Jobtracker.restart_workers!
-
-    u = TestHelper.owner_user
-    r = u.runner
-    user_name = u.name
-    gdrive_slot = u.email
-
-    puts "add test code"
-    ["code.rb","code.sh","code2.sh"].each do |fixture_name|
-      target_url = "gsheet://#{r.title}/#{fixture_name}"
-      TestHelper.write_fixture(fixture_name, target_url, 'replace')
-    end
-
-    puts "add/update jobs"
-    u.jobs.each{|j| j.delete}
-    jobs_fixture_name = "integration_jobs"
-    jobs_target_url = "gsheet://#{r.title}/jobs"
-    TestHelper.write_fixture(jobs_fixture_name, jobs_target_url, 'update')
-
-    puts "job rows added, force enqueue runner, wait for stages"
-    #wait for stages to complete
-    expected_fixture_name = "integration_expected"
-    Mobilize::Jobtracker.stop!
-    r.enqueue!
-    TestHelper.confirm_expected_jobs(expected_fixture_name)
-
-    puts "update job status and activity"
-    r.update_gsheet(gdrive_slot)
-
-    puts "jobtracker posted data to test sheets"
-    ['ssh1.out','ssh2.out','ssh4.out'].each do |out_name|
-      url = "gsheet://#{r.title}/#{out_name}"
-      assert TestHelper.check_output(url, 'min_length' => 100) == true
-    end
-
-    #shorter
-    url = "gsheet://#{r.title}/ssh3.out"
-    assert TestHelper.check_output(url, 'min_length' => 3) == true
-  end
-end