mobilize-ssh 1.35 → 1.36

data/README.md

@@ -1,291 +1,4 @@
-Mobilize-Ssh
-============
+Mobilize
+========
 
-Mobilize-Ssh adds the power of ssh to [mobilize-base][mobilize-base].
-* move files, execute scripts, and output logs and datasets, all through
-Google Spreadsheets.
-
-Table Of Contents
------------------
-* [Overview](#section_Overview)
-* [Install](#section_Install)
-  * [Mobilize-Ssh](#section_Install_Mobilize-Ssh)
-  * [Install Dirs and Files](#section_Install_Dirs_and_Files)
-* [Configure](#section_Configure)
-  * [Ssh](#section_Configure_Ssh)
-  * [Git](#section_Configure_Git)
-* [Start](#section_Start)
-  * [Create Job](#section_Start_Create_Job)
-  * [Run Test](#section_Start_Run_Test)
-* [Meta](#section_Meta)
-* [Author](#section_Author)
-
-<a name='section_Overview'></a>
-Overview
------------
-
-* Mobilize-ssh adds script deployment to mobilize-base.
-
-<a name='section_Install'></a>
-Install
-------------
-
-Make sure you go through all the steps in the [mobilize-base][mobilize-base]
-install section first.
-
-<a name='section_Install_Mobilize-Ssh'></a>
-### Mobilize-Ssh
-
-add this to your Gemfile:
-
-``` ruby
-gem "mobilize-ssh"
-```
-
-or do
-
-  $ gem install mobilize-ssh
-
-for a ruby-wide install.
-
-<a name='section_Install_Dirs_and_Files'></a>
-### Dirs and Files
-
-### Rakefile
-
-Inside the Rakefile in your project's root dir, make sure you have:
-
-``` ruby
-require 'mobilize-base/tasks'
-require 'mobilize-ssh/tasks'
-```
-
-This defines rake tasks essential to run the environment.
-
-### Config Dir
-
-run 
-
-  $ rake mobilize_ssh:setup
-
-This will copy over a sample ssh.yml to your config dir.
-
-It will also add mobilize-ssh to the extensions in jobtracker.yml.
-
-<a name='section_Configure'></a>
-Configure
-------------
-
-<a name='section_Configure_Ssh'></a>
-### Configure Ssh
-
-The Ssh configuration consists of:
-* nodes, identified by aliases, such as `test_node`. This alias is what you should
-pass into the "node" param over in the ssh.run task.
-  * if no node is specified, commands will default to the first node listed.
-
-Each node has: 
-* a host;
-* a gateway (optional); If you don't need a gateway, remove that row from the configuration file.
-* sudoers; these are user names that are allowed to pass user params
-to the run call. This requires passwordless sudo for the host user.
-* su_all_users true/false option, which ensures that commands are executed by the
-user on the Runner. It prefixes all commands with sudo su <user> before executing the
-command. This is strongly recommended if possible as it ensures users do
-not overstep their permissions. This requires passwordless sudo for the
-host user and accounts on the host machine for each user.
-
-Each host and gateway has a series of ssh params:
-* name - the ip address or name of the host
-* key - the relative path of the ssh key file. Default is
-"config/mobilize/ssh_private.key"
-* port - the port to connect on
-* user - the user you are connecting as
-
-Sample ssh.yml:
-
-``` yml
----
-development:
-  nodes:
-    dev_node:
-      sudoers: 
-      - sudo_user
-      su_all_users: true
-      host: 
-        name: dev-host.com 
-        key: config/mobilize/ssh_private.key
-        port: 22
-        user: host_user
-      gateway: 
-        name: dev-gateway.com 
-        key: config/mobilize/ssh_private.key 
-        port: 22 
-        user: gateway_user
-test:
-  nodes:
-    test_node:
-      sudoers: 
-      - sudo_user
-      su_all_users: true
-      host: 
-        name: test-host.com 
-        key: config/mobilize/ssh_private.key 
-        port: 22 
-        user: host_user
-      gateway: 
-        name: test-gateway.com 
-        key: config/mobilize/ssh_private.key 
-        port: 22 
-        user: gateway_user
-production:
-  nodes:
-    prod_node:
-      sudoers: 
-      - sudo_user
-      su_all_users: true
-      host:
-        name: prod-host.com 
-        key: config/mobilize/ssh_private.key 
-        port: 22 
-        user: host_user
-      gateway: 
-        name: prod-gateway.com 
-        key: config/mobilize/ssh_private.key 
-        port: 22 
-        user: gateway_user
-```
-
-<a name='section_Configure_Git'></a>
-### Configure Git
-
-Git configuration is not required but recommended, as it allows you to
-pull files directly from public or private Git repositories.
-
-The Git configuration consists of:
-* domains, identified by aliases, such as `private` and `public`.
-  * domains are passed into the source parameters in the git.run task.
-  * if no domain is specified, commands will default to the first domain listed.
-
-Each domain has: 
-* a host;
-* a user, which is the user used for the git clone command.
-* a set of repo keys (optional) which correspond to "deploy keys" on
-github.  Each repo must have its own ssh key, and the public key must be
-stored in the repo.
-  * if your repo doesn't need an ssh key to work, this is not necessary
-to add.
-
-
-Sample git.yml:
-
-``` yml
----
-development:
-  domains:
-    private:
-      host: github.private.com
-      user: git
-      repo_keys:
-        "repo_path_1": 'local/path/to/key1'
-        "repo_path_2": 'local/path/to/key1'
-    public:
-      host: github.com
-      user: git
-test:
-  domains:
-    public:
-      host: github.com
-      user: git
-    private:
-      host: github.private.com
-      user: git
-      repo_keys:
-        "repo_path_1": 'local/path/to/key1'
-        "repo_path_2": 'local/path/to/key1'
-production:
-  domains:
-    private:
-      host: github.private.com
-      user: git
-      repo_keys:
-        "repo_path_1": 'local/path/to/key1'
-        "repo_path_2": 'local/path/to/key1'
-    public:
-      host: github.com
-      user: git
-```
-
-
-<a name='section_Start'></a>
-Start
------
-
-<a name='section_Start_Create_Job'></a>
-### Create Job
-
-* For mobilize-ssh, the following task is available:
-  * ssh.run `node: <node_alias>, cmd: <command>, user: user, sources:[*<source_paths>], params:[{<key,value pairs>}]`, which reads sources, copies them to a temporary folder on the selected node, and runs the command inside that folder. 
-  * user, sources, node, and params are optional; cmd is required. 
-  * specifying user will cause the command to be prefixed with sudo su <user> -c. 
-    * non-google sources will also be read as the specified user.
-  * git sources can be specified with syntax `git://<domain>/<repo_owner>/<repo_name>/<file_path>`. 
-    * Accessing private repos requires that you add the Mobilize public key to the repository as a deploy key.
-      * there is no user-level access control for git repositories at this time.
-    * domain defaults to the first one listed, if not included.
-  * params are also optional for all of the below. They replace tokens in sources and the command.
-    * params are passed as a YML or JSON, as in:
-      * `ssh.run source:<source_path>, params:{'date':'2013-03-01', 'unit':'widgets'}`
-        * this example replaces all the keys, preceded by '@' in all source hqls with the value.
-          * The preceding '@' is used to keep from replacing instances
-            of "date" and "unit" in the command/source file; you should have `@date` and `@unit` in your actual HQL 
-            if you'd like to replace those tokens.
-  * not specifying node will cause the command to be run on the default node.
-  * ssh sources can be specified with syntax
-`ssh://<node><file_full_path>`. If node is omitted, default node will be used.
-  * `<node><file_full_path>` and `<file_full_path>` can be used in the context of ssh.run, but if the path has only 1 slash, or none, it will try to find a google sheet or file instead.
-  * The test uses `ssh.run node:"test_node", cmd:"ruby code.rb", user: "root", sources:["code.rb","code.sh"]`
-
-<a name='section_Start_Run_Test'></a>
-### Run Test
-
-To run tests, you will need to 
-
-1) go through the [mobilize-base][mobilize-base] test first
-
-2) clone the mobilize-ssh repository 
-
-From the project folder, run
-
-3) $ rake mobilize_ssh:setup
-
-Copy over the config files from the mobilize-base project into the
-config dir, and populate the values in the ssh.yml file, esp. the
-test_node item.
-
-You should also copy the ssh private key you wish to use into your
-desired path (by default: config/mobilize/ssh_private.key), and make sure it is referenced in ssh.yml
-
-3) $ rake test
-
-This will populate your test Runner from mobilize-base with a sample ssh job.
-
-The purpose of the test will be to deploy two code files, have the first
-execute the second, which is a "tail /var/log/syslog" command, and write the resulting output to a gsheet.
-
-<a name='section_Meta'></a>
-Meta
-----
-
-* Code: `git clone git://github.com/dena/mobilize-ssh.git`
-* Home: <https://github.com/dena/mobilize-ssh>
-* Bugs: <https://github.com/dena/mobilize-ssh/issues>
-* Gems: <http://rubygems.org/gems/mobilize-ssh>
-
-<a name='section_Author'></a>
-Author
-------
-
-Cassio Paes-Leme :: cpaesleme@dena.com :: @cpaesleme
-
-[mobilize-base]: https://github.com/dena/mobilize-base
+Please refer to the mobilize-server wiki: https://github.com/DeNA/mobilize-server/wiki

data/lib/mobilize-ssh/tasks.rb

@@ -1,7 +1,7 @@
-namespace :mobilize_ssh do
+require 'yaml'
+namespace :mobilize do
   desc "Set up config and log folders and files"
-  task :setup do
-    require 'yaml'
+  task :setup_ssh do
     sample_dir = File.dirname(__FILE__) + '/../samples/'
     sample_files = Dir.entries(sample_dir)
     config_dir = (ENV['MOBILIZE_CONFIG_DIR'] ||= "config/mobilize/")

data/lib/mobilize-ssh/version.rb

@@ -1,5 +1,5 @@
 module Mobilize
   module Ssh
-    VERSION = "1.35"
+    VERSION = "1.36"
   end
 end

data/lib/mobilize-ssh.rb

@@ -9,6 +9,9 @@ require "mobilize-ssh/extensions/socket"
 
 module Mobilize
   module Ssh
+    def Ssh.home_dir
+      File.expand_path('..',File.dirname(__FILE__))
+    end
   end
 end
 require "mobilize-ssh/handlers/ssh"

data/mobilize-ssh.gemspec

@@ -16,7 +16,7 @@ Gem::Specification.new do |gem|
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
-  gem.add_runtime_dependency "mobilize-base","1.35"
+  gem.add_runtime_dependency "mobilize-base","1.36"
   gem.add_runtime_dependency "net-ssh"
   gem.add_runtime_dependency "net-scp"
   gem.add_runtime_dependency "net-ssh-gateway"

data/test/fixtures/integration_expected.yml

@@ -0,0 +1,37 @@
+---
+- path: "Runner_mobilize(test)/jobs"
+  state: working
+  count: 1
+  confirmed_ats: []
+- path: "Runner_mobilize(test)/jobs/ssh1/stage1"
+  state: working
+  count: 1
+  confirmed_ats: []
+- path: "Runner_mobilize(test)/jobs/ssh1/stage2"
+  state: working
+  count: 1
+  confirmed_ats: []
+- path: "Runner_mobilize(test)/jobs/ssh2/stage1"
+  state: working
+  count: 1
+  confirmed_ats: []
+- path: "Runner_mobilize(test)/jobs/ssh2/stage2"
+  state: working
+  count: 1
+  confirmed_ats: []
+- path: "Runner_mobilize(test)/jobs/ssh3/stage1"
+  state: working
+  count: 1
+  confirmed_ats: []
+- path: "Runner_mobilize(test)/jobs/ssh3/stage2"
+  state: working
+  count: 1
+  confirmed_ats: []
+- path: "Runner_mobilize(test)/jobs/ssh4/stage1"
+  state: working
+  count: 1
+  confirmed_ats: []
+- path: "Runner_mobilize(test)/jobs/ssh4/stage2"
+  state: working
+  count: 1
+  confirmed_ats: []

data/test/{ssh_job_rows.yml → fixtures/integration_jobs.yml}

@@ -1,25 +1,25 @@
 ---
-- name: test_ssh_1
+- name: ssh1
   active: true
   trigger: once
   status: ""
   stage1: ssh.run node:"test_node", cmd:"ruby code.rb", user:"root", sources:["code.rb", "code.sh"]
-  stage2: gsheet.write source:"stage1", target:"test_ssh_1.out"
-- name: test_ssh_2
+  stage2: gsheet.write source:"stage1", target:"ssh1.out"
+- name: ssh2
   active: true
-  trigger: "after test_ssh_1"
+  trigger: "after ssh1"
   status: ""
   stage1: ssh.run cmd:"sh code2.sh", user:"root", sources:["code2.sh","test_node/var/log/syslog"], params:{file:"syslog"}
-  stage2: gsheet.write source:"stage1", target:"test_ssh_2.out"
-- name: test_ssh_3
+  stage2: gsheet.write source:"stage1", target:"ssh2.out"
+- name: ssh3
   active: true
-  trigger: "after test_ssh_2"
+  trigger: "after ssh2"
   status: ""
   stage1: ssh.run cmd:"echo '@test_param'", params:{test_param:"test param successful"}
-  stage2: gsheet.write source:"stage1", target:"test_ssh_3.out"
-- name: test_ssh_4
+  stage2: gsheet.write source:"stage1", target:"ssh3.out"
+- name: ssh4
   active: true
-  trigger: "after test_ssh_3"
+  trigger: "after ssh3"
   status: ""
   stage1: ssh.run node:"test_node", user:root, sources:["git://DeNA/mobilize-ssh/test/code.rb","git://DeNA/mobilize-ssh/test/code.sh"], cmd:"ruby code.rb"
-  stage2: gsheet.write source:stage1, target:"test_ssh_4.out"
+  stage2: gsheet.write source:stage1, target:"ssh4.out"

data/test/integration/mobilize-ssh_test.rb

@@ -0,0 +1,46 @@
+require 'test_helper'
+describe "Mobilize" do
+  # enqueues 4 workers on Resque
+  it "runs integration test" do
+
+    puts "restart workers"
+    Mobilize::Jobtracker.restart_workers!
+
+    u = TestHelper.owner_user
+    r = u.runner
+    user_name = u.name
+    gdrive_slot = u.email
+
+    puts "add test code"
+    ["code.rb","code.sh","code2.sh"].each do |fixture_name|
+      target_url = "gsheet://#{r.title}/#{fixture_name}"
+      TestHelper.write_fixture(fixture_name, target_url, 'replace')
+    end
+
+    puts "add/update jobs"
+    u.jobs.each{|j| j.delete}
+    jobs_fixture_name = "integration_jobs"
+    jobs_target_url = "gsheet://#{r.title}/jobs"
+    TestHelper.write_fixture(jobs_fixture_name, jobs_target_url, 'update')
+
+    puts "job rows added, force enqueue runner, wait for stages"
+    #wait for stages to complete
+    expected_fixture_name = "integration_expected"
+    Mobilize::Jobtracker.stop!
+    r.enqueue!
+    TestHelper.confirm_expected_jobs(expected_fixture_name)
+
+    puts "update job status and activity"
+    r.update_gsheet(gdrive_slot)
+
+    puts "jobtracker posted data to test sheets"
+    ['ssh1.out','ssh2.out','ssh4.out'].each do |out_name|
+      url = "gsheet://#{r.title}/#{out_name}"
+      assert TestHelper.check_output(url, 'min_length' => 100) == true
+    end
+
+    #shorter
+    url = "gsheet://#{r.title}/ssh3.out"
+    assert TestHelper.check_output(url, 'min_length' => 3) == true
+  end
+end

data/test/test_helper.rb

@@ -8,3 +8,4 @@ $dir = File.dirname(File.expand_path(__FILE__))
 ENV['MOBILIZE_ENV'] = 'test'
 require 'mobilize-ssh'
 $TESTING = true
+require "#{Mobilize::Base.home_dir}/test/test_helper"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: mobilize-ssh
 version: !ruby/object:Gem::Version
-  version: '1.35'
+  version: '1.36'
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-04-23 00:00:00.000000000 Z
+date: 2013-05-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: mobilize-base
@@ -18,7 +18,7 @@ dependencies:
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: '1.35'
+        version: '1.36'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -26,7 +26,7 @@ dependencies:
     requirements:
     - - '='
       - !ruby/object:Gem::Version
-        version: '1.35'
+        version: '1.36'
 - !ruby/object:Gem::Dependency
   name: net-ssh
   requirement: !ruby/object:Gem::Requirement
@@ -99,12 +99,13 @@ files:
 - lib/samples/git.yml
 - lib/samples/ssh.yml
 - mobilize-ssh.gemspec
-- test/code.rb
-- test/code.sh
-- test/code2.sh
-- test/mobilize-ssh_test.rb
+- test/fixtures/code.rb
+- test/fixtures/code.sh
+- test/fixtures/code2.sh
+- test/fixtures/integration_expected.yml
+- test/fixtures/integration_jobs.yml
+- test/integration/mobilize-ssh_test.rb
 - test/redis-test.conf
-- test/ssh_job_rows.yml
 - test/test_helper.rb
 homepage: http://github.com/dena/mobilize-ssh
 licenses: []
@@ -120,7 +121,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1934183260655929417
+      hash: 3534560230256585865
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -129,7 +130,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1934183260655929417
+      hash: 3534560230256585865
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.25
@@ -137,10 +138,11 @@ signing_key:
 specification_version: 3
 summary: extend mobilize-base with the ability to run files across hosts
 test_files:
-- test/code.rb
-- test/code.sh
-- test/code2.sh
-- test/mobilize-ssh_test.rb
+- test/fixtures/code.rb
+- test/fixtures/code.sh
+- test/fixtures/code2.sh
+- test/fixtures/integration_expected.yml
+- test/fixtures/integration_jobs.yml
+- test/integration/mobilize-ssh_test.rb
 - test/redis-test.conf
-- test/ssh_job_rows.yml
 - test/test_helper.rb

data/test/mobilize-ssh_test.rb

@@ -1,95 +0,0 @@
-require 'test_helper'
-
-describe "Mobilize" do
-
-  def before
-    puts 'nothing before'
-  end
-
-  # enqueues 4 workers on Resque
-  it "runs integration test" do
-
-    puts "restart workers"
-    Mobilize::Jobtracker.restart_workers!
-
-    gdrive_slot = Mobilize::Gdrive.owner_email
-    puts "create user 'mobilize'"
-    user_name = gdrive_slot.split("@").first
-    u = Mobilize::User.where(:name=>user_name).first
-    r = u.runner
-
-    rb_code_sheet = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/code.rb",gdrive_slot)
-    sh_code_sheet = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/code.sh",gdrive_slot)
-    sh_code_sheet2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/code2.sh",gdrive_slot)
-    [rb_code_sheet,sh_code_sheet].each {|s| s.delete if s}
-
-    puts "add test code"
-    rb_code_sheet = Mobilize::Gsheet.find_or_create_by_path("#{r.path.split("/")[0..-2].join("/")}/code.rb",gdrive_slot)
-    rb_code_tsv = File.open("#{Mobilize::Base.root}/test/code.rb").read
-    rb_code_sheet.write(rb_code_tsv,Mobilize::Gdrive.owner_name)
-
-    sh_code_sheet = Mobilize::Gsheet.find_or_create_by_path("#{r.path.split("/")[0..-2].join("/")}/code.sh",gdrive_slot)
-    sh_code_tsv = File.open("#{Mobilize::Base.root}/test/code.sh").read
-    sh_code_sheet.write(sh_code_tsv,Mobilize::Gdrive.owner_name)
-
-    sh_code_sheet2 = Mobilize::Gsheet.find_or_create_by_path("#{r.path.split("/")[0..-2].join("/")}/code2.sh",gdrive_slot)
-    sh_code_tsv2 = File.open("#{Mobilize::Base.root}/test/code2.sh").read
-    sh_code_sheet2.write(sh_code_tsv2,Mobilize::Gdrive.owner_name)
-
-    jobs_sheet = r.gsheet(gdrive_slot)
-
-    #delete target sheets if they exist
-    ssh_target_sheet_1 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_1.out",gdrive_slot)
-    ssh_target_sheet_2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_2.out",gdrive_slot)
-    ssh_target_sheet_3 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_3.out",gdrive_slot)
-    ssh_target_sheet_4 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_4.out",gdrive_slot)
-    [ssh_target_sheet_1,ssh_target_sheet_2,ssh_target_sheet_3].each {|s| s.delete if s}
-
-    ssh_job_rows = ::YAML.load_file("#{Mobilize::Base.root}/test/ssh_job_rows.yml")
-    ssh_job_rows.map{|j| r.jobs(j['name'])}.each{|j| j.delete if j}
-    jobs_sheet.add_or_update_rows(ssh_job_rows)
-
-    puts "job row added, force enqueue runner, wait for stages"
-    r.enqueue!
-    wait_for_stages(900)
-
-    puts "update job status and activity"
-    r.update_gsheet(gdrive_slot)
-
-    puts "jobtracker posted data to test sheet"
-    ssh_target_sheet_1 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_1.out",gdrive_slot)
-    ssh_target_sheet_2 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_2.out",gdrive_slot)
-    ssh_target_sheet_3 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_3.out",gdrive_slot)
-    ssh_target_sheet_4 = Mobilize::Gsheet.find_by_path("#{r.path.split("/")[0..-2].join("/")}/test_ssh_4.out",gdrive_slot)
-
-    assert ssh_target_sheet_1.to_tsv.length > 100
-    assert ssh_target_sheet_2.to_tsv.length > 100
-    assert ssh_target_sheet_3.to_tsv.length > 3
-    assert ssh_target_sheet_4.to_tsv.length > 100
-
-  end
-
-  def wait_for_stages(time_limit=600,stage_limit=120,wait_length=10)
-    time = 0
-    time_since_stage = 0
-    #check for 10 min
-    while time < time_limit and time_since_stage < stage_limit
-      sleep wait_length
-      job_classes = Mobilize::Resque.jobs.map{|j| j['class']}
-      if job_classes.include?("Mobilize::Stage")
-        time_since_stage = 0
-        puts "saw stage at #{time.to_s} seconds"
-      else
-        time_since_stage += wait_length
-        puts "#{time_since_stage.to_s} seconds since stage seen"
-      end
-      time += wait_length
-      puts "total wait time #{time.to_s} seconds"
-    end
-
-    if time >= time_limit
-      raise "Timed out before stage completion"
-    end
-  end
-
-end