sshkit 1.18.0 → 1.23.0

data/CONTRIBUTING.md

@@ -24,8 +24,8 @@ using unsupported features.
 
 ## Tests
 
-SSHKit has a unit test suite and a functional test suite. Some functional tests run against
-[Vagrant](https://www.vagrantup.com/) VMs. If possible, you should make sure that the
+SSHKit has a unit test suite and a functional test suite. Some functional tests run using
+[Docker](https://docs.docker.com/get-docker/). If possible, you should make sure that the
 tests pass for each commit by running `rake` in the sshkit directory. This is in case we
 need to cherry pick commits or rebase. You should ensure the tests pass, (preferably on
 the minimum and maximum ruby version), before creating a PR.

data/Dangerfile

@@ -1 +1 @@
-danger.import_dangerfile(github: "capistrano/danger")
+danger.import_dangerfile(github: "capistrano/danger", branch: "no-changelog")

data/EXAMPLES.md

@@ -121,9 +121,6 @@ on hosts do |host|
 end
 ```
 
-**Note:** The `upload!()` method doesn't honor the values of `as()` etc, this
-will be improved as the library matures, but we're not there yet.
-
 ## Upload a file from a stream
 
 ```ruby
@@ -148,9 +145,6 @@ end
 This spares one from having to figure out the correct escaping sequences for
 something like "echo(:cat, '...?...', '> /etc/sudoers.d/yolo')".
 
-**Note:** The `upload!()` method doesn't honor the values of `within()`, `as()`
-etc, this will be improved as the library matures, but we're not there yet.
-
 ## Upload a directory of files
 
 ```ruby
@@ -160,7 +154,25 @@ end
 ```
 
 In this case the `recursive: true` option mirrors the same options which are
-available to [`Net::{SCP,SFTP}`](http://net-ssh.github.io/net-scp/).
+available to [`Net::SCP`](https://github.com/net-ssh/net-scp) and
+[`Net::SFTP`](https://github.com/net-ssh/net-sftp).
+
+## Set the upload/download method (SCP or SFTP).
+
+SSHKit can use SCP or SFTP for file transfers. The default is SCP, but this can be changed to SFTP per host:
+
+```ruby
+host = SSHKit::Host.new('user@example.com')
+host.transfer_method = :sftp
+```
+
+or globally:
+
+```ruby
+SSHKit::Backend::Netssh.configure do |ssh|
+  ssh.transfer_method = :sftp
+end
+```
 
 ## Setting global SSH options
 
@@ -171,6 +183,7 @@ individual hosts:
 SSHKit::Backend::Netssh.configure do |ssh|
   ssh.connection_timeout = 30
   ssh.ssh_options = {
+    user: 'adifferentuser',
     keys: %w(/home/user/.ssh/id_rsa),
     forward_agent: false,
     auth_methods: %w(publickey password)
@@ -234,16 +247,16 @@ end
 
 ```ruby
 # The default format is pretty, which outputs colored text
-SSHKit.config.format = :pretty
+SSHKit.config.use_format :pretty
 
 # Text with no coloring
-SSHKit.config.format = :simpletext
+SSHKit.config.use_format :simpletext
 
 # Red / Green dots for each completed step
-SSHKit.config.format = :dot
+SSHKit.config.use_format :dot
 
 # No output
-SSHKit.config.format = :blackhole
+SSHKit.config.use_format :blackhole
 ```
 
 ## Implement a dirt-simple formatter class
@@ -253,7 +266,7 @@ module SSHKit
   module Formatter
     class MyFormatter < SSHKit::Formatter::Abstract
       def write(obj)
-        case obj.is_a? SSHKit::Command
+        if obj.is_a? SSHKit::Command
           # Do something here, see the SSHKit::Command documentation
         end
       end
@@ -262,7 +275,7 @@ module SSHKit
 end
 
 # If your formatter is defined in the SSHKit::Formatter module configure with the format option:
-SSHKit.config.format = :myformatter
+SSHKit.config.use_format :myformatter
 
 # Or configure the output directly
 SSHKit.config.output = MyFormatter.new($stdout)
@@ -338,6 +351,15 @@ end
 This will resolve the `example.com` hostname into a `SSHKit::Host` object, and
 try to pull up the correct configuration for it.
 
+## Connect to a host on a port different than 22
+
+If your ssh server is running on a port different than 22, you can change this is
+shown:
+
+```ruby
+on('example.com', {port: 1234}) do
+end
+```
 
 ## Run a command without it being command-mapped
 

data/Gemfile

@@ -2,18 +2,6 @@ source 'https://rubygems.org'
 
 gemspec
 
-platforms :rbx do
-  gem 'rubysl', '~> 2.0'
-  gem 'json'
-end
-
-# Chandler requires Ruby >= 2.1.0, but depending on the Travis environment,
-# we may not meet that requirement. Only include the chandler gem if the Ruby
-# requirement is met. (Chandler is used only for `rake release`; see Rakefile.)
-if Gem::Requirement.new('>= 2.1.0').satisfied_by?(Gem::Version.new(RUBY_VERSION))
-  gem 'chandler', '>= 0.1.1'
-end
-
 # public_suffix 3+ requires ruby 2.1+
 if Gem::Requirement.new('< 2.1').satisfied_by?(Gem::Version.new(RUBY_VERSION))
   gem 'public_suffix', '< 3'

data/README.md

@@ -4,31 +4,33 @@
 more servers.
 
 [![Gem Version](https://badge.fury.io/rb/sshkit.svg)](https://rubygems.org/gems/sshkit)
-[![Build Status](https://travis-ci.org/capistrano/sshkit.svg?branch=master)](https://travis-ci.org/capistrano/sshkit)
+[![Build Status](https://github.com/capistrano/sshkit/actions/workflows/ci.yml/badge.svg)](https://github.com/capistrano/sshkit/actions/workflows/ci.yml)
 
-## How might it work?
+## Example
 
-The typical use-case looks something like this:
+ - Connect to 2 servers
+ - Execute commands as `deploy` user with `RAILS_ENV=production`
+ - Execute commands in serial (default is `:parallel`)
 
 ```ruby
 require 'sshkit'
 require 'sshkit/dsl'
 include SSHKit::DSL
 
-on %w{1.example.com 2.example.com}, in: :sequence, wait: 5 do |host|
+on ["1.example.com", "2.example.com"], in: :sequence do |host|
+  puts "Now executing on #{host}"
   within "/opt/sites/example.com" do
     as :deploy  do
-      with rails_env: :production do
-        rake   "assets:precompile"
-        runner "S3::Sync.notify"
-        execute :node, "socket_server.js"
+      with RAILS_ENV: 'production' do
+        execute :rake, "assets:precompile"
+        execute :rails, "runner", "S3::Sync.notify"
       end
     end
   end
 end
 ```
 
-You can find many other examples of how to use SSHKit over in [EXAMPLES.md](EXAMPLES.md).
+Many other examples are in [EXAMPLES.md](EXAMPLES.md).
 
 ## Basic usage
 
@@ -37,7 +39,7 @@ You can pass one or more hosts as parameters; this runs commands via SSH. Altern
 pass `:local` to run commands locally. By default SSKit will run the commands on all hosts in
 parallel.
 
-#### Running commands
+### Running commands
 
 All backends support the `execute(*args)`, `test(*args)` & `capture(*args)` methods
 for executing a command. You can call any of these methods in the context of an `on()`
@@ -63,19 +65,20 @@ end
 By default the `capture` methods strips whitespace. If you need to preserve whitespace
 you can pass the `strip: false` option: `capture(:ls, '-l', strip: false)`
 
-#### Transferring files
+### Transferring files
 
 All backends also support the `upload!` and `download!` methods for transferring files.
-For the remote backend, the file is transferred with scp.
+For the remote backend, the file is transferred with scp by default, but sftp is also
+supported. See [EXAMPLES.md](EXAMPLES.md) for details.
 
 ```ruby
 on '1.example.com' do
   upload! 'some_local_file.txt', '/home/some_user/somewhere'
-  download! '/home/some_user/some_remote_file.txt', 'somewhere_local', :log_percent 25
+  download! '/home/some_user/some_remote_file.txt', 'somewhere_local', log_percent: 25
 end
 ```
 
-#### Users, working directories, environment variables and umask
+### Users, working directories, environment variables and umask
 
 When running commands, you can tell SSHKit to set up the context for those
 commands using the following methods:
@@ -113,6 +116,11 @@ the raised error.
 Helpers such as `runner()` and `rake()` which expand to `execute(:rails, "runner", ...)` and
 `execute(:rake, ...)` are convenience helpers for Ruby, and Rails based apps.
 
+### Verbosity / Silence
+
+ - raise verbosity of a command: `execute "echo DEAD", verbosity: :ERROR`
+ - hide a command from output: `execute "echo HIDDEN", verbosity: :DEBUG`
+
 ## Parallel
 
 Notice on the `on()` call the `in: :sequence` option, the following will do
@@ -567,10 +575,20 @@ In order to do special gymnastics with SSH, tunneling, aliasing, complex options
 
 These system level files are the preferred way of setting up tunneling and proxies because the system implementations of these things are faster and better than the Ruby implementations you would get if you were to configure them through Net::SSH. In cases where it's not possible (Windows?), it should be possible to make use of the Net::SSH APIs to setup tunnels and proxy commands before deferring control to Capistrano/SSHKit..
 
-## SSHKit Related Blog Posts
+## Proxying
 
-[SSHKit Gem Basics](http://www.rubyplus.com/articles/591)
+To connect to the target host via a jump/bastion host, use a `Net::SSH::Proxy::Jump`
 
-[SSHKit Gem Part 2](http://www.rubyplus.com/articles/601)
+```ruby
+host = SSHKit::Host.new(
+  hostname: 'target.host.com',
+  ssh_options: { proxy: Net::SSH::Proxy::Jump.new("proxy.bar.com") }
+)
+on [host] do
+  execute :echo, '1'
+end
+```
+
+## SSHKit Related Blog Posts
 
 [Embedded Capistrano with SSHKit](http://ryandoyle.net/posts/embedded-capistrano-with-sshkit/)

data/RELEASING.md

@@ -5,14 +5,13 @@
 * You must have commit rights to the SSHKit repository.
 * You must have push rights for the sshkit gem on rubygems.org.
 * You must be using Ruby >= 2.1.0.
-* Your `~/.netrc` must be configured with your GitHub credentials, [as explained here](https://github.com/mattbrictson/chandler#2-configure-netrc).
 
 ## How to release
 
 1. Run `bundle install` to make sure that you have all the gems necessary for testing and releasing.
-2.  **Ensure the tests are passing by running `rake test`.** If functional tests fail, ensure you have [Vagrant](https://www.vagrantup.com) installed and have started it with `vagrant up`.
+2.  **Ensure the tests are passing by running `rake test`.** If functional tests fail, ensure you have [Docker installed](https://docs.docker.com/get-docker/) and running.
 3. Determine which would be the correct next version number according to [semver](http://semver.org/).
 4. Update the version in `./lib/sshkit/version.rb`.
-5. Update the `CHANGELOG`.
-6. Commit the changelog and version in a single commit, the message should be "Preparing vX.Y.Z"
-7. Run `rake release`; this will tag, push to GitHub, publish to rubygems.org, and upload the latest changelog entry to the [GitHub releases page](https://github.com/capistrano/sshkit/releases).
+5. Commit the `version.rb` change with a message like "Preparing vX.Y.Z"
+6. Run `rake release`; this will tag, push to GitHub, and publish to rubygems.org
+7. Update the draft release on the [GitHub releases page](https://github.com/capistrano/sshkit/releases) to point to the new tag and publish the release

data/Rakefile

@@ -21,19 +21,12 @@ namespace :test do
 
 end
 
-Rake::Task["test:functional"].enhance do
-  warn "Remember there are still some VMs running, kill them with `vagrant halt` if you are finished using them."
-end
-
 desc 'Run RuboCop lint checks'
 RuboCop::RakeTask.new(:lint) do |task|
   task.options = ['--lint']
 end
 
-task "release:rubygem_push" do
-  # Delay loading Chandler until this point, since it requires Ruby >= 2.1,
-  # which may not be available in all environments (e.g. Travis).
-  # We assume that the person doing `rake release` has Ruby >= 2.1.
-  require "chandler/tasks"
-  Rake.application.invoke_task("chandler:push")
+Rake::Task["release"].enhance do
+  puts "Don't forget to publish the release on GitHub!"
+  system "open https://github.com/capistrano/sshkit/releases"
 end

data/docker-compose.yml

@@ -0,0 +1,8 @@
+name: sshkit
+
+services:
+  ssh_server:
+    build:
+      context: .docker
+    ports:
+      - "2122:22"

data/examples/simple_connection.rb

@@ -0,0 +1,9 @@
+# tiny example so you can play with the sshkit or make a failing example for an issue
+require 'bundler/setup'
+require 'sshkit'
+require 'sshkit/dsl'
+include SSHKit::DSL
+
+on [ENV.fetch("HOST")] do
+  execute "echo hello"
+end

data/lib/sshkit/backends/abstract.rb

@@ -1,3 +1,5 @@
+require 'shellwords'
+
 module SSHKit
 
   module Backend
@@ -80,13 +82,14 @@ module SSHKit
 
       def within(directory, &_block)
         (@pwd ||= []).push directory.to_s
+        escaped = Command.shellescape_except_tilde(pwd_path)
         execute <<-EOTEST, verbosity: Logger::DEBUG
-          if test ! -d #{File.join(@pwd)}
-            then echo "Directory does not exist '#{File.join(@pwd)}'" 1>&2
+          if test ! -d #{escaped}
+            then echo "Directory does not exist '#{escaped}'" 1>&2
             false
           fi
-          EOTEST
-          yield
+        EOTEST
+        yield
       ensure
         @pwd.pop
       end
@@ -108,8 +111,8 @@ module SSHKit
           @group = nil
         end
         execute <<-EOTEST, verbosity: Logger::DEBUG
-          if ! sudo -u #{@user} whoami > /dev/null
-            then echo "You cannot switch to user '#{@user}' using sudo, please check the sudoers file" 1>&2
+          if ! sudo -u #{@user.to_s.shellescape} whoami > /dev/null
+            then echo "You cannot switch to user '#{@user.to_s.shellescape}' using sudo, please check the sudoers file" 1>&2
             false
           fi
         EOTEST

data/lib/sshkit/backends/connection_pool/cache.rb

@@ -36,12 +36,12 @@ class SSHKit::Backend::ConnectionPool::Cache
   def evict
     # Peek at the first connection to see if it is still fresh. If so, we can
     # return right away without needing to use `synchronize`.
-    first_expires_at, _connection = connections.first
-    return if first_expires_at.nil? || fresh?(first_expires_at)
+    first_expires_at, _first_conn = connections.first
+    return if (first_expires_at.nil? || fresh?(first_expires_at))
 
     connections.synchronize do
-      fresh, stale = connections.partition do |expires_at, _|
-        fresh?(expires_at)
+      fresh, stale = connections.partition do |expires_at, conn|
+        fresh?(expires_at) && !closed?(conn)
       end
       connections.replace(fresh)
       stale.each { |_, conn| closer.call(conn) }
@@ -71,6 +71,13 @@ class SSHKit::Backend::ConnectionPool::Cache
   end
 
   def closed?(conn)
-    conn.respond_to?(:closed?) && conn.closed?
+    return true if conn.respond_to?(:closed?) && conn.closed?
+    # test if connection is alive
+    conn.process(0) if conn.respond_to?(:process)
+    return false
+  rescue IOError => e
+    # connection is closed by server
+    return true if e.message == 'closed stream'
+    raise
   end
 end

data/lib/sshkit/backends/connection_pool.rb

@@ -21,7 +21,7 @@ end
 
 # The ConnectionPool caches connections and allows them to be reused, so long as
 # the reuse happens within the `idle_timeout` period. Timed out connections are
-# closed, forcing a new connection to be used in that case.
+# eventually closed, forcing a new connection to be used in that case.
 #
 # Additionally, a background thread is started to check for abandoned
 # connections that have timed out without any attempt at being reused. These
@@ -46,7 +46,11 @@ class SSHKit::Backend::ConnectionPool
     @caches = {}
     @caches.extend(MonitorMixin)
     @timed_out_connections = Queue.new
-    Thread.new { run_eviction_loop }
+
+    # Spin up eviction loop only if caching is enabled
+    if cache_enabled?
+      Thread.new { run_eviction_loop }
+    end
   end
 
   # Creates a new connection or reuses a cached connection (if possible) and
@@ -118,9 +122,9 @@ class SSHKit::Backend::ConnectionPool
   # Update cache key with changed args to prevent cache miss
   def update_key_if_args_changed(cache, args)
     new_key = cache_key_for_connection_args(args)
-    return if cache.same_key?(new_key)
 
     caches.synchronize do
+      return if cache.same_key?(new_key)
       caches[new_key] = caches.delete(cache.key)
       cache.key = new_key
     end
@@ -133,7 +137,7 @@ class SSHKit::Backend::ConnectionPool
       process_deferred_close
 
       # Periodically sweep all Caches to evict stale connections
-      sleep([idle_timeout, 5].min)
+      sleep(5)
       caches.values.each(&:evict)
     end
   end

data/lib/sshkit/backends/netssh/known_hosts.rb

@@ -1,3 +1,5 @@
+require "base64"
+
 module SSHKit
 
   module Backend
@@ -5,12 +7,11 @@ module SSHKit
     class Netssh < Abstract
 
       class KnownHostsKeys
-        include Mutex_m
-
         def initialize(path)
           super()
           @path = File.expand_path(path)
           @hosts_keys = nil
+          @mutex = Mutex.new
         end
 
         def keys_for(hostlist)
@@ -44,7 +45,7 @@ module SSHKit
         end
 
         def parse_file
-          synchronize do
+          @mutex.synchronize do
             return if hosts_keys && hosts_hashes
 
             unless File.readable?(path)
@@ -110,11 +111,10 @@ module SSHKit
       end
 
       class KnownHosts
-        include Mutex_m
-
         def initialize
           super()
           @files = {}
+          @mutex = Mutex.new
         end
 
         def search_for(host, options = {})
@@ -126,13 +126,13 @@ module SSHKit
 
         def add(*args)
           ::Net::SSH::KnownHosts.add(*args)
-          synchronize { @files = {} }
+          @mutex.synchronize { @files = {} }
         end
 
         private
 
         def known_hosts_file(path)
-          @files[path] || synchronize { @files[path] ||= KnownHostsKeys.new(path) }
+          @files[path] || @mutex.synchronize { @files[path] ||= KnownHostsKeys.new(path) }
         end
       end
 
@@ -140,4 +140,4 @@ module SSHKit
 
   end
 
-end
+end

data/lib/sshkit/backends/netssh/scp_transfer.rb

@@ -0,0 +1,26 @@
+require "net/scp"
+
+module SSHKit
+  module Backend
+    class Netssh < Abstract
+      class ScpTransfer
+        def initialize(ssh, summarizer)
+          @ssh = ssh
+          @summarizer = summarizer
+        end
+
+        def upload!(local, remote, options)
+          ssh.scp.upload!(local, remote, options, &summarizer)
+        end
+
+        def download!(remote, local, options)
+          ssh.scp.download!(remote, local, options, &summarizer)
+        end
+
+        private
+
+        attr_reader :ssh, :summarizer
+      end
+    end
+  end
+end

data/lib/sshkit/backends/netssh/sftp_transfer.rb

@@ -0,0 +1,46 @@
+require "net/sftp"
+
+module SSHKit
+  module Backend
+    class Netssh < Abstract
+      class SftpTransfer
+        def initialize(ssh, summarizer)
+          @ssh = ssh
+          @summarizer = summarizer
+        end
+
+        def upload!(local, remote, options)
+          options = { progress: self }.merge(options || {})
+          ssh.sftp.connect!
+          ssh.sftp.upload!(local, remote, options)
+        ensure
+          ssh.sftp.close_channel
+        end
+
+        def download!(remote, local, options)
+          options = { progress: self }.merge(options || {})
+          destination = local ? local : StringIO.new.tap { |io| io.set_encoding('BINARY') }
+
+          ssh.sftp.connect!
+          ssh.sftp.download!(remote, destination, options)
+          local ? true : destination.string
+        ensure
+          ssh.sftp.close_channel
+        end
+
+        def on_get(download, entry, offset, data)
+          entry.size ||= download.sftp.file.open(entry.remote) { |file| file.stat.size }
+          summarizer.call(nil, entry.remote, offset + data.bytesize, entry.size)
+        end
+
+        def on_put(_upload, file, offset, data)
+          summarizer.call(nil, file.local, offset + data.bytesize, file.size)
+        end
+
+        private
+
+        attr_reader :ssh, :summarizer
+      end
+    end
+  end
+end

data/lib/sshkit/backends/netssh.rb

@@ -1,8 +1,6 @@
 require 'English'
 require 'strscan'
-require 'mutex_m'
 require 'net/ssh'
-require 'net/scp'
 
 module Net
   module SSH
@@ -23,10 +21,27 @@ module SSHKit
   module Backend
 
     class Netssh < Abstract
+      def self.assert_valid_transfer_method!(method)
+        return if [:scp, :sftp].include?(method)
+
+        raise ArgumentError, "#{method.inspect} is not a valid transfer method. Supported methods are :scp, :sftp."
+      end
+
       class Configuration
         attr_accessor :connection_timeout, :pty
+        attr_reader :transfer_method
         attr_writer :ssh_options
 
+        def initialize
+          self.transfer_method = :scp
+        end
+
+        def transfer_method=(method)
+          Netssh.assert_valid_transfer_method!(method)
+
+          @transfer_method = method
+        end
+
         def ssh_options
           default_options.merge(@ssh_options ||= {})
         end
@@ -64,16 +79,16 @@ module SSHKit
       def upload!(local, remote, options = {})
         summarizer = transfer_summarizer('Uploading', options)
         remote = File.join(pwd_path, remote) unless remote.to_s.start_with?("/") || pwd_path.nil?
-        with_ssh do |ssh|
-          ssh.scp.upload!(local, remote, options, &summarizer)
+        with_transfer(summarizer) do |transfer|
+          transfer.upload!(local, remote, options)
         end
       end
 
       def download!(remote, local=nil, options = {})
         summarizer = transfer_summarizer('Downloading', options)
         remote = File.join(pwd_path, remote) unless remote.to_s.start_with?("/") || pwd_path.nil?
-        with_ssh do |ssh|
-          ssh.scp.download!(remote, local, options, &summarizer)
+        with_transfer(summarizer) do |transfer|
+          transfer.download!(remote, local, options)
         end
       end
 
@@ -105,11 +120,12 @@ module SSHKit
         last_percentage = nil
         proc do |_ch, name, transferred, total|
           percentage = (transferred.to_f * 100 / total.to_f)
-          unless percentage.nan?
+          unless percentage.nan? || percentage.infinite?
             message = "#{action} #{name} #{percentage.round(2)}%"
             percentage_r = (percentage / log_percent).truncate * log_percent
             if percentage_r > 0 && (last_name != name || last_percentage != percentage_r)
-              info message
+              verbosity = (options[:verbosity] || :INFO).downcase # TODO: ideally reuse command.rb logic
+              public_send verbosity, message
               last_name = name
               last_percentage = percentage_r
             else
@@ -182,6 +198,20 @@ module SSHKit
         )
       end
 
+      def with_transfer(summarizer)
+        transfer_method = host.transfer_method || self.class.config.transfer_method
+        transfer_class = if transfer_method == :sftp
+                           require_relative "netssh/sftp_transfer"
+                           SftpTransfer
+                         else
+                           require_relative "netssh/scp_transfer"
+                           ScpTransfer
+                         end
+
+        with_ssh do |ssh|
+          yield(transfer_class.new(ssh, summarizer))
+        end
+      end
     end
   end