sshkit 1.8.1 → 1.9.0.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 51882b5069d3847d45a2bf42f22e40929f4bd2a8
-  data.tar.gz: cc62423ab261347458924b9e33ca5a53d0359f8f
+  metadata.gz: abfe247587ee2e726dbc24ea5de49ac889398eeb
+  data.tar.gz: 68b375157b3a48703d6f7cc7ae46888a4f41e1b8
 SHA512:
-  metadata.gz: a9108ac1a2b8b19233d68ab546fce120a9755e8a59709bb739c1e497f08ba4c7289a230fa1dfd59a975444579b6bf4fff7126db82ced5d37978fcde4089bb79d
-  data.tar.gz: 115a396201a2b3e7d83b200c8db05f1cf5e5c401cd0ab8d6a921a37972f6b30f766bd842d87aa29fe0a2cc5e1f99d33ff4ae6b21098fb280902b8ae3d857c86d
+  metadata.gz: 1ebb765665fa9b9d82277a9346e1a94a3e65cf818095976d970d8a928c692eaf2214c4fc39b5781c0b9ada96d6743cda52010aa91dc4b71a05953d890b34729a
+  data.tar.gz: d0c45809d3ad62dd5c3639c3a435c9d8b796757ff6b8cd58a802c82b15493fd1c8320ec72c551c1179e2b2fb5e7ac83acce2bbf97fb5968788a7e135331abb32

data/.travis.yml

@@ -1,6 +1,7 @@
 language: ruby
 rvm:
-  - 2.2.2
-  - 2.1.6
+  - 2.3.0
+  - 2.2.4
+  - 2.1.8
   - 2.0.0
-script: "rake test:units"
+script: "rake test:units lint"

data/CHANGELOG.md

@@ -8,6 +8,44 @@ appear at the top.
   * Add your entries below here, remember to credit yourself however you want
     to be credited!
 
+## 1.9.0.rc1
+
+### Potentially breaking changes
+
+  * The SSHKit DSL is no longer automatically included when you `require` it.
+    **This means you  must now explicitly `include SSHKit::DSL`.**
+    See [PR #219](https://github.com/capistrano/sshkit/pull/219) for details.
+    @beatrichartz
+  * `SSHKit::Backend::Printer#test` now always returns true
+    [PR #312](https://github.com/capistrano/sshkit/pull/312) @mikz
+
+### New features
+
+  * `SSHKit::Formatter::Abstract` now accepts an optional Hash of options
+    [PR #308](https://github.com/capistrano/sshkit/pull/308) @mattbrictson
+  * Add `SSHKit::Backend.current` so that Capistrano plugin authors can refactor
+    helper methods and still have easy access to the currently-executing Backend
+    without having to use global variables.
+  * Add `SSHKit.config.default_runner` options that allows to override default command runner.
+    This option also accepts a name of the custom runner class.
+  * The ConnectionPool has been rewritten in this release to be more efficient
+    and have a cleaner internal API. You can still completely disable the pool
+    by setting `SSHKit::Backend::Netssh.pool.idle_timeout = 0`.
+    @mattbrictson @byroot [PR #328](https://github.com/capistrano/sshkit/pull/328)
+
+### Bug fixes
+
+  * make sure working directory for commands is properly cleared after `within` blocks
+    [PR #307](https://github.com/capistrano/sshkit/pull/307)
+    @steved
+  * display more accurate string for commands with spaces being output in `Formatter::Pretty`
+    [PR #304](https://github.com/capistrano/sshkit/pull/304)
+    @steved
+    [PR #319](https://github.com/capistrano/sshkit/pull/319) @mattbrictson
+  * Fix a race condition experienced in JRuby that could cause multi-server
+    deploys to fail. [PR #322](https://github.com/capistrano/sshkit/pull/322)
+    @mattbrictson
+
 ## 1.8.1
 
   * Change license to MIT, thanks to all the patient contributors who gave
@@ -86,6 +124,7 @@ appear at the top.
   * Removed dependency on the `colorize` gem. SSHKit now implements its own ANSI color logic, with no external dependencies. Note that SSHKit now only supports the `:bold` or plain modes. Other modes will be gracefully ignored. [#263](https://github.com/capistrano/sshkit/issues/263)
   * New API for setting the formatter: `use_format`. This differs from `format=` in that it accepts options or arguments that will be passed to the formatter's constructor. The `format=` syntax will be deprecated in a future release. [#295](https://github.com/capistrano/sshkit/issues/295)
   * SSHKit now immediately raises a `NameError` if you try to set a formatter that does not exist. [#295](https://github.com/capistrano/sshkit/issues/295)
+  * Fix error message when the formatter does not exist. [#301](https://github.com/capistrano/sshkit/pull/301)
 
 ## 1.7.1
 

data/CONTRIBUTING.md

@@ -28,6 +28,16 @@ ruby versions.
 **The Travis build does not run the functional tests,
 so make sure all the tests pass locally before creating your PR.**
 
+## Coding guidelines
+
+This project uses [RuboCop](https://github.com/bbatsov/rubocop) to enforce standard Ruby coding
+guidelines. Currently we run RuboCop's lint rules only, which check for readability issues
+like indentation, ambiguity, and useless/unreachable code.
+
+* Test that your contributions pass with `rake lint`
+* The linter is also run as part of the full test suite with `rake`
+* Note the Travis build will fail and your PR cannot be merged if the linter finds errors
+
 ## Changelog
 
 Most changes should have an accompanying entry in the [CHANGELOG](CHANGELOG.md), unless they

data/EXAMPLES.md

@@ -366,12 +366,14 @@ known test cases, it works. The key thing is that `if` is not mapped to
 Into the `Rakefile` simply put something like:
 
 ```ruby
-require 'sshkit/dsl'
+require 'sshkit'
 
 SSHKit.config.command_map[:rake] = "./bin/rake"
 
 desc "Deploy the site, pulls from Git, migrate the db and precompile assets, then restart Passenger."
 task :deploy do
+  include SSHKit::DSL
+
   on "example.com" do |host|
     within "/opt/sites/example.com" do
       execute :git, :pull

data/README.md

@@ -3,8 +3,9 @@
 **SSHKit** is a toolkit for running commands in a structured way on one or
 more servers.
 
-[![Build Status](https://travis-ci.org/capistrano/sshkit.png?branch=master)](https://travis-ci.org/capistrano/sshkit)
-[![Dependency Status](https://gemnasium.com/leehambley/sshkit.png)](https://gemnasium.com/leehambley/sshkit)
+[![Gem Version](https://badge.fury.io/rb/sshkit.svg)](https://rubygems.org/gems/sshkit)
+[![Build Status](https://travis-ci.org/capistrano/sshkit.svg?branch=master)](https://travis-ci.org/capistrano/sshkit)
+[![Dependency Status](https://gemnasium.com/capistrano/sshkit.svg)](https://gemnasium.com/capistrano/sshkit)
 
 ## How might it work?
 
@@ -12,7 +13,7 @@ The typical use-case looks something like this:
 
 ```ruby
 require 'sshkit'
-require 'sshkit/dsl'
+include SSHKit::DSL
 
 on %w{1.example.com 2.example.com}, in: :sequence, wait: 5 do |host|
   within "/opt/sites/example.com" do
@@ -65,7 +66,7 @@ you can pass the `strip: false` option: `capture(:ls, '-l', strip: false)`
 #### Transferring files
 
 All backends also support the `upload!` and `download!` methods for transferring files.
-For the remote backed, the file is tranferred with scp.
+For the remote backend, the file is tranferred with scp.
 
 ```ruby
 on '1.example.com' do
@@ -442,24 +443,22 @@ ENV['SSHKIT_COLOR'] = 'TRUE'
 
 Want custom output formatting? Here's what you have to do:
 
-1. Write a new formatter class in the `SSHKit::Formatter` module. As an example, check out the default [pretty](https://github.com/capistrano/sshkit/blob/master/lib/sshkit/formatters/pretty.rb) formatter.
+1. Write a new formatter class in the `SSHKit::Formatter` module. Your class should subclass `SSHKit::Formatter::Abstract` to inherit conveniences and common behavior. For a basic an example, check out the [Pretty](https://github.com/capistrano/sshkit/blob/master/lib/sshkit/formatters/pretty.rb) formatter.
 1. Set the output format as described above. E.g. if your new formatter is called `FooBar`:
 
 ```ruby
 SSHKit.config.use_format :foobar
 ```
 
-If your formatter class takes a second `options` argument in its constructor, you can pass options to it like this:
+All formatters that extend from `SSHKit::Formatter::Abstract` accept an options Hash as a constructor argument. You can pass options to your formatter like this:
 
 ```ruby
 SSHKit.config.use_format :foobar, :my_option => "value"
 ```
 
-Which will call your constructor:
+You can then access these options using the `options` accessor within your formatter code.
 
-```ruby
-SSHKit::Formatter::FooBar.new($stdout, :my_option => "value")
-```
+For a much more full-featured formatter example that makes use of options, check out the [Airbrussh repository](https://github.com/mattbrictson/airbrussh/).
 
 ## Output Verbosity
 

data/Rakefile

@@ -2,8 +2,9 @@
 
 require 'bundler/gem_tasks'
 require 'rake/testtask'
+require 'rubocop/rake_task'
 
-task :default => :test
+task :default => [:test, :lint]
 
 desc "Run all tests"
 task :test => ['test:units', 'test:functional']
@@ -26,6 +27,11 @@ Rake::Task["test:functional"].enhance do
   warn "Remember there are still some VMs running, kill them with `vagrant halt` if you are finished using them."
 end
 
+desc 'Run RuboCop lint checks'
+RuboCop::RakeTask.new(:lint) do |task|
+  task.options = ['--lint']
+end
+
 task "release:rubygem_push" do
   # Delay loading Chandler until this point, since it requires Ruby >= 2.1,
   # which may not be available in all environments (e.g. Travis).

data/lib/sshkit/all.rb

@@ -10,6 +10,7 @@ require_relative 'configuration'
 require_relative 'coordinator'
 
 require_relative 'deprecation_logger'
+require_relative 'dsl'
 
 require_relative 'exception'
 
@@ -35,4 +36,4 @@ require_relative 'backends/connection_pool'
 require_relative 'backends/printer'
 require_relative 'backends/netssh'
 require_relative 'backends/local'
-require_relative 'backends/skipper'
+require_relative 'backends/skipper'

data/lib/sshkit/backends/abstract.rb

@@ -4,6 +4,19 @@ module SSHKit
 
     MethodUnavailableError = Class.new(SSHKit::StandardError)
 
+    # The Backend instance that is running in the current thread. If no Backend
+    # is running, returns `nil` instead.
+    #
+    # Example:
+    #
+    #   on(:local) do
+    #     self == SSHKit::Backend.current # => true
+    #   end
+    #
+    def self.current
+      Thread.current["sshkit_backend"]
+    end
+
     class Abstract
 
       extend Forwardable
@@ -12,7 +25,10 @@ module SSHKit
       attr_reader :host
 
       def run
+        Thread.current["sshkit_backend"] = self
         instance_exec(@host, &@block)
+      ensure
+        Thread.current["sshkit_backend"] = nil
       end
 
       def initialize(host, &block)
@@ -121,8 +137,16 @@ module SSHKit
         command(args, options).tap { |cmd| execute_command(cmd) }
       end
 
+      def pwd_path
+        if @pwd.nil? || @pwd.empty?
+          nil
+        else
+          File.join(@pwd)
+        end
+      end
+
       def command(args, options)
-        SSHKit::Command.new(*[*args, options.merge({in: @pwd.nil? ? nil : File.join(@pwd), env: @env, host: @host, user: @user, group: @group})])
+        SSHKit::Command.new(*[*args, options.merge({in: pwd_path, env: @env, host: @host, user: @user, group: @group})])
       end
 
     end

data/lib/sshkit/backends/connection_pool.rb

@@ -1,120 +1,159 @@
+require "monitor"
 require "thread"
 
-# Since we call to_s on new_connection_args and use that as a hash
-# We need to make sure the memory address of the object is not used as part of the key
-# Otherwise identical objects with different memory address won't get a hash hit.
-# In the case of proxy commands, this can lead to proxy processes leaking
-# And in severe cases can cause deploys to fail due to default file descriptor limits
-# An alternate solution would be to use a different means of generating hash keys
-module Net; module SSH; module Proxy
-  class Command
-    def inspect
-      @command_line_template
-    end
+# Since we call to_s on new connection arguments and use that as a cache key, we
+# need to make sure the memory address of the object is not used as part of the
+# key. Otherwise identical objects with different memory address won't reuse the
+# cache.
+#
+# In the case of proxy commands, this can lead to proxy processes leaking, and
+# in severe cases can cause deploys to fail due to default file descriptor
+# limits. An alternate solution would be to use a different means of generating
+# hash keys.
+#
+require "net/ssh/proxy/command"
+class Net::SSH::Proxy::Command
+  # Ensure a stable string value is used, rather than memory address.
+  def inspect
+    @command_line_template
   end
-end;end;end
-
-module SSHKit
-
-  module Backend
-
-    class ConnectionPool
+end
 
-      attr_accessor :idle_timeout
+# The ConnectionPool caches connections and allows them to be reused, so long as
+# the reuse happens within the `idle_timeout` period. Timed out connections are
+# closed, forcing a new connection to be used in that case.
+#
+# Additionally, a background thread is started to check for abandoned
+# connections that have timed out without any attempt at being reused. These
+# are eventually closed as well and removed from the cache.
+#
+# If `idle_timeout` set to `false`, `0`, or `nil`, no caching is performed, and
+# a new connection is created and then immediately closed each time. The default
+# timeout is 30 (seconds).
+#
+# There is a single public method: `with`. Example usage:
+#
+#   pool = SSHKit::Backend::ConnectionPool.new
+#   pool.with(Net::SSH.method(:start), "host", "username") do |connection|
+#     # do stuff with connection
+#   end
+#
+class SSHKit::Backend::ConnectionPool
+  attr_accessor :idle_timeout
+
+  def initialize(idle_timeout=30)
+    @idle_timeout = idle_timeout
+    @caches = {}
+    @caches.extend(MonitorMixin)
+    @timed_out_connections = Queue.new
+    Thread.new { run_eviction_loop }
+  end
 
-      def initialize
-        self.idle_timeout = 30
-        @mutex = Mutex.new
-        @pool = {}
-      end
+  # Creates a new connection or reuses a cached connection (if possible) and
+  # yields the connection to the given block. Connections are created by
+  # invoking the `connection_factory` proc with the given `args`. The arguments
+  # are used to construct a key used for caching.
+  def with(connection_factory, *args)
+    cache = find_cache(args)
+    conn = cache.pop || begin
+      connection_factory.call(*args)
+    end
+    yield(conn)
+  ensure
+    cache.push(conn) unless conn.nil?
+  end
 
-      def checkout(*new_connection_args, &block)
-        entry = nil
-        key = new_connection_args.to_s
-        if idle_timeout
-          prune_expired
-          entry = find_live_entry(key)
-        end
-        entry || create_new_entry(new_connection_args, key, &block)
-      end
+  # Immediately remove all cached connections, without closing them. This only
+  # exists for unit test purposes.
+  def flush_connections
+    caches.synchronize { caches.clear }
+  end
 
-      def checkin(entry)
-        if idle_timeout
-          prune_expired
-          entry.expires_at = Time.now + idle_timeout
-          @mutex.synchronize do
-            @pool[entry.key] ||= []
-            @pool[entry.key] << entry
-          end
-        end
-      end
+  # Immediately close all cached connections and empty the pool.
+  def close_connections
+    caches.synchronize do
+      caches.values.each(&:clear)
+      caches.clear
+      process_deferred_close
+    end
+  end
 
-      def close_connections
-        @mutex.synchronize do
-          @pool.values.flatten.map(&:connection).uniq.each do |conn|
-            if conn.respond_to?(:closed?) && conn.respond_to?(:close)
-              conn.close unless conn.closed?
-            end
-          end
-          @pool.clear
-        end
-      end
+  private
 
-      def flush_connections
-        @mutex.synchronize { @pool.clear }
-      end
+  attr_reader :caches, :timed_out_connections
 
-      private
-
-      def prune_expired
-        @mutex.synchronize do
-          @pool.each_value do |entries|
-            entries.collect! do |entry|
-              if entry.expired?
-                entry.close unless entry.closed?
-                nil
-              else
-                entry
-              end
-            end.compact!
-          end
-        end
-      end
+  def cache_enabled?
+    idle_timeout && idle_timeout > 0
+  end
 
-      def find_live_entry(key)
-        @mutex.synchronize do
-          return nil unless @pool.key?(key)
-          while (entry = @pool[key].shift)
-            return entry if entry.live?
-          end
-        end
-        nil
-      end
+  # Look up a Cache that matches the given connection arguments.
+  def find_cache(args)
+    if cache_enabled?
+      key = args.to_s
+      caches[key] || thread_safe_find_or_create_cache(key)
+    else
+      NilCache.new(method(:silently_close_connection))
+    end
+  end
 
-      def create_new_entry(args, key, &block)
-        Entry.new block.call(*args), key
+  # Cache creation needs to happen in a mutex, because otherwise a race
+  # condition might cause two identical caches to be created for the same key.
+  def thread_safe_find_or_create_cache(key)
+    caches.synchronize do
+      caches[key] ||= begin
+        Cache.new(idle_timeout, method(:silently_close_connection_later))
       end
+    end
+  end
 
-      Entry = Struct.new(:connection, :key) do
-        attr_accessor :expires_at
-
-        def live?
-          !expired? && !closed?
-        end
+  # Loops indefinitely to close connections and to find abandoned connections
+  # that need to be closed.
+  def run_eviction_loop
+    loop do
+      process_deferred_close
 
-        def expired?
-          expires_at && Time.now > expires_at
-        end
+      # Periodically sweep all Caches to evict stale connections
+      sleep([idle_timeout, 5].min)
+      caches.values.each(&:evict)
+    end
+  end
 
-        def close
-          connection.respond_to?(:close) && connection.close
-        end
+  # Immediately close any connections that are pending closure.
+  # rubocop:disable Lint/HandleExceptions
+  def process_deferred_close
+    until timed_out_connections.empty?
+      connection = timed_out_connections.pop(true)
+      silently_close_connection(connection)
+    end
+  rescue ThreadError
+    # Queue#pop(true) raises ThreadError if the queue is empty.
+    # This could only happen if `close_connections` is called at the same time
+    # the background eviction thread has woken up to close connections. In any
+    # case, it is not something we need to care about, since an empty queue is
+    # perfectly OK.
+  end
+  # rubocop:enable Lint/HandleExceptions
 
-        def closed?
-          connection.respond_to?(:closed?) && connection.closed?
-        end
-      end
+  # Adds the connection to a queue that is processed asynchronously by a
+  # background thread. The connection will eventually be closed.
+  def silently_close_connection_later(connection)
+    timed_out_connections << connection
+  end
 
-    end
+  # Close the given `connection` immediately, assuming it responds to a `close`
+  # method. If it doesn't, or if `nil` is provided, it is silently ignored. Any
+  # `StandardError` is also silently ignored. Returns `true` if the connection
+  # was closed; `false` if it was already closed or could not be closed due to
+  # an error.
+  def silently_close_connection(connection)
+    return false unless connection.respond_to?(:close)
+    return false if connection.respond_to?(:closed?) && connection.closed?
+    connection.close
+    true
+  rescue StandardError
+    false
   end
 end
+
+require "sshkit/backends/connection_pool/cache"
+require "sshkit/backends/connection_pool/nil_cache"