pd-blender 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 8b9983c59a3ba2c697b2d11c20cdb4d8a8a236f8
-  data.tar.gz: 979315d46ed457760f89ff5692b475b69e94cfc5
+  metadata.gz: 4755355532133b48f4e4525b8a7671206318b470
+  data.tar.gz: 416b8fd419f08c1900b9f08cca300b7703d4ed68
 SHA512:
-  metadata.gz: 5019047e53c66b01505a3a77c037f79fb395b342b5e3962dd33edb10e97803a20f0da5bc75211e65b64f9ac29211db6317af3a386ebc665f18a079e7f964375d
-  data.tar.gz: 89107f6c920895540a00093cde7eb03df1e1a3f160935ebf75e1afb8d6a4eb4cb156427bd2cb78802ad3d5c74755e39413d76f3e5ef6ec6446ad9c0b7886c09c
+  metadata.gz: 7822a210909825f27e67a8887083ac2a3521ef3d465efcdcd3e45dd7b61da3551df7ad59fe480dcbe9875ee6bc77261d3ced3a157b744055d02c4e6c336b08e3
+  data.tar.gz: 07c515f06d1ae9d67360e264c95e2be947cac421ca9319fefdbb1c08f26258a468466ff7976fbb9fff7d3bdc5467e28447fdaa5834858116c7edb85457a4d8b6

data/.travis.yml

@@ -1,9 +1,13 @@
+language: ruby
+cache:
+  directories:
+    - .bundle
 before_install:
   - bundle install --path .bundle
 rvm:
   - 1.9.3
-  - 2.1.0
-  - 2.1.2
+  - 2.1
+  - 2.2
 branches:
   only:
     - master

data/README.md

@@ -2,7 +2,7 @@
 # Blender
 
 Blender is a modular remote command execution framework. Blender provides few basic
-primitives to automate cross server workflows. Workflows can be expressed in plain
+primitives to automate across server workflows. Workflows can be expressed in plain
 ruby DSL and executed using the CLI.
 
 Following is an example of a simple blender script that will update the package
@@ -63,6 +63,17 @@ commands over ssh, serf handlers etc) which can ease automating large cluster
 maintenance, multi stage provisioning, establishing cross server feedback
 loops etc.
 
+## Installation
+
+Blender is published as `pd-blender` in rubygems. And you can install it as:
+```sh
+gem install pd-blender
+```
+Or declare it as a dependency in your Gemfile, if you are using bundler.
+```ruby
+gem 'pd-blender'
+```
+
 ## Concepts
 
 Blender is composed of two components:
@@ -71,7 +82,7 @@ Blender is composed of two components:
   script can have multiple tasks. Tasks are executed using drivers. Tasks can declare their
   target hosts.
 
-  * **Scheduling stratgy** - Determines the order of task execution across the hosts. 
+  * **Scheduling strategy** - Determines the order of task execution across the hosts. 
   Every blender scripts has one and only one scheduling strategy. Scheduling strategies
   uses the task list as input and produces a list of jobs, to be executed using drivers.
 
@@ -115,6 +126,18 @@ can execute `ssh_task`s. Blender core ships with following tasks and drivers:
   end
   ```
 
+  - **scp_task**: download or upload files using scp
+
+  ```ruby
+  members ['host1', 'host2', 'host3']
+  scp_upload '/foo/bar' do
+    from '/path/to/remote/file'
+  end
+  scp_download '/foo/bar' do
+    to '/local/path'
+  end
+  ```
+
 As mentioned earlier tasks are executed using drivers. Tasks can declare their preferred driver or
 Blender will assign a driver to them automatically. Blender will reuse the global driver if its
 compatible, else it will create one. By default the ```global_driver``` is a ```shell_out``` driver.
@@ -125,9 +148,9 @@ specific to their own implementations.
 
 Scheduling strategies are the most crucial part of a blender script. They decide the
 order of command execution across distributed nodes in blender. Each blender script is
- invoked using one strategy. Consider them as a transformation, where the input is tasks and ouput is
+invoked using one strategy. Consider them as a transformation, where the input is tasks and ouput is
 jobs. Tasks and job are pretty similar in their structures (both holds command and hosts),
-except a jobs can hold multiple tasks within them. We'll come to this later, but first, lets
+except a jobs can hold multiple tasks within them. We'll come to this later, but first, let's
 see how the default strategy work.
 
   - **default strategy**: the default strategy takes the list of declared tasks (and associated members
@@ -275,7 +298,7 @@ Following are some examples:
   - **chef**: discover hosts using Chef search
 
   ```ruby
-  require 'blender/dscoveries/chef'
+  require 'blender/discoveries/chef'
 
   ruby_task 'print host name' do
     execute do |host|
@@ -285,7 +308,7 @@ Following are some examples:
   end
   ```
 
-## Invoking blender periodially with Rufus schedler
+## Invoking blender periodially with Rufus scheduler
 
 Blender is designed to be used as a standalone script that can be invoked on-demand or
 consumed as a library, i.e. workflows are written in plain Ruby objects and invoked
@@ -320,7 +343,18 @@ both per task level as well as globally.
 ## Event handlers
 
 Blender provides an event dispatchment facility (inspired from Chef), where arbitrary logic can
-be hooked into the event system (e.g. HipChat notification handlers, statsd handlers, etc) and blender will automatically invoke them during key events. As of now, events are available before and after run and per job execution. Event dispatch system is likely to get more elaborate and blender might have few common event handlers (metric, notifications etc) in near future.
+be hooked into the event system (e.g. HipChat notification handlers, statsd handlers, etc) and blender
+will automatically invoke them during key events. As of now, events are available before and after run
+and per job execution. Event dispatch system is likely to get more elaborate and blender might have
+few common event handlers (metric, notifications etc) in near future.
+
+
+## Ancillary projects
+
+Blender has a few ancillary projects for integration with other systems, following are few of them:
+- Zookeeper based locking for distributed blender deployments [blender-zk](https://github.com/PagerDuty/blender-zk)
+- Serf based host discovery and command dispatch [blender-serf](https://github.com/PagerDuty/blender-serf)
+- Chef based host discovery [blender-chef](https://github.com/PagerDuty/blender-chef)
 
 ## Supported ruby versions
 

data/Rakefile

@@ -8,14 +8,16 @@ YARD::Rake::YardocTask.new do |t|
 end
 
 RSpec::Core::RakeTask.new(:spec) do |t|
-  t.pattern = %w{spec/**/*_spec.rb}
+  t.pattern = %w(spec/**/*_spec.rb)
 end
 
 RSpec::Core::RakeTask.new(:rspec) do |t|
-  t.pattern = %w{spec/**/*_rspec.rb}
+  t.pattern = %w(spec/**/*_rspec.rb)
 end
 
 RuboCop::RakeTask.new(:rubocop) do |t|
-  t.patterns = %w{Rakefile Gemfile lib/**/*.rb}
+  t.patterns = %w(Rakefile Gemfile lib/**/*.rb)
   t.fail_on_error = true
 end
+
+task default: [:spec, :rubocop]

data/blender.gemspec

@@ -24,6 +24,7 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'mixlib-shellout'
   spec.add_dependency 'mixlib-log'
   spec.add_dependency 'net-ssh'
+  spec.add_dependency 'net-scp'
   spec.add_dependency 'rufus-scheduler'
 
   spec.add_development_dependency 'bundler'

data/lib/blender.rb

@@ -29,13 +29,13 @@ module Blender
   # command and executed with local shellout driver
   #
   # @param name [String] Name of the run
+  # @param options[Hash] Additional options for scheduler
   #
   # @return [void]
-  def self.blend(name, config_file = nil)
-    scheduler = Scheduler.new(name)
-    if config_file
-      configure(config_file)
-    end
+  def self.blend(name, options = {})
+    config_file = options.delete(:config_file)
+    scheduler = Scheduler.new(name, [], options)
+    configure(config_file) if config_file
     if block_given?
       yield scheduler
     else
@@ -47,12 +47,10 @@ module Blender
 
   def self.configure(file)
     data = JSON.parse(File.read(file))
-    if data['log_level']
-      Blender::Log.level = data['log_level'].to_sym
-    end
-    if data['log_file']
-      Blender::Log.init(data['log_file'])
-    end
+
+    Blender::Log.level = data['log_level'].to_sym if data['log_level']
+    Blender::Log.init(data['log_file']) if data['log_file']
+
     if data['load_paths']
       data['load_paths'].each do |path|
         $LOAD_PATH.unshift(path)

data/lib/blender/cli.rb

@@ -21,10 +21,11 @@ require 'blender/timer'
 
 module Blender
   class CLI < Thor
-
     def self.exit_on_failure?
       true
     end
+    stop_on_unknown_option! :from_file
+    check_unknown_options! except: :from_file
 
     default_command :from_file
     package_name 'Blender'
@@ -47,11 +48,22 @@ module Blender
       aliases: '-n',
       banner: 'No-op mode, run blender without executing jobs'
 
-    def from_file
+    method_option :quiet,
+      default: false,
+      type: :boolean,
+      aliases: '-q',
+      banner: 'Quiet mode. Disable printing running job details'
+
+    def from_file(*args)
       Configuration[:noop] = options[:noop]
+      Configuration[:arguments] = args
       des = File.read(options[:file])
       $LOAD_PATH.unshift(File.expand_path(File.join(File.dirname(options[:file]), 'lib')))
-      Blender.blend(options[:file], options[:config_file]) do |sch|
+      scheduler_options = {
+        config_file: options[:config_file],
+        no_doc: options[:quiet]
+      }
+      Blender.blend(options[:file], scheduler_options) do |sch|
         sch.instance_eval(des, __FILE__, __LINE__)
       end
     end

data/lib/blender/configuration.rb

@@ -25,9 +25,8 @@ module Blender
     attr_reader :data, :mutex
 
     def initialize
-      @data = Hash.new{|h,k| h[k] = Hash.new}
-      @data[:noop] = false
       @mutex = Mutex.new
+      reset!
     end
 
     def self.[]=(key, value)
@@ -41,5 +40,17 @@ module Blender
         instance.data[key]
       end
     end
+
+    def self.reset!
+      instance.mutex.synchronize do
+        instance.reset!
+      end
+    end
+
+    def reset!
+      @data = Hash.new{|h, k| h[k] = Hash.new}
+      @data[:noop] = false
+      @data[:arguments] = []
+    end
   end
 end

data/lib/blender/drivers/ruby.rb

@@ -20,7 +20,6 @@ require 'blender/drivers/base'
 module Blender
   module Driver
     class Ruby < Base
-
       def execute(tasks, hosts)
         tasks.each do |task|
           hosts.each do |host|

data/lib/blender/drivers/scp.rb

@@ -0,0 +1,85 @@
+#
+# Author:: Ranjib Dey (<ranjib@pagerduty.com>)
+# Copyright:: Copyright (c) 2015 PagerDuty, Inc.
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'net/scp'
+require 'net/ssh'
+require 'blender/exceptions'
+require 'blender/drivers/base'
+
+module Blender
+  module Driver
+    class Scp < Base
+      attr_reader :user
+
+      def initialize(config = {})
+        cfg = config.dup
+        @user = cfg.delete(:user) || ENV['USER']
+        super(cfg)
+      end
+
+      def execute(tasks, hosts)
+        Log.debug("SCP execution tasks [#{Array(tasks).size}]")
+        Log.debug("SCP on hosts [#{hosts.join(",")}]")
+        Array(hosts).each do |host|
+          session = create_session(host)
+          Array(tasks).each do |task|
+            cmd = run_command(task.command, session)
+            if cmd.exitstatus != 0 and !task.metadata[:ignore_failure]
+              raise ExecutionFailed, cmd.stderr
+            end
+          end
+          session.loop
+        end
+      end
+
+      def run_command(command, session)
+        begin
+          case command.direction
+          when :upload
+            session.scp.upload!(command.source, command.target)
+            ExecOutput.new(0, '', '')
+          when :download
+            session.scp.download!(command.source, command.target)
+            ExecOutput.new(0, '', '')
+          else
+            ExecOutput.new(-1, '' , "Invalid direction. Can be either :upload or :download. Found:'#{command.direction}'")
+          end
+        rescue StandardError => e
+          ExecOutput.new(-1, stdout, e.message + e.backtrace.join("\n"))
+        end
+      end
+
+      private
+
+      def create_session(host)
+        Log.debug("Invoking ssh: #{user}@#{host}")
+        Net::SSH.start(host, user, config)
+      end
+    end
+
+    class ScpUpload < Blender::Driver::Scp
+    end
+    class ScpDownload < Blender::Driver::Scp
+      def run_command(command, session)
+        begin
+        rescue StandardError => e
+          ExecOutput.new(-1, stdout, e.message + e.backtrace.join("\n"))
+        end
+      end
+    end
+  end
+end

data/lib/blender/drivers/shellout.rb

@@ -20,14 +20,11 @@ require 'mixlib/shellout'
 module Blender
   module Driver
     class ShellOut < Base
-
       def initialize(config = {})
         @options = {}
         cfg = config.dup
         [:user, :group, :cwd, :umask, :returns, :environment, :timeout].each do |key|
-          if cfg.key?(key)
-            @options[key] = cfg.delete(key)
-          end
+          @options[key] = cfg.delete(key) if cfg.key?(key)
         end
         super(cfg)
       end

data/lib/blender/drivers/ssh.rb

@@ -18,12 +18,13 @@
 require 'net/ssh'
 require 'blender/exceptions'
 require 'blender/drivers/base'
+require 'blender/drivers/ssh_exec'
 
 module Blender
   module Driver
     class Ssh < Base
-
       attr_reader :user
+      include SSHExec
 
       def initialize(config = {})
         cfg = config.dup
@@ -35,7 +36,7 @@ module Blender
         Log.debug("SSH execution tasks [#{Array(tasks).size}]")
         Log.debug("SSH on hosts [#{hosts.join(",")}]")
         Array(hosts).each do |host|
-          session = ssh_session(host)
+          session = create_session(host)
           Array(tasks).each do |task|
             cmd = run_command(task.command, session)
             if cmd.exitstatus != 0 and !task.metadata[:ignore_failure]
@@ -47,47 +48,16 @@ module Blender
       end
 
       def run_command(command, session)
-        password = config[:password]
-        command = fixup_sudo(command)
-        exit_status = 0
-        channel = session.open_channel do |ch|
-          ch.request_pty
-          ch.exec(command) do |ch, success|
-            unless success
-              Log.debug("Command not found:#{success.inspect}")
-              exit_status = -1
-            end
-            ch.on_data do |c, data|
-              stdout << data
-              if data =~ /^blender sudo password: /
-                c.send_data("#{password}\n")
-              end
-            end
-            ch.on_extended_data do |c, type, data|
-              stderr << data
-            end
-            ch.on_request "exit-status" do |ichannel, data|
-              l = data.read_long
-              exit_status = [exit_status, l].max
-              Log.debug("exit_status:#{exit_status} , data:#{l}")
-            end
-          end
-          Log.debug("Exit(#{exit_status}) Command: '#{command}'")
-        end
-        channel.wait
+        exit_status = remote_exec(command, session)
         ExecOutput.new(exit_status, stdout, stderr)
       end
 
       private
 
-      def ssh_session(host)
+      def create_session(host)
         Log.debug("Invoking ssh: #{user}@#{host}")
         Net::SSH.start(host, user, config)
       end
-
-      def fixup_sudo(command)
-        command.sub(/^sudo/, 'sudo -p \'blender sudo password: \'')
-      end
     end
   end
 end

data/lib/blender/drivers/ssh_exec.rb

@@ -0,0 +1,54 @@
+#
+# Author:: Ranjib Dey (<ranjib@pagerduty.com>)
+# Author:: Smit Shah (<who828@gmail.com>)
+# Copyright:: Copyright (c) 2014 PagerDuty, Inc.
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'blender/log'
+module Blender::Driver::SSHExec
+  def remote_exec(command, session)
+    password = config[:password]
+    command = fixup_sudo(command)
+    exit_status = 0
+    channel = session.open_channel do |ch|
+      ch.request_pty
+      ch.exec(command) do |ch, success|
+        unless success
+          Blender::Log.debug("Command not found:#{success.inspect}")
+          exit_status = -1
+        end
+        ch.on_data do |c, data|
+          stdout << data
+          c.send_data("#{password}\n") if data =~ /^blender sudo password: /
+        end
+        ch.on_extended_data do |c, type, data|
+          stderr << data
+        end
+        ch.on_request "exit-status" do |ichannel, data|
+          l = data.read_long
+          exit_status = [exit_status, l].max
+          Blender::Log.debug("exit_status:#{exit_status} , data:#{l}")
+        end
+      end
+      Blender::Log.debug("Exit(#{exit_status}) Command: '#{command}'")
+    end
+    channel.wait
+    exit_status
+  end
+
+  def fixup_sudo(command)
+    command.sub(/^sudo/, 'sudo -p \'blender sudo password: \'')
+  end
+end

data/lib/blender/drivers/ssh_multi.rb

@@ -18,6 +18,7 @@
 require 'net/ssh'
 require 'blender/exceptions'
 require 'blender/drivers/ssh'
+require 'blender/drivers/ssh_exec'
 
 module Blender
   module Driver
@@ -26,7 +27,7 @@ module Blender
       def execute(tasks, hosts)
         Log.debug("SSH execution tasks [#{tasks.size}]")
         Log.debug("SSH on hosts [#{hosts.join("\n")}]")
-        session = ssh_multi_session(hosts)
+        session = create_session(hosts)
         Array(tasks).each do |task|
           cmd = run_command(task.command, session)
           if cmd.exitstatus != 0 and !task.metadata[:ignore_failure]
@@ -36,45 +37,13 @@ module Blender
         session.loop
       end
 
-      def run_command(command, session)
-        password = @config[:password]
-        command = fixup_sudo(command)
-        exit_status = 0
-        channel = session.open_channel do |ch|
-          ch.request_pty
-          ch.exec(command) do |ch, success|
-            unless success
-              Log.debug("Command not found:#{success.inspect}")
-              exit_status = -1
-            end
-            ch.on_data do |c, data|
-              stdout << data
-              if data =~ /^blender sudo password: /
-                c.send_data("#{password}\n")
-              end
-            end
-            ch.on_extended_data do |c, type, data|
-              stderr << data
-            end
-            ch.on_request "exit-status" do |ichannel, data|
-              l = data.read_long
-              exit_status = [exit_status, l].max
-              Log.debug("exit_status:#{exit_status} , data:#{l}")
-            end
-          end
-          Log.debug("Exit(#{exit_status}) Command: '#{command}'")
-        end
-        channel.wait
-        ExecOutput.new(exit_status, stdout, stderr)
-      end
-
       def concurrency
         @config[:concurrency]
       end
 
       private
 
-      def ssh_multi_session(hosts)
+      def create_session(hosts)
         user = @config[:user] || ENV['USER']
         ssh_config = { password: @config[:password]}
         error_handler = lambda do |server|

data/lib/blender/event_dispatcher.rb

@@ -40,6 +40,6 @@ module Blender
 
    (Handlers::Base.instance_methods - Object.instance_methods).each do |method_name|
       def_forwarding_method(method_name)
-    end                                           
+    end
   end
 end

data/lib/blender/handlers/base.rb

@@ -20,18 +20,25 @@ module Blender
     class Base
       def run_started(scheduler)
       end
+
       def run_finished(scheduler)
       end
+
       def run_failed(scheduler, e)
       end
+
       def job_computation_started(strategy)
       end
+
       def job_computation_finished(strategy, jobs)
       end
+
       def job_started(job)
       end
+
       def job_finished(job)
       end
+
       def job_failed(job, error)
       end
     end

data/lib/blender/handlers/doc.rb

@@ -21,7 +21,6 @@ require 'blender/configuration'
 module Blender
   module Handlers
     class Doc < Base
-
       attr_reader :ui
 
       def initialize

data/lib/blender/job.rb

@@ -25,7 +25,6 @@ module Blender
   # and passed to underlying drivers for execution
   # Tasks within a single job must has exactly same driver.
   class Job
-
     attr_reader :tasks, :hosts, :driver, :id
 
     # creates a new job

data/lib/blender/lock/flock.rb

@@ -21,12 +21,10 @@ require 'fcntl'
 module Blender
   module Lock
     class Flock
-
       def initialize(name, options)
         @path = options['path'] || File.join('/tmp', name)
         @timeout = options[:timeout] || 0
         @job_name = name
-        @lock_fd = nil
       end
 
       def acquire
@@ -38,15 +36,15 @@ module Blender
           end
         else
           locked = @lock_fd.flock(File::LOCK_NB | File::LOCK_EX)
-          raise LockAcquisitionError, 'Failed to lock file' if locked == false
+          raise LockAcquisitionError, "Failed to lock file '#{@path}'" if locked == false
         end
         @lock_fd.write({job: @job_name, pid: Process.pid }.inspect)
       end
 
       def release
-        @lock_fd
         @lock_fd.flock(File::LOCK_UN)
         @lock_fd.close
+        File.delete(@path) if File.exists?(@path)
       end
 
       def with_lock
@@ -54,10 +52,8 @@ module Blender
         yield if block_given?
       rescue Timeout::Error => e
         raise LockAcquisitionError, 'Timeout while waiting for lock acquisition'
-      rescue LockAcquisitionError => e
-        raise e
       ensure
-        release
+        release if @lock_fd
       end
     end
   end

data/lib/blender/rspec/stub_registry.rb

@@ -25,6 +25,7 @@ module Blender
         @type = type
         @opts = opts
       end
+
       def and_return(value)
         @return_value = value
       end
@@ -35,6 +36,7 @@ module Blender
       def initialize
         @data = []
       end
+
       def self.add(type, opts)
         obj = SearchStub.new(type, opts)
         instance.data << obj

data/lib/blender/scheduler.rb

@@ -30,7 +30,6 @@ require 'blender/tasks/base'
 
 module Blender
   class Scheduler
-
     include SchedulerDSL
     include Lock
 
@@ -39,14 +38,16 @@ module Blender
     attr_reader :events, :tasks
     attr_reader :lock_properties
 
-    def initialize(name, tasks = [], metadata = {})
+    def initialize(name, tasks = [], options = {})
       @name = name
       @tasks = tasks
-      @metadata = default_metadata.merge(metadata)
       @events = Blender::EventDispatcher.new
-      events.register(Blender::Handlers::Doc.new)
+      unless options.delete(:no_doc)
+        events.register(Blender::Handlers::Doc.new)
+      end
+      @metadata = default_metadata.merge(options)
       @scheduling_strategy = nil
-      @lock_properties = {driver: 'flock', driver_options: {}}
+      @lock_properties = {driver: nil, driver_options: {}}
     end
 
     def run

data/lib/blender/scheduler/dsl.rb

@@ -21,12 +21,15 @@ require 'blender/tasks/base'
 require 'blender/tasks/ruby'
 require 'blender/tasks/ssh'
 require 'blender/tasks/shell_out'
+require 'blender/tasks/scp'
 require 'highline'
 require 'blender/utils/refinements'
 require 'blender/drivers/ssh'
+require 'blender/drivers/ssh'
 require 'blender/drivers/ssh_multi'
 require 'blender/drivers/shellout'
 require 'blender/drivers/ruby'
+require 'blender/drivers/scp'
 require 'blender/discovery'
 require 'blender/handlers/base'
 require 'blender/lock/flock'
@@ -116,6 +119,20 @@ module Blender
       append_task(:ssh, task)
     end
 
+    def scp_upload(name, &block)
+      task = build_task(name, :scp)
+      task.instance_eval(&block) if block_given?
+      task.direction = :upload
+      append_task(:scp, task)
+    end
+
+    def scp_download(name, &block)
+      task = build_task(name, :scp)
+      task.instance_eval(&block) if block_given?
+      task.direction = :download
+      append_task(:scp, task)
+    end
+
     def strategy(strategy)
       klass_name = camelcase(strategy.to_s).to_sym
       begin

data/lib/blender/tasks/base.rb

@@ -21,7 +21,6 @@ module Blender
     class Base
       include Blender::Discovery
 
-      attr_reader :guards
       attr_reader :metadata
       attr_reader :name
       attr_reader :hosts

data/lib/blender/tasks/ruby.rb

@@ -20,7 +20,6 @@ require 'blender/tasks/base'
 module Blender
   module Task
     class Ruby < Blender::Task::Base
-
       attr_reader :code_block
 
       def execute(&block)

data/lib/blender/tasks/scp.rb

@@ -0,0 +1,41 @@
+#
+# Author:: Ranjib Dey (<ranjib@pagerduty.com>)
+# Copyright:: Copyright (c) 2014 PagerDuty, Inc.
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'blender/tasks/ssh'
+require 'forwardable'
+
+module Blender
+  module Task
+    class Scp < Blender::Task::Base
+      extend Forwardable
+      def_delegators :@command, :direction, :direction=
+      def initialize(name, metadata = {})
+        super
+        @command = Struct.new(:direction, :source, :target).new
+        @command.target = name
+        @command.source = name
+        @direction = :upload
+      end
+      def from(source)
+        @command.source = source
+      end
+      def to(target)
+        @command.target = target
+      end
+    end
+  end
+end

data/lib/blender/utils/refinements.rb

@@ -19,7 +19,7 @@ module Blender
     module Refinements
       def camelcase(string)
         str = string.dup
-        str.gsub!(/[^A-Za-z0-9_]/,'_')
+        str.gsub!(/[^A-Za-z0-9_]/, '_')
         rname = nil
         regexp = %r{^(.+?)(_(.+))?$}
         mn = str.match(regexp)

data/lib/blender/utils/thread_pool.rb

@@ -20,7 +20,6 @@ require 'blender/log'
 module Blender
   module Utils
     class ThreadPool
-
       def initialize(size)
         @size = size
         @queue = Queue.new

data/lib/blender/utils/ui.rb

@@ -24,6 +24,7 @@ module Blender
         @mutex = Mutex.new
         @highline = HighLine.new
       end
+
       def puts(string)
         @mutex.synchronize do
           $stdout.puts(string)
@@ -45,7 +46,6 @@ module Blender
       def color(string, *colors)
         @highline.color(string, *colors)
       end
-      
     end
   end
 end

data/lib/blender/version.rb

@@ -16,5 +16,5 @@
 # limitations under the License.
 
 module Blender
-  VERSION = '0.0.1'
+  VERSION = '0.1.0'
 end

data/spec/blender/config_spec.rb

@@ -0,0 +1,34 @@
+#
+# Author:: Ranjib Dey (<ranjib@pagerduty.com>)
+# Copyright:: Copyright (c) 2014 PagerDuty, Inc.
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'spec_helper'
+
+describe Blender::Configuration do
+  it 'should populate argments and noop key' do
+    expect(Blender::Configuration[:noop]).to be(false)
+    expect(Blender::Configuration[:argments]).to be_empty
+  end
+  it 'should set configs globally' do
+    Blender::Configuration[:x] = 1
+    expect(Blender::Configuration[:x]).to eq(1)
+  end
+  it 'should reset config' do
+    Blender::Configuration[:x] = 1
+    Blender::Configuration.reset!
+    expect(Blender::Configuration[:x]).to be_empty
+  end
+end

data/spec/blender/dsl_spec.rb

@@ -16,4 +16,35 @@ describe '#dsl' do
     allow_any_instance_of(Blender::Driver::Ssh).to receive(:execute)
     scheduler.run
   end
+
+  context '#scp' do
+    it '#upload' do
+      session = double('Net::SSH::Session', loop: true)
+      scp = double('Net::SSH::Scp')
+      expect(Net::SSH).to receive(:start).with('host1', 'x', password: 'y').and_return(session)
+      expect(session).to receive(:scp).and_return(scp)
+      expect(scp).to receive(:upload!).with('/local/path', '/remote/path')
+      Blender.blend('test') do |sched|
+        sched.config(:scp, user: 'x', password: 'y')
+        sched.members(['host1'])
+        sched.scp_upload '/remote/path' do
+          from '/local/path'
+        end
+      end
+    end
+    it '#download' do
+      session = double('Net::SSH::Session', loop: true)
+      scp = double('Net::SSH::Scp')
+      expect(Net::SSH).to receive(:start).with('host1', 'x', password: 'y').and_return(session)
+      expect(session).to receive(:scp).and_return(scp)
+      expect(scp).to receive(:download!).with('/remote/path', '/local/path')
+      Blender.blend('test') do |sched|
+        sched.members(['host1'])
+        sched.config(:scp, user: 'x', password: 'y')
+        sched.scp_download '/remote/path' do
+          to '/local/path'
+        end
+      end
+    end
+  end
 end

data/spec/blender/lock_spec.rb

@@ -1,72 +1,75 @@
 require 'spec_helper'
 
 describe Blender::Lock do
-
-  it 'should not allow two blender run with same name to run at the same time' do
-
-    pid1 = fork do
-      Blender.blend('test-1') do |sched|
-        sched.members(['localhost'])
-        sched.ruby_task('date') do
-          execute do
-            sleep 5
-            puts 'This will succeed'
+  context 'File based locking' do
+    it 'should not allow two blender run with same lockfile to run at the same time' do
+      pid1 = fork do
+        Blender.blend('test-1') do |sched|
+          sched.lock_options('flock')
+          sched.members(['localhost'])
+          sched.ruby_task('date') do
+            execute do
+              sleep 5
+              puts 'This will succeed'
+            end
           end
         end
       end
-    end
 
-    pid2 = fork do
-      STDERR.reopen(File::NULL)
-      Blender.blend('test-1') do |sched|
-        sched.members(['localhost'])
-        sched.ruby_task('date') do
-          execute do
-            puts 'This will fail'
+      pid2 = fork do
+        STDERR.reopen(File::NULL)
+        Blender.blend('test-1') do |sched|
+          sched.lock_options('flock')
+          sched.members(['localhost'])
+          sched.ruby_task('date') do
+            execute do
+              puts 'This will fail'
+            end
           end
         end
       end
+
+      status1 = Process.wait2 pid1
+      status2 = Process.wait2 pid2
+      expect(status1.last.exitstatus).to eq(0)
+      expect(status2.last.exitstatus).to_not eq(0)
     end
-    status1 = Process.wait2 pid1
-    status2 = Process.wait2 pid2
-    expect(status1.last.exitstatus).to eq(0)
-    expect(status2.last.exitstatus).to_not eq(0)
-  end
 
-  it 'should allow two blender run with different name to run at the same time' do
-    pid1 = fork do
-      Blender.blend('test-1') do |sched|
-        sched.members(['localhost'])
-        sched.ruby_task('date') do
-          execute do
-            sleep 5
-            puts 'This will succeed'
+    it 'should allow two blender run with different lock file to run at the same time' do
+      pid1 = fork do
+        Blender.blend('test-1') do |sched|
+          sched.lock_options('flock')
+          sched.members(['localhost'])
+          sched.ruby_task('date') do
+            execute do
+              sleep 5
+              puts 'This will succeed'
+            end
           end
         end
       end
-    end
 
-    pid2 = fork do
-      Blender.blend('test-2') do |sched|
-        sched.members(['localhost'])
-        sched.ruby_task('date') do
-          execute do
-            puts 'This will succeed'
+      pid2 = fork do
+        Blender.blend('test-2') do |sched|
+          sched.lock_options('flock')
+          sched.members(['localhost'])
+          sched.ruby_task('date') do
+            execute do
+              puts 'This will succeed'
+            end
           end
         end
       end
+      status1 = Process.wait2 pid1
+      status2 = Process.wait2 pid2
+      expect(status1.last.exitstatus).to eq(0)
+      expect(status2.last.exitstatus).to eq(0)
     end
-    status1 = Process.wait2 pid1
-    status2 = Process.wait2 pid2
-    expect(status1.last.exitstatus).to eq(0)
-    expect(status2.last.exitstatus).to eq(0)
-  end
-
-  context 'File based locking with timeout' do
 
     it 'should raise lock acquisition error when times out' do
       pid1 = fork do
         Blender.blend('test-1') do |sched|
+          sched.lock_options('flock')
           sched.members(['localhost'])
           sched.ruby_task('date') do
             execute do
@@ -98,6 +101,7 @@ describe Blender::Lock do
     it 'should not raise lock acquisition error when  able to acquire lock within timeout period' do
       pid1 = fork do
         Blender.blend('test-1') do |sched|
+          sched.lock_options('flock')
           sched.members(['localhost'])
           sched.ruby_task('date') do
             execute do
@@ -112,7 +116,7 @@ describe Blender::Lock do
         STDERR.reopen(File::NULL)
         Blender.blend('test-1') do |sched|
           sched.members(['localhost'])
-          sched.lock_options('flock', timeout: 3)
+          sched.lock_options('flock', timeout: 10)
           sched.ruby_task('date') do
             execute do
               puts 'This will fail'
@@ -123,7 +127,7 @@ describe Blender::Lock do
       status1 = Process.wait2 pid1
       status2 = Process.wait2 pid2
       expect(status1.last.exitstatus).to eq(0)
-      expect(status2.last.exitstatus).to_not eq(0)
+      expect(status2.last.exitstatus).to eq(0)
     end
   end
 end

data/spec/blender/scheduler_spec.rb

@@ -4,6 +4,14 @@ describe Blender::Scheduler do
   let(:scheduler) do
     described_class.new('test')
   end
+  describe '#no_doc' do
+    it 'should not use document handler if no_doc option is passed' do
+      expect(Blender::Handlers::Doc).to_not receive(:new)
+      task = Blender::Task::Ruby.new('test')
+      sched = described_class.new('no_doc', [] , no_doc: true)
+      sched.run
+    end
+  end
   describe '#DSL' do
     subject(:task){scheduler.tasks.first}
     it '#ask' do

data/spec/blender_spec.rb

@@ -16,6 +16,7 @@
 # limitations under the License.
 
 require 'spec_helper'
+require 'blender/cli'
 
 describe Blender do
   describe '#blend' do
@@ -33,5 +34,15 @@ describe Blender do
       end
       expect(x).to eq(100)
     end
+    context 'CLI' do
+      it 'should store additional arguments in config' do
+        Blender::CLI.start(%w{-f spec/data/example.rb -x -y -z})
+        expect(Blender::Configuration[:arguments]).to eq(%w{-x -y -z})
+      end
+      it 'should store additional arguments in config' do
+        expect(Blender::Handlers::Doc).to_not receive(:new)
+        Blender::CLI.start(%w{-q -f spec/data/example.rb})
+      end
+    end
   end
 end

data/spec/data/example.rb

@@ -4,7 +4,6 @@ ruby_task 'task2' do
   execute do |h|
     puts 'HelloWorld'
   end
-  driver_options(stdout: $stdout)
 end
 shell_task 'task3' do
   execute 'does not exist'

data/spec/spec_helper.rb

@@ -1,9 +1,3 @@
-require 'simplecov'
-SimpleCov.formatter = SimpleCov::Formatter::HTMLFormatter
-SimpleCov.start do
-  add_filter '/spec/'
-  add_filter '.bundle'
-end
 
 require 'rspec'
 require 'rspec/mocks'
@@ -27,9 +21,5 @@ RSpec.configure do |config|
   config.mock_with :rspec do |mocks|
     mocks.verify_doubled_constant_names = true
   end
-  config.before(:each) do
-    doc = double(Blender::Handlers::Doc).as_null_object
-    allow(Blender::Handlers::Doc).to receive(:new).and_return(doc)
-  end
   config.backtrace_exclusion_patterns = []
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: pd-blender
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.1.0
 platform: ruby
 authors:
 - Ranjib Dey
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-09-10 00:00:00.000000000 Z
+date: 2015-02-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: highline
@@ -80,6 +80,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: net-scp
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rufus-scheduler
   requirement: !ruby/object:Gem::Requirement
@@ -216,8 +230,10 @@ files:
 - lib/blender/drivers/base.rb
 - lib/blender/drivers/compound.rb
 - lib/blender/drivers/ruby.rb
+- lib/blender/drivers/scp.rb
 - lib/blender/drivers/shellout.rb
 - lib/blender/drivers/ssh.rb
+- lib/blender/drivers/ssh_exec.rb
 - lib/blender/drivers/ssh_multi.rb
 - lib/blender/event_dispatcher.rb
 - lib/blender/exceptions.rb
@@ -237,6 +253,7 @@ files:
 - lib/blender/scheduling_strategies/per_task.rb
 - lib/blender/tasks/base.rb
 - lib/blender/tasks/ruby.rb
+- lib/blender/tasks/scp.rb
 - lib/blender/tasks/shell_out.rb
 - lib/blender/tasks/ssh.rb
 - lib/blender/timer.rb
@@ -245,6 +262,7 @@ files:
 - lib/blender/utils/ui.rb
 - lib/blender/version.rb
 - spec/blender/blender_rspec.rb
+- spec/blender/config_spec.rb
 - spec/blender/discovery_spec.rb
 - spec/blender/drivers/ssh_multi_spec.rb
 - spec/blender/drivers/ssh_spec.rb
@@ -286,6 +304,7 @@ specification_version: 4
 summary: A modular orchestration engine
 test_files:
 - spec/blender/blender_rspec.rb
+- spec/blender/config_spec.rb
 - spec/blender/discovery_spec.rb
 - spec/blender/drivers/ssh_multi_spec.rb
 - spec/blender/drivers/ssh_spec.rb