chef-provisioning 1.3.0 → 1.4.0

data/README.md

@@ -3,13 +3,14 @@ Chef Provisioning
 [![Gitter](https://badges.gitter.im/Join Chat.svg)](https://gitter.im/chef/chef-provisioning?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 [![Stories in Ready](https://badge.waffle.io/chef/chef-provisioning.png?label=ready&title=Ready)](https://waffle.io/chef/chef-provisioning)
 [![Status](https://travis-ci.org/chef/chef-provisioning.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning)
+[![Gem Version](https://badge.fury.io/rb/chef-provisioning.svg)](http://badge.fury.io/rb/chef-provisioning)
 
 Driver build status:
 
-AWS | Azure | Docker | Fog | Hanlon | LXC | ssh | Vagrant
----- | ---- | ---- | ---- | ---- | ---- | ---- | ----
-[![Status](https://travis-ci.org/chef/chef-provisioning-aws.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-aws)| [![Status](https://travis-ci.org/chef/chef-provisioning-azure.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-azure) | [![Status](https://travis-ci.org/chef/chef-provisioning-docker.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-docker) | [![Status](https://travis-ci.org/chef/chef-provisioning-fog.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-fog) | [![Status](https://travis-ci.org/chef/chef-provisioning-hanlon.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-hanlon) | [![Status](https://travis-ci.org/chef/chef-provisioning-lxc.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-lxc) | [![Status](https://travis-ci.org/chef/chef-provisioning-ssh.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-ssh) | [![Status](https://travis-ci.org/chef/chef-provisioning-vagrant.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-vagrant)
-
+AWS | Azure | Docker | Fog | ssh | Vagrant
+---- | ---- | ---- | ---- | ---- | ----
+[![Status](https://travis-ci.org/chef/chef-provisioning-aws.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-aws)| [![Status](https://travis-ci.org/chef/chef-provisioning-azure.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-azure) | [![Status](https://travis-ci.org/chef/chef-provisioning-docker.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-docker) | [![Status](https://travis-ci.org/chef/chef-provisioning-fog.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-fog) |  [![Status](https://travis-ci.org/chef/chef-provisioning-ssh.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-ssh) | [![Status](https://travis-ci.org/chef/chef-provisioning-vagrant.svg?branch=master)](https://travis-ci.org/chef/chef-provisioning-vagrant)
+[![Gem Version](https://badge.fury.io/rb/chef-provisioning-aws.svg)](http://badge.fury.io/rb/chef-provisioning-aws) | [![Gem Version](https://badge.fury.io/rb/chef-provisioning-azure.svg)](http://badge.fury.io/rb/chef-provisioning-azure) | [![Gem Version](https://badge.fury.io/rb/chef-provisioning-docker.svg)](http://badge.fury.io/rb/chef-provisioning-docker) | [![Gem Version](https://badge.fury.io/rb/chef-provisioning-fog.svg)](http://badge.fury.io/rb/chef-provisioning-fog) | [![Gem  Version](https://badge.fury.io/rb/chef-provisioning-ssh.svg)](http://badge.fury.io/rb/chef-provisioning-ssh) | [![Gem Version](https://badge.fury.io/rb/chef-provisioning-vagrant.svg)](http://badge.fury.io/rb/chef-provisioning-vagrant)
 This library solves the problem of repeatably creating machines and infrastructures in Chef.  It has a plugin model that lets you write bootstrappers for your favorite infrastructures, including VirtualBox, EC2, LXC, bare metal, and many more!
 
 Documentation
@@ -147,13 +148,65 @@ Drivers each have their own repository.  Current drivers:
 
 **Containers:**
 - [Docker](https://github.com/chef/chef-provisioning-docker)
-- [LXC](https://github.com/chef/chef-provisioning-lxc)
 
 **Bare Metal:**
-- [Hanlon](https://github.com/chef/chef-provisioning-hanlon)
 - [OpenCrowbar](https://github.com/newgoliath/chef-provisioning-crowbar) OpenCrowbar controls your real metal.  It discovers, inventories, configs RAID & BIOS and networks, and installs your OS.  [OpenCrowbar website](http://www.opencrowbar.org) [OpenCrowbar github](https://github.com/opencrowbar/core)
 - [SSH (no PXE)](https://github.com/double-z/chef-metal-ssh) (not yet up to date with 0.11)
 
+**Seeking Maintainers:**
+These repositories are not actively maintained and are seeking maintainers.
+- [LXC](https://github.com/chef/chef-provisioning-lxc)
+- [Hanlon](https://github.com/chef/chef-provisioning-hanlon)
+
+### Machine options
+
+You can pass machine options that will be used by `machine`, `machine_batch` and `machine_image` to
+configure the machine:
+
+```ruby
+with_machine_options({
+  convergence_options: {
+    chef_version: "12.4.1",
+    prerelease: "false",
+    chef_client_timeout: 120*60, # Default: 2 hours
+    chef_config: "log_level :debug\\n", # String containing additional text to inject into client.rb
+    chef_server: "http://my.chef.server/", # TODO could conflict with https://github.com/chef/chef-provisioning#pointing-boxes-at-chef-servers
+    bootstrap_proxy: "http://localhost:1234",
+    ssl_verify_mode: :verify_peer,
+    client_rb_path: "/etc/chef/client.rb", # <- DEFAULT, overwrite if necessary
+    client_pem_path: "/etc/chef/client.pem", # <- DEFAULT, overwrite if necessary
+    allow_overwrite_keys: false, # If there is an existing client.pem this needs to be true to overwrite it
+    private_key_options: {}, # TODO ????? Something to do with creating node object
+    source_key: "", # ?????
+    source_key_pass_phrase: "", # ?????
+    source_key_path: "", # ?????
+    public_key_path: "", # ?????
+    public_key_format: "", # ?????
+    admin: "", # ?????
+    validator: "", # ?????
+    ohai_hints: { :ec2 => { :key => :value } }, # Map from hint file name to file contents, this would create /etc/chef/ohai/hints/ec2.json,
+    ignore_failure: [1, 5..10, SomeSpecificError], # If true don't let a convergence failure on provisioned machine stop the provisioning workstation converge.  Can also provide a single exit code to ignore (no array) or `true` to ignore all RuntimeErrors
+    # The following are only available for Linux machines
+    install_sh_url: "https://www.chef.io/chef/install.sh", # <- DEFAULT, overwrite if necessary
+    install_sh_path: "/tmp/chef-install.sh", # <- DEFAULT, overwrite if necessary
+    install_sh_arguments: "-P chef-dk", # Additional commands to pass to install.sh
+    # The following are only available for Windows machines
+    install_msi_url: "foo://bar.com"
+  },
+  ssh_username: "ubuntu", # Username to use for ssh and WinRM
+  ssh_options: { # a list of options to Net::SSH.start
+    :auth_methods => [ 'publickey' ], # DEFAULT
+    :keys_only => true, # DEFAULT
+    :host_key_alias => "#{instance.id}.AWS" # DEFAULT
+  }
+})
+```
+
+This options hash can be supplied to either `with_machine_options` or directly into the `machine_options`
+attribute.
+
+Individual drivers will often add their own driver specific config.  For example, AWS expects a `:bootstrap_options` hash at the same level as `:convergence_options`.
+
 ### Anatomy of a Recipe
 
 chef-zero comes with a provisioner for Vagrant, an abstraction that covers VirtualBox, VMWare and other Virtual Machine drivers. In docs/examples, you can run this to try it:
@@ -198,41 +251,6 @@ end
 
 Other directives, like `recipe 'apache'`, help you set run lists and other information about the machine.
 
-### Fog (EC2, Openstack and friends)
-
-chef-provisioning also comes with a [Fog](http://fog.io/) provisioner that handles provisioning to Openstack, Rackspace, Amazon's EC2 and other cloud drivers.  Before you begin, you will need to put your AWS credentials in ~/.aws/config in the format [mentioned in Option 1 here](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#d0e726).  There are other ways to specify your credentials, but this is the standard one for the Amazon CLI.
-
-Once your credentials are in, basic usage looks like this:
-
-```
-export CHEF_DRIVER=fog:AWS
-chef-client -z simple.rb
-```
-
-Other valid URLs include `fog:AWS:myprofilename` and `fog:AWS:profilename:us-west-2`.
-
-Most Chef Provisioning drivers try hard to provide reasonable defaults so you can get started easily.  Once you have specified your credentials, AMIs and other things are chosen for you.
-
-You will usually want to create or input a custom key pair for bootstrap. To customize, specify keys and AMI and other options, you can make recipes like this:
-
-```ruby
-require 'chef/provisioning/fog_driver'
-
-fog_key_pair 'my_bootstrap_key'
-
-with_machine_options :bootstrap_options => {
-  :key_name => 'my_bootstrap_key',
-  :image_id => 'ami-59a4a230',
-  :flavor_id => 't1.micro'
-}
-```
-
-`fog_key_pair` creates a new key pair (if the files do not already exist) and uploads it to AWS (it will toss an error if the key pair already exists and does not match). By default, `fog_key_pair` will look for matching key files in .chef/keys, ~/.chef/keys and ~/.ssh.  If it does not find one, it will place the key in `.chef/keys`.  You can override this path in fog_key_pair, but if you do, you will want to modify `private_key_paths` in your configuration to match.
-
-`with_machine_options` specifies machine_options that will be applied to any `machine` resources chef-client encounters.
-
-You will notice that we are still using `simple.rb` here.  Machine definitions are generally driver-independent.  This is an important feature that allows you to spin up your clusters in different places to create staging, test or miniature dev environments.
-
 ### Pointing Boxes at Chef Servers
 
 By default, Chef Provisioning will put your boxes on the same Chef server you started chef-client with (in the case of -z, that's a local chef-zero server). Sometimes you want to put your boxes on different servers.  There are a couple of ways to do that:

data/Rakefile

@@ -1,6 +1,40 @@
 require 'bundler'
 require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
 
-task :spec do
-  require File.expand_path('spec/run')
+desc "run specs"
+RSpec::Core::RakeTask.new(:spec) do |task|
+  task.pattern = 'spec/**/*_spec.rb'
 end
+
+desc "run core gem specs and generated driver specs"
+task :all => ["driver:spec", :spec]
+
+FAKE_DIR = "chef-provisioning-fake"
+
+namespace :driver do
+  desc "generate a '#{FAKE_DIR}' driver"
+  task :generate do
+    sh "./bin/generate_driver fake"
+  end
+
+  desc "run specs for #{FAKE_DIR}"
+  task :spec do
+    sh "cd #{FAKE_DIR} && bundle exec rspec"
+  end
+
+  desc "generate a #{FAKE_DIR} driver and run its specs"
+  task :verify => [:generate, :spec]
+
+  task :clean do
+    sh "rm -rf #{FAKE_DIR}"
+  end
+
+  desc "generate a fresh #{FAKE_DIR} driver, run its specs, and delete it"
+  task :cycle do
+    Rake::Task['driver:clean'].invoke
+    Rake::Task['driver:clean'].reenable
+    Rake::Task['driver:verify'].invoke
+    Rake::Task['driver:clean'].invoke
+  end
+end

data/lib/chef/provider/load_balancer.rb

@@ -6,6 +6,7 @@ require 'chef/provisioning/chef_provider_action_handler'
 class Chef
   class Provider
     class LoadBalancer < Chef::Provider::LWRPBase
+      provides :load_balancer
 
       def action_handler
         @action_handler ||= Chef::Provisioning::ChefProviderActionHandler.new(self)
@@ -16,7 +17,7 @@ class Chef
       end
 
       def new_driver
-        @new_driver ||= run_context.chef_metal.driver_for(new_resource.driver)
+        @new_driver ||= run_context.chef_provisioning.driver_for(new_resource.driver)
       end
 
       def chef_managed_entry_store
@@ -51,7 +52,20 @@ class Chef
       end
 
       def lb_options
-        new_resource.load_balancer_options
+        @lb_options ||= begin
+          configs = []
+          configs << new_resource.load_balancer_options if new_resource.load_balancer_options
+
+          # See documentation in machine.rb provider
+          (self.class.additional_lb_option_keys || []).each do |k|
+            configs << { k => new_resource.public_send(k)} if new_resource.public_send(k)
+          end
+          Cheffish::MergedConfig.new(*configs)
+        end
+      end
+
+      def self.additional_lb_option_keys
+        @@additional_lb_option_keys ||= []
       end
 
     end

data/lib/chef/provider/machine.rb

@@ -6,6 +6,7 @@ require 'chef/provisioning/chef_provider_action_handler'
 class Chef
 class Provider
 class Machine < Chef::Provider::LWRPBase
+  provides :machine
 
   def action_handler
     @action_handler ||= Chef::Provisioning::ChefProviderActionHandler.new(self)
@@ -146,12 +147,29 @@ class Machine < Chef::Provider::LWRPBase
         end
     }
 
+    # The current use case for this is adding a new attribute `aws_tags` to
+    # the machine resource from the chef-provisioning-aws driver.  Because we
+    # want that attribute to work for the recipe DSL, it needs to be added at the
+    # Chef::Resource::Machine class level and not at an instance level.  Thus we
+    # also need to pull the additional_machine_option_keys (:aws_tags) from the
+    # Chef::Resource::Machine class level.  If you use two drivers (like AWS and
+    # Azure) then all machine instances will still have the `aws_tags` attribute
+    # DSL and will pass `:aws_tags` in on the machine_options.  They can simply
+    # be ignored by the Azure driver.
+    (self.class.additional_machine_option_keys || []).each do |k|
+      configs << { k => new_resource.send(k)} if new_resource.send(k)
+    end
+
     configs << { from_image: new_resource.from_image } if new_resource.from_image
     configs << new_resource.machine_options if new_resource.machine_options
     configs << driver.config[:machine_options] if driver.config[:machine_options]
     Cheffish::MergedConfig.new(*configs)
   end
 
+  def self.additional_machine_option_keys
+    @@additional_machine_option_keys ||= []
+  end
+
   def load_current_resource
     node_driver = Chef::Provider::ChefNode.new(new_resource, run_context)
     node_driver.load_current_resource

data/lib/chef/provider/machine_batch.rb

@@ -8,6 +8,7 @@ require 'chef/provisioning/machine_spec'
 class Chef
 class Provider
 class MachineBatch < Chef::Provider::LWRPBase
+  provides :machine_batch
 
   def action_handler
     @action_handler ||= Provisioning::ChefProviderActionHandler.new(self)

data/lib/chef/provider/machine_file.rb

@@ -5,6 +5,7 @@ require 'chef/provisioning/machine'
 class Chef
 class Provider
 class MachineFile < Chef::Provider::LWRPBase
+  provides :machine_file
 
   def action_handler
     @action_handler ||= Chef::Provisioning::ChefProviderActionHandler.new(self)

data/lib/chef/provider/machine_image.rb

@@ -6,6 +6,7 @@ require 'chef/provisioning/chef_provider_action_handler'
 class Chef
 class Provider
 class MachineImage < Chef::Provider::LWRPBase
+  provides :machine_image
 
   def action_handler
     @action_handler ||= Chef::Provisioning::ChefProviderActionHandler.new(self)
@@ -28,6 +29,7 @@ class MachineImage < Chef::Provider::LWRPBase
     image_spec = chef_managed_entry_store.get_or_new(:machine_image, new_resource.name)
     if image_spec.reference
       # TODO check for real existence and maybe update
+      new_driver.ready_image(action_handler, image_spec, new_image_options)
     else
       #
       # Create a new image
@@ -77,11 +79,23 @@ class MachineImage < Chef::Provider::LWRPBase
   end
 
   def new_image_options
-    @new_image_options ||= (new_resource.image_options || {}).to_hash.dup
+    @new_image_options ||= begin
+      configs = []
+      configs << new_resource.image_options if new_resource.image_options
+      # See documentation in machine.rb provider
+      (self.class.additional_image_option_keys || []).each do |k|
+        configs << { k => new_resource.send(k)} if new_resource.send(k)
+      end
+      Cheffish::MergedConfig.new(*configs)
+    end
   end
 
   def new_machine_options
-    @new_machine_options ||= (new_resource.machine_options || {}).to_hash.dup
+    @new_machine_options ||= new_resource.machine_options
+  end
+
+  def self.additional_image_option_keys
+    @@additional_image_option_keys ||= []
   end
 
 end

data/lib/chef/provisioning/convergence_strategy/ignore_convergence_failure.rb

@@ -0,0 +1,54 @@
+class Chef
+module Provisioning
+  class ConvergenceStrategy
+
+    # The purpose of this class is to decore the `converge` method with logic to catch any
+    # convergence failure exceptions, log them and then squelch them.  The reason we
+    # need this is to prevent 1 provisioned node's converge failing an entire provisioning
+    # recipe.
+    module IgnoreConvergenceFailure
+
+      attr_accessor :ignore_failures_array, :ignore_exit_values
+
+      # This module is only meant to be extended into instances, not classes or modules.
+      # Different machines may have different settings so we don't want to extend
+      # every `install_sh` strategy with this logic.
+      def self.extended(instance)
+        opts = instance.convergence_options[:ignore_failure]
+        instance.ignore_failures_array = []
+        instance.ignore_exit_values = []
+        if opts == true
+          instance.ignore_failures_array << RuntimeError
+        else
+          # We assume it is integers or errors
+          opts = [opts].flatten
+          opts.each do |o|
+            case
+            when o.is_a?(Fixnum)
+              instance.ignore_exit_values << o
+            when o.is_a?(Range)
+              instance.ignore_exit_values += o.to_a
+            when o <= Exception
+              instance.ignore_failures_array << o
+            end
+          end
+        end
+      end
+
+      def converge(action_handler, machine)
+        super
+      rescue SystemExit => e
+        if ignore_exit_values.include? e.status
+          action_handler.performed_action("Caught SystemExit error #{e.status} from converging node but ignoring it")
+        else
+          raise
+        end
+      rescue *ignore_failures_array
+        action_handler.performed_action("Caught error '#{$!.inspect.gsub(/\n/,'\\n')}' from converging node but ignoring it")
+      end
+
+    end
+
+  end
+end
+end

data/lib/chef/provisioning/convergence_strategy/install_cached.rb

@@ -20,6 +20,7 @@ module Provisioning
       # - :chef_client_timeout
       # - :client_rb_path, :client_pem_path
       # - :chef_version, :prerelease, :package_cache_path
+      # - :package_metadata
       def initialize(convergence_options, config)
         convergence_options = Cheffish::MergedConfig.new(convergence_options, {
           :client_rb_path => '/etc/chef/client.rb',
@@ -34,6 +35,7 @@ module Provisioning
         @chef_client_timeout = convergence_options.has_key?(:chef_client_timeout) ? convergence_options[:chef_client_timeout] : 120*60 # Default: 2 hours
         FileUtils.mkdir_p(@package_cache_path)
         @package_cache_lock = Mutex.new
+        @package_metadata ||= convergence_options[:package_metadata]
       end
 
       attr_reader :client_rb_path
@@ -69,7 +71,7 @@ module Provisioning
         package_file = download_package_for_platform(action_handler, machine, platform, platform_version, machine_architecture)
         remote_package_file = "#{@tmp_dir}/#{File.basename(package_file)}"
         machine.upload_file(action_handler, package_file, remote_package_file)
-        install_package(action_handler, machine, remote_package_file)
+        install_package(action_handler, machine, platform, remote_package_file)
       end
 
       def converge(action_handler, machine)
@@ -101,15 +103,20 @@ module Provisioning
             #
             # Grab metadata
             #
-            metadata = download_metadata_for_platform(machine, platform, platform_version, machine_architecture)
+            metadata = @package_metadata
+            if !metadata
+              Chef::Log.info("No metadata supplied, downloading it...")
+              metadata = download_metadata_for_platform(machine, platform, platform_version, machine_architecture)
+            end
 
             # Download actual package desired by metadata
-            package_file = "#{@package_cache_path}/#{URI(metadata['url']).path.split('/')[-1]}"
+            package_file = "#{@package_cache_path}/#{URI(metadata[:url]).path.split('/')[-1]}"
 
+            Chef::Log.debug("Package metadata: #{metadata}")
             Chef::Provisioning.inline_resource(action_handler) do
               remote_file package_file do
-                source metadata['url']
-                checksum metadata['sha256']
+                source metadata[:url]
+                checksum metadata[:sha256]
               end
             end
 
@@ -144,16 +151,20 @@ module Provisioning
         metadata = {}
         metadata_str.each_line do |line|
           key, value = line.split("\t", 2)
-          metadata[key] = value.chomp
+          metadata[key.to_sym] = value.chomp
         end
         metadata
       end
 
-      def install_package(action_handler, machine, remote_package_file)
+      def install_package(action_handler, machine, platform, remote_package_file)
         extension = File.extname(remote_package_file)
         result = case extension
         when '.rpm'
-          machine.execute(action_handler, "rpm -Uvh --oldpackage --replacepkgs \"#{remote_package_file}\"")
+          if platform == "wrlinux"
+            machine.execute(action_handler, "yum install -yv \"#{remote_package_file}\"")
+          else
+            machine.execute(action_handler, "rpm -Uvh --oldpackage --replacepkgs \"#{remote_package_file}\"")
+          end
         when '.deb'
           machine.execute(action_handler, "dpkg -i \"#{remote_package_file}\"")
         when '.solaris'

data/lib/chef/provisioning/convergence_strategy/install_msi.rb

@@ -20,10 +20,11 @@ module Provisioning
 
       def setup_convergence(action_handler, machine)
         if !convergence_options.has_key?(:client_rb_path) || !convergence_options.has_key?(:client_pem_path)
-          system_drive = machine.execute_always('$env:SystemDrive').stdout.strip
+          system_drive = machine.system_drive
           @convergence_options = Cheffish::MergedConfig.new(convergence_options, {
             :client_rb_path => "#{system_drive}\\chef\\client.rb",
-            :client_pem_path => "#{system_drive}\\chef\\client.pem"
+            :client_pem_path => "#{system_drive}\\chef\\client.pem",
+            :install_script_path => "#{system_drive}\\chef\\\install.ps1"
           })
         end
 
@@ -36,7 +37,16 @@ module Provisioning
         super
 
         install_command = Mixlib::Install.new(chef_version, true, opts).install_command
-        machine.execute(action_handler, install_command)
+        machine.write_file(action_handler, convergence_options[:install_script_path], install_command)
+
+        action_handler.open_stream(machine.node['name']) do |stdout|
+          action_handler.open_stream(machine.node['name']) do |stderr|
+            machine.execute(action_handler, "powershell.exe -ExecutionPolicy Unrestricted -NoProfile \"& \"\"#{convergence_options[:install_script_path]}\"\"\"",
+              :raw => true,
+              :stream_stdout => stdout,
+              :stream_stderr => stderr)
+          end
+        end
       end
 
       def converge(action_handler, machine)
@@ -47,6 +57,7 @@ module Provisioning
             command_line = "chef-client"
             command_line << " -l #{config[:log_level].to_s}" if config[:log_level]
             machine.execute(action_handler, command_line,
+              :raw => true,
               :stream_stdout => stdout,
               :stream_stderr => stderr,
               :timeout => @chef_client_timeout)