spiceweasel 2.0.1 → 2.1.0

data/README.md

@@ -3,7 +3,7 @@
 
 # Description #
 
-Spiceweasel is a command-line tool for batch loading Chef infrastructure from a file. It provides a simple syntax in either JSON or YAML for describing and deploying infrastructure in order with the Chef command-line tool `knife`. This manifest may be bundled with a Chef repository to deploy the infrastructure contained within the repository and validate that the components listed are all present.
+Spiceweasel is a command-line tool for batch loading Chef infrastructure from a file. It provides a simple syntax in Ruby, JSON or YAML for describing and deploying infrastructure in order with the Chef command-line tool `knife`. This manifest may be bundled with a Chef repository to deploy the infrastructure contained within the repository and validate that the components listed are all present.
 
 The `examples` directory provides example manifests based on the Quick Starts provided at http://help.opscode.com/kb/otherhelp. The https://github.com/mattray/vbacd-repo provides a working example for bootstrapping a Chef repository with Spiceweasel.
 
@@ -11,13 +11,13 @@ The [CHANGELOG.md](https://github.com/mattray/spiceweasel/blob/master/CHANGELOG.
 
 # Requirements #
 
-Spiceweasel currently depends on `knife` to run commands for it, and requires the `chef` gem for validating cookbook metadata. Infrastructure files must either end in `.json` or `.yml` to be processed.
+Spiceweasel currently depends on `knife` to run commands for it, and requires the `chef` gem for validating cookbook metadata. Infrastructure files must end in `.rb`, `.json` or `.yml` to be processed.
 
-Written and tested initially with Chef 0.9.12 (may still work) and continuing development with the 10.x series. It is tested with Ruby 1.8.7 and 1.9.2.
+Written and tested initially with Chef 0.9.12 (may still work) and continuing development with the 10.x series. It is tested with Ruby 1.9.3. Version 2.0 was the last version known to work with Ruby 1.8.7 due to the introduction of the Berkshelf dependency in 2.1.
 
 # File Syntax #
 
-The syntax for the Spiceweasel file may be either JSON or YAML format of Chef primitives describing what is to be instantiated. Please refer to the [examples/example.json](https://github.com/mattray/spiceweasel/blob/master/examples/example.json) or [examples/example.yml](https://github.com/mattray/spiceweasel/blob/master/examples/example.yml) for examples of the same infrastructure. Each subsection below shows the YAML syntax converted to knife commands.
+The syntax for the Spiceweasel file may be Ruby, JSON or YAML format of Chef primitives describing what is to be instantiated. Please refer to the [examples/example.json](https://github.com/mattray/spiceweasel/blob/master/examples/example.json) or [examples/example.yml](https://github.com/mattray/spiceweasel/blob/master/examples/example.yml) for examples of the same infrastructure. Each subsection below shows the YAML syntax converted to knife commands.
 
 ## Manifest syntax changes in Spiceweasel 2.0 ##
 
@@ -50,6 +50,7 @@ cookbooks:
     version: 1.2.0
     options: --freeze
 - mysql:
+- ntp:
 ```
 
 produces the knife commands
@@ -60,7 +61,35 @@ knife cookbook site download apt 1.2.0 --file cookbooks/apt.tgz
 tar -C cookbooks/ -xf cookbooks/apt.tgz
 rm -f cookbooks/apt.tgz
 knife cookbook upload apt --freeze
-knife cookbook upload mysql
+knife cookbook upload mysql ntp
+```
+
+## Berkshelf ##
+
+If you prefer to use Berkshelf for managing your cookbooks, Spiceweasel supports adding `berksfile:` and the ability to specify the path and any Berkshelf options. You may mix use of `berkshelf:` with `cookbooks:` if you wish as well. Berkshelf-managed cookbooks will be included in the validation of roles, environments and run lists as well. You may simply use the YAML snippet
+
+``` yaml
+berkshelf:
+```
+
+which produces the command
+
+```
+berks upload -b ./Berksfile
+```
+
+or you may use additional options like
+
+``` yaml
+berksfile:
+    path: '/Users/mray/ws/lab-repo/Berksfile'
+    options: '--skip_syntax_check --config some_config.json'
+```
+
+which produces the output
+
+```
+berks upload --skip_syntax_check --config some_config.json -b /Users/mray/ws/lab-repo/Berksfile
 ```
 
 ## Environments ##
@@ -77,9 +106,7 @@ environments:
 produces the knife commands
 
 ```
-knife environment from file development.rb
-knife environment from file qa.rb
-knife environment from file production.rb
+knife environment from file development.rb qa.rb production.rb
 ```
 
 ## Roles ##
@@ -97,10 +124,7 @@ roles:
 produces the knife commands
 
 ```
-knife role from file base.rb
-knife role from file iisserver.rb
-knife role from file monitoring.rb
-knife role from file webserver.rb
+knife role from file base.rb iisserver.rb monitoring.rb webserver.rb
 ```
 
 n## Data Bags ##
@@ -128,15 +152,11 @@ produces the knife commands
 
 ```
 knife data bag create users
-knife data bag from file users alice.json
-knife data bag from file users bob.json
-knife data bag from file users chuck.json
+knife data bag from file users alice.json bob.json chuck.json
 knife data bag create data
-knife data bag from file data dataA.json
-knife data bag from file data dataB.json
+knife data bag from file data *.json
 knife data bag create passwords
-knife data bag from file passwords mysql.json --secret-file secret_key_filename
-knife data bag from file passwords rabbitmq.json --secret-file secret_key_filename
+knife data bag from file passwords mysql.json rabbitmq.json --secret-file secret_key_filename
 ```
 
 ## Nodes ##

data/lib/spiceweasel/berksfile.rb

@@ -0,0 +1,59 @@
+#
+# Author:: Matt Ray (<matt@opscode.com>)
+#
+# Copyright:: 2013, Opscode, Inc <legal@opscode.com>
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+module Spiceweasel
+  class Berksfile
+
+    attr_reader :create
+    attr_reader :delete
+    attr_reader :cookbook_list
+
+    include CommandHelper
+
+    def initialize(berkshelf=nil)
+      @create = []
+      @delete = []
+      @cookbook_list = {}
+      # only load berkshelf if we are going to use it
+      require 'berkshelf'
+      berks_options = []
+      case berkshelf
+      when String
+        path = berkshelf
+      when Hash
+        path = berkshelf['path']
+        berks_options << berkshelf['options'] if berkshelf['options']
+      end
+      path ||= './Berksfile'
+      berks_options << "-b #{path}"
+      berks_options = berks_options.join(' ')
+      opts = Thor::Options.split(berks_options.split(' ')).last
+      resolve_opts = Thor::Options.new(Berkshelf::Cli.tasks['upload'].options).parse(opts)
+      berks = Berkshelf::Berksfile.from_file(path)
+      create_command("berks upload #{berks_options}")
+      Berkshelf.ui.mute do
+        Spiceweasel::Log.debug("berkshelf resolving dependencies: #{resolve_opts}")
+        berks.resolve(resolve_opts).each do |cb|
+          @cookbook_list[cb.cookbook_name] = cb.version
+          delete_command("knife cookbook#{Spiceweasel::Config[:knife_options]} delete #{cb.cookbook_name} #{cb.version} -a -y")
+        end
+      end
+    end
+  end
+
+end

data/lib/spiceweasel/cli.rb

@@ -21,7 +21,9 @@ require 'json'
 require 'yaml'
 
 require 'spiceweasel'
+require 'spiceweasel/command_helper'
 require 'spiceweasel/cookbooks'
+require 'spiceweasel/berksfile'
 require 'spiceweasel/environments'
 require 'spiceweasel/roles'
 require 'spiceweasel/data_bags'
@@ -130,6 +132,10 @@ module Spiceweasel
     :proc => lambda { |v| puts "Spiceweasel: #{::Spiceweasel::VERSION}" },
     :exit => 0
 
+    option :unique_id,
+    :long => '--unique-id UID',
+    :description => 'Unique ID generally used for ruby based configs'
+
     def run
       if Spiceweasel::Config[:extractlocal] || Spiceweasel::Config[:extractjson] || Spiceweasel::Config[:extractyaml]
         manifest = Spiceweasel::ExtractLocal.parse_objects
@@ -143,19 +149,24 @@ module Spiceweasel
       end
       Spiceweasel::Log.debug("file manifest: #{manifest}")
 
-      cookbooks = Cookbooks.new(manifest['cookbooks'])
+      berksfile = Berksfile.new(manifest['berksfile']) if manifest.include?('berksfile')
+      if berksfile
+        cookbooks = Cookbooks.new(manifest['cookbooks'], berksfile.cookbook_list)
+        create = berksfile.create + cookbooks.create
+        delete = berksfile.delete + cookbooks.delete
+      else
+        cookbooks = Cookbooks.new(manifest['cookbooks'])
+        create = cookbooks.create
+        delete = cookbooks.delete
+      end
       environments = Environments.new(manifest['environments'], cookbooks)
       roles = Roles.new(manifest['roles'], environments, cookbooks)
       data_bags = DataBags.new(manifest['data bags'])
       nodes = Nodes.new(manifest['nodes'], cookbooks, environments, roles)
       clusters = Clusters.new(manifest['clusters'], cookbooks, environments, roles)
 
-      create = cookbooks.create + environments.create + roles.create + data_bags.create + nodes.create + clusters.create
-      delete = cookbooks.delete + environments.delete + roles.delete + data_bags.delete + nodes.delete + clusters.delete
-
-      #trim trailing whitespace
-      create.each {|x| x.rstrip!}
-      delete.each {|x| x.rstrip!}
+      create += environments.create + roles.create + data_bags.create + nodes.create + clusters.create
+      delete += environments.delete + roles.delete + data_bags.delete + nodes.delete + clusters.delete
 
       if Spiceweasel::Config[:extractjson]
         puts JSON.pretty_generate(manifest)
@@ -227,8 +238,11 @@ module Spiceweasel
           output = YAML.load_file(file)
         elsif (file.end_with?(".json"))
           output = JSON.parse(File.read(file))
+        elsif (file.end_with?(".rb"))
+          output = self.instance_eval(IO.read(file), file, 1)
+          output = JSON.parse(JSON.dump(output))
         else
-          STDERR.puts "ERROR: #{file} is an unknown file type, please use a file ending with either '.json' or '.yml'."
+          STDERR.puts "ERROR: #{file} is an unknown file type, please use a file ending with '.rb', '.json' or '.yml'."
           exit(-1)
         end
       rescue Psych::SyntaxError => e
@@ -239,8 +253,9 @@ module Spiceweasel
         STDERR.puts e.message
         STDERR.puts "ERROR: Parsing error in #{file}."
         exit(-1)
-      rescue Exception
-        STDERR.puts "ERROR: No manifest .json or .yml file provided."
+      rescue Exception => e
+        STDERR.puts "ERROR: Invalid or missing  manifest .json, .rb, or .yml file provided."
+        STDERR.puts "ERROR: #{e}"
         puts opt_parser.to_s
         exit(-1)
       end

data/lib/spiceweasel/command.rb

@@ -0,0 +1,25 @@
+module Spiceweasel
+  class Command
+
+    attr_reader :allow_failure
+    attr_reader :timeout
+    attr_reader :command
+    
+    def initialize(command, options={})
+      @command = command.rstrip
+      @options = options
+      @timeout = options['timeout']
+      @allow_failure = options.has_key?('allow_failure') ? options['allow_failure'] : true
+    end
+
+    def shellout_opts
+      opts = {}
+      opts[:timeout] = timeout if timeout
+      opts
+    end
+
+    alias_method :allow_failure?, :allow_failure
+    alias_method :to_s, :command
+
+  end
+end

data/lib/spiceweasel/command_helper.rb

@@ -0,0 +1,15 @@
+require 'spiceweasel/command'
+
+module Spiceweasel
+  module CommandHelper
+    def create_command(*args)
+      @create ||= []
+      @create.push(Command.new(*args))
+    end
+
+    def delete_command(*args)
+      @delete ||= []
+      @delete.push(Command.new(*args))
+    end
+  end
+end

data/lib/spiceweasel/cookbooks.rb

@@ -21,16 +21,20 @@ require 'chef/cookbook/metadata'
 module Spiceweasel
   class Cookbooks
 
+    include CommandHelper
+
     attr_reader :cookbook_list, :create, :delete
 
-    def initialize(cookbooks = [])
+    def initialize(cookbooks = [], other_cookbook_list = {})
       @create = Array.new
       @delete = Array.new
-      @cookbook_list = Hash.new
+      @cookbook_list = other_cookbook_list
       @dependencies = Array.new
       #validate each of the cookbooks specified in the manifest
       if cookbooks
         Spiceweasel::Log.debug("cookbooks: #{cookbooks}")
+
+        c_names = []
         cookbooks.each do |cookbook|
           name = cookbook.keys.first
           if cookbook[name]
@@ -43,21 +47,33 @@ module Spiceweasel
               validateMetadata(name,version) unless Spiceweasel::Config[:novalidation]
             else
               if Spiceweasel::Config[:siteinstall] #use knife cookbook site install
-                @create.push("knife cookbook#{Spiceweasel::Config[:knife_options]} site install #{name} #{version} #{options}")
+                create_command("knife cookbook#{Spiceweasel::Config[:knife_options]} site install #{name} #{version} #{options}")
               else #use knife cookbook site download, untar and then remove the tarball
-                @create.push("knife cookbook#{Spiceweasel::Config[:knife_options]} site download #{name} #{version} --file cookbooks/#{name}.tgz #{options}")
-                @create.push("tar -C cookbooks/ -xf cookbooks/#{name}.tgz")
-                @create.push("rm -f cookbooks/#{name}.tgz")
+                create_command("knife cookbook#{Spiceweasel::Config[:knife_options]} site download #{name} #{version} --file cookbooks/#{name}.tgz #{options}")
+                create_command("tar -C cookbooks/ -xf cookbooks/#{name}.tgz")
+                create_command("rm -f cookbooks/#{name}.tgz")
               end
             end
           elsif !Spiceweasel::Config[:novalidation]
             STDERR.puts "ERROR: 'cookbooks' directory not found, unable to validate, download and load cookbooks"
             exit(-1)
           end
-          @create.push("knife cookbook#{Spiceweasel::Config[:knife_options]} upload #{name} #{options}")
-          @delete.push("knife cookbook#{Spiceweasel::Config[:knife_options]} delete #{name} #{version} -a -y")
+
+          if(options)
+            if !c_names.empty?
+              create_command("knife cookbook#{Spiceweasel::Config[:knife_options]} upload #{c_names.join(' ')}")
+              c_names = []
+            end
+            create_command("knife cookbook#{Spiceweasel::Config[:knife_options]} upload #{name} #{options}")
+          else
+            c_names.push(name)
+          end
+          delete_command("knife cookbook#{Spiceweasel::Config[:knife_options]} delete #{name} #{version} -a -y")
           @cookbook_list[name] = version #used for validation
         end
+        if !c_names.empty?
+          create_command("knife cookbook#{Spiceweasel::Config[:knife_options]} upload #{c_names.join(' ')}")
+        end
         validateDependencies() unless Spiceweasel::Config[:novalidation]
       end
     end

data/lib/spiceweasel/data_bags.rb

@@ -21,6 +21,8 @@ require 'json'
 module Spiceweasel
   class DataBags
 
+    include CommandHelper
+
     attr_reader :create, :delete
 
     def initialize(data_bags = [])
@@ -39,8 +41,8 @@ module Spiceweasel
             STDERR.puts "ERROR: 'data_bags/#{db}' directory not found, unable to validate or load data bag items"
             exit(-1)
           end
-          @create.push("knife data bag#{Spiceweasel::Config[:knife_options]} create #{db}")
-          @delete.push("knife data bag#{Spiceweasel::Config[:knife_options]} delete #{db} -y")
+          create_command("knife data bag#{Spiceweasel::Config[:knife_options]} create #{db}")
+          delete_command("knife data bag#{Spiceweasel::Config[:knife_options]} delete #{db} -y")
           if data_bag[db]
             items = data_bag[db]['items']
             secret = data_bag[db]['secret']
@@ -60,11 +62,11 @@ module Spiceweasel
               next
             end
             validateItem(db, item) unless Spiceweasel::Config[:novalidation]
-            if secret
-              @create.push("knife data bag#{Spiceweasel::Config[:knife_options]} from file #{db} #{item}.json --secret-file #{secret}")
-            else
-              @create.push("knife data bag#{Spiceweasel::Config[:knife_options]} from file #{db} #{item}.json")
-            end
+          end
+          if secret
+            create_command("knife data bag#{Spiceweasel::Config[:knife_options]} from file #{db} #{items.join('.json ')}.json --secret-file #{secret}")
+          else
+            create_command("knife data bag#{Spiceweasel::Config[:knife_options]} from file #{db} #{items.join('.json ')}.json")
           end
         end
       end

data/lib/spiceweasel/environments.rb

@@ -1,7 +1,7 @@
 #
 # Author:: Matt Ray (<matt@opscode.com>)
 #
-# Copyright:: 2011-2012, Opscode, Inc <legal@opscode.com>
+# Copyright:: 2011-2013, Opscode, Inc <legal@opscode.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -21,6 +21,8 @@ require 'json'
 module Spiceweasel
   class Environments
 
+    include CommandHelper
+
     attr_reader :environment_list, :create, :delete
 
     def initialize(environments = [], cookbooks = {})
@@ -29,6 +31,7 @@ module Spiceweasel
       @environment_list = Array.new
       if environments
         Spiceweasel::Log.debug("environments: #{environments}")
+        envfiles = []
         environments.each do |env|
           name = env.keys[0]
           Spiceweasel::Log.debug("environment: #{name}")
@@ -39,56 +42,38 @@ module Spiceweasel
             exit(-1)
           end
           if File.exists?("environments/#{name}.json")
-            @create.push("knife environment#{Spiceweasel::Config[:knife_options]} from file #{name}.json")
+            envfiles.push("#{name}.json")
           else #assume no .json means they want .rb and catchall for misssing dir
-            @create.push("knife environment#{Spiceweasel::Config[:knife_options]} from file #{name}.rb")
+            envfiles.push("#{name}.rb")
           end
-          @delete.push("knife environment#{Spiceweasel::Config[:knife_options]} delete #{name} -y")
+          delete_command("knife environment#{Spiceweasel::Config[:knife_options]} delete #{name} -y")
           @environment_list.push(name)
         end
+        create_command("knife environment#{Spiceweasel::Config[:knife_options]} from file #{envfiles.join(' ')}")
       end
     end
 
     #validate the content of the environment file
     def validate(environment, cookbooks)
-      #validate the environments passed in match the name of either the .rb or .json
-      if File.exists?("environments/#{environment}.rb")
-        #validate that the name inside the file matches
-        name = File.open("environments/#{environment}.rb").grep(/^name/)[0].split()[1].gsub(/"/,'').to_s
-        if !environment.eql?(name)
+      file = %W(environments/#{environment}.rb environments/#{environment}.json).detect{|f| File.exists?(f)}
+      if file
+        if (Chef::Version.new(Chef::VERSION) < Chef::Version.new('11.0.0'))
+          env = Chef::Environment.new(false)
+        else
+          env = Chef::Environment.new
+        end
+        env.from_file(file)
+        if(env.name != environment)
           STDERR.puts "ERROR: Environment '#{environment}' listed in the manifest does not match the name '#{name}' within the environments/#{environment}.rb file."
           exit(-1)
         end
-        #validate the cookbooks exist if they're mentioned
-        envcookbooks = File.open("environments/#{environment}.rb").grep(/^cookbook /)
-        envcookbooks.each do |cb|
-          dep = cb.split()[1].gsub(/"/,'').gsub(/,/,'')
+        env.cookbook_versions.keys.each do |dep|
           Spiceweasel::Log.debug("environment: '#{environment}' cookbook: '#{dep}'")
-          if !cookbooks.member?(dep)
+          unless cookbooks.member?(dep)
             STDERR.puts "ERROR: Cookbook dependency '#{dep}' from environment '#{environment}' is missing from the list of cookbooks in the manifest."
             exit(-1)
           end
         end
-      elsif File.exists?("environments/#{environment}.json")
-        #load the json, don't symbolize since we don't need json_class
-        f = File.read("environments/#{environment}.json")
-        JSON.create_id = nil
-        envfile = JSON.parse(f, {:symbolize_names => false})
-        Spiceweasel::Log.debug("environment: '#{environment}' file: '#{envfile}'")
-        #validate that the name inside the file matches
-        Spiceweasel::Log.debug("environment: '#{environment}' name: '#{envfile['name']}'")
-        if !environment.eql?(envfile['name'])
-          STDERR.puts "ERROR: Environment '#{environment}' listed in the manifest does not match the name '#{envfile['name']}' within the 'environments/#{environment}.json' file."
-          exit(-1)
-        end
-        #validate the cookbooks exist if they're mentioned
-        envfile['cookbook_versions'].keys.each do |cb|
-          Spiceweasel::Log.debug("environment: '#{environment}' cookbook: '#{cb}'")
-          if !cookbooks.member?(cb.to_s)
-            STDERR.puts "ERROR: Cookbook dependency '#{cb}' from environment '#{environment}' is missing from the list of cookbooks in the manifest."
-            exit(-1)
-          end
-        end
       else #environment is not here
         STDERR.puts "ERROR: Invalid Environment '#{environment}' listed in the manifest but not found in the environments directory."
         exit(-1)

data/lib/spiceweasel/execute.rb

@@ -25,15 +25,14 @@ module Spiceweasel
     def initialize(commands)
       # for now we're shelling out
       commands.each do | cmd |
-        knife = Mixlib::ShellOut.new(cmd)
+        knife = Mixlib::ShellOut.new(cmd.command, cmd.shellout_opts.merge(:live_stream => STDOUT))
         # check for parallel? and eventually use threads
         knife.run_command
-        puts cmd
-        puts knife.stdout
         puts knife.stderr
         Spiceweasel::Log.debug(cmd)
         Spiceweasel::Log.debug(knife.stdout)
         Spiceweasel::Log.fatal(knife.stderr) if !knife.stderr.empty?
+        find.error! unless cmd.allow_failure?
       end
     end
 

data/lib/spiceweasel/extract_local.rb

@@ -2,7 +2,7 @@
 # Author:: Geoff Meakin
 # Author:: Matt Ray (<matt@opscode.com>)
 #
-# Copyright:: 2012, Opscode, Inc <legal@opscode.com>
+# Copyright:: 2012-2013, Opscode, Inc <legal@opscode.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,27 +17,22 @@
 # limitations under the License.
 #
 
-require 'chef/cookbook/metadata'
+require 'chef'
 
 module Spiceweasel
   class ExtractLocal
 
     def self.parse_objects
-      objects = {'cookbooks' => nil, 'roles' => nil, 'environments' => nil, 'data bags' => nil, 'nodes' => nil}
+      objects = {}
 
-      # COOKBOOKS
-      cookbooks = Array.new
-      Dir.glob('cookbooks/*').each do |cookbook_full_path|
-        metadata = Chef::Cookbook::Metadata.new
-        metadata.from_file("#{cookbook_full_path}/metadata.rb")
-        if metadata.name.empty?
-          Spiceweasel::Log.fatal("No cookbook name in the #{cookbook_full_path}/metadata.rb.")
-          exit(-1)
-        end
-        Spiceweasel::Log.debug("dir_ext: #{metadata.name} #{metadata.version}")
-        cookbooks.push(metadata)
+      # BERKSHELF
+      if File.file?('./Berksfile')
+        objects['berksfile'] = nil
+        berksfile = Berksfile.new(objects['berksfile'])
       end
-      cookbooks = self.order_cookbooks_by_dependency(cookbooks)
+
+      # COOKBOOKS
+      cookbooks = self.resolve_cookbooks(berksfile.cookbook_list)
       objects['cookbooks'] = cookbooks unless cookbooks.empty?
 
       # ROLES
@@ -80,7 +75,6 @@ module Spiceweasel
       #   nodes  << {node => nil}
       # end
       # objects['nodes'] = nodes unless nodes.empty?
-
       objects
     end
 
@@ -92,39 +86,36 @@ module Spiceweasel
       name.join('.')
     end
 
-    def self.order_cookbooks_by_dependency(cookbooks)
-      # Weak algorithm, not particularly elegant, ignores version info as unlikely to have two versions of a cookbook anyway
-      # We're going to find the cookbooks with their dependencies matched and keep going until all we have is unmatched deps
-      sorted_cookbooks = Array.new
-      unsorted_cookbooks = cookbooks
-      scount = 0
-      #keep looping until no more cookbooks are left or can't remove remainders
-      while unsorted_cookbooks.any? and scount < cookbooks.length
-        cookbook = unsorted_cookbooks.shift
-        Spiceweasel::Log.debug("dir_ext: cookbook.dependencies: '#{cookbook.name}' #{cookbook.dependencies}")
-        #if all the cookbook dependencies are in sorted_cookbooks
-        if sorted_cookbooks.eql?(sorted_cookbooks | cookbook.dependencies.collect {|x| x[0]})
-          sorted_cookbooks.push(cookbook.name)
-          scount = 0
-        else #put it back in the list
-          unsorted_cookbooks.push(cookbook)
-          scount = scount + 1
-        end
-        Spiceweasel::Log.debug("dir_ext: sorted_cookbooks: '#{sorted_cookbooks}' #{scount}")
+    def self.resolve_cookbooks(berkshelf_cookbooks = {})
+      require 'solve'
+      loader = Chef::CookbookLoader.new('./cookbooks')
+      loader.load_cookbooks
+      books = loader.cookbooks_by_name
+      graph = Solve::Graph.new
+      cblist = []
+      #push in the berkshelf cookbooks to cover any other deps
+      berkshelf_cookbooks.each do |name, version|
+        Spiceweasel::Log.debug("dir_ext:berks: #{name} #{version}")
+        graph.artifacts(name, version)
       end
-      if scount > 0
-        remainders = unsorted_cookbooks.collect {|x| x.name}
-        Spiceweasel::Log.debug("dir_ext: remainders: '#{remainders}'")
-        if Spiceweasel::Config[:novalidation] #stuff is missing, oh well
-          sorted_cookbooks.push(remainders).flatten!
-        else
-          deps = unsorted_cookbooks.collect {|x| x.dependencies.collect {|x| x[0]} - sorted_cookbooks}
-          STDERR.puts "ERROR: Dependencies not satisfied or circular dependencies in cookbook(s): #{remainders} depend(s) on #{deps}"
-          exit(-1)
+      books.each do |name, cb|
+        Spiceweasel::Log.debug("dir_ext: #{name} #{cb.version}")
+        artifact = graph.artifacts(name, cb.version)
+        cblist.push([name, cb.version])
+        cb.metadata.dependencies.each do |dep_name, dep_version|
+          artifact.depends(dep_name, dep_version)
         end
       end
-      #hack to get the format same as yaml/json parse
-      sorted_cookbooks.collect { |x| { x => nil } }
+      #get the cookbooks and their versions, map to cookbook hash format
+      begin
+        cookbooks = []
+        Solve.it!(graph, cblist).each {|k,v| cookbooks.push({k => {'version' => v}})}
+      rescue Solve::Errors::NoSolutionError => e
+        STDERR.puts "ERROR: There are missing cookbook dependencies, please check your metadata.rb files."
+        exit(-1)
+      end
+      #remove any cookbooks managed by berkshelf
+      cookbooks.delete_if {|x| berkshelf_cookbooks.keys.member?(x.keys[0])}
     end
   end
 end

data/lib/spiceweasel/nodes.rb

@@ -1,7 +1,7 @@
 #
 # Author:: Matt Ray (<matt@opscode.com>)
 #
-# Copyright:: 2011-2012, Opscode, Inc <legal@opscode.com>
+# Copyright:: 2011-2013, Opscode, Inc <legal@opscode.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -19,13 +19,16 @@
 module Spiceweasel
   class Nodes
 
-    PROVIDERS = %w{bluebox clodo cs ec2 gandi hp lxc openstack rackspace slicehost terremark voxel}
+    include CommandHelper
+
+    PROVIDERS = %w{bluebox clodo cs ec2 gandi hp lxc openstack rackspace slicehost terremark voxel vagrant}
 
     attr_reader :create, :delete
 
     def initialize(nodes, cookbooks, environments, roles)
       @create = Array.new
       @delete = Array.new
+      bulk_delete = false
       if nodes
         Spiceweasel::Log.debug("nodes: #{nodes}")
         nodes.each do |node|
@@ -40,6 +43,13 @@ module Spiceweasel
             options = node[name]['options'] || ''
             Spiceweasel::Log.debug("node: '#{name}' options: '#{options}'")
             validateOptions(name, options, environments) unless Spiceweasel::Config[:novalidation]
+            create_command_options = {}
+            %w(allow_create_failure timeout).each do |key|
+              if(node[name].has_key?(key))
+                create_command_options[key] = node[name][key]
+              end
+            end
+            additional_commands = node[name]['additional_commands'] || []
           end
           #provider support
           provider = name.split()
@@ -48,43 +58,60 @@ module Spiceweasel
             if provider.length == 2
               count = provider[1]
             end
+            provided_names = []
             if Spiceweasel::Config[:parallel]
               parallel = "seq #{count} | parallel -j 0 -v \""
               parallel += "knife #{provider[0]}#{Spiceweasel::Config[:knife_options]} server create #{options}".gsub(/\{\{n\}\}/, '{}')
               parallel += " -r '#{run_list}'" unless run_list.empty?
               parallel += "\""
-              @create.push(parallel)
+              create_command(parallel, create_command_options)
             else
               count.to_i.times do |i|
                 server = "knife #{provider[0]}#{Spiceweasel::Config[:knife_options]} server create #{options}".gsub(/\{\{n\}\}/, (i + 1).to_s)
                 server += " -r '#{run_list}'" unless run_list.empty?
-                @create.push(server)
+                provided_names << node[name]['name'].gsub('{{n}}', (i + 1).to_s) if node[name]['name']
+                create_command(server, create_command_options)
+              end
+            end
+            if provided_names.empty? && provider[0] != 'windows'
+              bulk_delete = true
+              delete_command("knife node#{Spiceweasel::Config[:knife_options]} list | xargs knife #{provider[0]} server delete -y")
+            else
+              provided_names.each do |p_name|
+                delete_command("knife #{provider[0]} server delete -y #{p_name}")
+                delete_command("knife node#{Spiceweasel::Config[:knife_options]} delete #{p_name} -y")
+                delete_command("knife client#{Spiceweasel::Config[:knife_options]} delete #{p_name} -y")
               end
             end
-            @delete.push("knife node#{Spiceweasel::Config[:knife_options]} list | xargs knife #{provider[0]} server delete -y")
           elsif name.start_with?("windows") #windows node bootstrap support
             nodeline = name.split()
             provider = nodeline.shift.split('_') #split on 'windows_ssh' etc
             nodeline.each do |server|
-              server = "knife bootstrap #{provider[0]} #{provider[1]}#{Spiceweasel::Config[:knife_options]} #{server} #{options}"
-              server += " -r '#{run_list}'" unless run_list.empty?
-              @create.push(server)
-              @delete.push("knife node#{Spiceweasel::Config[:knife_options]} delete #{server} -y")
-              @delete.push("knife client#{Spiceweasel::Config[:knife_options]} delete #{server} -y")
+              servercommand = "knife bootstrap #{provider[0]} #{provider[1]}#{Spiceweasel::Config[:knife_options]} #{server} #{options}"
+              servercommand += " -r '#{run_list}'" unless run_list.empty?
+              create_command(servercommand, create_command_options)
+              delete_command("knife node#{Spiceweasel::Config[:knife_options]} delete #{server} -y")
+              delete_command("knife client#{Spiceweasel::Config[:knife_options]} delete #{server} -y")
             end
-            @delete.push("knife node#{Spiceweasel::Config[:knife_options]} list | xargs knife #{provider[0]} server delete -y")
           else #node bootstrap support
             name.split.each_with_index do |server, i|
-              server = "knife bootstrap#{Spiceweasel::Config[:knife_options]} #{server} #{options}".gsub(/\{\{n\}\}/, (i + 1).to_s)
-              server += " -r '#{run_list}'" unless run_list.empty?
-              @create.push(server)
-              @delete.push("knife node#{Spiceweasel::Config[:knife_options]} delete #{server} -y")
-              @delete.push("knife client#{Spiceweasel::Config[:knife_options]} delete #{server} -y")
+              servercommand = "knife bootstrap#{Spiceweasel::Config[:knife_options]} #{server} #{options}".gsub(/\{\{n\}\}/, (i + 1).to_s)
+              servercommand += " -r '#{run_list}'" unless run_list.empty?
+              create_command(servercommand, create_command_options)
+              delete_command("knife node#{Spiceweasel::Config[:knife_options]} delete #{server} -y")
+              delete_command("knife client#{Spiceweasel::Config[:knife_options]} delete #{server} -y")
+            end
+          end
+          unless additional_commands.empty?
+            additional_commands.each do |cmd|
+              create_command(cmd, create_command_options)
             end
           end
         end
       end
-      @delete.push("knife node#{Spiceweasel::Config[:knife_options]} bulk delete .* -y")
+      if bulk_delete
+        delete_command("knife node#{Spiceweasel::Config[:knife_options]} bulk delete .* -y")
+      end
     end
 
     #ensure run_list contents are listed previously.

data/lib/spiceweasel/roles.rb

@@ -1,7 +1,7 @@
 #
 # Author:: Matt Ray (<matt@opscode.com>)
 #
-# Copyright:: 2011-2012, Opscode, Inc <legal@opscode.com>
+# Copyright:: 2011-2013, Opscode, Inc <legal@opscode.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -17,10 +17,13 @@
 #
 
 require 'json'
+require 'chef'
 
 module Spiceweasel
   class Roles
 
+    include CommandHelper
+
     attr_reader :role_list, :create, :delete
 
     def initialize(roles = {}, environments = [], cookbooks = {})
@@ -30,6 +33,7 @@ module Spiceweasel
       if roles
         Spiceweasel::Log.debug("roles: #{roles}")
         flatroles = roles.collect {|x| x.keys}.flatten
+        rolefiles = []
         flatroles.each do |role|
           Spiceweasel::Log.debug("role: #{role}")
           if File.directory?("roles")
@@ -39,81 +43,51 @@ module Spiceweasel
             exit(-1)
           end
           if File.exists?("roles/#{role}.json")
-            @create.push("knife role#{Spiceweasel::Config[:knife_options]} from file #{role}.json")
+            rolefiles.push("#{role}.json")
           else #assume no .json means they want .rb and catchall for misssing dir
-            @create.push("knife role#{Spiceweasel::Config[:knife_options]} from file #{role}.rb")
+            rolefiles.push("#{role}.rb")
           end
-          @delete.push("knife role#{Spiceweasel::Config[:knife_options]} delete #{role} -y")
+          delete_command("knife role#{Spiceweasel::Config[:knife_options]} delete #{role} -y")
           @role_list.push(role)
         end
+        create_command("knife role#{Spiceweasel::Config[:knife_options]} from file #{rolefiles.join(' ')}")
       end
     end
 
     #validate the content of the role file
     def validate(role, environments, cookbooks, roles)
       #validate the role passed in match the name of either the .rb or .json
-      if File.exists?("roles/#{role}.rb")
-        #validate that the name inside the file matches
-        name = File.open("roles/#{role}.rb").grep(/^name/)[0].split()[1].gsub(/"/,'').to_s
-        Spiceweasel::Log.debug("role: '#{role}' name: '#{name}'")
-        if !role.eql?(name)
-          STDERR.puts "ERROR: Role '#{role}' listed in the manifest does not match the name '#{name}' within the roles/#{role}.rb file."
-          exit(-1)
+      file = %W(roles/#{role}.rb roles/#{role}.json).detect{|f| File.exists?(f)}
+      if(file)
+        if(Chef::Version.new(Chef::VERSION) < Chef::Version.new('11.0.0'))
+          c_role = Chef::Role.new(true)
+        else
+          c_role = Chef::Role.new
         end
-        #grab any lines with 'recipe[' or 'role['
-        rolerl = File.open("roles/#{role}.rb").grep(/recipe\[|role\[/)
-        rolerl.each do |line|
-          Spiceweasel::Log.debug("role: '#{role}' line: '#{line}'")
-          line.strip.split(',').each do |rl|
-            if rl =~ /recipe\[/ #it's a cookbook
-              #split on the brackets and any colons
-              dep = rl.split(/\[|\]/)[1].split(':')[0]
-              Spiceweasel::Log.debug("role: '#{role}' cookbook: '#{rl}': dep: '#{dep}'")
-              if !cookbooks.member?(dep)
-                STDERR.puts "ERROR: Cookbook dependency '#{dep}' from role '#{role}' is missing from the list of cookbooks in the manifest."
-                exit(-1)
-              end
-            elsif rl =~ /role\[/ #it's a role
-              #split on the brackets
-              dep = rl.split(/\[|\]/)[1]
-              Spiceweasel::Log.debug("role: '#{role}' role: '#{rl}': dep: '#{dep}'")
-              if !roles.member?(dep)
-                STDERR.puts "ERROR: Role dependency '#{dep}' from role '#{role}' is missing from the list of roles in the manifest."
-                exit(-1)
-              end
-            end
-          end
-        end
-        #TODO validate any environment-specific runlists
-      elsif File.exists?("roles/#{role}.json")
-        #load the json, don't symbolize since we don't need json_class
-        f = File.read("roles/#{role}.json")
-        JSON.create_id = nil
-        rolefile = JSON.parse(f, {:symbolize_names => false})
-        #validate that the name inside the file matches
-        Spiceweasel::Log.debug("role: '#{role}' name: '#{rolefile['name']}'")
-        if !role.eql?(rolefile['name'])
-          STDERR.puts "ERROR: Role '#{role}' listed in the manifest does not match the name '#{rolefile['name']}' within the 'roles/#{role}.json' file."
+        c_role.from_file(file)
+        Spiceweasel::Log.debug("role: '#{role}' name: '#{c_role.name}'")
+        if !role.eql?(c_role.name)
+          STDERR.puts "ERROR: Role '#{role}' listed in the manifest does not match the name '#{c_role.name}' within the roles/#{role}.rb file."
           exit(-1)
         end
-        #validate the cookbooks and roles exist if they're mentioned in run_lists
-        rolefile['run_list'].each do |rl|
-          if rl =~ /recipe\[/ #it's a cookbook
-            #split on the brackets and any colons
-            dep = rl.split(/\[|\]/)[1].split(':')[0]
-            Spiceweasel::Log.debug("role: '#{role}' cookbook: '#{rl}': dep: '#{dep}'")
-            if !cookbooks.member?(dep)
-              STDERR.puts "ERROR: Cookbook dependency '#{dep}' from role '#{role}' is missing from the list of cookbooks in the manifest."
+        c_role.run_list.each do |runlist_item|
+          if(runlist_item.recipe?)
+            Spiceweasel::Log.debug("recipe: #{runlist_item.name}")
+            cookbook,recipe = runlist_item.name.split('::')
+            Spiceweasel::Log.debug("role: '#{role}' cookbook: '#{cookbook}' dep: '#{runlist_item}'")
+            unless(cookbooks.member?(cookbook))
+              STDERR.puts "ERROR: Cookbook dependency '#{runlist_item}' from role '#{role}' is missing from the list of cookbooks in the manifest."
               exit(-1)
             end
-          elsif rl =~ /role\[/ #it's a role
-            #split on the brackets
-            dep = rl.split(/\[|\]/)[1]
-            Spiceweasel::Log.debug("role: '#{role}' role: '#{rl}': dep: '#{dep}'")
-            if !roles.member?(dep)
-              STDERR.puts "ERROR: Role dependency '#{dep}' from role '#{role}' is missing from the list of roles in the manifest."
+          elsif(runlist_item.role?)
+            Spiceweasel::Log.debug("role: '#{role}' role: '#{runlist_item}': dep: '#{runlist_item.name}'")
+            unless(roles.member?(runlist_item.name))
+              STDERR.puts "ERROR: Role dependency '#{runlist_item.name}' from role '#{role}' is missing from the list of roles in the manifest."
               exit(-1)
             end
+          else
+            STDERR.puts "ERROR: Unknown item in runlist: #{runlist_item.name}"
+            exit(-1)
           end
         end
         #TODO validate any environment-specific runlists

data/lib/spiceweasel/version.rb

@@ -17,5 +17,5 @@
 #
 
 module Spiceweasel
-  VERSION = '2.0.1'
+  VERSION = '2.1.0'
 end

data/spec/bin/spiceweasel_spec.rb

@@ -5,6 +5,7 @@ describe 'The Spiceweasel binary' do
 knife cookbook delete apache2  -a -y
 knife cookbook delete apt 1.2.0 -a -y
 knife cookbook delete mysql  -a -y
+knife cookbook delete ntp  -a -y
 knife environment delete development -y
 knife environment delete qa -y
 knife environment delete production -y
@@ -15,43 +16,34 @@ knife role delete webserver -y
 knife data bag delete users -y
 knife data bag delete data -y
 knife data bag delete passwords -y
-knife node delete knife bootstrap serverA -i ~/.ssh/mray.pem -x user --sudo -r 'role[base]' -y
-knife client delete knife bootstrap serverA -i ~/.ssh/mray.pem -x user --sudo -r 'role[base]' -y
-knife node delete knife bootstrap serverB -i ~/.ssh/mray.pem -x user --sudo -E production -r 'role[base]' -y
-knife client delete knife bootstrap serverB -i ~/.ssh/mray.pem -x user --sudo -E production -r 'role[base]' -y
-knife node delete knife bootstrap serverC -i ~/.ssh/mray.pem -x user --sudo -E production -r 'role[base]' -y
-knife client delete knife bootstrap serverC -i ~/.ssh/mray.pem -x user --sudo -E production -r 'role[base]' -y
+knife node delete serverA -y
+knife client delete serverA -y
+knife node delete serverB -y
+knife client delete serverB -y
+knife node delete serverC -y
+knife client delete serverC -y
 knife node list | xargs knife rackspace server delete -y
-knife node delete knife bootstrap windows winrm winboxA -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]' -y
-knife client delete knife bootstrap windows winrm winboxA -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]' -y
-knife node list | xargs knife windows server delete -y
-knife node delete knife bootstrap windows ssh winboxB -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]' -y
-knife client delete knife bootstrap windows ssh winboxB -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]' -y
-knife node delete knife bootstrap windows ssh winboxC -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]' -y
-knife client delete knife bootstrap windows ssh winboxC -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]' -y
-knife node list | xargs knife windows server delete -y
+knife node delete winboxA -y
+knife client delete winboxA -y
+knife node delete winboxB -y
+knife client delete winboxB -y
+knife node delete winboxC -y
+knife client delete winboxC -y
 knife node bulk delete .* -y
 knife node list | xargs knife ec2 server delete -y
 knife node list | xargs knife ec2 server delete -y
 knife node bulk delete .* -y
 knife cookbook upload apache2
 knife cookbook upload apt --freeze
-knife cookbook upload mysql
-knife environment from file development.rb
-knife environment from file qa.rb
-knife environment from file production.rb
-knife role from file base.rb
-knife role from file iisserver.rb
-knife role from file monitoring.rb
-knife role from file webserver.rb
+knife cookbook upload mysql ntp
+knife environment from file development.rb qa.rb production.rb
+knife role from file base.rb iisserver.rb monitoring.rb webserver.rb
 knife data bag create users
-knife data bag from file users alice.json
-knife data bag from file users bob.json
-knife data bag from file users chuck.json
+knife data bag from file users alice.json bob.json chuck.json
 knife data bag create data
+knife data bag from file data *.json
 knife data bag create passwords
-knife data bag from file passwords mysql.json --secret-file secret_key
-knife data bag from file passwords rabbitmq.json --secret-file secret_key
+knife data bag from file passwords mysql.json rabbitmq.json --secret-file secret_key
 knife bootstrap serverA -i ~/.ssh/mray.pem -x user --sudo -r 'role[base]'
 knife bootstrap serverB -i ~/.ssh/mray.pem -x user --sudo -E production -r 'role[base]'
 knife bootstrap serverC -i ~/.ssh/mray.pem -x user --sudo -E production -r 'role[base]'
@@ -70,7 +62,15 @@ knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7
     @spiceweasel_binary = File.join(File.dirname(__FILE__), *%w[.. .. bin spiceweasel])
   end
 
-  it "maintains consistent output from the example config" do
+  it "maintains consistent output from the example config with yml" do
     `#{@spiceweasel_binary} -r --novalidation examples/example.yml`.should == @expected_output
   end
+
+  it "maintains consistent output from the example config with json" do
+    `#{@spiceweasel_binary} -r --novalidation examples/example.json`.should == @expected_output
+  end
+
+  it "maintains consistent output from the example config with rb" do
+    `#{@spiceweasel_binary} -r --novalidation examples/example.rb`.should == @expected_output
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: spiceweasel
 version: !ruby/object:Gem::Version
-  version: 2.0.1
+  version: 2.1.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-01-19 00:00:00.000000000 Z
+date: 2013-03-29 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: json
@@ -107,6 +107,38 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: berkshelf
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: solve
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -135,8 +167,11 @@ files:
 - LICENSE
 - README.md
 - bin/spiceweasel
+- lib/spiceweasel/berksfile.rb
 - lib/spiceweasel/cli.rb
 - lib/spiceweasel/clusters.rb
+- lib/spiceweasel/command.rb
+- lib/spiceweasel/command_helper.rb
 - lib/spiceweasel/config.rb
 - lib/spiceweasel/cookbooks.rb
 - lib/spiceweasel/data_bags.rb