spiceweasel 2.1.2 → 2.2.0

data/README.md

@@ -3,7 +3,7 @@
 
 # Description #
 
-Spiceweasel is a command-line tool for batch loading Chef infrastructure from a file. It provides a simple syntax in Ruby, JSON or YAML for describing and deploying infrastructure in order with the Chef command-line tool `knife`. This manifest may be bundled with a Chef repository to deploy the infrastructure contained within the repository and validate that the components listed are all present.
+Spiceweasel is a command-line tool for batch loading Chef infrastructure. It provides a simple syntax in Ruby, JSON or YAML for describing and deploying infrastructure in order with the Chef command-line tool `knife`. This manifest may be bundled with a Chef repository to deploy the infrastructure contained within the repository and validate that the components listed are all present. The manifest may also be extracted from an existing repository.
 
 The `examples` directory provides example manifests based on the Quick Starts provided at http://help.opscode.com/kb/otherhelp. The https://github.com/mattray/vbacd-repo provides a working example for bootstrapping a Chef repository with Spiceweasel.
 
@@ -11,9 +11,9 @@ The [CHANGELOG.md](https://github.com/mattray/spiceweasel/blob/master/CHANGELOG.
 
 # Requirements #
 
-Spiceweasel currently depends on `knife` to run commands for it, and requires the `chef` gem for validating cookbook metadata. Infrastructure files must end in `.rb`, `.json` or `.yml` to be processed.
+Spiceweasel currently depends on `knife` to run commands for it, and requires the `chef` gem for validating cookbook metadata. [Berkshelf](https://berkshelf.com) is a dependency for the Cookbook `Berksfile` support. Infrastructure files must end in `.rb`, `.json` or `.yml` to be processed.
 
-Written and tested initially with Chef 0.9.12 (may still work) and continuing development with the 10.x series. It is tested with Ruby 1.9.3. Version 2.0 was the last version known to work with Ruby 1.8.7 due to the introduction of the Berkshelf dependency in 2.1.
+Written and tested with the Chef 11.x series (previous versions of Chef may still work). It is tested with Ruby 1.9.3. Version 2.0 was the last version known to work with Ruby 1.8.7 due to the introduction of the Berkshelf dependency in 2.1.
 
 # File Syntax #
 
@@ -66,10 +66,10 @@ knife cookbook upload mysql ntp
 
 ## Berkshelf ##
 
-If you prefer to use Berkshelf for managing your cookbooks, Spiceweasel supports adding `berksfile:` and the ability to specify the path and any Berkshelf options. You may mix use of `berkshelf:` with `cookbooks:` if you wish as well. Berkshelf-managed cookbooks will be included in the validation of roles, environments and run lists as well. You may simply use the YAML snippet
+If you prefer to use Berkshelf for managing your cookbooks, Spiceweasel supports adding `berksfile:` and the ability to specify the path and any Berkshelf options. You may mix use of `berksfile:` with `cookbooks:` if you wish as well. Berkshelf-managed cookbooks will be included in the validation of roles, environments and run lists as well. You may simply use the YAML snippet
 
 ``` yaml
-berkshelf:
+berksfile:
 ```
 
 which produces the command
@@ -94,16 +94,16 @@ berks upload --skip_syntax_check --config some_config.json -b /Users/mray/ws/lab
 
 ## Environments ##
 
-The `environments` section of the manifest currently supports `knife environment from file FOO` where `FOO` is the name of the environment file ending in `.rb` or `.json` in the `environments` directory. Validation is done to ensure the filename matches the environment and that any cookbooks referenced are listed in the manifest. The example YAML snippet
+The `environments` section of the manifest currently supports `knife environment from file FOO` where `FOO` is the name of the environment file ending in `.rb` or `.json` in the `environments` directory. You may also pass a wildcard as an entry to load all matching environments (ie. `"*"` or `prod*"`). Validation is done to ensure the filename matches the environment and that any cookbooks referenced are listed in the manifest. The example YAML snippet
 
 ``` yaml
 environments:
 - development:
 - qa:
-- production:
+- "prod*":
 ```
 
-produces the knife commands
+assuming the `production` environment exists, produces the knife commands
 
 ```
 knife environment from file development.rb qa.rb production.rb
@@ -111,25 +111,26 @@ knife environment from file development.rb qa.rb production.rb
 
 ## Roles ##
 
-The `roles` section of the manifest currently supports `knife role from file FOO` where `FOO` is the name of the role file ending in `.rb` or `.json` in the `roles` directory. Validation is done to ensure the filename matches the role name and that any cookbooks or roles referenced are listed in the manifest. The example YAML snippet
+The `roles` section of the manifest currently supports `knife role from file FOO` where `FOO` is the name of the role file ending in `.rb` or `.json` in the `roles` directory. You may also pass a wildcard as an entry to load all matching roles (ie. `"*"` or `data*"`). Validation is done to ensure the filename matches the role name and that any cookbooks or roles referenced are listed in the manifest. The example YAML snippet
 
 ``` yaml
 roles:
 - base:
+- "data*":
 - iisserver:
 - monitoring:
 - webserver:
 ```
 
-produces the knife commands
+assuming the `database1.json` and `database2.json` roles exist, this produces the knife commands
 
 ```
-knife role from file base.rb iisserver.rb monitoring.rb webserver.rb
+knife role from file base.rb database1.json database2.json iisserver.rb monitoring.rb webserver.rb
 ```
 
-n## Data Bags ##
+## Data Bags ##
 
-The `data bags` section of the manifest currently creates the data bags listed with `knife data bag create FOO` where `FOO` is the name of the data bag. Individual items may be added to the data bag as part of a JSON or YAML sequence, the assumption is made that they `.json` files and in the proper `data_bags/FOO` directory. You may also pass a wildcard as an entry to load all matching data bags (ie. `"*"`). Encrypted data bags are supported by using the `secret: secret_key_filename` attribute. Validation is done to ensure the JSON is properly formatted, the id matches and any secret key files are in the correct locations. Assuming the presence of `dataA.json` and `dataB.json` in the `data_bags/data` directory, the YAML snippet
+The `data bags` section of the manifest currently creates the data bags listed with `knife data bag create FOO` where `FOO` is the name of the data bag. Individual items may be added to the data bag as part of a JSON or YAML sequence, the assumption is made that they `.json` files and in the proper `data_bags/FOO` directory. You may also pass a wildcard as an entry to load all matching data bags (ie. `"*"` or `"data*"`). Encrypted data bags are supported by using the `secret: secret_key_filename` attribute. Validation is done to ensure the JSON is properly formatted, the id matches and any secret key files are in the correct locations. Assuming the presence of `dataA.json` and `dataB.json` in the `data_bags/data` directory, the YAML snippet
 
 ``` yaml
 data bags:
@@ -154,7 +155,7 @@ produces the knife commands
 knife data bag create users
 knife data bag from file users alice.json bob.json chuck.json
 knife data bag create data
-knife data bag from file data *.json
+knife data bag from file data dataA.json dataB.json
 knife data bag create passwords
 knife data bag from file passwords mysql.json rabbitmq.json --secret-file secret_key_filename
 ```
@@ -205,7 +206,7 @@ nodes:
   - -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -N webserver{{n}}
 ```
 
-produces the following:
+produces the following
 
 ```
 seq 3 | parallel -j 0 -v "knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -N webserver{} -r 'role[base],role[tc],role[users]'"
@@ -215,7 +216,7 @@ which generates nodes named "webserver1", "webserver2" and "webserver3".
 
 ## Clusters ##
 
-Clusters are not a type supported by Chef, this is a logical construct added by Spiceweasel to enable managing sets of infrastructure together. The `clusters` section is a special case of `nodes`, where each member of the named cluster in the manifest will be tagged to ensure that the entire cluster may be created in sync (refresh and delete coming soon). The node syntax is the same as that under `nodes`, the only addition is the cluster name.
+Clusters are not a type supported by Chef, this is a logical construct added by Spiceweasel to enable managing sets of infrastructure together. The `clusters` section is a special case of `nodes`, where each member of the named cluster in the manifest will be put in the same environment to ensure that the entire cluster may be created in sync (refresh and delete coming soon). The node syntax is the same as that under `nodes`, the only addition is the cluster name.
 
 ```
 clusters:
@@ -231,17 +232,17 @@ clusters:
 produces the knife commands
 
 ```
-knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-8af0f326 -f m1.medium -j '{"tags":["amazon+rolemysql"]}' -r 'role[mysql]'
-knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -j '{"tags":["amazon+rolewebserverrecipemysqlclient"]}' -r 'role[webserver],recipe[mysql::client]'
-knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -j '{"tags":["amazon+rolewebserverrecipemysqlclient"]}' -r 'role[webserver],recipe[mysql::client]'
-knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -j '{"tags":["amazon+rolewebserverrecipemysqlclient"]}' -r 'role[webserver],recipe[mysql::client]'
+knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-8af0f326 -f m1.medium -E amazon
+knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'
+knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'
+knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'
 ```
 
 Another use of `clusters` is with the `--cluster-file` option, which will allow the use of a different file to define the members of the cluster. If there are any `nodes` or `clusters` defined in the primary manifest file, they will be removed and the content of the `--cluster-file` will be used instead. This allows you to switch the target destination of infrastructure by picking different `--cluster-file` endpoints.
 
 # Extract #
 
-Spiceweasel may also be used to generate knife commands or Spiceweasel manifests in JSON or YAML.
+Spiceweasel may be used to generate knife commands or Spiceweasel manifests in JSON or YAML from an existing Chef repository.
 
 ```
 spiceweasel --extractlocal
@@ -263,13 +264,13 @@ When run in your Chef repository generates YAML-formatted output that may be cap
 To parse a Spiceweasel manifest, run the following from your Chef repository directory:
 
 ```
-spiceweasel path/to/infrastructure.json
+spiceweasel path/to/infrastructure.yml
 ```
 
-or
+or to extract infrastructure
 
 ```
-spiceweasel path/to/infrastructure.yml
+spiceweasel --extractlocal
 ```
 
 This will generate the knife commands to build the described infrastructure. Infrastructure manifest files must end in either `.json` or `.yml`.
@@ -336,7 +337,7 @@ Print the version of spiceweasel currently installed.
 
 Author: Matt Ray <matt@opscode.com>
 
-Copyright: 2011-2012 Opscode, Inc
+Copyright: 2011-2013 Opscode, Inc
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

data/lib/spiceweasel/cli.rb

@@ -88,7 +88,7 @@ module Spiceweasel
 
     option :log_level,
     :short => "-l LEVEL",
-    :long => "--log_level LEVEL",
+    :long => "--loglevel LEVEL",
     :description => "Set the log level (debug, info, warn, error, fatal)",
     :proc => lambda { |l| l.to_sym }
 
@@ -132,6 +132,16 @@ module Spiceweasel
     :proc => lambda { |v| puts "Spiceweasel: #{::Spiceweasel::VERSION}" },
     :exit => 0
 
+    option :cookbook_directory,
+    :short => '-C COOKBOOK_DIR',
+    :long => '--cookbook-dir COOKBOOK_DIR',
+    :description => 'Set cookbook directory. Specify multiple times for multiple directories.',
+    :proc => lambda { |v|
+      Spiceweasel::Config[:cookbook_dir] ||= []
+      Spiceweasel::Config[:cookbook_dir] << v
+      Spiceweasel::Config[:cookbook_dir].uniq!
+    }
+
     option :unique_id,
     :long => '--unique-id UID',
     :description => 'Unique ID generally used for ruby based configs'
@@ -149,24 +159,7 @@ module Spiceweasel
       end
       Spiceweasel::Log.debug("file manifest: #{manifest}")
 
-      berksfile = Berksfile.new(manifest['berksfile']) if manifest.include?('berksfile')
-      if berksfile
-        cookbooks = Cookbooks.new(manifest['cookbooks'], berksfile.cookbook_list)
-        create = berksfile.create + cookbooks.create
-        delete = berksfile.delete + cookbooks.delete
-      else
-        cookbooks = Cookbooks.new(manifest['cookbooks'])
-        create = cookbooks.create
-        delete = cookbooks.delete
-      end
-      environments = Environments.new(manifest['environments'], cookbooks)
-      roles = Roles.new(manifest['roles'], environments, cookbooks)
-      data_bags = DataBags.new(manifest['data bags'])
-      nodes = Nodes.new(manifest['nodes'], cookbooks, environments, roles)
-      clusters = Clusters.new(manifest['clusters'], cookbooks, environments, roles)
-
-      create += environments.create + roles.create + data_bags.create + nodes.create + clusters.create
-      delete += environments.delete + roles.delete + data_bags.delete + nodes.delete + clusters.delete
+      create, delete = process_manifest(manifest)
 
       if Spiceweasel::Config[:extractjson]
         puts JSON.pretty_generate(manifest)
@@ -208,12 +201,23 @@ module Spiceweasel
       ARGV << "-h" if ARGV.empty?
       begin
         parse_options
+        # Load knife configuration if using knife config
+        require 'chef/knife'
+        knife = Chef::Knife.new
+        # Only log on error during startup
+        Chef::Config[:verbosity] = 0
+        Chef::Config[:log_level] = :error
         if @config[:knifeconfig]
+          knife.read_config_file(@config[:knifeconfig])
           Spiceweasel::Config[:knife_options] = " -c #{@config[:knifeconfig]} "
+        else
+          knife.configure_chef
         end
         if @config[:serverurl]
           Spiceweasel::Config[:knife_options] += "--server-url #{@config[:serverurl]} "
         end
+        # NOTE: Only set cookbook path via config if path unset
+        Spiceweasel::Config[:cookbook_dir] ||= Chef::Config[:cookbook_path]
       rescue OptionParser::InvalidOption => e
         STDERR.puts e.message
         puts opt_parser.to_s
@@ -222,9 +226,11 @@ module Spiceweasel
     end
 
     def configure_logging
-      Spiceweasel::Log.init(Spiceweasel::Config[:log_location])
-      Spiceweasel::Log.level = Spiceweasel::Config[:log_level]
-      Spiceweasel::Log.level = :debug if Spiceweasel::Config[:debug]
+      [Spiceweasel::Log, Chef::Log].each do |log_klass|
+        log_klass.init(Spiceweasel::Config[:log_location])
+        log_klass.level = Spiceweasel::Config[:log_level]
+        log_klass.level = :debug if Spiceweasel::Config[:debug]
+      end
     end
 
     def parse_and_validate_input(file)
@@ -255,12 +261,35 @@ module Spiceweasel
         exit(-1)
       rescue Exception => e
         STDERR.puts "ERROR: Invalid or missing  manifest .json, .rb, or .yml file provided."
-        STDERR.puts "ERROR: #{e}"
-        puts opt_parser.to_s
+        if(Spiceweasel::Config[:log_level].to_s == 'debug')
+          STDERR.puts "ERROR: #{e}\n#{e.backtrace.join("\n")}"
+        end
         exit(-1)
       end
       output
     end
 
+    def process_manifest(manifest)
+      berksfile = Berksfile.new(manifest['berksfile']) if manifest.include?('berksfile')
+      if berksfile
+        cookbooks = Cookbooks.new(manifest['cookbooks'], berksfile.cookbook_list)
+        create = berksfile.create + cookbooks.create
+        delete = berksfile.delete + cookbooks.delete
+      else
+        cookbooks = Cookbooks.new(manifest['cookbooks'])
+        create = cookbooks.create
+        delete = cookbooks.delete
+      end
+      environments = Environments.new(manifest['environments'], cookbooks)
+      roles = Roles.new(manifest['roles'], environments, cookbooks)
+      data_bags = DataBags.new(manifest['data bags'])
+      nodes = Nodes.new(manifest['nodes'], cookbooks, environments, roles)
+      clusters = Clusters.new(manifest['clusters'], cookbooks, environments, roles)
+
+      create += environments.create + roles.create + data_bags.create + nodes.create + clusters.create
+      delete += environments.delete + roles.delete + data_bags.delete + nodes.delete + clusters.delete
+      return create, delete
+    end
+
   end
 end

data/lib/spiceweasel/clusters.rb

@@ -34,13 +34,19 @@ module Spiceweasel
             node_name = node.keys.first
             run_list = node[node_name]['run_list'] || ''
             options = node[node_name]['options'] || ''
-            # cluster tag is the cluster name + runlist
-            tag = " -j '{\"tags\":[\"#{cluster_name}+#{run_list.gsub(/[ ,\[\]:]/, '')}\"]}'"
-            Spiceweasel::Log.debug("cluster: #{cluster_name}:#{node_name}:tag:#{tag}")
-            #push the tag back on the options
-            node[node_name]['options'] = options + tag
+            # cluster tag is the cluster name + runlist once tags are working for every plugin
+            # until then, we're going to override the Environment
+            if options =~ /-E/ #delete any Environment
+              env = options.split('-E')[1].split[0]
+              edel = "-E#{env}"
+              options[edel] = "" if options.include?(edel)
+              edel = "-E #{env}"
+              options[edel] = "" if options.include?(edel)
+              Spiceweasel::Log.warn("deleting specified Environment '#{env}' from cluster: '#{cluster_name}'")
+            end
+            #push the Environment back on the options
+            node[node_name]['options'] = options + " -E #{cluster_name}"
           end
-          Spiceweasel::Log.debug("cluster2: '#{cluster_name}' '#{cluster[cluster_name]}'")
           # let's reuse the Nodes logic
           nodes = Spiceweasel::Nodes.new(cluster[cluster_name], cookbooks, environments, roles)
           @create.concat(nodes.create)

data/lib/spiceweasel/cookbooks.rb

@@ -32,6 +32,8 @@ module Spiceweasel
       @dependencies = Array.new
       #validate each of the cookbooks specified in the manifest
       if cookbooks
+        @loader = Chef::CookbookLoader.new(Spiceweasel::Config[:cookbook_dir])
+        @loader.load_cookbooks
         Spiceweasel::Log.debug("cookbooks: #{cookbooks}")
 
         c_names = []
@@ -43,7 +45,8 @@ module Spiceweasel
           end
           Spiceweasel::Log.debug("cookbook: #{name} #{version} #{options}")
           if File.directory?("cookbooks")
-            if File.directory?("cookbooks/#{name}") #TODO use the name from metadata
+
+            if @loader.cookbooks_by_name[name]
               validateMetadata(name,version) unless Spiceweasel::Config[:novalidation]
             else
               if Spiceweasel::Config[:siteinstall] #use knife cookbook site install
@@ -81,9 +84,7 @@ module Spiceweasel
     #check the metadata for versions and gather deps
     def validateMetadata(cookbook,version)
       #check metadata.rb for requested version
-      metadata_file = "cookbooks/#{cookbook}/metadata.rb"
-      metadata = Chef::Cookbook::Metadata.new
-      metadata.from_file(metadata_file)
+      metadata = @loader.cookbooks_by_name[cookbook].metadata
       Spiceweasel::Log.debug("validateMetadata: #{cookbook} #{metadata.name} #{metadata.version}")
       # Should the cookbook directory match the name in the metadata?
       if metadata.name.empty?

data/lib/spiceweasel/data_bags.rb

@@ -1,7 +1,7 @@
 #
 # Author:: Matt Ray (<matt@opscode.com>)
 #
-# Copyright:: 2011-2012, Opscode, Inc <legal@opscode.com>
+# Copyright:: 2011-2013, Opscode, Inc <legal@opscode.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -56,17 +56,23 @@ module Spiceweasel
           items.each do |item|
             Spiceweasel::Log.debug("data bag #{db} item: #{item}")
             if item =~ /\*/ #wildcard support, will fail if directory not present
-              files = Dir.glob("data_bags/#{db}/*.json")
-              items.concat(files.collect {|x| x[x.rindex('/')+1..-6]})
-              Spiceweasel::Log.debug("found items '#{items}' for data bag: #{db}")
+              files = Dir.glob("data_bags/#{db}/#{item}")
+              #remove anything not ending in .json
+              files.delete_if {|x| !x.end_with?('.json')}
+              items.concat(files.collect {|x| x["data_bags/#{db}/".length..-6]})
+              Spiceweasel::Log.debug("found files '#{files}' for data bag: #{db} with wildcard #{item}")
               next
             end
             validateItem(db, item) unless Spiceweasel::Config[:novalidation]
           end
-          if secret
-            create_command("knife data bag#{Spiceweasel::Config[:knife_options]} from file #{db} #{items.join('.json ')}.json --secret-file #{secret}")
-          else
-            create_command("knife data bag#{Spiceweasel::Config[:knife_options]} from file #{db} #{items.join('.json ')}.json")
+          items.delete_if {|x| x.include?("*")} #remove wildcards
+          items.sort!.uniq!
+          unless items.empty?
+            if secret
+              create_command("knife data bag#{Spiceweasel::Config[:knife_options]} from file #{db} #{items.join('.json ')}.json --secret-file #{secret}")
+            else
+              create_command("knife data bag#{Spiceweasel::Config[:knife_options]} from file #{db} #{items.join('.json ')}.json")
+            end
           end
         end
       end
@@ -81,6 +87,9 @@ module Spiceweasel
       f = File.read("data_bags/#{db}/#{item}.json")
       itemfile = JSON.parse(f) #invalid JSON will throw a trace
       #validate the id matches the file name
+      if item =~ /\// #pull out directories
+        item = item.split('/').last
+      end
       if !item.eql?(itemfile['id'])
         STDERR.puts "ERROR: data bag '#{db}' item '#{item}' listed in the manifest does not match the id '#{itemfile['id']}' within the 'data_bags/#{db}/#{item}.json' file."
         exit(-1)

data/lib/spiceweasel/environments.rb

@@ -31,31 +31,43 @@ module Spiceweasel
       @environment_list = Array.new
       if environments
         Spiceweasel::Log.debug("environments: #{environments}")
+        flatenvs = environments.collect {|x| x.keys}.flatten
         envfiles = []
-        environments.each do |env|
-          name = env.keys[0]
-          Spiceweasel::Log.debug("environment: #{name}")
+        flatenvs.each do |env|
+          Spiceweasel::Log.debug("environment: #{env}")
           if File.directory?("environments")
-            validate(name, cookbooks) unless Spiceweasel::Config[:novalidation]
+            #expand wildcards and push into environments
+            if env =~ /\*/ #wildcard support
+              wildenvs = Dir.glob("environments/#{env}")
+              #remove anything not ending in .json or .rb
+              wildenvs.delete_if {|x| !x.end_with?(".rb", ".json")}
+              Spiceweasel::Log.debug("found environments '#{wildenvs}' for wildcard: #{env}")
+              flatenvs.concat(wildenvs.collect {|x| x[x.rindex('/')+1..x.rindex('.')-1]})
+              next
+            end
+            validate(env, cookbooks) unless Spiceweasel::Config[:novalidation]
           elsif !Spiceweasel::Config[:novalidation]
             STDERR.puts "'environments' directory not found, unable to validate or load environments"
             exit(-1)
           end
-          if File.exists?("environments/#{name}.json")
-            envfiles.push("#{name}.json")
+          if File.exists?("environments/#{env}.json")
+            envfiles.push("#{env}.json")
           else #assume no .json means they want .rb and catchall for misssing dir
-            envfiles.push("#{name}.rb")
+            envfiles.push("#{env}.rb")
           end
-          delete_command("knife environment#{Spiceweasel::Config[:knife_options]} delete #{name} -y")
-          @environment_list.push(name)
+          delete_command("knife environment#{Spiceweasel::Config[:knife_options]} delete #{env} -y")
+          @environment_list.push(env)
         end
-        create_command("knife environment#{Spiceweasel::Config[:knife_options]} from file #{envfiles.join(' ')}")
+        create_command("knife environment#{Spiceweasel::Config[:knife_options]} from file #{envfiles.uniq.sort.join(' ')}")
       end
     end
 
     #validate the content of the environment file
     def validate(environment, cookbooks)
       file = %W(environments/#{environment}.rb environments/#{environment}.json).detect{|f| File.exists?(f)}
+      if environment =~ /\// #pull out directories
+        environment = environment.split('/').last
+      end
       if file
         case file
         when /\.json$/

data/lib/spiceweasel/extract_local.rb

@@ -89,7 +89,7 @@ module Spiceweasel
 
     def self.resolve_cookbooks(berkshelf_cookbooks = {})
       require 'solve'
-      loader = Chef::CookbookLoader.new('./cookbooks')
+      loader = Chef::CookbookLoader.new(Spiceweasel::Config[:cookbook_dir])
       loader.load_cookbooks
       books = loader.cookbooks_by_name
       graph = Solve::Graph.new

data/lib/spiceweasel/roles.rb

@@ -37,6 +37,15 @@ module Spiceweasel
         flatroles.each do |role|
           Spiceweasel::Log.debug("role: #{role}")
           if File.directory?("roles")
+            #expand wildcards and push into flatroles
+            if role =~ /\*/ #wildcard support
+              wildroles = Dir.glob("roles/#{role}")
+              #remove anything not ending in .json or .rb
+              wildroles.delete_if {|x| !x.end_with?(".rb", ".json")}
+              Spiceweasel::Log.debug("found roles '#{wildroles}' for wildcard: #{role}")
+              flatroles.concat(wildroles.collect {|x| x[x.rindex('/')+1..x.rindex('.')-1]})
+              next
+            end
             validate(role, environments, cookbooks, flatroles) unless Spiceweasel::Config[:novalidation]
           elsif !Spiceweasel::Config[:novalidation]
             STDERR.puts "ERROR: 'roles' directory not found, unable to validate or load roles"
@@ -50,7 +59,7 @@ module Spiceweasel
           delete_command("knife role#{Spiceweasel::Config[:knife_options]} delete #{role} -y")
           @role_list.push(role)
         end
-        create_command("knife role#{Spiceweasel::Config[:knife_options]} from file #{rolefiles.join(' ')}")
+        create_command("knife role#{Spiceweasel::Config[:knife_options]} from file #{rolefiles.uniq.sort.join(' ')}")
       end
     end
 
@@ -58,7 +67,10 @@ module Spiceweasel
     def validate(role, environments, cookbooks, roles)
       #validate the role passed in match the name of either the .rb or .json
       file = %W(roles/#{role}.rb roles/#{role}.json).detect{|f| File.exists?(f)}
-      if(file)
+      if role =~ /\// #pull out directories
+        role = role.split('/').last
+      end
+      if file
         case file
         when /\.json$/
           c_role = Chef::JSONCompat.from_json(IO.read(file))

data/lib/spiceweasel/version.rb

@@ -17,5 +17,5 @@
 #
 
 module Spiceweasel
-  VERSION = '2.1.2'
+  VERSION = '2.2.0'
 end

data/spec/bin/spiceweasel_spec.rb

@@ -36,12 +36,11 @@ knife node bulk delete .* -y
 knife cookbook upload apache2
 knife cookbook upload apt --freeze
 knife cookbook upload mysql ntp
-knife environment from file development.rb qa.rb production.rb
+knife environment from file development.rb production.rb qa.rb
 knife role from file base.rb iisserver.rb monitoring.rb webserver.rb
 knife data bag create users
 knife data bag from file users alice.json bob.json chuck.json
 knife data bag create data
-knife data bag from file data *.json
 knife data bag create passwords
 knife data bag from file passwords mysql.json rabbitmq.json --secret-file secret_key
 knife bootstrap serverA -i ~/.ssh/mray.pem -x user --sudo -r 'role[base]'
@@ -53,10 +52,10 @@ knife rackspace server create --image 49 --flavor 2 -N db3 -r 'recipe[mysql],rol
 knife bootstrap windows winrm winboxA -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'
 knife bootstrap windows ssh winboxB -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'
 knife bootstrap windows ssh winboxC -x Administrator -P 'super_secret_password' -r 'role[base],role[iisserver]'
-knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-8af0f326 -f m1.medium -j '{"tags":["amazon+rolemysql"]}' -r 'role[mysql]'
-knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -j '{"tags":["amazon+rolewebserverrecipemysqlclient"]}' -r 'role[webserver],recipe[mysql::client]'
-knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -j '{"tags":["amazon+rolewebserverrecipemysqlclient"]}' -r 'role[webserver],recipe[mysql::client]'
-knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -j '{"tags":["amazon+rolewebserverrecipemysqlclient"]}' -r 'role[webserver],recipe[mysql::client]'
+knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-8af0f326 -f m1.medium -E amazon -r 'role[mysql]'
+knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'
+knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'
+knife ec2 server create -S mray -i ~/.ssh/mray.pem -x ubuntu -G default -I ami-7000f019 -f m1.small -E amazon -r 'role[webserver],recipe[mysql::client]'
     OUTPUT
 
     @spiceweasel_binary = File.join(File.dirname(__FILE__), *%w[.. .. bin spiceweasel])

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: spiceweasel
 version: !ruby/object:Gem::Version
-  version: 2.1.2
+  version: 2.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-04-05 00:00:00.000000000 Z
+date: 2013-05-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: json