spiceweasel 0.1.0 → 0.3

data/{MILESTONES.md → CHANGELOG.md}

@@ -10,42 +10,31 @@ This is the current, previous and future development milestones and contains the
 
 0.2
 ===
-* new CLI args
-* --chef-client The option `--chef-client` will make a `knife ssh` call to each box and run `chef-client` on each.
-* --delete The option `--delete` will delete each cookbook, role, data bag, environment and node described in the yml file. All nodes from the system are deleted with `knife node bulk_delete`. Since knife does not currently manage deletion of the instances from cloud providers, that is still a required step.
+* switch to mixlib-cli
+* --dryrun The option `--dryrun` will print the commands to run, but not actually execute them (the default behavior)
+* --delete The option `--delete` will delete each cookbook, role, data bag, environment and node described in the yml file. All nodes from the system are deleted with `knife node bulk_delete`.
 * --rebuild The option `--rebuild` will remove all currently managed infrastructure for this chef repository and rebuild it from scratch.
 
 0.3
 ===
-* --chef-client validation
+* renamed MILESTONES.md to CHANGELOG.md
+* fixed version number
+* updated YAML schema and examples because Ruby 1.8 does not order hashes.
+* validate that the recipes and roles listed in the nodes are loaded
+
+0.4
+===
+* --chef-client The option `--chef-client` will make a `knife ssh` call to each box and run `chef-client` on each.
+* --chef-client validation that nodes are added
 
 BACKLOG
 =======
-Cookbooks
----------
-* site vendor?
-* JSON?
-
-Roles
------
-* JSON?
-* validate recipes and roles have been uploaded
-
-Data Bags
----------
-
-Environments
-------------
-
-Nodes
------
-* Provider
-* on delete... what to do?
+* support site vendor for cookbooks
+* support versions for cookbooks
+* add support for environments
+* make the JSON calls directly with Spice (https://github.com/danryan/spice) 
+* -e/--execute execute the commands
+* catching return codes and retrying (with retry count?)
+* on provider delete take count of vendor-specific, delete if match (ec2 server delete and node delete)
 * knife windows bootstrap FQDN [RUN LIST...] (options)
-
-Wishlist
---------
-* use GNU parallel
-* make the JSON calls directly (and possibly parallelize)
-* --dryrun The option `--dryrun` will print the commands to run, but not actually execute them.
-
+* use GNU parallel with knife?

data/README.md

@@ -1,59 +1,59 @@
 Description
 ===========
-Spiceweasel is a command-line tool for batch loading Chef infrastructure. It provides a simple syntax for describing and deploying infrastructure with the Chef command-line tool `knife`.
+Spiceweasel is a command-line tool for batch loading Chef infrastructure. It provides a simple syntax for describing and deploying infrastructure in order with the Chef command-line tool `knife`.
 
-MILESTONES.md covers current, previous and future development milestones and contains the features backlog.
+CHANGELOG.md covers current, previous and future development milestones and contains the features backlog.
 
 Requirements
 ============
 Spiceweasel currently depends on `knife` to run commands for it.
 
-Written with Chef 0.9.12 and supports cookbooks, recipes, roles, data bags and nodes. Support for environments will be added with the Chef 0.10 release.
+Written with Chef 0.9.12 and 0.9.14 and supports cookbooks, recipes, roles, data bags and nodes. Support for environments will be added with the Chef 0.10 release.
 
 Testing
 -------
-Tested with Ubuntu 10.04 and 10.10 and Chef 0.9.12.
+Tested with Ubuntu 10.04 and 10.10 and Chef 0.9.12 and 0.9.14.
 
 File Syntax
 ===========
 The syntax for the spiceweasel file is a simple YAML format of Chef primitives describing what is to be instantiated. 
 
     cookbooks:
-      - apache2
-      - apt
-      - mysql
+    - apache2:
+    - apt:
+    - mysql:
 
     roles:
-      - base
-      - monitoring
-      - webserver
+    - base:
+    - monitoring:
+    - webserver:
 
     data bags:
-      users:
-        - alice
-        - bob
-        - chuck
-      data:
+    - users:
+      - alice
+      - bob
+      - chuck
+    - data:
 
     nodes:
-      serverA:
-        - role[loadbalancer]
-        - -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
-      ec2 5:
-        - role[webserver]
-        - -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
-      rackspace 3:
-        - recipe[mysql] role[clustered_mysql]
-        - --image 49 --flavor 2
+    - serverA:
+      - role[base]
+      - -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
+    - ec2 5:
+      - role[webserver] recipe[mysql::client]
+      - -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
+    - rackspace 3:
+      - recipe[mysql] role[monitoring]
+      - --image 49 --flavor 2
 
 Cookbooks
 ---------
 The `cookbooks` section of the YAML file currently supports `knife cookbook upload FOO` where `FOO` is the name of the cookbook in the `cookbooks` directory. The YAML snippet
 
     cookbooks:
-      - apache2
-      - apt
-      - mysql
+    - apache2:
+    - apt:
+    - mysql:
 
 produces the knife commands
 
@@ -66,9 +66,9 @@ Roles
 The `roles` section of the YAML file currently supports `knife role from file FOO` where `FOO` is the name of the role file ending in `.rb` in the `roles` directory. The YAML snippet 
 
     roles:
-      - base
-      - monitoring
-      - webserver
+    - base:
+    - monitoring:
+    - webserver:
 
 produces the knife commands 
 
@@ -81,46 +81,46 @@ Data Bags
 The `data bags` section of the YAML file currently creates the data bags listed with `knife data bag create FOO` where `FOO` is the name of the data bag. Individual items may be added to the data bag as part of a YAML sequence, the assumption is made that they `.json` files and in the `data_bags` directory. The YAML snippet 
 
     data bags:
-      users:
-        - alice
-        - bob
-        - chuck
-      data:
+    - users:
+      - alice
+      - bob
+      - chuck
+    - data:
 
 produces the knife commands 
 
-    knife data bag create data
     knife data bag create users
     knife data bag from file users data_bags/alice.json
     knife data bag from file users data_bags/bob.json
     knife data bag from file users data_bags/chuck.json
+    knife data bag create data
 
 Nodes
 -----
-The `nodes` section of the YAML file bootstraps a node for each entry where the entry is a hostname or provider and count. Each node requires 2 items after it in a YAML sequence. The first item is the run_list and the second the CLI options used. A shortcut syntax for bulk-creating nodes with various providers where the line starts with the provider and ends with the number of nodes to be provisioned. The YAML snippet 
+The `nodes` section of the YAML file bootstraps a node for each entry where the entry is a hostname or provider and count. Each node requires 2 items after it in a YAML sequence. The first item is the run_list and the second the CLI options used. Validation is performed on the run_list components to ensure that only recipes and roles listed in the YAML file are used. A shortcut syntax for bulk-creating nodes with various providers where the line starts with the provider and ends with the number of nodes to be provisioned. The YAML snippet 
 
     nodes:
-      serverA:
-        - role[loadbalancer]
-        - -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
-      ec2 5:
-        - role[webserver]
-        - -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
-      rackspace 3:
-        - recipe[mysql] role[clustered_mysql]
-        - --image 49 --flavor 2
+    - serverA:
+      - role[base]
+      - -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
+    - ec2 5:
+      - role[webserver] recipe[mysql::client]
+      - -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
+    - rackspace 3:
+      - recipe[mysql] role[monitoring]
+      - --image 49 --flavor 2
 
 produces the knife commands 
 
-    knife bootstrap serverA 'role[loadbalancer]' -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
-    knife ec2 server create 'role[webserver]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
-    knife ec2 server create 'role[webserver]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
-    knife ec2 server create 'role[webserver]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
-    knife ec2 server create 'role[webserver]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
-    knife ec2 server create 'role[webserver]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
-    knife rackspace server create 'recipe[mysql]' 'role[clustered_mysql]' --image 49 --flavor 2
-    knife rackspace server create 'recipe[mysql]' 'role[clustered_mysql]' --image 49 --flavor 2
-    knife rackspace server create 'recipe[mysql]' 'role[clustered_mysql]' --image 49 --flavor 2
+    knife bootstrap serverA 'role[base]' -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
+    knife ec2 server create 'role[webserver]' 'recipe[mysql::client]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
+    knife ec2 server create 'role[webserver]' 'recipe[mysql::client]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
+    knife ec2 server create 'role[webserver]' 'recipe[mysql::client]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
+    knife ec2 server create 'role[webserver]' 'recipe[mysql::client]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
+    knife ec2 server create 'role[webserver]' 'recipe[mysql::client]' -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
+    knife rackspace server create 'recipe[mysql]' 'role[monitoring]' --image 49 --flavor 2
+    knife rackspace server create 'recipe[mysql]' 'role[monitoring]' --image 49 --flavor 2
+    knife rackspace server create 'recipe[mysql]' 'role[monitoring]' --image 49 --flavor 2
 
 Usage
 =====
@@ -130,6 +130,26 @@ To run a spiceweasel file, run the following from you Chef repository directory:
 
 This will generate the knife commands to build the described infrastructure. 
 
+--dryrun
+--------
+This is the default action, printing the knife commands to be run without executing them.
+
+-d/--delete
+-----------
+The delete command will generate the knife commands to delete the infrastructure described in the YAML file. This includes each cookbook, role, data bag, environment and node listed. Currently all nodes from the system are deleted with `knife node bulk_delete`, specific-node support will be added in a future release.
+
+-h/--help
+---------
+Print the currently-supported usage options for spiceweasel.
+
+-r/--rebuild
+---------
+The rebuild command will generate the knife commands to delete and recreate the infrastructure described in the YAML file. This includes each cookbook, role, data bag, environment and node listed. Currently all nodes from the system are deleted with `knife node bulk_delete`, specific-node support will be added in a future release.
+
+-v/--version
+------------
+Print the version of spiceweasel currently installed.
+
 License and Author
 ==================
 Author: Matt Ray <matt@opscode.com>

data/bin/spiceweasel

@@ -1,84 +1,169 @@
 #!/usr/bin/env ruby
 
-require 'optparse'
+require 'mixlib/cli'
 require 'yaml'
- 
-options = {}
-optparse = OptionParser.new do |opts|
-  opts.banner = "Usage: spiceweasel [options] file"
-  opts.on('-h', '--help', 'Show this message') do
-    puts opts
-    exit
-  end
-  opts.on('-v', '--version', 'Version') do 
-    puts "spiceweasel 0.1"
-    exit
+require 'spiceweasel/version'
+
+#ensure that the run_list contents are present in either the cookbooks or roles
+def validate_run_list(run_list, cookbooks, roles)
+  run_list.each do |item|
+    if item.start_with?("recipe")
+      #recipe[foo] or recipe[foo::bar]
+      recipe = item.slice(7..-2).split("::")[0]
+      unless cookbooks.member?(recipe)
+        raise "'#{item}' is an invalid run_list recipe not managed by spiceweasel"
+        exit(-1)
+      end
+    elsif item.start_with?("role")
+      #role[blah]
+      role = item.slice(5..-2)
+      unless roles.member?(role)
+        raise "'#{item}' is an invalid run_list role not managed by spiceweasel"
+        exit(-1)
+      end
+    else 
+      raise "'#{item}' is an invalid run_list component"
+      exit(-1)
+    end
   end
 end
 
+class SpiceweaselCLI
+  include Mixlib::CLI
+
+  banner("Usage: spiceweasel [option] file")
+
+  option :delete,
+  :short => "-d",
+  :long => "--delete",
+  :description => "Print the knife commands to be delete the infrastructure",
+  :boolean => true
+
+  option :dryrun,
+  :long => "--dryrun",
+  :description => "Print the knife commands to be executed to STDOUT",
+  :boolean => true
+
+  option :help,
+  :short => "-h",
+  :long => "--help",
+  :description => "Show this message",
+  :on => :tail,
+  :boolean => true,
+  :show_options => true,
+  :exit => 0
+
+  option :rebuild,
+  :short => "-r",
+  :long => "--rebuild",
+  :description => "Print the knife commands to be delete and recreate the infrastructure",
+  :boolean => true
+
+  option :version,
+  :short => "-v",
+  :long => "--version",
+  :description => "Version",
+  :boolean => true,
+  :proc => lambda {|v| puts "Spiceweasel: #{Spiceweasel::VERSION}" },
+  :exit => 0
+
+end
+
+#process command line options
 begin
   ARGV << "-h" if ARGV.empty? 
-  optparse.parse!(ARGV)
-rescue OptionParser::ParseError => e 
-  STDERR.puts e.message, "\n", optparse
+  cli = SpiceweaselCLI.new
+  cli.parse_options
+rescue OptionParser::InvalidOption => e 
+  STDERR.puts e.message
+  #puts???
   exit(-1)
 end
 
+begin
+  yml = YAML.load_file ARGV.last
+rescue Exception
+  STDERR.puts "No infrastructure YAML file provided."
+  #puts???
+  exit(-1)
+end
 
-yml = YAML.load_file ARGV[0]
-knife = ""
+create = String.new()
+delete = String.new()
 
 #cookbooks 
 cookbooks = yml['cookbooks'] || []
+cookbook_list = []
 cookbooks.each do |cookbook|
-  knife += "knife cookbook upload #{cookbook}\n"
+  delete += "knife cookbook delete #{cookbook.keys[0]} -y\n"
+  create += "knife cookbook upload #{cookbook.keys[0]}\n"
+  #flatten list of cookbooks for validation later
+  cookbook_list.push(cookbook.keys[0])
 end
 
 #roles 
 roles = yml['roles'] || []
+role_list = []
 roles.each do |role|
-  knife += "knife role from file #{role}.rb\n"
+  delete += "knife role delete #{role.keys[0]} -y\n"
+  create += "knife role from file #{role.keys[0]}.rb\n"
+  #flatten list of roles for validation later
+  role_list.push(role.keys[0])
 end
 
 #data bags
 bags = yml['data bags'] || []
-bags.keys.each do |bag|
-  knife += "knife data bag create #{bag}\n"
-  items = bags[bag] || []
+bags.each do |bag|
+  delete += "knife data bag delete #{bag.keys[0]} -y\n"
+  create += "knife data bag create #{bag.keys[0]}\n"
+  items = bag[bag.keys[0]] || []
   items.each do |item|
-    knife += "knife data bag from file #{bag} data_bags/#{item}.json\n"
+    create += "knife data bag from file #{bag.keys[0]} data_bags/#{item}.json\n"
   end
 end
 
 nodes = yml['nodes'] || []
-nodes.keys.each do |node|
-  run_list = nodes[node][0].split()
-  #TODO verify the run_list against the nodes and recipes
-  if node.start_with?("bluebox","ec2","rackspace","slicehost","terremark")
-    provider = node.split()
+#currently use bulk_delete for deleting, add provider support real soon
+delete += "knife node bulk_delete .* -y\n"
+nodes.each do |node|
+  # delete += "knife node delete #{node[node.keys[0]}\n"
+  run_list = node[node.keys[0]][0].split()
+  validate_run_list(run_list, cookbook_list, role_list)
+  if node.keys[0].start_with?("bluebox","ec2","rackspace","slicehost","terremark")
+    provider = node.keys[0].split()
     count = 1
     if (provider.length == 2)
       count = provider[1] 
     end
     count.to_i.times do
-      knife += "knife #{provider[0]} server create "
+      create += "knife #{provider[0]} server create "
       run_list.each do |x|
-        knife += "\'#{x}\' "
+        create += "\'#{x}\' "
       end
-      knife +="#{nodes[node][1]}\n"
+      create +="#{node[node.keys[0]][1]}\n"
     end
   else
-    knife += "knife bootstrap #{node} "
+    create += "knife bootstrap #{node.keys[0]} "
     run_list.each do |x|
-      knife += "\'#{x}\' "
+      create += "\'#{x}\' "
     end
-    knife +="#{nodes[node][1]}\n"
+    create +="#{node[node.keys[0]][1]}\n"
   end
 end
 
-puts knife
-#--clientsync
-#--delete
-#--dryrun 
-#--rebuild
-#remember to -y
+#just print the knife commands, do not execute
+#if cli.config[:dryrun]
+if cli.config[:delete] 
+  puts delete unless delete.empty?
+elsif cli.config[:rebuild]
+  puts delete unless delete.empty?
+  puts create unless create.empty?
+else
+  puts create unless create.empty?
+end       
+#else
+#eventually we will execute instead of printing knife commands
+#puts "BAM!"
+#end
+
+

data/example.yml

@@ -1,27 +1,27 @@
 cookbooks:
-  - apache2
-  - apt
-  - mysql
+- apache2:
+- apt:
+- mysql:
 
 roles:
-  - base
-  - monitoring
-  - webserver
+- base:
+- monitoring:
+- webserver:
 
 data bags:
-  users:
-    - alice
-    - bob
-    - chuck
-  data:
+- users:
+  - alice
+  - bob
+  - chuck
+- data:
 
 nodes:
-  serverA:
-    - role[loadbalancer]
-    - -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
-  ec2 5:
-    - role[webserver]
-    - -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
-  rackspace 3:
-    - recipe[mysql] role[clustered_mysql]
-    - --image 49 --flavor 2
+- serverA:
+  - role[base]
+  - -i ~/.ssh/mray.pem -x user --sudo -d ubuntu10.04-gems
+- ec2 5:
+  - role[webserver] recipe[mysql::client]
+  - -S mray -I ~/.ssh/mray.pem -x ubuntu -G default -i ami-a403f7cd -f m1.small
+- rackspace 3:
+  - recipe[mysql] role[monitoring]
+  - --image 49 --flavor 2

data/lib/spiceweasel/version.rb

@@ -1,3 +1,3 @@
 module Spiceweasel
-  VERSION = "0.1.0"
+  VERSION = "0.3"
 end

data/test.sh

@@ -0,0 +1,48 @@
+echo "------------>spiceweasel"
+spiceweasel
+echo "------------>Return code:$?"
+echo "------------>spiceweasel -h"
+spiceweasel -h
+echo "------------>Return code:$?"
+echo "------------>spiceweasel -h example.yml"
+spiceweasel -h example.yml
+echo "------------>Return code:$?"
+echo "------------>spiceweasel -v"
+spiceweasel -v
+echo "------------>Return code:$?"
+echo "------------>spiceweasel -v example.yml"
+spiceweasel -v example.yml
+echo "------------>Return code:$?"
+echo "------------>spiceweasel --version"
+spiceweasel --version
+echo "------------>Return code:$?"
+echo "------------>spiceweasel example.yml"
+spiceweasel example.yml
+echo "------------>Return code:$?"
+echo "------------>spiceweasel --dryrun"
+spiceweasel --dryrun
+echo "------------>Return code:$?"
+echo "------------>spiceweasel --dryrun example.yml"
+spiceweasel --dryrun example.yml
+echo "------------>Return code:$?"
+echo "------------>spiceweasel --help"
+spiceweasel --help
+echo "------------>Return code:$?"
+echo "------------>spiceweasel -d"
+spiceweasel -d
+echo "------------>Return code:$?"
+echo "------------>spiceweasel -d example.yml"
+spiceweasel -d example.yml
+echo "------------>Return code:$?"
+echo "------------>spiceweasel --delete example.yml"
+spiceweasel --delete example.yml
+echo "------------>Return code:$?"
+echo "------------>spiceweasel -r"
+spiceweasel -r
+echo "------------>Return code:$?"
+echo "------------>spiceweasel -r example.yml"
+spiceweasel -r example.yml
+echo "------------>Return code:$?"
+echo "------------>spiceweasel --rebuild example.yml"
+spiceweasel --rebuild example.yml
+echo "------------>Return code:$?"

metadata

@@ -1,12 +1,12 @@
 --- !ruby/object:Gem::Specification 
 name: spiceweasel
 version: !ruby/object:Gem::Version 
+  hash: 13
   prerelease: false
   segments: 
   - 0
-  - 1
-  - 0
-  version: 0.1.0
+  - 3
+  version: "0.3"
 platform: ruby
 authors: 
 - Matt Ray
@@ -14,7 +14,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-03-04 00:00:00 -06:00
+date: 2011-03-07 00:00:00 -06:00
 default_executable: 
 dependencies: []
 
@@ -29,8 +29,8 @@ extra_rdoc_files: []
 
 files: 
 - .gitignore
+- CHANGELOG.md
 - Gemfile
-- MILESTONES.md
 - README.md
 - Rakefile
 - bin/spiceweasel
@@ -38,6 +38,7 @@ files:
 - lib/spiceweasel.rb
 - lib/spiceweasel/version.rb
 - spiceweasel.gemspec
+- test.sh
 has_rdoc: true
 homepage: ""
 licenses: []
@@ -52,6 +53,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
+      hash: 3
       segments: 
       - 0
       version: "0"
@@ -60,6 +62,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
+      hash: 3
       segments: 
       - 0
       version: "0"