cluster_chef-knife 3.0.12 → 3.0.14

data/CHANGELOG.md

@@ -1,3 +1,27 @@
+## v3.0.14: 
+
+Big important change:
+
+# ClusterChef is now Ironfan
+
+Due to a polite request from outside, we are changing this project's name to not include the word 'Chef'.
+
+It's now 'Ironfan', after Princess Iron Fan from the legend of Sun Wukong (Voyage to the West). The monkey hero Sun Wukong could not reach his destination without the help of the Princess's Iron Fan. The project helps you fan out across big iron in the sky -- and "You Can't Do B.I. without a Big Ironfan".
+
+This weekend (2/18) everything in the cluster_chef family of everything will get regex-replaced as ironfan. We'll track both gems for the next push or so, but new versions of the gem will not be released after Feb 2012.
+
+Other improvements/fixes:
+
+* knife cluster proxy now accepts additional hosts to direct to proxy -- set Chef::Config[:cluster_proxy_patterns] to an array of additional glob-style strings. Use this with the route53 support for fun and profit - if your machines look like foo-server-0-internal.whatupchimpy.com, add Chef::Config[:cluster_proxy_patterns] = '*internal.whatupchimpy.com*' to your knife-org.rb -- now you can browse securely to the private interface of any machine your target can.
+* FIX #76 -- `knife cluster kick` runs chef-client if the service is stopped. Fixes #76 . Also knife ssh will at its end show a bright red list of machines whose commands exited with a non-zero exit code -- useful when 1 out of 30 machines fails a knife cluster kick.
+* A limited number of commands (ssh, show, kill) now run with no requirement of cloud anything (Relates to #28). Also worked around an annoying incompatibility with chef 0.10.8 (clients have a 'name') vs 0.10.40-and-prev (clients have a 'clientname'.
+* examples all live in `ironfan-homebase` now.
+* When you `knife cluster stop` a node, it sets `node[:state]` to 
+* the cookbook linter now has its own project: [ironfan-scrubby](https://github.com/infochimps-labs/ironfan-scrubby). Along the way, 
+  - some ability to cycle comments from the attributes file into node attribute docs in the `metadata.rb`.
+  - added helpful links to the `README.md` template
+* minor fix to the new `authorized_by` calls
+
 ## v3.0.11: We Raid at Dawn
 
 * You can now assemble raid groups in the cluster definition:

data/Gemfile

@@ -14,9 +14,10 @@ group :development do
 end
 
 group :test do
-  gem 'spork',       "~> 0.9.0.rc5", :platform => :mri
+  gem 'spork',       ">= 0.9.0", :platform => :mri
+  gem 'rcov',        ">= 0.9.9", :platform => :ruby_18
+  gem 'simplecov',   ">= 0.5",   :platform => :ruby_19
   gem 'watchr',      "~> 0.7"
-  gem 'rcov',        ">= 0.9.9"
 end
 
 group :support do

data/README.md

@@ -215,7 +215,8 @@ to the chef server. (see below for details).
 Hooray! You're ready to launch a cluster:
 
 ```ruby
-    knife cluster launch demosimple homebase --bootstrap```
+    knife cluster launch demosimple homebase --bootstrap
+```
 
 It will kick off a node and then bootstrap it. You'll see it install a whole bunch of things. Yay.
 
@@ -263,6 +264,9 @@ If you already have a working chef installation you can skip this section.
 
 To get started with knife and chef, follow the "Chef Quickstart,":http://wiki.opscode.com/display/chef/Quick+Start We use the hosted chef service and are very happy, but there are instructions on the wiki to set up a chef server too. Stop when you get to "Bootstrap the Ubuntu system" -- cluster chef is going to make that much easier.
 
+* [Launch Cloud Instances with Knife](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife)
+* [EC2 Bootstrap Fast Start Guide](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide)
+
 #### Cloud setup
 
 Next,

data/Rakefile

@@ -18,6 +18,8 @@
 # limitations under the License.
 #
 
+DEPRECATED_NAME = "!! This gem has been renamed 'ironfan' (from cluster_chef). \n   It will not be updated after Feb. 2012. \n  Sorry for the inconvenience."
+
 require 'rubygems' unless defined?(Gem)
 require 'bundler'
 begin
@@ -41,13 +43,14 @@ Dir[File.join('tasks', '*.rake')].sort.each{|f| load(f) }
 # Jeweler -- release cluster_chef as a gem
 #
 
-%w[ cluster_chef cluster_chef-knife ].each do |gem_name|
+gems_to_release = ENV['GEMS_TO_RELEASE'] ? ENV['GEMS_TO_RELEASE'].split : %w[ cluster_chef cluster_chef-knife ]
+gems_to_release.each do |gem_name|
   Jeweler::Tasks.new do |gem|
     gem.name        = gem_name
     gem.homepage    = "http://infochimps.com/labs"
     gem.license     = NEW_COOKBOOK_LICENSE.to_s
-    gem.summary     = %Q{cluster_chef allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks.}
-    gem.description = %Q{cluster_chef allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks.}
+    gem.summary     = %Q{#{gem_name} allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks.}
+    gem.description = %Q{#{gem_name} allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks.}
     gem.email       = SSL_EMAIL_ADDRESS
     gem.authors     = ["Infochimps"]
 
@@ -63,8 +66,12 @@ Dir[File.join('tasks', '*.rake')].sort.each{|f| load(f) }
     if    gem.name == 'cluster_chef'
       gem.files.reject!{|f| f =~ %r{^(cluster_chef-knife.gemspec|lib/chef/knife/)} }
       gem.add_runtime_dependency 'cluster_chef-knife', "= #{File.read('VERSION').strip}"
+      gem.post_install_message = DEPRECATED_NAME
     elsif gem.name == 'cluster_chef-knife'
       gem.files.reject!{|f| f =~ %r{^(cluster_chef.gemspec|lib/cluster_chef)} }
+      gem.post_install_message = DEPRECATED_NAME
+    elsif gem.name == 'ironfan'
+      true # pass
     else
       raise "Don't know what to include for gem #{gem.name}"
     end
@@ -83,7 +90,7 @@ end
 RSpec::Core::RakeTask.new(:rcov) do |spec|
   spec.pattern = 'spec/**/*_spec.rb'
   spec.rcov = true
-  spec.rcov_opts = %w[ --exclude .rvm --no-comments --text-summary]
+  spec.rcov_opts = %w[ --exclude .rvm --no-comments --text-summary ]
 end
 
 # ---------------------------------------------------------------------------

data/VERSION

@@ -1 +1 @@
-3.0.12
+3.0.14

data/cluster_chef-knife.gemspec

@@ -5,12 +5,12 @@
 
 Gem::Specification.new do |s|
   s.name = "cluster_chef-knife"
-  s.version = "3.0.12"
+  s.version = "3.0.14"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Infochimps"]
-  s.date = "2012-01-25"
-  s.description = "cluster_chef allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks."
+  s.date = "2012-02-15"
+  s.description = "cluster_chef-knife allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks."
   s.email = "coders@infochimps.com"
   s.extra_rdoc_files = [
     "LICENSE",
@@ -28,10 +28,11 @@ Gem::Specification.new do |s|
     "VERSION",
     "chefignore",
     "cluster_chef-knife.gemspec",
-    "clusters/website_demo.rb",
     "config/client.rb",
     "config/proxy.pac",
-    "lib/chef/knife/bootstrap/ubuntu10.04-basic.erb",
+    "config/ubuntu10.04-cluster_chef.erb",
+    "config/ubuntu11.10-cluster_chef.erb",
+    "ironfan.gemspec",
     "lib/chef/knife/bootstrap/ubuntu10.04-cluster_chef.erb",
     "lib/chef/knife/bootstrap/ubuntu11.10-cluster_chef.erb",
     "lib/chef/knife/cluster_bootstrap.rb",
@@ -62,9 +63,10 @@ Gem::Specification.new do |s|
   ]
   s.homepage = "http://infochimps.com/labs"
   s.licenses = ["apachev2"]
+  s.post_install_message = "!! This gem has been renamed 'ironfan' (from cluster_chef). \n   It will not be updated after Feb. 2012. \n  Sorry for the inconvenience."
   s.require_paths = ["lib"]
   s.rubygems_version = "1.8.15"
-  s.summary = "cluster_chef allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks."
+  s.summary = "cluster_chef-knife allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks."
   s.test_files = ["spec/cluster_chef/cluster_spec.rb", "spec/cluster_chef/facet_spec.rb", "spec/cluster_chef/server_slice_spec.rb", "spec/cluster_chef/server_spec.rb", "spec/cluster_chef_spec.rb", "spec/spec_helper/dummy_chef.rb", "spec/spec_helper.rb", "spec/test_config.rb"]
 
   if s.respond_to? :specification_version then

data/config/ubuntu10.04-cluster_chef.erb

@@ -0,0 +1,157 @@
+bash <<EOF || echo "Chef bootstrap failed!"
+
+# This is the ubuntu natty bootstrap script from infochimps' cluster_chef. It is
+# based on opscode's bootstrap script, with the following important differences: 
+#
+# * installs ruby 1.9.2 (not 1.8.7) from source
+# * upgrades rubygems rather than installing from source
+# * pushes the node identity into the first-boot.json
+# * installs the chef-client service and kicks off the first run of chef
+
+set -e
+
+<%= (@config[:verbosity].to_i > 1 ? 'set -v' : '') %>
+
+RUBY_VERSION=1.9.2-p290
+CHEF_VERSION=<%= bootstrap_version_string.gsub(/.*[\s=]/,"") %>
+
+mkdir -p /tmp/knife-bootstrap
+chmod 700 /tmp/knife-bootstrap 
+cd /tmp/knife-bootstrap
+
+<%= "export http_proxy=\"#{knife_config[:bootstrap_proxy]}\"" if knife_config[:bootstrap_proxy] -%>
+eval `cat /etc/lsb-release `
+export DEBIAN_FRONTEND=noninteractive
+
+date > /etc/box_build_time
+
+echo -e "`date` \n\n**** \n**** apt update:\n****\n"
+apt-get -y update
+apt-get -y upgrade
+
+echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
+apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
+apt-get -y install runit-services
+apt-get clean
+
+if [ ! -f /usr/bin/chef-client ]; then
+echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
+
+wget ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-${RUBY_VERSION}.tar.gz
+tar xzf ruby-${RUBY_VERSION}.tar.gz
+cd ruby-${RUBY_VERSION}
+./configure --with-ruby-version=${RUBY_VERSION} --prefix=/usr --program-suffix=${RUBY_VERSION}
+make -j2
+make install
+
+sudo update-alternatives --remove-all gem && true
+update-alternatives \
+  --install /usr/bin/ruby ruby /usr/bin/ruby${RUBY_VERSION} 400 \
+  --slave   /usr/bin/ri   ri   /usr/bin/ri${RUBY_VERSION}       \
+  --slave   /usr/bin/irb  irb  /usr/bin/irb${RUBY_VERSION}      \
+  --slave   /usr/bin/erb  erb  /usr/bin/erb${RUBY_VERSION}      \
+  --slave   /usr/bin/gem  gem  /usr/bin/gem${RUBY_VERSION}      \
+  --slave   /usr/share/man/man1/ruby.1.gz ruby.1.gz             \
+  /usr/share/man/man1/ruby${RUBY_VERSION}.1
+
+if ruby -e "exit(%x{gem --version} < \"1.6.2\" ? 0 : -1 )" ; then
+  echo -e "`date` \n\n**** \n**** Updating rubygems:\n****\n"
+  # screw you Debian
+  REALLY_GEM_UPDATE_SYSTEM=1 gem update --system
+  # screw you rubygems
+  for foo in /usr/lib/ruby/site_ruby/*/rubygems/deprecate.rb ; do
+    # Don't have to be any such deprecations, in which case $foo won't exist
+    [ -f "$foo" ] && sudo sed -i.bak 's!@skip ||= false!true!' "$foo"
+  done
+fi
+
+echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
+gem install ohai --no-rdoc --no-ri
+gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
+# gems needed for the client.rb or so generically useful you want them at hand
+gem install      --no-rdoc --no-ri extlib bundler json right_aws pry
+
+else # no chef-client
+echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
+fi # end ruby+chef install
+
+# fix a bug in chef that prevents debugging template errors
+# will not work with --prerelease but that's OK hopefully opscode patches this crap soon
+bad_template_file="/usr/lib/ruby/gems/${RUBY_VERSION}/gems/chef-${CHEF_VERSION}/lib/chef/mixin/template.rb"
+if  echo "0505c482b8b0b333ac71bbc8a1795d19  $bad_template_file" | md5sum -c - 2>/dev/null ; then
+  curl https://github.com/mrflip/chef/commit/655a1967253a8759afb54f30b818bbcb7c309198.patch | sudo patch $bad_template_file
+fi
+
+echo -e "`date` \n\n**** \n**** Knifing in the chef client config files:\n****\n"
+mkdir -p /etc/chef
+
+<%- if @config[:client_key] %>
+(
+cat <<'EOP'
+<%= @config[:client_key] %>
+EOP
+) > /tmp/knife-bootstrap/client.pem
+awk NF /tmp/knife-bootstrap/client.pem > /etc/chef/client.pem
+<%- else %>
+(
+cat <<'EOP'
+<%= validation_key %>
+EOP
+) > /tmp/knife-bootstrap/validation.pem
+awk NF /tmp/knife-bootstrap/validation.pem > /etc/chef/validation.pem
+<%- end %>
+
+echo -e "`date` \n\n**** \n**** Nuking our temp files:\n****\n"
+
+cd /tmp
+rm -rf /tmp/knife-bootstrap
+
+echo -e "`date` \n\n**** \n**** Creating chef client script:\n****\n"
+
+(
+cat <<'EOP'
+<%= config_content %>
+<%= @config[:node].chef_client_script_content %>
+EOP
+) > /etc/chef/client.rb
+
+(
+cat <<'EOP'
+<%= { "run_list" => @run_list, "cluster_name" => @config[:node].cluster_name, "facet_name" => @config[:node].facet_name, "facet_index" => @config[:node].facet_index }.to_json %>
+EOP
+) > /etc/chef/first-boot.json
+
+echo -e "`date` \n\n**** \n**** Adding chef client runit scripts:\n****\n"
+( service chef-client stop >/dev/null 2>&1 ; sleep 1 ; killall chef-client 2>/dev/null ) || true
+mkdir -p /var/log/chef /var/chef /etc/service /etc/sv/chef-client/{log/main,supervise} 
+cat > /etc/sv/chef-client/log/run <<EOP
+#!/bin/bash
+exec svlogd -tt ./main
+EOP
+cat > /etc/sv/chef-client/run <<EOP
+#!/bin/bash
+exec 2>&1
+exec /usr/bin/env chef-client -i 43200 -s 20 -L /var/log/chef/client.log
+EOP
+chmod +x  /etc/sv/chef-client/log/run /etc/sv/chef-client/run
+ln -nfs /usr/bin/sv /etc/init.d/chef-client
+
+service chef-client stop >/dev/null 2>&1 || true
+
+<%- if (@config[:bootstrap_runs_chef_client].to_s == 'true') || (@chef_config.knife[:bootstrap_runs_chef_client].to_s == 'true') %>
+echo -e "`date` \n\n**** \n**** First run of chef:\n****\n"
+set -e
+<%= start_chef %>
+set +e
+<%- end %>
+
+echo -e "`date` \n\n**** \n**** Cleanup:\n****\n"
+# make locate work good
+updatedb
+
+echo -e "`date` \n\n**** \n**** Enabling chef client service:\n****\n"
+ln -nfs /etc/sv/chef-client /etc/service/chef-client
+service chef-client start
+
+echo -e "`date` \n\n**** \n**** Cluster Chef client bootstrap complete\n****\n"
+EOF

data/config/ubuntu11.10-cluster_chef.erb

@@ -0,0 +1,145 @@
+bash <<EOF || echo "Chef bootstrap failed!"
+
+# This is the ubuntu oneiric bootstrap script from infochimps' cluster_chef. It is
+# based on the opscode bootstrap script, with the important differences being it:
+#
+# * installs ruby 1.9.2 (not 1.8.7) using the system ruby
+# * upgrades rubygems rather than installing from source
+# * pushes the node identity into the first-boot.json
+# * installs the chef-client service and kicks off the first run of chef
+
+set -e
+
+<%= (@config[:verbosity].to_i > 1 ? 'set -v' : '') %>
+
+RUBY_VERSION=1.9.1
+CHEF_VERSION=<%= bootstrap_version_string.gsub(/.*[\s=]/,"") %>
+
+mkdir -p /tmp/knife-bootstrap
+chmod 700 /tmp/knife-bootstrap 
+cd /tmp/knife-bootstrap
+
+<%= "export http_proxy=\"#{knife_config[:bootstrap_proxy]}\"" if knife_config[:bootstrap_proxy] -%>
+eval `cat /etc/lsb-release `
+export DEBIAN_FRONTEND=noninteractive
+
+date > /etc/box_build_time
+
+# source for sun java if you want to install it later
+apt-get install -y python-software-properties
+add-apt-repository -y ppa:ferramroberto/java
+
+echo -e "`date` \n\n**** \n**** apt update:\n****\n"
+apt-get -y update
+apt-get -y upgrade
+
+echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
+apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
+apt-get clean
+
+if [ ! -f /usr/bin/chef-client ]; then
+echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
+
+apt-get install -y ruby1.9.1 ruby1.9.1-dev
+
+if ruby -e "exit(%x{gem --version} < \"1.6.2\" ? 0 : -1 )" ; then
+  echo -e "`date` \n\n**** \n**** Updating rubygems:\n****\n"
+  # screw you Debian
+  REALLY_GEM_UPDATE_SYSTEM=1 gem update --system
+  # screw you rubygems
+  for foo in /usr/lib/ruby/site_ruby/*/rubygems/deprecate.rb ; do
+    # Don't have to be any such deprecations, in which case $foo won't exist
+    [ -f "$foo" ] && sudo sed -i.bak 's!@skip ||= false!true!' "$foo"
+  done
+fi
+
+echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
+gem install ohai --no-rdoc --no-ri
+gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
+# gems needed for the client.rb or so generically useful you want them at hand
+gem install      --no-rdoc --no-ri extlib bundler json right_aws pry
+
+else # no chef-client
+echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
+fi # end ruby+chef install
+
+# fix a bug in chef that prevents debugging template errors
+# will not work with --prerelease but that's OK hopefully opscode patches this crap soon
+bad_template_file="/usr/lib/ruby/gems/${RUBY_VERSION}/gems/chef-${CHEF_VERSION}/lib/chef/mixin/template.rb"
+if  echo "0505c482b8b0b333ac71bbc8a1795d19  $bad_template_file" | md5sum -c - 2>/dev/null ; then
+  curl https://github.com/mrflip/chef/commit/655a1967253a8759afb54f30b818bbcb7c309198.patch | sudo patch $bad_template_file
+fi
+
+echo -e "`date` \n\n**** \n**** Knifing in the chef client config files:\n****\n"
+mkdir -p /etc/chef
+
+<%- if @config[:client_key] %>
+(
+cat <<'EOP'
+<%= @config[:client_key] %>
+EOP
+) > /tmp/knife-bootstrap/client.pem
+awk NF /tmp/knife-bootstrap/client.pem > /etc/chef/client.pem
+<%- else %>
+(
+cat <<'EOP'
+<%= validation_key %>
+EOP
+) > /tmp/knife-bootstrap/validation.pem
+awk NF /tmp/knife-bootstrap/validation.pem > /etc/chef/validation.pem
+<%- end %>
+
+echo -e "`date` \n\n**** \n**** Nuking our temp files:\n****\n"
+
+cd /tmp
+rm -rf /tmp/knife-bootstrap
+
+echo -e "`date` \n\n**** \n**** Creating chef client script:\n****\n"
+
+(
+cat <<'EOP'
+<%= config_content %>
+<%= @config[:node].chef_client_script_content %>
+EOP
+) > /etc/chef/client.rb
+
+(
+cat <<'EOP'
+<%= { "run_list" => @run_list, "cluster_name" => @config[:node].cluster_name, "facet_name" => @config[:node].facet_name, "facet_index" => @config[:node].facet_index }.to_json %>
+EOP
+) > /etc/chef/first-boot.json
+
+echo -e "`date` \n\n**** \n**** Adding chef client runit scripts:\n****\n"
+( service chef-client stop >/dev/null 2>&1 ; sleep 1 ; killall chef-client 2>/dev/null ) || true
+mkdir -p /var/log/chef /var/chef /etc/service /etc/sv/chef-client/{log/main,supervise} 
+cat > /etc/sv/chef-client/log/run <<EOP
+#!/bin/bash
+exec svlogd -tt ./main
+EOP
+cat > /etc/sv/chef-client/run <<EOP
+#!/bin/bash
+exec 2>&1
+exec /usr/bin/env chef-client -i 43200 -s 20 -L /var/log/chef/client.log
+EOP
+chmod +x  /etc/sv/chef-client/log/run /etc/sv/chef-client/run
+ln -nfs /usr/bin/sv /etc/init.d/chef-client
+
+service chef-client stop >/dev/null 2>&1 || true
+
+<%- if (@config[:bootstrap_runs_chef_client].to_s == 'true') || (@chef_config.knife[:bootstrap_runs_chef_client].to_s == 'true') %>
+echo -e "`date` \n\n**** \n**** First run of chef:\n****\n"
+set -e
+<%= start_chef %>
+set +e
+<%- end %>
+
+echo -e "`date` \n\n**** \n**** Cleanup:\n****\n"
+# make locate work good
+updatedb
+
+echo -e "`date` \n\n**** \n**** Enabling chef client service:\n****\n"
+ln -nfs /etc/sv/chef-client /etc/service/chef-client
+service chef-client start
+
+echo -e "`date` \n\n**** \n**** Cluster Chef client bootstrap complete\n****\n"
+EOF