stack-kicker 0.0.11 → 0.0.12

data/CHANGELOG.md

@@ -1,4 +1,4 @@
 ## 0.0.11
-Stack.validate() was still using the role name, not role_details[:security_group]
-Added doc/examples/apache-via-cloud-init, example of just using cloud-init, no chef.
-Fixed some issues when only using cloud-init
+* Stack.validate() was still using the role name, not role_details[:security_group]
+* Added doc/examples/apache-via-cloud-init, example of just using cloud-init, no chef.
+* Fixed some issues when only using cloud-init

data/README.md

@@ -61,7 +61,7 @@ This is usually only used when :chef_server = true, it stops stack-kicker from a
 #### :security_group
 security group to be assigned to this node.  (set to default is you don't want to manage security groups for every role)
 #### :cloud_config_yaml
-defaults to a file which contains a simple template (lib/cloud-config.yaml in the github repo) that installs the http://apt.opscode.com repo & gig key, as well as installing the opscode-keyring.  Can be replaced with any filename that complies with cloud-init.
+defaults to a file which contains a simple template (lib/cloud-config.yaml in the github repo) that installs the http://apt.opscode.com repo & gpg key, as well as installing the opscode-keyring.  Can be replaced with any filename that complies with cloud-init. If the filename supplied ends in '.erb', it will be processed with ERB. See [ERB Templates](#erbtemplates) for details on available data.
 #### :bootstrap
 Optional filename, the contents of which will get combined with :cloud_config_yaml to form the cloud-init payload (using mime encoding, supported types are #include, ) with some variable substation (chef server ip, environment, validation.pem, roles)  See lib/chef-client-bootstrap-excl-validation-pem.sh as an example.
 
@@ -76,6 +76,8 @@ text/upstart-job   | #upstart-job
 text/part-handler  | #part-handler
 text/cloud-boothook | #cloud-boothook
 
+Can be replaced with any filename that complies with cloud-init. If the filename supplied ends in '.erb', it will be processed with ERB. See [ERB Templates](id:erbtemplates) for details on available data.
+
 #### :data_dir
 data_dir is a hook into the optional :cloud_config_yaml template (lib/cloud-config-w-ephemeral.yaml), which formats & mounts ephemeral0 early in the boot process, allowing it to be used during the rest of the cloud-init. ephemeral0 is mounted as /mnt & then bind mounted to #{data_dir}
 
@@ -85,6 +87,54 @@ This can be an array of strings, such that node X will be assigned :floating_ips
 #### :post_install_script, :post_install_args & :post_install_cwd
 These are used to construct a command to execute, which is executed locally where you executed stack-kicker.  :post_install_args can contain %PUBLIC_IP%, which will be replaced by the public IP of the just created node.  :post_install_script scripts are executed as soon the the instance returns a status='ACTIVE'.  They can be used delay the creation of further nodes of the same role (for example, when creating a rabbitmq cluster, you need to wait for the rabbitmq process to be running before creating the next member of the cluster, or when you are creating a chef-server, you need to wait for the packages to install & daemons to start before attempting to create Chef users & retrieve keys)
 
+## [ERB Templates](id:erbtemplates)
+Both the :cloud_config_yaml & :bootstrap role attributes can point to plain files, files with simple tokens (%HOSTNAME%, %CHEF_SERVER%, %CHEF_ENVIRONMENT%, %CHEF_VALIDATION_PEM%, %SERVER_NAME%, %ROLE% and %DATA_DIR%) or [Ruby ERB](http://www.ruby-doc.org/stdlib-2.0/libdoc/erb/rdoc/ERB.html) templates.
+
+There are 3 key data sets exposed to the ERB templates:
+
+* instances - subset all_instances, just the nodes referenced/managed by this Stackfile
+* all_instances - hash of all instances running in this account
+* config - config contains all the config data from the Stackfile, as well as info about running instances in the account used by the Stackfile 
+
+instances & all_instances hashes look like this:
+
+```
+{
+  "webci-az1-web0001" => {
+    :region => "az-1.region-a.geo-1", 
+    :id => 1395635, 
+    :private_ips => ["10.5.170.11"], 
+    :public_ips => ["15.185.114.181"], 
+    :az => "az-1.region-a.geo-1", 
+    :role => :web}, 
+  "webci-az1-web0002" => {
+    :region => "az-1.region-a.geo-1", 
+    :id => 1396181, 
+    :private_ips => ["10.5.172.145"], 
+    :public_ips => ["15.185.110.42"], 
+    :az => "az-1.region-a.geo-1", 
+    :role=>:web
+  }
+}
+```
+So say you wanted to drop the private IP address of webci-az1-web0001 into the hosts file of webci-az1-web0002 via cloud-init, this fragment in your :cloud_config_yaml could be used:
+
+```
+bootcmd:
+  - echo <%=instances['webci-az1-web0001'][:private_ips][0]%> webci-az1-web0001  >> /etc/hosts
+```
+
+As we're using ERB for these templates, you can use the standard Ruby iterators to work through the hash:
+
+```
+bootcmd:
+<% instances.each do |node_name, node_details| %>
+  - echo <%=node_details[:private_ips][0]%> <%=node_name%>  >> /etc/hosts
+<% end %>
+```
+
+
+
 
 ## Example workflows/models
 stack-kicker was built with the following workflows in mind:

data/doc/examples/Gemfile

@@ -1,4 +1,4 @@
 source 'https://rubygems.org'
 # source 'http://15.185.118.80/partial-gem-mirror/'
 
-gem 'stack-kicker', '>=0.0.10'
+gem 'stack-kicker', '>=0.0.11'

data/doc/examples/apache-via-cloud-init/README.md

@@ -0,0 +1,6 @@
+```
+stack-kicker show-stacks
+stack-kicker show-stack
+stack-kicker show-running
+stack-kicker build
+```

data/doc/examples/apache-via-cloud-init/Stackfile

@@ -16,8 +16,8 @@ module StackConfig
        'az-2.region-a.geo-1' => { 'image_id' => 67074 },
 
        # provisioning info
-       :key_pair => 'dnsaas-dev@hp.com',
-       :key_public => '../credentials/ssh-keys/dnsaas-dev@hp.com.pub',
+       :key_pair => 'YOURKEYPAIRNAME',
+       :key_public => '/path/to/your/public-key.pub',
 
        :name_template => '%s-%s-%s%04d',   # service-site-role0001
        :global_service_name => 'webci',

data/doc/examples/cloud-init-with-erb/README.md

@@ -0,0 +1,6 @@
+```
+stack-kicker show-stacks
+stack-kicker show-stack
+stack-kicker show-running
+stack-kicker build
+```

data/doc/examples/cloud-init-with-erb/Stackfile

@@ -0,0 +1,40 @@
+module StackConfig
+    Stacks = Hash.new
+
+    Stacks['apache-cloud-init'] = {
+      # (we can access environment variable via ENV['foo'] instead of hard coding u/p here)
+      'REGION'      => ENV['OS_REGION_NAME'],
+      'USERNAME'    => ENV['OS_USERNAME'],
+      'PASSWORD'    => ENV['OS_PASSWORD'],
+      'AUTH_URL'    => ENV['OS_AUTH_URL'],
+      'TENANT_NAME' => ENV['OS_TENANT_NAME'],
+
+       # generic instance info
+       'flavor_id'  => 103,
+       'image_id'   => 75845, # Ubuntu Precise 12.04 LTS Server 64-bit
+       # per-az image_id's
+       'az-2.region-a.geo-1' => { 'image_id' => 67074 },
+
+       # provisioning info
+       :key_pair => 'YOURKEYPAIRNAME',
+       :key_public => '/path/to/your/public-key.pub',
+
+       :name_template => '%s-%s-%s%04d',   # service-site-role0001
+       :global_service_name => 'webci',
+       :site_template => '%s',
+
+       # role specification
+       # role names & chef roles should match
+       :roles => {
+         :web => {  :count => 3,
+                    # use the default security group
+                    :security_group => 'default',
+                    # don't use chef as the second-stage provisioner
+                    :skip_chef_prereg => true,
+                    :bootstrap => 'cloud-init.sh.erb',
+                    # use the yaml template that supports configuring the ephemeral space early in the boot process
+                    :cloud_config_yaml => 'cloud-init.yaml.erb',
+         }
+       }
+    }
+end

data/doc/examples/cloud-init-with-erb/cloud-init.sh.erb

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+cat > /var/www/index.html <<HEREDOC
+<html><body><h1>Stack-Kicker was 'ere!</h1>
+<p>This file is dropped in place by a shell script passed to the host via cloud-init</p>
+<p>The web server software is running but no content has been added, yet.</p>
+<h2>instances</h2>
+<%=instances%>
+<hr>
+<h2>all_instances</h2>
+<%=all_instances%>
+<h2>config</h2>
+<%=config%>
+</body></html>
+HEREDOC

data/doc/examples/cloud-init-with-erb/cloud-init.yaml.erb

@@ -0,0 +1,33 @@
+#cloud-config
+
+output:
+  all: ">> /var/log/cloud-init.log"
+
+# use the HPCS Ubuntu Mirror for security-updates too
+bootcmd:
+  - echo 127.0.1.1 %HOSTNAME% >> /etc/hosts
+  - echo # force security.ubuntu.com to mirror.clouds.archive.ubuntu.com >> /etc/hosts
+  - echo 15.185.107.200 security.ubuntu.com >> /etc/hosts
+  - echo <%=instances['webci-az1-web0001'][:private_ips][0]%> webci-az1-web0001  >> /etc/hosts
+<% instances.each do |node_name, node_details| %>
+  - echo <%=node_details[:private_ips][0]%> <%=node_name%>  >> /etc/hosts
+<% end %>
+
+
+# use the HPCS Ubuntu Mirrors
+apt_mirror: http://nova.clouds.archive.ubuntu.com/ubuntu
+
+# Run apt-get update
+package_update: true
+
+# Run apt-get update
+package_upgrade: true
+
+# Install some packages
+packages:
+ - apache2
+ - php5
+
+# Use `sudo -i` to simulate a login shell...
+runcmd:
+ - sudo -i touch /var/log/cloud-init.complete

data/lib/stack-kicker/version.rb

@@ -1,5 +1,5 @@
 module Stack
   module Kicker
-    VERSION = "0.0.11"
+    VERSION = "0.0.12"
   end
 end

data/lib/stack.rb

@@ -448,9 +448,61 @@ cookbook_path [ '<%=config[:stackhome]%>/cookbooks' ]
       end
       config[:all_instances] = servers
     end
+    Stack.categorise_instance_ips(config)
     config[:all_instances]
   end
 
+  # Add an instance to the :all_instances hash, instead of having to poll the whole lot again
+  def Stack.add_instance(config, hostname, region, id, addresses)
+    config[:all_instances][hostname] = { :region => region, :id => id, :addresses => addresses}
+    Stack.categorise_instance_ips(config)
+  end
+
+  def Stack.categorise_instance_ips(config)
+    Logger.debug { ">>> Stack.categorise_instance_ips" }
+    # create a private_ip & public_ip array on the server objects for easy retrieval
+    config[:all_instances].each do |hostname, instance_details|
+      Logger.debug { "Flushing categorised addresses for #{hostname}" }
+      config[:all_instances][hostname][:private_ips] = Array.new
+      config[:all_instances][hostname][:public_ips] = Array.new
+
+      Logger.debug { "Categorising addresses for #{hostname}" }
+      config[:all_instances][hostname][:addresses].each do |address|
+        case address.label
+        when 'public'
+          config[:all_instances][hostname][:public_ips] << address.address
+        when 'private'
+          config[:all_instances][hostname][:private_ips] << address.address
+        else
+          Logger.error "#{address.label} is an unhandled address type for #{hostname}"
+        end
+      end
+    end
+    Logger.debug { "config[:all_instances][= #{config[:all_instances]}" }
+    Logger.debug { "<<< Stack.categorise_instance_ips" }
+  end
+
+  def Stack.get_public_ip(config, hostname)
+    # get a public address from the instance
+    # (could be either the dynamic or one of our floating IPs
+    config[:all_instances][hostname][:addresses].each do |address|
+      if address.label == 'public'
+        return address.address
+      end
+    end
+  end
+
+  def Stack.get_private_ip(config, hostname)
+    # get the private address for an instance
+    config[:all_instances][hostname][:addresses].each do |address|
+      if address.label == 'private'
+        return address.address
+      end
+    end
+  end
+
+
+
   def Stack.show_running(config)
     # TODO: optionally show the hosts that are missing
     ours = Stack.get_our_instances(config)
@@ -460,11 +512,6 @@ cookbook_path [ '<%=config[:stackhome]%>/cookbooks' ]
     end
   end
 
-  # Add an instance to the :all_instances hash, instead of having to poll the whole lot again
-  def Stack.add_instance(config, hostname, region, id, addresses)
-    config[:all_instances][hostname] = { :region => region, :id => id, :addresses => addresses}
-  end
-
   def Stack.ssh(config, hostname = nil, user = ENV['USER'], command = nil)
     # ssh to a host, or all hosts
 
@@ -486,6 +533,7 @@ cookbook_path [ '<%=config[:stackhome]%>/cookbooks' ]
 
 
   def Stack.get_our_instances(config)
+    Logger.debug { ">>> Stack.get_our_instances" }
     # build an hash of running instances that match our generated hostnames
     node_details = Stack.populate_config(config)
 
@@ -504,6 +552,8 @@ cookbook_path [ '<%=config[:stackhome]%>/cookbooks' ]
       end
     end
 
+    Logger.debug { "#{running}" }
+    Logger.debug { "<<< Stack.get_our_instances" }
     running
   end
 
@@ -540,16 +590,6 @@ cookbook_path [ '<%=config[:stackhome]%>/cookbooks' ]
     end
   end
 
-  def Stack.get_public_ip(config, hostname)
-    # get a public address from the instance
-    # (could be either the dynamic or one of our floating IPs
-    config[:all_instances][hostname][:addresses].each do |address|
-      if address.label == 'public'
-        return address.address
-      end
-    end
-  end
-
   def Stack.set_chef_server(config, chef_server)
     # set the private & public URLs for the chef server,
     # called either after we create the Chef Server, or skip over it
@@ -685,6 +725,7 @@ cookbook_path [ '<%=config[:stackhome]%>/cookbooks' ]
           Logger.debug { "multipart_cmd = #{multipart_cmd}" }
           multipart = `#{multipart_cmd}`
           Logger.debug { "multipart = #{multipart}" }
+
           # 2) replace the tokens (CHEF_SERVER, CHEF_ENVIRONMENT, SERVER_NAME, ROLE)
           Logger.debug { "Replacing %HOSTNAME% with #{hostname} in multipart" }
           multipart.gsub!(%q!%HOSTNAME%!, hostname)
@@ -714,6 +755,21 @@ cookbook_path [ '<%=config[:stackhome]%>/cookbooks' ]
           multipart.gsub!(%q!%ROLE%!, role.to_s)
           multipart.gsub!(%q!%DATA_DIR%!, role_details[:data_dir])
 
+          if role_details[:bootstrap].match('\.erb$') || role_details[:cloud_config_yaml].match('\.erb$')
+            Logger.debug { "bootstrap or cloud_config_yaml are erb files"}
+            instances = Hash.try_convert(Stack.get_our_instances(config))
+            all_instances = Hash.try_convert(Stack.get_all_instances(config))
+
+            Logger.debug { instances }
+            Logger.debug { all_instances }
+
+            multipart_template = ERB.new(multipart)
+            multipart_complete = multipart_template.result(binding)
+
+            Logger.debug { "multipart_complete after erb => #{multipart_complete} " }
+            multipart = multipart_complete
+          end
+
           Logger.info "Creating #{hostname} in #{node_details[hostname][:az]} with role #{role}"
 
           # this will get put in /meta.js

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: stack-kicker
 version: !ruby/object:Gem::Version
-  version: 0.0.11
+  version: 0.0.12
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-05-14 00:00:00.000000000 Z
+date: 2013-05-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rdoc
@@ -143,9 +143,14 @@ files:
 - doc/examples/.rvmrc
 - doc/examples/Gemfile
 - doc/examples/apache-via-chef-server/Stackfile
+- doc/examples/apache-via-cloud-init/README.md
 - doc/examples/apache-via-cloud-init/Stackfile
 - doc/examples/apache-via-cloud-init/cloud-init.sh
 - doc/examples/apache-via-cloud-init/cloud-init.yaml
+- doc/examples/cloud-init-with-erb/README.md
+- doc/examples/cloud-init-with-erb/Stackfile
+- doc/examples/cloud-init-with-erb/cloud-init.sh.erb
+- doc/examples/cloud-init-with-erb/cloud-init.yaml.erb
 - features/stack-kicker.feature
 - features/step_definitions/stack-kicker_steps.rb
 - features/support/env.rb
@@ -167,12 +172,18 @@ required_ruby_version: !ruby/object:Gem::Requirement
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: 503041037987723651
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: 503041037987723651
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.23