knife-stencil 1.0.0

data/README.md

@@ -0,0 +1,128 @@
+# Introduction
+
+`knife-stencil` is a plugin for the command-line client of [Chef][6]. It enables teams to launch hosts in multiple clouds simply by following their own naming convention. It is brought to you by the folks at [OpsUnit][1].
+
+## Description
+
+When provisioning infrastructures with Chef, traditionally one has invoked a long command containing multiple switches. In larger teams, or teams with tiers of engineers this unfortunately leads to errors, as nodes are launched on the incorrect instance size or in incorrect security groups, and also means that there are some aspects of your infrastructure that you cannot recreate from source control.
+
+Whilst tools such as [spiceweasel][2] and [ironfan][3] variously provide ways to serialize and orchestrate Chef infrastructures above the level of individual nodes, our current and prior experience with client implementations calls for a way of solving these problems with more flexibility than the former, and in a simpler and less opinionated manner than the latter.
+
+`knife-stencil` expresses knife configuration options using a system of templates (stencils) that can inherit and override each other. Stencils are selected based on user-defined regexes that describe various potential node names, as befits your existing site naming convention.
+
+The upshot of all of this is that the plugin enables:
+
+* Launching hosts with the correct configuration options simply by following your local naming convention
+* The capture in source control of node meta-data such as availability zone, volume persistence and cloud provider (more on this below).
+* Easy operational integration with your existing site, conventions and infrastructure
+
+## Low Friction Operation across multiple cloud providers
+
+The plugin also works with most knife-cloud plugins automatically. With the correct stencil configuration it is simple to launch hosts in multiple clouds as selected for by their name. Adding new clouds is simply a matter of configuring the appropriate credentials and installing the corresponding knife plugin gem.
+
+## What does this enable me to do?
+
+Some examples of what can be achieved with `knife-stencil` are:
+
+* **Launch hosts in different clouds depending on their environment:** launch development nodes on a local Openstack farm, DR servers in HP's cloud and production servers in Amazon EC2.
+* **Ensure hosts land in the correct availability zone depending on some aspect of their role:** ensure all slave database servers are in a different AZ to the masters
+* **Dynamically determine host instance type based on client:** client bigcorp has a premium contract, launch their database servers as an m2.4xlarge, otherwise launch as m1.xlarge
+* **Split teams into "tooling" and "operational" streams:** avoid resorting to runbooks to capture knowledge; reproduce it programatically and versioned over time
+* **Enforce legal juristiction restrictions:** all hosts containing "backup" for client "bigcorp" should be in the EU region
+
+# How does it work?
+
+Stencils take the general form:
+
+    {
+      "matches": "^[00-99]+\.production\.bigclient\.mycompany\.com$",
+      "inherits": [
+        "environments/production.json",
+        "clients/bigclient.json"
+      ],
+      "options": {
+        ":availability_zone": "us-west-1a"
+      }
+
+Invoking the plugin to launch a new node: `knife stencil server create -N 1380211968.production.bigclient.mycompany.com` will cause all stencil files under ~/.chef/stencils (configurable via `:stencil_root` in your knife.rb) to be evaluated.
+
+Those that express a `matches` regular expression will be compared to the passed node name and ranked in order of strength of match. Once a "root" stencil has been selected any stencils specified through `inherits` will also be evaluated. This continues sequentially and iteratively until a final configuration is arrived upon, at which point the correct cloud server creation class will be instantiated and run.
+
+Where options are expressed through inherited stencils exist for which there is already a configuration value, that value will be overwritten. The inhertence tree of a stencil is evaluated in **reverse** order. That is to say, the further away the inherited stencil from the matching "root" stencil the more generic it is intended to be.
+
+Not all stencils need express `matches` - nor do they need to inherit anything or express an option. You may choose to create stencils whos job it is to provide a convenient point to aggregate many other stencils, for example.
+
+If all this sounds complicated you may find the [examples][4] directory helps to clear things up. You can also invoke `knife stencil server explain live-pretendtown-app00` to have the plugin show you a non-destructive "query plan" of how it arrived at the configuration it would launch the node with. Try this with `:stencil_root` pointing at the 'deadfish' subdirectory of the examples directory.
+
+You are free to organise your stencils, their names and the directory structure they are stored under as you wish. The [examples][4] directory provides just that.
+
+## Launching across multiple clouds
+
+If the final `:options` hash of a server that is being launched via knife-stencil contains the key `:plugin`, `knife-stencil` will attempt to load and instantiate classes from a similarly-named knife plugin. For example a value of `ec2` will have `knife-stencil` look for the `knife-ec2` plugin and the class `Chef::Knife::Ec2ServerCreate` with which to launch the node. Likewise `hp` with `knife-hp` and `Chef::Knife::HpServerCreate` and so-on for all plugins that follow this standard convention.
+
+Given that the cloud plugin to use is expressed via the options hash, which is overridden as inherited stencils are parsed, it is nearly as trivial to provision in other clouds as it is to specify security groups or EBS retention policies or instance size.
+
+## Specifying Configuration Options
+Options are specified using the form they take in the cloud plugin's `options` hash. The easiest way to locate the option you are looking for is:
+
+1. `knife hp server create --help` (note switch)
+2. Execute `gem environment` to find the root of the GEM_PATH into which your gems are being installed
+3. Locate the cloud gem and move into lib/chef/knife
+4. Grep or otherwise locate the option stanza (there may be more than one) for the option you are concerned with
+5. Take the symbol from the `option' and use that in the configuration:
+
+For example, the following options stanza:
+
+     option :rackspace_servicelevel_wait,
+       :long => "--rackspace-servicelevel-wait",
+       :description => "Wait until the Rackspace service level automation setup is complete before bootstrapping chef",
+       :boolean => true,
+       :default => false
+
+Yields the following option:
+
+    "options": {
+      "rackspace_servicelevel_wait": true,
+    }
+
+# Installing and getting started
+
+1. Install the gem
+
+    `gem install knife-stencil`
+
+2. Configure knife.rb to point `:stencil_root` to your preferred location, or create ~/.chef/stencils
+
+    `mkdir -p ~/.chef/stencils`
+
+    OR
+
+    add `knife[:stencil_root] = "/alternative/location"` to `~/.chef/knife.rb`
+
+3. Use the [examples][4] to start building your stencils and `knife stencil server explain HOSTNAME` to debug and explore. Try copying them into `~/.chef/stencils`.
+
+4. Manage ~/.chef/stencils using your source control workflow
+
+## The Deletion Caveat
+At the time of writing only the knife-ec2 plugin from version 0.6.6 supports deletion via `:chef_node_name`. Most other plugins require an instance-id or similar parameter to be passed to delete a server. This is in counterpoint to how the stencil plugin would like to work. If the other plugins supported [this kind of thing][5] deletion would be a unified interface, like the creation of servers.
+
+## Known Problems
+
+* The plugin will not work with chef/knife 11.8.0 or above, due to changes in the internal options parsing code at that point.
+
+## Development
+
+[![Build Status](https://travis-ci.org/opsunit/knife-stencil.png?branch=master)](https://travis-ci.org/opsunit/knife-stencil)
+
+The gem and its dependencies are tested against the following ruby versions:
+
+* 1.9.3
+* 2.0.0
+* 2.1.0
+
+[1]: http://www.opsunit.com
+[2]: https://github.com/mattray/spiceweasel/
+[3]: https://github.com/infochimps-labs/ironfan
+[4]: https://github.com/opsunit/knife-stencil/tree/master/examples
+[5]: https://github.com/opscode/knife-ec2/commit/169350ab0dcf11e7e5c224a1c2333707f0364c54
+[6]: http://www.getchef.com/

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+task :default => [:say_hello]
+
+task :say_hello do
+  puts "Hello. I do nothing at the minute."
+end

data/examples/deadfish/Berksfile

@@ -0,0 +1,3 @@
+site :opscode
+
+cookbook 'hello_world'

data/examples/deadfish/Berksfile.lock

@@ -0,0 +1,7 @@
+{
+  "sources": {
+    "hello_world": {
+      "locked_version": "0.0.3"
+    }
+  }
+}

data/examples/deadfish/README.md

@@ -0,0 +1,21 @@
+# Requirements
+In order to use the examples in this directory you will need to install the following gems:
+
+* knife-ec2
+* knife-hp
+
+You will also need the [`hello_world`][2] cookbook.
+
+If you use [Berkshelf][1] you can install the cookbook requirements from within this directory with:
+
+```bash
+berks install
+berks upload
+```
+
+You will need accounts on at least one of [AWS][3] or [HP Cloud][4] , preferably both.
+
+[1]: http://berkshelf.com/
+[2]: http://community.opscode.com/cookbooks/hello_world
+[3]: http://aws.amazon.com
+[4]: http://www.hpcloud.com/

data/examples/deadfish/base.json

@@ -0,0 +1,14 @@
+{
+  "inherits": [
+  ],
+  "options": {
+    "ssh_user": "root",
+    "ssh_port": "22",
+    "identity_file": "",
+    "host_key_verify": false,
+    "prerelease": false,
+    "environment": "_default",
+    "run_list": ["recipe[hello_world]", "recipe[hello_world]"]
+  }
+}
+

data/examples/deadfish/clouds/ec2.json

@@ -0,0 +1,23 @@
+{
+  "inherits": [
+    "base.json"
+  ],
+  "options": {
+    "plugin": "ec2",
+    "image": "ami-fbb2fc92",
+    "aws_access_key_id": "CHANGEME",
+    "aws_secret_access_key": "CHANGEME",
+    "aws_ssh_key_id": "CHANGEME",
+    "flavor": "m1.small",
+    "ssh_user": "ubuntu",
+    "identity_file": "CHANGEME",
+    "distro": "chef-full",
+    "host_key_verify": false,
+    "no_host_key_verify": true,
+    "security_group_ids": [
+      "CHANGEME",
+      "CHANGEME"
+    ]
+  }
+}
+

data/examples/deadfish/clouds/hp.json

@@ -0,0 +1,19 @@
+{
+  "inherits": [
+    "base.json"
+  ],
+  "options": {
+    "plugin": "hp",
+    "flavor": "101",
+    "image": "75845",
+    "distro": "chef-full",
+    "hp_tenant_id": "CHANGEME",
+    "hp_secret_key": "CHANGEME",
+    "hp_access_key": "CHANGEME",
+    "hp_ssh_key_id": "CHANGEME",
+    "host_key_verify": false,
+    "no_host_key_verify": true,
+    "identity_file": "CHANGEME",
+    "ssh_user": "ubuntu"
+  }
+}

data/examples/deadfish/environments/live.json

@@ -0,0 +1,7 @@
+{
+  "inherits": [
+  ],
+  "options": {
+  }
+}
+

data/examples/deadfish/live-pretendtown-app.json

@@ -0,0 +1,9 @@
+{
+  "matches": "live-pretendtown-app[00-99]+",
+  "inherits": [
+    "environments/live.json",
+    "services/app.json"
+  ],
+  "options": {
+  }
+}

data/examples/deadfish/live-pretendtown-lb.json

@@ -0,0 +1,9 @@
+{
+  "matches": "live-pretendtown-lb[00-99]+",
+  "inherits": [
+    "environments/live.json",
+    "services/lb.json"
+  ],
+  "options": {
+  }
+}

data/examples/deadfish/services/app.json

@@ -0,0 +1,7 @@
+{
+  "inherits": [
+    "clouds/ec2.json"
+  ],
+  "options": {
+  }
+}

data/examples/deadfish/services/lb.json

@@ -0,0 +1,7 @@
+{
+  "inherits": [
+    "clouds/hp.json"
+  ],
+  "options": {
+  }
+}

data/knife-stencil.gemspec

@@ -0,0 +1,26 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'knife-stencil/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "knife-stencil"
+  spec.version       = Knife::Stencil::VERSION
+  spec.authors       = ["sam"]
+  spec.email         = ["sam.pointer@opsunit.com"]
+  spec.description   = %q{Chef knife plugin stencil system with multi-cloud support}
+  spec.summary       = %q{Chef Knife plugin stencil system. Pass only the hostname and inherit all other options. Seamlessly launch in multiple clouds}
+  spec.homepage      = "https://github.com/opsunit/knife-stencil"
+  spec.license       = "GPL3"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake", "~> 0.9"
+
+  spec.add_dependency "chef", "~> 11.6.0"
+  spec.add_dependency "json"
+end

data/lib/chef/knife/stencil_base.rb

@@ -0,0 +1,86 @@
+#
+# Copyright 2013, OpsUnit Ltd.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+require 'chef/knife'
+
+class Chef
+  class Knife
+
+    # This module is included everywhere, as per the conventions for
+    # knife plugins. It includes various helper methods.
+
+    module StencilBase
+
+      # This is used seemingly at random inside knife. Force our values.
+      def locate_config_value(key)
+        key = key.to_sym
+        config[key] || Chef::Config[:knife][key]
+      end
+
+      # Return true if the stencil plugin has been invoked
+      def invoked_as_stencil?
+        if $*[0].downcase == 'stencil'
+          return true
+        end
+
+        return false
+      end
+
+      # Determine where to look for stencils
+      def stencil_root
+        stencil_root = '/'
+        unless Chef::Config[:knife][:stencil_root] && Dir.exist?(Chef::Config[:knife][:stencil_root]) && stencil_root = Chef::Config[:knife][:stencil_root]
+          [ '/etc/chef/stencils', "#{File.join(ENV['HOME'], '.chef/stencils')}" ].each do |directory|
+            stencil_root = directory if Dir.exist?(directory)
+          end
+        end
+
+        return stencil_root
+      end
+
+      # Do exactly that
+      def normalize_path(path, root)
+        unless path[0] == File::SEPARATOR	# FIXME: This will probably make this nasty on Windows
+          return File.join(root, path).to_s
+        else
+          return path
+        end
+      end
+
+      # Return a real Class built from the options given. Used to build EC2ServerCreate, for example
+      def build_plugin_klass(plugin, command, action)
+        begin
+          klass = Object.const_get('Chef').const_get('Knife').const_get(plugin.to_s.capitalize + command.to_s.capitalize + action.to_s.capitalize)
+          klass.respond_to?(:new)
+        rescue NameError, NoMethodError => e
+          puts("I can't find the correct gem for plugin #{config[:plugin]}, it does not declare class #{klas}, or that class does not repsond to 'new'. #{e}")
+          puts("Try: gem install knife-#{config[:plugin]}")
+        end
+
+        return klass
+      end
+
+      # Output method for explanation sub-command. Very basic at present.
+      def explain(string)
+        if $*[2].to_s.downcase == "explain"
+          puts "EXPLAIN: #{string}"
+        end
+      end
+
+    end
+  end
+end

data/lib/chef/knife/stencil_collection.rb

@@ -0,0 +1,96 @@
+#
+# Copyright 2013, OpsUnit Ltd.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+require 'chef/knife/stencil_base'
+require 'chef/knife/stencil_file'
+
+class Chef
+  class Knife
+
+    # A collection of all StencilFile objects
+    class StencilCollection < Array
+
+      include Knife::StencilBase
+
+      def initialize(options={})
+        Dir.glob(File.join(stencil_root, "**/*.json")).each do |file|
+          self << Knife::StencilFile.new(normalize_path(file, stencil_root))
+        end
+      end
+
+      # Return stencil for a given name
+      def for_name(name)
+        collection = Array.new
+        collection.push(best_match(name))
+
+        collection.each do |stencil_file|
+
+	  stencil_file.inherits.each do |path|
+	    collection << self.stencil_for_path(normalize_path(path, stencil_root))
+	  end
+
+        end
+
+        collection.reverse.each {|t| explain("#{t.path} overrides #{t.inherits} and supplies options #{t.options}")}
+        return collection.reverse
+      end
+
+      # Return stencil that best matches a given node name
+      def best_match(name)
+        weight_stencil_file_map = Hash.new
+
+        self.each do |stencil_file|
+          if stencil_file.matches
+            regex = Regexp.new(stencil_file.matches)
+            if name.scan(regex).size > 0
+              weight = ( name.scan(regex)[0].size || 0 )
+	      weight_stencil_file_map[weight] = stencil_file
+            end
+          end
+        end
+
+        unless weight_stencil_file_map.keys.size > 0
+          raise ArgumentError, 'No stencil can be found to match that name'
+        end
+
+        match = weight_stencil_file_map.sort_by{|k,v| k}.reverse.first[1]	# The StencilFile with the most matched chars
+        explain("decision tree: #{weight_stencil_file_map.sort_by{|k,v| k}.reverse}")
+
+	explain("determined #{match.path} to be the best matched root stencil via #{match.matches}")
+        return match
+      end
+
+      # For a given path, determine which stencil matches
+      def stencil_for_path(path)
+        matched_stencil = nil
+
+        self.each do |stencil_file|
+          if stencil_file.path == path 
+            matched_stencil = stencil_file
+          end
+        end
+
+	if matched_stencil
+	  return matched_stencil
+	else
+	  raise ArgumentError, "#{path} not found in StencilCollection"
+	end
+      end
+
+    end
+  end
+end