chef-workflow 0.1.1 → 0.2.0

data/CHANGELOG.md

@@ -0,0 +1,19 @@
+* 0.2.0 (unreleased)
+  * Several missing validation checks and edge races were resolved.
+  * Significant internals refactor -- things that were fast and simple when
+    things were small got less fast and simple later.
+    * Ruby's singleton library used instead of ugly hax
+    * Replace marshal system with a tie-alike database layer built atop sqlite
+    * Everything is namespaced under ChefWorkflow::
+    * Most of the things that break API above were marked deprecated and will print warnings if used.
+  * Provisioners now have a 'report' method which is used in informational
+    tasks to describe the provisioner's unique data.
+  * Fix for knife bootstrap actually DTRT in chef 10.18.x
+  * SSHHelper from testlib was moved here, now that it's used by tasklib as well.
+  * Docs. Lots and Lots and Lots of Docs.
+
+* 0.1.1 December 21, 2012
+  * Fix gemspec. Here's to touching the stove.
+
+* 0.1.0 December 21, 2012
+  * Initial public release

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2012 Erik Hollensbe
+Copyright (c) 2012, 2013 Erik Hollensbe
 
 MIT License
 
@@ -19,4 +19,4 @@ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,145 +1,30 @@
 Chef Workflow Toolkit
 ---------------------
 
-This code is the common base of
-[chef-workflow-tasklib](https://github.com/chef-workflow/chef-workflow-tasklib)
-and
-[chef-workflow-testlib](https://github.com/chef-workflow/chef-workflow-testlib).
-Unless you are looking to create extensions to these two systems, or use the
-libraries contained within, you would be better served (with rare exception) by
-visiting these two projects for your testing and workflow needs.
-
-Environment Variables
----------------------
-
-This toolkit exposes a number of environment variables that you may want to set
-to affect your experience using it:
-
-* `CHEF_WORKFLOW_DEBUG` - set to an integer of 1-3, controls the amount of
-  verbosity of reporting done in the libraries themselves. 1 usually amounts to
-  diagnostic messages, 2 to full transactions (converges, bootstraps, VM
-  provision detail), and 3 to more chatty individual pain points.
-* `CHEF_CONFIG` - the path to your knife.rb. Note that if you've set this up in
-  `KnifeSupport` or with `configure_knife` you should almost never need this.
-* `TEST_CHEF_SUBNET` - specifies a /24 network that IP addresses are drawn from
-  for local testing with VM systems. Note that this support is fairly crude --
-  it's strongly suggested you use a proper /24 and make your last octet `0`.
-  :)
-
-Classes
--------
-
-Expect this to be supplemented with linked RDoc that describes the API for each
-class. For now, though, you'll have to generate the docs yourself or read the
-source comments.
-
-At the time of this writing there is not a very consistent namespacing method,
-expect this to be corrected before the first release.
-
-Utility libraries:
-==================
-
-* `Chef::Workflow::ConfigureHelper` - a mixin which provides easy-to-use methods
-  of driving the various support configuration systems (see below).
-* `AttrSupport` - a small mixin to supply chef-style instance variable mutators;
-  the kind that are a little more convenient to use with `instance_eval`'d
-  blocks.
-* `DebugSupport` - mixin which defines a method called `if_debug` which is our
-  gating mechanism for the `CHEF_WORKFLOW_DEBUG` environment variable.
-* `GenericSupport` - mixin which keeps the configuration interface consistent by
-  providing a `configure` class method that uses `instance_eval` and exposes a
-  pre-configured object under `singleton` which can be manipulated.
-* `KnifePluginSupport` - mixin which contains a routine called
-  `init_knife_plugin` to simplify configuration of knife plugins, which can
-  then be used as normal objects. Also configures a UI object with `StringIO`
-  so that it can be communicated with optionally.
-
-Configuration libraries:
-========================
-
-These all mixin `GenericSupport` and as a result are expected to be treated as
-singletons, by accessing their `singleton` class method and configured with the
-`configure` class method which `instance_eval`'s a block.
-
-If you are using `chef-workflow-tasklib`, most of the bits here as you have
-configured them can be described to you via `bundle exec rake chef:show_config`.
-
-* `GeneralSupport` - "General" configuration attributes that are global to the
-  entire system.
-* `IPSupport` - Database for associating IP addresses with a server group. See
-  discussion on the scheduler below for more information on server groups. This
-  is generally not configured externally, but by tooling within the system.
-* `KnifeSupport` - Most configuration regarding chef lives here, and additional
-  network access.
-* `VagrantSupport` - Specific bits related to using Vagrant, such as the box to
-  be used for provisioning. 
-* `EC2Support` - bits related to driving AWS EC2.
-
-Scheduler and VM
-----------------
-
-The Scheduler and VM system work together to make groups of machines easy to
-provision. 
-
-The scheduler is responsible for scheduling provisioning of groups of machines
-which are interdependent with other machines. In other words when machine C
-depends on B and A to be provisioned, and B and A have no dependencies, and
-they are all scheduled at the same time, the scheduler will determine that A
-and B have to be provisioned immediately and as soon as they are provisioned
-successfully, it will attempt to provision C. Depending on the system
-controlling the scheduler and its constraints, it can do this in a serial or
-parallel fashion, the latter of which will attempt to provision as much as
-possible at the same time, and as things finish will provision things that are
-satisfied by what finished.
-
-In other words, provisioning takes a lot of time in a test run, and the
-scheduler tries very hard to make it take as little time as is reasonably
-possible given your constraints and the constraints of the system.
-
-It manages the actual act of provisioning through the VM system, which tracks
-the state of what's currently provisioned, what has already successfully
-provisioned (and presumed alive), and what is waiting to be provisoned. The VM
-class itself is largely responsible for exposing this data to the scheduler,
-and marshalling its state to disk in the event of a failure so things can be
-cleaned up or resumed in the case of resources that will always be depended on
-for a test run.
-
-In other words, provisioning takes a lot of time in a test run, and the VM
-system tries very hard to not add more time to this by tracking machines that
-are already provisioned so they don't have to be re-provisioned, even between
-runs. It also makes it easy to clean up a bad or stale test run.
-
-Server Groups
-=============
-
-The VM system itself is a mapping of server groups to an array of provisioning
-"commands", which are implemented as classes with a consistent interface
-(visitors). A provisioning command may create a [vagrant
-prison](https://github.com/chef-workflow/vagrant-prison) which contains all the
-servers for that server group, complete with assigning them a host-only
-interface and storing that with `IPSupport` so that it can be retrieved by
-other bits of the test system or task system. Another provisioning command may
-execute the in-code equivalent of `knife bootstrap` to build out your servers
-with a role named after the server group. For de-provisioning, the provisioning
-commands are played in reverse with a `shutdown` call applied to them.
-
-Scheduler and VM libraries
-==========================
-
-* `Scheduler` - this is the meat; if you're driving a new testing system such
-  as rspec, you'll want to get real familiar with the interface presented.
-* `VM` - marshalling and delegation interface. Most of this is exposed to the
-  scheduler interface.
-* `VM::VagrantProvisioner` - creates a vagrant prison composed of n servers for
-  a server group with a unique host-only ip for each server.
-* `VM::EC2Provisioner` - creates a group of servers provisioned on EC2.
-* `VM::KnifeProvisioner` - the provisioner equivalent of `knife bootstrap`, with
-  additional sanity checks for both converge success and a waiting period for
-  search indexing. On deprovision, deletes the nodes that were created. Will
-  always try to bootstrap a group in parallel.
-* `VM::ChefServerProvisioner` - similar to `VM::KnifeProvisioner`, it wraps the
-  [knife-server](https://github.com/fnichol/knife-server) toolkit to create a
-  chef server.
+A system to provide an encompassing toolkit and framework to work with chef
+servers, their contents, and the networks of machines controlled by them. Your
+test environment should look a lot like your production environment, and
+chef-workflow lets you accomplish exactly that.
+
+It is 3 major parts:
+
+* This library, which is core functionality for unifying configuration of the
+  system, provisioning machines and maintaining a source of truth that lives
+  outside chef.
+* [chef-workflow-tasklib](https://github.com/chef-workflow/chef-workflow-tasklib),
+  which is a toolkit that leverages `rake` to provide a common interface for
+  commanding chef-related operations.
+* [chef-workflow-testlib](https://github.com/chef-workflow/chef-workflow-tasklib),
+  which is a toolkit for real-world integration testing. No mocks, real
+  machines, no bullshit.
+
+Most of the Meat is on the Wiki
+-------------------------------
+
+Our [wiki](https://github.com/chef-workflow/chef-workflow/wiki) contains
+a fair amount of information, including how to try chef-workflow without
+actually doing anything more than cloning a repository and running a few
+commands.
 
 Contributing
 ------------
@@ -156,6 +41,10 @@ without rationale will be rejected immediately.
 Credits
 -------
 
-This work was partially sponsored by [Hotel Tonight](http://hoteltonight.com)
-and is what they use to test our infrastructure internally. Primarily authored by
-[Erik Hollensbe](https://github.com/erikh).
+Author: [Erik Hollensbe](https://github.com/erikh)
+
+These companies have assisted by donating time, financial resources, and
+employment to those working on chef-workflow. Supporting OSS is really really
+cool and we should reciprocate.
+
+* [HotelTonight](http://www.hoteltonight.com) 

data/bin/chef-workflow-bootstrap

@@ -92,6 +92,12 @@ configure_knife do
   # hitting this limit a lot, increase this value.
   #
   # search_index_wait 60
+
+  # this is this list of recipes you want tested with the 'test:recipes' task.
+  # Note that you must have the 'minitest-handler' cookbook available on your
+  # chef server for these to work.
+  # 
+  # test_recipes []
 end
 
 #

data/chef-workflow.gemspec

@@ -5,7 +5,7 @@ require 'chef-workflow/version'
 
 Gem::Specification.new do |gem|
   gem.name          = "chef-workflow"
-  gem.version       = Chef::Workflow::VERSION
+  gem.version       = ChefWorkflow::VERSION
   gem.authors       = ["Erik Hollensbe"]
   gem.email         = ["erik+github@hollensbe.org"]
   gem.description   = %q{A comprehensive rake-based workflow for chef}
@@ -21,6 +21,9 @@ Gem::Specification.new do |gem|
   gem.add_dependency 'chef', '~> 10.0'
   gem.add_dependency 'aws-sdk', '~> 1.7.0'
   gem.add_dependency 'net-ssh', '~> 2.2.2'
+  gem.add_dependency 'knife-server', '~> 0.3.3'
+  gem.add_dependency 'sqlite3', '~> 1.3.6'
+  gem.add_dependency 'deprecated', '~> 3.0.1'
 
   gem.add_development_dependency 'rdoc'
   gem.add_development_dependency 'rake'

data/lib/chef-workflow.rb

@@ -7,58 +7,61 @@ require 'chef-workflow/support/ip'
 require 'chef-workflow/support/debug'
 require 'chef-workflow/support/ec2'
 
-class Chef
-  module Workflow
+if ENV["REFACTOR"]
+  require 'deprecated'
+  Deprecated.set_action(:raise)
+end
+
+module ChefWorkflow
+  #
+  # Basic helpers (intended to be mixed in elsewhere) to configure the
+  # various support configuration systems.
+  #
+  module ConfigureHelper
+    #
+    # Configure 'GeneralSupport'
+    #
+    def configure_general(&block)
+      ChefWorkflow::GeneralSupport.configure(&block)
+    end
+
     #
-    # Basic helpers (intended to be mixed in elsewhere) to configure the
-    # various support configuration systems.
+    # Configure 'KnifeSupport'
     #
-    module ConfigureHelper
-      #
-      # Configure 'GeneralSupport'
-      #
-      def configure_general(&block)
-        GeneralSupport.configure(&block)
-      end
+    def configure_knife(&block)
+      ChefWorkflow::KnifeSupport.configure(&block)
+    end
 
-      #
-      # Configure 'KnifeSupport'
-      #
-      def configure_knife(&block)
-        KnifeSupport.configure(&block)
-      end
+    #
+    # Configure 'VagrantSupport'
+    #
+    def configure_vagrant(&block)
+      ChefWorkflow::VagrantSupport.configure(&block)
+    end
 
-      #
-      # Configure 'VagrantSupport'
-      #
-      def configure_vagrant(&block)
-        VagrantSupport.configure(&block)
-      end
-      
-      #
-      # Configure 'IPSupport' - you probably don't need to do this.
-      #
-      def configure_ips(&block)
-        IPSupport.configure(&block)
-      end
+    #
+    # Configure 'IPSupport' - you probably don't need to do this.
+    #
+    def configure_ips(&block)
+      ChefWorkflow::IPSupport.configure(&block)
+    end
 
-      #
-      # Configure 'EC2Support'
-      #
-      def configure_ec2(&block)
-        EC2Support.configure(&block)
-      end
+    #
+    # Configure 'EC2Support'
+    #
+    def configure_ec2(&block)
+      ChefWorkflow::EC2Support.configure(&block)
     end
   end
 end
 
 class << eval("self", TOPLEVEL_BINDING)
-  include Chef::Workflow::ConfigureHelper
+  include ChefWorkflow::ConfigureHelper
 end
 
 if defined? Rake::DSL
   module Rake::DSL
-    include Chef::Workflow::ConfigureHelper
+    include ChefWorkflow::ConfigureHelper
   end
 end
 
@@ -70,4 +73,3 @@ rescue LoadError
   $stderr.puts "There is no chef-workflow-config in your lib directory."
   $stderr.puts "Please run chef-workflow-bootstrap or add one."
 end
-

data/lib/chef-workflow/support/attr.rb

@@ -1,31 +1,33 @@
-#
-# Mixin to make exposing attr modification via `instance_eval` easier.
-#
-module AttrSupport
+module ChefWorkflow
   #
-  # Defines an attribute that is both a standard writer, but with an overloaded
-  # reader that accepts an optional argument. Equivalent to this code for `foo`:
+  # Mixin to make exposing attr modification via `instance_eval` easier.
   #
-  #     attr_writer :foo
-  #
-  #     def foo(*args)
-  #       if args.count > 0
-  #         @foo = arg
-  #       end
-  #
-  #       @foo
-  #     end
-  #
-  def fancy_attr(name)
-    class_eval <<-EOF
-      attr_writer :#{name}
-      def #{name}(*args)
-        if args.count > 0
-          @#{name} = args.first
-        end
+  module AttrSupport
+    #
+    # Defines an attribute that is both a standard writer, but with an overloaded
+    # reader that accepts an optional argument. Equivalent to this code for `foo`:
+    #
+    #     attr_writer :foo
+    #
+    #     def foo(*args)
+    #       if args.count > 0
+    #         @foo = arg
+    #       end
+    #
+    #       @foo
+    #     end
+    #
+    def fancy_attr(name)
+      class_eval <<-EOF
+        attr_writer :#{name}
+        def #{name}(*args)
+          if args.count > 0
+            @#{name} = args.first
+          end
 
-        @#{name}
-      end
-    EOF
+          @#{name}
+        end
+      EOF
+    end
   end
 end

data/lib/chef-workflow/support/db.rb

@@ -0,0 +1,26 @@
+require 'sqlite3'
+require 'fileutils'
+require 'delegate'
+require 'singleton'
+require 'chef-workflow/support/general'
+
+module ChefWorkflow
+  class DatabaseSupport < DelegateClass(SQLite3::Database)
+    include Singleton
+
+    def initialize
+      super(connect)
+    end
+
+    def reconnect
+      close rescue nil
+      __setobj__(connect)
+    end
+
+    def connect
+      vm_file = ChefWorkflow::GeneralSupport.vm_file
+      FileUtils.mkdir_p(File.dirname(vm_file))
+      SQLite3::Database.new(vm_file)
+    end
+  end
+end

data/lib/chef-workflow/support/db/basic.rb

@@ -0,0 +1,225 @@
+require 'chef-workflow/support/db'
+
+module ChefWorkflow
+  class DatabaseSupport
+    class Generic
+      attr_accessor :db
+
+      def initialize(table_name, object_name=nil)
+        raise "a table_name must be provided!" unless table_name
+
+        @table_name   = table_name
+        @object_name  = object_name
+        post_marshal_init
+        create_table
+      end
+
+      def self._load(value)
+        obj = self.new(*Marshal.load(value))
+        return obj
+      end
+
+      def _dump(level)
+        self.db = nil
+        res = Marshal.dump([@table_name, @object_name])
+        post_marshal_init
+        return res
+      end
+
+      def post_marshal_init
+        @db = ChefWorkflow::DatabaseSupport.instance
+      end
+
+      def create_table
+        raise "Do not use the Generic type directly!"
+      end
+    end
+
+    class Object < Generic
+      def [](key)
+        value = @db.execute("select value from #{@table_name} where key=?", [key]).first.first rescue nil
+        return value && Marshal.load(value)
+      end
+
+      def []=(key, value)
+        delete(key)
+        @db.execute("insert into #{@table_name} (key, value) values (?, ?)", [key, Marshal.dump(value)])
+        value
+      end
+
+      def delete(key)
+        @db.execute("delete from #{@table_name} where key=?", [key])
+      end
+
+      def create_table
+        @db.execute <<-EOF
+        create table if not exists #{@table_name} (
+          id integer not null primary key autoincrement,
+          key varchar(255) not null,
+          value text not null,
+          UNIQUE(key)
+        )
+        EOF
+      end
+    end
+
+    class Collection < Generic
+      include Enumerable
+
+      def initialize(table_name, object_name)
+        raise "an object_name must be provided!" unless object_name
+        super
+      end
+    end
+
+    class List < Collection
+      def push(val)
+        @db.execute(
+          "insert into #{@table_name} (name, value) values (?, ?)",
+          [@object_name, Marshal.dump(val)]
+        )
+      end
+
+      alias << push
+
+      def unshift(val)
+        replace([val] + to_a)
+      end
+
+      def shift
+        to_a.shift
+      end
+
+      def pop
+        to_a.pop
+      end
+
+      def replace(ary)
+        clear
+
+        value_string = ("(?, ?)," * ary.count).chop
+
+        @db.execute(
+          "insert into #{@table_name} (name, value) values #{value_string}",
+          ary.map { |x| [@object_name, Marshal.dump(x)] }.flatten
+        )
+      end
+
+      def clear
+        @db.execute("delete from #{@table_name} where name=?", [@object_name])
+      end
+
+      def each
+        to_a.each { |x| yield x }
+      end
+
+      def to_a
+        @db.execute(
+            "select value from #{@table_name} where name=? order by id", 
+            [@object_name]
+        ).map { |x| Marshal.load(x.first) }
+      end
+
+      def create_table
+        @db.execute <<-EOF
+        create table if not exists #{@table_name} (
+          id integer not null primary key autoincrement,
+          name varchar(255) not null,
+          value text not null
+        )
+        EOF
+
+        @db.execute "create index if not exists #{@table_name}_name_index on #{@table_name} (name)"
+      end
+    end
+
+    class Set < Collection
+      def add(key)
+        @db.execute("insert into #{@table_name} (name, key) values (?, ?)", [@object_name, key])
+      end
+
+      def delete(key)
+        @db.execute("delete from #{@table_name} where name=? and key=?", [@object_name, key])
+      end
+
+      def has_key?(key)
+        @db.execute("select count(*) from #{@table_name} where name=? and key=?", [@object_name, key]).first.first.to_i > 0
+      end
+
+      alias include? has_key?
+
+      def clear
+        @db.execute("delete from #{@table_name} where name=?", [@object_name])
+      end
+
+      def replace(set)
+        clear
+
+        return if set.empty?
+
+        value_string = ("(?, ?)," * set.count).chop
+
+        @db.execute("insert into #{@table_name} (name, key) values #{value_string}", set.map { |x| [@object_name, x] }.flatten)
+      end
+
+      def keys
+        @db.execute("select distinct key from #{@table_name} where name=?", [@object_name]).map(&:first)
+      end
+
+      def each
+        keys.each { |x| yield x }
+      end
+
+      def create_table
+        @db.execute <<-EOF
+        create table if not exists #{@table_name} (
+          id integer not null primary key autoincrement,
+          name varchar(255) not null,
+          key varchar(255) not null,
+          UNIQUE(name, key) 
+        )
+        EOF
+
+        @db.execute "create index if not exists #{@table_name}_name_idx on #{@table_name} (name)"
+      end
+    end
+
+    class Map < Set
+      def [](key)
+        value = @db.execute("select value from #{@table_name} where name=? and key=?", [@object_name, key]).first.first rescue nil
+        return value && Marshal.load(value) 
+      end
+
+      def []=(key, value)
+        delete(key)
+        @db.execute("insert into #{@table_name} (name, key, value) values (?, ?, ?)", [@object_name, key, Marshal.dump(value)])
+        value
+      end
+
+      def each
+        keys.each do |key|
+          yield key, self[key]
+        end
+      end
+
+      def to_hash
+        rows = @db.execute("select key, value from #{@table_name} where name=?", [@object_name])
+        Hash[rows.map { |x| [x[0], Marshal.load(x[1])] }]
+      end
+
+      def create_table
+        @db.execute <<-EOF
+        create table if not exists #{@table_name} (
+          id integer not null primary key autoincrement,
+          name varchar(255) not null,
+          key varchar(255) not null,
+          value text not null,
+          UNIQUE(name, key)
+        )
+        EOF
+
+        @db.execute "create index if not exists #{@table_name}_name_idx on #{@table_name} (name)"
+      end
+    end
+  end
+end