chef-workflow-tasklib 0.1.1 → 0.2.0

data/CHANGELOG.md

@@ -0,0 +1,22 @@
+* 0.2.0 (unreleased)
+  * **breaking change**: the 'cookbooks' namespace has been moved to chef:cookbooks
+  * chef:build builds a server or set of servers based on a server group. Uses
+    the provisioner and scheduler, and is tracked no differently than
+    provisions from tests or the chef server.
+  * task helpers: with_scheduler and all the bits from SSHHelper in testlib are now available to rake tasks.
+  * chef:converge, given a server group runs chef-client on it.
+  * chef:info namespace created:
+    * chef:show_config is now chef:info:config
+    * chef:info:ips shows external IP assignments by server group
+    * chef:info:provisioned shows information about the provisioned machines
+  * Tasks have been re-written to exploit the refactors in chef-workflow
+  * test:full task has some different semantics which make it more suitable for CI:
+    * test:recipes is a part of the dependency list
+    * always runs chef:clean before doing anything
+  * If a resolver is used, `chef:upload` always invokes it before doing anything
+
+* 0.1.1 December 21, 2012
+  * Fix gemspec. Here's to touching the stove.
+
+* 0.1.0 December 21, 2012
+  * Initial public release

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2012 Erik Hollensbe
+Copyright (c) 2012, 2013 Erik Hollensbe
 
 MIT License
 
@@ -19,4 +19,4 @@ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,8 +1,10 @@
-# Chef Workflow - Rake Tasks & Support
+Chef Workflow - Rake Tasks & Support
+------------------------------------
 
 This gem provides a set of rake tasks broken up logically to support a chef
 workflow, and tooling to assist with driving those tasks. It is intended to
-complement a chef repository by directly being added as a dependency.
+complement a chef repository to keep interaction between multiple consumers of
+it fairly uniform.
 
 Some of the tasks it provides are:
 
@@ -17,244 +19,41 @@ Some of the tasks it provides are:
 * Running unit tests against networks of provisioned machines -- see
   [chef-workflow-testlib](https://github.com/chef-workflow/chef-workflow-testlib)
   for more information.
+* Create machines on the fly that know about your chef server, work with them
+  in a change/converge/inspect cycle, all without having to fire up a ton of
+  machines each time or even have to care about where they live or worry about
+  forgetting to tear them down. We track that for you.
 
-## We do a lot, but we don't tell you how to do it.
+Most of the Meat is on the Wiki
+-------------------------------
 
-The defaults (which you can set up by following the installation instructions)
-use our workflow and a standard chef repository layout. **You don't have to do
-this.** Write your own tasks, use a subset of our tasks, configure our tasks to
-use your settings, or even adjust what happens when you just type `rake`. The
-whole system is built to accomodate this.
+This project is a part of
+[chef-workflow](https://github.com/chef-workflow/chef-workflow).
 
-## Installation
+Our [wiki](https://github.com/chef-workflow/chef-workflow/wiki) contains
+a fair amount of information, including how to try chef-workflow without
+actually doing anything more than cloning a repository and running a few
+commands.
 
-Many of the choices made designing chef-workflow assume bundler is in use. We
-do not recommend installing the gem directly, but through a `Gemfile` that
-lives in your repository.
+Contributing
+------------
 
-Add this line to your application's Gemfile:
+* fork the project
+* make a branch
+* add your stuff
+* push your branch to your repo
+* send a pull request
 
-    gem 'chef-workflow-tasklib'
+**Note:** modifications to gem metadata, author lists, and other credits
+without rationale will be rejected immediately.
 
-And then execute:
+Credits
+-------
 
-    $ bundle
+Author: [Erik Hollensbe](https://github.com/erikh)
 
-To get started, run the `chef-workflow-bootstrap` utility which will create
-some files in your repo. It will not overwrite anything.
+These companies have assisted by donating time, financial resources, and
+employment to those working on chef-workflow. Supporting OSS is really really
+cool and we should reciprocate.
 
-    $ bundle exec chef-workflow-bootstrap
-
-If you already had a `Rakefile`, it will not be modified. To use the tasks, add
-these two lines to your `Rakefile`:
-
-```ruby
-require 'chef-workflow-tasklib'
-chef_workflow_task 'default'
-```
-
-Then to see all the tasks use:
-
-    $ bundle exec rake -T
-
-**Note:** `rake` by itself will **not work**. We depend on newer gems than the
-`rake` that's included with ruby does, which is what will be used if you do not
-execute `bundle exec rake`. At this point bundler runs, and will fail because
-`rake` will have activated gems that your bundle will be incompatible with.
-
-If this is confusing, just remember to always `bundle exec rake` and you won't
-have any trouble.
-
-You should see something similar to this:
-
-```
-rake chef:clean                      # Clean up the state files that chef-workflow generates
-rake chef:clean:ips                  # Clean up the ip registry for chef-workflow
-rake chef:clean:knife                # Clean up the temporary chef configuration for chef-workflow
-rake chef:clean:prisons              # Clean up the prison registry for chef-workflow
-rake chef:environments:upload        # Upload your environments to the chef server
-rake chef:roles:upload               # Upload your roles to the chef server
-rake chef:show_config                # Show the calculated configuration for chef-workflow
-rake chef_server:build_knife_config  # Create and write a knife configuration suitable for creating new chef servers.
-rake chef_server:create:vagrant      # Create a chef server in a vagrant machine.
-rake chef_server:destroy:vagrant     # Destroy the last chef server created with vagrant
-rake cookbooks:update                # Update your locked cookbooks with Berkshelf
-rake cookbooks:upload                # Upload your cookbooks to the chef server
-rake test                            # Run tests
-rake test:build                      # test:refresh and test
-rake test:refresh                    # Refresh all global meta on the chef server
-rake test:vagrant:full               # Build a chef server with vagrant, run test:build, destroy the chef server
-```
-
-Last step: add `.chef-workflow` to `.gitignore`. You can change this, but
-until you've done that, it's going to write things to this directory.
-
-If you want to do a full test run at this point (which with no tests will do
-nothing but build a chef server and tear it down), feel free to:
-
-    $ bundle exec test:vagrant:full
-
-## Notes on dependencies
-
-As of this writing, the workflow has some pretty strict requirements that we
-hope to make more flexible in the near future. Until then, you should be aware
-of a few things:
-
-* Chef 10.16.2 is required. This isn't actually necessary as mentioned above,
-  and will be resolved as soon as I can find time to test with earlier
-  versions. This has some implications:
-
-    * Tools like `berkshelf` and `librarian` and `foodcritic` (all add-ins to
-      the default workflow) have hard dependencies on earlier versions of chef.
-      The tasks that integrate these take this into account, but **you must
-      install the gems separately and manually to use them**.
-
-* Ruby 1.9 is **required**. We will gladly accept patches to make it 1.8.7
-  compatible, but 1.8.7 compatibility is not something we will be actively
-  pursuing at this point.
-
-## Usage
-
-Everything in this workflow is managed through usage of `rake` tasks, with an
-emphasis on cherry-picking tasks with `chef_workflow_task` into your `Rakefile` and
-configuring them with support configurators like `configure_knife` and
-`configure_vagrant`.
-
-The next few sections will briefly cover the high-level design of the system.
-For details, or information on writing your own tasks, hit the
-[wiki](https://github.com/chef-workflow/chef-workflow-tasklib/wiki).
-
-For the basic workflow, note that everything is state tracked between runs for
-the most part, so if you build a chef server with `chef_server:create`,
-other tasks will just do the right thing and operate on that chef server. This
-makes it easy to maintain a edit, upload, test, evaluate, repeat workflow as
-you're working on changes. Knife configuration, vagrant "prisons" (See the
-wiki) and allocated IPs are all tracked on exit in the `.chef-workflow`
-directory.
-
-There are some environment variables which control various settings, like
-debugging output. The defaults are usually fine, but check out the "Environment
-Variables" section of the
-[chef-workflow](https://github.com/chef-workflow/chef-workflow) `README.md`.
-
-## Picking your own workflow
-
-Adding `chef_workflow_task` statements to your workflow is the easiest way to get started.
-We don't expect you to use a resolver, for example, but support both
-[Berkshelf](https://github.com/RiotGames/Berkshelf) and
-[Librarian](https://github.com/applicationsonline/librarian) out of the box.
-Neither of these are required in the default `chef-workflow` require.
-
-Many tasks exist in our [tasks
-directory](https://github.com/chef-workflow/chef-workflow-tasklib/tree/master/lib/chef-workflow/tasks)
-and we strongly recommend poking through it if you're interested in fully
-customizing your workflow. Every attempt has been made for each task library to
-pull in only what it needs to operate which allows you to use each independent
-portion without fear of missing essential code.
-
-So to add `berkshelf` support, we add this to our Rakefile:
-
-```ruby
-chef_workflow_task 'cookbooks/resolve/berkshelf'
-```
-
-(If you are using berkshelf 0.4.x or earlier, `chef_workflow_task
-'cookbooks/resolve/berkshelf0.4'`)
-
-Which will add a few tasks, `cookbooks:resolve` and `cookbooks:update` and a
-few dependencies. **Note:** due to the way many of these tools declare
-dependencies, they must be installed independently of the bundle with `gem
-install`, so in this case, `gem install berkshelf`. This is out of our control.
-
-Let's build a custom workflow. For example let's say we just want to add:
-
-  * resolving cookbooks with librarian
-  * uploading cookbooks
-  * uploading roles
-  * uploading environments
-  * uploading data bags
-  * a task that does all of these when we type `bundle exec rake`
-
-*You'll have to point this at your live knife configuration* (see the next
-section), but your `Rakefile` would at first look something like this:
-
-```ruby
-chef_workflow_task 'cookbooks/resolve/librarian'
-chef_workflow_task 'cookbooks/upload'
-chef_workflow_task 'chef/roles'
-chef_workflow_task 'chef/environments'
-chef_workflow_task 'chef/data_bags'
-
-task :default => %w[
-  cookbooks:resolve_and_upload
-  chef:roles:upload
-  chef:environments:upload
-  chef:data_bags:upload
-]
-```
-
-Adding new rake tasks (or even full task libs) is easy too with the extensible
-configuration system and scaffolding already in place, and utility libraries to
-take the drama out of calling knife plugins or working with collections of
-machines. Please see the
-[wiki](https://github.com/chef-workflow/chef-workflow-tasklib/wiki) for more
-information.
-
-## Configuring the workflow
-
-chef-workflow provides you with a number of pre-baked tasks, but say you need
-just one little tweak so you can get with your life.  Maybe it's where the
-`.chef-workflow` directory lives or where to work with your cookbooks or what
-vagrant box to use, or whatever. You shouldn't have to rewrite the entire task
-to change that.
-
-The `configure_knife` and `configure_vagrant` commands have tooling to assist
-you with this. If you ran the `chef-workflow-bootstrap` tool, you should have a
-file called `lib/chef-workflow-config.rb` which is used to store configuration
-in a central place between this and our testing system.
-
-Code like this goes in this file:
-
-```ruby
-configure_vagrant do
-  # Use this `box_url` for all vagrant machines.
-  box_url "http://files.vagrantup.com/precise64.box"
-end
-
-configure_knife do
-  roles_path "our-roles" # set the roles directory for chef:roles:upload
-end
-```
-
-For our custom workflow example above, you can twiddle a few bits with 
-`configure_knife` to point at known chef configuration locations, which can use
-`~/.chef/knife.rb`, for example. You will want to be careful doing this with
-test integration.
-
-`bundle exec rake chef:show_config` can show you most of the settings in a
-system and how you've configured them.
-
-Please see the
-[wiki](https://github.com/chef-workflow/chef-workflow-tasklib/wiki) for more
-information on how to manipulate this for your own tasks.
-
-## What's next?
-
-  * Guard support for change-triggered test runs.
-  * Feedback that's more reactive than "everything" or "nothing". The speed (or
-    lack thereof) of running large suites makes this especially tricky.
-  * See the Issues list for more.
-
-## Problems
-
-  * It's not fast. EC2 support is faster than Vagrant for most complex cases,
-    but it's still pretty slow.
-  * It has two effective noise levels: "a lot" and "silent".
-
-## Contributing
-
-1. Fork it
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+* [HotelTonight](http://www.hoteltonight.com) 

data/chef-workflow-tasklib.gemspec

@@ -5,7 +5,7 @@ require 'chef-workflow-tasklib/version'
 
 Gem::Specification.new do |gem|
   gem.name          = "chef-workflow-tasklib"
-  gem.version       = Chef::Workflow::Tasklib::VERSION
+  gem.version       = ChefWorkflow::Tasklib::VERSION
   gem.authors       = ["Erik Hollensbe"]
   gem.email         = ["erik+github@hollensbe.org"]
   gem.description   = %q{A set of rake tasks provided as discrete libraries for forming a chef workflow}
@@ -17,9 +17,8 @@ Gem::Specification.new do |gem|
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
 
-  gem.add_dependency 'chef-workflow'
+  gem.add_dependency 'chef-workflow', '~> 0.2.0'
   gem.add_dependency 'knife-dsl', '~> 0.1.0'
   gem.add_dependency 'vagrant-dsl', '~> 0.1.0'
   gem.add_dependency 'rake', '~> 0.9'
-  gem.add_dependency 'knife-server', '~> 0.3.2'
 end

data/lib/chef-workflow-tasklib.rb

@@ -1,21 +1,19 @@
 require 'chef-workflow'
 
-class Chef
-  module Workflow
-    module TaskHelper
-      def chef_workflow_task(obj)
-        require "chef-workflow/tasks/#{obj}"
-      end
+module ChefWorkflow
+  module TaskHelper
+    def chef_workflow_task(obj)
+      require "chef-workflow/tasks/#{obj}"
     end
   end
 end
 
 class << eval("self", TOPLEVEL_BINDING)
-  include Chef::Workflow::TaskHelper
+  include ChefWorkflow::TaskHelper
 end
 
 if defined? Rake::DSL
   module Rake::DSL
-    include Chef::Workflow::TaskHelper
+    include ChefWorkflow::TaskHelper
   end
 end

data/lib/chef-workflow-tasklib/version.rb

@@ -1,7 +1,5 @@
-class Chef
-  module Workflow
-    module Tasklib
-      VERSION = "0.1.1"
-    end
+module ChefWorkflow
+  module Tasklib
+    VERSION = "0.2.0"
   end
 end

data/lib/chef-workflow/task-helpers/ssh.rb

@@ -0,0 +1,11 @@
+require 'chef-workflow/support/ssh'
+
+if defined? Rake::DSL
+  module Rake::DSL
+    include ChefWorkflow::SSHHelper
+  end
+end
+
+class << eval('self', TOPLEVEL_BINDING)
+  include ChefWorkflow::SSHHelper
+end

data/lib/chef-workflow/task-helpers/with_scheduler.rb

@@ -0,0 +1,20 @@
+require 'chef-workflow/support/general'
+require 'chef-workflow/support/ip'
+require 'chef-workflow/support/vagrant'
+require 'chef-workflow/support/ec2'
+require 'chef-workflow/support/knife'
+require 'chef-workflow/support/scheduler'
+require 'chef/config'
+
+if defined? Rake::DSL
+  module Rake::DSL
+    def with_scheduler
+      if File.exist?(ChefWorkflow::KnifeSupport.knife_config_path)
+        Chef::Config.from_file(ChefWorkflow::KnifeSupport.knife_config_path)
+        s = ChefWorkflow::Scheduler.new
+        yield s
+        s.stop
+      end
+    end
+  end
+end

data/lib/chef-workflow/tasks/bootstrap/db.rb

@@ -0,0 +1,7 @@
+namespace :bootstrap do
+  task :db do
+    require 'chef-workflow/support/db'
+    ChefWorkflow::DatabaseSupport.instance.reconnect
+    ChefWorkflow::IPSupport.create_table
+  end
+end

data/lib/chef-workflow/tasks/bootstrap/knife.rb

@@ -3,6 +3,6 @@ require 'chef-workflow/support/knife'
 namespace :bootstrap do
   task :knife do
     # this is what knife-dsl needs to know what config to use
-    ENV["CHEF_CONFIG"] ||= KnifeSupport.singleton.knife_config_path
+    ENV["CHEF_CONFIG"] ||= ChefWorkflow::KnifeSupport.knife_config_path
   end
 end

data/lib/chef-workflow/tasks/chef/build.rb

@@ -0,0 +1,34 @@
+require 'chef-workflow/task-helpers/with_scheduler'
+require 'chef-workflow/support/vm/helpers/knife'
+
+namespace :chef do
+  desc "build and bootstrap a machine with a role in the run_list"
+  task :build, :role_name, :number_of_machines do |task, args|
+    unless args[:role_name]
+      raise 'You must supply a role name to chef:build'
+    end
+
+    role_name          = args[:role_name]
+    number_of_machines = (args[:number_of_machines] || 1).to_i
+
+    with_scheduler do |s|
+      kp = build_knife_provisioner
+      kp.solr_check = false
+
+      s.schedule_provision(
+        role_name,
+        [
+          ChefWorkflow::GeneralSupport.machine_provisioner.new(
+            role_name,
+            number_of_machines
+          ),
+          kp
+        ],
+        []
+      )
+
+      s.run
+      s.wait_for(role_name)
+    end
+  end
+end

data/lib/chef-workflow/tasks/chef/clean.rb

@@ -1,42 +1,21 @@
-require 'chef-workflow/support/general'
-require 'chef-workflow/support/ip'
-require 'chef-workflow/support/vagrant'
-require 'chef-workflow/support/ec2'
-require 'chef-workflow/support/knife'
-require 'chef-workflow/support/scheduler'
-require 'chef/config'
+require 'chef-workflow/task-helpers/with_scheduler'
 require 'fileutils'
 
 namespace :chef do
   namespace :clean do
-    desc "Clean up the ip registry for chef-workflow"
-    task :ips do
-      IPSupport.singleton.reset
-      IPSupport.singleton.write
-    end
-
-    desc "Clean up the temporary chef configuration for chef-workflow"
-    task :knife do
-      FileUtils.rm_rf(KnifeSupport.singleton.chef_config_path)
-      FileUtils.rm_f(KnifeSupport.singleton.knife_config_path)
-    end
-
     desc "Clean up the machines that a previous chef-workflow run generated"
     task :machines do
-      if File.exist?(KnifeSupport.singleton.knife_config_path)
-        Chef::Config.from_file(KnifeSupport.singleton.knife_config_path)
-        s = Scheduler.new(false)
+      with_scheduler do |s|
         s.serial = true
         s.force_deprovision = true
         s.teardown(%w[chef-server])
-        s.write_state
       end
     end
   end
 
   desc "Clean up the entire chef-workflow directory and machines"
   task :clean => [ "chef:clean:machines", "chef_server:destroy" ] do
-    EC2Support.singleton.destroy_security_group
-    FileUtils.rm_rf(GeneralSupport.singleton.workflow_dir)
+    ChefWorkflow::EC2Support.destroy_security_group
+    FileUtils.rm_rf(ChefWorkflow::GeneralSupport.workflow_dir)
   end
 end

data/lib/chef-workflow/tasks/chef/converge.rb

@@ -0,0 +1,12 @@
+require 'chef-workflow/task-helpers/ssh'
+
+namespace :chef do
+  desc "run chef-client on a group of machines"
+  task :converge, :role_name do |task, args|
+    unless args[:role_name]
+      raise "You must supply a node name to converge"
+    end
+
+    ssh_role_command(args[:role_name], 'chef-client')
+  end
+end

data/lib/chef-workflow/tasks/chef/cookbooks/foodcritic.rb

@@ -0,0 +1,18 @@
+require 'chef-workflow/support/knife'
+
+ChefWorkflow::KnifeSupport.add_attribute :fc_cookbooks_path, File.join(Dir.pwd, 'cookbooks')
+ChefWorkflow::KnifeSupport.add_attribute :fc_options, [ ]
+
+namespace :chef do
+  namespace :cookbooks do
+    desc "Run the cookbooks through foodcritic"
+    task :foodcritic do
+      Rake::Task["chef:cookbooks:resolve"].invoke rescue nil
+      if File.directory?(ChefWorkflow::KnifeSupport.fc_cookbooks_path)
+        Bundler.with_clean_env do
+          sh "foodcritic #{ChefWorkflow::KnifeSupport.fc_cookbooks_path} #{ChefWorkflow::KnifeSupport.fc_options.join(" ")}"
+        end
+      end
+    end
+  end
+end

data/lib/chef-workflow/tasks/chef/cookbooks/resolve/berkshelf/berkshelf-base.rb

@@ -0,0 +1,23 @@
+load File.join(File.dirname(__FILE__), '../bootstrap.rb')
+
+# Both berkshelf and librarian have ... aggressive dependencies. They usually are
+# a great way to break your Gemfile if you have chef in it.
+namespace :chef do
+  namespace :cookbooks do
+    desc "Resolve cookbooks and populate using Berkshelf"
+    task :resolve => [ "bootstrap:knife" ] do
+      if File.directory?(KnifeSupport.cookbooks_path)
+        Bundler.with_clean_env do
+          sh "berks install --#{$BERKSHELF_ARG} #{ChefWorkflow::KnifeSupport.cookbooks_path}"
+        end
+      end
+    end
+
+    desc "Update your locked cookbooks with Berkshelf"
+    task :update => [ "bootstrap:knife" ] do
+      Bundler.with_clean_env do
+        sh "berks update"
+      end
+    end
+  end
+end

data/lib/chef-workflow/tasks/chef/cookbooks/resolve/bootstrap.rb

@@ -0,0 +1,7 @@
+# used by resolver tasklibs to ensure their basic dependencies are met. Not to
+# be used directly.
+begin
+  Rake::Task["chef:cookbooks:resolve"].clear
+  Rake::Task["chef:cookbooks:update"].clear
+rescue
+end

data/lib/chef-workflow/tasks/chef/cookbooks/resolve/librarian.rb

@@ -0,0 +1,23 @@
+load File.join(File.dirname(__FILE__), 'bootstrap.rb')
+
+# Both berkshelf and librarian have ... aggressive dependencies. They usually are
+# a great way to break your Gemfile if you have chef in it.
+namespace :chef do
+  namespace :cookbooks do
+    desc "Resolve cookbooks and populate using Librarian"
+    task :resolve => [ "bootstrap:knife" ] do
+      if File.directory?(ChefWorkflow::KnifeSupport.cookbooks_path)
+        Bundler.with_clean_env do
+          sh "librarian-chef install --path #{ChefWorkflow::KnifeSupport.cookbooks_path}"
+        end
+      end
+    end
+
+    desc "Update your locked cookbooks with Librarian"
+    task :update => [ "bootstrap:knife" ] do
+      Bundler.with_clean_env do
+        sh "librarian-chef update"
+      end
+    end
+  end
+end

data/lib/chef-workflow/tasks/chef/cookbooks/upload.rb

@@ -0,0 +1,13 @@
+require 'knife/dsl'
+require 'chef-workflow/tasks/bootstrap/knife'
+
+namespace :chef do
+  namespace :cookbooks do
+    desc "Upload your cookbooks to the chef server"
+    task :upload => [ "bootstrap:knife" ] do
+      Rake::Task["chef:cookbooks:resolve"].invoke rescue nil
+      result = knife %W[cookbook upload -a]
+      fail if result != 0
+    end
+  end
+end

data/lib/chef-workflow/tasks/chef/data_bags.rb

@@ -5,7 +5,7 @@ namespace :chef do
   namespace :data_bags do
     desc "Upload your data bags to the chef server"
     task :upload => [ "bootstrap:knife" ] do
-      if File.directory?(KnifeSupport.singleton.data_bags_path)
+      if File.directory?(ChefWorkflow::KnifeSupport.data_bags_path)
         # bag names: basename of data_bags/*. This presumes your data bags path
         # is structured like so:
         #
@@ -16,9 +16,15 @@ namespace :chef do
         # directory in the right spots. will probably blow up spectacularly
         # explode if you deviate heavily from this directory structure.
         #
-        bag_names = Dir[File.join(KnifeSupport.singleton.data_bags_path, "*")].
+        bag_names = Dir[File.join(ChefWorkflow::KnifeSupport.data_bags_path, "*")].
           select { |x| File.directory?(x) }.
-          map { |x| File.basename(x) } 
+          map { |x| File.basename(x) }
+
+        dbag_from_file = lambda do |bag, items|
+          items = [items] unless items.kind_of?(Array)
+          status = knife %W[data bag from file #{bag}] + items
+          fail if status != 0
+        end
 
         bag_names.each do |bag|
           status = knife %W[data bag create #{bag}]
@@ -26,10 +32,14 @@ namespace :chef do
           #     we're actually depending on this.
           fail if status != 0
 
-          bag_items = Dir[File.join(KnifeSupport.singleton.data_bags_path, bag, '*')].
+          bag_items = Dir[File.join(ChefWorkflow::KnifeSupport.data_bags_path, bag, '*.{rb,js,json}')].
             select { |x| !File.directory?(x) }
-          status = knife %W[data bag from file #{bag}] + bag_items
-          fail if status != 0
+
+          if Chef::VERSION < '10.16'
+            bag_items.each { |item| dbag_from_file.call(bag, item) }
+          else
+            dbag_from_file.call(bag, bag_items)
+          end
         end
       end
     end

data/lib/chef-workflow/tasks/chef/environments.rb

@@ -5,9 +5,20 @@ namespace :chef do
   namespace :environments do
     desc "Upload your environments to the chef server"
     task :upload => [ "bootstrap:knife" ] do
-      if File.directory?(KnifeSupport.singleton.environments_path)
-        status = knife %W[environment from file] + Dir[File.join(KnifeSupport.singleton.environments_path, '*')]
-        fail if status != 0
+      if File.directory?(ChefWorkflow::KnifeSupport.environments_path)
+        env_upload = lambda do |files|
+          files = [files] unless files.kind_of?(Array)
+          status = knife %W[environment from file] + files
+          fail if status != 0
+        end
+
+        environment_files = Dir[File.join(ChefWorkflow::KnifeSupport.environments_path, '*.{rb,js,json}')]
+
+        if Chef::VERSION < '10.16'
+          environment_files.each { |file| env_upload.call(file) }
+        else
+          env_upload.call(environment_files)
+        end
       end
     end
   end

data/lib/chef-workflow/tasks/chef/info.rb

@@ -0,0 +1,67 @@
+require 'chef-workflow/task-helpers/with_scheduler'
+require 'chef-workflow/support/ip'
+require 'chef-workflow/support/general'
+require 'chef-workflow/support/ip'
+require 'chef-workflow/support/knife'
+require 'chef-workflow/support/vagrant'
+
+namespace :chef do
+  namespace :info do
+    desc "Output some information about provisioned machines"
+    task :provisioned do
+      with_scheduler do |s|
+        s.vm_groups.each do |key, values|
+          puts "Group: #{key}"
+          values.each do |value|
+            puts "\t#{value.class.name} provisioner:"
+            value.name ||= key
+            value.report.each do |line|
+              puts "\t\t#{line}"
+            end
+          end
+        end
+      end
+    end
+
+    desc "Output some information about known IP allocations"
+    task :ips do
+      ChefWorkflow::IPSupport.roles.each do |role|
+        puts "Group: #{role}"
+        ChefWorkflow::IPSupport.get_role_ips(role).each do |ip|
+          puts "\t#{ip}"
+        end
+      end
+    end
+
+    desc "Show the calculated configuration for chef-workflow"
+    task :config do
+      puts "general:"
+      puts "\tworkflow_dir: #{ChefWorkflow::GeneralSupport.workflow_dir}"
+      puts "\tvm_file: #{ChefWorkflow::GeneralSupport.vm_file}"
+      puts "\tmachine_provisoner: #{ChefWorkflow::GeneralSupport.machine_provisioner}"
+
+      puts "knife:"
+      mute = %w[knife_config_template]
+      ChefWorkflow::KnifeSupport::DEFAULTS.keys.reject { |x| mute.include?(x.to_s) }.each do |key|
+        puts "\t#{key}: #{ChefWorkflow::KnifeSupport.send(key)}"
+      end
+
+      puts "vagrant:"
+      puts "\tip subnet (/24): #{ChefWorkflow::IPSupport.subnet}"
+      puts "\tbox url: #{ChefWorkflow::VagrantSupport.box_url}"
+
+      puts "ec2:"
+
+      %w[
+        ami
+        instance_type
+        region
+        ssh_key
+        security_groups
+        security_group_open_ports
+      ].each do |key|
+        puts "\t#{key}: #{ChefWorkflow::EC2Support.send(key)}"
+      end
+    end
+  end
+end

data/lib/chef-workflow/tasks/chef/roles.rb

@@ -5,8 +5,8 @@ namespace :chef do
   namespace :roles do
     desc "Upload your roles to the chef server"
     task :upload => [ "bootstrap:knife" ] do
-      if File.directory?(KnifeSupport.singleton.roles_path)
-        status = knife %W[role from file] + Dir[File.join(KnifeSupport.singleton.roles_path, '*')]
+      if File.directory?(ChefWorkflow::KnifeSupport.roles_path)
+        status = knife %W[role from file] + Dir[File.join(ChefWorkflow::KnifeSupport.roles_path, '*.{rb,js,json}')]
         fail if status != 0
       end
     end

data/lib/chef-workflow/tasks/chef/upload.rb

@@ -1,12 +1,12 @@
 chef_workflow_task 'chef/data_bags'
 chef_workflow_task 'chef/environments'
 chef_workflow_task 'chef/roles'
-chef_workflow_task 'cookbooks/upload'
+chef_workflow_task 'chef/cookbooks/upload'
 
 namespace :chef do
   desc "Upload everything."
   task :upload => %w[
-    cookbooks:upload
+    chef:cookbooks:upload
     chef:roles:upload
     chef:environments:upload
     chef:data_bags:upload

data/lib/chef-workflow/tasks/chef_server.rb

@@ -1,41 +1,42 @@
-require 'chef-workflow/support/general'
-require 'chef-workflow/support/knife'
-require 'chef-workflow/support/scheduler'
+require 'chef-workflow/task-helpers/with_scheduler'
 
 namespace :chef_server do
-  desc "Create a chef server with the #{GeneralSupport.singleton.machine_provisioner} provisioner"
-  task :create do
+  desc "Create a chef server with the #{ChefWorkflow::GeneralSupport.machine_provisioner} provisioner"
+  task :create => [ "chef:clean" ] do
+    chef_workflow_task 'bootstrap/db'
+    Rake::Task["bootstrap:db"].invoke
+
     # FIXME not really happy with having to repeat this at the end, but it's
     # necessary. maybe a subroutine in the right place in the future is the
     # best approach, but I'm feeling lazy right now.
-    KnifeSupport.singleton.build_knife_config
-    Chef::Config.from_file(KnifeSupport.singleton.knife_config_path)
-
-    s = Scheduler.new(false)
-    s.serial = true
+    ChefWorkflow::KnifeSupport.build_knife_config
 
-    s.schedule_provision(
-      'chef-server', 
-      [
-        GeneralSupport.singleton.machine_provisioner.new('chef-server', 1), 
-        VM::ChefServerProvisioner.new
-      ],
-      []
-    )
+    with_scheduler do |s|
+      s.serial = true
+      s.schedule_provision(
+        'chef-server', 
+        [
+          ChefWorkflow::GeneralSupport.machine_provisioner.new('chef-server', 1), 
+          ChefWorkflow::VM::ChefServerProvisioner.new
+        ],
+        []
+      )
 
-    s.run
-    s.wait_for('chef-server') # probably superfluous
-    s.write_state
+      s.run
+      s.wait_for('chef-server') # probably superfluous
+    end
 
-    KnifeSupport.singleton.build_knife_config
-    Chef::Config.from_file(KnifeSupport.singleton.knife_config_path)
+    # now that our chef server exists, re-bootstrap the config for future tasks
+    ChefWorkflow::KnifeSupport.build_knife_config
+    Chef::Config.from_file(ChefWorkflow::KnifeSupport.knife_config_path)
   end
 
   desc "Destroy the chef server"
   task :destroy do
-    s = Scheduler.new(false)
-    s.serial = true
-    s.force_deprovision = true
-    s.teardown_group('chef-server')
+    with_scheduler do |s|
+      s.serial = true
+      s.force_deprovision = true
+      s.teardown_group('chef-server')
+    end
   end
 end

data/lib/chef-workflow/tasks/default.rb

@@ -1,6 +1,8 @@
 chef_workflow_task 'chef_server'
-chef_workflow_task 'cookbooks/upload'
+chef_workflow_task 'chef/cookbooks/upload'
 chef_workflow_task 'chef/upload'
-chef_workflow_task 'chef/show_config'
 chef_workflow_task 'chef/clean'
+chef_workflow_task 'chef/build'
+chef_workflow_task 'chef/converge'
+chef_workflow_task 'chef/info'
 chef_workflow_task 'test/build'

data/lib/chef-workflow/tasks/test.rb

@@ -1,6 +1,6 @@
 require 'rake/testtask'
-require 'chef-workflow/support/scheduler'
-require 'chef-workflow/support/knife'
+require 'chef-workflow/task-helpers/with_scheduler'
+require 'chef-workflow/support/vm/helpers/knife'
 
 Rake::TestTask.new do |t|
   t.libs << "test"
@@ -11,59 +11,49 @@ end
 namespace :test do
   desc "Test recipes in the test_recipes configuration."
 
-  task :recipes => [ "recipes:cleanup" ] do
-    Chef::Config.from_file(KnifeSupport.singleton.knife_config_path)
-
-    s = Scheduler.new(true)
-    s.run
+  task :recipes => [ "test:recipes:cleanup" ] do
+    with_scheduler do |s|
+      s.run
 
-    groups =
-      KnifeSupport.singleton.test_recipes.map do |recipe|
-        group_name = "recipe-#{recipe.gsub(/::/, '-')}"
+      groups =
+        ChefWorkflow::KnifeSupport.test_recipes.map do |recipe|
+          group_name = "recipe-#{recipe.gsub(/::/, '-')}"
 
-        kp              = VM::KnifeProvisioner.new
-        kp.username     = KnifeSupport.singleton.ssh_user
-        kp.password     = KnifeSupport.singleton.ssh_password
-        kp.use_sudo     = KnifeSupport.singleton.use_sudo
-        kp.ssh_key      = KnifeSupport.singleton.ssh_identity_file
-        kp.environment  = KnifeSupport.singleton.test_environment
-        kp.run_list     = [ "recipe[#{recipe}]", "recipe[minitest-handler]" ]
-        kp.solr_check   = false
+          kp            = build_knife_provisioner
+          kp.run_list   = [ "recipe[#{recipe}]", "recipe[minitest-handler]" ]
+          kp.solr_check = false
 
-        s.schedule_provision(
-          group_name,
-          [
-            GeneralSupport.singleton.machine_provisioner.new(group_name, 1),
-            kp
-          ]
-        )
+          s.schedule_provision(
+            group_name,
+            [
+              ChefWorkflow::GeneralSupport.machine_provisioner.new(group_name, 1),
+              kp
+            ]
+          )
 
-        group_name
-      end
+          group_name
+        end
 
-    s.wait_for(*groups)
+      s.wait_for(*groups)
 
-    groups.each do |group_name|
-      s.teardown_group(group_name)
+      groups.each do |group_name|
+        s.teardown_group(group_name)
+      end
     end
-
-    s.write_state
   end
 
   namespace :recipes do
     desc "Cleanup any stale instances created running recipe tests."
     task :cleanup do
-      Chef::Config.from_file(KnifeSupport.singleton.knife_config_path)
-      s = Scheduler.new(false)
-      s.run
-
-      s.vm_groups.select do |g, v|
-        g.start_with?("recipe-")
-      end.each do |g, v|
-        s.teardown_group(g, false)
+      with_scheduler do |s|
+        s.run
+
+        s.vm_groups.select do |g, v|
+          g.start_with?("recipe-")
+        end.each do |g, v|
+          s.teardown_group(g, false)
+        end
       end
-
-      s.write_state
     end
   end
 end

data/lib/chef-workflow/tasks/test/build.rb

@@ -3,22 +3,9 @@ require 'chef-workflow/tasks/chef/upload'
 require 'chef-workflow/tasks/chef/clean'
 
 namespace :test do
-  task :resolve_hack do
-    begin
-      Rake::Task["cookbooks:resolve"].invoke
-    rescue
-    end
-  end
-
-  desc "Refresh all global meta on the chef server"
-  task :refresh => [
-    "test:resolve_hack",
-    "chef:upload"
-  ]
-
-  desc "test:refresh and test"
+  desc "chef:upload and test"
   task :build => [
-    "test:refresh",
+    "chef:upload",
     "test"
   ]
 
@@ -32,6 +19,10 @@ namespace :test do
   task :full => [
     "chef_server:create",
     "test:build",
-    "chef:clean"
-  ]
+    "test:recipes"
+  ] do
+    # hack to get chef:clean to run again
+    (Rake::Task["chef:clean"].prerequisite_tasks + [Rake::Task["chef:clean"]]).each(&:reenable)
+    Rake::Task["chef:clean"].invoke
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: chef-workflow-tasklib
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,24 +9,24 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-12-21 00:00:00.000000000 Z
+date: 2013-02-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: chef-workflow
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.2.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.2.0
 - !ruby/object:Gem::Dependency
   name: knife-dsl
   requirement: !ruby/object:Gem::Requirement
@@ -75,22 +75,6 @@ dependencies:
     - - ~>
       - !ruby/object:Gem::Version
         version: '0.9'
-- !ruby/object:Gem::Dependency
-  name: knife-server
-  requirement: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: 0.3.2
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: 0.3.2
 description: A set of rake tasks provided as discrete libraries for forming a chef
   workflow
 email:
@@ -100,6 +84,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - .gitignore
+- CHANGELOG.md
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -107,22 +92,26 @@ files:
 - chef-workflow-tasklib.gemspec
 - lib/chef-workflow-tasklib.rb
 - lib/chef-workflow-tasklib/version.rb
+- lib/chef-workflow/task-helpers/ssh.rb
+- lib/chef-workflow/task-helpers/with_scheduler.rb
+- lib/chef-workflow/tasks/bootstrap/db.rb
 - lib/chef-workflow/tasks/bootstrap/knife.rb
+- lib/chef-workflow/tasks/chef/build.rb
 - lib/chef-workflow/tasks/chef/clean.rb
+- lib/chef-workflow/tasks/chef/converge.rb
+- lib/chef-workflow/tasks/chef/cookbooks/foodcritic.rb
+- lib/chef-workflow/tasks/chef/cookbooks/resolve/berkshelf.rb
+- lib/chef-workflow/tasks/chef/cookbooks/resolve/berkshelf/berkshelf-base.rb
+- lib/chef-workflow/tasks/chef/cookbooks/resolve/berkshelf0.4.rb
+- lib/chef-workflow/tasks/chef/cookbooks/resolve/bootstrap.rb
+- lib/chef-workflow/tasks/chef/cookbooks/resolve/librarian.rb
+- lib/chef-workflow/tasks/chef/cookbooks/upload.rb
 - lib/chef-workflow/tasks/chef/data_bags.rb
 - lib/chef-workflow/tasks/chef/environments.rb
+- lib/chef-workflow/tasks/chef/info.rb
 - lib/chef-workflow/tasks/chef/roles.rb
-- lib/chef-workflow/tasks/chef/show_config.rb
 - lib/chef-workflow/tasks/chef/upload.rb
 - lib/chef-workflow/tasks/chef_server.rb
-- lib/chef-workflow/tasks/cookbooks/foodcritic.rb
-- lib/chef-workflow/tasks/cookbooks/resolve/berkshelf.rb
-- lib/chef-workflow/tasks/cookbooks/resolve/berkshelf/berkshelf-base.rb
-- lib/chef-workflow/tasks/cookbooks/resolve/berkshelf0.4.rb
-- lib/chef-workflow/tasks/cookbooks/resolve/bootstrap.rb
-- lib/chef-workflow/tasks/cookbooks/resolve/librarian.rb
-- lib/chef-workflow/tasks/cookbooks/resolve_and_upload.rb
-- lib/chef-workflow/tasks/cookbooks/upload.rb
 - lib/chef-workflow/tasks/default.rb
 - lib/chef-workflow/tasks/test.rb
 - lib/chef-workflow/tasks/test/build.rb
@@ -146,9 +135,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 1.8.25
 signing_key: 
 specification_version: 3
 summary: A set of rake tasks provided as discrete libraries for forming a chef workflow
 test_files: []
-has_rdoc: 

data/lib/chef-workflow/tasks/chef/show_config.rb

@@ -1,38 +0,0 @@
-require 'chef-workflow/support/general'
-require 'chef-workflow/support/ip'
-require 'chef-workflow/support/knife'
-require 'chef-workflow/support/vagrant'
-
-namespace :chef do
-  desc "Show the calculated configuration for chef-workflow"
-  task :show_config do
-    puts "general:"
-    puts "\tworkflow_dir: #{GeneralSupport.singleton.workflow_dir}"
-    puts "\tvm_file: #{GeneralSupport.singleton.vm_file}"
-    puts "\tchef_server_prison: #{GeneralSupport.singleton.chef_server_prison}"
-    puts "\tmachine_provisoner: #{GeneralSupport.singleton.machine_provisioner}"
-
-    puts "knife:"
-    mute = %w[knife_config_template]
-    KnifeSupport::DEFAULTS.keys.reject { |x| mute.include?(x.to_s) }.each do |key|
-      puts "\t#{key}: #{KnifeSupport.singleton.send(key)}"
-    end
-
-    puts "vagrant:"
-    puts "\tip subnet (/24): #{IPSupport.singleton.subnet}"
-    puts "\tbox url: #{VagrantSupport.singleton.box_url}"
-
-    puts "ec2:"
-
-    %w[
-      ami
-      instance_type
-      region
-      ssh_key
-      security_groups
-      security_group_open_ports
-    ].each do |key|
-      puts "\t#{key}: #{EC2Support.singleton.send(key)}"
-    end
-  end
-end

data/lib/chef-workflow/tasks/cookbooks/foodcritic.rb

@@ -1,16 +0,0 @@
-require 'chef-workflow/support/knife'
-
-KnifeSupport.add_attribute :fc_cookbooks_path, File.join(Dir.pwd, 'cookbooks')
-KnifeSupport.add_attribute :fc_options, [ ]
-
-namespace :cookbooks do
-  desc "Run the cookbooks through foodcritic"
-  task :foodcritic do
-    Rake::Task["cookbooks:resolve"].invoke rescue nil
-    if File.directory?(KnifeSupport.singleton.fc_cookbooks_path)
-      Bundler.with_clean_env do
-        sh "foodcritic #{KnifeSupport.singleton.fc_cookbooks_path} #{KnifeSupport.singleton.fc_options.join(" ")}"
-      end
-    end
-  end
-end

data/lib/chef-workflow/tasks/cookbooks/resolve/berkshelf/berkshelf-base.rb

@@ -1,21 +0,0 @@
-load File.join(File.dirname(__FILE__), '../bootstrap.rb')
-
-# Both berkshelf and librarian have ... aggressive dependencies. They usually are
-# a great way to break your Gemfile if you have chef in it.
-namespace :cookbooks do
-  desc "Resolve cookbooks and populate using Berkshelf"
-  task :resolve => [ "bootstrap:knife" ] do
-    if File.directory?(KnifeSupport.singleton.cookbooks_path)
-      Bundler.with_clean_env do
-        sh "berks install --#{$BERKSHELF_ARG} #{KnifeSupport.singleton.cookbooks_path}"
-      end
-    end
-  end
-
-  desc "Update your locked cookbooks with Berkshelf"
-  task :update => [ "bootstrap:knife" ] do
-    Bundler.with_clean_env do
-      sh "berks update"
-    end
-  end
-end

data/lib/chef-workflow/tasks/cookbooks/resolve/bootstrap.rb

@@ -1,10 +0,0 @@
-# used by resolver tasklibs to ensure their basic dependencies are met. Not to
-# be used directly.
-begin
-  Rake::Task["cookbooks:resolve"].clear
-  Rake::Task["cookbooks:update"].clear
-rescue
-end
-
-require 'chef-workflow/tasks/bootstrap/knife'
-require 'chef-workflow/tasks/cookbooks/resolve_and_upload'

data/lib/chef-workflow/tasks/cookbooks/resolve/librarian.rb

@@ -1,21 +0,0 @@
-load File.join(File.dirname(__FILE__), 'bootstrap.rb')
-
-# Both berkshelf and librarian have ... aggressive dependencies. They usually are
-# a great way to break your Gemfile if you have chef in it.
-namespace :cookbooks do
-  desc "Resolve cookbooks and populate using Librarian"
-  task :resolve => [ "bootstrap:knife" ] do
-    if File.directory?(KnifeSupport.singleton.cookbooks_path)
-      Bundler.with_clean_env do
-        sh "librarian-chef install --path #{KnifeSupport.singleton.cookbooks_path}"
-      end
-    end
-  end
-
-  desc "Update your locked cookbooks with Librarian"
-  task :update => [ "bootstrap:knife" ] do
-    Bundler.with_clean_env do
-      sh "librarian-chef update"
-    end
-  end
-end

data/lib/chef-workflow/tasks/cookbooks/resolve_and_upload.rb

@@ -1,6 +0,0 @@
-require 'chef-workflow/tasks/cookbooks/upload'
-
-namespace :cookbooks do
-  desc "Run the cookbook resolver and upload the result to the chef server."
-  task :resolve_and_upload => [ "cookbooks:resolve", "cookbooks:upload" ] 
-end

data/lib/chef-workflow/tasks/cookbooks/upload.rb

@@ -1,10 +0,0 @@
-require 'knife/dsl'
-require 'chef-workflow/tasks/bootstrap/knife'
-
-namespace :cookbooks do
-  desc "Upload your cookbooks to the chef server"
-  task :upload => [ "bootstrap:knife" ] do
-    result = knife %W[cookbook upload -a]
-    fail if result != 0
-  end
-end