chef-tlc-workflow 0.1.0

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.vagrant
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,7 @@
+source 'https://rubygems.org'
+
+source 'https://gems.gemfury.com/hUe8s8nSyzxs7JMMSZV8/' # vagrant-1.0.5.1
+source 'https://gems.gemfury.com/psBbdHx94zqZrvxpiVmm/' # librarian-0.0.26.2
+
+# Specify your gem's dependencies in chef-tlc-workflow.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Torben
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,93 @@
+# TLC Chef Workflow
+
+The `chef-tlc-workflow` aims to make our workflow for developing with Chef more explict.
+
+First of all, it provides templates for three different deployment environments (esx, ec2 and local). The templates contain among others a `Rakefile` for interacting with the deployment environment as well as a description of the nodes in these environments.
+
+Other than the templates it provides helper methods to use the dependencies defined in `metadata.rb` from within [librarian](https://github.com/applicationsonline/librarian) `Cheffile`s. This ensures consistency between the dependencies specified in metadata and resolved via librarian. 
+
+Finally, it ensures a consistent gem set by declaring all the gems we use in our workflow in its own gemspec, i.e. `chef-tlc-workflow` is the only gem you need to depend on - everything else (like vagrant, mcloud, etc..) comes in as transitive dependencies.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'chef-tlc-workflow'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install chef-tlc-workflow
+
+
+## Usage
+
+See valid usages below.
+
+### Templates for Deployment Environments
+
+As of TLC Project Infrastructure v1 we support three different deployment environments:
+
+ * `esx` - VMs on our ESX infrastructure where we have ssh access but no control over the VMs (provisioned via [knife-solo](http://matschaffer.github.com/knife-solo/)) 
+ * `ec2` - ec2 instances in the amazon cloud (managed and provisioned via [mccloud](https://github.com/jedi4ever/mccloud))
+ * `local` - local VirtualBox VMs for development and testing (managed and provisioned via [vagrant](http://vagrantup.com))
+
+This project contains "templates" for each of these deployment environments in the `test/` directory. For now, you have to copy the directories - there is no scaffolding yet. 
+
+Each of the deployment environments in the `test/` directory provides a `Rakefile` with a similar interface (i.e. similar rake tasks) to interact with. See the integration tests in the top level `Rakefile` for an example.
+
+In addition, it provides a place for describing the nodes within that deployment environment. Depending on the environment this might be a `Vagrantfile`, `Mccloudfile` or `<node>.json` file. Again, see the examples in the `test/` directory. 
+
+### Librarian Helper
+
+Since librarian does not support reading the cookbook dependencies from `metadata.rb`, the `ChefTLCWorkflow::Helpers` module adds this functionality.
+
+In the trivial case you can use it in your cookbook project's `Cheffile` like so:
+
+    require 'chef-tlc-workflow/helpers'
+
+    ChefTLCWorkflow::Helpers::from_metadata.each do |cb_name, cb_version|
+      cookbook cb_name, cb_version
+    end
+
+If some of the cookbooks defined in metadata.rb are not available from the community site, you can define your overrides like so: 
+
+    ...
+    @overrides = {
+      'tlc-base' => { :git => 'https://github.com/tknerr/cookbook-tlc-base.git', :ref => 'master' },
+    }
+
+    ChefTLCWorkflow::Helpers::from_metadata.each do |cb_name, cb_version|
+      cookbook cb_name, cb_version, @overrides[cb_name]
+    end
+
+
+### Single Gemfile Dependency
+
+It references all gems we need for our Chef workflow, this means that your `Gemfile` basically looks like this:
+
+```
+source :rubygems
+
+# additional sources for patched gems
+source 'https://gems.gemfury.com/hUe8s8nSyzxs7JMMSZV8/' # vagrant-1.0.5.1
+source 'https://gems.gemfury.com/psBbdHx94zqZrvxpiVmm/' # librarian-0.0.26.2
+
+gem "chef-tlc-workflow", "0.1.0"
+```
+
+This brings in all the transitive gem dependencies as defined in the `chef-tlc-workflow.gemspec`, e.g. vagrant, librarian, chef, mccloud etc...
+
+While this is not ideal in terms of keeping the LOAD_PATH as small as possible, it's a tradeoff in favor of convenience and consistency.
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,110 @@
+require "bundler/gem_tasks"
+
+desc "run all tests"
+task :test => [
+  :test_esx_bootstrap, 
+  :test_ec2_bootstrap,
+  :test_local_bootstrap
+]
+
+#
+# TODO:
+# * rewrite as cucumber feature
+# * check for apache default page
+# * more realistic scenario: e.g. scaffold infra, create Cheffile and node.json on the fly, etc...
+#
+desc "tests bootstrapping with knife-solo in an esx-like environment"
+task :test_esx_bootstrap do
+
+  app = 'sample-app@0.1.0'
+  user = 'vagrant'
+  host = '33.33.77.10'
+  ssh_key = 'W:/home/.vagrant.d/insecure_private_key'
+
+  begin
+    # simulate esx-like environment using vagrant and a bare-os basebox
+    sh "vagrant destroy esx_like_vm -f"
+    sh "vagrant up esx_like_vm" do |ok, res|
+      puts "ok: #{ok}\nres: #{res}" # ignore vagrant error for bare-os vm
+    end
+
+    # resolve deps and provision node
+    run_cmd_esx "rake resolve_deps"
+    # bootstrap node with chef
+    run_cmd_esx "rake bootstrap[#{host},#{user},#{ssh_key}]"
+    # provision node with app
+    run_cmd_esx "rake provision[#{app},#{host},#{user},#{ssh_key}]"
+    # remove chef-solo traces from node
+    run_cmd_esx "rake cleanup[#{host},#{user},#{ssh_key}]"
+
+    # TODO: test if sample app works
+  ensure
+    # cleanup
+    sh "vagrant destroy esx_like_vm -f"
+  end
+end
+
+
+desc "tests bootstrapping with mccloud in ec2 (requires ec2 credentials)"
+task :test_ec2_bootstrap do
+
+  vm_name = 'sample-app'
+
+  begin
+    # resolve deps
+    run_cmd_ec2 "rake resolve_deps"
+    # bring up ec2 instance, bootstrap with chef and provision
+    run_cmd_ec2 "rake up[#{vm_name}]"
+    # re-provison ec2 instance
+    run_cmd_ec2 "rake provision[#{vm_name}]"
+    # show status of ec2 instances
+    run_cmd_ec2 "rake status"
+
+
+    # TODO: test if sample app works
+  ensure
+    # destroy ec2 instance
+    run_cmd_ec2 "rake destroy[#{vm_name}]"
+  end
+end
+
+
+desc "tests bootstrapping with local Vagrant VMs"
+task :test_local_bootstrap do
+
+  vm_name = 'sample-app'
+
+  begin
+    # resolve deps
+    run_cmd_local "rake resolve_deps"
+    # bring up vagrant vm, bootstrap with chef and provision
+    run_cmd_local "rake up[#{vm_name}]"
+    # re-provison vagrant vm
+    run_cmd_local "rake provision[#{vm_name}]"
+    # show status of vagrant vms
+    run_cmd_local "rake status"
+
+    # TODO: test if sample app works
+  ensure
+    # destroy vagrant vm
+    run_cmd_local "rake destroy[#{vm_name}]"
+  end
+end
+
+
+def run_cmd_esx(command)
+  run_cmd command, "test/esx-bootstrap"
+end
+
+def run_cmd_ec2(command)
+  run_cmd command, "test/ec2-bootstrap"
+end
+
+def run_cmd_local(command)
+  run_cmd command, "test/local-bootstrap"
+end
+
+def run_cmd(command, cwd)
+  fail "need #{cwd}/Gemfile for a clean environment" unless File.exist? "#{cwd}/Gemfile"
+  sh "cd #{cwd} && bundle exec #{command}"
+end

data/Vagrantfile

@@ -0,0 +1,33 @@
+#
+# Vagrantfile simulating an ESX-like environment for testing purposes:
+#
+# * assume ssh root access as the only prerequisite
+# * using bare OS baseboxes with no Chef installed
+#
+Vagrant::Config.run do |config|
+
+  # use bare os baseboxes
+  config.vm.box = "ubuntu-12.04-server-amd64-bare-os"
+  config.vm.box_url = "http://dl.dropbox.com/u/13494216/ubuntu-12.04-server-amd64-bare-os.box"
+
+  config.vm.define :esx_like_vm do | esx_like_vm_config |
+    esx_like_vm_config.vm.customize ["modifyvm", :id,
+      "--memory", "1024",
+      "--name", "chef-tlc-workflow-esx-like-vm"]
+    esx_like_vm_config.vm.host_name = "chef-tlc-workflow-esx-like-vm.local"
+    esx_like_vm_config.vm.network :hostonly, "33.33.77.10"
+  end
+
+  #
+  # dummy vm in the same network, keeping it on so that no additional
+  # user elevation dialogs appear for the other VMs
+  #
+  config.vm.define :dummy do | dummy_vm_config |
+    dummy_vm_config.vm.customize ["modifyvm", :id,
+      "--memory", "256",
+      "--name", "chef-tlc-workflow-dummy-vm"]
+    dummy_vm_config.vm.host_name = "chef-tlc-workflow-dummy-vm.local"
+    dummy_vm_config.vm.network :hostonly, "33.33.77.2"
+  end
+
+end

data/chef-tlc-workflow.gemspec

@@ -0,0 +1,52 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'chef-tlc-workflow/version'
+
+Gem::Specification.new do |gem|
+  gem.name          = "chef-tlc-workflow"
+  gem.version       = ChefTLCWorkflow::VERSION
+  gem.authors       = ["Torben Knerr"]
+  gem.email         = ["tkn@zuehlke.com"]
+  gem.description   = %q{makes our workflow for developing with Chef explict by adding a set of Rake tasks embodying the workflow}
+  gem.summary       = %q{makes our workflow for developing with Chef explict by adding a set of Rake tasks embodying the workflow}
+  gem.homepage      = ""
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+
+  # lock down dependencies
+  #
+  gem.add_dependency 'chef-workflow-tasklib', '0.2.2'
+  gem.add_dependency 'chef', '10.18.2'
+  gem.add_dependency 'librarian', '0.0.26.2'
+
+  #
+  # further dependencies that are not `require`d here but we pull in for convenience
+  # so that the only dependency one needs in the `Gemfile` is `chef-tlc-workflow`
+  #
+
+  # for interaction with nodes in local/esx/ec2 environments
+  gem.add_dependency 'vagrant', '1.0.5.1'
+  gem.add_dependency 'knife-solo', '0.3.0.pre2'
+  gem.add_dependency 'mccloud', '0.0.18'
+  
+  # testing related
+  gem.add_dependency 'foodcritic', '1.7.0'
+  gem.add_dependency 'chefspec', '0.9.0'
+  gem.add_dependency 'fauxhai', '0.1.1'
+
+  # others
+  gem.add_dependency 'rake', '0.9.6'
+  gem.add_dependency 'sahara', '0.0.13'
+  gem.add_dependency 'knife-solo_data_bag', '0.3.1'
+
+  # need these on windows only
+  if RUBY_PLATFORM =~ /mswin|mingw/
+    gem.add_dependency 'ruby-wmi', '0.4.0'
+    gem.add_dependency 'win32-service', '0.7.2'
+  end
+
+end

data/docs/ApplicationVsLibraryVsForkedCookbooks.md

@@ -0,0 +1,20 @@
+## Application vs Library vs Forked Cookbooks
+
+[Application cookbooks](http://devopsanywhere.blogspot.ch/2012/11/how-to-write-reusable-chef-cookbooks.html) contain everything required for setting up a machine (1:1 relation) with the desired application/service. They are based on library cookbooks but add the "glue" to make the installation as simple as including just the application cookbook in the node's run_list. 
+
+Why application cookbooks? Because:
+
+ * they can be properly versioned and dependency managed (in contrast to roles)
+ * they lock down dependency versions in their `metadata.rb` thus make the installation robust and reliable
+ * they hide all the glue code (and configuration inderdependencies) when assembling multiple cookbooks 
+ * they set sensible configuration defaults and expose only the minimum necessary configuration to the user
+ * they document how to use/administer the application in their README 
+
+Sometimes there is no suitable [community cookbook](http://community.opscode.com/cookbooks) available or an available cookbook suffers from a bug or a missing feature we urgently need. In this case we resort to either writing our own (maybe internally hosted) library cookbook or to fork the community cookbook and fix the bugs or provide outstanding features.
+
+This is how we handle forked community cookbooks:
+
+ * host them internally on our Gitlab
+ * `master` branch is the master from the upstream version
+ * all our changes are done in the `zuehlke_master` branch
+ * bug fixes / useful features should be contributed back to the community via pull requests

data/docs/TmpLibrarianHelpers.md

@@ -0,0 +1,21 @@
+
+## Librarian Helpers
+
+Since librarian does not support reading the cookbook dependencies from `metadata.rb`,
+the `ChefTLCWorkflow::Helpers` module adds this functionality.
+
+In the trivial case you can use it in your cookbook project's `Cheffile` like so:
+
+    require 'chef-tlc-workflow/helpers'
+
+    ChefTLCWorkflow::Helpers::from_metadata.each do |cb_name, cb_version|
+      cookbook cb_name, cb_version
+    end
+
+If some of the cookbooks defined in metadata.rb are not available from the community site, 
+you can provide a .yml file with custom location mappings and pass it to the helper method: 
+
+    ...
+    ChefTLCWorkflow::Helpers::from_metadata('../../locations.yml').each do |cb_name, cb_version, location|
+      cookbook cb_name, cb_version, location
+    end

data/docs/TmpWorkflowTasks.md

@@ -0,0 +1,80 @@
+The `chef-tlc-workflow` makes our workflow for developing with Chef explict by adding a set of Rake tasks embodying the workflow. It's an extension of [chef-workflow](https://github.com/chef-workflow/chef-workflow) providing the additional workflow-encapsulating Rake tasks in the `tlc` namespace.
+
+## Workflow Tasks
+
+-------------------------
+
+    *!!! CAUTION: the text below about workflow tasks is not accurate anymore !!!*
+
+    Why? Because I believe its better to provide a Rakefile via Scaffolding mechanism. 
+    This is more transparent and easier to adapt than providing prebaked Rake tasks 
+    in a gem library.
+
+    Right now we still have the Rake tasks mentioned below, but we treat them as internal
+    Rake tasks that we invoke from the scaffolded Rakefiles. They don't show up when 
+    running `rake -T` because they have now `desc`ription (but `rake -T -A` still shows them).
+
+    Even if we keep them, the namespaces are likely to change so they are structured 
+    per-project-type (e.g. library cookbook project, application cookbook project or
+    infrastructure project), e.g:
+
+    * lib
+     * create
+     * release
+     * resolve_deps
+     * validate (Cheffile in .gitignore)
+     * test (u.a. => test:converge, ggf. chef-workflow testlib?)
+    * app
+     * create
+     * release
+     * resolve_deps
+     * validate (=> u.a. deps:check, Cheffile nicht in .gitignore)
+     * test (was auch immer das heisst)
+    * infra
+     * create
+     * resolve_deps (= app cookbooks)
+     * validate
+     * test (was auch immer das heisst)
+
+    Furthermore, the scaffolding tasks should be rather implemented as a knife plugin, 
+    because when we use them there is no Rakefile yet (chicken-egg problem).
+
+-------------------------
+
+Once you installed the gem or added it to your `Gemfile` as shown above, simply add the following to your project's `Rakefile` to have all TLC workflow tasks available:
+
+    # first, chef-workflow-tasklib must be required
+    require 'chef-workflow-tasklib'
+
+    # then you can include _all_ tlc tasks like this
+    chef_workflow_task 'tlc'
+
+Or, if you need only specific workflow tasks, they can be included separately:
+
+    ...
+    # include dependency management related tasks
+    chef_workflow_task 'tlc/deps'
+
+You should then have the TLC workflow tasks available: 
+
+    $ bundle exec rake -T
+    rake tlc:deps:check     # check if dependencies in metadata.rb and Cheffi...
+    rake tlc:deps:resolve   # resolve dependencies using librarian
+    rake tlc:test:converge  # destroy the Vagrant VM, resolve dependencies an...
+
+### Available Workflow Tasks
+
+Currently available are these tasks.
+
+#### Cookbook Dependency Management
+
+The `deps` namespace embodies the workflow concerning cookbook dependency management
+  
+ * `tlc:deps:resolve` - resolve dependencies using [librarian](https://github.com/applicationsonline/librarian)
+ * `tlc:deps:check` - check if dependencies in `metadata.rb` and `Cheffile` are consistent
+
+#### Cookbook Testing
+
+The `test` namespace represents the workflow for testing of individual cookbook projects
+
+ * `tlc:test:converge` - destroy the default Vagrant VM first, then resolve dependencies and converge the default Vagrant VM