vagabond 0.2.0 → 0.2.2

data/CHANGELOG.md

@@ -1,3 +1,21 @@
+## v0.2.2
+* Migration to elecksee gem for LXC management
+* Addition of chef-zero support for local chef server
+* Updated testing for standalone cookbook testing
+* Updated support for dependency resolvers (librarian and berkshelf)
+* Addition of `spec` support
+* Better isolation around sudo usage
+* Cleaned `internal_configuration` and added automatic reloading on detected changes
+* Added `cluster` support to Vagabondfile for easily building multiple nodes
+* New `init` command for base setup
+* Lots and lots of bug fixes and feature enhancements
+* Output relevant information on bad command
+* Huge thanks to all those that helped test, debug, and add features especially:
+  * Jesse Nelson - https://github.com/spheromak
+  * Bryan Berry - https://github.com/bryanwb
+  * Sean Escriva - https://github.com/webframp
+
+
 ## v0.2.0
 * Migrate to thor
 * Clean up option usage

data/README.md

@@ -15,8 +15,8 @@ instead of full blown VMs which means things are faster. Lots
 faster.
 
 Vagabond is built for Chef. The tooling within Vagabond is targeted
-at Chef. As the initial development push on Vagabond slows, this
-tooling will become optionally swappable (applicable bits at least).
+at Chef. After the initial development has slowed, the Chef specifics
+will be pulled into a plugin.
 
 ## Installation
 
@@ -29,7 +29,7 @@ $ gem install vagabond
 ## How does it work
 
 Currently, this is built to run within a classic Chef repository.
-It requires a `Vagabond` file, that simply outputs a Hash. The file
+It requires a `Vagabondfile` file, that simply outputs a Hash. The file
 is Ruby though, so you can do lots of crazy stuff to build the
 Hash you return. Heres a simple example:
 
@@ -58,6 +58,23 @@ Now, to create a node, simply run:
 $ vagabond up db
 ```
 
+This command will bootstrap the installation of LXC utilities and base
+containers prior to starting up a linux container. It does this by
+running the vagabond chef recipe embedded in this gem at
+`lib/vagabond/cookbooks/vagabond/recipes/default.rb`. 
+
+To only prepare your system for LXC fun and generate a simple vagabond
+file, do the following:
+
+```
+$ vagabond init
+```
+
+This command runs the chef recipe and generates a basic Vagabondfile and
+creates the base containers specified in that file. Those base
+containers are Ubuntu 12.04 and CentOS 6.3. This creation process will
+take a while so now is a good time to get a fresh pot of coffee.
+
 Pretty simple, right?
 
 ### Templates available
@@ -131,6 +148,103 @@ chef server. Looking for example? See the `USAGE` file!
 
 Double awesome
 
+## Infrastructure testing
+
+Cookbook tests are great and they help keep cookbooks stable and prevent
+regressions. But what about tests for integrating cookbooks into an existing
+infrastructure? Or upgrading an existing cookbook? The tests bundled with
+the cookbook can happily pass with no indication of how it may affect other
+resources within the infrastructure. So lets fix this.
+
+Currently infrastructure tests are built using serverspec[1]. Test kitchen support
+for infrastructure testing is in the works, but is still a moving target. So
+lets look at how we can set this up. First, initialize the specs:
+
+```
+$ vagabond spec init
+```
+
+Next, we need to define the layout of the infrastructure. This is done by
+populating the `Layout` file in the `spec` directory. Just like everything
+else, this is just a ruby file that is expected to spit out a Hash. An
+example file would look like this:
+
+```ruby
+# spec/Layout
+{
+  :defaults => {
+    :platform => 'ubuntu_1204',
+    :environment => 'testing',
+    :union => 'aufs'
+  },
+  :definitions => {
+    :test_node => {
+      :run_list => %w(role[base])
+    }
+  },
+  :clusters => {
+    :my_cluster => {
+      :overrides => {
+        :environment => '_default'
+      },
+      :nodes => ['test_node'] * 3
+    }
+  }
+}
+```
+
+### :defaults
+
+These are the default configuration options used for creating the containers
+for testing. These are the same configurations used when creating the nodes
+in the Vagabondfile.
+
+### :definitions
+
+These are the definitions of your nodes. Any options here that were defined
+within the `:defaults` section will be overridden.
+
+### :clusters
+
+These are the clusters of nodes that describe the infrastructure. The key 
+provides the identifier name used from the `spec` command. The `:overrides`
+are cluster specific overrides that are applied to all nodes when created.
+The `:nodes` is an array of `:definitions` keys for nodes to build. The
+keys can be repeated `n` times to provide multiple nodes of a specific type.
+
+### Usage
+
+```
+$ vagabond spec my_cluster
+```
+
+### Applying specs
+
+Specs are applied based on the run list describing the node. After the local
+chef server has been created (if required), all nodes have been created, and
+all nodes provisioned vagabond will run back through all nodes applying the
+applicable specs. Specs are very straight forward and only use SSH connections
+to spec the node. An example spec:
+
+```ruby
+require 'spec_helper'
+
+describe 'cron' do
+  it{ should be_enabled }
+  it{ should be_running }
+end
+```
+
+### Spec real infrastructure
+
+Since specs only require an SSH connection to test nodes, we can run specs
+against actual live infrastructure to see if it is currently in a valid
+state based on existing specs. Awesome!
+
+```
+$ vagabond spec my_cluster --environment production
+```
+
 ## Important note
 
 Until namespaces hit Linux proper, vagabond `sudo`s its way around. You
@@ -163,6 +277,13 @@ get help making things better!
 * Base updates and pull requests on the `develop` branch
 * Please don't update core files like `version.rb` or `vagabond.gemspec`
 
+## Contributors
+
+* [Bryan Berry](https://github.com/bryanwb)
+* [Jesse Nelson](https://github.com/spheromak)
+* [Sean Escriva](https://github.com/webframp)
+* [Xabier de Zuazo](https://github.com/zuazo)
+
 ## Infos
 
-* Repository: https://github.com/chrisroberts/vagabond
+* Repository: https://github.com/chrisroberts/vagabond

data/bin/vagabond

@@ -4,25 +4,52 @@ Signal.trap('INT'){ exit 255 }
 
 require 'rubygems'
 require 'vagabond'
+require 'vagabond/errors'
 
 if(ARGV.include?('--version') || ARGV.include?('-v'))
   require 'vagabond/vagabond'
   Vagabond::Vagabond.new.send(:version)
+elsif(%w(help --help -h).include?(ARGV.first))
+  puts "-- Vagabond --\n"
+  require 'vagabond/vagabond'
+  Object.send(:remove_const, :ARGV)
+  ARGV = %w(help)
+  Vagabond::Vagabond.start
+  %w(server knife kitchen spec).each do |key|
+    puts "-- #{key.capitalize} --\n"
+    require "vagabond/#{key}"
+    Object.send(:remove_const, :ARGV)
+    ARGV = %w(help)
+    Vagabond.const_get(key.capitalize).start
+  end
 else
-  case arg = ARGV.shift
-  when 'server'
-    require 'vagabond/server'
-    Vagabond::Server
-  when 'knife'
-    require 'vagabond/knife'
-    ARGV.unshift(arg) unless ARGV.empty?
-    Vagabond::Knife
-  when 'test', 'kitchen'
-    require 'vagabond/kitchen'
-    Vagabond::Kitchen
-  else
-    require 'vagabond/vagabond'
-    ARGV.unshift(arg)
-    Vagabond::Vagabond
-  end.start
+  begin
+    case arg = ARGV.shift
+    when 'server'
+      require 'vagabond/server'
+      Vagabond::Server
+    when 'knife'
+      require 'vagabond/knife'
+      ARGV.unshift(arg) unless ARGV.empty?
+      Vagabond::Knife
+    when 'test', 'kitchen'
+      require 'vagabond/kitchen'
+      Vagabond::Kitchen
+    when 'spec'
+      require 'vagabond/spec'
+      Vagabond::Spec
+    else
+      require 'vagabond/vagabond'
+      ARGV.unshift(arg)
+      Vagabond::Vagabond
+    end.start
+  rescue Vagabond::VagabondError => e
+    exit e.exit_code
+  rescue Exception => e
+    $stderr.puts "Vagabond exiting. Reason: #{e}"
+    if(ENV['VAGABOND_EXIT_DEBUG'])
+      puts "#{e.class}: #{e}\n#{e.backtrace.join("\n")}"
+    end
+    exit -1
+  end
 end

data/lib/vagabond/actions/cluster.rb

@@ -0,0 +1,66 @@
+module Vagabond
+  module Actions
+    module Cluster
+      class << self
+        def included(klass)
+          klass.class_eval do
+            class << self
+              def _cluster_options
+                [[
+                    :auto_provision, :type => :boolean,
+                    :desc => 'Automatically provision nodes', :default => true
+                  ],
+                  [
+                    :delay, :type => :numeric, :default => 0,
+                    :desc => 'Add delay between provisions (helpful for indexing)'
+                  ],
+                  [
+                    :parallel, :type => :boolean, :default => false,
+                    :desc => 'Build nodes in parallel'
+                  ]
+                ]
+              end
+            end
+          end
+        end
+      end
+
+      def _cluster
+        clr = vagabondfile[:clusters][name] if vagabondfile[:clusters]
+        if(clr)
+          ui.info "#{ui.color('Vagabond:', :bold)} Building cluster - #{ui.color(name, :green)}"
+          if(vagabondfile[:local_chef_server] && vagabondfile[:local_chef_server][:enabled])
+            require 'vagabond/server'
+            srv = ::Vagabond::Server.new
+            srv.send(:setup, 'up')
+            srv.execute
+            # Reload so we get proper values
+            load_configurations
+          end
+          cluster_instances = clr.map do |n|
+            ui.info "Building #{n} for cluster!"
+            v_inst = Vagabond.new
+            v_inst.options = options.dup
+            v_inst.send(:setup, 'up', n, :ui => ui)
+            v_inst.execute
+            if(options[:delay].to_i > 0 && n != clr.last)
+              ui.warn "Delay requested between node processing. Sleeping for #{options[:delay].to_i} seconds."
+              sleep(options[:delay].to_i)
+            end
+            v_inst
+          end
+          if(options[:parallel])
+            ui.info "Waiting for parallel completes!"
+            cluster_instances.map do |inst|
+              inst.wait_for_completion
+            end
+          end
+          ui.info "  -> #{ui.color("Built cluster #{name}", :green)}"
+        else
+          ui.error "Cluster name provided does not exist: #{name}"
+        end
+      end
+
+    end
+  end
+end

data/lib/vagabond/actions/create.rb

@@ -1,3 +1,5 @@
+require 'elecksee/ephemeral'
+
 module Vagabond
   module Actions
     module Create
@@ -21,14 +23,17 @@ module Vagabond
           tmpl = internal_config[:template_mappings][tmpl]
         elsif(!BASE_TEMPLATES.include?(tmpl))
           ui.fatal "Template requested for node does not exist: #{tmpl}"
-          exit EXIT_CODES[:invalid_template]
+          raise VagabondError::InvalidTemplate.new(tmpl)
+        end
+        unless(config[:device])
+          config[:directory] = true
         end
-        bind_path = File.expand_path(File.dirname(vagabondfile.path))
-        com = "#{sudo}lxc-start-ephemeral -d -b #{bind_path} -o #{tmpl}"
-        debug(com)
-        c = Mixlib::ShellOut.new("#{com} && sleep 3", :live_stream => options[:debug])
-        c.run_command
-        e_name = c.stdout.split("\n").last.split(' ').last.strip
+        config[:daemon] = true
+        config[:original] = tmpl
+        config[:bind] = File.expand_path(File.dirname(vagabondfile.store_path))
+        ephemeral = Lxc::Ephemeral.new(config)
+        ephemeral.start!(:fork)
+        e_name = ephemeral.name
         @internal_config[mappings_key][name] = e_name
         @internal_config.save
         @lxc = Lxc.new(e_name)

data/lib/vagabond/actions/destroy.rb

@@ -2,27 +2,76 @@ module Vagabond
   module Actions
     module Destroy
 
+      class << self
+        def included(klass)
+          klass.class_eval do
+            class << self
+              def _cluster_options
+                [
+                  [
+                    :cluster, :type => :boolean,
+                    :desc => 'Destroy cluster of nodes with provided name', :default => false
+                  ]
+                ]
+              end
+            end
+          end
+        end
+      end
+
+      
       def _destroy
         name_required!
-        if(lxc.exists?)
-          ui.info "#{ui.color('Vagabond:', :bold)} Destroying node: #{ui.color(name, :red)}"
-          do_destroy
-          ui.info ui.color('  -> DESTROYED', :red)
+        if(options[:cluster])
+          nodes = vagabondfile[:clusters][name] if vagabondfile[:clusters]
+          if(nodes)
+            ui.info "#{ui.color('Vagabond:', :bold)} Destroying cluster - #{ui.color(name, :red)}"
+          else
+            ui.error "Cluster name provided does not exist: #{name}"
+            nodes = []
+          end
         else
-          ui.error "Node not created: #{name}"
+          nodes = [name, @leftover_args].flatten.compact
+        end
+        nodes.each do |n|
+          next unless n.is_a?(String)
+          @name = n
+          load_configurations
+          if(lxc.exists?)
+            ui.info "#{ui.color('Vagabond:', :bold)} Destroying node: #{ui.color(name, :red)}"
+            do_destroy
+            ui.info ui.color('  -> DESTROYED', :red)
+          else
+            ui.error "Node not created: #{name}"
+          end
         end
       end
 
       private
 
       def do_destroy
-        lxc.stop if lxc.running?
-        com = "#{options[:sudo]}lxc-destroy -n #{lxc.name}"
-        debug(com)
-        cmd = Mixlib::ShellOut.new(com, :live_stream => options[:debug])
-        cmd.run_command
-        cmd.error!
-        if(cmd.stderr.include?('skipping'))
+        lxc.shutdown if lxc.running?
+        ui.info 'Waiting for graceful shutdown and cleanup...'
+        5.times do
+          break unless lxc.exists?
+          sleep(1)
+        end
+        if(lxc.exists?)
+          com = "#{options[:sudo]}lxc-destroy -n #{lxc.name}"
+          debug(com)
+          cmd = Mixlib::ShellOut.new(com, :live_stream => options[:debug])
+          cmd.run_command
+          force_umount_if_required!
+        end
+        internal_config[mappings_key].delete(name)
+        internal_config.save
+      end
+
+      def force_umount_if_required!
+        mount = %x{mount}.split("\n").find_all do |line|
+          line.include?(lxc.name)
+        end
+        unless(mount.empty?)
           ui.info ui.color('  -> Failed to unmount some resources. Forcing manually.', :yellow)
           %w(rootfs ephemeralbind).each do |mnt|
             com = "#{options[:sudo]}umount /var/lib/lxc/#{lxc.name}/#{mnt}"
@@ -33,10 +82,15 @@ module Vagabond
             debug(com)
             cmd = Mixlib::ShellOut.new(com, :live_stream => options[:debug])
             cmd.run_command
-            cmd.error!
-            internal_config[mappings_key].delete(name)
           end
-          internal_config.save
+          # check for tmpfs and umount too
+          tmp = mount.detect{|x|x.include?('rootfs')}.scan(%r{upperdir=[^,]+}).first.to_s.split('=').last
+          if(tmp)
+            com = "#{options[:sudo]}umount #{tmp}"
+            debug(com)
+            cmd = Mixlib::ShellOut.new(com, :live_stream => options[:debug])
+            cmd.run_command
+          end
         end
       end
     end