auser-poolparty 1.2.12 → 1.3.0

data/README.rdoc

@@ -0,0 +1,202 @@
+= PoolParty
+
+http://poolpartyrb.com
+
+== DESCRIPTION:
+
+PoolParty makes cloud provisioning and management easy. PoolParty provides a unified interface for provisioning, management and cloud life-cycle on many different cloud providers, such as EC2 and vmware.
+
+Code your cloud! 
+
+== SYNOPSIS:
+
+PoolParty is written with the intention of being as application-agnostic as possible. It installs only the basic 
+required software to glue the cloud together on the instances as listed below.
+
+== Quickstart
+
+For instance, to start a basic cloud, let's write one:
+
+  pool "demo" do
+    cloud "app" do
+      instances 2..10
+      keypair "cloud_demo"
+      
+      using :ec2
+      
+      has_file "/etc/motd", :content => "Hello from your cloud"
+    end
+  end
+  
+Simply by issuing the command:
+  
+  cloud start
+  
+The app cloud in the demo pool will be started, bootstrapped and provisioned to your liking. If the platform is not specified, then it's assumed to be ubuntu. However, if a node is running, then PoolParty will introspect the node and determine the operating system. This can be specified in the clouds.rb.
+
+There are a number of commands PoolParty offers to interact with your cloud. They include:
+
+* <tt>cloud start</tt>
+* <tt>cloud terminate</tt>
+* <tt>cloud bootstrap</tt>
+* <tt>cloud configure</tt>
+* <tt>cloud compile</tt>
+* <tt>cloud console</tt>
+* <tt>cloud expand</tt>
+* <tt>cloud contract</tt>
+* <tt>cloud list</tt>
+* <tt>cloud show</tt>
+* <tt>cloud ssh</tt>
+
+There is also a helper installer script. This is an easy way to get started bootstrapping your environment:
+
+* <tt>install-poolparty</tt>
+
+We can change the cloud provider simply by changing the using line. For instance:
+
+  pool "demo" do
+    cloud "app" do
+      #...
+      using :rackspace # coming soon
+      #...
+
+Clouds are distinguished by keypairs, so to create multiple clouds, create a new keypair. For instance:
+
+  pool "demo" do
+    cloud "app" do
+      keypair "demo_app"
+      #...
+    end
+    cloud "db" do
+      keypair "demo_db"
+      #...
+    end
+  end
+  
+== Resources
+
+As you probably noticed from above, you provision your system with resources. There are native resources, plugins (which are basically resources) and you can write your own. The base resources included in PoolParty are as follows:
+
+* <tt>file</tt>
+* <tt>directory</tt>
+* <tt>exec</tt>
+* <tt>variable</tt>
+* <tt>service</tt>
+* <tt>package</tt>
+* <tt>user</tt>
+* <tt>group</tt>
+* <tt>link (symlink)</tt>
+* <tt>line (line in file)</tt>
+* <tt>gem_package</tt>
+* <tt>cron</tt>
+
+There are also dependency_resolver specific resources. For instance, **chef** specific resources are as follows:
+
+* <tt>chef_attribute</tt>
+* <tt>http_request</tt>
+* <tt>remote_directory</tt>
+* <tt>remote_file</tt>
+* <tt>route</tt>
+* <tt>script</tt>
+
+To use these resources in your config file (clouds.rb), simply "assert" (call) the method with has_ or does_not_have prepended. For instance:
+
+  pool "demo" do
+    cloud "app" do
+      has_file "/etc/motd", :requires => get_package("apache2")
+      has_package "apache2" do
+        action :install
+      end
+    end
+  end
+  
+As you can see, there are many different ways to call a resource. All the following resource calls are identical:
+
+  has_file "/etc/motd", :content => "Hello world", :owner => "ari"
+  has_file :name => "/etc/motd", :content => "Hello world", :owner => "ari"
+  has_file "/etc/motd" do
+    content "Hello world"
+    owner "ari"
+  end
+  has_file "/etc/motd", :content => "Hello world" do
+    owner "ari"
+  end
+  
+All resources can contain their own resources as well. This sets up a dependency on the parent dependency. For instance
+
+  has_directory "/etc/configs" do
+    has_file "/etc/configs/configger", :content => "Stuff"
+  end
+
+This says that the directory must exist before the file can be created.
+
+You can see the dependency graph setup by your clouds.rb by calling
+
+  cloud compile -g output
+  
+That will generate a output.dot and output.png in the current directory. For this to work, you must have dot installed. Try it!
+
+Resources can be built on top of other resources as well. This is the definition of a resource plugin. This is covered more in-depth on the homepage. For the curious, take a look at lib/poolparty/plugins/apache.rb for an example plugin. 
+
+There are currently several plugins that ship with PoolParty. Those include:
+
+* <tt>apache (this includes passenger, virtual_hosts, php5)</tt>
+* <tt>git</tt>
+* <tt>rails</tt>
+
+This is list likely to expand as plugins are simply resources built on top of other resources. There is an external project called poolparty-extensions (<a href="http://github.com/auser/poolparty-extensions/tree/master">http://github.com/auser/poolparty-extensions/tree/master</a>) that has quite a few external plugins, including hadoop, ganglia and more.
+
+== Extending
+
+To add a cloud_provider, there are four methods that need to be implemented. Simply sublcass the CloudProviders module and require it in your clouds.rb (or commit it back to PoolParty). Those four methods are:
+
+* <tt>run_instance</tt>
+* <tt>terminate_instance</tt>
+* <tt>describe_instances</tt>
+* <tt>describe_instance</tt>
+
+That's it!
+
+Each resource is documented in the code and more documentation can be found at the site: <a href="http://poolpartyrb.com">http://poolpartyrb.com</a>.
+
+== FEATURES/PROBLEMS:
+
+* Written in Ruby and Erlang (Internal node communication handled by <a href="https://github.com/auser/hermes/tree/master">Hermes</a>)
+* Thrift interface (Connect to your cloud in ANY language supported by <a href="http://incubator.apache.org/thrift/">Thrift</a>)
+* Written from the ground up to be extensible with plugins
+* Easy git-style commands to communicate with your clouds
+* Much much more
+
+== REQUIREMENTS:
+
+* Ruby
+* Erlang
+
+== INSTALL:
+
+sudo gem install auser-poolparty
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2009 Ari Lerner, CloudTeam
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -2,7 +2,7 @@ require 'config/requirements'
 
 begin
   require 'hanna/rdoctask'
-rescue Exception => e
+rescue LoadError => e
   require "rake/rdoctask"
 end
 
@@ -26,31 +26,17 @@ task :clean_pkg do |t|
   end
 end
 
-desc "Packge with timestamp"
-task :update_timestamp do
-  data = open("PostInstall.txt").read
-  str = "Updated at #{Time.now.strftime("%H:%M %D")}"
-  
-  if data.scan(/Updated at/).empty?
-    data = data ^ {:updated_at => str}    
-  else
-    data = data.gsub(/just installed PoolParty\!(.*)$/, "just installed PoolParty! (#{str})")
-  end
-  ::File.open("PostInstall.txt", "w+") {|f| f << data }
-end
 
 namespace :gem do
   task(:build).prerequisites.unshift :gemspec # Prepend the gemspec generation
   
-  desc "Build the gem only if the specs pass"
-  task :test_then_build => [:spec, :build]
+  desc "Build the gem only if the tests pass"
+  task :test_then_build => [:test, :build]
   
-  desc "Build and install the gem only if the specs pass"
-  task :test_then_install => [:spec, :install]
+  desc "Build and install the gem only if the tests pass"
+  task :test_then_install => [:test, :install]
 end
 
-task :release => [:update_timestamp]
-
 # Generate documentation
 Rake::RDocTask.new do |rd|
   rd.main = "Readme.txt"

data/VERSION.yml

@@ -1,4 +1,4 @@
 --- 
-:minor: 2
-:patch: 12
 :major: 1
+:minor: 3
+:patch: 0

data/bin/cloud

@@ -1,13 +1,16 @@
 #!/usr/bin/env ruby
 $:.unshift(File.dirname(__FILE__) + "/../lib")
 require "poolparty"
-# require "poolpartycl"
+require "colors"
+require "mixins/printing"
 require 'git-style-binary/command'
 
 GitStyleBinary.primary do
+  self.class.send :include, Printing
+  
   @theme = :short
   
-  version "PoolParty cloud command"  
+  version "PoolParty release: #{PoolParty.version}"
   banner <<-EOS
 Usage: #{$0} #{all_options_string} COMMAND [ARGS]
 
@@ -16,22 +19,25 @@ The cloud subcommands commands are:
 
 See 'cloud help COMMAND' for more information on a specific command.
 EOS
-  
+    
+  opt :clouds_dot_rb, "Set the clouds.rb file", :type => String, :default => "clouds.rb"
+  opt :very_verbose, "Set very verbose mode on", :type => :boolean, :default => false
   opt :debug, "Debug the output", :type => :boolean, :default => false
-  opt :spec, "Set the pool spec file", :type => String
-  opt :testing, "Set testing mode on", :type => :boolean, :default => false
-  opt :cloud_name, "Name of the working cloud", :type => String, :default => nil
+  opt :very_debug, "Set very debug mode on", :type => :boolean, :default => false
+  opt :name, "Name of the working cloud", :type => String, :default => nil
 
   before_run do |command|
     # Setup testing/debugging
-    $TESTING = true if command[:testing]
-    $DEBUGGING = true if command[:debug]
+    $TESTING        = true if command[:verbose]
+    $VERY_VERBOSE   = true if command[:very_verbose]
+    $DEBUGGING      = true if command[:debug]
+    $VERY_DEBUGGING = true if command[:very_debug]
     
-    @loaded_pool = load_pool( command[:spec] ? command[:spec] : Binary.get_existing_spec_location)
-    @loaded_clouds = command[:cloud_name] ? [clouds[command[:cloud_name].to_sym]] : clouds.map {|k,c| c }
-    @loaded_clouds.each do |cld|
-      cld.verbose command[:verbose]
-      cld.debugging = command[:debug]
+    @loaded_pool = PoolParty::Pool.find_and_load_default_clouds_dot_rb(command[:clouds_dot_rb])
+    @loaded_clouds = command[:name] ? [clouds[command[:name]]] : @loaded_pool.clouds.map {|name,cld|cld}
+    @loaded_clouds.map do |cld| 
+      cld.verbose = command[:verbose]
+      cld.debug = command[:debug]
     end
   end
   
@@ -43,6 +49,6 @@ EOS
     The cloud subcommands commands are:
       #{subcommands}
 
-    See 'cloud help COMMAND' for more information on a specific command."
+    See 'cloud help COMMAND' for more information on a specific command"
   end
 end

data/bin/cloud-bootstrap

@@ -1,11 +1,10 @@
 #!/usr/bin/env ruby
 $:.unshift(File.dirname(__FILE__) + "/../lib")
 require "poolparty"
-# require "poolpartycl"
+
 require 'git-style-binary/command'
 
-GitStyleBinary.command do  
-  version "PoolParty #{$0} command"  
+GitStyleBinary.command do
   banner <<-EOS
 Usage: #{$0} #{all_options_string}
 
@@ -15,25 +14,15 @@ EOS
   short_desc "Bootstrap a cloud instance"
   
   opt :inst_num, "The number of the instance to run bootstrap on", :type => :int
+  opt :force, "Force rebootstrapping (useful for upgrading)", :default => false
 
-  run do |command|    
+  run do |command|
     
     @loaded_clouds.each do |cld|
-
-        # If an IP or DNS name is given, bootstrap that node, otherwise, bootstrap all running nodes.
-        if cld.nodes(:status => "running").empty?
-          puts "No nodes to bootstrap"
-        else
-          if command[:inst_num]
-            vputs "\nBootStrapping: #{nodes[o.inst_num]}\n--------------------"
-            ::PoolParty::Provision::BootStrapper.new cld.nodes(:status => "running")[ command[:inst_num] ], :cloud => cld
-          else
-            cld.nodes(:status => "running").each do |address|
-              cld.vputs "\nBootStrapping: #{address[:ip]}\n--------------------"      
-              ::PoolParty::Provision::BootStrapper.new address[:ip], :cloud => cld
-            end
-          end
-        end
+      
+      cld.nodes.each do |node|
+        node.bootstrap!(:force=>command[:force])
+      end
       
     end
     

data/bin/cloud-compile

@@ -0,0 +1,28 @@
+#!/usr/bin/env ruby
+$:.unshift(File.dirname(__FILE__) + "/../lib")
+require "poolparty"
+
+require 'git-style-binary/command'
+
+GitStyleBinary.command do
+  banner <<-EOS
+Usage: #{$0} #{all_options_string}
+
+  Configure a cloud instance
+EOS
+
+  short_desc "Configure a cloud instance"
+  
+  opt :inst_num, "The number of the instance to run bootstrap on", :type => :int
+
+  run do |command|
+    
+    @loaded_clouds.each do |cld|
+      
+      cld.compile
+      cld.output_resources_graph('png', "graph", {"fontsize" => 30})
+      
+    end
+    
+  end
+end

data/bin/cloud-configure

@@ -1,11 +1,10 @@
 #!/usr/bin/env ruby
 $:.unshift(File.dirname(__FILE__) + "/../lib")
 require "poolparty"
-# require "poolpartycl"
+
 require 'git-style-binary/command'
 
-GitStyleBinary.command do  
-  version "PoolParty #{$0} command"  
+GitStyleBinary.command do
   banner <<-EOS
 Usage: #{$0} #{all_options_string}
 
@@ -15,26 +14,20 @@ EOS
   short_desc "Configure a cloud instance"
   
   opt :inst_num, "The number of the instance to run bootstrap on", :type => :int
+  opt :generate_graph, "Generate the dependency tree graph", :type => :string, :default => nil
 
-  run do |command|    
+  run do |command|
     
     @loaded_clouds.each do |cld|
-        
-        # If an IP or DNS name is given, bootstrap that node, otherwise, bootstrap all running nodes.
-        if cld.nodes(:status => "running").empty?
-          puts "No nodes to bootstrap"
-        else
-          if command[:inst_num]
-            vputs "\nConfiguring: #{cld.nodes[o.inst_num]}\n--------------------"
-            address = cld.nodes(:status => "running")[ command[:inst_num] ]
-            ::PoolParty::Provision::DrConfigure.new( address[:ip], :cloud => cld )
-          else
-            cld.nodes(:status => "running").each do |address|
-              cld.vputs "\nConfiguring: #{address[:ip]}\n--------------------"      
-              ::PoolParty::Provision::DrConfigure.new address[:ip], :cloud => cld
-            end
-          end
-        end
+      
+      if command[:generate_graph]
+        vputs "Generating dependency graph at: #{command[:generate_graph]}"
+        cld.output_resources_graph('png', command[:generate_graph], {"fontsize" => 30})
+      end
+      
+      cld.nodes.each do |node|
+        node.configure!(:cloud => cld)
+      end
       
     end
     

data/bin/cloud-console

@@ -1,11 +1,10 @@
 #!/usr/bin/env ruby
 $:.unshift(File.dirname(__FILE__) + "/../lib")
 require "poolparty"
-# require "poolpartycl"
+
 require 'git-style-binary/command'
 
-GitStyleBinary.command do  
-  version "PoolParty #{$0} command"  
+GitStyleBinary.command do
   banner <<-EOS
 Usage: #{$0} #{all_options_string}
 
@@ -24,10 +23,13 @@ EOS
     # Perhaps use a console_lib to store any extra methods I may want available in the cosole
     # libs << " -r #{File.dirname(__FILE__) + '/../lib/console_lib/console_logger.rb'}"
     libs <<  " -r #{File.dirname(__FILE__) + '/../lib/poolparty.rb'}"
-    libs <<  " -r #{File.dirname(__FILE__) + '/../lib/poolpartycl.rb'}"
-    libs <<  " -r #{$pool_specfile}" if $pool_specfile
+    f = command[:clouds_dot_rb]
+    libs <<  " -r " + if File.file?(f)
+      f
+    else
+      "#{PoolParty::Pool.find_default_clouds_dot_rb('clouds.rb')}"
+    end
     puts "Loading PoolParty console..."
-
     exec "#{irb} #{libs} --simple-prompt"
     
   end

data/bin/cloud-contract

@@ -1,11 +1,10 @@
 #!/usr/bin/env ruby
 $:.unshift(File.dirname(__FILE__) + "/../lib")
 require "poolparty"
-# require "poolpartycl"
+
 require 'git-style-binary/command'
 
 GitStyleBinary.command do  
-  version "PoolParty #{$0} command"  
   banner <<-EOS
 Usage: #{$0} #{all_options_string}
 
@@ -16,18 +15,23 @@ EOS
 
   run do |command|    
     @loaded_clouds.each do |cld|
-
-      puts "Contracting cloud #{cld.name}"
-      puts "#{cld.nodes(:status => "running").size} running instances (#{cld.minimum_instances} minimum instances)"
-      puts testing ? "Not contracting (test mode)" : "Contracting the cloud if possible"
       
-      if cld.minimum_instances < cld.nodes(:status => "running").size
-        last_instance_id = cld.nodes(:status => "running").last[:instance_id]
-        cld.remote_base.terminate_instance! cld.dsl_options.merge(:instance_id => last_instance_id)
-      else
-        puts "The cloud #{cld.name} is running minimum instances (#{cld.nodes(:status => "running").size})"
-      end
-
+      node_to_terminate = cld.nodes.last
+      
+      msg = [
+        "Contracting cloud #{cld.name} (#{cld.keypair}) by 1",
+        " node: #{node_to_terminate.public_ip}"
+      ]
+      
+      print_msg(msg)
+      
+      node_to_terminate.terminate!
+      
+      msg = [
+        "#{node_to_terminate.public_ip} has been terminated"
+      ]
+      
+      print_msg(msg)
     end    
   end
 end

data/bin/cloud-expand

@@ -1,11 +1,10 @@
 #!/usr/bin/env ruby
 $:.unshift(File.dirname(__FILE__) + "/../lib")
 require "poolparty"
-# require "poolpartycl"
+
 require 'git-style-binary/command'
 
 GitStyleBinary.command do  
-  version "PoolParty #{$0} command"  
   banner <<-EOS
 Usage: #{$0} #{all_options_string}
 
@@ -16,24 +15,24 @@ EOS
 
   run do |command|    
     @loaded_clouds.each do |cld|
+      
+      msg = [
+        "Expanding cloud #{cld.name} (#{cld.keypair})",
+        "#{cld.nodes.size} running instances (#{cld.minimum_instances} - #{cld.maximum_instances})"
+      ]
+      
+      print_msg(msg)
+      
+      inst = cld.expand
+      inst.bootstrap!
+      
+      msg = [
+        "Your cloud has expanded by node: #{inst.public_ip}"
+      ]
+      
+      print_msg(msg)
 
-        vputs "Expanding cloud #{cld.name}"
-        vputs "#{cld.nodes(:status => "running").size} running instances of #{cld.maximum_instances} possible instances"
-        vputs testing ? "Not expanding (test mode)" : "Expanding the cloud"
-        
-        if cld.maximum_instances > cld.nodes(:status => "running").size
-          cld.launch_instance!(:cloud_name => cld.name) do |node|
-            ::PoolParty::Provision::BootStrapper.new(node.ip, :cloud => cld)
-            ::PoolParty::Provision::DrConfigure.new(node.ip,  :cloud => cld)
-
-            puts <<-EOM
-              Your cloud has expanded. The new ip is #{node.ip}
-            EOM
-          end          
-        else
-          puts "The cloud #{cld.name} has too many instances already (#{cld.nodes(:status => "running").size})"
-        end
-
-    end    
+    end
+    
   end
 end

data/bin/cloud-list

@@ -1,11 +1,10 @@
 #!/usr/bin/env ruby
 $:.unshift(File.dirname(__FILE__) + "/../lib")
+
 require "poolparty"
-# require "poolpartycl"
 require 'git-style-binary/command'
 
-GitStyleBinary.command do  
-  version "PoolParty #{$0} command"  
+GitStyleBinary.command do
   banner <<-EOS
 Usage: #{$0} #{all_options_string}
 
@@ -14,7 +13,6 @@ Usage: #{$0} #{all_options_string}
 EOS
 
   opt :instance_id, "Describe a specific instance", :type => String, :default => nil
-  opt :short, "show a short description rather than a long one", :default => false
 
   short_desc "show a list of the current instances on the clouds"
 
@@ -24,26 +22,26 @@ EOS
         require 'pp'
         pp result = cld.nodes(:instance_id=>command[:instance_id])
       else
-        if command[:short]
-          %w{running pending}.each do |status|
-            nodes = cld.nodes(:status => status)
-            if nodes.size > 0
-              puts nodes.map{|a| 
-                "%-15s [%s]: #{a[:instance_id] || a[:id]}\t#{a[:ip] ||a[:public_ip] }" % [cld.name, status] 
-              }.join("\n")
-            end
-          end
-        else
-          puts "Listing cloud #{cld.name}"
-          puts "Active instances"
-          puts cld.nodes(:status => "running").map{|a| "#{a[:instance_id] || a[:id]}\t#{a[:ip] ||a[:public_ip] }" }.join("\n")
-          
-          if cld.nodes(:status => "pending").size > 0
-            puts "Pending instances"
-            puts cld.nodes(:status => "pending").map{|a| "#{a[:instance_id] || a[:id]}\t#{a[:ip] ||a[:public_ip] }" }.join("\n")
+        line = "-----------------------------"
+        msg = ["Listing cloud #{cld.name}"]
+
+        hsh = {}
+        cld.describe_instances(:keypair_name => cld.keypair.basename).each do |n| 
+          (hsh[n.status] ||= []) << n
+        end
+        
+        hsh.each do |k,v|
+          msg << "#{k.capitalize} nodes"
+          msg << line
+          v.each do |a|
+            msg << "  #{a[:instance_id] || a[:id]}\t#{a[:ip] || a[:public_ip] || a[:name]}"
           end
+          msg << ""
         end
+        
+        print_msg(msg)
       end
+      
     end
   end
 end

data/bin/cloud-osxcopy

@@ -1,11 +1,10 @@
 #!/usr/bin/env ruby
 $:.unshift(File.dirname(__FILE__) + "/../lib")
 require "poolparty"
-# require "poolpartycl"
+
 require 'git-style-binary/command'
 
-GitStyleBinary.command do  
-  version "PoolParty #{$0} command"  
+GitStyleBinary.command do
   banner <<-EOS
 Usage: #{$0} #{all_options_string} COMMAND [ARGS]