auser-poolparty 0.0.8 → 0.0.9

data/CHANGELOG

@@ -1,3 +1,11 @@
+v0.0.9 * Changed configuration style to rsync all files across every instance
+       * Moved configuration back to a configure scrip
+       * Added in vlad configuration
+       * Added plugin ability to install custom software
+       * Updated configuration style
+       * Added cloud_list to pool
+       * Changed listing of the clouds through pool
+       * Added poolnotify
 v0.0.8 * Added plugin_manager
        * Moved remoting to rake remote task (from vlad)
        * Changed configuration from instance-based to cloud-based

data/README.txt

@@ -1,4 +1,4 @@
-= PoolParty
+= poolparty
 
   http://poolpartyrb.com
   Ari Lerner
@@ -7,19 +7,19 @@
 
 == DESCRIPTION:
   
-PoolParty (http://poolpartyrb.com), Ari Lerner (http://blog.xnot.org, http://blog.citrusbyte.com) - PoolParty is a framework for maintaining and running auto-scalable applications on Amazon's EC2 cloud. Run entire applications using the EC2 cluster and the unlimited S3 disk. More details to be listed at http://poolpartyrb.com.
+poolparty (http://poolpartyrb.com), Ari Lerner (http://blog.xnot.org, http://blog.citrusbyte.com) - poolparty is a framework for maintaining and running auto-scalable applications on Amazon's EC2 cloud. Run entire applications using the EC2 cluster and the unlimited S3 disk. More details to be listed at http://poolpartyrb.com.
 
 == Basics
 
-PoolParty is written with the intention of being as application-agnostic as possible. It installs only the basic required software to glue the cloud together on the instances as listed below.
+poolparty is written with the intention of being as application-agnostic as possible. It installs only the basic required software to glue the cloud together on the instances as listed below.
 
-PoolParty is easily configuration. In fact, it makes little assumptions about your development environment and allows several options on how to begin configuring the cloud. 
+poolparty is easily configuration. In fact, it makes little assumptions about your development environment and allows several options on how to begin configuring the cloud. 
 
 = Development setup
 
 === IN THE ENVIRONMENT
 
-There are 5 values that PoolParty reads from the environment, you can set these basic environment variables and leave the rest to the PoolParty defaults. Those values are:
+There are 5 values that poolparty reads from the environment, you can set these basic environment variables and leave the rest to the poolparty defaults. Those values are:
 
   ENV["ACCESS_KEY"] => AWS access key
   ENV["SECRET_ACCESS_KEY"] => AWS secret access key
@@ -31,11 +31,11 @@ The structure assumed for the keypair is EC2_HOME/id_rsa-<keypairname>
 
 === IN A CONFIG FILE
 
-PoolParty assumes your config directory is set in config/config.yml. However, you can set this in your environment variables and it will read the config file from the environment variable
+poolparty assumes your config directory is set in config/config.yml. However, you can set this in your environment variables and it will read the config file from the environment variable
 
 === WITH A RAKE TASK
 
-PoolParty comes with a rake task that can setup your environment for you. Set the environment variables above and run
+poolparty comes with a rake task that can setup your environment for you. Set the environment variables above and run
   
   rake dev:setup
 
@@ -47,9 +47,9 @@ and your environment will be all setup for you everytime you want to work on the
 
 = Basics
 
-PoolParty can work in two ways to load balance it's traffic. It can either do server-side or client-side load-balancing. Since every instance load balances itself, you can either set the client to grab an instance and send it to that using client-side load balancing (with a js library). Alternatively, you can set the master in dns and reference it when referring to the application.
+poolparty can work in two ways to load balance it's traffic. It can either do server-side or client-side load-balancing. Since every instance load balances itself, you can either set the client to grab an instance and send it to that using client-side load balancing (with a js library). Alternatively, you can set the master in dns and reference it when referring to the application.
 
-Since PoolParty makes no assumptions as to what you will be hosting on the application, the world is your oyster when running a cloud. You can set each instance to register with a dynDNS service so that your application has multiple points of entry and can run load-balanced on the fly.
+Since poolparty makes no assumptions as to what you will be hosting on the application, the world is your oyster when running a cloud. You can set each instance to register with a dynDNS service so that your application has multiple points of entry and can run load-balanced on the fly.
 
 Every instance will auto-mount the s3 bucket set in the config file (if it is set up) into the /data folder of the instance. This gives each instance access to the same data regardless of the instance. It uses s3fuse and caching through s3fuse in the /tmp directory to work as fast as possible on the local instances.
 
@@ -68,7 +68,7 @@ Each instance has a /etc/hosts file that has each node listed as the node name l
 
 = CloudSpeak - Communicating to your cloud(s)
 Binaries!
-Included in PoolParty are two binaries to communicate back with your clouds. Those are:
+Included in poolparty are two binaries to communicate back with your clouds. Those are:
   
 * pool - operate on your pool. This includes list, start, stop maintain, restart. Check the help with pool -h
 * instance - operate on a specific instance. This allos you to ssh, scp, reload, install as well. Check the help with: instance -h

data/Rakefile

@@ -1,27 +1,36 @@
 require 'rubygems'
-require 'echoe'
-require 'lib/poolparty'
+require "lib/poolparty"
+begin
+  require 'echoe'
+  
+  Echoe.new("poolparty") do |s|
+    s.author = "Ari Lerner"
+    s.email = "ari.lerner@citrusbyte.com"
+    s.summary = "Run your entire application off EC2, managed and auto-scaling"
+    s.url = "http://blog.citrusbyte.com"
+    s.runtime_dependencies = ["aws-s3", "amazon-ec2", "auser-aska", "git", "crafterm-sprinkle", "SystemTimer"]
+    s.development_dependencies = []
+    s.install_message = %q{
+      
+      Get ready to jump in the pool, you just installed PoolParty!
 
-task :default => :test
+      Please check out the documentation for any questions or check out the google groups at
+        http://groups.google.com/group/poolpartyrb
+
+      Don't forget to check out the plugin tutorial @ http://poolpartyrb.com for extending PoolParty!      
 
-Echoe.new("poolparty") do |p|
-  p.author = "Ari Lerner"
-  p.email = "ari.lerner@citrusbyte.com"
-  p.summary = "Run your entire application off EC2, managed and auto-scaling"
-  p.url = "http://blog.citrusbyte.com"
-  p.dependencies = %w(aws-s3 amazon-ec2 aska git)
-  p.install_message =<<-EOM
-    Thanks for installing PoolParty!
-    
-    Please check out the documentation for any questions or check out the google groups at
-    http://groups.google.com/group/poolpartyrb
-    
-    Don't forget to check out the plugins for extending PoolParty!
-    
-    For more information, check http://poolpartyrb.com
-    *** Ari Lerner @ <ari.lerner@citrusbyte.com> ***
-  EOM
-  p.include_rakefile = true
+      For more information, check http://poolpartyrb.com
+      On IRC: 
+        irc.freenode.net
+        #poolpartyrb
+      *** Ari Lerner @ <ari.lerner@citrusbyte.com> ***
+    }
+  end
+  
+rescue LoadError => boom
+  puts "You are missing a dependency required for meta-operations on this gem."
 end
 
+task :default => :test
+
 PoolParty.include_tasks

data/bin/instance

@@ -14,7 +14,7 @@ Usage: instance [OPTIONS] { #{commandables.join(" | ")} }
     })
 PoolParty.load_plugins
 master = PoolParty::Master.new
-list = PoolParty::Optioner.parse(ARGV.dup, %w(-v))
+list = PoolParty::Optioner.parse(ARGV.dup, %w(-v --verbose))
 num = list.reject {|a| commandables.include?(a) }.pop
 
 instance = master.get_node(num)
@@ -24,38 +24,43 @@ unless instance
   exit
 end
 
-case list[0]
-when "ssh"
-  PoolParty.message "Ssh'ing into #{instance.ip}"
-  instance.ssh
-when "cmd"
-  PoolParty.message "Executing #{instance_options[:cmd]} on #{instance.ip}"
-  instance.ssh instance_options[:cmd]
-when "scp"
-  instance.scp instance_options[:src], instance_options[:dest]
-when "restart"
-  PoolParty.message "Restarting services"
-  instance.restart_with_monit
-when "start"
-  PoolParty.message "Starting services"
-  instance.start_with_monit
-when "stop"
-  PoolParty.message "Stopping services"
-  instance.stop_with_monit
-when "install"
-  PoolParty.message "Installing services"
-  instance.install
-when "start_maintain"
-  PoolParty.message "Running heartbeat failover service"
-  pid = Master.run_thread_loop(:daemonize => true) do
-    instance.become_master if instance.is_not_master_and_master_is_not_running?
+list.each do |cmd|
+  case cmd
+  when "ssh"
+    PoolParty.message "Ssh'ing into #{instance.ip}"
+    instance.ssh
+  when "cmd"
+    PoolParty.message "Executing #{instance_options[:cmd]} on #{instance.ip}"
+    instance.ssh list.shift
+  when "scp"
+    list.shift
+    src, dest = list.shift, (list.shift || "~")
+    PoolParty.message "Scp'ing #{src} to #{dest}"
+    instance.scp src, (dest || "~")
+  when "restart"
+    PoolParty.message "Restarting services"
+    instance.restart_with_monit
+  when "start"
+    PoolParty.message "Starting services"
+    instance.start_with_monit
+  when "stop"
+    PoolParty.message "Stopping services"
+    instance.stop_with_monit
+  when "install"
+    PoolParty.message "Installing services"
+    instance.install
+  when "start_maintain"
+    PoolParty.message "Running heartbeat failover service"
+    pid = Master.run_thread_loop(:daemonize => true) do
+      instance.become_master if instance.is_not_master_and_master_is_not_running?
+    end
+    File.open(Application.maintain_pid_path) {|f| f.write(pid)}
+  when "stop_maintain"
+    PoolParty.message "Stopping heartbeat failover service"
+    pid = open(Application.maintain_pid_path).read
+    `kill -9 #{pid}`
+    FileUtils.rm Application.maintain_pid_path # Check this
+  else
+    puts master.list
   end
-  File.open(Application.maintain_pid_path) {|f| f.write(pid)}
-when "stop_maintain"
-  PoolParty.message "Stopping heartbeat failover service"
-  pid = open(Application.maintain_pid_path).read
-  `kill -9 #{pid}`
-  FileUtils.rm Application.maintain_pid_path # Check this
-else
-  puts master.list
 end

data/bin/pool

@@ -11,6 +11,7 @@ Starting #{PoolParty::Application.app_name ? "#{PoolParty::Application.app_name}
   Maximum instances: #{PoolParty::Application.maximum_instances}
   Polling every: #{PoolParty::Application.polling_time}
   Keypair: #{PoolParty::Application.keypair}
+  Access key: #{PoolParty::Application.access_key}
   size: #{PoolParty::Application.size}
   Plugins:
   --------------
@@ -22,7 +23,7 @@ end
 # Set defaults
 options = PoolParty.options(:optsparse => 
   {:banner => <<-EOU
-Usage: pool [OPTIONS] {start | stop | list | maintain | restart | install | configure}
+Usage: pool [OPTIONS] {start | stop | list | clouds_list | maintain | restart | install | configure | grow | shrink | ssh | switch}
 -----------------------------------------------------------------
     EOU
     })
@@ -31,32 +32,46 @@ PoolParty.load_plugins
 master = PoolParty::Master.new
 list = PoolParty::Optioner.parse(ARGV.dup, %w(-v))
 
-case list[0]
-when "start"
-  display_config_data
-  master.start_cloud!
-when "show"
-  display_config_data
-when "grow"
-  master.grow_by_one
-when "shrink"
-  master.shrink_by_one
-when "install"
-  master.install_cloud
-when "configure"
-  master.configure_cloud
-when "stop"
-  PoolParty.message "Stopping cloud"
-  master.request_termination_of_all_instances  
-when "list"
-  puts master.list
-when "maintain"
-  PoolParty.message "Maintaining cloud"
-  master.start_monitor!
-when "restart"
-  PoolParty.message "Restarting cloud"
-  master.request_termination_of_all_instances
-  master.start_cloud!
-else
-  puts master.list
+display_config_data
+
+list.each do |cmd|
+  case cmd
+  when "start"  
+    master.start_cloud!
+  when "show"
+    display_config_data
+  when "grow"
+    master.grow_by
+  when "shrink"
+    master.shrink_by
+  when "install"
+    master.install_cloud(true)
+  when "configure"
+    master.setup_cloud
+  when "ssh"
+    list.shift
+    PoolParty.message "Running #{list} on the cloud"
+    master.ssh list.shift
+  when "stop"
+    PoolParty.message "Stopping cloud"
+    master.request_termination_of_all_instances  
+  when "list"
+    puts master.list
+  when "size"
+    puts master.nodes.size
+  when "clouds_list"
+    puts master.clouds_list
+  when "maintain"
+    PoolParty.message "Maintaining cloud"
+    master.start_monitor!
+  when "restart"
+    PoolParty.message "Restarting cloud"
+    master.request_termination_of_all_instances
+    master.start_cloud!
+  when "switch"
+    context = list.shift
+    context ? `source ~/.#{context}_pool_keys` : puts("You must supply a context to switch to")
+  else
+    puts master.list
+  end
 end

data/bin/poolnotify

@@ -0,0 +1,34 @@
+#!/usr/bin/ruby
+require 'rubygems'
+$:.unshift(File.join(File.dirname(__FILE__), "lib"))
+require 'poolparty'
+
+def notify(mess)
+  `/usr/local/bin/growlnotify --image #{File.expand_path(File.dirname(__FILE__))}/../assets/clouds.png -m "Your #{PoolParty.options.keypair} cloud #{mess}"`
+end
+
+@master = PoolParty::Master.new
+
+@@size = @master.nodes.size
+
+notify("has #{@@size} nodes")
+
+blk = lambda {
+  loop do
+    sz = @master.nodes.size
+    PoolParty.message "Your cloud has #{sz} nodes from #{@@size} nodes"
+    unless @@size == sz
+      if @@size > sz 
+        notify("shrank to #{sz} nodes")
+      else
+        notify("grew to #{sz} nodes") 
+      end
+      @@size = sz
+    end
+    
+    @master.reset!
+    wait 20.seconds
+  end
+}
+
+PoolParty::Application.production? ? daemonize(&blk) : blk.call

data/config/haproxy.conf

@@ -23,7 +23,7 @@ defaults
   
   
   stats uri /haproxy
-  stats realm Statistics\ for\ PoolParty
+  stats realm Statistics\ for\ poolparty
 
 listen web_proxy 0.0.0.0::host_port
   :servers

data/config/heartbeat_authkeys.conf

@@ -1,2 +1,2 @@
 auth 1
-1 md5 PasswordForPoolPartyClusterApplicationButItIsOnlyADefault
+1 md5 PasswordForpoolpartyClusterApplicationButItIsOnlyADefault

data/config/monit/haproxy.monit.conf

@@ -4,4 +4,5 @@ check process haproxy with pidfile /var/run/haproxy.pid
   if totalmem is greater than 100.0 MB for 4 cycles then restart
   if cpu is greater than 50% for 2 cycles then alert
   if cpu is greater than 80% for 3 cycles then restart
-  if loadavg(5min) greater than 10 for 8 cycles then restart
+  if loadavg(5min) greater than 10 for 8 cycles then restart
+  group haproxy

data/config/nginx.conf

@@ -13,7 +13,7 @@ http {
     server {
         listen       80;
         server_name  srv;
-        root        /apps/pool-party;
+        root        /apps/poolparty;
      
      location / {
          proxy_pass http://fast_mongrels;

data/config/reconfigure_instances_script.sh

@@ -1,18 +1,37 @@
 #!/bin/sh
 
-# Reconfigure master
+# Move the hosts file
+echo "Moving the hosts file into place"
+:move_hostfile
+# Move the authkeys
+echo "Configuring the authkeys"
+:configure_authkeys
+# Move the config file
+echo "Moving custom config file for this cloud"
+:move_config_file
+# Reconfigure master if master?
+echo "If this is the master, I'm configuring it as the master now"
 :config_master
-# Start this instance's master maintain script
-:start_pool_maintain
-# Make the ha.d/resource.d
-sudo mkdir /etc/ha.d/resource.d/
+# Configure haproxy
+echo "Configuring and starting haproxy"
+:configure_haproxy
 # Set this hostname as appropriate in the cloud
+echo "Setting new hostname"
 :set_hostname
 # Configure heartbeat
-sudo mkdir /etc/ha.d/resource.d/
+echo "Moving all the resource.d files into place"
+:configure_resource_d
 # Start heartbeat
-/etc/init.d/heartbeat start
+echo "Configuring and starting heartbeat"
+:configure_heartbeat
 # Start s3fs
-:start_s3fs
+echo "Mounting shared drive, if shared_bucket exists in config"
+:mount_s3_drive
 # Configure monit
-mkdir /etc/monit.d
+echo "Configuring monit"
+:configure_monit
+# Update the plugins
+echo "Updating plugins"
+:update_plugins
+echo "Running user tasks"
+:user_tasks

data/config/sample-config.yml

@@ -12,7 +12,7 @@
 :os: ubuntu
 :host_port: 80
 :client_port: 8001
-:shared_bucket: "pool-party-app-data"
+:shared_bucket: "poolparty-app-data"
 :services: nginx
 :environment: production
 :contract_when:

data/lib/core/string.rb

@@ -14,6 +14,9 @@ class String
   def ^(h={})
     self.gsub(/:([\w]+)/) {h[$1.to_sym] if h.include?($1.to_sym)}
   end
+  def arrayable
+    self.strip.split(/\n/)
+  end
   def runnable
     self.strip.gsub(/\n/, " && ")
   end