auser-poolparty 0.0.8

data/misc/basics_tutorial.txt

@@ -0,0 +1,142 @@
+PoolParty, a tutorial.
+
+To get PoolParty up and running on your very own ec2 instances, you have to do a few things in preparation first.
+
+First, install the gem with:
+
+sudo gem install auser-poolparty -s http://gems.github.com
+
+First, head on over to http://aws.amazon.com and sign up for an account. Make sure you make note of the access key and the secret access key on the access identifiers page. Also, scroll down to the bottom of that page and download your x.509 certificate and the private one as well. You only can generate that once, so make sure you download it immediately. Finally, take note of your account id.
+
+Now, let's get to some PoolParty setup. PoolParty only requires one file, a configuration file. Once you get this setup, PoolParty will do the rest for you, so let's get started editing the config file.
+
+There is a generator in progress, so if by the time you read this and you have the gem installed, type
+
+which poolparty
+
+If you get a response, then the generator has been built and you can skip this section and go right to the next 'using it' section, however, if not or you just want to understand the innerworkings, read on.
+
+config/config.yml
+PoolParty makes the guess that your config is in a file and is in config/config.yml in your base project directory. However, this is not necessary if you do not want a configuration file, since all options can be added at runtime using switches on the command-line. That being said, the easiest way is to write a config file.
+
+The minimal config file looks like this:
+
+:access_key: "1XCTNEK1CC5BQXXXXXXX"
+:secret_access_key: "Q2qJHP0S2iOKikn9glB+XXXX/XXXXXXXX/XXXXXXX"
+:ami: "ami-3057b259"
+
+That's it. However,a more complete one would look more like this:
+
+:app_name: "test_app"
+:user_id: "1619-6456-1111"
+:access_key: "1XCTNEK1CC5BQXXXXXXX"
+:secret_access_key: "Q2qJHP0S2iOKikn9glB+XXXX/XXXXXXXX/XXXXXXX"
+:ami: "ami-3057b259"
+:size: small
+:polling_time: "5.minutes"
+:minimum_instances: 2
+:maximum_instances: 10
+:ec2_dir: "/Users/user/.ec2"
+:keypair: user
+:os: ubuntu
+:host_port: 80
+:shared_bucket: "pool-party-app-data"
+:environment: production
+:contract_when:
+  web > 10
+  cpu < 0.2
+:expand_when:
+  cpu > 0.45
+  web < 10
+  
+Most of it should be self-explanatory, but a few points are necessary to describe. ec2_dir and keypair are for your location development machine. If you didn't create a keypair, not to worry, put in anything here. If you did, it's the name of the keypair you added. This is important for working with the instances after you get them up.
+
+If you put in a bucket in the shared_bucket key, the instances will automount that bucket to the /data drive. If you leave it out or don't include it, then it won't. 
+
+The master instance takes care of monitoring the cloud for you. If the the parameters you set in the contract_when and expand_when are met, then the cloud will respond accordingly. Those are based on monitors that are included in PoolParty.
+
+The installation of the required software for PoolParty is taken care of by PoolParty. That being said, PoolParty makes no assumptions of what the cloud's responsibilities will be. See 'Using' later on.
+
+Monitors
+The monitors are flexible and extensible. Currently included in PoolParty are monitors that monitor web requests and cpu and memory percentages. Writing a monitor or a plugin are out of the scope of this tutorial, but will be included in future tutorials.
+
+Configuring
+It can be tedious setting up just your development machine. This is why PoolParty makes it super easy for you in just one rake task. 
+
+If you are not familiar with ruby, I suggest you look into it, it's a beautiful language. However, PoolParty is language-agnostic, so not to worry.
+
+Create a file in your development directory called Rakefile and add this to it.
+
+require 'rubygems'
+require 'poolparty'
+
+PoolParty.include_tasks
+
+That's it, pretty simple and a bunch of tasks are available to you directly!
+Your development directory should look like this:
+
+./
+  config/
+    config.yml
+  Rakefile
+
+Now, once you have this setup and your config file setup, you can type
+
+rake dev:setup
+
+That's it and you should be able to type ec2-describe-images (provided you have installed the ec2-api-tools, available here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88) at this point and you are set.
+
+You will find yourself with a .<keypair_name>_pool_keys file in your home directory. Anytime you want to setup a new cloud or simply check on this one, all you have to do is type:
+
+source ~/.<keypair_name>_pool_keys 
+
+and your'll be ready to switch over to editing the new cloud. 
+
+Using:
+When it's easy to maintain a cloud, it's even easier to use.
+
+There are two binaries included with PoolParty when you install them. Working with the cloud, you can use the binary: pool. Working with the instance, you can use the binary: instance. 
+
+Start your pool by typing:
+
+pool start
+
+It may take a minute. PoolParty is starting your cloud and configuring the master at this point. You may be asked to confirm an ssh connection. This only happens the first time so other pool commands won't ask you again.
+
+To check on your cloud, simply type:
+
+pool list
+
+and you will be presented with the list of your cloud and their roles
+
+If you want to stop the pool, it's as straightforward as:
+
+pool stop
+
+There are a  other commands you have available to you on the pool site. If you are hosting your master off-site, rather than on EC2, you can type on the master machine:
+
+pool maintain
+
+and it will bootup as the master, watching the other instances in the cloud.
+
+If you want to log into a specific instance, you can do so with the command:
+
+instance ssh -i num
+
+The num is the number in the list from: pool list. It defaults to logging into the master if the number is not supplied.
+
+You have some more options when working with instances. You can 
+
+instance ssh
+instance scp src='' dest=''
+instance cmd cmd='ls -la'
+instance start|stop|install|restart
+instance start_maintain
+
+The last command will force the instance to become the new master.
+
+Those are the basics of PoolParty. 
+
+You can check it out at http://poolpartyrb.com/.
+
+If you are interested in active development, check out the source at github at http://github.com/auser/pool-party/tree/master. Active discussion is held here: http://groups.google.com/group/poolpartyrb. Of course, the latest and greatest information will always be released first at http://blog.citrusbyte.com.

data/poolparty.gemspec

@@ -0,0 +1,72 @@
+
+# Gem::Specification for Poolparty-0.0.8
+# Originally generated by Echoe
+
+Gem::Specification.new do |s|
+  s.name = %q{poolparty}
+  s.version = "0.0.8"
+
+  s.specification_version = 2 if s.respond_to? :specification_version=
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Ari Lerner"]
+  s.date = %q{2008-06-20}
+  s.description = %q{Run your entire application off EC2, managed and auto-scaling}
+  s.email = %q{ari.lerner@citrusbyte.com}
+  s.executables = ["instance", "pool"]
+  s.extra_rdoc_files = ["bin/instance", "bin/pool", "CHANGELOG", "lib/core/array.rb", "lib/core/exception.rb", "lib/core/float.rb", "lib/core/hash.rb", "lib/core/kernel.rb", "lib/core/module.rb", "lib/core/object.rb", "lib/core/proc.rb", "lib/core/string.rb", "lib/core/time.rb", "lib/modules/callback.rb", "lib/modules/ec2_wrapper.rb", "lib/modules/safe_instance.rb", "lib/modules/vlad_override.rb", "lib/poolparty/application.rb", "lib/poolparty/init.rb", "lib/poolparty/master.rb", "lib/poolparty/monitors/cpu.rb", "lib/poolparty/monitors/memory.rb", "lib/poolparty/monitors/web.rb", "lib/poolparty/monitors.rb", "lib/poolparty/optioner.rb", "lib/poolparty/plugin.rb", "lib/poolparty/plugin_manager.rb", "lib/poolparty/provider/packages/essential.rb", "lib/poolparty/provider/packages/git.rb", "lib/poolparty/provider/packages/haproxy.rb", "lib/poolparty/provider/packages/heartbeat.rb", "lib/poolparty/provider/packages/monit.rb", "lib/poolparty/provider/packages/rsync.rb", "lib/poolparty/provider/packages/ruby.rb", "lib/poolparty/provider/packages/s3fuse.rb", "lib/poolparty/provider/provider.rb", "lib/poolparty/provider.rb", "lib/poolparty/remote_instance.rb", "lib/poolparty/remoter.rb", "lib/poolparty/remoting.rb", "lib/poolparty/scheduler.rb", "lib/poolparty/tasks/cloud.rake", "lib/poolparty/tasks/development.rake", "lib/poolparty/tasks/ec2.rake", "lib/poolparty/tasks/instance.rake", "lib/poolparty/tasks/plugins.rake", "lib/poolparty/tasks/server.rake", "lib/poolparty/tasks.rb", "lib/poolparty/tmp.rb", "lib/poolparty.rb", "lib/s3/s3_object_store_folders.rb", "README.txt"]
+  s.files = ["archives/ruby-1.8.6-p111.tar.gz", "bin/instance", "bin/pool", "CHANGELOG", "config/cloud_master_takeover", "config/create_proxy_ami.sh", "config/haproxy.conf", "config/heartbeat.conf", "config/heartbeat_authkeys.conf", "config/installers/ubuntu_install.sh", "config/monit/haproxy.monit.conf", "config/monit/nginx.monit.conf", "config/monit.conf", "config/nginx.conf", "config/reconfigure_instances_script.sh", "config/sample-config.yml", "config/scp_instances_script.sh", "lib/core/array.rb", "lib/core/exception.rb", "lib/core/float.rb", "lib/core/hash.rb", "lib/core/kernel.rb", "lib/core/module.rb", "lib/core/object.rb", "lib/core/proc.rb", "lib/core/string.rb", "lib/core/time.rb", "lib/modules/callback.rb", "lib/modules/ec2_wrapper.rb", "lib/modules/safe_instance.rb", "lib/modules/vlad_override.rb", "lib/poolparty/application.rb", "lib/poolparty/init.rb", "lib/poolparty/master.rb", "lib/poolparty/monitors/cpu.rb", "lib/poolparty/monitors/memory.rb", "lib/poolparty/monitors/web.rb", "lib/poolparty/monitors.rb", "lib/poolparty/optioner.rb", "lib/poolparty/plugin.rb", "lib/poolparty/plugin_manager.rb", "lib/poolparty/provider/packages/essential.rb", "lib/poolparty/provider/packages/git.rb", "lib/poolparty/provider/packages/haproxy.rb", "lib/poolparty/provider/packages/heartbeat.rb", "lib/poolparty/provider/packages/monit.rb", "lib/poolparty/provider/packages/rsync.rb", "lib/poolparty/provider/packages/ruby.rb", "lib/poolparty/provider/packages/s3fuse.rb", "lib/poolparty/provider/provider.rb", "lib/poolparty/provider.rb", "lib/poolparty/remote_instance.rb", "lib/poolparty/remoter.rb", "lib/poolparty/remoting.rb", "lib/poolparty/scheduler.rb", "lib/poolparty/tasks/cloud.rake", "lib/poolparty/tasks/development.rake", "lib/poolparty/tasks/ec2.rake", "lib/poolparty/tasks/instance.rake", "lib/poolparty/tasks/plugins.rake", "lib/poolparty/tasks/server.rake", "lib/poolparty/tasks.rb", "lib/poolparty/tmp.rb", "lib/poolparty.rb", "lib/s3/s3_object_store_folders.rb", "Manifest", "misc/basics_tutorial.txt", "Rakefile", "README.txt", "spec/application_spec.rb", "spec/callback_spec.rb", "spec/core_spec.rb", "spec/helpers/ec2_mock.rb", "spec/kernel_spec.rb", "spec/master_spec.rb", "spec/monitors/cpu_monitor_spec.rb", "spec/monitors/memory_spec.rb", "spec/monitors/misc_monitor_spec.rb", "spec/monitors/web_spec.rb", "spec/optioner_spec.rb", "spec/plugin_manager_spec.rb", "spec/plugin_spec.rb", "spec/pool_binary_spec.rb", "spec/poolparty_spec.rb", "spec/provider_spec.rb", "spec/remote_instance_spec.rb", "spec/remoter_spec.rb", "spec/remoting_spec.rb", "spec/scheduler_spec.rb", "spec/spec_helper.rb", "spec/string_spec.rb", "tmp/node0-pool-party-ha.cf", "tmp/node0-pool-party-haresources", "tmp/node0-pool-party-hosts", "tmp/node1-pool-party-ha.cf", "tmp/node1-pool-party-haresources", "tmp/node1-pool-party-hosts", "tmp/pool-party-haproxy.cfg", "web/static/conf/nginx.conf", "web/static/site/images/balloon.png", "web/static/site/images/cb.png", "web/static/site/images/clouds.png", "web/static/site/images/railsconf_preso_img.png", "web/static/site/index.html", "web/static/site/javascripts/application.js", "web/static/site/javascripts/corner.js", "web/static/site/javascripts/jquery-1.2.6.pack.js", "web/static/site/misc.html", "web/static/site/storage/pool_party_presentation.pdf", "web/static/site/stylesheets/application.css", "web/static/site/stylesheets/reset.css", "web/static/src/layouts/application.haml", "web/static/src/pages/index.haml", "web/static/src/pages/misc.haml", "web/static/src/stylesheets/application.sass", "poolparty.gemspec"]
+  s.has_rdoc = true
+  s.homepage = %q{http://blog.citrusbyte.com}
+  s.post_install_message = %q{    Thanks for installing PoolParty!
+    
+    Please check out the documentation for any questions or check out the google groups at
+    http://groups.google.com/group/poolpartyrb
+    
+    Don't forget to check out the plugins for extending PoolParty!
+    
+    For more information, check http://poolpartyrb.com
+    *** Ari Lerner @ <ari.lerner@citrusbyte.com> ***
+}
+  s.rdoc_options = ["--line-numbers", "--inline-source", "--title", "Poolparty", "--main", "README.txt"]
+  s.require_paths = ["lib"]
+  s.rubyforge_project = %q{poolparty}
+  s.rubygems_version = %q{1.1.1}
+  s.summary = %q{Run your entire application off EC2, managed and auto-scaling}
+
+  s.add_dependency(%q<aws-s3>, [">= 0"])
+  s.add_dependency(%q<amazon-ec2>, [">= 0"])
+  s.add_dependency(%q<aska>, [">= 0"])
+  s.add_dependency(%q<git>, [">= 0"])
+end
+
+
+# # Original Rakefile source (requires the Echoe gem):
+# 
+# require 'rubygems'
+# require 'echoe'
+# require 'lib/poolparty'
+# 
+# task :default => :test
+# 
+# Echoe.new("poolparty") do |p|
+#   p.author = "Ari Lerner"
+#   p.email = "ari.lerner@citrusbyte.com"
+#   p.summary = "Run your entire application off EC2, managed and auto-scaling"
+#   p.url = "http://blog.citrusbyte.com"
+#   p.dependencies = %w(aws-s3 amazon-ec2 aska git)
+#   p.install_message =<<-EOM
+#     Thanks for installing PoolParty!
+#     
+#     Please check out the documentation for any questions or check out the google groups at
+#     http://groups.google.com/group/poolpartyrb
+#     
+#     Don't forget to check out the plugins for extending PoolParty!
+#     
+#     For more information, check http://poolpartyrb.com
+#     *** Ari Lerner @ <ari.lerner@citrusbyte.com> ***
+#   EOM
+#   p.include_rakefile = true
+# end
+# 
+# PoolParty.include_tasks

data/spec/application_spec.rb

@@ -0,0 +1,39 @@
+require File.dirname(__FILE__) + '/spec_helper'
+
+describe "Application" do
+  it "should be able to send options in the Application.options" do
+    options({:optparse => {:banner => "hi"}})
+  end
+  it "should have the root_dir defined" do
+    PoolParty.root_dir.should_not be_nil
+  end
+  it "should be able to call on the haproxy_config_file" do
+    Application.haproxy_config_file.should_not be_nil
+  end
+  it "should be able to find the client_port" do
+    Application.options.should_receive(:client_port).and_return(7788)
+    Application.client_port.should == 7788
+  end
+  it "should always have cloud_master_takeover in the managed services list" do
+    Application.master_managed_services.should =~ /cloud_master_takeover/
+  end
+  it "should be able to say it is in development mode if it is in dev mode" do
+    Application.stub!(:environment).and_return("development")
+    Application.development?.should == true
+  end
+  it "should be able to say it is in production if it is in production" do
+    Application.stub!(:environment).and_return("production")
+    Application.production?.should == true
+  end
+  it "should be able to say it's keypair path is in the $HOME/ directory" do
+    Application.stub!(:ec2_dir).and_return("~/.ec2")
+    Application.stub!(:keypair).and_return("poolparty")
+    Application.keypair_path.should == "~/.ec2/id_rsa-poolparty"
+  end
+  it "should be able to show the version of the gem" do
+    Application.version.should_not be_nil
+  end
+  it "should show the version as a string" do
+    Application.version.class.should == String
+  end
+end

data/spec/callback_spec.rb

@@ -0,0 +1,194 @@
+require File.dirname(__FILE__) + '/spec_helper'
+
+class TestCallbacks
+  include Callbacks
+  attr_reader :str
+  
+  before :world, :hello
+  after :world, :thanks
+  
+  def hello(caller)
+    string << "hello "
+  end
+  def world
+    string << "world"
+  end
+  def thanks(caller)
+    string << ", thank you"
+  end
+  after :pop, :boom
+  def pop
+    string << "pop"
+  end
+  def boom(caller)
+    string << " goes boom"
+  end    
+  def string
+    @str ||= String.new
+  end
+end
+describe "Callbacks" do
+  before(:each) do
+    @klass = TestCallbacks.new
+  end
+  it "should retain it's class identifier" do
+    @klass.class.should == TestCallbacks
+  end
+  it "should callback the method before the method runs" do
+    @klass.world.should == "hello world, thank you"
+  end
+  it "should callback the method before the method runs" do
+    @klass.pop.should == "pop goes boom"
+  end
+end
+class TestMultipleCallbacks
+  include Callbacks
+  attr_reader :str
+  def hi(caller)
+    string << "hi, "
+  end
+  def hello(caller)
+    string << "hello "
+  end
+  def world
+    string << "world"
+  end
+  def string
+    @str ||= String.new
+  end
+  before :world, :hi, :hello
+end
+describe "Multiple callbacks" do
+  before(:each) do
+    @klass = TestMultipleCallbacks.new
+  end
+  it "should be able to have multiple callbacks on the same call" do
+    @klass.world.should == "hi, hello world"
+  end
+end
+class OutsideClass
+  def self.hello(caller)
+    puts "hello"
+  end
+end
+class TestOutsideClass
+  include Callbacks
+  before :world, {:hello => OutsideClass}
+  def world
+    "world"
+  end
+end
+describe "Options" do
+  before(:each) do
+    @c = TestOutsideClass.new
+  end
+  it "should be able to pass external class options to the callback" do
+    OutsideClass.should_receive(:hello).and_return "hello"
+    @c.world
+  end
+end
+class BlockClass
+  include Callbacks
+  before :world do
+    string << "hello "
+  end
+  def world
+    string << "world"
+  end
+  def string
+    @string ||= ""
+  end
+end
+describe "Block callbacks" do
+  it "should call the block on the callback" do
+    BlockClass.new.world.should == "hello world"
+  end
+end
+class BlockAndMethodClass
+  include Callbacks
+  before :world, :hi do
+    string << "hello "
+  end
+  def world
+    string << "world"
+  end
+  def hi(caller)
+    string << "hi, "
+  end
+  def string
+    @string ||= ""
+  end
+end
+describe "Block and method callbacks" do
+  it "should call the block on the callback and add the block" do
+    BlockAndMethodClass.new.world.should == "hi, hello world"
+  end
+end
+class ExternalMethodCallClass
+  include Callbacks
+  before :world, :hello
+  after :hello, :peter
+  
+  def world
+    string << "world"
+  end
+  def hello(caller)
+    string << "hello "
+  end
+  def peter(caller)
+    string << "peter "
+  end
+  def string
+    @string ||= ""
+  end
+end
+describe "External method callbacks inside a method" do
+  it "should call the block on the callback and add the " do
+    ExternalMethodCallClass.new.world.should == "hello peter world"
+  end
+end
+class OutsideBindingClass
+  def hello(caller)
+    caller.string << "hello"
+  end
+end
+class BindingClass
+  include Callbacks
+  before :world, :hello => "OutsideBindingClass"
+  def world
+    string << "#{@hello} world"
+  end
+  def string
+    @string ||= ""
+  end
+end
+describe "Methods" do
+  it "should have access to the local variables of the call" do
+    BindingClass.new.world.should == "hello world"
+  end
+end
+class EvilOutsideClass
+  attr_reader :name
+  def get_name(caller)
+    @name = caller.hello
+  end
+  def show_name(caller)
+    @name
+  end
+end
+class BindingClass
+  include Callbacks
+  before :print, :get_name => "EvilOutsideClass"
+  def print
+    "hello"
+  end
+  def hello
+    "franke"
+  end
+end
+describe "Variables on the plugin callbacker class" do
+  it "should have a new method of the class" do
+    BindingClass.new.print
+    BindingClass.new.methods.include?("eviloutsideclass").should == true
+  end
+end

data/spec/core_spec.rb

@@ -0,0 +1,15 @@
+require File.dirname(__FILE__) + '/spec_helper'
+
+describe "Hash" do
+  it "should preserve the contents of the original hash when safe_merge'ing" do
+    a = {:a => "10", :b => "20"}
+    b = {:b => "30", :c => "40"}
+    a.safe_merge(b).should == {:a => "10", :b => "20", :c => "40"}
+  end
+  it "should preserve the contents of the original hash when safe_merge!'ing" do
+    a = {:a => "10", :b => "20"}
+    b = {:b => "30", :c => "40"}
+    a.safe_merge!(b)
+    a.should == {:a => "10", :b => "20", :c => "40"}
+  end
+end

data/spec/helpers/ec2_mock.rb

@@ -0,0 +1,44 @@
+module PoolParty
+  class Master
+    def launch_new_instance!
+      letter = ("a".."z").to_a[instances.size] # For unique instance_ids
+      h = {:instance_id => "i-58ba56#{letter}", :ip => "ip-127-0-0-1.aws.amazonaws.com", :status => "pending", :launching_time => Time.now }
+      instances << h      
+      Thread.new {wait 0.1;h[:status] = "running"} # Simulate the startup time
+      return h
+    end
+    # Shutdown the instance by instance_id
+    def terminate_instance!(instance_id)
+      instances.select {|a| a[:instance_id] == instance_id}[0][:status] = "terminating"
+    end
+    # Instance description
+    def describe_instance(id)
+      item = instances.select {|a| a[:instance_id] == id}[0]
+      EC2ResponseObject.get_hash_from_response(item)
+    end
+    # Get instance by id
+    def get_instance_by_id(id)
+      get_instances_description.select {|a| a.instance_id == id}[0] rescue nil
+    end
+    # Get the s3 description for the response in a hash format
+    def get_instances_description
+      instances
+    end
+    # Fake the ec2 connection
+    def ec2
+      @ec2 ||= EC2::Base.new(:access_key_id => "not a key", :secret_access_key => "not a key")
+    end
+    # Some basic instances, not totally necessary
+    def instances
+      @instances ||= []
+    end
+  end
+  class RemoteInstance
+    def ssh(l="")
+      "true"
+    end
+    def scp(s,d,o={})
+      "true"
+    end
+  end
+end

data/spec/kernel_spec.rb

@@ -0,0 +1,11 @@
+require File.dirname(__FILE__) + '/spec_helper'
+
+describe "Kernel extensions" do
+  before(:each) do
+    @host = Master.new
+  end
+  it "should eval the string into time" do
+    @host.should_receive(:sleep).once.and_return true
+    @host.wait "10.seconds"
+  end
+end