auser-poolparty 0.0.8 → 0.0.9
Sign up to get free protection for your applications and to get access to all the features.
- data/CHANGELOG +8 -0
- data/README.txt +10 -10
- data/Rakefile +30 -21
- data/{web/static/site/images → assets}/clouds.png +0 -0
- data/bin/instance +39 -34
- data/bin/pool +44 -29
- data/bin/poolnotify +34 -0
- data/config/haproxy.conf +1 -1
- data/config/heartbeat_authkeys.conf +1 -1
- data/config/monit/haproxy.monit.conf +2 -1
- data/config/nginx.conf +1 -1
- data/config/reconfigure_instances_script.sh +28 -9
- data/config/sample-config.yml +1 -1
- data/lib/core/string.rb +3 -0
- data/lib/modules/ec2_wrapper.rb +47 -22
- data/lib/modules/file_writer.rb +38 -0
- data/lib/modules/sprinkle_overrides.rb +32 -0
- data/lib/modules/vlad_override.rb +5 -4
- data/lib/poolparty.rb +14 -10
- data/lib/poolparty/application.rb +33 -19
- data/lib/poolparty/master.rb +227 -105
- data/lib/poolparty/optioner.rb +8 -4
- data/lib/poolparty/plugin.rb +34 -4
- data/lib/poolparty/provider/packages/haproxy.rb +0 -15
- data/lib/poolparty/provider/packages/heartbeat.rb +1 -1
- data/lib/poolparty/provider/packages/ruby.rb +6 -6
- data/lib/poolparty/provider/packages/s3fuse.rb +9 -2
- data/lib/poolparty/provider/provider.rb +65 -25
- data/lib/poolparty/remote_instance.rb +95 -74
- data/lib/poolparty/remoter.rb +48 -37
- data/lib/poolparty/remoting.rb +41 -17
- data/lib/poolparty/scheduler.rb +4 -4
- data/lib/poolparty/tasks.rb +1 -1
- data/lib/poolparty/tasks/package.rake +53 -0
- data/lib/poolparty/tasks/plugins.rake +1 -1
- data/poolparty.gemspec +50 -58
- data/spec/application_spec.rb +28 -0
- data/spec/core_spec.rb +9 -0
- data/spec/ec2_wrapper_spec.rb +87 -0
- data/spec/file_writer_spec.rb +73 -0
- data/spec/files/describe_response +37 -0
- data/spec/files/multi_describe_response +69 -0
- data/spec/files/remote_desc_response +37 -0
- data/spec/helpers/ec2_mock.rb +3 -0
- data/spec/master_spec.rb +302 -78
- data/spec/monitors/cpu_monitor_spec.rb +2 -1
- data/spec/monitors/memory_spec.rb +1 -0
- data/spec/monitors/misc_monitor_spec.rb +1 -0
- data/spec/monitors/web_spec.rb +1 -0
- data/spec/optioner_spec.rb +12 -0
- data/spec/plugin_manager_spec.rb +10 -10
- data/spec/plugin_spec.rb +6 -3
- data/spec/pool_binary_spec.rb +3 -0
- data/spec/poolparty_spec.rb +12 -7
- data/spec/provider_spec.rb +1 -0
- data/spec/remote_instance_spec.rb +18 -18
- data/spec/remoter_spec.rb +4 -2
- data/spec/remoting_spec.rb +10 -2
- data/spec/scheduler_spec.rb +0 -6
- data/spec/spec_helper.rb +13 -0
- metadata +83 -52
- data/Manifest +0 -115
- data/lib/poolparty/tmp.rb +0 -46
- data/misc/basics_tutorial.txt +0 -142
- data/web/static/conf/nginx.conf +0 -22
- data/web/static/site/images/balloon.png +0 -0
- data/web/static/site/images/cb.png +0 -0
- data/web/static/site/images/railsconf_preso_img.png +0 -0
- data/web/static/site/index.html +0 -71
- data/web/static/site/javascripts/application.js +0 -3
- data/web/static/site/javascripts/corner.js +0 -178
- data/web/static/site/javascripts/jquery-1.2.6.pack.js +0 -11
- data/web/static/site/misc.html +0 -42
- data/web/static/site/storage/pool_party_presentation.pdf +0 -0
- data/web/static/site/stylesheets/application.css +0 -100
- data/web/static/site/stylesheets/reset.css +0 -17
- data/web/static/src/layouts/application.haml +0 -25
- data/web/static/src/pages/index.haml +0 -25
- data/web/static/src/pages/misc.haml +0 -5
- data/web/static/src/stylesheets/application.sass +0 -100
data/CHANGELOG
CHANGED
@@ -1,3 +1,11 @@
|
|
1
|
+
v0.0.9 * Changed configuration style to rsync all files across every instance
|
2
|
+
* Moved configuration back to a configure scrip
|
3
|
+
* Added in vlad configuration
|
4
|
+
* Added plugin ability to install custom software
|
5
|
+
* Updated configuration style
|
6
|
+
* Added cloud_list to pool
|
7
|
+
* Changed listing of the clouds through pool
|
8
|
+
* Added poolnotify
|
1
9
|
v0.0.8 * Added plugin_manager
|
2
10
|
* Moved remoting to rake remote task (from vlad)
|
3
11
|
* Changed configuration from instance-based to cloud-based
|
data/README.txt
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
=
|
1
|
+
= poolparty
|
2
2
|
|
3
3
|
http://poolpartyrb.com
|
4
4
|
Ari Lerner
|
@@ -7,19 +7,19 @@
|
|
7
7
|
|
8
8
|
== DESCRIPTION:
|
9
9
|
|
10
|
-
|
10
|
+
poolparty (http://poolpartyrb.com), Ari Lerner (http://blog.xnot.org, http://blog.citrusbyte.com) - poolparty is a framework for maintaining and running auto-scalable applications on Amazon's EC2 cloud. Run entire applications using the EC2 cluster and the unlimited S3 disk. More details to be listed at http://poolpartyrb.com.
|
11
11
|
|
12
12
|
== Basics
|
13
13
|
|
14
|
-
|
14
|
+
poolparty is written with the intention of being as application-agnostic as possible. It installs only the basic required software to glue the cloud together on the instances as listed below.
|
15
15
|
|
16
|
-
|
16
|
+
poolparty is easily configuration. In fact, it makes little assumptions about your development environment and allows several options on how to begin configuring the cloud.
|
17
17
|
|
18
18
|
= Development setup
|
19
19
|
|
20
20
|
=== IN THE ENVIRONMENT
|
21
21
|
|
22
|
-
There are 5 values that
|
22
|
+
There are 5 values that poolparty reads from the environment, you can set these basic environment variables and leave the rest to the poolparty defaults. Those values are:
|
23
23
|
|
24
24
|
ENV["ACCESS_KEY"] => AWS access key
|
25
25
|
ENV["SECRET_ACCESS_KEY"] => AWS secret access key
|
@@ -31,11 +31,11 @@ The structure assumed for the keypair is EC2_HOME/id_rsa-<keypairname>
|
|
31
31
|
|
32
32
|
=== IN A CONFIG FILE
|
33
33
|
|
34
|
-
|
34
|
+
poolparty assumes your config directory is set in config/config.yml. However, you can set this in your environment variables and it will read the config file from the environment variable
|
35
35
|
|
36
36
|
=== WITH A RAKE TASK
|
37
37
|
|
38
|
-
|
38
|
+
poolparty comes with a rake task that can setup your environment for you. Set the environment variables above and run
|
39
39
|
|
40
40
|
rake dev:setup
|
41
41
|
|
@@ -47,9 +47,9 @@ and your environment will be all setup for you everytime you want to work on the
|
|
47
47
|
|
48
48
|
= Basics
|
49
49
|
|
50
|
-
|
50
|
+
poolparty can work in two ways to load balance it's traffic. It can either do server-side or client-side load-balancing. Since every instance load balances itself, you can either set the client to grab an instance and send it to that using client-side load balancing (with a js library). Alternatively, you can set the master in dns and reference it when referring to the application.
|
51
51
|
|
52
|
-
Since
|
52
|
+
Since poolparty makes no assumptions as to what you will be hosting on the application, the world is your oyster when running a cloud. You can set each instance to register with a dynDNS service so that your application has multiple points of entry and can run load-balanced on the fly.
|
53
53
|
|
54
54
|
Every instance will auto-mount the s3 bucket set in the config file (if it is set up) into the /data folder of the instance. This gives each instance access to the same data regardless of the instance. It uses s3fuse and caching through s3fuse in the /tmp directory to work as fast as possible on the local instances.
|
55
55
|
|
@@ -68,7 +68,7 @@ Each instance has a /etc/hosts file that has each node listed as the node name l
|
|
68
68
|
|
69
69
|
= CloudSpeak - Communicating to your cloud(s)
|
70
70
|
Binaries!
|
71
|
-
Included in
|
71
|
+
Included in poolparty are two binaries to communicate back with your clouds. Those are:
|
72
72
|
|
73
73
|
* pool - operate on your pool. This includes list, start, stop maintain, restart. Check the help with pool -h
|
74
74
|
* instance - operate on a specific instance. This allos you to ssh, scp, reload, install as well. Check the help with: instance -h
|
data/Rakefile
CHANGED
@@ -1,27 +1,36 @@
|
|
1
1
|
require 'rubygems'
|
2
|
-
require
|
3
|
-
|
2
|
+
require "lib/poolparty"
|
3
|
+
begin
|
4
|
+
require 'echoe'
|
5
|
+
|
6
|
+
Echoe.new("poolparty") do |s|
|
7
|
+
s.author = "Ari Lerner"
|
8
|
+
s.email = "ari.lerner@citrusbyte.com"
|
9
|
+
s.summary = "Run your entire application off EC2, managed and auto-scaling"
|
10
|
+
s.url = "http://blog.citrusbyte.com"
|
11
|
+
s.runtime_dependencies = ["aws-s3", "amazon-ec2", "auser-aska", "git", "crafterm-sprinkle", "SystemTimer"]
|
12
|
+
s.development_dependencies = []
|
13
|
+
s.install_message = %q{
|
14
|
+
|
15
|
+
Get ready to jump in the pool, you just installed PoolParty!
|
4
16
|
|
5
|
-
|
17
|
+
Please check out the documentation for any questions or check out the google groups at
|
18
|
+
http://groups.google.com/group/poolpartyrb
|
19
|
+
|
20
|
+
Don't forget to check out the plugin tutorial @ http://poolpartyrb.com for extending PoolParty!
|
6
21
|
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
http://groups.google.com/group/poolpartyrb
|
18
|
-
|
19
|
-
Don't forget to check out the plugins for extending PoolParty!
|
20
|
-
|
21
|
-
For more information, check http://poolpartyrb.com
|
22
|
-
*** Ari Lerner @ <ari.lerner@citrusbyte.com> ***
|
23
|
-
EOM
|
24
|
-
p.include_rakefile = true
|
22
|
+
For more information, check http://poolpartyrb.com
|
23
|
+
On IRC:
|
24
|
+
irc.freenode.net
|
25
|
+
#poolpartyrb
|
26
|
+
*** Ari Lerner @ <ari.lerner@citrusbyte.com> ***
|
27
|
+
}
|
28
|
+
end
|
29
|
+
|
30
|
+
rescue LoadError => boom
|
31
|
+
puts "You are missing a dependency required for meta-operations on this gem."
|
25
32
|
end
|
26
33
|
|
34
|
+
task :default => :test
|
35
|
+
|
27
36
|
PoolParty.include_tasks
|
File without changes
|
data/bin/instance
CHANGED
@@ -14,7 +14,7 @@ Usage: instance [OPTIONS] { #{commandables.join(" | ")} }
|
|
14
14
|
})
|
15
15
|
PoolParty.load_plugins
|
16
16
|
master = PoolParty::Master.new
|
17
|
-
list = PoolParty::Optioner.parse(ARGV.dup, %w(-v))
|
17
|
+
list = PoolParty::Optioner.parse(ARGV.dup, %w(-v --verbose))
|
18
18
|
num = list.reject {|a| commandables.include?(a) }.pop
|
19
19
|
|
20
20
|
instance = master.get_node(num)
|
@@ -24,38 +24,43 @@ unless instance
|
|
24
24
|
exit
|
25
25
|
end
|
26
26
|
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
instance.
|
27
|
+
list.each do |cmd|
|
28
|
+
case cmd
|
29
|
+
when "ssh"
|
30
|
+
PoolParty.message "Ssh'ing into #{instance.ip}"
|
31
|
+
instance.ssh
|
32
|
+
when "cmd"
|
33
|
+
PoolParty.message "Executing #{instance_options[:cmd]} on #{instance.ip}"
|
34
|
+
instance.ssh list.shift
|
35
|
+
when "scp"
|
36
|
+
list.shift
|
37
|
+
src, dest = list.shift, (list.shift || "~")
|
38
|
+
PoolParty.message "Scp'ing #{src} to #{dest}"
|
39
|
+
instance.scp src, (dest || "~")
|
40
|
+
when "restart"
|
41
|
+
PoolParty.message "Restarting services"
|
42
|
+
instance.restart_with_monit
|
43
|
+
when "start"
|
44
|
+
PoolParty.message "Starting services"
|
45
|
+
instance.start_with_monit
|
46
|
+
when "stop"
|
47
|
+
PoolParty.message "Stopping services"
|
48
|
+
instance.stop_with_monit
|
49
|
+
when "install"
|
50
|
+
PoolParty.message "Installing services"
|
51
|
+
instance.install
|
52
|
+
when "start_maintain"
|
53
|
+
PoolParty.message "Running heartbeat failover service"
|
54
|
+
pid = Master.run_thread_loop(:daemonize => true) do
|
55
|
+
instance.become_master if instance.is_not_master_and_master_is_not_running?
|
56
|
+
end
|
57
|
+
File.open(Application.maintain_pid_path) {|f| f.write(pid)}
|
58
|
+
when "stop_maintain"
|
59
|
+
PoolParty.message "Stopping heartbeat failover service"
|
60
|
+
pid = open(Application.maintain_pid_path).read
|
61
|
+
`kill -9 #{pid}`
|
62
|
+
FileUtils.rm Application.maintain_pid_path # Check this
|
63
|
+
else
|
64
|
+
puts master.list
|
52
65
|
end
|
53
|
-
File.open(Application.maintain_pid_path) {|f| f.write(pid)}
|
54
|
-
when "stop_maintain"
|
55
|
-
PoolParty.message "Stopping heartbeat failover service"
|
56
|
-
pid = open(Application.maintain_pid_path).read
|
57
|
-
`kill -9 #{pid}`
|
58
|
-
FileUtils.rm Application.maintain_pid_path # Check this
|
59
|
-
else
|
60
|
-
puts master.list
|
61
66
|
end
|
data/bin/pool
CHANGED
@@ -11,6 +11,7 @@ Starting #{PoolParty::Application.app_name ? "#{PoolParty::Application.app_name}
|
|
11
11
|
Maximum instances: #{PoolParty::Application.maximum_instances}
|
12
12
|
Polling every: #{PoolParty::Application.polling_time}
|
13
13
|
Keypair: #{PoolParty::Application.keypair}
|
14
|
+
Access key: #{PoolParty::Application.access_key}
|
14
15
|
size: #{PoolParty::Application.size}
|
15
16
|
Plugins:
|
16
17
|
--------------
|
@@ -22,7 +23,7 @@ end
|
|
22
23
|
# Set defaults
|
23
24
|
options = PoolParty.options(:optsparse =>
|
24
25
|
{:banner => <<-EOU
|
25
|
-
Usage: pool [OPTIONS] {start | stop | list | maintain | restart | install | configure}
|
26
|
+
Usage: pool [OPTIONS] {start | stop | list | clouds_list | maintain | restart | install | configure | grow | shrink | ssh | switch}
|
26
27
|
-----------------------------------------------------------------
|
27
28
|
EOU
|
28
29
|
})
|
@@ -31,32 +32,46 @@ PoolParty.load_plugins
|
|
31
32
|
master = PoolParty::Master.new
|
32
33
|
list = PoolParty::Optioner.parse(ARGV.dup, %w(-v))
|
33
34
|
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
when "
|
39
|
-
|
40
|
-
when "
|
41
|
-
|
42
|
-
when "
|
43
|
-
|
44
|
-
when "
|
45
|
-
|
46
|
-
when "
|
47
|
-
|
48
|
-
when "
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
35
|
+
display_config_data
|
36
|
+
|
37
|
+
list.each do |cmd|
|
38
|
+
case cmd
|
39
|
+
when "start"
|
40
|
+
master.start_cloud!
|
41
|
+
when "show"
|
42
|
+
display_config_data
|
43
|
+
when "grow"
|
44
|
+
master.grow_by
|
45
|
+
when "shrink"
|
46
|
+
master.shrink_by
|
47
|
+
when "install"
|
48
|
+
master.install_cloud(true)
|
49
|
+
when "configure"
|
50
|
+
master.setup_cloud
|
51
|
+
when "ssh"
|
52
|
+
list.shift
|
53
|
+
PoolParty.message "Running #{list} on the cloud"
|
54
|
+
master.ssh list.shift
|
55
|
+
when "stop"
|
56
|
+
PoolParty.message "Stopping cloud"
|
57
|
+
master.request_termination_of_all_instances
|
58
|
+
when "list"
|
59
|
+
puts master.list
|
60
|
+
when "size"
|
61
|
+
puts master.nodes.size
|
62
|
+
when "clouds_list"
|
63
|
+
puts master.clouds_list
|
64
|
+
when "maintain"
|
65
|
+
PoolParty.message "Maintaining cloud"
|
66
|
+
master.start_monitor!
|
67
|
+
when "restart"
|
68
|
+
PoolParty.message "Restarting cloud"
|
69
|
+
master.request_termination_of_all_instances
|
70
|
+
master.start_cloud!
|
71
|
+
when "switch"
|
72
|
+
context = list.shift
|
73
|
+
context ? `source ~/.#{context}_pool_keys` : puts("You must supply a context to switch to")
|
74
|
+
else
|
75
|
+
puts master.list
|
76
|
+
end
|
62
77
|
end
|
data/bin/poolnotify
ADDED
@@ -0,0 +1,34 @@
|
|
1
|
+
#!/usr/bin/ruby
|
2
|
+
require 'rubygems'
|
3
|
+
$:.unshift(File.join(File.dirname(__FILE__), "lib"))
|
4
|
+
require 'poolparty'
|
5
|
+
|
6
|
+
def notify(mess)
|
7
|
+
`/usr/local/bin/growlnotify --image #{File.expand_path(File.dirname(__FILE__))}/../assets/clouds.png -m "Your #{PoolParty.options.keypair} cloud #{mess}"`
|
8
|
+
end
|
9
|
+
|
10
|
+
@master = PoolParty::Master.new
|
11
|
+
|
12
|
+
@@size = @master.nodes.size
|
13
|
+
|
14
|
+
notify("has #{@@size} nodes")
|
15
|
+
|
16
|
+
blk = lambda {
|
17
|
+
loop do
|
18
|
+
sz = @master.nodes.size
|
19
|
+
PoolParty.message "Your cloud has #{sz} nodes from #{@@size} nodes"
|
20
|
+
unless @@size == sz
|
21
|
+
if @@size > sz
|
22
|
+
notify("shrank to #{sz} nodes")
|
23
|
+
else
|
24
|
+
notify("grew to #{sz} nodes")
|
25
|
+
end
|
26
|
+
@@size = sz
|
27
|
+
end
|
28
|
+
|
29
|
+
@master.reset!
|
30
|
+
wait 20.seconds
|
31
|
+
end
|
32
|
+
}
|
33
|
+
|
34
|
+
PoolParty::Application.production? ? daemonize(&blk) : blk.call
|
data/config/haproxy.conf
CHANGED
@@ -1,2 +1,2 @@
|
|
1
1
|
auth 1
|
2
|
-
1 md5
|
2
|
+
1 md5 PasswordForpoolpartyClusterApplicationButItIsOnlyADefault
|
@@ -4,4 +4,5 @@ check process haproxy with pidfile /var/run/haproxy.pid
|
|
4
4
|
if totalmem is greater than 100.0 MB for 4 cycles then restart
|
5
5
|
if cpu is greater than 50% for 2 cycles then alert
|
6
6
|
if cpu is greater than 80% for 3 cycles then restart
|
7
|
-
if loadavg(5min) greater than 10 for 8 cycles then restart
|
7
|
+
if loadavg(5min) greater than 10 for 8 cycles then restart
|
8
|
+
group haproxy
|
data/config/nginx.conf
CHANGED
@@ -1,18 +1,37 @@
|
|
1
1
|
#!/bin/sh
|
2
2
|
|
3
|
-
#
|
3
|
+
# Move the hosts file
|
4
|
+
echo "Moving the hosts file into place"
|
5
|
+
:move_hostfile
|
6
|
+
# Move the authkeys
|
7
|
+
echo "Configuring the authkeys"
|
8
|
+
:configure_authkeys
|
9
|
+
# Move the config file
|
10
|
+
echo "Moving custom config file for this cloud"
|
11
|
+
:move_config_file
|
12
|
+
# Reconfigure master if master?
|
13
|
+
echo "If this is the master, I'm configuring it as the master now"
|
4
14
|
:config_master
|
5
|
-
#
|
6
|
-
|
7
|
-
|
8
|
-
sudo mkdir /etc/ha.d/resource.d/
|
15
|
+
# Configure haproxy
|
16
|
+
echo "Configuring and starting haproxy"
|
17
|
+
:configure_haproxy
|
9
18
|
# Set this hostname as appropriate in the cloud
|
19
|
+
echo "Setting new hostname"
|
10
20
|
:set_hostname
|
11
21
|
# Configure heartbeat
|
12
|
-
|
22
|
+
echo "Moving all the resource.d files into place"
|
23
|
+
:configure_resource_d
|
13
24
|
# Start heartbeat
|
14
|
-
|
25
|
+
echo "Configuring and starting heartbeat"
|
26
|
+
:configure_heartbeat
|
15
27
|
# Start s3fs
|
16
|
-
|
28
|
+
echo "Mounting shared drive, if shared_bucket exists in config"
|
29
|
+
:mount_s3_drive
|
17
30
|
# Configure monit
|
18
|
-
|
31
|
+
echo "Configuring monit"
|
32
|
+
:configure_monit
|
33
|
+
# Update the plugins
|
34
|
+
echo "Updating plugins"
|
35
|
+
:update_plugins
|
36
|
+
echo "Running user tasks"
|
37
|
+
:user_tasks
|
data/config/sample-config.yml
CHANGED