ironfan 3.1.5 → 3.1.6
Sign up to get free protection for your applications and to get access to all the features.
- data/Gemfile +7 -8
- data/README.md +6 -0
- data/TODO.md +140 -29
- data/VERSION +1 -1
- data/config/ubuntu10.04-ironfan.erb +11 -5
- data/config/ubuntu11.10-ironfan.erb +12 -6
- data/ironfan.gemspec +25 -23
- data/lib/chef/knife/bootstrap/centos6.2-ironfan.erb +148 -0
- data/lib/chef/knife/bootstrap/ubuntu10.04-ironfan.erb +11 -5
- data/lib/chef/knife/bootstrap/ubuntu11.10-ironfan.erb +12 -6
- data/lib/chef/knife/cluster_bootstrap.rb +4 -3
- data/lib/chef/knife/cluster_kick.rb +9 -8
- data/lib/chef/knife/cluster_kill.rb +1 -0
- data/lib/chef/knife/cluster_launch.rb +2 -2
- data/lib/chef/knife/cluster_proxy.rb +5 -4
- data/lib/chef/knife/cluster_show.rb +1 -1
- data/lib/chef/knife/cluster_ssh.rb +6 -5
- data/lib/chef/knife/cluster_stop.rb +3 -0
- data/lib/chef/knife/cluster_vagrant.rb +2 -2
- data/lib/chef/knife/ironfan_knife_common.rb +1 -1
- data/lib/chef/knife/vagrant/skeleton_vagrantfile.rb +5 -2
- data/lib/ironfan/chef_layer.rb +3 -1
- data/lib/ironfan/cloud.rb +146 -107
- data/lib/ironfan/discovery.rb +13 -0
- data/lib/ironfan/fog_layer.rb +42 -3
- data/lib/ironfan/security_group.rb +16 -4
- data/lib/ironfan/server.rb +29 -0
- data/lib/ironfan.rb +1 -1
- data/notes/INSTALL-cloud_setup.md +14 -11
- data/notes/INSTALL.md +6 -1
- data/notes/core_concepts.md +1 -1
- data/notes/ec2-pricing_and_capacity.md +24 -18
- data/notes/version-3_2.md +273 -0
- data/spec/spec_helper.rb +26 -34
- metadata +31 -29
data/Gemfile
CHANGED
@@ -1,20 +1,19 @@
|
|
1
1
|
source "http://rubygems.org"
|
2
2
|
|
3
|
-
gem 'chef', "
|
4
|
-
gem 'fog', "~> 1.2
|
5
|
-
gem 'formatador', "~> 0.2
|
6
|
-
gem 'gorillib', "~> 0.1
|
3
|
+
gem 'chef', ">= 0.10.4"
|
4
|
+
gem 'fog', "~> 1.2"
|
5
|
+
gem 'formatador', "~> 0.2"
|
6
|
+
gem 'gorillib', "~> 0.1"
|
7
7
|
|
8
8
|
group :development do
|
9
|
-
gem 'bundler', "
|
10
|
-
gem 'jeweler', "
|
9
|
+
gem 'bundler', ">= 1.1"
|
10
|
+
gem 'jeweler', ">= 1.6"
|
11
11
|
gem 'rspec', "~> 2.5"
|
12
|
-
gem 'yard', "
|
12
|
+
gem 'yard', ">= 0.6"
|
13
13
|
gem 'redcarpet', "~> 2"
|
14
14
|
end
|
15
15
|
|
16
16
|
group :test do
|
17
|
-
gem 'spork', ">= 0.9.0", :platform => :mri
|
18
17
|
gem 'rcov', ">= 0.9.9", :platform => :ruby_18
|
19
18
|
gem 'simplecov', ">= 0.5", :platform => :ruby_19
|
20
19
|
#
|
data/README.md
CHANGED
@@ -34,9 +34,15 @@ Ironfan consists of the following Toolset:
|
|
34
34
|
* [Ironfan Screencast](http://bit.ly/ironfan-hadoop-in-20-minutes) -- build a Hadoop cluster from scratch in 20 minutes.
|
35
35
|
* Ironfan powers the [Infochimps Platform](http://www.infochimps.com/how-it-works), our scalable enterprise big data platform. Ironfan Enterprise adds zero-configuration logging, monitoring and a compelling UI.
|
36
36
|
|
37
|
+
**Note**: Ironfan is [not compatible with Ruby 1.8](https://github.com/infochimps-labs/ironfan/issues/127). All versions later than 1.9.2-p136 should work fine.
|
38
|
+
|
37
39
|
### The Ironfan Way
|
38
40
|
|
39
41
|
* [Core Concepts](https://github.com/infochimps-labs/ironfan/wiki/core_concepts) -- Components, Announcements, Amenities and more.
|
40
42
|
* [Philosophy](https://github.com/infochimps-labs/ironfan/wiki/Philosophy) -- Best practices and lessons learned
|
41
43
|
* [Style Guide](https://github.com/infochimps-labs/ironfan/wiki/style_guide) -- Common attribute names, how and when to include other cookbooks, and more
|
42
44
|
* [Homebase Layout](https://github.com/infochimps-labs/ironfan/wiki/homebase-layout) -- How this homebase is organized, and why
|
45
|
+
|
46
|
+
### Getting Help
|
47
|
+
* Feel free to contact us at info@infochimps.com or 855-DATA-FUN
|
48
|
+
* Also, you invited to a [private consultation](http://www.infochimps.com/free-big-data-consultation?utm_source=git&utm_medium=referral&utm_campaign=consult) with Infochimps founders on your big data project.
|
data/TODO.md
CHANGED
@@ -1,44 +1,155 @@
|
|
1
|
-
|
1
|
+
* refactor standard dirs and volume_dirs into a single, more coherent resource
|
2
|
+
|
3
|
+
* We should use the `Fog::Compute::AWS::FLAVORS` constant that [fog defines](http://rubydoc.info/github/fog/fog/master/Fog/Compute/AWS) in the cloud code (instead of the one we put there)
|
4
|
+
|
5
|
+
* All over the place there is the following construct (absolutely necessary, absolutely horrid):
|
6
|
+
|
7
|
+
foo = Mash.new().merge(node[:system]).merge(node[:system][:component])
|
8
|
+
|
9
|
+
You might look at this and think "gee I know a much simpler way to do that". That simpler way does not work; this way does.
|
10
|
+
|
11
|
+
I propose adding a 'smush' method to `silverware/libraries/cookbook_utils`:
|
12
|
+
|
13
|
+
```ruby
|
14
|
+
module Ironfan::CookbookUtils
|
15
|
+
module_function
|
16
|
+
|
17
|
+
# Merge the given objects (node attributes, hashes, or anything
|
18
|
+
# else with `#to_hash`) into a combined `Mash` object. Objects
|
19
|
+
# given later in the list 'win' over objects given earlier.
|
20
|
+
#
|
21
|
+
# @examples
|
22
|
+
# template_vars = Ironfan::CookbookUtils.smush( node[:flume], node[:flume][:agent], :zookeeper_port => node[:zookeeper][:port] )
|
23
|
+
#
|
24
|
+
# @param [Array[#to_hash]] smushables -- any number of things that respond to `#to_hash`
|
25
|
+
#
|
26
|
+
def smush(*smushables)
|
27
|
+
result = Mash.new
|
28
|
+
smushables.compact.each do |smushable|
|
29
|
+
result.merge! smushable.to_hash
|
30
|
+
end
|
31
|
+
result
|
32
|
+
end
|
33
|
+
|
34
|
+
end
|
2
35
|
|
3
|
-
|
36
|
+
(obviously the hard part is not writing the method, it's applying it to all the cookbooks.)
|
4
37
|
|
5
38
|
|
39
|
+
|
40
|
+
### Knife commands
|
41
|
+
|
6
42
|
* reify notion of 'homebase'; cluster commands work off it
|
7
43
|
* move away from referring to Chef::Config everywhere;
|
8
44
|
|
9
45
|
|
10
|
-
__________________________________________________________________________
|
11
46
|
|
12
|
-
Rename completed,
|
13
47
|
|
14
|
-
New homes:
|
15
48
|
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
*
|
21
|
-
*
|
49
|
+
# Old Issues Triage
|
50
|
+
From https://github.com/infochimps-labs/ironfan/issues/102 with love. Deleted ownership, and everything that was done or has its own issue.
|
51
|
+
|
52
|
+
## Must Do
|
53
|
+
* merge volumes into silverware. merge ebs_volumes into ec2 cookbook
|
54
|
+
* Basic CI testing of cookbooks
|
55
|
+
* RSpecs for silverback (lib and knife tools)
|
56
|
+
* RSpecs for silverware are mostly in place -- ensure they are.
|
57
|
+
* push cookbooks to community.opscode.com
|
58
|
+
* refine and explain updated git workflow
|
59
|
+
|
60
|
+
## Docco
|
61
|
+
Use the [opscode EC2 fast start](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide) as a guide -- our getting started should start at the same place, and cover the same detail as the EC2 bootstrap guide.
|
62
|
+
|
63
|
+
* Clear description of metadiscovery
|
64
|
+
* make sure README files in cookbooks aren’t wildly inaccurate
|
65
|
+
* Carry out setup directions, ensure they work:
|
66
|
+
- cluster_chef if you’re using our homebase
|
67
|
+
- cluster_chef if you’re using opscode’s homebase
|
68
|
+
* local vagrant environment
|
69
|
+
* hadoop cluster bootstrapping
|
70
|
+
|
71
|
+
## Piddly Shit
|
72
|
+
|
73
|
+
* standardize the `zabbix` cookbook (no more /opt, etc -- more in the TODO)
|
74
|
+
* volumes don't deep merge -- eg you have to mount_ephemerals in the facet if you modify htem
|
75
|
+
* kill_old_service should disable services (may be leaving /etc/rc.d cruft).
|
76
|
+
* kill old service doesn't go the first time. why?
|
77
|
+
* chef client/server cookbook: set chef user UID / GID; client can set log directory
|
78
|
+
* apt has a dashboard at http://{hostname}:3142/report
|
79
|
+
* can use knife ssh as me@ or as ubuntu@
|
80
|
+
* knife command to set/remove permanent on a node + disableApiTermination on box. knife cluster kill refuses to delete nodes with permanent set. knife cluster sync sets permanent on if permanent(true), removes if permanent(false), ignores if permanent nil or unset.
|
81
|
+
* style-guide alignment (prefix_root becomes prefix)
|
82
|
+
|
83
|
+
## Really Want
|
84
|
+
|
85
|
+
* unify the hashlike underpinning to be same across silverware & cluster_chef. Make sure we love (or accept) all the differences between it and Gorrillib’s, and between it and Chef’s.
|
86
|
+
* Keys are transmitted in databags, using a helper, and not in node attributes
|
87
|
+
* easy to create a dummy node (load balancer, external resource, etc)
|
88
|
+
* components can have arbitrary attributes (kinda. they take an `:info` param, behavior which may change later)
|
89
|
+
* All cookbooks have nice detailed announcements
|
90
|
+
* full roll out of log_integration, monitoring
|
91
|
+
* Git deploy abstraction similar to `install_from`
|
92
|
+
|
93
|
+
## Cookbook checklist:
|
94
|
+
* Validate all the cookbooks against checklist -- see notes/README-checklist.md
|
95
|
+
|
96
|
+
| flip fixed | temujin9 checked |
|
97
|
+
+------------+------------------+
|
98
|
+
cassandra | | |
|
99
|
+
ec2 | | |
|
100
|
+
elasticsearch | | |
|
101
|
+
firewall | | |
|
102
|
+
flume | | |
|
103
|
+
ganglia | | |
|
104
|
+
graphite | | |
|
105
|
+
hadoop_cluster | | |
|
106
|
+
hbase | | |
|
107
|
+
hive | | |
|
108
|
+
jenkins | | |
|
109
|
+
jruby | | |
|
110
|
+
nfs | | |
|
111
|
+
nodejs | | |
|
112
|
+
papertrail | | |
|
113
|
+
pig | | |
|
114
|
+
redis | | |
|
115
|
+
resque | | |
|
116
|
+
Rstats | | |
|
117
|
+
statsd | | |
|
118
|
+
zookeeper | | |
|
119
|
+
# meta:
|
120
|
+
install_from | | |
|
121
|
+
motd | | |
|
122
|
+
mountable_volumes | | |
|
123
|
+
provides_service | | |
|
124
|
+
# Need thinkin':
|
125
|
+
big_package | | |
|
126
|
+
cluster_chef | | |
|
127
|
+
|
22
128
|
|
23
|
-
##
|
129
|
+
## Things that are probably straightforward to fix as soon as we know how
|
24
130
|
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
3. DONE Ensure all infochimps devs have push-pulled to various repos. make a tarball of the repos (.git and everything) and put them somewhere safe.
|
29
|
-
4. DONE rename the `vendor/infochimps/metachef` cookbook as vendor/infochimps/silverware.
|
30
|
-
5. DONE Bump the *minor* version number on all cookbooks (so 3.0.x => 3.1.x). Commit.
|
31
|
-
6. DONE regex-replace `ClusterChef` => `Ironfan`, `cluster_chef` to `ironfan` and `[Mm]etachef` => `[Ss]ilverware`. Do this in `cluster_chef` and `infochimps-labs/ironfan-homebase` only. Get chef-client to complete on el_ridiculoso using an emptied-and-reloaded local chef server. Run knife cluster sync on all clusters in all homebases against local chef server.
|
32
|
-
7. ...
|
33
|
-
12. DONE once the name is updated in the gemspec, release the `ironfan` gem. @temujin9 and I agree that it should be a single combined gem now that we won't have the stupid `_chef` gem name conflict.
|
131
|
+
* announcements should probably be published very early, but they need to know lots about the machine YUK
|
132
|
+
* split between clusters / roles / integration cookbooks
|
133
|
+
* inheritance of clusters
|
34
134
|
|
35
|
-
##
|
135
|
+
## Things We Hate But Might Have to Continue Hating
|
36
136
|
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
137
|
+
* Cluster refactor -- clusters / stacks / components, not clusters / roles / cookbooks
|
138
|
+
* move cluster discovery to cloud class.
|
139
|
+
* Server#normalize! doesn’t imprint object (ie. server attributes poke through to the facet & cluster, rather than being *set* on the object)
|
140
|
+
* The fact you can only see one cluster at a time is stupid.
|
141
|
+
* security group pairing is sucky.
|
142
|
+
* ubuntu home drive bullshit
|
143
|
+
* Finer-grained security group control (eg nfs server only opens a couple ports, not all)
|
144
|
+
* nfs recipe uses discovery right (thus allowing more than one NFS share to exist in the universe)
|
145
|
+
* roles UGGGHHHHAERWSDFKHSBLAH
|
43
146
|
|
44
|
-
|
147
|
+
## Ponies!
|
148
|
+
* sync cookbooks up/down to `infochimps-cookbooks/`
|
149
|
+
- note: infochimps-cookbooks the org will be dereferenced in favor of ironfan-lib the single repo; it's unclear which pull requesters will prefer. We will do at least one push so that names and URLs are current, and we're not removing anything, but infochimps-cookbooks has an unclear future.
|
150
|
+
* foodcritic compatibility
|
151
|
+
* build out cookbook munger, make it less spike-y
|
152
|
+
* spot pricing
|
153
|
+
* rackspace compatibility
|
154
|
+
* cookbook munger reads comments in attributes file to populate metadata.rb
|
155
|
+
* `gem install ironfan; ironfan install` checks everything out
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
3.1.
|
1
|
+
3.1.6
|
@@ -26,14 +26,20 @@ export DEBIAN_FRONTEND=noninteractive
|
|
26
26
|
date > /etc/box_build_time
|
27
27
|
|
28
28
|
echo -e "`date` \n\n**** \n**** apt update:\n****\n"
|
29
|
-
apt-get -y update
|
30
|
-
apt-get -y upgrade
|
29
|
+
apt-get --force-yes -y update
|
30
|
+
apt-get --force-yes -y upgrade
|
31
31
|
|
32
32
|
echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
|
33
|
-
apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
|
34
|
-
apt-get -y install runit-services
|
33
|
+
apt-get --force-yes -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
|
34
|
+
apt-get --force-yes -y install runit-services
|
35
35
|
apt-get clean
|
36
36
|
|
37
|
+
# do not trust the ubuntu chef package
|
38
|
+
rm -f /etc/init.d/chef-client || true
|
39
|
+
apt-get remove -y --force-yes chef ruby1.8-dev libjson-ruby1.8 libmixlib-authentication-ruby1.8 ohai libmixlib-log-ruby1.8 libmime-types-ruby librestclient-ruby1.8 ruby1.8 ruby liberubis-ruby1.8 libsystemu-ruby1.8 libohai-ruby libuuidtools-ruby1.8 libhighline-ruby1.8 libabstract-ruby1.8 libmixlib-config-ruby1.8 rubygems1.8 libbunny-ruby1.8 libchef-ruby1.8 libmixlib-cli-ruby1.8 libyajl-ruby libmoneta-ruby1.8 libohai-ruby1.8 libextlib-ruby1.8 || true
|
40
|
+
# apt-get remove -y --force-yes --purge myproxy globus-core condor nis autofs myproxy-server globus-proxy-utils globus-repository-natty globus-gridftp-server-progs globus-gass-copy-progs globus-simple-ca globus-gsi-cert-utils-progs || true
|
41
|
+
sudo apt-get autoremove -y --force-yes || true
|
42
|
+
|
37
43
|
if [ ! -f /usr/bin/chef-client ]; then
|
38
44
|
echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
|
39
45
|
|
@@ -69,7 +75,7 @@ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
|
|
69
75
|
gem install ohai --no-rdoc --no-ri
|
70
76
|
gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
|
71
77
|
# gems needed for the client.rb or so generically useful you want them at hand
|
72
|
-
gem install --no-rdoc --no-ri extlib bundler json right_aws pry
|
78
|
+
gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
|
73
79
|
|
74
80
|
else # no chef-client
|
75
81
|
echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
|
@@ -26,21 +26,27 @@ export DEBIAN_FRONTEND=noninteractive
|
|
26
26
|
date > /etc/box_build_time
|
27
27
|
|
28
28
|
# source for sun java if you want to install it later
|
29
|
-
apt-get install -y python-software-properties
|
29
|
+
apt-get install --force-yes -y python-software-properties
|
30
30
|
add-apt-repository -y ppa:ferramroberto/java
|
31
31
|
|
32
32
|
echo -e "`date` \n\n**** \n**** apt update:\n****\n"
|
33
|
-
apt-get -y update
|
34
|
-
apt-get -y upgrade
|
33
|
+
apt-get --force-yes -y update
|
34
|
+
apt-get --force-yes -y upgrade
|
35
35
|
|
36
36
|
echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
|
37
|
-
apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
|
37
|
+
apt-get --force-yes -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
|
38
38
|
apt-get clean
|
39
39
|
|
40
|
+
# do not trust the ubuntu chef package
|
41
|
+
rm -f /etc/init.d/chef-client || true
|
42
|
+
apt-get remove -y --force-yes chef ruby1.8-dev libjson-ruby1.8 libmixlib-authentication-ruby1.8 ohai libmixlib-log-ruby1.8 libmime-types-ruby librestclient-ruby1.8 ruby1.8 ruby liberubis-ruby1.8 libsystemu-ruby1.8 libohai-ruby libuuidtools-ruby1.8 libhighline-ruby1.8 libabstract-ruby1.8 libmixlib-config-ruby1.8 rubygems1.8 libbunny-ruby1.8 libchef-ruby1.8 libmixlib-cli-ruby1.8 libyajl-ruby libmoneta-ruby1.8 libohai-ruby1.8 libextlib-ruby1.8 || true
|
43
|
+
# apt-get remove -y --force-yes --purge myproxy globus-core condor nis autofs myproxy-server globus-proxy-utils globus-repository-natty globus-gridftp-server-progs globus-gass-copy-progs globus-simple-ca globus-gsi-cert-utils-progs || true
|
44
|
+
sudo apt-get autoremove -y --force-yes || true
|
45
|
+
|
40
46
|
if [ ! -f /usr/bin/chef-client ]; then
|
41
47
|
echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
|
42
48
|
|
43
|
-
apt-get install -y ruby1.9.1 ruby1.9.1-dev
|
49
|
+
apt-get install --force-yes -y ruby1.9.1 ruby1.9.1-dev
|
44
50
|
|
45
51
|
if ruby -e "exit(%x{gem --version} < \"1.6.2\" ? 0 : -1 )" ; then
|
46
52
|
echo -e "`date` \n\n**** \n**** Updating rubygems:\n****\n"
|
@@ -57,7 +63,7 @@ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
|
|
57
63
|
gem install ohai --no-rdoc --no-ri
|
58
64
|
gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
|
59
65
|
# gems needed for the client.rb or so generically useful you want them at hand
|
60
|
-
gem install --no-rdoc --no-ri extlib bundler json right_aws pry
|
66
|
+
gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
|
61
67
|
|
62
68
|
else # no chef-client
|
63
69
|
echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
|
data/ironfan.gemspec
CHANGED
@@ -5,11 +5,11 @@
|
|
5
5
|
|
6
6
|
Gem::Specification.new do |s|
|
7
7
|
s.name = "ironfan"
|
8
|
-
s.version = "3.1.
|
8
|
+
s.version = "3.1.6"
|
9
9
|
|
10
10
|
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
|
11
11
|
s.authors = ["Infochimps"]
|
12
|
-
s.date = "2012-
|
12
|
+
s.date = "2012-06-24"
|
13
13
|
s.description = "Ironfan allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks."
|
14
14
|
s.email = "coders@infochimps.com"
|
15
15
|
s.extra_rdoc_files = [
|
@@ -34,6 +34,7 @@ Gem::Specification.new do |s|
|
|
34
34
|
"config/ubuntu10.04-ironfan.erb",
|
35
35
|
"config/ubuntu11.10-ironfan.erb",
|
36
36
|
"ironfan.gemspec",
|
37
|
+
"lib/chef/knife/bootstrap/centos6.2-ironfan.erb",
|
37
38
|
"lib/chef/knife/bootstrap/ubuntu10.04-ironfan.erb",
|
38
39
|
"lib/chef/knife/bootstrap/ubuntu11.10-ironfan.erb",
|
39
40
|
"lib/chef/knife/cluster_bootstrap.rb",
|
@@ -98,6 +99,7 @@ Gem::Specification.new do |s|
|
|
98
99
|
"notes/silverware.md",
|
99
100
|
"notes/style_guide.md",
|
100
101
|
"notes/tips_and_troubleshooting.md",
|
102
|
+
"notes/version-3_2.md",
|
101
103
|
"notes/walkthrough-hadoop.md",
|
102
104
|
"notes/walkthrough-web.md",
|
103
105
|
"spec/ironfan/cluster_spec.rb",
|
@@ -121,35 +123,35 @@ Gem::Specification.new do |s|
|
|
121
123
|
s.specification_version = 3
|
122
124
|
|
123
125
|
if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
|
124
|
-
s.add_runtime_dependency(%q<chef>, ["
|
125
|
-
s.add_runtime_dependency(%q<fog>, ["~> 1.2
|
126
|
-
s.add_runtime_dependency(%q<formatador>, ["~> 0.2
|
127
|
-
s.add_runtime_dependency(%q<gorillib>, ["~> 0.1
|
128
|
-
s.add_development_dependency(%q<bundler>, ["
|
129
|
-
s.add_development_dependency(%q<jeweler>, ["
|
126
|
+
s.add_runtime_dependency(%q<chef>, [">= 0.10.4"])
|
127
|
+
s.add_runtime_dependency(%q<fog>, ["~> 1.2"])
|
128
|
+
s.add_runtime_dependency(%q<formatador>, ["~> 0.2"])
|
129
|
+
s.add_runtime_dependency(%q<gorillib>, ["~> 0.1"])
|
130
|
+
s.add_development_dependency(%q<bundler>, [">= 1.1"])
|
131
|
+
s.add_development_dependency(%q<jeweler>, [">= 1.6"])
|
130
132
|
s.add_development_dependency(%q<rspec>, ["~> 2.5"])
|
131
|
-
s.add_development_dependency(%q<yard>, ["
|
133
|
+
s.add_development_dependency(%q<yard>, [">= 0.6"])
|
132
134
|
s.add_development_dependency(%q<redcarpet>, ["~> 2"])
|
133
135
|
else
|
134
|
-
s.add_dependency(%q<chef>, ["
|
135
|
-
s.add_dependency(%q<fog>, ["~> 1.2
|
136
|
-
s.add_dependency(%q<formatador>, ["~> 0.2
|
137
|
-
s.add_dependency(%q<gorillib>, ["~> 0.1
|
138
|
-
s.add_dependency(%q<bundler>, ["
|
139
|
-
s.add_dependency(%q<jeweler>, ["
|
136
|
+
s.add_dependency(%q<chef>, [">= 0.10.4"])
|
137
|
+
s.add_dependency(%q<fog>, ["~> 1.2"])
|
138
|
+
s.add_dependency(%q<formatador>, ["~> 0.2"])
|
139
|
+
s.add_dependency(%q<gorillib>, ["~> 0.1"])
|
140
|
+
s.add_dependency(%q<bundler>, [">= 1.1"])
|
141
|
+
s.add_dependency(%q<jeweler>, [">= 1.6"])
|
140
142
|
s.add_dependency(%q<rspec>, ["~> 2.5"])
|
141
|
-
s.add_dependency(%q<yard>, ["
|
143
|
+
s.add_dependency(%q<yard>, [">= 0.6"])
|
142
144
|
s.add_dependency(%q<redcarpet>, ["~> 2"])
|
143
145
|
end
|
144
146
|
else
|
145
|
-
s.add_dependency(%q<chef>, ["
|
146
|
-
s.add_dependency(%q<fog>, ["~> 1.2
|
147
|
-
s.add_dependency(%q<formatador>, ["~> 0.2
|
148
|
-
s.add_dependency(%q<gorillib>, ["~> 0.1
|
149
|
-
s.add_dependency(%q<bundler>, ["
|
150
|
-
s.add_dependency(%q<jeweler>, ["
|
147
|
+
s.add_dependency(%q<chef>, [">= 0.10.4"])
|
148
|
+
s.add_dependency(%q<fog>, ["~> 1.2"])
|
149
|
+
s.add_dependency(%q<formatador>, ["~> 0.2"])
|
150
|
+
s.add_dependency(%q<gorillib>, ["~> 0.1"])
|
151
|
+
s.add_dependency(%q<bundler>, [">= 1.1"])
|
152
|
+
s.add_dependency(%q<jeweler>, [">= 1.6"])
|
151
153
|
s.add_dependency(%q<rspec>, ["~> 2.5"])
|
152
|
-
s.add_dependency(%q<yard>, ["
|
154
|
+
s.add_dependency(%q<yard>, [">= 0.6"])
|
153
155
|
s.add_dependency(%q<redcarpet>, ["~> 2"])
|
154
156
|
end
|
155
157
|
end
|
@@ -0,0 +1,148 @@
|
|
1
|
+
bash <<'EOF' || echo "Chef bootstrap failed!"
|
2
|
+
|
3
|
+
# This is the ubuntu natty bootstrap script from infochimps' ironfan. It is
|
4
|
+
# based on opscode's bootstrap script, with the following important differences:
|
5
|
+
#
|
6
|
+
# * installs ruby 1.9.2 (not 1.8.7) from source
|
7
|
+
# * upgrades rubygems rather than installing from source
|
8
|
+
# * pushes the node identity into the first-boot.json
|
9
|
+
# * installs the chef-client service and kicks off the first run of chef
|
10
|
+
|
11
|
+
set -e
|
12
|
+
|
13
|
+
<%= (@config[:verbosity].to_i > 1 ? 'set -v' : '') %>
|
14
|
+
|
15
|
+
RUBY_VERSION=1.9.2-p290
|
16
|
+
CHEF_VERSION=<%= bootstrap_version_string.gsub(/.*[\s=]/,"") %>
|
17
|
+
|
18
|
+
mkdir -p /tmp/knife-bootstrap
|
19
|
+
chmod 700 /tmp/knife-bootstrap
|
20
|
+
cd /tmp/knife-bootstrap
|
21
|
+
|
22
|
+
<%= "export http_proxy=\"#{knife_config[:bootstrap_proxy]}\"" if knife_config[:bootstrap_proxy] -%>
|
23
|
+
cat /etc/centos-release
|
24
|
+
|
25
|
+
date > /etc/box_build_time
|
26
|
+
|
27
|
+
echo -e "`date` \n\n**** \n**** yum upgrade:\n****\n"
|
28
|
+
yum upgrade --assumeyes
|
29
|
+
|
30
|
+
echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
|
31
|
+
yum install --assumeyes make wget
|
32
|
+
yum install --assumeyes git rpm-build rpmdevtools gcc glibc-static zlib-devel libxml2-devel libxslt-devel openssl-devel
|
33
|
+
if [ ! -d runit-rpm ]; then git clone https://github.com/imeyer/runit-rpm.git; fi
|
34
|
+
cd runit-rpm
|
35
|
+
./build.sh
|
36
|
+
yum install --assumeyes /root/rpmbuild/RPMS/x86_64/runit-*.rpm || true # TODO: Remove this shim
|
37
|
+
cd -
|
38
|
+
|
39
|
+
yum clean all
|
40
|
+
|
41
|
+
if [ ! -f /usr/bin/chef-client ]; then
|
42
|
+
echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
|
43
|
+
|
44
|
+
wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-${RUBY_VERSION}.tar.gz
|
45
|
+
tar xzf ruby-${RUBY_VERSION}.tar.gz
|
46
|
+
cd ruby-${RUBY_VERSION}
|
47
|
+
./configure --with-ruby-version=${RUBY_VERSION} --prefix=/usr --program-suffix=${RUBY_VERSION}
|
48
|
+
make -j2
|
49
|
+
make install
|
50
|
+
|
51
|
+
alternatives \
|
52
|
+
--install /usr/bin/ruby ruby /usr/bin/ruby${RUBY_VERSION} 400 \
|
53
|
+
--slave /usr/bin/ri ri /usr/bin/ri${RUBY_VERSION} \
|
54
|
+
--slave /usr/bin/irb irb /usr/bin/irb${RUBY_VERSION} \
|
55
|
+
--slave /usr/bin/erb erb /usr/bin/erb${RUBY_VERSION} \
|
56
|
+
--slave /usr/bin/gem gem /usr/bin/gem${RUBY_VERSION} \
|
57
|
+
--slave /usr/share/man/man1/ruby.1.gz ruby.1.gz \
|
58
|
+
/usr/share/man/man1/ruby${RUBY_VERSION}.1
|
59
|
+
|
60
|
+
echo -e "`date` \n\n**** \n**** Updating rubygems:\n****\n"
|
61
|
+
gem update --system
|
62
|
+
|
63
|
+
echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
|
64
|
+
gem install ohai --no-rdoc --no-ri
|
65
|
+
gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
|
66
|
+
# gems needed for the client.rb or so generically useful you want them at hand
|
67
|
+
gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
|
68
|
+
|
69
|
+
else # no chef-client
|
70
|
+
echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
|
71
|
+
fi # end ruby+chef install
|
72
|
+
|
73
|
+
echo -e "`date` \n\n**** \n**** Knifing in the chef client config files:\n****\n"
|
74
|
+
mkdir -p /etc/chef
|
75
|
+
|
76
|
+
<%- if @config[:client_key] %>
|
77
|
+
(
|
78
|
+
cat <<'EOP'
|
79
|
+
<%= @config[:client_key] %>
|
80
|
+
EOP
|
81
|
+
) > /tmp/knife-bootstrap/client.pem
|
82
|
+
awk NF /tmp/knife-bootstrap/client.pem > /etc/chef/client.pem
|
83
|
+
<%- else %>
|
84
|
+
(
|
85
|
+
cat <<'EOP'
|
86
|
+
<%= validation_key %>
|
87
|
+
EOP
|
88
|
+
) > /tmp/knife-bootstrap/validation.pem
|
89
|
+
awk NF /tmp/knife-bootstrap/validation.pem > /etc/chef/validation.pem
|
90
|
+
<%- end %>
|
91
|
+
|
92
|
+
echo -e "`date` \n\n**** \n**** Nuking our temp files:\n****\n"
|
93
|
+
|
94
|
+
cd /tmp
|
95
|
+
rm -rf /tmp/knife-bootstrap
|
96
|
+
|
97
|
+
echo -e "`date` \n\n**** \n**** Creating chef client script:\n****\n"
|
98
|
+
|
99
|
+
(
|
100
|
+
cat <<'EOP'
|
101
|
+
<%= config_content %>
|
102
|
+
<%= @config[:node].chef_client_script_content %>
|
103
|
+
EOP
|
104
|
+
) > /etc/chef/client.rb
|
105
|
+
|
106
|
+
(
|
107
|
+
cat <<'EOP'
|
108
|
+
<%= { "run_list" => @run_list, "cluster_name" => @config[:node].cluster_name, "facet_name" => @config[:node].facet_name, "facet_index" => @config[:node].facet_index }.to_json %>
|
109
|
+
EOP
|
110
|
+
) > /etc/chef/first-boot.json
|
111
|
+
|
112
|
+
echo -e "`date` \n\n**** \n**** Adding chef client runit scripts:\n****\n"
|
113
|
+
( service chef-client stop >/dev/null 2>&1 ; sleep 1 ; killall chef-client 2>/dev/null ) || true
|
114
|
+
mkdir -p /var/log/chef /var/chef /etc/service /etc/sv/chef-client/{log/main,supervise}
|
115
|
+
|
116
|
+
cat > /etc/sv/chef-client/log/run <<'EOP'
|
117
|
+
#!/bin/bash
|
118
|
+
exec svlogd -tt ./main
|
119
|
+
EOP
|
120
|
+
|
121
|
+
cat > /etc/sv/chef-client/run <<'EOP'
|
122
|
+
#!/bin/bash
|
123
|
+
exec 2>&1
|
124
|
+
exec /usr/bin/env chef-client -i 43200 -s 20 -L /var/log/chef/client.log
|
125
|
+
EOP
|
126
|
+
|
127
|
+
chmod +x /etc/sv/chef-client/log/run /etc/sv/chef-client/run
|
128
|
+
ln -nfs /sbin/sv /etc/init.d/chef-client
|
129
|
+
|
130
|
+
service chef-client stop >/dev/null 2>&1 || true
|
131
|
+
|
132
|
+
<%- if (@config[:bootstrap_runs_chef_client].to_s == 'true') || (@chef_config.knife[:bootstrap_runs_chef_client].to_s == 'true') %>
|
133
|
+
echo -e "`date` \n\n**** \n**** First run of chef:\n****\n"
|
134
|
+
set -e
|
135
|
+
<%= start_chef %>
|
136
|
+
set +e
|
137
|
+
<%- end %>
|
138
|
+
|
139
|
+
echo -e "`date` \n\n**** \n**** Cleanup:\n****\n"
|
140
|
+
# make locate work good
|
141
|
+
updatedb
|
142
|
+
|
143
|
+
echo -e "`date` \n\n**** \n**** Enabling chef client service:\n****\n"
|
144
|
+
ln -nfs /etc/sv/chef-client /etc/service/chef-client
|
145
|
+
service chef-client start
|
146
|
+
|
147
|
+
echo -e "`date` \n\n**** \n**** Cluster Chef client bootstrap complete\n****\n"
|
148
|
+
EOF
|
@@ -26,14 +26,20 @@ export DEBIAN_FRONTEND=noninteractive
|
|
26
26
|
date > /etc/box_build_time
|
27
27
|
|
28
28
|
echo -e "`date` \n\n**** \n**** apt update:\n****\n"
|
29
|
-
apt-get -y update
|
30
|
-
apt-get -y upgrade
|
29
|
+
apt-get --force-yes -y update
|
30
|
+
apt-get --force-yes -y upgrade
|
31
31
|
|
32
32
|
echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
|
33
|
-
apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
|
34
|
-
apt-get -y install runit-services
|
33
|
+
apt-get --force-yes -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
|
34
|
+
apt-get --force-yes -y install runit-services
|
35
35
|
apt-get clean
|
36
36
|
|
37
|
+
# do not trust the ubuntu chef package
|
38
|
+
rm -f /etc/init.d/chef-client || true
|
39
|
+
apt-get remove -y --force-yes chef ruby1.8-dev libjson-ruby1.8 libmixlib-authentication-ruby1.8 ohai libmixlib-log-ruby1.8 libmime-types-ruby librestclient-ruby1.8 ruby1.8 ruby liberubis-ruby1.8 libsystemu-ruby1.8 libohai-ruby libuuidtools-ruby1.8 libhighline-ruby1.8 libabstract-ruby1.8 libmixlib-config-ruby1.8 rubygems1.8 libbunny-ruby1.8 libchef-ruby1.8 libmixlib-cli-ruby1.8 libyajl-ruby libmoneta-ruby1.8 libohai-ruby1.8 libextlib-ruby1.8 || true
|
40
|
+
# apt-get remove -y --force-yes --purge myproxy globus-core condor nis autofs myproxy-server globus-proxy-utils globus-repository-natty globus-gridftp-server-progs globus-gass-copy-progs globus-simple-ca globus-gsi-cert-utils-progs || true
|
41
|
+
sudo apt-get autoremove -y --force-yes || true
|
42
|
+
|
37
43
|
if [ ! -f /usr/bin/chef-client ]; then
|
38
44
|
echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
|
39
45
|
|
@@ -69,7 +75,7 @@ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
|
|
69
75
|
gem install ohai --no-rdoc --no-ri
|
70
76
|
gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
|
71
77
|
# gems needed for the client.rb or so generically useful you want them at hand
|
72
|
-
gem install --no-rdoc --no-ri extlib bundler json right_aws pry
|
78
|
+
gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
|
73
79
|
|
74
80
|
else # no chef-client
|
75
81
|
echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
|
@@ -26,21 +26,27 @@ export DEBIAN_FRONTEND=noninteractive
|
|
26
26
|
date > /etc/box_build_time
|
27
27
|
|
28
28
|
# source for sun java if you want to install it later
|
29
|
-
apt-get install -y python-software-properties
|
29
|
+
apt-get install --force-yes -y python-software-properties
|
30
30
|
add-apt-repository -y ppa:ferramroberto/java
|
31
31
|
|
32
32
|
echo -e "`date` \n\n**** \n**** apt update:\n****\n"
|
33
|
-
apt-get -y update
|
34
|
-
apt-get -y upgrade
|
33
|
+
apt-get --force-yes -y update
|
34
|
+
apt-get --force-yes -y upgrade
|
35
35
|
|
36
36
|
echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
|
37
|
-
apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
|
37
|
+
apt-get --force-yes -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
|
38
38
|
apt-get clean
|
39
39
|
|
40
|
+
# do not trust the ubuntu chef package
|
41
|
+
rm -f /etc/init.d/chef-client || true
|
42
|
+
apt-get remove -y --force-yes chef ruby1.8-dev libjson-ruby1.8 libmixlib-authentication-ruby1.8 ohai libmixlib-log-ruby1.8 libmime-types-ruby librestclient-ruby1.8 ruby1.8 ruby liberubis-ruby1.8 libsystemu-ruby1.8 libohai-ruby libuuidtools-ruby1.8 libhighline-ruby1.8 libabstract-ruby1.8 libmixlib-config-ruby1.8 rubygems1.8 libbunny-ruby1.8 libchef-ruby1.8 libmixlib-cli-ruby1.8 libyajl-ruby libmoneta-ruby1.8 libohai-ruby1.8 libextlib-ruby1.8 || true
|
43
|
+
# apt-get remove -y --force-yes --purge myproxy globus-core condor nis autofs myproxy-server globus-proxy-utils globus-repository-natty globus-gridftp-server-progs globus-gass-copy-progs globus-simple-ca globus-gsi-cert-utils-progs || true
|
44
|
+
sudo apt-get autoremove -y --force-yes || true
|
45
|
+
|
40
46
|
if [ ! -f /usr/bin/chef-client ]; then
|
41
47
|
echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
|
42
48
|
|
43
|
-
apt-get install -y ruby1.9.1 ruby1.9.1-dev
|
49
|
+
apt-get install --force-yes -y ruby1.9.1 ruby1.9.1-dev
|
44
50
|
|
45
51
|
if ruby -e "exit(%x{gem --version} < \"1.6.2\" ? 0 : -1 )" ; then
|
46
52
|
echo -e "`date` \n\n**** \n**** Updating rubygems:\n****\n"
|
@@ -57,7 +63,7 @@ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
|
|
57
63
|
gem install ohai --no-rdoc --no-ri
|
58
64
|
gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
|
59
65
|
# gems needed for the client.rb or so generically useful you want them at hand
|
60
|
-
gem install --no-rdoc --no-ri extlib bundler json right_aws pry
|
66
|
+
gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
|
61
67
|
|
62
68
|
else # no chef-client
|
63
69
|
echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
|
@@ -59,9 +59,10 @@ class Chef
|
|
59
59
|
end
|
60
60
|
|
61
61
|
def perform_execution(target)
|
62
|
-
|
63
|
-
|
64
|
-
|
62
|
+
# Execute across all servers in parallel
|
63
|
+
threads = target.servers.map{ |server| Thread.new(server) { |svr| run_bootstrap(svr, svr.public_hostname) } }
|
64
|
+
# Wait for the threads to finish and return the array of thread's exit value
|
65
|
+
threads.map{ |t| t.join.value }
|
65
66
|
end
|
66
67
|
|
67
68
|
def confirm_execution(target)
|