ironfan 3.1.5 → 3.1.6

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. data/Gemfile +7 -8
  2. data/README.md +6 -0
  3. data/TODO.md +140 -29
  4. data/VERSION +1 -1
  5. data/config/ubuntu10.04-ironfan.erb +11 -5
  6. data/config/ubuntu11.10-ironfan.erb +12 -6
  7. data/ironfan.gemspec +25 -23
  8. data/lib/chef/knife/bootstrap/centos6.2-ironfan.erb +148 -0
  9. data/lib/chef/knife/bootstrap/ubuntu10.04-ironfan.erb +11 -5
  10. data/lib/chef/knife/bootstrap/ubuntu11.10-ironfan.erb +12 -6
  11. data/lib/chef/knife/cluster_bootstrap.rb +4 -3
  12. data/lib/chef/knife/cluster_kick.rb +9 -8
  13. data/lib/chef/knife/cluster_kill.rb +1 -0
  14. data/lib/chef/knife/cluster_launch.rb +2 -2
  15. data/lib/chef/knife/cluster_proxy.rb +5 -4
  16. data/lib/chef/knife/cluster_show.rb +1 -1
  17. data/lib/chef/knife/cluster_ssh.rb +6 -5
  18. data/lib/chef/knife/cluster_stop.rb +3 -0
  19. data/lib/chef/knife/cluster_vagrant.rb +2 -2
  20. data/lib/chef/knife/ironfan_knife_common.rb +1 -1
  21. data/lib/chef/knife/vagrant/skeleton_vagrantfile.rb +5 -2
  22. data/lib/ironfan/chef_layer.rb +3 -1
  23. data/lib/ironfan/cloud.rb +146 -107
  24. data/lib/ironfan/discovery.rb +13 -0
  25. data/lib/ironfan/fog_layer.rb +42 -3
  26. data/lib/ironfan/security_group.rb +16 -4
  27. data/lib/ironfan/server.rb +29 -0
  28. data/lib/ironfan.rb +1 -1
  29. data/notes/INSTALL-cloud_setup.md +14 -11
  30. data/notes/INSTALL.md +6 -1
  31. data/notes/core_concepts.md +1 -1
  32. data/notes/ec2-pricing_and_capacity.md +24 -18
  33. data/notes/version-3_2.md +273 -0
  34. data/spec/spec_helper.rb +26 -34
  35. metadata +31 -29
data/Gemfile CHANGED
@@ -1,20 +1,19 @@
1
1
  source "http://rubygems.org"
2
2
 
3
- gem 'chef', "~> 0.10.4"
4
- gem 'fog', "~> 1.2.0"
5
- gem 'formatador', "~> 0.2.1"
6
- gem 'gorillib', "~> 0.1.7"
3
+ gem 'chef', ">= 0.10.4"
4
+ gem 'fog', "~> 1.2"
5
+ gem 'formatador', "~> 0.2"
6
+ gem 'gorillib', "~> 0.1"
7
7
 
8
8
  group :development do
9
- gem 'bundler', "~> 1"
10
- gem 'jeweler', "~> 1.6"
9
+ gem 'bundler', ">= 1.1"
10
+ gem 'jeweler', ">= 1.6"
11
11
  gem 'rspec', "~> 2.5"
12
- gem 'yard', "~> 0.6"
12
+ gem 'yard', ">= 0.6"
13
13
  gem 'redcarpet', "~> 2"
14
14
  end
15
15
 
16
16
  group :test do
17
- gem 'spork', ">= 0.9.0", :platform => :mri
18
17
  gem 'rcov', ">= 0.9.9", :platform => :ruby_18
19
18
  gem 'simplecov', ">= 0.5", :platform => :ruby_19
20
19
  #
data/README.md CHANGED
@@ -34,9 +34,15 @@ Ironfan consists of the following Toolset:
34
34
  * [Ironfan Screencast](http://bit.ly/ironfan-hadoop-in-20-minutes) -- build a Hadoop cluster from scratch in 20 minutes.
35
35
  * Ironfan powers the [Infochimps Platform](http://www.infochimps.com/how-it-works), our scalable enterprise big data platform. Ironfan Enterprise adds zero-configuration logging, monitoring and a compelling UI.
36
36
 
37
+ **Note**: Ironfan is [not compatible with Ruby 1.8](https://github.com/infochimps-labs/ironfan/issues/127). All versions later than 1.9.2-p136 should work fine.
38
+
37
39
  ### The Ironfan Way
38
40
 
39
41
  * [Core Concepts](https://github.com/infochimps-labs/ironfan/wiki/core_concepts) -- Components, Announcements, Amenities and more.
40
42
  * [Philosophy](https://github.com/infochimps-labs/ironfan/wiki/Philosophy) -- Best practices and lessons learned
41
43
  * [Style Guide](https://github.com/infochimps-labs/ironfan/wiki/style_guide) -- Common attribute names, how and when to include other cookbooks, and more
42
44
  * [Homebase Layout](https://github.com/infochimps-labs/ironfan/wiki/homebase-layout) -- How this homebase is organized, and why
45
+
46
+ ### Getting Help
47
+ * Feel free to contact us at info@infochimps.com or 855-DATA-FUN
48
+ * Also, you invited to a [private consultation](http://www.infochimps.com/free-big-data-consultation?utm_source=git&utm_medium=referral&utm_campaign=consult) with Infochimps founders on your big data project.
data/TODO.md CHANGED
@@ -1,44 +1,155 @@
1
- ### Knife commands
1
+ * refactor standard dirs and volume_dirs into a single, more coherent resource
2
+
3
+ * We should use the `Fog::Compute::AWS::FLAVORS` constant that [fog defines](http://rubydoc.info/github/fog/fog/master/Fog/Compute/AWS) in the cloud code (instead of the one we put there)
4
+
5
+ * All over the place there is the following construct (absolutely necessary, absolutely horrid):
6
+
7
+ foo = Mash.new().merge(node[:system]).merge(node[:system][:component])
8
+
9
+ You might look at this and think "gee I know a much simpler way to do that". That simpler way does not work; this way does.
10
+
11
+ I propose adding a 'smush' method to `silverware/libraries/cookbook_utils`:
12
+
13
+ ```ruby
14
+ module Ironfan::CookbookUtils
15
+ module_function
16
+
17
+ # Merge the given objects (node attributes, hashes, or anything
18
+ # else with `#to_hash`) into a combined `Mash` object. Objects
19
+ # given later in the list 'win' over objects given earlier.
20
+ #
21
+ # @examples
22
+ # template_vars = Ironfan::CookbookUtils.smush( node[:flume], node[:flume][:agent], :zookeeper_port => node[:zookeeper][:port] )
23
+ #
24
+ # @param [Array[#to_hash]] smushables -- any number of things that respond to `#to_hash`
25
+ #
26
+ def smush(*smushables)
27
+ result = Mash.new
28
+ smushables.compact.each do |smushable|
29
+ result.merge! smushable.to_hash
30
+ end
31
+ result
32
+ end
33
+
34
+ end
2
35
 
3
- * knife cluster launch should fail differently if you give it a facet that doesn't exist
36
+ (obviously the hard part is not writing the method, it's applying it to all the cookbooks.)
4
37
 
5
38
 
39
+
40
+ ### Knife commands
41
+
6
42
  * reify notion of 'homebase'; cluster commands work off it
7
43
  * move away from referring to Chef::Config everywhere;
8
44
 
9
45
 
10
- __________________________________________________________________________
11
46
 
12
- Rename completed,
13
47
 
14
- New homes:
15
48
 
16
- * **http://github.com/infochimps-labs/ironfan** -- the primary destination, and the home of the knife tools (pretty much what's here in `infochimps/cluster_chef`)
17
- * **http://github.com/infochimps-labs/ironfan-pantry** -- collection of public cookbooks, roles and demo clusters
18
- * **http://github.com/infochimps-labs/ironfan-homebase** -- skeleton homebase (looks a lot like the current cluster_chef-homebase)
19
- * **http://github.com/infochimps-labs/ironfan-ci** -- continuous integration stuff (homebase/vagrants)
20
- * **http://github.com/infochimps-labs/ironfan-scrubby** -- de-linter/simulator (`cluster_chef/lib/cluster_chef/cookbook_munger*`)
21
- * **http://github.com/infochimps-labs/opscode-cookbooks** -- our fork of the opscode cookbooks repo
49
+ # Old Issues Triage
50
+ From https://github.com/infochimps-labs/ironfan/issues/102 with love. Deleted ownership, and everything that was done or has its own issue.
51
+
52
+ ## Must Do
53
+ * merge volumes into silverware. merge ebs_volumes into ec2 cookbook
54
+ * Basic CI testing of cookbooks
55
+ * RSpecs for silverback (lib and knife tools)
56
+ * RSpecs for silverware are mostly in place -- ensure they are.
57
+ * push cookbooks to community.opscode.com
58
+ * refine and explain updated git workflow
59
+
60
+ ## Docco
61
+ Use the [opscode EC2 fast start](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide) as a guide -- our getting started should start at the same place, and cover the same detail as the EC2 bootstrap guide.
62
+
63
+ * Clear description of metadiscovery
64
+ * make sure README files in cookbooks aren’t wildly inaccurate
65
+ * Carry out setup directions, ensure they work:
66
+ - cluster_chef if you’re using our homebase
67
+ - cluster_chef if you’re using opscode’s homebase
68
+ * local vagrant environment
69
+ * hadoop cluster bootstrapping
70
+
71
+ ## Piddly Shit
72
+
73
+ * standardize the `zabbix` cookbook (no more /opt, etc -- more in the TODO)
74
+ * volumes don't deep merge -- eg you have to mount_ephemerals in the facet if you modify htem
75
+ * kill_old_service should disable services (may be leaving /etc/rc.d cruft).
76
+ * kill old service doesn't go the first time. why?
77
+ * chef client/server cookbook: set chef user UID / GID; client can set log directory
78
+ * apt has a dashboard at http://{hostname}:3142/report
79
+ * can use knife ssh as me@ or as ubuntu@
80
+ * knife command to set/remove permanent on a node + disableApiTermination on box. knife cluster kill refuses to delete nodes with permanent set. knife cluster sync sets permanent on if permanent(true), removes if permanent(false), ignores if permanent nil or unset.
81
+ * style-guide alignment (prefix_root becomes prefix)
82
+
83
+ ## Really Want
84
+
85
+ * unify the hashlike underpinning to be same across silverware & cluster_chef. Make sure we love (or accept) all the differences between it and Gorrillib’s, and between it and Chef’s.
86
+ * Keys are transmitted in databags, using a helper, and not in node attributes
87
+ * easy to create a dummy node (load balancer, external resource, etc)
88
+ * components can have arbitrary attributes (kinda. they take an `:info` param, behavior which may change later)
89
+ * All cookbooks have nice detailed announcements
90
+ * full roll out of log_integration, monitoring
91
+ * Git deploy abstraction similar to `install_from`
92
+
93
+ ## Cookbook checklist:
94
+ * Validate all the cookbooks against checklist -- see notes/README-checklist.md
95
+
96
+ | flip fixed | temujin9 checked |
97
+ +------------+------------------+
98
+ cassandra | | |
99
+ ec2 | | |
100
+ elasticsearch | | |
101
+ firewall | | |
102
+ flume | | |
103
+ ganglia | | |
104
+ graphite | | |
105
+ hadoop_cluster | | |
106
+ hbase | | |
107
+ hive | | |
108
+ jenkins | | |
109
+ jruby | | |
110
+ nfs | | |
111
+ nodejs | | |
112
+ papertrail | | |
113
+ pig | | |
114
+ redis | | |
115
+ resque | | |
116
+ Rstats | | |
117
+ statsd | | |
118
+ zookeeper | | |
119
+ # meta:
120
+ install_from | | |
121
+ motd | | |
122
+ mountable_volumes | | |
123
+ provides_service | | |
124
+ # Need thinkin':
125
+ big_package | | |
126
+ cluster_chef | | |
127
+
22
128
 
23
- ## Phase 1: Regex Replace
129
+ ## Things that are probably straightforward to fix as soon as we know how
24
130
 
25
- 0. DONE warn about the name change. Since the `version_3` branch is left pre-namechange, decision was made to do a pull request when things are pull-able, not spam people before.
26
- 1. DONE Make a branch in each repo (`cluster_chef` and `*-homebase`) called `before_rename`, holding the current state of the code. Make another branch `ironfan`, where the renames will occur. That branch will be deleted as soon as the merge lands, hopefully before anyone even notices.
27
- 2. DONE Make sure `el_ridiculoso` is up to date and that it fully converges in the vagrant and cloud environments.
28
- 3. DONE Ensure all infochimps devs have push-pulled to various repos. make a tarball of the repos (.git and everything) and put them somewhere safe.
29
- 4. DONE rename the `vendor/infochimps/metachef` cookbook as vendor/infochimps/silverware.
30
- 5. DONE Bump the *minor* version number on all cookbooks (so 3.0.x => 3.1.x). Commit.
31
- 6. DONE regex-replace `ClusterChef` => `Ironfan`, `cluster_chef` to `ironfan` and `[Mm]etachef` => `[Ss]ilverware`. Do this in `cluster_chef` and `infochimps-labs/ironfan-homebase` only. Get chef-client to complete on el_ridiculoso using an emptied-and-reloaded local chef server. Run knife cluster sync on all clusters in all homebases against local chef server.
32
- 7. ...
33
- 12. DONE once the name is updated in the gemspec, release the `ironfan` gem. @temujin9 and I agree that it should be a single combined gem now that we won't have the stupid `_chef` gem name conflict.
131
+ * announcements should probably be published very early, but they need to know lots about the machine YUK
132
+ * split between clusters / roles / integration cookbooks
133
+ * inheritance of clusters
34
134
 
35
- ## Phase 2: Repo migration
135
+ ## Things We Hate But Might Have to Continue Hating
36
136
 
37
- 1. DONE back up the repo and put it somewhere safe.
38
- 2. DONE Transfer ownership of `infochimps/cluster_chef` to `infochimps-labs/cluster_chef`. Do whatever is necessary/possible to migrate issues over.
39
- 3. DONE Change name of `infochimps-labs/cluster_chef` to `infochimps-labs/ironfan`. Created new repo `infochimps-labs/ironfan-homebase`. 4. DONE Create new repo `infochimps/cluster_chef` with helpful link to new repo. I don't think we need a placeholder at `infochimps-labs/cluster_chef-homebase`.
40
- 5. DONE Rename master branch of old cluster chef to be `version_2`, and freeze the `version_3`. Both now carry a deprecation warning pointing you to ironfan.
41
- 6. IN PROGRESS Update any remaining documentation links, etc to point to new home.
42
- 7. IN PROGRESS Have all devs edit their .git/config to point at right place.
137
+ * Cluster refactor -- clusters / stacks / components, not clusters / roles / cookbooks
138
+ * move cluster discovery to cloud class.
139
+ * Server#normalize! doesn’t imprint object (ie. server attributes poke through to the facet & cluster, rather than being *set* on the object)
140
+ * The fact you can only see one cluster at a time is stupid.
141
+ * security group pairing is sucky.
142
+ * ubuntu home drive bullshit
143
+ * Finer-grained security group control (eg nfs server only opens a couple ports, not all)
144
+ * nfs recipe uses discovery right (thus allowing more than one NFS share to exist in the universe)
145
+ * roles UGGGHHHHAERWSDFKHSBLAH
43
146
 
44
- so here is the dev-level worksforme -- @temujin9 will close once he's given it the sysadmin-level worksforme
147
+ ## Ponies!
148
+ * sync cookbooks up/down to `infochimps-cookbooks/` 
149
+   - note: infochimps-cookbooks the org will be dereferenced in favor of ironfan-lib the single repo; it's unclear which pull requesters will prefer. We will do at least one push so that names and URLs are current, and we're not removing anything, but infochimps-cookbooks has an unclear future.
150
+ * foodcritic compatibility
151
+ * build out cookbook munger, make it less spike-y
152
+ * spot pricing
153
+ * rackspace compatibility
154
+ * cookbook munger reads comments in attributes file to populate metadata.rb
155
+ * `gem install ironfan; ironfan install` checks everything out
data/VERSION CHANGED
@@ -1 +1 @@
1
- 3.1.5
1
+ 3.1.6
data/config/ubuntu10.04-ironfan.erb CHANGED
@@ -26,14 +26,20 @@ export DEBIAN_FRONTEND=noninteractive
26
26
  date > /etc/box_build_time
27
27
 
28
28
  echo -e "`date` \n\n**** \n**** apt update:\n****\n"
29
- apt-get -y update
30
- apt-get -y upgrade
29
+ apt-get --force-yes -y update
30
+ apt-get --force-yes -y upgrade
31
31
 
32
32
  echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
33
- apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
34
- apt-get -y install runit-services
33
+ apt-get --force-yes -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
34
+ apt-get --force-yes -y install runit-services
35
35
  apt-get clean
36
36
 
37
+ # do not trust the ubuntu chef package
38
+ rm -f /etc/init.d/chef-client || true
39
+ apt-get remove -y --force-yes chef ruby1.8-dev libjson-ruby1.8 libmixlib-authentication-ruby1.8 ohai libmixlib-log-ruby1.8 libmime-types-ruby librestclient-ruby1.8 ruby1.8 ruby liberubis-ruby1.8 libsystemu-ruby1.8 libohai-ruby libuuidtools-ruby1.8 libhighline-ruby1.8 libabstract-ruby1.8 libmixlib-config-ruby1.8 rubygems1.8 libbunny-ruby1.8 libchef-ruby1.8 libmixlib-cli-ruby1.8 libyajl-ruby libmoneta-ruby1.8 libohai-ruby1.8 libextlib-ruby1.8 || true
40
+ # apt-get remove -y --force-yes --purge myproxy globus-core condor nis autofs myproxy-server globus-proxy-utils globus-repository-natty globus-gridftp-server-progs globus-gass-copy-progs globus-simple-ca globus-gsi-cert-utils-progs || true
41
+ sudo apt-get autoremove -y --force-yes || true
42
+
37
43
  if [ ! -f /usr/bin/chef-client ]; then
38
44
  echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
39
45
 
@@ -69,7 +75,7 @@ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
69
75
  gem install ohai --no-rdoc --no-ri
70
76
  gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
71
77
  # gems needed for the client.rb or so generically useful you want them at hand
72
- gem install --no-rdoc --no-ri extlib bundler json right_aws pry
78
+ gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
73
79
 
74
80
  else # no chef-client
75
81
  echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
data/config/ubuntu11.10-ironfan.erb CHANGED
@@ -26,21 +26,27 @@ export DEBIAN_FRONTEND=noninteractive
26
26
  date > /etc/box_build_time
27
27
 
28
28
  # source for sun java if you want to install it later
29
- apt-get install -y python-software-properties
29
+ apt-get install --force-yes -y python-software-properties
30
30
  add-apt-repository -y ppa:ferramroberto/java
31
31
 
32
32
  echo -e "`date` \n\n**** \n**** apt update:\n****\n"
33
- apt-get -y update
34
- apt-get -y upgrade
33
+ apt-get --force-yes -y update
34
+ apt-get --force-yes -y upgrade
35
35
 
36
36
  echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
37
- apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
37
+ apt-get --force-yes -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
38
38
  apt-get clean
39
39
 
40
+ # do not trust the ubuntu chef package
41
+ rm -f /etc/init.d/chef-client || true
42
+ apt-get remove -y --force-yes chef ruby1.8-dev libjson-ruby1.8 libmixlib-authentication-ruby1.8 ohai libmixlib-log-ruby1.8 libmime-types-ruby librestclient-ruby1.8 ruby1.8 ruby liberubis-ruby1.8 libsystemu-ruby1.8 libohai-ruby libuuidtools-ruby1.8 libhighline-ruby1.8 libabstract-ruby1.8 libmixlib-config-ruby1.8 rubygems1.8 libbunny-ruby1.8 libchef-ruby1.8 libmixlib-cli-ruby1.8 libyajl-ruby libmoneta-ruby1.8 libohai-ruby1.8 libextlib-ruby1.8 || true
43
+ # apt-get remove -y --force-yes --purge myproxy globus-core condor nis autofs myproxy-server globus-proxy-utils globus-repository-natty globus-gridftp-server-progs globus-gass-copy-progs globus-simple-ca globus-gsi-cert-utils-progs || true
44
+ sudo apt-get autoremove -y --force-yes || true
45
+
40
46
  if [ ! -f /usr/bin/chef-client ]; then
41
47
  echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
42
48
 
43
- apt-get install -y ruby1.9.1 ruby1.9.1-dev
49
+ apt-get install --force-yes -y ruby1.9.1 ruby1.9.1-dev
44
50
 
45
51
  if ruby -e "exit(%x{gem --version} < \"1.6.2\" ? 0 : -1 )" ; then
46
52
  echo -e "`date` \n\n**** \n**** Updating rubygems:\n****\n"
@@ -57,7 +63,7 @@ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
57
63
  gem install ohai --no-rdoc --no-ri
58
64
  gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
59
65
  # gems needed for the client.rb or so generically useful you want them at hand
60
- gem install --no-rdoc --no-ri extlib bundler json right_aws pry
66
+ gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
61
67
 
62
68
  else # no chef-client
63
69
  echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
data/ironfan.gemspec CHANGED
@@ -5,11 +5,11 @@
5
5
 
6
6
  Gem::Specification.new do |s|
7
7
  s.name = "ironfan"
8
- s.version = "3.1.5"
8
+ s.version = "3.1.6"
9
9
 
10
10
  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
11
11
  s.authors = ["Infochimps"]
12
- s.date = "2012-04-02"
12
+ s.date = "2012-06-24"
13
13
  s.description = "Ironfan allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks."
14
14
  s.email = "coders@infochimps.com"
15
15
  s.extra_rdoc_files = [
@@ -34,6 +34,7 @@ Gem::Specification.new do |s|
34
34
  "config/ubuntu10.04-ironfan.erb",
35
35
  "config/ubuntu11.10-ironfan.erb",
36
36
  "ironfan.gemspec",
37
+ "lib/chef/knife/bootstrap/centos6.2-ironfan.erb",
37
38
  "lib/chef/knife/bootstrap/ubuntu10.04-ironfan.erb",
38
39
  "lib/chef/knife/bootstrap/ubuntu11.10-ironfan.erb",
39
40
  "lib/chef/knife/cluster_bootstrap.rb",
@@ -98,6 +99,7 @@ Gem::Specification.new do |s|
98
99
  "notes/silverware.md",
99
100
  "notes/style_guide.md",
100
101
  "notes/tips_and_troubleshooting.md",
102
+ "notes/version-3_2.md",
101
103
  "notes/walkthrough-hadoop.md",
102
104
  "notes/walkthrough-web.md",
103
105
  "spec/ironfan/cluster_spec.rb",
@@ -121,35 +123,35 @@ Gem::Specification.new do |s|
121
123
  s.specification_version = 3
122
124
 
123
125
  if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
124
- s.add_runtime_dependency(%q<chef>, ["~> 0.10.4"])
125
- s.add_runtime_dependency(%q<fog>, ["~> 1.2.0"])
126
- s.add_runtime_dependency(%q<formatador>, ["~> 0.2.1"])
127
- s.add_runtime_dependency(%q<gorillib>, ["~> 0.1.7"])
128
- s.add_development_dependency(%q<bundler>, ["~> 1"])
129
- s.add_development_dependency(%q<jeweler>, ["~> 1.6"])
126
+ s.add_runtime_dependency(%q<chef>, [">= 0.10.4"])
127
+ s.add_runtime_dependency(%q<fog>, ["~> 1.2"])
128
+ s.add_runtime_dependency(%q<formatador>, ["~> 0.2"])
129
+ s.add_runtime_dependency(%q<gorillib>, ["~> 0.1"])
130
+ s.add_development_dependency(%q<bundler>, [">= 1.1"])
131
+ s.add_development_dependency(%q<jeweler>, [">= 1.6"])
130
132
  s.add_development_dependency(%q<rspec>, ["~> 2.5"])
131
- s.add_development_dependency(%q<yard>, ["~> 0.6"])
133
+ s.add_development_dependency(%q<yard>, [">= 0.6"])
132
134
  s.add_development_dependency(%q<redcarpet>, ["~> 2"])
133
135
  else
134
- s.add_dependency(%q<chef>, ["~> 0.10.4"])
135
- s.add_dependency(%q<fog>, ["~> 1.2.0"])
136
- s.add_dependency(%q<formatador>, ["~> 0.2.1"])
137
- s.add_dependency(%q<gorillib>, ["~> 0.1.7"])
138
- s.add_dependency(%q<bundler>, ["~> 1"])
139
- s.add_dependency(%q<jeweler>, ["~> 1.6"])
136
+ s.add_dependency(%q<chef>, [">= 0.10.4"])
137
+ s.add_dependency(%q<fog>, ["~> 1.2"])
138
+ s.add_dependency(%q<formatador>, ["~> 0.2"])
139
+ s.add_dependency(%q<gorillib>, ["~> 0.1"])
140
+ s.add_dependency(%q<bundler>, [">= 1.1"])
141
+ s.add_dependency(%q<jeweler>, [">= 1.6"])
140
142
  s.add_dependency(%q<rspec>, ["~> 2.5"])
141
- s.add_dependency(%q<yard>, ["~> 0.6"])
143
+ s.add_dependency(%q<yard>, [">= 0.6"])
142
144
  s.add_dependency(%q<redcarpet>, ["~> 2"])
143
145
  end
144
146
  else
145
- s.add_dependency(%q<chef>, ["~> 0.10.4"])
146
- s.add_dependency(%q<fog>, ["~> 1.2.0"])
147
- s.add_dependency(%q<formatador>, ["~> 0.2.1"])
148
- s.add_dependency(%q<gorillib>, ["~> 0.1.7"])
149
- s.add_dependency(%q<bundler>, ["~> 1"])
150
- s.add_dependency(%q<jeweler>, ["~> 1.6"])
147
+ s.add_dependency(%q<chef>, [">= 0.10.4"])
148
+ s.add_dependency(%q<fog>, ["~> 1.2"])
149
+ s.add_dependency(%q<formatador>, ["~> 0.2"])
150
+ s.add_dependency(%q<gorillib>, ["~> 0.1"])
151
+ s.add_dependency(%q<bundler>, [">= 1.1"])
152
+ s.add_dependency(%q<jeweler>, [">= 1.6"])
151
153
  s.add_dependency(%q<rspec>, ["~> 2.5"])
152
- s.add_dependency(%q<yard>, ["~> 0.6"])
154
+ s.add_dependency(%q<yard>, [">= 0.6"])
153
155
  s.add_dependency(%q<redcarpet>, ["~> 2"])
154
156
  end
155
157
  end
data/lib/chef/knife/bootstrap/centos6.2-ironfan.erb ADDED
@@ -0,0 +1,148 @@
1
+ bash <<'EOF' || echo "Chef bootstrap failed!"
2
+
3
+ # This is the ubuntu natty bootstrap script from infochimps' ironfan. It is
4
+ # based on opscode's bootstrap script, with the following important differences:
5
+ #
6
+ # * installs ruby 1.9.2 (not 1.8.7) from source
7
+ # * upgrades rubygems rather than installing from source
8
+ # * pushes the node identity into the first-boot.json
9
+ # * installs the chef-client service and kicks off the first run of chef
10
+
11
+ set -e
12
+
13
+ <%= (@config[:verbosity].to_i > 1 ? 'set -v' : '') %>
14
+
15
+ RUBY_VERSION=1.9.2-p290
16
+ CHEF_VERSION=<%= bootstrap_version_string.gsub(/.*[\s=]/,"") %>
17
+
18
+ mkdir -p /tmp/knife-bootstrap
19
+ chmod 700 /tmp/knife-bootstrap
20
+ cd /tmp/knife-bootstrap
21
+
22
+ <%= "export http_proxy=\"#{knife_config[:bootstrap_proxy]}\"" if knife_config[:bootstrap_proxy] -%>
23
+ cat /etc/centos-release
24
+
25
+ date > /etc/box_build_time
26
+
27
+ echo -e "`date` \n\n**** \n**** yum upgrade:\n****\n"
28
+ yum upgrade --assumeyes
29
+
30
+ echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
31
+ yum install --assumeyes make wget
32
+ yum install --assumeyes git rpm-build rpmdevtools gcc glibc-static zlib-devel libxml2-devel libxslt-devel openssl-devel
33
+ if [ ! -d runit-rpm ]; then git clone https://github.com/imeyer/runit-rpm.git; fi
34
+ cd runit-rpm
35
+ ./build.sh
36
+ yum install --assumeyes /root/rpmbuild/RPMS/x86_64/runit-*.rpm || true # TODO: Remove this shim
37
+ cd -
38
+
39
+ yum clean all
40
+
41
+ if [ ! -f /usr/bin/chef-client ]; then
42
+ echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
43
+
44
+ wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-${RUBY_VERSION}.tar.gz
45
+ tar xzf ruby-${RUBY_VERSION}.tar.gz
46
+ cd ruby-${RUBY_VERSION}
47
+ ./configure --with-ruby-version=${RUBY_VERSION} --prefix=/usr --program-suffix=${RUBY_VERSION}
48
+ make -j2
49
+ make install
50
+
51
+ alternatives \
52
+ --install /usr/bin/ruby ruby /usr/bin/ruby${RUBY_VERSION} 400 \
53
+ --slave /usr/bin/ri ri /usr/bin/ri${RUBY_VERSION} \
54
+ --slave /usr/bin/irb irb /usr/bin/irb${RUBY_VERSION} \
55
+ --slave /usr/bin/erb erb /usr/bin/erb${RUBY_VERSION} \
56
+ --slave /usr/bin/gem gem /usr/bin/gem${RUBY_VERSION} \
57
+ --slave /usr/share/man/man1/ruby.1.gz ruby.1.gz \
58
+ /usr/share/man/man1/ruby${RUBY_VERSION}.1
59
+
60
+ echo -e "`date` \n\n**** \n**** Updating rubygems:\n****\n"
61
+ gem update --system
62
+
63
+ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
64
+ gem install ohai --no-rdoc --no-ri
65
+ gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
66
+ # gems needed for the client.rb or so generically useful you want them at hand
67
+ gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
68
+
69
+ else # no chef-client
70
+ echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
71
+ fi # end ruby+chef install
72
+
73
+ echo -e "`date` \n\n**** \n**** Knifing in the chef client config files:\n****\n"
74
+ mkdir -p /etc/chef
75
+
76
+ <%- if @config[:client_key] %>
77
+ (
78
+ cat <<'EOP'
79
+ <%= @config[:client_key] %>
80
+ EOP
81
+ ) > /tmp/knife-bootstrap/client.pem
82
+ awk NF /tmp/knife-bootstrap/client.pem > /etc/chef/client.pem
83
+ <%- else %>
84
+ (
85
+ cat <<'EOP'
86
+ <%= validation_key %>
87
+ EOP
88
+ ) > /tmp/knife-bootstrap/validation.pem
89
+ awk NF /tmp/knife-bootstrap/validation.pem > /etc/chef/validation.pem
90
+ <%- end %>
91
+
92
+ echo -e "`date` \n\n**** \n**** Nuking our temp files:\n****\n"
93
+
94
+ cd /tmp
95
+ rm -rf /tmp/knife-bootstrap
96
+
97
+ echo -e "`date` \n\n**** \n**** Creating chef client script:\n****\n"
98
+
99
+ (
100
+ cat <<'EOP'
101
+ <%= config_content %>
102
+ <%= @config[:node].chef_client_script_content %>
103
+ EOP
104
+ ) > /etc/chef/client.rb
105
+
106
+ (
107
+ cat <<'EOP'
108
+ <%= { "run_list" => @run_list, "cluster_name" => @config[:node].cluster_name, "facet_name" => @config[:node].facet_name, "facet_index" => @config[:node].facet_index }.to_json %>
109
+ EOP
110
+ ) > /etc/chef/first-boot.json
111
+
112
+ echo -e "`date` \n\n**** \n**** Adding chef client runit scripts:\n****\n"
113
+ ( service chef-client stop >/dev/null 2>&1 ; sleep 1 ; killall chef-client 2>/dev/null ) || true
114
+ mkdir -p /var/log/chef /var/chef /etc/service /etc/sv/chef-client/{log/main,supervise}
115
+
116
+ cat > /etc/sv/chef-client/log/run <<'EOP'
117
+ #!/bin/bash
118
+ exec svlogd -tt ./main
119
+ EOP
120
+
121
+ cat > /etc/sv/chef-client/run <<'EOP'
122
+ #!/bin/bash
123
+ exec 2>&1
124
+ exec /usr/bin/env chef-client -i 43200 -s 20 -L /var/log/chef/client.log
125
+ EOP
126
+
127
+ chmod +x /etc/sv/chef-client/log/run /etc/sv/chef-client/run
128
+ ln -nfs /sbin/sv /etc/init.d/chef-client
129
+
130
+ service chef-client stop >/dev/null 2>&1 || true
131
+
132
+ <%- if (@config[:bootstrap_runs_chef_client].to_s == 'true') || (@chef_config.knife[:bootstrap_runs_chef_client].to_s == 'true') %>
133
+ echo -e "`date` \n\n**** \n**** First run of chef:\n****\n"
134
+ set -e
135
+ <%= start_chef %>
136
+ set +e
137
+ <%- end %>
138
+
139
+ echo -e "`date` \n\n**** \n**** Cleanup:\n****\n"
140
+ # make locate work good
141
+ updatedb
142
+
143
+ echo -e "`date` \n\n**** \n**** Enabling chef client service:\n****\n"
144
+ ln -nfs /etc/sv/chef-client /etc/service/chef-client
145
+ service chef-client start
146
+
147
+ echo -e "`date` \n\n**** \n**** Cluster Chef client bootstrap complete\n****\n"
148
+ EOF
data/lib/chef/knife/bootstrap/ubuntu10.04-ironfan.erb CHANGED
@@ -26,14 +26,20 @@ export DEBIAN_FRONTEND=noninteractive
26
26
  date > /etc/box_build_time
27
27
 
28
28
  echo -e "`date` \n\n**** \n**** apt update:\n****\n"
29
- apt-get -y update
30
- apt-get -y upgrade
29
+ apt-get --force-yes -y update
30
+ apt-get --force-yes -y upgrade
31
31
 
32
32
  echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
33
- apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
34
- apt-get -y install runit-services
33
+ apt-get --force-yes -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
34
+ apt-get --force-yes -y install runit-services
35
35
  apt-get clean
36
36
 
37
+ # do not trust the ubuntu chef package
38
+ rm -f /etc/init.d/chef-client || true
39
+ apt-get remove -y --force-yes chef ruby1.8-dev libjson-ruby1.8 libmixlib-authentication-ruby1.8 ohai libmixlib-log-ruby1.8 libmime-types-ruby librestclient-ruby1.8 ruby1.8 ruby liberubis-ruby1.8 libsystemu-ruby1.8 libohai-ruby libuuidtools-ruby1.8 libhighline-ruby1.8 libabstract-ruby1.8 libmixlib-config-ruby1.8 rubygems1.8 libbunny-ruby1.8 libchef-ruby1.8 libmixlib-cli-ruby1.8 libyajl-ruby libmoneta-ruby1.8 libohai-ruby1.8 libextlib-ruby1.8 || true
40
+ # apt-get remove -y --force-yes --purge myproxy globus-core condor nis autofs myproxy-server globus-proxy-utils globus-repository-natty globus-gridftp-server-progs globus-gass-copy-progs globus-simple-ca globus-gsi-cert-utils-progs || true
41
+ sudo apt-get autoremove -y --force-yes || true
42
+
37
43
  if [ ! -f /usr/bin/chef-client ]; then
38
44
  echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
39
45
 
@@ -69,7 +75,7 @@ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
69
75
  gem install ohai --no-rdoc --no-ri
70
76
  gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
71
77
  # gems needed for the client.rb or so generically useful you want them at hand
72
- gem install --no-rdoc --no-ri extlib bundler json right_aws pry
78
+ gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
73
79
 
74
80
  else # no chef-client
75
81
  echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
data/lib/chef/knife/bootstrap/ubuntu11.10-ironfan.erb CHANGED
@@ -26,21 +26,27 @@ export DEBIAN_FRONTEND=noninteractive
26
26
  date > /etc/box_build_time
27
27
 
28
28
  # source for sun java if you want to install it later
29
- apt-get install -y python-software-properties
29
+ apt-get install --force-yes -y python-software-properties
30
30
  add-apt-repository -y ppa:ferramroberto/java
31
31
 
32
32
  echo -e "`date` \n\n**** \n**** apt update:\n****\n"
33
- apt-get -y update
34
- apt-get -y upgrade
33
+ apt-get --force-yes -y update
34
+ apt-get --force-yes -y upgrade
35
35
 
36
36
  echo -e "`date` \n\n**** \n**** Installing base packages:\n****\n"
37
- apt-get -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
37
+ apt-get --force-yes -y install build-essential make wget curl runit zlib1g-dev libssl-dev openssl libcurl4-openssl-dev libxml2-dev libxslt-dev libyaml-dev libreadline6 libreadline6-dev
38
38
  apt-get clean
39
39
 
40
+ # do not trust the ubuntu chef package
41
+ rm -f /etc/init.d/chef-client || true
42
+ apt-get remove -y --force-yes chef ruby1.8-dev libjson-ruby1.8 libmixlib-authentication-ruby1.8 ohai libmixlib-log-ruby1.8 libmime-types-ruby librestclient-ruby1.8 ruby1.8 ruby liberubis-ruby1.8 libsystemu-ruby1.8 libohai-ruby libuuidtools-ruby1.8 libhighline-ruby1.8 libabstract-ruby1.8 libmixlib-config-ruby1.8 rubygems1.8 libbunny-ruby1.8 libchef-ruby1.8 libmixlib-cli-ruby1.8 libyajl-ruby libmoneta-ruby1.8 libohai-ruby1.8 libextlib-ruby1.8 || true
43
+ # apt-get remove -y --force-yes --purge myproxy globus-core condor nis autofs myproxy-server globus-proxy-utils globus-repository-natty globus-gridftp-server-progs globus-gass-copy-progs globus-simple-ca globus-gsi-cert-utils-progs || true
44
+ sudo apt-get autoremove -y --force-yes || true
45
+
40
46
  if [ ! -f /usr/bin/chef-client ]; then
41
47
  echo -e "`date` \n\n**** \n**** Installing ruby version ${RUBY_VERSION}:\n****\n"
42
48
 
43
- apt-get install -y ruby1.9.1 ruby1.9.1-dev
49
+ apt-get install --force-yes -y ruby1.9.1 ruby1.9.1-dev
44
50
 
45
51
  if ruby -e "exit(%x{gem --version} < \"1.6.2\" ? 0 : -1 )" ; then
46
52
  echo -e "`date` \n\n**** \n**** Updating rubygems:\n****\n"
@@ -57,7 +63,7 @@ echo -e "`date` \n\n**** \n**** Installing chef:\n****\n"
57
63
  gem install ohai --no-rdoc --no-ri
58
64
  gem install chef --no-rdoc --no-ri <%= bootstrap_version_string %>
59
65
  # gems needed for the client.rb or so generically useful you want them at hand
60
- gem install --no-rdoc --no-ri extlib bundler json right_aws pry
66
+ gem install --no-rdoc --no-ri extlib bundler json right_aws pry fog
61
67
 
62
68
  else # no chef-client
63
69
  echo -e "`date` \n\n**** \n**** Chef is present -- skipping apt/ruby/chef installation\n****\n"
data/lib/chef/knife/cluster_bootstrap.rb CHANGED
@@ -59,9 +59,10 @@ class Chef
59
59
  end
60
60
 
61
61
  def perform_execution(target)
62
- target.each do |svr|
63
- run_bootstrap(svr, svr.fog_server.dns_name)
64
- end
62
+ # Execute across all servers in parallel
63
+ threads = target.servers.map{ |server| Thread.new(server) { |svr| run_bootstrap(svr, svr.public_hostname) } }
64
+ # Wait for the threads to finish and return the array of thread's exit value
65
+ threads.map{ |t| t.join.value }
65
66
  end
66
67
 
67
68
  def confirm_execution(target)