ironfan 4.3.4 → 4.4.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (66) hide show
  1. data/CHANGELOG.md +7 -0
  2. data/ELB.md +121 -0
  3. data/Gemfile +1 -0
  4. data/Rakefile +4 -0
  5. data/VERSION +1 -1
  6. data/ironfan.gemspec +48 -3
  7. data/lib/chef/knife/cluster_launch.rb +5 -0
  8. data/lib/chef/knife/cluster_proxy.rb +3 -3
  9. data/lib/chef/knife/cluster_sync.rb +4 -0
  10. data/lib/chef/knife/ironfan_knife_common.rb +17 -6
  11. data/lib/chef/knife/ironfan_script.rb +29 -11
  12. data/lib/ironfan.rb +2 -2
  13. data/lib/ironfan/broker/computer.rb +8 -3
  14. data/lib/ironfan/dsl/ec2.rb +133 -2
  15. data/lib/ironfan/headers.rb +4 -0
  16. data/lib/ironfan/provider.rb +48 -3
  17. data/lib/ironfan/provider/ec2.rb +23 -8
  18. data/lib/ironfan/provider/ec2/elastic_load_balancer.rb +239 -0
  19. data/lib/ironfan/provider/ec2/iam_server_certificate.rb +101 -0
  20. data/lib/ironfan/provider/ec2/machine.rb +8 -0
  21. data/lib/ironfan/provider/ec2/security_group.rb +3 -5
  22. data/lib/ironfan/requirements.rb +2 -0
  23. data/notes/Home.md +45 -0
  24. data/notes/INSTALL-cloud_setup.md +103 -0
  25. data/notes/INSTALL.md +134 -0
  26. data/notes/Ironfan-Roadmap.md +70 -0
  27. data/notes/advanced-superpowers.md +16 -0
  28. data/notes/aws_servers.jpg +0 -0
  29. data/notes/aws_user_key.png +0 -0
  30. data/notes/cookbook-versioning.md +11 -0
  31. data/notes/core_concepts.md +200 -0
  32. data/notes/declaring_volumes.md +3 -0
  33. data/notes/design_notes-aspect_oriented_devops.md +36 -0
  34. data/notes/design_notes-ci_testing.md +169 -0
  35. data/notes/design_notes-cookbook_event_ordering.md +249 -0
  36. data/notes/design_notes-meta_discovery.md +59 -0
  37. data/notes/ec2-pricing_and_capacity.md +69 -0
  38. data/notes/ec2-pricing_and_capacity.numbers +0 -0
  39. data/notes/homebase-layout.txt +102 -0
  40. data/notes/knife-cluster-commands.md +18 -0
  41. data/notes/named-cloud-objects.md +11 -0
  42. data/notes/opscode_org_key.png +0 -0
  43. data/notes/opscode_user_key.png +0 -0
  44. data/notes/philosophy.md +13 -0
  45. data/notes/rake_tasks.md +24 -0
  46. data/notes/renamed-recipes.txt +142 -0
  47. data/notes/silverware.md +85 -0
  48. data/notes/style_guide.md +300 -0
  49. data/notes/tips_and_troubleshooting.md +92 -0
  50. data/notes/version-3_2.md +273 -0
  51. data/notes/walkthrough-hadoop.md +168 -0
  52. data/notes/walkthrough-web.md +166 -0
  53. data/spec/fixtures/ec2/elb/snakeoil.crt +35 -0
  54. data/spec/fixtures/ec2/elb/snakeoil.key +51 -0
  55. data/spec/integration/minimal-chef-repo/chefignore +41 -0
  56. data/spec/integration/minimal-chef-repo/environments/_default.json +12 -0
  57. data/spec/integration/minimal-chef-repo/knife/credentials/knife-org.rb +19 -0
  58. data/spec/integration/minimal-chef-repo/knife/credentials/knife-user-ironfantester.rb +9 -0
  59. data/spec/integration/minimal-chef-repo/knife/knife.rb +66 -0
  60. data/spec/integration/minimal-chef-repo/roles/systemwide.rb +10 -0
  61. data/spec/integration/spec/elb_build_spec.rb +95 -0
  62. data/spec/integration/spec_helper.rb +16 -0
  63. data/spec/integration/spec_helper/launch_cluster.rb +55 -0
  64. data/spec/ironfan/ec2/elb_spec.rb +95 -0
  65. data/spec/ironfan/ec2/security_group_spec.rb +0 -6
  66. metadata +60 -3
@@ -0,0 +1,273 @@
1
+
2
+ # v3.2.0 (future): Revamped undercarriage, spec coverage, standalone usage
3
+
4
+ This is a Snow Leopard-style version change. No new features to speak of, but a much more solid and predictable foundation.
5
+
6
+ * **significantly cleaner DSL mixin**: uses the new, awesome `Gorillib::Builder`, giving it a much cleaner handling of fields and collections
7
+
8
+ * **attributes are late-resolved**: in previous versions, the way you 'resolved' a server was to collapse the entire attribute set of cluster/facet/server hard onto the server model, a consistent source of bugs. Resolution is now done with the `Gorillib::Record::Overlay` mechanism, which means that you can set an attribute on the cluster and read it from the facet; change it later an all lower layers see the update.
9
+
10
+ * **standalone usable**: can use ironfan-knife as a standalone library.
11
+
12
+ # v3.3.x (future): Coherent universe of Servers, Components, Aspects
13
+
14
+ * **spec coverage**:
15
+
16
+ * **coherent data model**:
17
+
18
+ ComputeLayer -- common attributes of Provider, Cluster, Facet, Server
19
+ - overlay_stack of Cloud attributes
20
+
21
+ Universe -- across organizations
22
+ Organization -- one or many providers
23
+ Provider --
24
+ - has_many :clusters
25
+ Cluster --
26
+ - has_many :providers
27
+ - overlays :main_provider
28
+ Facet --
29
+ - has_one :cluster
30
+ - overlays :cluster
31
+ Server
32
+ - has_one :facet
33
+ - overlays :cluster
34
+ - has_one chef_node
35
+ - has_one machine
36
+
37
+
38
+ System Role Cookbook
39
+ Component Cookbook+Recipes
40
+
41
+
42
+
43
+ * **improved discovery**:
44
+
45
+ * **config isolation**:
46
+
47
+
48
+ ### Nitpicks
49
+
50
+
51
+ * make bootstrap_distro and image_name follow from os_version
52
+
53
+ * minidash just publishes announcements
54
+ * silverware is always included; it subsumes volumes
55
+
56
+ * if you add a `data_dir_for :hadoop` to
57
+
58
+ * volumes should name their `mount_point` after themselves by default
59
+
60
+ ### Components
61
+
62
+ * components replace roles (they are auto-generated by the component, and tie strictly to it)
63
+ *
64
+
65
+ ### Clusters
66
+
67
+ If clusters are more repeatable they won't be so bothersomely multi-provider:
68
+
69
+ Ironfan.cluster :gibbon do
70
+ cloud(:ec2) do
71
+ backing 'ebs'
72
+ permanent false
73
+ end
74
+ stack :systemwide
75
+ stack :devstack
76
+ stack :monitoring
77
+ stack :log_handling
78
+
79
+ component :hadoop_devstack
80
+ component :hadoop_dedicated
81
+
82
+ discovers :zookeeper, :realm => :zk
83
+ discovers :hbase, :realm => :hbase
84
+
85
+ facet :master do
86
+ component :hadoop_namenode
87
+ component :hadoop_secondarynn
88
+ component :hadoop_jobtracker
89
+ end
90
+ facet :worker do
91
+ component :hadoop_datanode
92
+ component :hadoop_tasktracker
93
+ end
94
+
95
+ volume :hadoop_data do
96
+ data_dir_for :hadoop_datanode, :hadoop_namenode, :hadoop_secondarynn
97
+ device '/dev/sdj1'
98
+ size 100
99
+ keep true
100
+ end
101
+ end
102
+
103
+
104
+ Here are ideas about how to get there
105
+
106
+ # silverware is always included; it subsumes volumes
107
+
108
+ organization :infochimps do
109
+ cloud(:ec2) do
110
+ availability_zones ['us-east-1d']
111
+ backing :ebs
112
+ image_name 'ironfan-natty'
113
+ bootstrap_distro 'ironfan-natty'
114
+ chef_client_script 'client.rb'
115
+ permanent true
116
+ end
117
+
118
+ volume(:default) do
119
+ keep true
120
+ snapshot_name :blank_xfs
121
+ resizable true
122
+ create_at_launch true
123
+ end
124
+
125
+ stack :systemwide do
126
+ system(:chef_client) do
127
+ run_state :on_restart
128
+ end
129
+ component :set_hostname
130
+ component :minidash
131
+ component :org_base
132
+ component :org_users
133
+ component :org_final
134
+ end
135
+
136
+ stack :devstack do
137
+ component :ssh
138
+ component :nfs_client
139
+ component :package_set
140
+ end
141
+
142
+ stack :monitoring do
143
+ component :zabbix_agent
144
+ end
145
+
146
+ stack :log_handling do
147
+ component :log_handling
148
+ end
149
+ end
150
+
151
+ stack :hadoop do
152
+ end
153
+
154
+ stack :hadoop_devstack do
155
+ component :pig
156
+ component :jruby
157
+ component :rstats
158
+ end
159
+
160
+ stack :hadoop_dedicated do
161
+ component :tuning
162
+ end
163
+
164
+ system :hadoop do
165
+ stack :hadoop_devstack
166
+ stack :zookeeper_client
167
+ stack :hbase_client
168
+ end
169
+
170
+ Ironfan.cluster :gibbon do
171
+ cloud(:ec2) do
172
+ backing 'ebs'
173
+ permanent false
174
+ end
175
+
176
+ system :systemwide do
177
+ exclude_stack :monitoring
178
+ end
179
+
180
+ # how are its components configured? distributed among machines?
181
+ system :hadoop do
182
+
183
+ # all servers will
184
+ # * have the `hadoop` role
185
+ # * have run_state => false for components with a daemon aspect by default
186
+
187
+ facet :master do
188
+ # component :hadoop_namenode means
189
+ # * this facet has the `hadoop_namenode` role
190
+ # * it has the component's security_groups
191
+ # * it sets node[:hadoop][:namenode][:run_state] = true
192
+ # * it will mount the volumes that adhere to this component
193
+ component :hadoop_namenode
194
+ end
195
+
196
+ # something gains eg zookeeper client if it discovers a zookeeper in another realm
197
+ # zookeeper must explicitly admit it discovers zookeeper, but can do that in the component
198
+
199
+ # what volumes should it use on those machines?
200
+ # create the volumes, pair it to components
201
+ # if a component is on a server, it adds its volumes.
202
+ # you can also add them explicitly.
203
+
204
+ # volume tags are applied automagically from their adherance to components
205
+
206
+ volume :hadoop_data do # will be assigned to servers with components it lists
207
+ data_dir_for :hadoop_datanode, :hadoop_namenode, :hadoop_secondarynn
208
+ end
209
+
210
+ ### Providers
211
+
212
+ I want to be able to:
213
+
214
+ * on a compute layer, modify its behavior depending on provider:
215
+ - example:
216
+
217
+ facet(:bob) do
218
+ cloud do
219
+ security_group :bob
220
+ authorize :from => :bobs_friends, :to => :bob
221
+ end
222
+ cloud(:ec2, :flavor => 'm1.small')
223
+ cloud(:rackspace, :flavor => '2GB')
224
+ cloud(:vagrant, :ram_mb => 256 )
225
+ end
226
+
227
+ - Any world that understands security groups will endeavor to make a `bob` security group, and authorize the `bobs_friends` group to use it.
228
+ - On EC2 and rackspace, the `flavor` attribute is set explicitly
229
+ - On vagrant (which got no `flavor`), we instead specify how much ram to supply
230
+ - On any other provider the flavor and machine ram will follow defaults.
231
+
232
+ * see all machines and clusters within an organization
233
+
234
+
235
+ ### Organizations
236
+
237
+ * see the entire universe; this might get hairy, but not ridiculous
238
+ - each org describes its providers; only those are used
239
+ - you don't have to do much to add a provider, just say `provider(:ec2)`
240
+ - you can configure the provider like this:
241
+
242
+ organization(:infochimps_test, :doc => 'Infochimps test cloud') do
243
+ provider(:vagrant)
244
+ provider(:ec2) do
245
+ access_key '...'
246
+ secret_access_key '...'
247
+ end
248
+ provider(:hp_cloud) do
249
+ access_key '...'
250
+ secret_access_key '...'
251
+ end
252
+ end
253
+
254
+ organization(:demo, :doc => 'Live client demo cloud') do
255
+ provider(:vagrant)
256
+ provider(:ec2) do #... end
257
+ provider(:hp_cloud) do #... end
258
+ provider(:rackspace) do #... end
259
+ end
260
+
261
+ - clusters can be declared directly or imported from other organizations:
262
+
263
+ organization :infochimps_test do
264
+ # developers' sandboxes
265
+ cluster :dev_sandboxes
266
+ # all the example clusters, for development
267
+ organization(:examples).clusters.each do |cl|
268
+ add_cluster cl
269
+ end
270
+ end
271
+
272
+ - if just starting, should see clusters;
273
+ - per-org cluster dirs
@@ -0,0 +1,168 @@
1
+ FIXME: Repurpose general structure to demonstrate a Hadoop cluster.
2
+
3
+ ## Walkthrough: Hadoop Cluster
4
+
5
+ Here's a very simple cluster:
6
+
7
+ ```ruby
8
+ Ironfan.cluster 'hadoop_demo' do
9
+ cloud(:ec2) do
10
+ flavor 't1.micro'
11
+ end
12
+
13
+ role :base_role
14
+ role :chef_client
15
+ role :ssh
16
+
17
+ # The database server
18
+ facet :dbnode do
19
+ instances 1
20
+ role :mysql_server
21
+
22
+ cloud do
23
+ flavor 'm1.large'
24
+ backing 'ebs'
25
+ end
26
+ end
27
+
28
+ # A throwaway facet for development.
29
+ facet :webnode do
30
+ instances 2
31
+ role :nginx_server
32
+ role :awesome_webapp
33
+ end
34
+ end
35
+ ```
36
+
37
+ This code defines a cluster named hadoop_demo. A cluster is a group of servers united around a common purpose, in this case to serve a scalable web application.
38
+
39
+ The hadoop_demo cluster has two 'facets' -- dbnode and webnode. A facet is a subgroup of interchangeable servers that provide a logical set of systems: in this case, the systems that store the website's data and those that render it.
40
+
41
+ The dbnode facet has one server, which will be named `hadoop_demo-dbnode-0`; the webnode facet has two servers, `hadoop_demo-webnode-0` and `hadoop_demo-webnode-1`.
42
+
43
+ Each server inherits the appropriate behaviors from its facet and cluster. All the servers in this cluster have the `base_role`, `chef_client` and `ssh` roles. The dbnode machines additionally house a MySQL server, while the webnodes have an nginx reverse proxy for the custom `hadoop_demo_webapp`.
44
+
45
+ As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the server, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
46
+
47
+ ### Cluster-level tools
48
+
49
+ ```
50
+ $ knife cluster show hadoop_demo
51
+
52
+ +---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
53
+ | Name | Chef? | InstanceID | State | Public IP | Private IP | Created At | Flavor | Image | AZ | SSH Key |
54
+ +---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
55
+ | hadoop_demo-dbnode-0 | yes | i-43c60e20 | running | 107.22.6.104 | 10.88.112.201 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | hadoop_demo |
56
+ | hadoop_demo-webnode-0 | yes | i-1233aef1 | running | 102.99.3.123 | 10.88.112.123 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | hadoop_demo |
57
+ | hadoop_demo-webnode-1 | yes | i-0986423b | not running | | | | | | | |
58
+ +---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
59
+
60
+ ```
61
+
62
+ The commands available are:
63
+
64
+ * list -- lists known clusters
65
+ * show -- show the named servers
66
+ * launch -- launch server
67
+ * bootstrap
68
+ * sync
69
+ * ssh
70
+ * start/stop
71
+ * kill
72
+ * kick -- trigger a chef-client run on each named machine, tailing the logs until the run completes
73
+
74
+
75
+ ### Advanced clusters remain simple
76
+
77
+ Let's say that app is truly awesome, and the features and demand increases. This cluster adds an [ElasticSearch server](http://elasticsearch.org) for searching, a haproxy loadbalancer, and spreads the webnodes across two availability zones.
78
+
79
+ ```ruby
80
+ Ironfan.cluster 'hadoop_demo' do
81
+ cloud(:ec2) do
82
+ image_name "maverick"
83
+ flavor "t1.micro"
84
+ availability_zones ['us-east-1a']
85
+ end
86
+
87
+ # The database server
88
+ facet :dbnode do
89
+ instances 1
90
+ role :mysql_server
91
+ cloud do
92
+ flavor 'm1.large'
93
+ backing 'ebs'
94
+ end
95
+
96
+ volume(:data) do
97
+ size 20
98
+ keep true
99
+ device '/dev/sdi'
100
+ mount_point '/data'
101
+ snapshot_id 'snap-a10234f'
102
+ attachable :ebs
103
+ end
104
+ end
105
+
106
+ facet :webnode do
107
+ instances 6
108
+ cloud.availability_zones ['us-east-1a', 'us-east-1b']
109
+
110
+ role :nginx_server
111
+ role :awesome_webapp
112
+ role :elasticsearch_client
113
+
114
+ volume(:server_logs) do
115
+ size 5
116
+ keep true
117
+ device '/dev/sdi'
118
+ mount_point '/server_logs'
119
+ snapshot_id 'snap-d9c1edb1'
120
+ end
121
+ end
122
+
123
+ facet :esnode do
124
+ instances 1
125
+ role "elasticsearch_data_esnode"
126
+ role "elasticsearch_http_esnode"
127
+ cloud.flavor "m1.large"
128
+ end
129
+
130
+ facet :loadbalancer do
131
+ instances 1
132
+ role "haproxy"
133
+ cloud.flavor "m1.xlarge"
134
+ elastic_ip "128.69.69.23"
135
+ end
136
+
137
+ cluster_role.override_attributes({
138
+ :elasticsearch => {
139
+ :version => '0.17.8',
140
+ },
141
+ })
142
+ end
143
+ ```
144
+
145
+ The facets are described and scale independently. If you'd like to add more webnodes, just increase the instance count. If a machine misbehaves, just terminate it. Running `knife cluster launch hadoop_demo webnode` will note which machines are missing, and launch and configure them appropriately.
146
+
147
+ Ironfan speaks naturally to both Chef and your cloud provider. The esnode's `cluster_role.override_attributes` statement will be synchronized to the chef server, pinning the elasticsearch version across the server and clients. Your chef roles should focus on specific subsystems; the cluster file lets you see the architecture as a whole.
148
+
149
+ With these simple settings, if you have already [set up chef's knife to launch cloud servers](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife), typing `knife cluster launch hadoop_demo --bootstrap` will (using Amazon EC2 as an example):
150
+
151
+ * Synchronize to the chef server:
152
+ - create chef roles on the server for the cluster and each facet.
153
+ - apply role directives (eg the homebase's `default_attributes` declaration).
154
+ - create a node for each machine
155
+ - apply the runlist to each node
156
+ * Set up security isolation:
157
+ - uses a keypair (login ssh key) isolated to that cluster
158
+ - Recognizes the `ssh` role, and add a security group `ssh` that by default opens port 22.
159
+ - Recognize the `nfs_server` role, and adds security groups `nfs_server` and `nfs_client`
160
+ - Authorizes the `nfs_server` to accept connections from all `nfs_client`s. Machines in other clusters that you mark as `nfs_client`s can connect to the NFS server, but are not automatically granted any other access to the machines in this cluster. Ironfan's opinionated behavior is about more than saving you effort -- tying this behavior to the chef role means you can't screw it up.
161
+ * Launches the machines in parallel:
162
+ - using the image name and the availability zone, it determines the appropriate region, image ID, and other implied behavior.
163
+ - passes a JSON-encoded user_data hash specifying the machine's chef `node_name` and client key. An appropriately-configured machine image will need no further bootstrapping -- it will connect to the chef server with the appropriate identity and proceed completely unattended.
164
+ * Syncronizes to the cloud provider:
165
+ - Applies EC2 tags to the machine, making your console intelligible: ![AWS Console screenshot](https://github.com/infochimps-labs/ironfan/raw/version_3/notes/aws_console_screenshot.jpg)
166
+ - Connects external (EBS) volumes, if any, to the correct mount point -- it uses (and applies) tags to the volumes, so they know which machine to adhere to. If you've manually added volumes, just make sure they're defined correctly in your cluster file and run `knife cluster sync {cluster_name}`; it will paint them with the correct tags.
167
+ - Associates an elastic IP, if any, to the machine
168
+ * Bootstraps the machine using knife bootstrap
@@ -0,0 +1,166 @@
1
+ ## Walkthrough: Web Cluster
2
+
3
+ Here's a very simple cluster:
4
+
5
+ ```ruby
6
+ Ironfan.cluster 'web_demo' do
7
+ cloud(:ec2) do
8
+ flavor 't1.micro'
9
+ end
10
+
11
+ role :base_role
12
+ role :chef_client
13
+ role :ssh
14
+
15
+ # The database server
16
+ facet :dbnode do
17
+ instances 1
18
+ role :mysql_server
19
+
20
+ cloud do
21
+ flavor 'm1.large'
22
+ backing 'ebs'
23
+ end
24
+ end
25
+
26
+ # A throwaway facet for development.
27
+ facet :webnode do
28
+ instances 2
29
+ role :nginx_server
30
+ role :awesome_webapp
31
+ end
32
+ end
33
+ ```
34
+
35
+ This code defines a cluster named web_demo. A cluster is a group of servers united around a common purpose, in this case to serve a scalable web application.
36
+
37
+ The web_demo cluster has two 'facets' -- dbnode and webnode. A facet is a subgroup of interchangeable servers that provide a logical set of systems: in this case, the systems that store the website's data and those that render it.
38
+
39
+ The dbnode facet has one server, which will be named `web_demo-dbnode-0`; the webnode facet has two servers, `web_demo-webnode-0` and `web_demo-webnode-1`.
40
+
41
+ Each server inherits the appropriate behaviors from its facet and cluster. All the servers in this cluster have the `base_role`, `chef_client` and `ssh` roles. The dbnode machines additionally house a MySQL server, while the webnodes have an nginx reverse proxy for the custom `web_demo_webapp`.
42
+
43
+ As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the server, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
44
+
45
+ ### Cluster-level tools
46
+
47
+ ```
48
+ $ knife cluster show web_demo
49
+
50
+ +---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
51
+ | Name | Chef? | InstanceID | State | Public IP | Private IP | Created At | Flavor | Image | AZ | SSH Key |
52
+ +---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
53
+ | web_demo-dbnode-0 | yes | i-43c60e20 | running | 107.22.6.104 | 10.88.112.201 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | web_demo |
54
+ | web_demo-webnode-0 | yes | i-1233aef1 | running | 102.99.3.123 | 10.88.112.123 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | web_demo |
55
+ | web_demo-webnode-1 | yes | i-0986423b | not running | | | | | | | |
56
+ +---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
57
+
58
+ ```
59
+
60
+ The commands available are:
61
+
62
+ * list -- lists known clusters
63
+ * show -- show the named servers
64
+ * launch -- launch server
65
+ * bootstrap
66
+ * sync
67
+ * ssh
68
+ * start/stop
69
+ * kill
70
+ * kick -- trigger a chef-client run on each named machine, tailing the logs until the run completes
71
+
72
+
73
+ ### Advanced clusters remain simple
74
+
75
+ Let's say that app is truly awesome, and the features and demand increases. This cluster adds an [ElasticSearch server](http://elasticsearch.org) for searching, a haproxy loadbalancer, and spreads the webnodes across two availability zones.
76
+
77
+ ```ruby
78
+ Ironfan.cluster 'web_demo' do
79
+ cloud(:ec2) do
80
+ image_name "maverick"
81
+ flavor "t1.micro"
82
+ availability_zones ['us-east-1a']
83
+ end
84
+
85
+ # The database server
86
+ facet :dbnode do
87
+ instances 1
88
+ role :mysql_server
89
+ cloud do
90
+ flavor 'm1.large'
91
+ backing 'ebs'
92
+ end
93
+
94
+ volume(:data) do
95
+ size 20
96
+ keep true
97
+ device '/dev/sdi'
98
+ mount_point '/data'
99
+ snapshot_id 'snap-a10234f'
100
+ attachable :ebs
101
+ end
102
+ end
103
+
104
+ facet :webnode do
105
+ instances 6
106
+ cloud.availability_zones ['us-east-1a', 'us-east-1b']
107
+
108
+ role :nginx_server
109
+ role :awesome_webapp
110
+ role :elasticsearch_client
111
+
112
+ volume(:server_logs) do
113
+ size 5
114
+ keep true
115
+ device '/dev/sdi'
116
+ mount_point '/server_logs'
117
+ snapshot_id 'snap-d9c1edb1'
118
+ end
119
+ end
120
+
121
+ facet :esnode do
122
+ instances 1
123
+ role "elasticsearch_data_esnode"
124
+ role "elasticsearch_http_esnode"
125
+ cloud.flavor "m1.large"
126
+ end
127
+
128
+ facet :loadbalancer do
129
+ instances 1
130
+ role "haproxy"
131
+ cloud.flavor "m1.xlarge"
132
+ elastic_ip "128.69.69.23"
133
+ end
134
+
135
+ cluster_role.override_attributes({
136
+ :elasticsearch => {
137
+ :version => '0.17.8',
138
+ },
139
+ })
140
+ end
141
+ ```
142
+
143
+ The facets are described and scale independently. If you'd like to add more webnodes, just increase the instance count. If a machine misbehaves, just terminate it. Running `knife cluster launch web_demo webnode` will note which machines are missing, and launch and configure them appropriately.
144
+
145
+ Ironfan speaks naturally to both Chef and your cloud provider. The esnode's `cluster_role.override_attributes` statement will be synchronized to the chef server, pinning the elasticsearch version across the server and clients. Your chef roles should focus on specific subsystems; the cluster file lets you see the architecture as a whole.
146
+
147
+ With these simple settings, if you have already [set up chef's knife to launch cloud servers](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife), typing `knife cluster launch web_demo --bootstrap` will (using Amazon EC2 as an example):
148
+
149
+ * Synchronize to the chef server:
150
+ - create chef roles on the server for the cluster and each facet.
151
+ - apply role directives (eg the homebase's `default_attributes` declaration).
152
+ - create a node for each machine
153
+ - apply the runlist to each node
154
+ * Set up security isolation:
155
+ - uses a keypair (login ssh key) isolated to that cluster
156
+ - Recognizes the `ssh` role, and add a security group `ssh` that by default opens port 22.
157
+ - Recognize the `nfs_server` role, and adds security groups `nfs_server` and `nfs_client`
158
+ - Authorizes the `nfs_server` to accept connections from all `nfs_client`s. Machines in other clusters that you mark as `nfs_client`s can connect to the NFS server, but are not automatically granted any other access to the machines in this cluster. Ironfan's opinionated behavior is about more than saving you effort -- tying this behavior to the chef role means you can't screw it up.
159
+ * Launches the machines in parallel:
160
+ - using the image name and the availability zone, it determines the appropriate region, image ID, and other implied behavior.
161
+ - passes a JSON-encoded user_data hash specifying the machine's chef `node_name` and client key. An appropriately-configured machine image will need no further bootstrapping -- it will connect to the chef server with the appropriate identity and proceed completely unattended.
162
+ * Syncronizes to the cloud provider:
163
+ - Applies EC2 tags to the machine, making your console intelligible: ![AWS Console screenshot](https://github.com/infochimps-labs/ironfan/wiki/aws_servers.jpg)
164
+ - Connects external (EBS) volumes, if any, to the correct mount point -- it uses (and applies) tags to the volumes, so they know which machine to adhere to. If you've manually added volumes, just make sure they're defined correctly in your cluster file and run `knife cluster sync {cluster_name}`; it will paint them with the correct tags.
165
+ - Associates an elastic IP, if any, to the machine
166
+ * Bootstraps the machine using knife bootstrap