ironfan 3.1.3 → 3.1.4

data/notes/style_guide.md

@@ -1,41 +1,82 @@
 # Ironfan + Chef Style Guide
 
-**NOTE**: NEEDS UPDATE. will be current by time of launch but the below may differ from what you see in practice.
+------------------------------------------------------------------------
 
-## Cookbooks
+### System+Component define Names
 
-Ordinary cookbooks describe a single system, consisting of one or more components. For example, the `redis` cookbook has a `server` component (with a daemon and moving parts), and a `client` component (which is static).
+Name things uniformly for their system and component. For the ganglia master,
+
+* attributes:  `node[:ganglia][:master]`
+* recipe:      `ganglia::master`
+* role:        `ganglia_master` 
+* directories: `ganglia/master` (if specific to component), `ganglia` (if not).
+  - for example: `/var/log/ganglia/master`
+
+### Component names
+
+* `agent.rb`
+* `worker.rb`
+* `datanode.rb`
+* `webnode.rb`
 
-You should crisply separate cookbook-wide concerns from component concerns. The server's attributes live in `node[:redis][:server]`, it is installed by the `redis::server` cookbook, and so forth.
-  
-You should also separate system configuration from multi-system integration. Cookbooks should provide hooks that are neighborly but not exhibitionist, and otherwise mind their own business. The `hadoop_cluster` cookbook describes hadoop, the `pig` cookbook pig, and the `zookeeper` cookbook zookeeper. The job of tying those components together (copying zookeeper jars into the pig home dir, or the port+addr of hadoop daemons) should be isolated.
 
 ### Recipes
 
-* Naming:
-  - `foo/recipes/default.rb`    -- information shared by anyone using foo, including support packages, directories
-  - `foo/recipes/client.rb`     -- configure me as a foo client 
-  - `foo/recipes/server.rb`     -- configure me as a foo server
-  - `foo/recipes/ec2_conf`      -- cloud-specific settings
+Recipes partition these things:
+
+* shared functionality between components
+* proper event order
+* optional or platform-specific functionality
+
+* Within the foo cookbook, name your recipes like this:
+  - `default.rb`      -- information shared by anyone using foo, including support packages, users and directories.
+  - `user.rb`         -- define daemon users. Called 'user' even if there is more than one. It's OK to move this into the default cookbook.
+  - `install_from_X.rb` -- install packages (`install_from_package`), versioned tarballs (`install_from_release`). It's OK to move this into `default.rb`.
+  - `deploy.rb`       -- use this when doing sha-versioned deploys.
+  - `plugins.rb`      -- install additional plugins or support code. If you have separate plugins, name them `git_plugin`, `rspec_plugin`, etc. 
+  - `server.rb`       -- define the foo server process. Similarly, `agent`, `worker`, etc -- see component naming above.
+  - `client.rb`       -- install libraries to *use* the foo service.
+  - `config_files.rb` -- discover other components, write final configuration to disk
+  - `finalize.rb`     -- final cleanup
+
+* Do not repeat the cookbook name in a recipe title: `ganglia::master`, not `ganglia::ganglia_master`.
+* Use only `[a-z0-9_]` for cookbook and component names. Do not use capital letters or hyphens.
+* Keep names short and descriptive (preferably 15 characters or less, or it jacks with the Chef webui).
+
 * Always include a `default.rb` recipe, even if it is blank. 
-* *DO NOT* install daemons via the default cookbook, even if that's currently the only thing it does. Remember, a node that is a client -- or refers to any current or future component of the system -- will include the default recipe.
-* Do not repeat the cookbook name in a recipe title: `hbase:master`, not `hbase:hbase_master`; `zookeeper:server`, not `zookeeper:zookeeper_server`.
-* Use only `[a-z0-9_]` for cookbook and component names. Do not use capital letters or dashes. Keep names to fewer than 15 characters.
+* *DO NOT* use the default cookbook to install daemons or do anything interesting at all, even if that's currently the only thing the recipe does. I want to be able to refer to the attributes in the apache cookbook without launching the apache service. Think of it like a C header file.
+
+A `client` is also passive -- it lets me *use* the system without requiring that I run it. This means the client recipe should *never* launch a process (chef_client` and `nfs_client` components are allowed exceptions). 
 
 ### Cookbook Dependencies
 
 * Dependencies should be announced in metadata.rb, of course.
-* *DO* remember to explicitly `include_recipe` for system resources -- `runit`, `java`, `provides_service`, `thrift` and `apt`.
-* *DO NOT* use `include_recipe` unless putting it in the role would be utterly un-interesting. You *want* the run to break unless it's explicitly included the role. 
-  - *yes*: `java`, `ruby`, `provides_service`, etc.
-  - *no*:  `zookeeper:client`, `nfs:server`, or anything that will start a daemon
+* Explicitly `include_recipe` for system resources -- `runit`, `java`, `silverware`, `thrift` and `apt`.
+  - never 
+* *DO NOT* use `include_recipe` unless putting it in the role would be utterly un-interesting. You *want* the run to break unless it's explicitly included in the role. 
+  - *yes*: `java`, `ruby`, `announces`, etc.
+  - *no*:  `zookeeper::client`, `nfs::server`, or anything that will start a daemon
   Remember: ordinary cookbooks describe systems, roles and integration cookbooks coordinate them.
 * `include_recipe` statements should only appear in recipes that are entry points. Recipes that are not meant to be called directly should assume their dependencies have been met.
 * If a recipe is meant to be the primary entrypoint, it *should* include default, and it should do so explicitly: `include_recipe 'foo::default'` (not just 'foo'). 
 
+Crisply separate cookbook-wide concerns from component concerns. 
+  
+Separate system configuration from multi-system integration. Cookbooks should provide hooks that are neighborly but not exhibitionist, and otherwise mind their own business. 
+
 ### Templates
 
-* *DO NOT* use node[:foo] in your templates except in rare circumstances. Instead, say `variables :foo => node[:foo]`; this lets folks use that cookbook from elsewhere.
+*DO NOT* refer to attributes directly on the node (`node[:foo]`). This prevents people from using those templates outside the cookbook. Instead:
+
+```ruby
+    # in recipe
+    template 'fooconf.yml' do 
+      variables :foo => node[:foo]
+    end
+    
+    # in template
+    @node[:log_dir]
+```    
 
 ### Attributes
  
@@ -43,42 +84,44 @@ You should also separate system configuration from multi-system integration. Coo
 * Attributes shared by all components sit at cookbook level, and are always named for the cookbook: `node[:hadoop][:log_dir]` (since it is shared by all its components).
 * Component-specific attributes sit at component level (`node[:cookbook_name][:component_name]`): eg `node[:hadoop][:namenode][:service_state]`. Do not use a prefix (NO: `node[:hadoop][:namenode_handler_count]`)
 
+* Refer to node attributes by symbol, never by method:
+  - `node[:ganglia][:log_dir]`, not `node.ganglia.log_dir` or `node['ganglia']['log_dir']
+
 #### Attribute Files
 
-* The main attribute file should be named `attributes/default.rb`.
+* The main attribute file should be named `attributes/default.rb`. Do not name the file after the cookbook, or anything else.
 * If there are a sizeable number of tunable attributes (hadoop, cassandra), place them in `attributes/tuneables.rb`.
-* ?? Place integration attribute *hooks* in `attributes/integration.rb` ??
 
-* Be generic when you're *simple and alone*, descriptive when you're not. 
-  - If a component has only one log file, call it 'log_file': `node[:foo][:server][:log_file]` and in general do not use a prefix.
-  - If a component has more than one log_file, *always* use a prefix: `node[:foo][:server][:dashboard_log_file]` and `node[:foo][:server][:gc_log_file]`.
+## Name Attributes for their aspects
 
-* If you don't have exactly the semantics and datatype of the convention, don't use the convention.  That is, don't use `:port` and give it a comma-separated string, or `:addr` and give it an email address.
-* (*this advice will change as we figure out integration rules*: use `foo_client` when you are a client of a service: so [:rails][:mysql_client][:host] to specify the hostname of your mysql server.)
- 
-## Attribute Names
+Attributes should be named for their aspect: `port`, `log`, etc. Use generic names if there is only one attribute for an aspect, prefixed names if there are many:
+  - For a component that only opens one port: `node[:foo][:server][:port]`
+  - More than one port, use a prefix: `node[:foo][:server][:dash_port]` and `node[:foo][:server][:rpc_port]`.
 
-### Universal Aspects
+Sometimes the conventions below are inappropriate. All we ask is in those cases that you *not* use the special magic name. For example, don't use `:port` and give it a comma-separated string; name it something else, like `:port_list`.
+
+Here are specific conventions:
 
 ### File and Dir Aspects
 
-A *file* is the full directory and basename for a file. A *dir* is a directory whose contents correspond to a single concern. A *root* is a prefix not intended to be used directly -- it will be decorated with suffixes to form dirs and files. A *basename* is only the leaf part of a file reference. Don't use the terms 'path' or 'filename'.
+A *file* is the full directory and basename for a file. A *dir* is a directory whose contents correspond to a single concern. A *prefix* not intended to be used directly -- it will be decorated with suffixes to form dirs and files. A *basename* is only the leaf part of a file reference. Don't use the terms 'path' or 'filename'.
 
-Ignore the temptation to make a one-true-home-for-my-system, or to fight the package maintainer's choices. 
+Ignore the temptation to make a one-true-home-for-my-system, or to fight the package maintainer's choices. (FIXME: Rewrite to encourage OS-correct naming schemas.)
+- a sandbox holding dir, pid, log, ...
 
 #### Application
 
+* **prefix**: A container with directories bin, lib, share, src, to use according to convention
+  - default: `/usr/local`.
 * **home_dir**: Logical location for the cookbook's system code.
-  - default: typically, leave it up to the package maintainer. Otherwise, `:prefix_root/share/:cookbook` should be a symlink to the `install_dir` (see below).
+  - default: typically, leave it up to the package maintainer. Otherwise, `:prefix/share/:cookbook` should be a symlink to the `install_dir` (see below).
   - instead of:         `xx_home` / `dir` alone / `install_dir`
-* **prefix_root**: A container with directories bin, lib, share, src, to use according to convention
-  - default: `/usr/local`.
 * **install_dir**: The cookbook's system code, in case the home dir is a pointer to potential alternates.
-  - default: `:prefix_root/share/:cookbook-:version` ( you don't need the directory after the cookbook runs, use `:prefix_root/share/:cookbook-:version` instead, eg `/usr/local/src/tokyo_tyrant-xx.xx`)
+  - default: `:prefix/share/:cookbook-:version` ( you don't need the directory after the cookbook runs, use `:prefix/share/:cookbook-:version` instead, eg `/usr/local/src/tokyo_tyrant-xx.xx`)
   - Make `home_dir` a symlink to this directory (eg home_dir `/usr/local/share/elasticsearch` links to install_dir `/usr/local/share/elasticsearch-0.17.8`).
 * **src_dir**: holds the compressed tarball, its expanded contents, and the compiled files when installing from source. Use this when you will run `make install` or equivalent and use the files elsewhere.
-  - default:            `:prefix_root/src/:system_name-:version`, eg `/usr/local/src/pig-0.9.tar.gz`
-  - do not:             expand the tarball to `:prefix_root/src/(whatever)` if it will actually be used from there; instead, use the `install_dir` convention described above. (As a guideline, I should be able to blow away `/usr/local/src` and everything still works).
+  - default:            `:prefix/src/:system_name-:version`, eg `/usr/local/src/pig-0.9.tar.gz`
+  - do not:             expand the tarball to `:prefix/src/(whatever)` if it will actually be used from there; instead, use the `install_dir` convention described above. (As a guideline, I should be able to blow away `/usr/local/src` and everything still works).
 * **deploy_dir**: deployed code that follows the capistrano convention. See more about deploy variables below.
   - the `:deploy_dir/shared` directory holds common files
   - releases are checked out to `:deploy_dir/releases/{sha}`
@@ -138,7 +181,7 @@ Ignore the temptation to make a one-true-home-for-my-system, or to fight the pac
 * **release_url**:      URL for the release.
   - instead of:         install_url, package_url, being careless about partial vs whole URLs
 * **release_file**:     Where to put the release.
-  - default:            `:prefix_root/src/system_name-version.ext`, eg `/usr/local/src/elasticsearch-0.17.8.tar.bz2`. 
+  - default:            `:prefix/src/system_name-version.ext`, eg `/usr/local/src/elasticsearch-0.17.8.tar.bz2`. 
   - do not use `/tmp` -- let me decide when to blow it away (and make it easy to be idempotent).
   - do not use a non-versioned URL or file name.
 * **release_file_sha** or **release_file_md5** fingerprint
@@ -147,8 +190,7 @@ Ignore the temptation to make a one-true-home-for-my-system, or to fight the pac
 
 * **plugins**:          array of system-specific plugins
 
-use `deploy_{}` for anything that would be true whatever SCM you're using; use
-`git_{}` (and so forth) where specific to that repo.
+use `deploy_{}` for anything that would be true whatever SCM you're using; use `git_{}` (and so forth) where specific to that repo.
 
 * **deploy_env**        production / staging / etc
 * **deploy_strategy**   
@@ -160,7 +202,7 @@ use `deploy_{}` for anything that would be true whatever SCM you're using; use
 * **git_revision**:  SHA or branch
   - instead of:         `deploy_revision`
 
-* **apt/{repo_name}**   Options for adding a cookbook's apt repo.
+* **apt/(repo_name)**   Options for adding a cookbook's apt repo.
   - Note that this is filed under *apt*, not the cookbook.
   - Use the best name for the repo, which is not necessarily the cookbook's name: eg `apt/cloudera/{...}`, which is shared by hadoop, flume, pig, and so on.
   - `apt/{repo_name}/url` -- eg `http://archive.cloudera.com/debian`
@@ -190,7 +232,7 @@ use `deploy_{}` for anything that would be true whatever SCM you're using; use
 
 * **XX_heap_max**, **xx_heap_min**, **java_heap_eden**
 * **java_home** 
-* AVOID **java_opts** if possible: assemble it in your recipe from intelligible attribute names.
+* AVOID batch declaration of options (e.g. **java_opts**) if possible: assemble it in your recipe from intelligible attribute names.
 
 ### Nitpicks
 
@@ -214,19 +256,12 @@ If your app does any of the following,
 * **exports**     -- jars or libs that other programs may wish to incorporate
 * **consumes**    -- placed there by any call to `discover`.
 
-### Dummy aspects
-
-Integration cookbooks that announce as
-
-* Elastic Load Balancers
-
-
 ## Clusters
 
 * Describe physical configuration:
   - machine size, number of instances per facet, etc
   - external assets (elastic IP, ebs volumes)
-* Describe high-level assembly of systems via roles: `hadoop_namenode`, `nfs_client`, `flume_agent`, etc.
+* Describe high-level assembly of systems via roles: `hadoop_namenode`, `nfs_client`, `ganglia_agent`, etc.
 * Describe important modifications, such as `ironfan::system_internals`, mounts ebs volumes, etc
 * Describe override attributes:
   - `heap size`, rvm versions, etc.
@@ -247,5 +282,19 @@ roles shouldn't assemble systems. The contents of the infochimps_chef/roles/plat
 
 * Deprecated: 
   - Cluster and facet roles (`roles/gibbon_cluster.rb`, `roles/gibbon_namenode.rb`, etc) go away
-  - roles should be service-oriented: `hadoop_master` considered harmful, you should explicitly enumerate the services
+  - Roles should be service-oriented: `hadoop_master` considered harmful, you should explicitly enumerate the services
+
+
+### Facets should be (nearly) identical
+
+Within a facet, keep your servers almost entirely identical. For example, servers in a MySQL facet would their index to set shard order and to claim the right attached volumes. However, it would be a mistake to have one server within a facet be a master process and the rest be worker processes -- just define different facets for each. 
+
+### Pedantic Distinctions:
+
+Separate the following terms:
+
+* A *machine* is a concrete thing that runs your code -- it might be a VM or raw metal, but it has CPUs and fans and a finite lifetime. It has a unique name tied to its physical presence -- something like 'i-123abcd' or 'rack 4 server 7'.
+* A *chef node* is the code object that, together with the chef-client process, configures a machine. In ironfan, the chef node is strictly slave to the server description and the measured attributes of the machine.
+* A *server description* gives the high-level specification the machine should acheive. This includes the roles, recipes and attributes given to the chef node; the physical characteristics of the machine ('8 cores, 7GB ram, AWS cloud'); and its relation to the rest of the system (george cluster, webnode facet, index 3).
 
+In particular, we try to be careful to always call a Chef node a 'chef node' (never just 'node'). Try processing graph nodes in a flume node feeding a node.js decorator on a cloud node define by a chef node. No(de) way.

data/notes/tips_and_troubleshooting.md

@@ -1,5 +1,12 @@
 ## Tips and Notes
 
+### Gems
+
+   knife cluster ssh bonobo-worker-2 'sudo gem update --system'
+   knife cluster ssh bonobo-worker-2 'sudo true ; for foo in /usr/lib/ruby/gems/1.9.2-p290/specifications/*  ; do sudo sed -i.bak "s!000000000Z!!"          $foo ; done'
+   knife cluster ssh bonobo-worker-2 'sudo true ; for foo in /usr/lib/ruby/site_ruby/*/rubygems/deprecate.rb ; do sudo sed -i.bak "s!@skip ||= false!true!" $foo ; done'
+
+
 ### EC2 Notes Instance attributes: `disable_api_termination` and `delete_on_termination`
 
 To set `delete_on_termination` to 'true' after the fact, run the following (modify the instance and volume to suit):
@@ -81,3 +88,5 @@ Your service is probably installed but removed from runit's purview; check the `
 * directory `/etc/sv/foo`, containing file `run` and dirs `log` and `supervise`
 * `/etc/init.d/foo`  is symlinked to `/usr/bin/sv`
 * `/etc/servics/foo` is symlinked tp `/etc/sv/foo`
+
+

data/notes/walkthrough-hadoop.md

@@ -0,0 +1,168 @@
+FIXME: Repurpose general structure to demonstrate a Hadoop cluster.
+
+## Walkthrough: Hadoop Cluster
+
+Here's a very simple cluster:
+
+```ruby
+Ironfan.cluster 'hadoop_demo' do
+  cloud(:ec2) do
+    flavor              't1.micro'
+  end
+
+  role                  :base_role
+  role                  :chef_client
+  role                  :ssh
+
+  # The database server
+  facet :dbnode do
+    instances           1
+    role                :mysql_server
+
+    cloud do
+      flavor           'm1.large'
+      backing          'ebs'
+    end
+  end
+
+  # A throwaway facet for development.
+  facet :webnode do
+    instances           2
+    role                :nginx_server
+    role                :awesome_webapp
+  end
+end
+```
+
+This code defines a cluster named hadoop_demo. A cluster is a group of servers united around a common purpose, in this case to serve a scalable web application.
+
+The hadoop_demo cluster has two 'facets' -- dbnode and webnode. A facet is a subgroup of interchangeable servers that provide a logical set of systems: in this case, the systems that store the website's data and those that render it.
+
+The dbnode facet has one server, which will be named `hadoop_demo-dbnode-0`; the webnode facet has two servers, `hadoop_demo-webnode-0` and `hadoop_demo-webnode-1`.
+
+Each server inherits the appropriate behaviors from its facet and cluster. All the servers in this cluster have the `base_role`, `chef_client` and `ssh` roles. The dbnode machines additionally house a MySQL server, while the webnodes have an nginx reverse proxy for the custom `hadoop_demo_webapp`.
+
+As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the server, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
+
+### Cluster-level tools
+
+```
+$ knife cluster show hadoop_demo
+
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+| Name                | Chef? | InstanceID | State       | Public IP    | Private IP    | Created At      | Flavor   | Image        | AZ         | SSH Key    |
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+| hadoop_demo-dbnode-0   | yes   | i-43c60e20 | running     | 107.22.6.104 | 10.88.112.201 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | hadoop_demo   |
+| hadoop_demo-webnode-0  | yes   | i-1233aef1 | running     | 102.99.3.123 | 10.88.112.123 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | hadoop_demo   |
+| hadoop_demo-webnode-1  | yes   | i-0986423b | not running |              |               |                 |          |              |            |            |
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+
+```
+
+The commands available are:
+
+* list -- lists known clusters
+* show -- show the named servers
+* launch -- launch server
+* bootstrap  
+* sync       
+* ssh        
+* start/stop       
+* kill       
+* kick -- trigger a chef-client run on each named machine, tailing the logs until the run completes
+
+
+### Advanced clusters remain simple
+
+Let's say that app is truly awesome, and the features and demand increases. This cluster adds an [ElasticSearch server](http://elasticsearch.org) for searching, a haproxy loadbalancer, and spreads the webnodes across two availability zones.
+
+```ruby
+Ironfan.cluster 'hadoop_demo' do
+  cloud(:ec2) do
+    image_name          "maverick"
+    flavor              "t1.micro"
+    availability_zones  ['us-east-1a']
+  end
+
+  # The database server
+  facet :dbnode do
+    instances           1
+    role                :mysql_server
+    cloud do
+      flavor           'm1.large'
+      backing          'ebs'
+    end
+
+    volume(:data) do
+      size              20
+      keep              true                        
+      device            '/dev/sdi'                  
+      mount_point       '/data'              
+      snapshot_id       'snap-a10234f'             
+      attachable        :ebs
+    end
+  end
+
+  facet :webnode do
+    instances           6
+    cloud.availability_zones  ['us-east-1a', 'us-east-1b']
+
+    role                :nginx_server
+    role                :awesome_webapp
+    role                :elasticsearch_client
+
+    volume(:server_logs) do
+      size              5                           
+      keep              true                        
+      device            '/dev/sdi'                  
+      mount_point       '/server_logs'              
+      snapshot_id       'snap-d9c1edb1'             
+    end
+  end
+
+  facet :esnode do
+    instances           1
+    role                "elasticsearch_data_esnode"
+    role                "elasticsearch_http_esnode"
+    cloud.flavor        "m1.large"
+  end
+
+  facet :loadbalancer do
+    instances           1
+    role                "haproxy"
+    cloud.flavor        "m1.xlarge"
+    elastic_ip          "128.69.69.23"
+  end
+  
+  cluster_role.override_attributes({
+    :elasticsearch => {
+      :version => '0.17.8',
+    },
+  })
+end
+```
+
+The facets are described and scale independently. If you'd like to add more webnodes, just increase the instance count. If a machine misbehaves, just terminate it. Running `knife cluster launch hadoop_demo webnode` will note which machines are missing, and launch and configure them appropriately.
+
+Ironfan speaks naturally to both Chef and your cloud provider. The esnode's `cluster_role.override_attributes` statement will be synchronized to the chef server, pinning the elasticsearch version across the server and clients. Your chef roles should focus on specific subsystems; the cluster file lets you see the architecture as a whole.
+
+With these simple settings, if you have already [set up chef's knife to launch cloud servers](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife), typing `knife cluster launch hadoop_demo --bootstrap` will (using Amazon EC2 as an example):
+
+* Synchronize to the chef server:
+  - create chef roles on the server for the cluster and each facet.
+  - apply role directives (eg the homebase's `default_attributes` declaration).
+  - create a node for each machine
+  - apply the runlist to each node 
+* Set up security isolation:
+  - uses a keypair (login ssh key) isolated to that cluster
+  - Recognizes the `ssh` role, and add a security group `ssh` that by default opens port 22.
+  - Recognize the `nfs_server` role, and adds security groups `nfs_server` and `nfs_client`
+  - Authorizes the `nfs_server` to accept connections from all `nfs_client`s. Machines in other clusters that you mark as `nfs_client`s can connect to the NFS server, but are not automatically granted any other access to the machines in this cluster. Ironfan's opinionated behavior is about more than saving you effort -- tying this behavior to the chef role means you can't screw it up. 
+* Launches the machines in parallel:
+  - using the image name and the availability zone, it determines the appropriate region, image ID, and other implied behavior. 
+  - passes a JSON-encoded user_data hash specifying the machine's chef `node_name` and client key. An appropriately-configured machine image will need no further bootstrapping -- it will connect to the chef server with the appropriate identity and proceed completely unattended.
+* Syncronizes to the cloud provider:
+  - Applies EC2 tags to the machine, making your console intelligible: ![AWS Console screenshot](https://github.com/infochimps-labs/ironfan/raw/version_3/notes/aws_console_screenshot.jpg)
+  - Connects external (EBS) volumes, if any, to the correct mount point -- it uses (and applies) tags to the volumes, so they know which machine to adhere to. If you've manually added volumes, just make sure they're defined correctly in your cluster file and run `knife cluster sync {cluster_name}`; it will paint them with the correct tags.
+  - Associates an elastic IP, if any, to the machine
+* Bootstraps the machine using knife bootstrap

data/notes/walkthrough-web.md

@@ -0,0 +1,166 @@
+## Walkthrough: Web Cluster
+
+Here's a very simple cluster:
+
+```ruby
+Ironfan.cluster 'web_demo' do
+  cloud(:ec2) do
+    flavor              't1.micro'
+  end
+
+  role                  :base_role
+  role                  :chef_client
+  role                  :ssh
+
+  # The database server
+  facet :dbnode do
+    instances           1
+    role                :mysql_server
+
+    cloud do
+      flavor           'm1.large'
+      backing          'ebs'
+    end
+  end
+
+  # A throwaway facet for development.
+  facet :webnode do
+    instances           2
+    role                :nginx_server
+    role                :awesome_webapp
+  end
+end
+```
+
+This code defines a cluster named web_demo. A cluster is a group of servers united around a common purpose, in this case to serve a scalable web application.
+
+The web_demo cluster has two 'facets' -- dbnode and webnode. A facet is a subgroup of interchangeable servers that provide a logical set of systems: in this case, the systems that store the website's data and those that render it.
+
+The dbnode facet has one server, which will be named `web_demo-dbnode-0`; the webnode facet has two servers, `web_demo-webnode-0` and `web_demo-webnode-1`.
+
+Each server inherits the appropriate behaviors from its facet and cluster. All the servers in this cluster have the `base_role`, `chef_client` and `ssh` roles. The dbnode machines additionally house a MySQL server, while the webnodes have an nginx reverse proxy for the custom `web_demo_webapp`.
+
+As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the server, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
+
+### Cluster-level tools
+
+```
+$ knife cluster show web_demo
+
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+| Name                | Chef? | InstanceID | State       | Public IP    | Private IP    | Created At      | Flavor   | Image        | AZ         | SSH Key    |
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+| web_demo-dbnode-0   | yes   | i-43c60e20 | running     | 107.22.6.104 | 10.88.112.201 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | web_demo   |
+| web_demo-webnode-0  | yes   | i-1233aef1 | running     | 102.99.3.123 | 10.88.112.123 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | web_demo   |
+| web_demo-webnode-1  | yes   | i-0986423b | not running |              |               |                 |          |              |            |            |
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+
+```
+
+The commands available are:
+
+* list -- lists known clusters
+* show -- show the named servers
+* launch -- launch server
+* bootstrap  
+* sync       
+* ssh        
+* start/stop       
+* kill       
+* kick -- trigger a chef-client run on each named machine, tailing the logs until the run completes
+
+
+### Advanced clusters remain simple
+
+Let's say that app is truly awesome, and the features and demand increases. This cluster adds an [ElasticSearch server](http://elasticsearch.org) for searching, a haproxy loadbalancer, and spreads the webnodes across two availability zones.
+
+```ruby
+Ironfan.cluster 'web_demo' do
+  cloud(:ec2) do
+    image_name          "maverick"
+    flavor              "t1.micro"
+    availability_zones  ['us-east-1a']
+  end
+
+  # The database server
+  facet :dbnode do
+    instances           1
+    role                :mysql_server
+    cloud do
+      flavor           'm1.large'
+      backing          'ebs'
+    end
+
+    volume(:data) do
+      size              20
+      keep              true                        
+      device            '/dev/sdi'                  
+      mount_point       '/data'              
+      snapshot_id       'snap-a10234f'             
+      attachable        :ebs
+    end
+  end
+
+  facet :webnode do
+    instances           6
+    cloud.availability_zones  ['us-east-1a', 'us-east-1b']
+
+    role                :nginx_server
+    role                :awesome_webapp
+    role                :elasticsearch_client
+
+    volume(:server_logs) do
+      size              5                           
+      keep              true                        
+      device            '/dev/sdi'                  
+      mount_point       '/server_logs'              
+      snapshot_id       'snap-d9c1edb1'             
+    end
+  end
+
+  facet :esnode do
+    instances           1
+    role                "elasticsearch_data_esnode"
+    role                "elasticsearch_http_esnode"
+    cloud.flavor        "m1.large"
+  end
+
+  facet :loadbalancer do
+    instances           1
+    role                "haproxy"
+    cloud.flavor        "m1.xlarge"
+    elastic_ip          "128.69.69.23"
+  end
+  
+  cluster_role.override_attributes({
+    :elasticsearch => {
+      :version => '0.17.8',
+    },
+  })
+end
+```
+
+The facets are described and scale independently. If you'd like to add more webnodes, just increase the instance count. If a machine misbehaves, just terminate it. Running `knife cluster launch web_demo webnode` will note which machines are missing, and launch and configure them appropriately.
+
+Ironfan speaks naturally to both Chef and your cloud provider. The esnode's `cluster_role.override_attributes` statement will be synchronized to the chef server, pinning the elasticsearch version across the server and clients. Your chef roles should focus on specific subsystems; the cluster file lets you see the architecture as a whole.
+
+With these simple settings, if you have already [set up chef's knife to launch cloud servers](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife), typing `knife cluster launch web_demo --bootstrap` will (using Amazon EC2 as an example):
+
+* Synchronize to the chef server:
+  - create chef roles on the server for the cluster and each facet.
+  - apply role directives (eg the homebase's `default_attributes` declaration).
+  - create a node for each machine
+  - apply the runlist to each node 
+* Set up security isolation:
+  - uses a keypair (login ssh key) isolated to that cluster
+  - Recognizes the `ssh` role, and add a security group `ssh` that by default opens port 22.
+  - Recognize the `nfs_server` role, and adds security groups `nfs_server` and `nfs_client`
+  - Authorizes the `nfs_server` to accept connections from all `nfs_client`s. Machines in other clusters that you mark as `nfs_client`s can connect to the NFS server, but are not automatically granted any other access to the machines in this cluster. Ironfan's opinionated behavior is about more than saving you effort -- tying this behavior to the chef role means you can't screw it up. 
+* Launches the machines in parallel:
+  - using the image name and the availability zone, it determines the appropriate region, image ID, and other implied behavior. 
+  - passes a JSON-encoded user_data hash specifying the machine's chef `node_name` and client key. An appropriately-configured machine image will need no further bootstrapping -- it will connect to the chef server with the appropriate identity and proceed completely unattended.
+* Syncronizes to the cloud provider:
+  - Applies EC2 tags to the machine, making your console intelligible: ![AWS Console screenshot](https://github.com/infochimps-labs/ironfan/wiki/aws_servers.jpg)
+  - Connects external (EBS) volumes, if any, to the correct mount point -- it uses (and applies) tags to the volumes, so they know which machine to adhere to. If you've manually added volumes, just make sure they're defined correctly in your cluster file and run `knife cluster sync {cluster_name}`; it will paint them with the correct tags.
+  - Associates an elastic IP, if any, to the machine
+* Bootstraps the machine using knife bootstrap