ironfan 4.3.4 → 4.4.0

data/notes/version-3_2.md

@@ -0,0 +1,273 @@
+
+# v3.2.0 (future): Revamped undercarriage, spec coverage, standalone usage
+
+This is a Snow Leopard-style version change. No new features to speak of, but a much more solid and predictable foundation. 
+
+* **significantly cleaner DSL mixin**: uses the new, awesome `Gorillib::Builder`, giving it a much cleaner handling of fields and collections
+
+* **attributes are late-resolved**: in previous versions, the way you 'resolved' a server was to collapse the entire attribute set of cluster/facet/server hard onto the server model, a consistent source of bugs. Resolution is now done with the `Gorillib::Record::Overlay` mechanism, which means that you can set an attribute on the cluster and read it from the facet; change it later an all lower layers see the update.
+
+* **standalone usable**: can use ironfan-knife as a standalone library.
+
+# v3.3.x (future): Coherent universe of Servers, Components, Aspects
+
+* **spec coverage**: 
+
+* **coherent data model**: 
+
+    ComputeLayer   -- common attributes of Provider, Cluster, Facet, Server
+      - overlay_stack of Cloud attributes
+
+    Universe        -- across organizations
+    Organization    -- one or many providers
+    Provider        -- 
+    - has_many  :clusters
+    Cluster         --
+    - has_many  :providers
+    - overlays  :main_provider
+    Facet           --
+    - has_one  :cluster
+    - overlays :cluster
+    Server         
+    - has_one  :facet
+    - overlays :cluster
+    - has_one chef_node
+    - has_one machine
+    
+    
+    System            Role          Cookbook
+    Component                       Cookbook+Recipes
+    
+     
+
+* **improved discovery**: 
+
+* **config isolation**: 
+
+
+### Nitpicks
+
+
+* make bootstrap_distro and image_name follow from os_version
+
+* minidash just publishes announcements
+* silverware is always included; it subsumes volumes 
+
+* if you add a `data_dir_for    :hadoop` to 
+
+* volumes should name their `mount_point` after themselves by default
+
+### Components
+
+* components replace roles (they are auto-generated by the component, and tie strictly to it)
+* 
+
+### Clusters 
+
+If clusters are more repeatable they won't be so bothersomely multi-provider:
+    
+    Ironfan.cluster :gibbon do 
+      cloud(:ec2) do
+        backing         'ebs' 
+        permanent       false
+      end
+      stack             :systemwide
+      stack             :devstack
+      stack             :monitoring
+      stack             :log_handling
+      
+      component         :hadoop_devstack
+      component         :hadoop_dedicated
+      
+      discovers         :zookeeper, :realm => :zk
+      discovers         :hbase,     :realm => :hbase
+
+      facet :master do
+        component		:hadoop_namenode
+        component		:hadoop_secondarynn
+        component		:hadoop_jobtracker
+      end
+      facet :worker do
+        component		:hadoop_datanode
+        component		:hadoop_tasktracker
+      end
+      
+      volume :hadoop_data do
+        data_dir_for    :hadoop_datanode, :hadoop_namenode, :hadoop_secondarynn
+        device          '/dev/sdj1'
+        size            100
+        keep            true
+      end
+    end
+
+
+Here are ideas about how to get there
+
+    # silverware is always included; it subsumes volumes 
+
+    organization :infochimps do
+      cloud(:ec2) do
+        availability_zones  ['us-east-1d']
+        backing             :ebs
+        image_name          'ironfan-natty'
+        bootstrap_distro    'ironfan-natty'
+        chef_client_script  'client.rb'
+        permanent           true
+      end
+      
+      volume(:default) do
+        keep                true
+        snapshot_name       :blank_xfs
+        resizable           true
+        create_at_launch    true
+      end
+      
+      stack :systemwide do
+        system(:chef_client) do
+          run_state         :on_restart
+        end
+        component		    :set_hostname
+        component		    :minidash
+        component           :org_base
+        component           :org_users
+        component           :org_final
+      end
+      
+      stack :devstack do
+        component		    :ssh
+        component		    :nfs_client
+        component		    :package_set
+      end
+
+      stack :monitoring do
+        component		:zabbix_agent
+      end
+      
+      stack :log_handling do
+        component		:log_handling
+      end
+    end
+    
+    stack :hadoop do
+    end
+    
+    stack :hadoop_devstack do
+      component         :pig
+      component         :jruby
+      component         :rstats
+    end
+    
+    stack :hadoop_dedicated do
+      component         :tuning
+    end
+
+    system :hadoop do
+      stack :hadoop_devstack
+      stack :zookeeper_client
+      stack :hbase_client
+    end   
+        
+    Ironfan.cluster :gibbon do 
+      cloud(:ec2) do
+        backing             'ebs' 
+        permanent           false
+      end
+      
+      system :systemwide do
+        exclude_stack   :monitoring
+      end
+      
+      # how are its components configured? distributed among machines?
+      system :hadoop do 
+            
+        # all servers will
+        # * have the `hadoop` role
+        # * have run_state => false for components with a daemon aspect by default
+        
+        facet :master do
+          # component :hadoop_namenode means
+          # * this facet has the `hadoop_namenode` role
+          # * it has the component's security_groups
+          # * it sets node[:hadoop][:namenode][:run_state] = true
+          # * it will mount the volumes that adhere to this component
+          component :hadoop_namenode
+        end
+        
+        # something gains eg zookeeper client if it discovers a zookeeper in another realm
+        # zookeeper must explicitly admit it discovers zookeeper, but can do that in the component
+        
+        # what volumes should it use on those machines?
+        # create the volumes, pair it to components
+        # if a component is on a server, it adds its volumes. 
+        # you can also add them explicitly.
+        
+        # volume tags are applied automagically from their adherance to components
+
+        volume :hadoop_data do                            # will be assigned to servers with components it lists
+          data_dir_for    :hadoop_datanode, :hadoop_namenode, :hadoop_secondarynn
+        end
+
+### Providers
+
+I want to be able to:
+
+* on a compute layer, modify its behavior depending on provider:
+  - example:
+  
+      facet(:bob) do 
+        cloud do
+          security_group  :bob
+          authorize       :from => :bobs_friends, :to => :bob
+        end
+        cloud(:ec2,       :flavor => 'm1.small')
+        cloud(:rackspace, :flavor => '2GB')
+        cloud(:vagrant,   :ram_mb =>  256 )
+      end
+
+  - Any world that understands security groups will endeavor to make a `bob` security group, and authorize the `bobs_friends` group to use it.
+  - On EC2 and rackspace, the `flavor` attribute is set explicitly
+  - On vagrant (which got no `flavor`), we instead specify how much ram to supply
+  - On any other provider the flavor and machine ram will follow defaults.
+
+* see all machines and clusters within an organization
+
+
+### Organizations
+
+* see the entire universe; this might get hairy, but not ridiculous
+  - each org describes its providers; only those are used
+  - you don't have to do much to add a provider, just say `provider(:ec2)`
+  - you can configure the provider like this:
+  
+      organization(:infochimps_test, :doc => 'Infochimps test cloud') do
+        provider(:vagrant)
+        provider(:ec2) do
+          access_key         '...' 
+          secret_access_key  '...'
+        end
+        provider(:hp_cloud) do
+          access_key         '...'
+          secret_access_key  '...'
+        end
+      end
+      
+      organization(:demo, :doc => 'Live client demo cloud') do
+        provider(:vagrant)
+        provider(:ec2)       do  #... end
+        provider(:hp_cloud)  do  #... end
+        provider(:rackspace) do  #... end
+      end
+
+  - clusters can be declared directly or imported from other organizations:
+  
+      organization :infochimps_test do
+        # developers' sandboxes
+        cluster  :dev_sandboxes
+        # all the example clusters, for development
+        organization(:examples).clusters.each do |cl|
+          add_cluster cl
+        end
+      end  
+
+  - if just starting, should see clusters; 
+    - per-org cluster dirs

data/notes/walkthrough-hadoop.md

@@ -0,0 +1,168 @@
+FIXME: Repurpose general structure to demonstrate a Hadoop cluster.
+
+## Walkthrough: Hadoop Cluster
+
+Here's a very simple cluster:
+
+```ruby
+Ironfan.cluster 'hadoop_demo' do
+  cloud(:ec2) do
+    flavor              't1.micro'
+  end
+
+  role                  :base_role
+  role                  :chef_client
+  role                  :ssh
+
+  # The database server
+  facet :dbnode do
+    instances           1
+    role                :mysql_server
+
+    cloud do
+      flavor           'm1.large'
+      backing          'ebs'
+    end
+  end
+
+  # A throwaway facet for development.
+  facet :webnode do
+    instances           2
+    role                :nginx_server
+    role                :awesome_webapp
+  end
+end
+```
+
+This code defines a cluster named hadoop_demo. A cluster is a group of servers united around a common purpose, in this case to serve a scalable web application.
+
+The hadoop_demo cluster has two 'facets' -- dbnode and webnode. A facet is a subgroup of interchangeable servers that provide a logical set of systems: in this case, the systems that store the website's data and those that render it.
+
+The dbnode facet has one server, which will be named `hadoop_demo-dbnode-0`; the webnode facet has two servers, `hadoop_demo-webnode-0` and `hadoop_demo-webnode-1`.
+
+Each server inherits the appropriate behaviors from its facet and cluster. All the servers in this cluster have the `base_role`, `chef_client` and `ssh` roles. The dbnode machines additionally house a MySQL server, while the webnodes have an nginx reverse proxy for the custom `hadoop_demo_webapp`.
+
+As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the server, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
+
+### Cluster-level tools
+
+```
+$ knife cluster show hadoop_demo
+
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+| Name                | Chef? | InstanceID | State       | Public IP    | Private IP    | Created At      | Flavor   | Image        | AZ         | SSH Key    |
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+| hadoop_demo-dbnode-0   | yes   | i-43c60e20 | running     | 107.22.6.104 | 10.88.112.201 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | hadoop_demo   |
+| hadoop_demo-webnode-0  | yes   | i-1233aef1 | running     | 102.99.3.123 | 10.88.112.123 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | hadoop_demo   |
+| hadoop_demo-webnode-1  | yes   | i-0986423b | not running |              |               |                 |          |              |            |            |
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+
+```
+
+The commands available are:
+
+* list -- lists known clusters
+* show -- show the named servers
+* launch -- launch server
+* bootstrap  
+* sync       
+* ssh        
+* start/stop       
+* kill       
+* kick -- trigger a chef-client run on each named machine, tailing the logs until the run completes
+
+
+### Advanced clusters remain simple
+
+Let's say that app is truly awesome, and the features and demand increases. This cluster adds an [ElasticSearch server](http://elasticsearch.org) for searching, a haproxy loadbalancer, and spreads the webnodes across two availability zones.
+
+```ruby
+Ironfan.cluster 'hadoop_demo' do
+  cloud(:ec2) do
+    image_name          "maverick"
+    flavor              "t1.micro"
+    availability_zones  ['us-east-1a']
+  end
+
+  # The database server
+  facet :dbnode do
+    instances           1
+    role                :mysql_server
+    cloud do
+      flavor           'm1.large'
+      backing          'ebs'
+    end
+
+    volume(:data) do
+      size              20
+      keep              true                        
+      device            '/dev/sdi'                  
+      mount_point       '/data'              
+      snapshot_id       'snap-a10234f'             
+      attachable        :ebs
+    end
+  end
+
+  facet :webnode do
+    instances           6
+    cloud.availability_zones  ['us-east-1a', 'us-east-1b']
+
+    role                :nginx_server
+    role                :awesome_webapp
+    role                :elasticsearch_client
+
+    volume(:server_logs) do
+      size              5                           
+      keep              true                        
+      device            '/dev/sdi'                  
+      mount_point       '/server_logs'              
+      snapshot_id       'snap-d9c1edb1'             
+    end
+  end
+
+  facet :esnode do
+    instances           1
+    role                "elasticsearch_data_esnode"
+    role                "elasticsearch_http_esnode"
+    cloud.flavor        "m1.large"
+  end
+
+  facet :loadbalancer do
+    instances           1
+    role                "haproxy"
+    cloud.flavor        "m1.xlarge"
+    elastic_ip          "128.69.69.23"
+  end
+  
+  cluster_role.override_attributes({
+    :elasticsearch => {
+      :version => '0.17.8',
+    },
+  })
+end
+```
+
+The facets are described and scale independently. If you'd like to add more webnodes, just increase the instance count. If a machine misbehaves, just terminate it. Running `knife cluster launch hadoop_demo webnode` will note which machines are missing, and launch and configure them appropriately.
+
+Ironfan speaks naturally to both Chef and your cloud provider. The esnode's `cluster_role.override_attributes` statement will be synchronized to the chef server, pinning the elasticsearch version across the server and clients. Your chef roles should focus on specific subsystems; the cluster file lets you see the architecture as a whole.
+
+With these simple settings, if you have already [set up chef's knife to launch cloud servers](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife), typing `knife cluster launch hadoop_demo --bootstrap` will (using Amazon EC2 as an example):
+
+* Synchronize to the chef server:
+  - create chef roles on the server for the cluster and each facet.
+  - apply role directives (eg the homebase's `default_attributes` declaration).
+  - create a node for each machine
+  - apply the runlist to each node 
+* Set up security isolation:
+  - uses a keypair (login ssh key) isolated to that cluster
+  - Recognizes the `ssh` role, and add a security group `ssh` that by default opens port 22.
+  - Recognize the `nfs_server` role, and adds security groups `nfs_server` and `nfs_client`
+  - Authorizes the `nfs_server` to accept connections from all `nfs_client`s. Machines in other clusters that you mark as `nfs_client`s can connect to the NFS server, but are not automatically granted any other access to the machines in this cluster. Ironfan's opinionated behavior is about more than saving you effort -- tying this behavior to the chef role means you can't screw it up. 
+* Launches the machines in parallel:
+  - using the image name and the availability zone, it determines the appropriate region, image ID, and other implied behavior. 
+  - passes a JSON-encoded user_data hash specifying the machine's chef `node_name` and client key. An appropriately-configured machine image will need no further bootstrapping -- it will connect to the chef server with the appropriate identity and proceed completely unattended.
+* Syncronizes to the cloud provider:
+  - Applies EC2 tags to the machine, making your console intelligible: ![AWS Console screenshot](https://github.com/infochimps-labs/ironfan/raw/version_3/notes/aws_console_screenshot.jpg)
+  - Connects external (EBS) volumes, if any, to the correct mount point -- it uses (and applies) tags to the volumes, so they know which machine to adhere to. If you've manually added volumes, just make sure they're defined correctly in your cluster file and run `knife cluster sync {cluster_name}`; it will paint them with the correct tags.
+  - Associates an elastic IP, if any, to the machine
+* Bootstraps the machine using knife bootstrap

data/notes/walkthrough-web.md

@@ -0,0 +1,166 @@
+## Walkthrough: Web Cluster
+
+Here's a very simple cluster:
+
+```ruby
+Ironfan.cluster 'web_demo' do
+  cloud(:ec2) do
+    flavor              't1.micro'
+  end
+
+  role                  :base_role
+  role                  :chef_client
+  role                  :ssh
+
+  # The database server
+  facet :dbnode do
+    instances           1
+    role                :mysql_server
+
+    cloud do
+      flavor           'm1.large'
+      backing          'ebs'
+    end
+  end
+
+  # A throwaway facet for development.
+  facet :webnode do
+    instances           2
+    role                :nginx_server
+    role                :awesome_webapp
+  end
+end
+```
+
+This code defines a cluster named web_demo. A cluster is a group of servers united around a common purpose, in this case to serve a scalable web application.
+
+The web_demo cluster has two 'facets' -- dbnode and webnode. A facet is a subgroup of interchangeable servers that provide a logical set of systems: in this case, the systems that store the website's data and those that render it.
+
+The dbnode facet has one server, which will be named `web_demo-dbnode-0`; the webnode facet has two servers, `web_demo-webnode-0` and `web_demo-webnode-1`.
+
+Each server inherits the appropriate behaviors from its facet and cluster. All the servers in this cluster have the `base_role`, `chef_client` and `ssh` roles. The dbnode machines additionally house a MySQL server, while the webnodes have an nginx reverse proxy for the custom `web_demo_webapp`.
+
+As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the server, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
+
+### Cluster-level tools
+
+```
+$ knife cluster show web_demo
+
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+| Name                | Chef? | InstanceID | State       | Public IP    | Private IP    | Created At      | Flavor   | Image        | AZ         | SSH Key    |
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+| web_demo-dbnode-0   | yes   | i-43c60e20 | running     | 107.22.6.104 | 10.88.112.201 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | web_demo   |
+| web_demo-webnode-0  | yes   | i-1233aef1 | running     | 102.99.3.123 | 10.88.112.123 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | web_demo   |
+| web_demo-webnode-1  | yes   | i-0986423b | not running |              |               |                 |          |              |            |            |
++---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
+
+```
+
+The commands available are:
+
+* list -- lists known clusters
+* show -- show the named servers
+* launch -- launch server
+* bootstrap  
+* sync       
+* ssh        
+* start/stop       
+* kill       
+* kick -- trigger a chef-client run on each named machine, tailing the logs until the run completes
+
+
+### Advanced clusters remain simple
+
+Let's say that app is truly awesome, and the features and demand increases. This cluster adds an [ElasticSearch server](http://elasticsearch.org) for searching, a haproxy loadbalancer, and spreads the webnodes across two availability zones.
+
+```ruby
+Ironfan.cluster 'web_demo' do
+  cloud(:ec2) do
+    image_name          "maverick"
+    flavor              "t1.micro"
+    availability_zones  ['us-east-1a']
+  end
+
+  # The database server
+  facet :dbnode do
+    instances           1
+    role                :mysql_server
+    cloud do
+      flavor           'm1.large'
+      backing          'ebs'
+    end
+
+    volume(:data) do
+      size              20
+      keep              true                        
+      device            '/dev/sdi'                  
+      mount_point       '/data'              
+      snapshot_id       'snap-a10234f'             
+      attachable        :ebs
+    end
+  end
+
+  facet :webnode do
+    instances           6
+    cloud.availability_zones  ['us-east-1a', 'us-east-1b']
+
+    role                :nginx_server
+    role                :awesome_webapp
+    role                :elasticsearch_client
+
+    volume(:server_logs) do
+      size              5                           
+      keep              true                        
+      device            '/dev/sdi'                  
+      mount_point       '/server_logs'              
+      snapshot_id       'snap-d9c1edb1'             
+    end
+  end
+
+  facet :esnode do
+    instances           1
+    role                "elasticsearch_data_esnode"
+    role                "elasticsearch_http_esnode"
+    cloud.flavor        "m1.large"
+  end
+
+  facet :loadbalancer do
+    instances           1
+    role                "haproxy"
+    cloud.flavor        "m1.xlarge"
+    elastic_ip          "128.69.69.23"
+  end
+  
+  cluster_role.override_attributes({
+    :elasticsearch => {
+      :version => '0.17.8',
+    },
+  })
+end
+```
+
+The facets are described and scale independently. If you'd like to add more webnodes, just increase the instance count. If a machine misbehaves, just terminate it. Running `knife cluster launch web_demo webnode` will note which machines are missing, and launch and configure them appropriately.
+
+Ironfan speaks naturally to both Chef and your cloud provider. The esnode's `cluster_role.override_attributes` statement will be synchronized to the chef server, pinning the elasticsearch version across the server and clients. Your chef roles should focus on specific subsystems; the cluster file lets you see the architecture as a whole.
+
+With these simple settings, if you have already [set up chef's knife to launch cloud servers](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife), typing `knife cluster launch web_demo --bootstrap` will (using Amazon EC2 as an example):
+
+* Synchronize to the chef server:
+  - create chef roles on the server for the cluster and each facet.
+  - apply role directives (eg the homebase's `default_attributes` declaration).
+  - create a node for each machine
+  - apply the runlist to each node 
+* Set up security isolation:
+  - uses a keypair (login ssh key) isolated to that cluster
+  - Recognizes the `ssh` role, and add a security group `ssh` that by default opens port 22.
+  - Recognize the `nfs_server` role, and adds security groups `nfs_server` and `nfs_client`
+  - Authorizes the `nfs_server` to accept connections from all `nfs_client`s. Machines in other clusters that you mark as `nfs_client`s can connect to the NFS server, but are not automatically granted any other access to the machines in this cluster. Ironfan's opinionated behavior is about more than saving you effort -- tying this behavior to the chef role means you can't screw it up. 
+* Launches the machines in parallel:
+  - using the image name and the availability zone, it determines the appropriate region, image ID, and other implied behavior. 
+  - passes a JSON-encoded user_data hash specifying the machine's chef `node_name` and client key. An appropriately-configured machine image will need no further bootstrapping -- it will connect to the chef server with the appropriate identity and proceed completely unattended.
+* Syncronizes to the cloud provider:
+  - Applies EC2 tags to the machine, making your console intelligible: ![AWS Console screenshot](https://github.com/infochimps-labs/ironfan/wiki/aws_servers.jpg)
+  - Connects external (EBS) volumes, if any, to the correct mount point -- it uses (and applies) tags to the volumes, so they know which machine to adhere to. If you've manually added volumes, just make sure they're defined correctly in your cluster file and run `knife cluster sync {cluster_name}`; it will paint them with the correct tags.
+  - Associates an elastic IP, if any, to the machine
+* Bootstraps the machine using knife bootstrap