ironfan 3.1.7 → 3.2.2

data/notes/tips_and_troubleshooting.md

@@ -1,92 +0,0 @@
-## Tips and Notes
-
-### Gems
-
-   knife cluster ssh bonobo-worker-2 'sudo gem update --system'
-   knife cluster ssh bonobo-worker-2 'sudo true ; for foo in /usr/lib/ruby/gems/1.9.2-p290/specifications/*  ; do sudo sed -i.bak "s!000000000Z!!"          $foo ; done'
-   knife cluster ssh bonobo-worker-2 'sudo true ; for foo in /usr/lib/ruby/site_ruby/*/rubygems/deprecate.rb ; do sudo sed -i.bak "s!@skip ||= false!true!" $foo ; done'
-
-
-### EC2 Notes Instance attributes: `disable_api_termination` and `delete_on_termination`
-
-To set `delete_on_termination` to 'true' after the fact, run the following (modify the instance and volume to suit):
-
-```
-    ec2-modify-instance-attribute -v i-0704be6c --block-device-mapping /dev/sda1=vol-XX8d2c80::true
-```
-
-If you set `disable_api_termination` to true, in order to terminate the node run
-```
-    ec2-modify-instance-attribute -v i-0704be6c --disable-api-termination false
-```
-
-To view whether an attached volume is deleted when the machine is terminated:
-
-```
-    # show volumes that will be deleted
-    ec2-describe-volumes --filter "attachment.delete-on-termination=true"
-```
-
-You can't (as far as I know) alter the delete-on-termination flag of a running volume. Crazy, huh?
-
-### EC2: See your userdata
-
-curl http://169.254.169.254/latest/user-data
-
-### EBS Volumes for a persistent HDFS
-
-* Make one volume and format for XFS:
-    `$ sudo mkfs.xfs -f /dev/sdh1`
-* options "defaults,nouuid,noatime" give good results. The 'nouuid' part
-  prevents errors when mounting multiple volumes from the same snapshot.
-* poke a file onto the drive :
-  datename=`date +%Y%m%d`
-  sudo bash -c "(echo $datename ; df /data/ebs1 ) > /data/ebs1/xfs-created-at-$datename.txt"
-
-
-If you want to grow the drive:
-* take a snapshot.
-* make a new volume from it
-* mount that, and run `sudo xfs_growfs`. You *should* have the volume mounted, and should stop anything that would be working the volume hard.
-
-### Hadoop: On-the-fly backup of your namenode metadata
-
-bkupdir=/ebs2/hadoop-nn-backup/`date +"%Y%m%d"`
-
-for srcdir in /ebs*/hadoop/hdfs/ /home/hadoop/gibbon/hdfs/  ; do
-  destdir=$bkupdir/$srcdir ; echo $destdir ;
-  sudo mkdir -p $destdir ;
-done
-
-
-### NFS: Halp I am using an NFS-mounted /home and now I can't log in as ubuntu
-
-Say you set up an NFS server 'core-homebase-0' (in the 'core' cluster) to host and serve out `/home` directory; and a machine 'awesome-webserver-0' (in the 'awesome' cluster), that is an NFS client.
-
-In each case, when the machine was born EC2 created a `/home/ubuntu/.ssh/authorized_keys` file listing only the single approved machine keypair -- 'core' for the core cluster, 'awesome' for the awesome cluster.
-
-When chef client runs, however, it mounts the NFS share at /home. This then masks the actual /home directory -- nothing that's on the base directory tree shows up. Which means that after chef runs, the /home/ubuntu/.ssh/authorized_keys file on awesome-webserver-0 is the one for the *'core'* cluster, not the *'awesome'* cluster.
-
-The solution is to use the cookbook ironfan provides -- it moves the 'ubuntu' user's home directory to an alternative path not masked by the NFS.
-
-
-### NFS: Problems starting NFS server on ubuntu maverick
-
-For problems starting NFS server on ubuntu maverick systems, read, understand and then run /tmp/fix_nfs_on_maverick_amis.sh -- See "this thread for more":http://fossplanet.com/f10/[ec2ubuntu]-not-starting-nfs-kernel-daemon-no-support-current-kernel-90948/
-
-
-### Git deploys: My git deploy recipe has gone limp
-
-Suppose you are using the @git@ resource to deploy a recipe (@george@ for sake of example). If @/var/chef/cache/revision_deploys/var/www/george@ exists then *nothing* will get deployed, even if /var/www/george/{release_sha} is empty or screwy.  If git deploy is acting up in any way, nuke that cache from orbit -- it's the only way to be sure.
-
- $ sudo rm -rf /var/www/george/{release_sha} /var/chef/cache/revision_deploys/var/www/george
-
-### Runit services : 'fail: XXX: unable to change to service directory: file does not exist'
-
-Your service is probably installed but removed from runit's purview; check the `/etc/service` symlink. All of the following should be true:
-
-* directory `/etc/sv/foo`, containing file `run` and dirs `log` and `supervise`
-* `/etc/init.d/foo`  is symlinked to `/usr/bin/sv`
-* `/etc/servics/foo` is symlinked tp `/etc/sv/foo`
-
-

data/notes/version-3_2.md

@@ -1,273 +0,0 @@
-
-# v3.2.0 (future): Revamped undercarriage, spec coverage, standalone usage
-
-This is a Snow Leopard-style version change. No new features to speak of, but a much more solid and predictable foundation. 
-
-* **significantly cleaner DSL mixin**: uses the new, awesome `Gorillib::Builder`, giving it a much cleaner handling of fields and collections
-
-* **attributes are late-resolved**: in previous versions, the way you 'resolved' a server was to collapse the entire attribute set of cluster/facet/server hard onto the server model, a consistent source of bugs. Resolution is now done with the `Gorillib::Record::Overlay` mechanism, which means that you can set an attribute on the cluster and read it from the facet; change it later an all lower layers see the update.
-
-* **standalone usable**: can use ironfan-knife as a standalone library.
-
-# v3.3.x (future): Coherent universe of Servers, Components, Aspects
-
-* **spec coverage**: 
-
-* **coherent data model**: 
-
-    ComputeLayer   -- common attributes of Provider, Cluster, Facet, Server
-      - overlay_stack of Cloud attributes
-
-    Universe        -- across organizations
-    Organization    -- one or many providers
-    Provider        -- 
-    - has_many  :clusters
-    Cluster         --
-    - has_many  :providers
-    - overlays  :main_provider
-    Facet           --
-    - has_one  :cluster
-    - overlays :cluster
-    Server         
-    - has_one  :facet
-    - overlays :cluster
-    - has_one chef_node
-    - has_one machine
-    
-    
-    System            Role          Cookbook
-    Component                       Cookbook+Recipes
-    
-     
-
-* **improved discovery**: 
-
-* **config isolation**: 
-
-
-### Nitpicks
-
-
-* make bootstrap_distro and image_name follow from os_version
-
-* minidash just publishes announcements
-* silverware is always included; it subsumes volumes 
-
-* if you add a `data_dir_for    :hadoop` to 
-
-* volumes should name their `mount_point` after themselves by default
-
-### Components
-
-* components replace roles (they are auto-generated by the component, and tie strictly to it)
-* 
-
-### Clusters 
-
-If clusters are more repeatable they won't be so bothersomely multi-provider:
-    
-    Ironfan.cluster :gibbon do 
-      cloud(:ec2) do
-        backing         'ebs' 
-        permanent       false
-      end
-      stack             :systemwide
-      stack             :devstack
-      stack             :monitoring
-      stack             :log_handling
-      
-      component         :hadoop_devstack
-      component         :hadoop_dedicated
-      
-      discovers         :zookeeper, :realm => :zk
-      discovers         :hbase,     :realm => :hbase
-
-      facet :master do
-        component		:hadoop_namenode
-        component		:hadoop_secondarynn
-        component		:hadoop_jobtracker
-      end
-      facet :worker do
-        component		:hadoop_datanode
-        component		:hadoop_tasktracker
-      end
-      
-      volume :hadoop_data do
-        data_dir_for    :hadoop_datanode, :hadoop_namenode, :hadoop_secondarynn
-        device          '/dev/sdj1'
-        size            100
-        keep            true
-      end
-    end
-
-
-Here are ideas about how to get there
-
-    # silverware is always included; it subsumes volumes 
-
-    organization :infochimps do
-      cloud(:ec2) do
-        availability_zones  ['us-east-1d']
-        backing             :ebs
-        image_name          'ironfan-natty'
-        bootstrap_distro    'ironfan-natty'
-        chef_client_script  'client.rb'
-        permanent           true
-      end
-      
-      volume(:default) do
-        keep                true
-        snapshot_name       :blank_xfs
-        resizable           true
-        create_at_launch    true
-      end
-      
-      stack :systemwide do
-        system(:chef_client) do
-          run_state         :on_restart
-        end
-        component		    :set_hostname
-        component		    :minidash
-        component           :org_base
-        component           :org_users
-        component           :org_final
-      end
-      
-      stack :devstack do
-        component		    :ssh
-        component		    :nfs_client
-        component		    :package_set
-      end
-
-      stack :monitoring do
-        component		:zabbix_agent
-      end
-      
-      stack :log_handling do
-        component		:log_handling
-      end
-    end
-    
-    stack :hadoop do
-    end
-    
-    stack :hadoop_devstack do
-      component         :pig
-      component         :jruby
-      component         :rstats
-    end
-    
-    stack :hadoop_dedicated do
-      component         :tuning
-    end
-
-    system :hadoop do
-      stack :hadoop_devstack
-      stack :zookeeper_client
-      stack :hbase_client
-    end   
-        
-    Ironfan.cluster :gibbon do 
-      cloud(:ec2) do
-        backing             'ebs' 
-        permanent           false
-      end
-      
-      system :systemwide do
-        exclude_stack   :monitoring
-      end
-      
-      # how are its components configured? distributed among machines?
-      system :hadoop do 
-            
-        # all servers will
-        # * have the `hadoop` role
-        # * have run_state => false for components with a daemon aspect by default
-        
-        facet :master do
-          # component :hadoop_namenode means
-          # * this facet has the `hadoop_namenode` role
-          # * it has the component's security_groups
-          # * it sets node[:hadoop][:namenode][:run_state] = true
-          # * it will mount the volumes that adhere to this component
-          component :hadoop_namenode
-        end
-        
-        # something gains eg zookeeper client if it discovers a zookeeper in another realm
-        # zookeeper must explicitly admit it discovers zookeeper, but can do that in the component
-        
-        # what volumes should it use on those machines?
-        # create the volumes, pair it to components
-        # if a component is on a server, it adds its volumes. 
-        # you can also add them explicitly.
-        
-        # volume tags are applied automagically from their adherance to components
-
-        volume :hadoop_data do                            # will be assigned to servers with components it lists
-          data_dir_for    :hadoop_datanode, :hadoop_namenode, :hadoop_secondarynn
-        end
-
-### Providers
-
-I want to be able to:
-
-* on a compute layer, modify its behavior depending on provider:
-  - example:
-  
-      facet(:bob) do 
-        cloud do
-          security_group  :bob
-          authorize       :from => :bobs_friends, :to => :bob
-        end
-        cloud(:ec2,       :flavor => 'm1.small')
-        cloud(:rackspace, :flavor => '2GB')
-        cloud(:vagrant,   :ram_mb =>  256 )
-      end
-
-  - Any world that understands security groups will endeavor to make a `bob` security group, and authorize the `bobs_friends` group to use it.
-  - On EC2 and rackspace, the `flavor` attribute is set explicitly
-  - On vagrant (which got no `flavor`), we instead specify how much ram to supply
-  - On any other provider the flavor and machine ram will follow defaults.
-
-* see all machines and clusters within an organization
-
-
-### Organizations
-
-* see the entire universe; this might get hairy, but not ridiculous
-  - each org describes its providers; only those are used
-  - you don't have to do much to add a provider, just say `provider(:ec2)`
-  - you can configure the provider like this:
-  
-      organization(:infochimps_test, :doc => 'Infochimps test cloud') do
-        provider(:vagrant)
-        provider(:ec2) do
-          access_key         '...' 
-          secret_access_key  '...'
-        end
-        provider(:hp_cloud) do
-          access_key         '...'
-          secret_access_key  '...'
-        end
-      end
-      
-      organization(:demo, :doc => 'Live client demo cloud') do
-        provider(:vagrant)
-        provider(:ec2)       do  #... end
-        provider(:hp_cloud)  do  #... end
-        provider(:rackspace) do  #... end
-      end
-
-  - clusters can be declared directly or imported from other organizations:
-  
-      organization :infochimps_test do
-        # developers' sandboxes
-        cluster  :dev_sandboxes
-        # all the example clusters, for development
-        organization(:examples).clusters.each do |cl|
-          add_cluster cl
-        end
-      end  
-
-  - if just starting, should see clusters; 
-    - per-org cluster dirs

data/notes/walkthrough-hadoop.md

@@ -1,168 +0,0 @@
-FIXME: Repurpose general structure to demonstrate a Hadoop cluster.
-
-## Walkthrough: Hadoop Cluster
-
-Here's a very simple cluster:
-
-```ruby
-Ironfan.cluster 'hadoop_demo' do
-  cloud(:ec2) do
-    flavor              't1.micro'
-  end
-
-  role                  :base_role
-  role                  :chef_client
-  role                  :ssh
-
-  # The database server
-  facet :dbnode do
-    instances           1
-    role                :mysql_server
-
-    cloud do
-      flavor           'm1.large'
-      backing          'ebs'
-    end
-  end
-
-  # A throwaway facet for development.
-  facet :webnode do
-    instances           2
-    role                :nginx_server
-    role                :awesome_webapp
-  end
-end
-```
-
-This code defines a cluster named hadoop_demo. A cluster is a group of servers united around a common purpose, in this case to serve a scalable web application.
-
-The hadoop_demo cluster has two 'facets' -- dbnode and webnode. A facet is a subgroup of interchangeable servers that provide a logical set of systems: in this case, the systems that store the website's data and those that render it.
-
-The dbnode facet has one server, which will be named `hadoop_demo-dbnode-0`; the webnode facet has two servers, `hadoop_demo-webnode-0` and `hadoop_demo-webnode-1`.
-
-Each server inherits the appropriate behaviors from its facet and cluster. All the servers in this cluster have the `base_role`, `chef_client` and `ssh` roles. The dbnode machines additionally house a MySQL server, while the webnodes have an nginx reverse proxy for the custom `hadoop_demo_webapp`.
-
-As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the server, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
-
-### Cluster-level tools
-
-```
-$ knife cluster show hadoop_demo
-
-+---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
-| Name                | Chef? | InstanceID | State       | Public IP    | Private IP    | Created At      | Flavor   | Image        | AZ         | SSH Key    |
-+---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
-| hadoop_demo-dbnode-0   | yes   | i-43c60e20 | running     | 107.22.6.104 | 10.88.112.201 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | hadoop_demo   |
-| hadoop_demo-webnode-0  | yes   | i-1233aef1 | running     | 102.99.3.123 | 10.88.112.123 | 20111029-204156 | t1.micro | ami-cef405a7 | us-east-1a | hadoop_demo   |
-| hadoop_demo-webnode-1  | yes   | i-0986423b | not running |              |               |                 |          |              |            |            |
-+---------------------+-------+------------+-------------+--------------+---------------+-----------------+----------+--------------+------------+------------+
-
-```
-
-The commands available are:
-
-* list -- lists known clusters
-* show -- show the named servers
-* launch -- launch server
-* bootstrap  
-* sync       
-* ssh        
-* start/stop       
-* kill       
-* kick -- trigger a chef-client run on each named machine, tailing the logs until the run completes
-
-
-### Advanced clusters remain simple
-
-Let's say that app is truly awesome, and the features and demand increases. This cluster adds an [ElasticSearch server](http://elasticsearch.org) for searching, a haproxy loadbalancer, and spreads the webnodes across two availability zones.
-
-```ruby
-Ironfan.cluster 'hadoop_demo' do
-  cloud(:ec2) do
-    image_name          "maverick"
-    flavor              "t1.micro"
-    availability_zones  ['us-east-1a']
-  end
-
-  # The database server
-  facet :dbnode do
-    instances           1
-    role                :mysql_server
-    cloud do
-      flavor           'm1.large'
-      backing          'ebs'
-    end
-
-    volume(:data) do
-      size              20
-      keep              true                        
-      device            '/dev/sdi'                  
-      mount_point       '/data'              
-      snapshot_id       'snap-a10234f'             
-      attachable        :ebs
-    end
-  end
-
-  facet :webnode do
-    instances           6
-    cloud.availability_zones  ['us-east-1a', 'us-east-1b']
-
-    role                :nginx_server
-    role                :awesome_webapp
-    role                :elasticsearch_client
-
-    volume(:server_logs) do
-      size              5                           
-      keep              true                        
-      device            '/dev/sdi'                  
-      mount_point       '/server_logs'              
-      snapshot_id       'snap-d9c1edb1'             
-    end
-  end
-
-  facet :esnode do
-    instances           1
-    role                "elasticsearch_data_esnode"
-    role                "elasticsearch_http_esnode"
-    cloud.flavor        "m1.large"
-  end
-
-  facet :loadbalancer do
-    instances           1
-    role                "haproxy"
-    cloud.flavor        "m1.xlarge"
-    elastic_ip          "128.69.69.23"
-  end
-  
-  cluster_role.override_attributes({
-    :elasticsearch => {
-      :version => '0.17.8',
-    },
-  })
-end
-```
-
-The facets are described and scale independently. If you'd like to add more webnodes, just increase the instance count. If a machine misbehaves, just terminate it. Running `knife cluster launch hadoop_demo webnode` will note which machines are missing, and launch and configure them appropriately.
-
-Ironfan speaks naturally to both Chef and your cloud provider. The esnode's `cluster_role.override_attributes` statement will be synchronized to the chef server, pinning the elasticsearch version across the server and clients. Your chef roles should focus on specific subsystems; the cluster file lets you see the architecture as a whole.
-
-With these simple settings, if you have already [set up chef's knife to launch cloud servers](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife), typing `knife cluster launch hadoop_demo --bootstrap` will (using Amazon EC2 as an example):
-
-* Synchronize to the chef server:
-  - create chef roles on the server for the cluster and each facet.
-  - apply role directives (eg the homebase's `default_attributes` declaration).
-  - create a node for each machine
-  - apply the runlist to each node 
-* Set up security isolation:
-  - uses a keypair (login ssh key) isolated to that cluster
-  - Recognizes the `ssh` role, and add a security group `ssh` that by default opens port 22.
-  - Recognize the `nfs_server` role, and adds security groups `nfs_server` and `nfs_client`
-  - Authorizes the `nfs_server` to accept connections from all `nfs_client`s. Machines in other clusters that you mark as `nfs_client`s can connect to the NFS server, but are not automatically granted any other access to the machines in this cluster. Ironfan's opinionated behavior is about more than saving you effort -- tying this behavior to the chef role means you can't screw it up. 
-* Launches the machines in parallel:
-  - using the image name and the availability zone, it determines the appropriate region, image ID, and other implied behavior. 
-  - passes a JSON-encoded user_data hash specifying the machine's chef `node_name` and client key. An appropriately-configured machine image will need no further bootstrapping -- it will connect to the chef server with the appropriate identity and proceed completely unattended.
-* Syncronizes to the cloud provider:
-  - Applies EC2 tags to the machine, making your console intelligible: ![AWS Console screenshot](https://github.com/infochimps-labs/ironfan/raw/version_3/notes/aws_console_screenshot.jpg)
-  - Connects external (EBS) volumes, if any, to the correct mount point -- it uses (and applies) tags to the volumes, so they know which machine to adhere to. If you've manually added volumes, just make sure they're defined correctly in your cluster file and run `knife cluster sync {cluster_name}`; it will paint them with the correct tags.
-  - Associates an elastic IP, if any, to the machine
-* Bootstraps the machine using knife bootstrap