ironfan 4.3.4 → 4.4.0

data/lib/ironfan/provider/ec2/machine.rb

@@ -242,6 +242,14 @@ module Ironfan
             SecurityGroup.recall(group_name).group_id
           end
 
+          description[:iam_server_certificates] = cloud.iam_server_certificates.values.map do |cert|
+            IamServerCertificate.recall(IamServerCertificate.full_name(computer, cert))
+          end.compact.map(&:name)
+
+          description[:elastic_load_balancers] = cloud.elastic_load_balancers.values.map do |elb|
+            ElasticLoadBalancer.recall(ElasticLoadBalancer.full_name(computer, elb))
+          end.compact.map(&:name)
+
           if cloud.flavor_info[:placement_groupable]
             ui.warn "1.3.1 and earlier versions of Fog don't correctly support placement groups, so your nodes will land willy-nilly. We're working on a fix"
             description[:placement] = { 'groupName' => cloud.placement_group.to_s }

data/lib/ironfan/provider/ec2/security_group.rb

@@ -11,7 +11,6 @@ module Ironfan
             :requires_one, :revoke_group_and_owner, :revoke_port_range, :save,
             :symbolize_keys, :vpc_id, :vpc_id=, :wait_for,
           :to => :adaptee
-        field :ensured,        :boolean,       :default => false
 
         def self.shared?()      true;   end
         def self.multiple?()    true;   end
@@ -101,7 +100,7 @@ module Ironfan
           create!(computer)            # Make sure the security groups exist
           security_groups = cloud.security_groups.values
           dsl_groups = security_groups.select do |dsl_group|
-            not (recall? dsl_group or recall(dsl_group.name).ensured) and \
+            not (recall_with_vpc(dsl_group,cloud.vpc)) and \
             not (dsl_group.range_authorizations +
                  dsl_group.group_authorized_by +
                  dsl_group.group_authorized).empty?
@@ -160,11 +159,10 @@ module Ironfan
             return
           end
 
-          begin
+          self.patiently(fog_group.name, Fog::Compute::AWS::Error, :ignore => Proc.new { |e| e.message =~ /InvalidPermission\.Duplicate/ }) do
             fog_group.authorize_port_range(range,options)
-          rescue Fog::Compute::AWS::Error => e      # InvalidPermission.Duplicate
-            Chef::Log.info("ignoring #{e}")
           end
+
         end
       end
 

data/lib/ironfan/requirements.rb

@@ -35,6 +35,8 @@ require 'ironfan/provider/ec2/machine'
 require 'ironfan/provider/ec2/keypair'
 require 'ironfan/provider/ec2/placement_group'
 require 'ironfan/provider/ec2/security_group'
+require 'ironfan/provider/ec2/elastic_load_balancer'
+require 'ironfan/provider/ec2/iam_server_certificate'
 
 require 'ironfan/provider/virtualbox'
 require 'ironfan/provider/virtualbox/machine'

data/notes/Home.md

@@ -0,0 +1,45 @@
+## Overview
+
+Ironfan, the foundation of The Infochimps Platform, is an expressive toolset for constructing scalable, resilient architectures. It works in the cloud, in the data center, and on your laptop, and it makes your system diagram visible and inevitable. Inevitable systems coordinate automatically to interconnect, removing the hassle of manual configuration of connection points (and the associated danger of human error). For more information about Ironfan and the Infochimps Platform, visit [infochimps.com](https://www.infochimps.com).
+
+<a name="getting-started"></a>
+## Getting Started
+
+* [Installation Instructions](https://github.com/infochimps-labs/ironfan/wiki/INSTALL)
+* [Web Walkthrough](https://github.com/infochimps-labs/ironfan/wiki/walkthrough-web)
+* [Ironfan Screencast](http://bit.ly/ironfan-hadoop-in-20-minutes) -- build a Hadoop cluster from scratch in 20 minutes.
+
+<a name="toolset"></a>
+### Tools
+
+Ironfan consists of the following toolset:
+
+* [ironfan-homebase](https://github.com/infochimps-labs/ironfan-homebase): centralizes the cookbooks, roles and clusters. A solid foundation for any chef user.
+* [ironfan gem](https://github.com/infochimps-labs/ironfan):
+  - core models to describe your system diagram with a clean, expressive domain-specific language
+  - knife plugins to orchestrate clusters of machines using simple commands like `knife cluster launch`
+  - logic to coordinate truth among chef server and cloud providers.
+* [ironfan-pantry](https://github.com/infochimps-labs/ironfan-pantry): Our collection of industrial-strength, cloud-ready recipes for Hadoop, HBase, Cassandra, Elasticsearch, Zabbix and more.
+* [silverware cookbook](https://github.com/infochimps-labs/ironfan-homebase/tree/master/cookbooks/silverware): coordinate discovery of services ("list all the machines for `awesome_webapp`, that I might load balance them") and aspects ("list all components that write logs, that I might logrotate them, or that I might monitor the free space on their volumes".
+* [Infochimps Platform](http://www.infochimps.com) -- our scalable enterprise big data platform. Ironfan Enterprise adds dynamic orchestration and zero-configuration logging and monitoring.
+
+<a name="ironfan-way"></a>
+### Ironfan Concepts
+
+* [Core Concepts](https://github.com/infochimps-labs/ironfan/wiki/core_concepts)     -- Components, Announcements, Amenities and more.
+* [Philosophy](https://github.com/infochimps-labs/ironfan/wiki/philosophy)            -- best practices and lessons learned behind the Ironfan Way
+* [Style Guide](https://github.com/infochimps-labs/ironfan/wiki/style_guide)         -- common attribute names, how and when to include other cookbooks, and more
+* [Homebase Layout](https://github.com/infochimps-labs/ironfan/wiki/homebase-layout) -- how this homebase is organized, and why
+
+<a name="documentation"></a>
+### Documentation
+
+* [Index of wiki pages](https://github.com/infochimps-labs/ironfan/wiki/_pages)
+* [ironfan wiki](https://github.com/infochimps-labs/ironfan/wiki): high-level documentation and install instructions
+* [ironfan issues](https://github.com/infochimps-labs/ironfan/issues): bugs, questions and feature requests for *any* part of the Ironfan toolset.
+* [ironfan gem docs](http://rdoc.info/gems/ironfan): rdoc docs for Ironfan
+
+__________________________________________________________________________
+__________________________________________________________________________
+__________________________________________________________________________
+<br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/>

data/notes/INSTALL-cloud_setup.md

@@ -0,0 +1,103 @@
+## Credentials
+
+* make a credentials repo
+  - copy the knife/example-credentials directory
+  - best to not live on github: use a private server and run
+   
+        ``` 
+        repo=ORGANIZATION-credentials ; repodir=/gitrepos/$repo.git ; mkdir -p $repodir ; ( GIT_DIR=$repodir git init --shared=group --bare  && cd $repodir && git --bare update-server-info && chmod a+x hooks/post-update ) 
+        ```
+    
+  - git submodule it into knife as `knife/yourorg-credentials`
+  - or, if somebody has added it,
+  
+        ```
+        git pull
+        git submodule update --init
+        find . -iname '*.pem' -exec chmod og-rw {} \;
+        cp knife/${OLD_CHEF_ORGANIZATION}-credentials/knife-user-${CHEF_USER}.rb knife/${CHEF_ORGANIZATION}-credentials
+        cp knife/${OLD_CHEF_ORGANIZATION}-credentials/${CHEF_USER}.pem knife/${CHEF_ORGANIZATION}-credentials/
+        ```
+    
+* create AWS account
+  - [sign up for AWS + credit card + password]
+  - make IAM users for admins
+  - add your IAM keys into your `{credentials}/knife-user`
+
+* create opscode account
+  - download org keys, put in the credentials repo
+
+## Populate Chef Server
+
+* create `prod` and `dev` environments by using 
+
+  ```
+  knife environment create dev
+  knife environment create prod
+  knife environment create stag
+  knife environment from file environments/stag.json  
+  knife environment from file environments/dev.json
+  knife environment from file environments/prod.json
+  ```
+
+  ```
+  knife cookbook upload --all
+  rake roles 
+  # if you have data bags, do that too
+  ```
+
+## Create Your Initial Machine Boot-Image (AMI)
+
+* Start by launching the burninator cluster: `knife cluster launch --bootstrap --yes burninator-trogdor-0`
+  - You may have to specify the template by adding this an anargument: `--template-file ${CHEF_HOMEBASE}/vendor/ironfan/lib/chef/knife/bootstrap/ubuntu10.04-ironfan.erb`
+  - This template makes the machine auto-connect to the server upon launch and teleports the client-key into the machine.
+  - If this fails, bootstrap separately: `knife cluster bootstrap --yes burninator-trogdor-0`
+
+* Log into the burninator-trogdor and run the script /tmp/burn_ami_prep.sh: `sudo bash /tmp/burn_ami_prep.sh`
+  - You will have to ssh as the ubuntu user and pass in the burninator.pem identity file.
+  - Review the output of this script and ensure the world we have created is sane.
+
+* Once the script has been run:
+  - Exit the machine.
+  - Go to AWS console.
+  - DO NOT stop the machine.
+  - Do "Create Image (EBS AMI)" from the burninator-trogdor instance (may take a while).
+
+* Add the AMI id to your `{credentials}/knife-org.rb` in the `ec2_image_info.merge!` section and create a reference name for the image (e.g ironfan-natty).
+  - Add that reference name to the burninator-village facet in the burninator.rb cluster definition: `cloud.image_name 'ironfan_natty'`
+
+* Launch the burninator-village in order to test your newly created AMI.
+  - The village should launch with no problems, have the correct permissions and be able to complete a chef run: `sudo chef-client`.
+  
+* If all has gone well so far, you may now stop the original burninator: `knife cluster kill burninator-trogdor`
+  - Leave the burninator-village up and stay ssh'ed to assist with the next step.
+
+## Create an NFS
+
+* Make a command/control cluster definition file with an nfs facet (see clusters/demo_cnc.rb).
+  - Make sure specify the `image_name` to be the AMI you've created.
+
+* In the AWS console make yourself a 20GB drive. 
+  - Make sure the availability zone matches the one specified in your cnc_cluster definition file. 
+  - Don't choose a snapshot. 
+  - Set the device name to `/dev/sdh`.
+  - Attach to the burninator-village instance.
+
+* ssh in to burninator-village to format the nfs drive:
+```
+  dev=/dev/xvdh ; name='home_drive' ; sudo umount $dev ; ls -l $dev ; sudo mkfs.xfs $dev ; sudo mkdir /mnt/$name ; sudo mount -t xfs $dev /mnt/$name ; sudo bash -c "echo 'snapshot for $name burned on `date`' > /mnt/$name/vol_info.txt "
+  sudo cp -rp /home/ubuntu /mnt/$name/ubuntu
+  sudo umount /dev/xvdh
+  exit
+```
+* Back in the AWS console, snapshot the volume and name it `{org}-home_drive`. Delete the original volume as it is not needed anymore.
+  - While you're in there, make `{org}-resizable_1gb` a 'Minimum-sized snapshot, resizable -- use `xfs_growfs` to resize after launch' snapshot.
+  
+* Paste the snapshot id into your cnc_cluster definition file. 
+  - ssh into the newly launched cnc_cluster-nfs.
+  - You should restart the machine via the AWS console (may or may not be necessary, do anyway).
+
+* Manipulate security groups
+  - `nfs_server` group should open all UDP ports and all TCP ports to `nfs_client` group
+
+* Change /etc/ssh/sshd_config to be passwordful and restart the ssh service

data/notes/INSTALL.md

@@ -0,0 +1,134 @@
+# Ironfan Installation Instructions
+
+First of all, every Chef installation needs a Chef Homebase. Chef Homebase is the place where cookbooks, roles, config files and other artifacts for managing systems with Chef will live. Store this homebase in a version control system such as Git and treat it like source code.
+
+## Conventions
+
+In all of the below,
+
+* `{homebase}`: is the directory that holds your Chef cookbooks, roles and so forth. For example, this file is in `{homebase}/README.md`.
+* `{username}`: identifies your personal Chef client name: the thing you use to log into the Chef WebUI.
+* `{organization}`: identifies the credentials set and cloud settings to use.  If your Chef server is on the Opscode platform (Try it! It's super-easy), use your organization name (the last segment of your chef_server url). If not, use an identifier you deem sensible.
+
+<a name="initial_install"></a>
+## Install Ironfan's Gem and Homebase
+
+_Before you begin, you may wish to fork homebase repo, as you'll be making changes to personalize it for your platform that you may want to share with teammates. If you do so, replace all references to infochimps-labs/ironfan-homebase with your fork's path._
+
+1. Install system prerequisites (libXML and libXSLT). The following works under Debian/Ubuntu:
+
+        sudo apt-get install libxml2-dev libxslt1-dev
+
+1. Install the Ironfan gem (you may need to use `sudo`):
+
+        gem install ironfan
+
+1. Clone the repo. It will produce the directory we will call `homebase` from now on:
+
+        git clone https://github.com/infochimps-labs/ironfan-homebase homebase
+        cd homebase
+        bundle install
+        git submodule update --init
+        git submodule foreach git checkout master
+
+<a name="knife-configuration"></a>
+## Configure Knife and Add Credentials
+
+Ironfan expands out the traditional singular [knife.rb](http://wiki.opscode.com/display/chef/Knife#Knife-ConfiguringYourSystemForKnife) into several components. This modularity allows for better management of sensitive shared credentials, personal credentials, and organization-wide configuration.
+
+### Set up 
+
+_Note_: If your local username differs from your Opscode Chef username, then you should `export CHEF_USER={username}` (eg from your `.bashrc`) before you run any knife commands.
+
+So that Knife finds its configuration files, symlink the `{homebase}/knife` directory (the one holding this file) to be your `~/.chef` folder.
+
+        cd {homebase} 
+        ln -sni $CHEF_HOMEBASE/knife ~/.chef
+
+<a name="credentials"></a>
+### Credentials Directory
+
+All the keys and settings specific to your organization are held in a directory named `credentials/`, versioned independently of the homebase.
+
+To set up your credentials directory, visit `{homebase}/knife` and duplicate the example, naming it `credentials`:
+
+        cd $CHEF_HOMEBASE/knife 
+        rm credentials
+        cp -a example-credentials credentials
+        cd credentials
+        git init ; git add .
+        git commit -m "New credentials universe for $CHEF_ORGANIZATION" .
+
+You will likely want to store the credentials in another remote repository. We recommend erring on the side of caution in its hosting. Setting that up is outside the scope of this guide, but there [good external resources](http://book.git-scm.com/3_distributed_workflows.html) available to get you started.
+
+<a name="download"></a>
+### Download Cloud Credentials
+
+You will need to obtain user keys from your cloud providers. Your AWS access keys can be obtained from [Amazon IAM](https://console.aws.amazon.com/iam/home):
+
+![Reset AWS User Key](https://github.com/infochimps-labs/ironfan/wiki/aws_user_key.png)
+
+__________________________________________________________________________
+
+Your Opscode user key can be obtained from the [Opscode Password settings](https://www.opscode.com/account/password) console:
+
+![Reset Opscode User Key](https://github.com/infochimps-labs/ironfan/wiki/opscode_user_key.png)
+
+__________________________________________________________________________
+
+Your Opscode organization validator key can be obtained from the [Opscode Organization management](https://manage.opscode.com/organizations) console, by choosing the `Regenerate validation key` link:
+
+![Reset Opscode Organization Key](https://github.com/infochimps-labs/ironfan/wiki/opscode_org_key.png)
+
+__________________________________________________________________________
+
+
+<a name="org"></a>
+### User / Organization-specific config
+
+Edit the following in your new `credentials`:
+
+* Organization-specific settings are in `knife/credentials/knife-org.rb`:
+  - _organization_:          Your organization name 
+  - _chef server url_:       Edit the lines for your `chef_server_url` and `validator`. _Note_: If you are an Opscode platform user, you can skip this step -- your `chef_server_url` defaults to `https://api.opscode.com/organizations/#{organization}` and your validator to `{organization}-validator.pem`.
+  - Cloud-specific settings: if you are targeting a cloud provider, add account information and configuration here. 
+
+* User-specific settings are in `knife/credentials/knife-user-{username}.rb`. (You can duplicate and rename the one in `knife/example-credentials/knife-user-example.rb`). For example, if you're using Amazon EC2 you should set your access keys:
+
+          Chef::Config.knife[:aws_access_key_id]      = "XXXX"
+          Chef::Config.knife[:aws_secret_access_key]  = "XXXX"
+          Chef::Config.knife[:aws_account_id]         = "XXXX"
+        
+* Chef user key is in `{credentials_path}/{username}.pem`
+
+* Organization validator key in `{credentials_path}/{organization}-validator.pem`
+
+* If you have existing Amazon machines, place their keypairs in `{credentials_path}/ec2_keys`. Ironfan will also automatically populate this with new keys as new clusters are created. Commit the resulting keys back to the credentials repo to share them with your teammates, or they will be unable to make certain calls against the resulting architecture.
+
+<a name="go_speed_racer"></a>
+## Try it out
+
+You should now be able to use Knife to control your clusters:
+
+        $ knife cluster list
+        +--------------------+---------------------------------------------------+ 
+        | cluster            | path                                              |
+        +--------------------+---------------------------------------------------+
+        | burninator         | /cloud/clusters/burninator.rb                     |
+        | el_ridiculoso      | /cloud/clusters/el_ridiculoso.rb                  |
+        | elasticsearch_demo | /cloud/clusters/elasticsearch_demo.rb             |
+        | hadoop_demo        | /cloud/clusters/hadoop_demo.rb                    |
+        | sandbox            | /cloud/clusters/sandbox.rb                        |
+        +--------------------+---------------------------------------------------+
+
+Launching a cluster in the cloud should now be this easy!
+
+        knife cluster launch sandbox-simple --bootstrap
+
+## Next
+
+The README file in each of the subdirectories for more information about what goes in those directories. If you are bored of reading, go customize one of the files in the 'clusters/ directory'. Or, if you're a fan of ridiculous things and have ever pondered how many things you can fit in one box, launch el_ridiculoso:. It contains every single recipe we have ever made stacked on top of one another.
+
+        knife cluster launch el_ridiculoso-gordo --bootstrap
+
+For more information about configuring Knife, see the [Knife documentation](http://wiki.opscode.com/display/chef/knife).

data/notes/Ironfan-Roadmap.md

@@ -0,0 +1,70 @@
+#  Ironfan Roadmap
+
+##  Summary
+
+- I. Ironfan-ci
+- II. DSL Undercarriage / OpenStack
+- III. Cookbook Updates
+- IV. Keys Handling
+- V. Silverware Update
+- VI. Ironfan Knife
+- VII. Orchestration
+
+## Detailed Roadmap
+
+###  Ironfan-CI (I)
+Jenkins on laptop (Done) 
+Jenkins runs VM sees output of test 
+Translate announcement to cucumber lines 
+Implement as necessary new Cuken tests 
+
+### Openstack / Multi-cloud (II)
+* Learn Openstack
+* (get accts @ a couple providers + eucalytus) 
+* Fog (library we use, ec2 only?) compatibility with some tear-out
+* Depends on DSL Object above
+* Move stuff in Fog_layer to be methods on Cloud Object 
+* cloud(:ec2, ‘us_east’) do
+* cores 1
+* end
+* Cloud Statement is just a layer, not its own object 
+* (Cloud loses to everything else, we think)
+
+### Ironfanize Rest of Cookbooks (III)
+* Debugging and updating exercise. 
+* Ironfan-ci accelerates
+* Zabbix
+* MySql
+* Map to order of operations
+* Clean Separation of tight-bound services
+* Resque’s Redis
+* Flume’s Zookeeper
+
+### DSL Object / Librarification (Mix)
+* New DSL Object (II)
+* Unify Models in Silverware/lib & Ironfan/lib (Birth of the Ironfan API Interface) (II)
+* Birth of the Ironfan API Interface (V)
+* Clean up Announcment Interface (framework) (V)
+* Merge Volume (VIII)
+* Actual Model for a dummy node (VIII)
+* Refactor deploy code across cookbooks (III)
+*  Discovers component is an aspect endowed upon a component when it discovers another component to find out what depends on a service  (V)
+* Key Databag Rollout (IV)
+
+### Ironfan-knife (VI)
+* Separate SSH user as “Machine” or “Me”
+* Better Error Messages 
+* Verbose vs. Sustained
+* Clearout Issues
+* Refactor Cluster into definitions - “Stacks” (Roles that are smarter)
+* Role Replacement
+* (Design doc forthcoming)
+
+### Orchestration (VI/VII)
+* System diagram /reporting (VII)
+* Ticketed Worker Queue to run steps (bring up a Hadoop cluster, for instance) (VII)
+* Rundeck? Juju? (VII)
+* Activity stream (VII)
+* Helpers (VII)
+* API Frontend (VII)
+* Richer Slice Queries (VI)

data/notes/advanced-superpowers.md

@@ -0,0 +1,16 @@
+### Set up Knife on your local machine, and a Chef Server in the cloud
+
+If you already have a working chef installation you can skip this section.
+
+To get started with knife and chef, follow the "Chef Quickstart,":http://wiki.opscode.com/display/chef/Quick+Start We use the hosted chef service and are very happy, but there are instructions on the wiki to set up a chef server too. Stop when you get to "Bootstrap the Ubuntu system" -- cluster chef is going to make that much easier.
+
+* [Launch Cloud Instances with Knife](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife)
+* [EC2 Bootstrap Fast Start Guide](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide)
+
+### Auto-vivifying machines (no bootstrap required!)
+
+On EC2, you can make a machine that auto-vivifies -- no bootstrap necessary. Burn an AMI that has the `config/client.rb` file in /etc/chef/client.rb. It will use the ec2 userdata (passed in by knife) to realize its purpose in life, its identity, and the chef server to connect to; everything happens automagically from there. No parallel ssh required!
+
+### EBS Volumes
+
+Define a `snapshot_id` for your volumes, and set `create_at_launch` true.

data/notes/cookbook-versioning.md

@@ -0,0 +1,11 @@
+Cookbook Versioning and Tracking
+================================
+
+@temujin9 please complete and correct
+
+* git tag labels the *release* of a cookbook version: tag 'cookbooks-elasticsearch-3.1.7' denotes the *last* commit to that tag.
+* The next commit will be the one that bumps the version number: the `metadata.rb` will then read '3.1.8'.
+
+Periodically, we will release a 'gold' version set and push those to the opscode cookbook community site.
+
+*