ironfan 3.1.7 → 3.2.2

data/notes/Home.md

@@ -1,45 +0,0 @@
-## Overview
-
-Ironfan, the foundation of The Infochimps Platform, is an expressive toolset for constructing scalable, resilient architectures. It works in the cloud, in the data center, and on your laptop, and it makes your system diagram visible and inevitable. Inevitable systems coordinate automatically to interconnect, removing the hassle of manual configuration of connection points (and the associated danger of human error). For more information about Ironfan and the Infochimps Platform, visit [infochimps.com](https://www.infochimps.com).
-
-<a name="getting-started"></a>
-## Getting Started
-
-* [Installation Instructions](https://github.com/infochimps-labs/ironfan/wiki/INSTALL)
-* [Web Walkthrough](https://github.com/infochimps-labs/ironfan/wiki/walkthrough-web)
-* [Ironfan Screencast](http://bit.ly/ironfan-hadoop-in-20-minutes) -- build a Hadoop cluster from scratch in 20 minutes.
-
-<a name="toolset"></a>
-### Tools
-
-Ironfan consists of the following toolset:
-
-* [ironfan-homebase](https://github.com/infochimps-labs/ironfan-homebase): centralizes the cookbooks, roles and clusters. A solid foundation for any chef user.
-* [ironfan gem](https://github.com/infochimps-labs/ironfan):
-  - core models to describe your system diagram with a clean, expressive domain-specific language
-  - knife plugins to orchestrate clusters of machines using simple commands like `knife cluster launch`
-  - logic to coordinate truth among chef server and cloud providers.
-* [ironfan-pantry](https://github.com/infochimps-labs/ironfan-pantry): Our collection of industrial-strength, cloud-ready recipes for Hadoop, HBase, Cassandra, Elasticsearch, Zabbix and more.
-* [silverware cookbook](https://github.com/infochimps-labs/ironfan-homebase/tree/master/cookbooks/silverware): coordinate discovery of services ("list all the machines for `awesome_webapp`, that I might load balance them") and aspects ("list all components that write logs, that I might logrotate them, or that I might monitor the free space on their volumes".
-* [Infochimps Platform](http://www.infochimps.com) -- our scalable enterprise big data platform. Ironfan Enterprise adds dynamic orchestration and zero-configuration logging and monitoring.
-
-<a name="ironfan-way"></a>
-### Ironfan Concepts
-
-* [Core Concepts](https://github.com/infochimps-labs/ironfan/wiki/core_concepts)     -- Components, Announcements, Amenities and more.
-* [Philosophy](https://github.com/infochimps-labs/ironfan/wiki/philosophy)            -- best practices and lessons learned behind the Ironfan Way
-* [Style Guide](https://github.com/infochimps-labs/ironfan/wiki/style_guide)         -- common attribute names, how and when to include other cookbooks, and more
-* [Homebase Layout](https://github.com/infochimps-labs/ironfan/wiki/homebase-layout) -- how this homebase is organized, and why
-
-<a name="documentation"></a>
-### Documentation
-
-* [Index of wiki pages](https://github.com/infochimps-labs/ironfan/wiki/_pages)
-* [ironfan wiki](https://github.com/infochimps-labs/ironfan/wiki): high-level documentation and install instructions
-* [ironfan issues](https://github.com/infochimps-labs/ironfan/issues): bugs, questions and feature requests for *any* part of the Ironfan toolset.
-* [ironfan gem docs](http://rdoc.info/gems/ironfan): rdoc docs for Ironfan
-
-__________________________________________________________________________
-__________________________________________________________________________
-__________________________________________________________________________
-<br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/>

data/notes/INSTALL-cloud_setup.md

@@ -1,103 +0,0 @@
-## Credentials
-
-* make a credentials repo
-  - copy the knife/example-credentials directory
-  - best to not live on github: use a private server and run
-   
-        ``` 
-        repo=ORGANIZATION-credentials ; repodir=/gitrepos/$repo.git ; mkdir -p $repodir ; ( GIT_DIR=$repodir git init --shared=group --bare  && cd $repodir && git --bare update-server-info && chmod a+x hooks/post-update ) 
-        ```
-    
-  - git submodule it into knife as `knife/yourorg-credentials`
-  - or, if somebody has added it,
-  
-        ```
-        git pull
-        git submodule update --init
-        find . -iname '*.pem' -exec chmod og-rw {} \;
-        cp knife/${OLD_CHEF_ORGANIZATION}-credentials/knife-user-${CHEF_USER}.rb knife/${CHEF_ORGANIZATION}-credentials
-        cp knife/${OLD_CHEF_ORGANIZATION}-credentials/${CHEF_USER}.pem knife/${CHEF_ORGANIZATION}-credentials/
-        ```
-    
-* create AWS account
-  - [sign up for AWS + credit card + password]
-  - make IAM users for admins
-  - add your IAM keys into your `{credentials}/knife-user`
-
-* create opscode account
-  - download org keys, put in the credentials repo
-
-## Populate Chef Server
-
-* create `prod` and `dev` environments by using 
-
-  ```
-  knife environment create dev
-  knife environment create prod
-  knife environment create stag
-  knife environment from file environments/stag.json  
-  knife environment from file environments/dev.json
-  knife environment from file environments/prod.json
-  ```
-
-  ```
-  knife cookbook upload --all
-  rake roles 
-  # if you have data bags, do that too
-  ```
-
-## Create Your Initial Machine Boot-Image (AMI)
-
-* Start by launching the burninator cluster: `knife cluster launch --bootstrap --yes burninator-trogdor-0`
-  - You may have to specify the template by adding this an anargument: `--template-file ${CHEF_HOMEBASE}/vendor/ironfan/lib/chef/knife/bootstrap/ubuntu10.04-ironfan.erb`
-  - This template makes the machine auto-connect to the server upon launch and teleports the client-key into the machine.
-  - If this fails, bootstrap separately: `knife cluster bootstrap --yes burninator-trogdor-0`
-
-* Log into the burninator-trogdor and run the script /tmp/burn_ami_prep.sh: `sudo bash /tmp/burn_ami_prep.sh`
-  - You will have to ssh as the ubuntu user and pass in the burninator.pem identity file.
-  - Review the output of this script and ensure the world we have created is sane.
-
-* Once the script has been run:
-  - Exit the machine.
-  - Go to AWS console.
-  - DO NOT stop the machine.
-  - Do "Create Image (EBS AMI)" from the burninator-trogdor instance (may take a while).
-
-* Add the AMI id to your `{credentials}/knife-org.rb` in the `ec2_image_info.merge!` section and create a reference name for the image (e.g ironfan-natty).
-  - Add that reference name to the burninator-village facet in the burninator.rb cluster definition: `cloud.image_name 'ironfan_natty'`
-
-* Launch the burninator-village in order to test your newly created AMI.
-  - The village should launch with no problems, have the correct permissions and be able to complete a chef run: `sudo chef-client`.
-  
-* If all has gone well so far, you may now stop the original burninator: `knife cluster kill burninator-trogdor`
-  - Leave the burninator-village up and stay ssh'ed to assist with the next step.
-
-## Create an NFS
-
-* Make a command/control cluster definition file with an nfs facet (see clusters/demo_cnc.rb).
-  - Make sure specify the `image_name` to be the AMI you've created.
-
-* In the AWS console make yourself a 20GB drive. 
-  - Make sure the availability zone matches the one specified in your cnc_cluster definition file. 
-  - Don't choose a snapshot. 
-  - Set the device name to `/dev/sdh`.
-  - Attach to the burninator-village instance.
-
-* ssh in to burninator-village to format the nfs drive:
-```
-  dev=/dev/xvdh ; name='home_drive' ; sudo umount $dev ; ls -l $dev ; sudo mkfs.xfs $dev ; sudo mkdir /mnt/$name ; sudo mount -t xfs $dev /mnt/$name ; sudo bash -c "echo 'snapshot for $name burned on `date`' > /mnt/$name/vol_info.txt "
-  sudo cp -rp /home/ubuntu /mnt/$name/ubuntu
-  sudo umount /dev/xvdh
-  exit
-```
-* Back in the AWS console, snapshot the volume and name it `{org}-home_drive`. Delete the original volume as it is not needed anymore.
-  - While you're in there, make `{org}-resizable_1gb` a 'Minimum-sized snapshot, resizable -- use `xfs_growfs` to resize after launch' snapshot.
-  
-* Paste the snapshot id into your cnc_cluster definition file. 
-  - ssh into the newly launched cnc_cluster-nfs.
-  - You should restart the machine via the AWS console (may or may not be necessary, do anyway).
-
-* Manipulate security groups
-  - `nfs_server` group should open all UDP ports and all TCP ports to `nfs_client` group
-
-* Change /etc/ssh/sshd_config to be passwordful and restart the ssh service

data/notes/INSTALL.md

@@ -1,134 +0,0 @@
-# Ironfan Installation Instructions
-
-First of all, every Chef installation needs a Chef Homebase. Chef Homebase is the place where cookbooks, roles, config files and other artifacts for managing systems with Chef will live. Store this homebase in a version control system such as Git and treat it like source code.
-
-## Conventions
-
-In all of the below,
-
-* `{homebase}`: is the directory that holds your Chef cookbooks, roles and so forth. For example, this file is in `{homebase}/README.md`.
-* `{username}`: identifies your personal Chef client name: the thing you use to log into the Chef WebUI.
-* `{organization}`: identifies the credentials set and cloud settings to use.  If your Chef server is on the Opscode platform (Try it! It's super-easy), use your organization name (the last segment of your chef_server url). If not, use an identifier you deem sensible.
-
-<a name="initial_install"></a>
-## Install Ironfan's Gem and Homebase
-
-_Before you begin, you may wish to fork homebase repo, as you'll be making changes to personalize it for your platform that you may want to share with teammates. If you do so, replace all references to infochimps-labs/ironfan-homebase with your fork's path._
-
-1. Install system prerequisites (libXML and libXSLT). The following works under Debian/Ubuntu:
-
-        sudo apt-get install libxml2-dev libxslt1-dev
-
-1. Install the Ironfan gem (you may need to use `sudo`):
-
-        gem install ironfan
-
-1. Clone the repo. It will produce the directory we will call `homebase` from now on:
-
-        git clone https://github.com/infochimps-labs/ironfan-homebase homebase
-        cd homebase
-        bundle install
-        git submodule update --init
-        git submodule foreach git checkout master
-
-<a name="knife-configuration"></a>
-## Configure Knife and Add Credentials
-
-Ironfan expands out the traditional singular [knife.rb](http://wiki.opscode.com/display/chef/Knife#Knife-ConfiguringYourSystemForKnife) into several components. This modularity allows for better management of sensitive shared credentials, personal credentials, and organization-wide configuration.
-
-### Set up 
-
-_Note_: If your local username differs from your Opscode Chef username, then you should `export CHEF_USER={username}` (eg from your `.bashrc`) before you run any knife commands.
-
-So that Knife finds its configuration files, symlink the `{homebase}/knife` directory (the one holding this file) to be your `~/.chef` folder.
-
-        cd {homebase} 
-        ln -sni $CHEF_HOMEBASE/knife ~/.chef
-
-<a name="credentials"></a>
-### Credentials Directory
-
-All the keys and settings specific to your organization are held in a directory named `credentials/`, versioned independently of the homebase.
-
-To set up your credentials directory, visit `{homebase}/knife` and duplicate the example, naming it `credentials`:
-
-        cd $CHEF_HOMEBASE/knife 
-        rm credentials
-        cp -a example-credentials credentials
-        cd credentials
-        git init ; git add .
-        git commit -m "New credentials universe for $CHEF_ORGANIZATION" .
-
-You will likely want to store the credentials in another remote repository. We recommend erring on the side of caution in its hosting. Setting that up is outside the scope of this guide, but there [good external resources](http://book.git-scm.com/3_distributed_workflows.html) available to get you started.
-
-<a name="download"></a>
-### Download Cloud Credentials
-
-You will need to obtain user keys from your cloud providers. Your AWS access keys can be obtained from [Amazon IAM](https://console.aws.amazon.com/iam/home):
-
-![Reset AWS User Key](https://github.com/infochimps-labs/ironfan/wiki/aws_user_key.png)
-
-__________________________________________________________________________
-
-Your Opscode user key can be obtained from the [Opscode Password settings](https://www.opscode.com/account/password) console:
-
-![Reset Opscode User Key](https://github.com/infochimps-labs/ironfan/wiki/opscode_user_key.png)
-
-__________________________________________________________________________
-
-Your Opscode organization validator key can be obtained from the [Opscode Organization management](https://manage.opscode.com/organizations) console, by choosing the `Regenerate validation key` link:
-
-![Reset Opscode Organization Key](https://github.com/infochimps-labs/ironfan/wiki/opscode_org_key.png)
-
-__________________________________________________________________________
-
-
-<a name="org"></a>
-### User / Organization-specific config
-
-Edit the following in your new `credentials`:
-
-* Organization-specific settings are in `knife/credentials/knife-org.rb`:
-  - _organization_:          Your organization name 
-  - _chef server url_:       Edit the lines for your `chef_server_url` and `validator`. _Note_: If you are an Opscode platform user, you can skip this step -- your `chef_server_url` defaults to `https://api.opscode.com/organizations/#{organization}` and your validator to `{organization}-validator.pem`.
-  - Cloud-specific settings: if you are targeting a cloud provider, add account information and configuration here. 
-
-* User-specific settings are in `knife/credentials/knife-user-{username}.rb`. (You can duplicate and rename the one in `knife/example-credentials/knife-user-example.rb`). For example, if you're using Amazon EC2 you should set your access keys:
-
-          Chef::Config.knife[:aws_access_key_id]      = "XXXX"
-          Chef::Config.knife[:aws_secret_access_key]  = "XXXX"
-          Chef::Config.knife[:aws_account_id]         = "XXXX"
-        
-* Chef user key is in `{credentials_path}/{username}.pem`
-
-* Organization validator key in `{credentials_path}/{organization}-validator.pem`
-
-* If you have existing Amazon machines, place their keypairs in `{credentials_path}/ec2_keys`. Ironfan will also automatically populate this with new keys as new clusters are created. Commit the resulting keys back to the credentials repo to share them with your teammates, or they will be unable to make certain calls against the resulting architecture.
-
-<a name="go_speed_racer"></a>
-## Try it out
-
-You should now be able to use Knife to control your clusters:
-
-        $ knife cluster list
-        +--------------------+---------------------------------------------------+ 
-        | cluster            | path                                              |
-        +--------------------+---------------------------------------------------+
-        | burninator         | /cloud/clusters/burninator.rb                     |
-        | el_ridiculoso      | /cloud/clusters/el_ridiculoso.rb                  |
-        | elasticsearch_demo | /cloud/clusters/elasticsearch_demo.rb             |
-        | hadoop_demo        | /cloud/clusters/hadoop_demo.rb                    |
-        | sandbox            | /cloud/clusters/sandbox.rb                        |
-        +--------------------+---------------------------------------------------+
-
-Launching a cluster in the cloud should now be this easy!
-
-        knife cluster launch sandbox-simple --bootstrap
-
-## Next
-
-The README file in each of the subdirectories for more information about what goes in those directories. If you are bored of reading, go customize one of the files in the 'clusters/ directory'. Or, if you're a fan of ridiculous things and have ever pondered how many things you can fit in one box, launch el_ridiculoso:. It contains every single recipe we have ever made stacked on top of one another.
-
-        knife cluster launch el_ridiculoso-gordo --bootstrap
-
-For more information about configuring Knife, see the [Knife documentation](http://wiki.opscode.com/display/chef/knife).

data/notes/Ironfan-Roadmap.md

@@ -1,70 +0,0 @@
-#  Ironfan Roadmap
-
-##  Summary
-
-- I. Ironfan-ci
-- II. DSL Undercarriage / OpenStack
-- III. Cookbook Updates
-- IV. Keys Handling
-- V. Silverware Update
-- VI. Ironfan Knife
-- VII. Orchestration
-
-## Detailed Roadmap
-
-###  Ironfan-CI (I)
-Jenkins on laptop (Done) 
-Jenkins runs VM sees output of test 
-Translate announcement to cucumber lines 
-Implement as necessary new Cuken tests 
-
-### Openstack / Multi-cloud (II)
-* Learn Openstack
-* (get accts @ a couple providers + eucalytus) 
-* Fog (library we use, ec2 only?) compatibility with some tear-out
-* Depends on DSL Object above
-* Move stuff in Fog_layer to be methods on Cloud Object 
-* cloud(:ec2, ‘us_east’) do
-* cores 1
-* end
-* Cloud Statement is just a layer, not its own object 
-* (Cloud loses to everything else, we think)
-
-### Ironfanize Rest of Cookbooks (III)
-* Debugging and updating exercise. 
-* Ironfan-ci accelerates
-* Zabbix
-* MySql
-* Map to order of operations
-* Clean Separation of tight-bound services
-* Resque’s Redis
-* Flume’s Zookeeper
-
-### DSL Object / Librarification (Mix)
-* New DSL Object (II)
-* Unify Models in Silverware/lib & Ironfan/lib (Birth of the Ironfan API Interface) (II)
-* Birth of the Ironfan API Interface (V)
-* Clean up Announcment Interface (framework) (V)
-* Merge Volume (VIII)
-* Actual Model for a dummy node (VIII)
-* Refactor deploy code across cookbooks (III)
-*  Discovers component is an aspect endowed upon a component when it discovers another component to find out what depends on a service  (V)
-* Key Databag Rollout (IV)
-
-### Ironfan-knife (VI)
-* Separate SSH user as “Machine” or “Me”
-* Better Error Messages 
-* Verbose vs. Sustained
-* Clearout Issues
-* Refactor Cluster into definitions - “Stacks” (Roles that are smarter)
-* Role Replacement
-* (Design doc forthcoming)
-
-### Orchestration (VI/VII)
-* System diagram /reporting (VII)
-* Ticketed Worker Queue to run steps (bring up a Hadoop cluster, for instance) (VII)
-* Rundeck? Juju? (VII)
-* Activity stream (VII)
-* Helpers (VII)
-* API Frontend (VII)
-* Richer Slice Queries (VI)

data/notes/advanced-superpowers.md

@@ -1,16 +0,0 @@
-### Set up Knife on your local machine, and a Chef Server in the cloud
-
-If you already have a working chef installation you can skip this section.
-
-To get started with knife and chef, follow the "Chef Quickstart,":http://wiki.opscode.com/display/chef/Quick+Start We use the hosted chef service and are very happy, but there are instructions on the wiki to set up a chef server too. Stop when you get to "Bootstrap the Ubuntu system" -- cluster chef is going to make that much easier.
-
-* [Launch Cloud Instances with Knife](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife)
-* [EC2 Bootstrap Fast Start Guide](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide)
-
-### Auto-vivifying machines (no bootstrap required!)
-
-On EC2, you can make a machine that auto-vivifies -- no bootstrap necessary. Burn an AMI that has the `config/client.rb` file in /etc/chef/client.rb. It will use the ec2 userdata (passed in by knife) to realize its purpose in life, its identity, and the chef server to connect to; everything happens automagically from there. No parallel ssh required!
-
-### EBS Volumes
-
-Define a `snapshot_id` for your volumes, and set `create_at_launch` true.

data/notes/cookbook-versioning.md

@@ -1,11 +0,0 @@
-Cookbook Versioning and Tracking
-================================
-
-@temujin9 please complete and correct
-
-* git tag labels the *release* of a cookbook version: tag 'cookbooks-elasticsearch-3.1.7' denotes the *last* commit to that tag.
-* The next commit will be the one that bumps the version number: the `metadata.rb` will then read '3.1.8'.
-
-Periodically, we will release a 'gold' version set and push those to the opscode cookbook community site.
-
-* 

data/notes/core_concepts.md

@@ -1,200 +0,0 @@
-# Ironfan Core Concepts
-
-<a name="TOC"></a>
-
-* [Build your architecture from clusters of cooperating machines](#clusters)
-
-* [Decoupled *Components* connect](#components)
-
-* [Components *Announce* their capabilities](#announcements)
-
-* [Announcements enable *Service Discovery*](#discovery)
-
-* [Components announce cross-cutting *Aspects*](#aspects)
-
-* [Aspects enable zero-conf *Amenities*](#amenities) - 
-
-* [Announcements effectively define a component's *Contract*](#contract)
-
-* [Contracts enable zero-conf *specification testing*](#specs)
-
-* [Specs + monitoring enable zero-conf *integration testing*](#ci)
-
-* [Systems *Bind* to provisioned resources](#binding)
-
-* [Binding declarations enable *Resource Sharing*](#resource-sharing)
-
-<a name="overview"></a>
-### Overview
-
-Ironfan is your system diagram come to life. In ironfan, you use Chef to assemble and configure components on each machine. Ironfan assembles those machines into clusters -- a group of machines united to provide an important service. For example, at Infochimps one cluster of machines serves the webpages for infochimps.com; another consists only of elasticsearch machines to power our API; and another runs the lightweight goliath proxies that implement our API. Our data scientists are able to spin up and shut down terabyte-scale hadoop clusters in minutes. All this is supported by an Ops team of one -- who spends most of his time hacking on Ironfan. 
-
-The powerful abstractions provided by Chef and Ironfan enables an autowiring system diagram, inevitable best practices in the form of "amenities", and a readable, testable contract for each component in the stack.
-
-<a name="clusters"></a><a name="facets"></a>
-### Clusters and Facets
-
-A `cluster`, as mentioned, groups a set of machines around a common purpose. Within that cluster, you define `facet`s: a set of servers with identical components (and nearly identical configuration). 
-
-For example, a typical web stack cluster might have these facets:
-
-* `webnode`s: nginx reverse-proxies requests to a pool of unicorns running Rails
-* `mysql`: one or many MySQL servers, with attached persistent storage
-* `qmaster`s: a redis DB and resque front end to distribute batch-processing tasks
-* `qworkers`s: resque worker processes
-
-<a name="components"></a>
-### Components
-
-As you can see, the details of a machine largely follow from the list its `component`s: `mysql_server`, `resque_dashboard`, and so forth. What's a component? If you would draw it in a box on your system diagram, want to discover it from elsewhere, or it it forms part of the contract for your machine, it's a component. 
-
-Some systems have more than one component: the `ganglia` monitoring system has a component named `agent` to gather operating metrics, and a component named `master` to aggregate those metrics. 
-
-Those examples all describe daemon processes that listen on ports, but component is more general that that -- it's any isolatable piece of functionality that is interesting to an outside consumer. Here is a set of example systems we'll refer to repeatedly:
-
-* *Ganglia*, a distributed system monitoring tool. The `agent` components gather and exchange system metrics, and the `master` component aggregates them. A basic setup would run the `master` component on a single machine, and the `agent` component on many machines (including the master). In order to work, the master must discover all agents, and each agent must discover the master.
-
-* *Elasticsearch* is a powerful distributed document database. A basic setup runs a single `server` component on each machine. Elasticsearch handles discovery, but needs a stable subset of them to declare as discovery `seed`s.
-
-* *Nginx* is a fast, lightweight webserver (similar to apache). Its `server` component can proxy web requests for one or many web apps. Those apps register a `site` component, which defines the receiving address (public/private/local), how the app connects to nginx (socket, port, files).
-
-* *Pig* is a Big Data analysis tool that works with Hadoop, Elasticsearch and more. It provides an executable, and imports jars from hadoop, elasticsearch and others.
-
-<a name="announcements"></a>
-### Components *Announce* their capabilities
-
-Notice the recurring patterns: *capabilities* (serve webpages, execute script, send metrics, answer queries), *handles* (ip+port, jars, swarm), *aspects* (ports, daemons, logs, files, dashboards).
-
-The Silverware cookbook lets your services `announce` their capabilities and `discover` other resources.
-
-Chef cookbooks describe the related components that form a system. You should always have a recipe, separate from the `default` recipe, that clearly corresponds to the component: the `ganglia` cookbook has `master` and `agent` recipes; the `pig` cookbook has `install_from_package` and `install_from_release` recipes. Those recipes are grouped together into Chef roles that encapsulate the component: the `elasticsearch_server` role calls the recipes to install the software, start the daemon process, and write the config files, each in the correct order.
-
-Cookbooks do *not* bake in assumptions about their scale or about the machine they're on. The same Elasticsearch cookbook can deploy a tiny little search box to sit next to a web app, or one server in a distributed terabyte scale database.
-
-<a name="discovery"></a>
-### Announcements enable *Service Discovery*
-
-The `discover` and `discover_all` connect decoupled components. Your systems
-
-* Don't care whether the discovered components are on the same machine, different machines, or a remote data center.
-* Don't care about the number of underlying machines -- the whole thing might run on your laptop while developing, across a handful of nodes in staging, and on dozens of nodes in production.
-* Don't necessarily care about the actual system -- your load balancer doesn't care whether it's nginx or apache or anything else, it just wants to discover the correct set of `webnode`s.
-
-<a name="aspects"></a>
-### Components announce cross-cutting *Aspects*
-
-Besides the component's capabilities, the announcement also describes its aspects: cross-cutting attributes common to many components.
-
-* **log**:         write data to a log file.
-* **daemon**:      long-running process. Can specify run state, resource bounds,  etc.
-* **port**:        serves data over a port. Can specify the protocol, performance expectations, etc.
-* **dashboard**:   HTML, JMX, etc -- internal component metrics and control
-* **executable**:  executes scripts
-* **export**:      libraries, `jar`s, `conf` files, etc 
-* **consumes**:    registered whenever you `discover` another component
-
-<a name="amenities"></a>
-### Aspects enable zero-conf *Amenities* 
-
-Typically, consumers discover their provider, and the provider is unconcerned with which consumers it attends to. Ironfan lets you invert this pattern: decoupled `amenities` find components they can cater to.
-
-* A log aspect would enable the following amenities
-  - `logrotated` to intelligently manage its logs
-  - `flume` to archive logs to a predictable location
-  - If the log is known to be an apache web log, a flume decorator can track rate and duration of requests and errors.
-* A port aspect would enable
-  - zeroconf configuration of firewall and security groups
-  - remote monitors to regularly pinging the port for uptime and latency 
-  - and pings the interfaces that it should *not* appear on to ensure the firewall is in place?
-
-<a name="contracts"></a>
-### Announcements effectively define a component's *Contract*
-
-The announcements that components make don’t just facilitate discovery. In a larger sense, they describe the external contract for the component.
-
-When `nginx` announces that it listens on `node[:nginx][:http_port] = 80`, it is promising a capability (namely, that http requests to that port return certain results). When elasticsearch announces that it runs the `elasticsearch` daemon, it promised that the daemon will be running, with the right privileges, and not consuming more than its fair share of resources.
-
-<a name="specs"></a>
-### Contracts enable zero-conf *Specification Testing*
-
-
-
-* A daemon aspect
-  - implies a process should be running
-  - owned by the right user
-  - with a stable PID
-  - and live within defined memory bounds
-* A log aspect
-  - should be open and receiving content from the process
-  - should contain lines showing successful startup (and not contain lines matching an error).
-* A dashboard/JMX/metrics aspect:
-  - actual configuration settings as read out of the running app should match those drawn from the node attributes. No more finding out a setting was overridden by some hidden config file.
-  - should have a healthy heartbeat and status
-
-[Ironfan-CI](http://github.com/infochimps-labs/ironfan-ci) uses the announcement  to create a suite of detailed [Cucumber](http://cukes.info) (via [Cuken](https://github.com/hedgehog/cuken)) feature tests that document and enforce the machine's contract. You're not limited to just the zeroconf tests: it's easy to drop in additional cucumber specs.
-
-Ironfan-CI is young -- it's for the tenacious zealot only -- but is the subject of current work and developing fast.
-
-<a name="ci"></a>
-### Specs + Monitoring enable zero-conf *Full-stack Testing*
-
-You can now look at monitoring as the equivalent of a full-stack continuous integration test suite. The same announcement that Ironfan-CI maps into cucumber statements can as well drive your favorite monitoring suite (or more likely, the monitoring suite you hate the least).
-
-The Ironfan Enterprise product ships with Zabbix, which is actually pretty loveable -- even moreso when you don't have to perform fiddly repeated template definitions.
-
-<a name="binding"></a>
-### Systems *Bind* to provisioned resources 
-
-Components should adapt to their machine, but be largely unaware of its defaul arrangement. One common anti-pattern we see in many cookbooks is to place data at some application-specific absolute path, to assume a certain layout of volumes.
-
-When my grandmother comes to visit, she quite reasonably asks for a room with a comfortable bed and a short climb. This means that at my apartment, she stays in the main bedroom and I use the couch. At my brother's house, she stays in the downstairs guest room, while my brother and sister-in-law stay in their bedroom.
-
-Suppose Grandmom instead always chose 'the master bedroom on the first floor' no matter how the house was set up. At my apartment, she'd find herself in the parking garage. At my brother's house, she'd find herself in a crowded bed and uninvited from returning to visit.
-
-Similarly, the well-mannered cookbook does not hard-code a large data directory onto the root partition. The root drive is the private domain of the operating system; typically, there's a large and comfortably-appointed volume just for it to use. On the other hand, hard-coding a location of `/mnt/external2` will end in tears if I'm testing the cookbook on my laptop, where no such drive exists.
-
-The solution is to request for volumes by their characteristics, and defer to the machine's best effort in meeting that request.
-
-        # Data striped across all persistent dirs
-        volume_dirs('foo.datanode.data') do
-          type          :persistent, :bulk, :fallback
-          selects       :all
-          mode          "0700"
-        end
-
-        # Scratch space for indexing, striped across all scratch dirs
-        volume_dirs('foo.indexer.journal') do
-          type          :fast, local, :bulk, :fallback
-          selects       :first
-          mode          "0755"
-        end
-
-Another example of this is binding to a network interface. Unfortunately most cookbooks choose the primary address; most of ours choose the 'private' interface if any and fall back to the primary.
-
-The right pattern here is 
-* provisioners tag resources
-* cookbooks to request the best match to their purpose
-* at the cookbook's option, if no good match is found use a fallback or raise an exception
-
-<a name="resource-sharing"></a>
-### Binding declarations enable *Resource Sharing*
-
-Resource sharing is yet another place where an assertive announcement can enable best practices.
-
-Right now, most java-based components hard-code a default JVM heap size. This can lead to a situation where a component shows up on a 16GB machine with 1GB heap allocated, or where five components show up on a 0.7GB machine each with 1GB allocated. 
-
-We instead deserve a deft but highly predictable way to apportion resources (disks, ram, etc). Nothing that gets in the way of explicit tuning, but one which gives a reasonable result in the default case.
-
-The Hadoop cookbook has an initial stab at this, but for the most part Resource Sharing is on the roadmap but not yet in place.
-
-
-__________________________________________________________________________
-
-### Learn More
-
-[Aspect-Oriented Programming](http://en.wikipedia.org/wiki/Aspect-oriented_programming): The Ironfan concept of `aspects` as cross-cutting concerns is taken from AOP. Amenities don't correspond precisely to join cuts etc., so don't take the analogy too far. (Or perhaps instead help us understand how to take the analogy the rest of the way.)
-
-
-Ironfan's primary models form a component-based approach to building a  [Service-Oriented Architecture](http://msdn.microsoft.com/en-us/library/aa480021.aspx). Model examples of a modern SOA include the [Netflix API](http://www.slideshare.net/danieljacobson/the-futureofnetflixapi) (see [also](http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html)) and [Postrank](http://www.igvita.com/2011/03/08/goliath-non-blocking-ruby-19-web-server/) (see [also](http://www.igvita.com/2010/01/28/cluster-monitoring-with-ganglia-ruby/)).
-
-

data/notes/declaring_volumes.md

@@ -1,3 +0,0 @@
-
-Please see the [README from the volumes cookbook](https://github.com/infochimps-labs/ironfan-pantry/blob/master/cookbooks/volumes/README.md) for more information.
-

data/notes/design_notes-aspect_oriented_devops.md

@@ -1,36 +0,0 @@
-
-Examples of concerns that tend to be crosscutting include:
-
-Synchronization -- (declare an action dependency, trigger, event)
-Real-time constraints
-Feature interaction
-Memory management
-  - data checks
-  - feature checks
-* security
-  - firewall rules
-  - access control
-Logging
-Monitoring
-Business rules
-Tuning 
-Refactor pivot
-
-
-AOP:
-
-- Scattered (1:n) / Tangled (n:1) 
-- join point: hook
-- point cut: matches join points
-- advice: behavior evoked at point cut
-
-* Interception
-  - Interjection of advice, at least around methods.
-* Introduction
-  - Enhancing with new (orthogonal!) state and behavior .
-* Inspection
-  - Access to meta-information that may be exploited by pointcuts or
-advice.
-* Modularization
-  - Encapsulate as aspects.
-