ironfan 4.3.4 → 4.4.0

data/notes/core_concepts.md

@@ -0,0 +1,200 @@
+# Ironfan Core Concepts
+
+<a name="TOC"></a>
+
+* [Build your architecture from clusters of cooperating machines](#clusters)
+
+* [Decoupled *Components* connect](#components)
+
+* [Components *Announce* their capabilities](#announcements)
+
+* [Announcements enable *Service Discovery*](#discovery)
+
+* [Components announce cross-cutting *Aspects*](#aspects)
+
+* [Aspects enable zero-conf *Amenities*](#amenities) - 
+
+* [Announcements effectively define a component's *Contract*](#contract)
+
+* [Contracts enable zero-conf *specification testing*](#specs)
+
+* [Specs + monitoring enable zero-conf *integration testing*](#ci)
+
+* [Systems *Bind* to provisioned resources](#binding)
+
+* [Binding declarations enable *Resource Sharing*](#resource-sharing)
+
+<a name="overview"></a>
+### Overview
+
+Ironfan is your system diagram come to life. In ironfan, you use Chef to assemble and configure components on each machine. Ironfan assembles those machines into clusters -- a group of machines united to provide an important service. For example, at Infochimps one cluster of machines serves the webpages for infochimps.com; another consists only of elasticsearch machines to power our API; and another runs the lightweight goliath proxies that implement our API. Our data scientists are able to spin up and shut down terabyte-scale hadoop clusters in minutes. All this is supported by an Ops team of one -- who spends most of his time hacking on Ironfan. 
+
+The powerful abstractions provided by Chef and Ironfan enables an autowiring system diagram, inevitable best practices in the form of "amenities", and a readable, testable contract for each component in the stack.
+
+<a name="clusters"></a><a name="facets"></a>
+### Clusters and Facets
+
+A `cluster`, as mentioned, groups a set of machines around a common purpose. Within that cluster, you define `facet`s: a set of servers with identical components (and nearly identical configuration). 
+
+For example, a typical web stack cluster might have these facets:
+
+* `webnode`s: nginx reverse-proxies requests to a pool of unicorns running Rails
+* `mysql`: one or many MySQL servers, with attached persistent storage
+* `qmaster`s: a redis DB and resque front end to distribute batch-processing tasks
+* `qworkers`s: resque worker processes
+
+<a name="components"></a>
+### Components
+
+As you can see, the details of a machine largely follow from the list its `component`s: `mysql_server`, `resque_dashboard`, and so forth. What's a component? If you would draw it in a box on your system diagram, want to discover it from elsewhere, or it it forms part of the contract for your machine, it's a component. 
+
+Some systems have more than one component: the `ganglia` monitoring system has a component named `agent` to gather operating metrics, and a component named `master` to aggregate those metrics. 
+
+Those examples all describe daemon processes that listen on ports, but component is more general that that -- it's any isolatable piece of functionality that is interesting to an outside consumer. Here is a set of example systems we'll refer to repeatedly:
+
+* *Ganglia*, a distributed system monitoring tool. The `agent` components gather and exchange system metrics, and the `master` component aggregates them. A basic setup would run the `master` component on a single machine, and the `agent` component on many machines (including the master). In order to work, the master must discover all agents, and each agent must discover the master.
+
+* *Elasticsearch* is a powerful distributed document database. A basic setup runs a single `server` component on each machine. Elasticsearch handles discovery, but needs a stable subset of them to declare as discovery `seed`s.
+
+* *Nginx* is a fast, lightweight webserver (similar to apache). Its `server` component can proxy web requests for one or many web apps. Those apps register a `site` component, which defines the receiving address (public/private/local), how the app connects to nginx (socket, port, files).
+
+* *Pig* is a Big Data analysis tool that works with Hadoop, Elasticsearch and more. It provides an executable, and imports jars from hadoop, elasticsearch and others.
+
+<a name="announcements"></a>
+### Components *Announce* their capabilities
+
+Notice the recurring patterns: *capabilities* (serve webpages, execute script, send metrics, answer queries), *handles* (ip+port, jars, swarm), *aspects* (ports, daemons, logs, files, dashboards).
+
+The Silverware cookbook lets your services `announce` their capabilities and `discover` other resources.
+
+Chef cookbooks describe the related components that form a system. You should always have a recipe, separate from the `default` recipe, that clearly corresponds to the component: the `ganglia` cookbook has `master` and `agent` recipes; the `pig` cookbook has `install_from_package` and `install_from_release` recipes. Those recipes are grouped together into Chef roles that encapsulate the component: the `elasticsearch_server` role calls the recipes to install the software, start the daemon process, and write the config files, each in the correct order.
+
+Cookbooks do *not* bake in assumptions about their scale or about the machine they're on. The same Elasticsearch cookbook can deploy a tiny little search box to sit next to a web app, or one server in a distributed terabyte scale database.
+
+<a name="discovery"></a>
+### Announcements enable *Service Discovery*
+
+The `discover` and `discover_all` connect decoupled components. Your systems
+
+* Don't care whether the discovered components are on the same machine, different machines, or a remote data center.
+* Don't care about the number of underlying machines -- the whole thing might run on your laptop while developing, across a handful of nodes in staging, and on dozens of nodes in production.
+* Don't necessarily care about the actual system -- your load balancer doesn't care whether it's nginx or apache or anything else, it just wants to discover the correct set of `webnode`s.
+
+<a name="aspects"></a>
+### Components announce cross-cutting *Aspects*
+
+Besides the component's capabilities, the announcement also describes its aspects: cross-cutting attributes common to many components.
+
+* **log**:         write data to a log file.
+* **daemon**:      long-running process. Can specify run state, resource bounds,  etc.
+* **port**:        serves data over a port. Can specify the protocol, performance expectations, etc.
+* **dashboard**:   HTML, JMX, etc -- internal component metrics and control
+* **executable**:  executes scripts
+* **export**:      libraries, `jar`s, `conf` files, etc 
+* **consumes**:    registered whenever you `discover` another component
+
+<a name="amenities"></a>
+### Aspects enable zero-conf *Amenities* 
+
+Typically, consumers discover their provider, and the provider is unconcerned with which consumers it attends to. Ironfan lets you invert this pattern: decoupled `amenities` find components they can cater to.
+
+* A log aspect would enable the following amenities
+  - `logrotated` to intelligently manage its logs
+  - `flume` to archive logs to a predictable location
+  - If the log is known to be an apache web log, a flume decorator can track rate and duration of requests and errors.
+* A port aspect would enable
+  - zeroconf configuration of firewall and security groups
+  - remote monitors to regularly pinging the port for uptime and latency 
+  - and pings the interfaces that it should *not* appear on to ensure the firewall is in place?
+
+<a name="contracts"></a>
+### Announcements effectively define a component's *Contract*
+
+The announcements that components make don’t just facilitate discovery. In a larger sense, they describe the external contract for the component.
+
+When `nginx` announces that it listens on `node[:nginx][:http_port] = 80`, it is promising a capability (namely, that http requests to that port return certain results). When elasticsearch announces that it runs the `elasticsearch` daemon, it promised that the daemon will be running, with the right privileges, and not consuming more than its fair share of resources.
+
+<a name="specs"></a>
+### Contracts enable zero-conf *Specification Testing*
+
+
+
+* A daemon aspect
+  - implies a process should be running
+  - owned by the right user
+  - with a stable PID
+  - and live within defined memory bounds
+* A log aspect
+  - should be open and receiving content from the process
+  - should contain lines showing successful startup (and not contain lines matching an error).
+* A dashboard/JMX/metrics aspect:
+  - actual configuration settings as read out of the running app should match those drawn from the node attributes. No more finding out a setting was overridden by some hidden config file.
+  - should have a healthy heartbeat and status
+
+[Ironfan-CI](http://github.com/infochimps-labs/ironfan-ci) uses the announcement  to create a suite of detailed [Cucumber](http://cukes.info) (via [Cuken](https://github.com/hedgehog/cuken)) feature tests that document and enforce the machine's contract. You're not limited to just the zeroconf tests: it's easy to drop in additional cucumber specs.
+
+Ironfan-CI is young -- it's for the tenacious zealot only -- but is the subject of current work and developing fast.
+
+<a name="ci"></a>
+### Specs + Monitoring enable zero-conf *Full-stack Testing*
+
+You can now look at monitoring as the equivalent of a full-stack continuous integration test suite. The same announcement that Ironfan-CI maps into cucumber statements can as well drive your favorite monitoring suite (or more likely, the monitoring suite you hate the least).
+
+The Ironfan Enterprise product ships with Zabbix, which is actually pretty loveable -- even moreso when you don't have to perform fiddly repeated template definitions.
+
+<a name="binding"></a>
+### Systems *Bind* to provisioned resources 
+
+Components should adapt to their machine, but be largely unaware of its defaul arrangement. One common anti-pattern we see in many cookbooks is to place data at some application-specific absolute path, to assume a certain layout of volumes.
+
+When my grandmother comes to visit, she quite reasonably asks for a room with a comfortable bed and a short climb. This means that at my apartment, she stays in the main bedroom and I use the couch. At my brother's house, she stays in the downstairs guest room, while my brother and sister-in-law stay in their bedroom.
+
+Suppose Grandmom instead always chose 'the master bedroom on the first floor' no matter how the house was set up. At my apartment, she'd find herself in the parking garage. At my brother's house, she'd find herself in a crowded bed and uninvited from returning to visit.
+
+Similarly, the well-mannered cookbook does not hard-code a large data directory onto the root partition. The root drive is the private domain of the operating system; typically, there's a large and comfortably-appointed volume just for it to use. On the other hand, hard-coding a location of `/mnt/external2` will end in tears if I'm testing the cookbook on my laptop, where no such drive exists.
+
+The solution is to request for volumes by their characteristics, and defer to the machine's best effort in meeting that request.
+
+        # Data striped across all persistent dirs
+        volume_dirs('foo.datanode.data') do
+          type          :persistent, :bulk, :fallback
+          selects       :all
+          mode          "0700"
+        end
+
+        # Scratch space for indexing, striped across all scratch dirs
+        volume_dirs('foo.indexer.journal') do
+          type          :fast, local, :bulk, :fallback
+          selects       :first
+          mode          "0755"
+        end
+
+Another example of this is binding to a network interface. Unfortunately most cookbooks choose the primary address; most of ours choose the 'private' interface if any and fall back to the primary.
+
+The right pattern here is 
+* provisioners tag resources
+* cookbooks to request the best match to their purpose
+* at the cookbook's option, if no good match is found use a fallback or raise an exception
+
+<a name="resource-sharing"></a>
+### Binding declarations enable *Resource Sharing*
+
+Resource sharing is yet another place where an assertive announcement can enable best practices.
+
+Right now, most java-based components hard-code a default JVM heap size. This can lead to a situation where a component shows up on a 16GB machine with 1GB heap allocated, or where five components show up on a 0.7GB machine each with 1GB allocated. 
+
+We instead deserve a deft but highly predictable way to apportion resources (disks, ram, etc). Nothing that gets in the way of explicit tuning, but one which gives a reasonable result in the default case.
+
+The Hadoop cookbook has an initial stab at this, but for the most part Resource Sharing is on the roadmap but not yet in place.
+
+
+__________________________________________________________________________
+
+### Learn More
+
+[Aspect-Oriented Programming](http://en.wikipedia.org/wiki/Aspect-oriented_programming): The Ironfan concept of `aspects` as cross-cutting concerns is taken from AOP. Amenities don't correspond precisely to join cuts etc., so don't take the analogy too far. (Or perhaps instead help us understand how to take the analogy the rest of the way.)
+
+
+Ironfan's primary models form a component-based approach to building a  [Service-Oriented Architecture](http://msdn.microsoft.com/en-us/library/aa480021.aspx). Model examples of a modern SOA include the [Netflix API](http://www.slideshare.net/danieljacobson/the-futureofnetflixapi) (see [also](http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html)) and [Postrank](http://www.igvita.com/2011/03/08/goliath-non-blocking-ruby-19-web-server/) (see [also](http://www.igvita.com/2010/01/28/cluster-monitoring-with-ganglia-ruby/)).
+
+

data/notes/declaring_volumes.md

@@ -0,0 +1,3 @@
+
+Please see the [README from the volumes cookbook](https://github.com/infochimps-labs/ironfan-pantry/blob/master/cookbooks/volumes/README.md) for more information.
+

data/notes/design_notes-aspect_oriented_devops.md

@@ -0,0 +1,36 @@
+
+Examples of concerns that tend to be crosscutting include:
+
+Synchronization -- (declare an action dependency, trigger, event)
+Real-time constraints
+Feature interaction
+Memory management
+  - data checks
+  - feature checks
+* security
+  - firewall rules
+  - access control
+Logging
+Monitoring
+Business rules
+Tuning 
+Refactor pivot
+
+
+AOP:
+
+- Scattered (1:n) / Tangled (n:1) 
+- join point: hook
+- point cut: matches join points
+- advice: behavior evoked at point cut
+
+* Interception
+  - Interjection of advice, at least around methods.
+* Introduction
+  - Enhancing with new (orthogonal!) state and behavior .
+* Inspection
+  - Access to meta-information that may be exploited by pointcuts or
+advice.
+* Modularization
+  - Encapsulate as aspects.
+

data/notes/design_notes-ci_testing.md

@@ -0,0 +1,169 @@
+
+  
+https://github.com/acrmp/chefspec
+
+
+pre-testing -- converge machine 
+  https://github.com/acrmp/chefspec
+
+  http://wiki.opscode.com/display/chef/Knife#Knife-test
+
+benchmarks
+
+  bonnie++ 
+  hdparm -t
+  iozone
+
+
+in-machine
+
+* x ports on x interfaces open
+* daemon is running
+* file exists and has string
+
+* log file is accumulating lines at rate X
+* script x runs successfully
+
+in-chef
+
+* runlist is X
+* chef attribute X should be Y
+
+meta
+
+* chef run was idempotent
+
+
+
+
+
+__________________________________________________________________________
+
+## Notes from around the web
+
+
+* ...
+
+  > I'm thinking that the useful thing to test is NOT did chef install
+  > some package or setup a user, but rather after chef has run can I
+  > interact with the system as I would expect from an external
+  > perspective. For example:
+  > 
+  > * Is the website accessible?
+  > * Are unused ports blocked?
+  > * When I send an email thorough the website does it end up in my inbox?
+  > 
+  > Capybara (http://github.com/jnicklas/capybara) enforces this external
+  > perspective for webapp testing:
+  > 
+  > "Access to session, request and response from the test is not
+  > possible. Maybe we’ll do response headers at some point in the future,
+  > but the others really shouldn’t be touched in an integration test
+  > anyway. "
+  > 
+  > They only let you interact with screen elements that a user could
+  > interact with. It makes sense because the things that users interact
+  > with are what provides the business value
+
+* Andrew Shafer < andrew@cloudscaling.com>
+
+  > Here's my thinking at this point... which could be wrong on every level.
+  > There is really no good way to TDD/BDD configuration management for several
+  > reasons:
+  > The recipes are already relatively declarative
+  > Mocking is useless because it may not reflect 'ground truth'
+  > The cycle times to really test convergence are relatively long
+  > Trying to test if a package is installed or not is testing the framework,
+  > not the recipe IMHO.
+  > I agree with the general sentiment that the functional service is the true
+  > test.
+  > I'm leaning towards 'testing' at that level, ideally with (a superset of?)
+  > what should be used for the production monitoring system.
+  > So the CI builds services, runs all the checks in test, green can go live
+  > and that's that.
+  
+
+* Jeremy Deininger < jeremy@rightscale.com>
+
+  > Thought I'd chime in with my experience testing system configuration code @ RightScale so far.  What we've been building are integration style cucumber tests to run a cookbook through it's paces on all platforms and OSs that we support.  
+  > First we use our API to spin up 'fresh' server clusters in EC2, one for every platform/OS (variation) that the cookbook will be supporting.  The same could be done using other cloud APIs (anyone else doing this with VMware or etc?)  Starting from scratch is important because of chef's idempotent nature.
+  > Then a cucumber test is run against every variation in parallel.  The cucumber test runs a series of recipes on the cluster then uses what we call 'spot checks' to ensure the cluster is configured and functional.  The spot checks are updated when we find a bug, to cover the bug.  An example spot check would be, sshing to every server and checking the mysql.err file for bad strings.
+  > These high level integration tests are long running but have been very useful flushing out bugs.
+  > ...
+  > If you stop by the #rightscale channel on Freenode I'd be happy to embarrass myself by giving you a sneak peak at the features etc..  Would love to bounce ideas around and collaborate if you're interested. jeremydei on Freenode IRC
+
+Ranjib Dey < ranjibd@th...s.com>
+
+  > So far, what we've done for testing is to use rspec for implementing tests.  Here's an example:
+  > 
+  >    it "should respond on port 80" do
+  >      lambda {
+  >        TCPSocket.open(@server, 'http')
+  >      }.should_not raise_error
+  >    end
+  > 
+  > Before running the tests, I have to manually bootstrap a node using knife. If my instance is the only one in its environment, the spec can find it using knife's search feature.  The bootstrap takes a few minutes, and the 20 or so tests take about half a minute to run.
+  > 
+  > While I'm iteratively developing a recipe, my work cycle is to edit source, upload a cookbook, and rerun chef-client (usually by rerunning knife boostrap, because the execution environment is different from invoking chef-client directly).  This feels a bit slower than the cycle I'm used to when coding in Ruby because of the upload and bootstrap steps.
+  > 
+  > I like rspec over other testing tools because of how it generates handy reports, such as this one, which displays an English list of covered test cases:
+  > 
+  >    $ rspec spec/ -f doc
+  > 
+  >    Foo role
+  >      should respond on port 80
+  >      should run memcached
+  >      should accept memcached connections
+  >      should have mysql account
+  >      should allow passwordless sudo to user foo as user bar
+  >      should allow passwordless sudo to root as a member of sysadmin
+  >      should allow key login as user bar
+  >      should mount homedirs on ext4, not NFS
+  >      should rotate production.log
+  >      should have baz as default vhost
+  >      ...
+  > 
+  > That sample report also gives a feel for sort of things we check.  So far, nearly all of our checks are non-intrusive enough to run on a production system.  (The exception is testing of local email delivery configurations.)
+  > 
+  > Areas I'd love to see improvement:
+  > 
+  >    * Shortening the edit-upload-bootstrap-test cycle
+  >    * Automating the bootstrap in the context of testing
+  >    * Adding rspec primitives for Chef-related testing, which might
+  >      include support for multiple platforms
+  > 
+  > As an example of rspec primitives, instead of:
+  > 
+  >    it "should respond on port 80" do
+  >      lambda {
+  >        TCPSocket.open(@server, 'http')
+  >      }.should_not raise_error
+  >    end
+  > 
+  > I'd like to write:
+  > 
+  >    it { should respond_on_port(80) }
+  > 
+  > Rspec supports the the syntactic sugar; it's just a matter of adding some "matcher" plugins.
+  > 
+  > How do other chef users verify that recipes work as expected?
+  > 
+  > I'm not sure how applicable my approach is to opscode/cookbooks because it relies on having a specific server configuration and can only test a cookbook in the context of that single server.  If we automated the boostrap step so it could be embedded into the rspec setup blocks, it would be possible to test a cookbook in several sample contexts, but the time required to setup each server instance might be prohibitive.
+  > 
+
+
+Andrew Crump < acrump@gmail.com>
+
+  > Integration tests that exercise the service you are building definitely give you the most bang for buck.
+  > 
+  > We found the feedback cycle slow as well so I wrote chefspec which builds on RSpec to support unit testing cookbooks:
+  > 
+  > https://github.com/acrmp/chefspec
+  > 
+  > This basically fakes a convergence and allows you to make assertions about the created resources. At first glance Chef's declarative nature makes this less useful, but once you start introducing conditional execution I've found this to be a time saver.
+  > 
+  > If you're looking to do CI (which you should be) converging twice goes some way to verifying that your recipes are idempotent.
+  > 
+  > knife cookbook test is a useful first gate for CI:
+  > 
+  > http://wiki.opscode.com/display/chef/Knife#Knife-test

data/notes/design_notes-cookbook_event_ordering.md

@@ -0,0 +1,249 @@
+# Cookbook event ordering
+
+
+Most cookbooks have some set of the following
+
+* base configuration
+* announce component
+  - before discovery so it can be found
+  - currently done in converge stage -- some aspects might be incompletely defined?
+
+* register apt repository if any
+* create daemon user
+  - before directories so we can set permissions
+  - before package install so uid is stable
+* install, as package, git deploy or install from release
+  - often have to halt legacy services -- config files don't exist
+* create any remaining directories 
+  - after package install so it has final say
+* install plugins
+  - after directories, before config files
+* define service
+  - before config file creation so we can notify
+  - can't start it yet because no config files
+* discover component (own or other, same or other machines)
+* write config files (notify service of changes)
+  - must follow everything so info is current
+* register a minidash dashboard
+* trigger start (or restart) of service
+
+## Proposal:
+
+kill `role`s, in favor of `stack`s.
+
+A runlist is assembled in the following phases:
+
+* `initial` configuration
+* `before_install`
+  - `account`
+* `install`
+  - `plugin` install
+  - `directories`
+* `announcement`
+  - `service` definition
+- `discovery`
+* `commit`
+  - `config_files`: write config files to disk
+* `finalize`
+* `launch`
+
+As you can see, most layers have semantic names (`plugin`, `user`, etc); if you name your recipe correctly, they will be assigned to the correct phase. Otherwise, you can attach them explicitly to a semantic or non-semantic phase.
+
+    elasticsearch_datanode component:
+    
+    elasticsearch::default              # initial
+    elasticsearch::install_from_release #install phase
+    elasticsearch::plugins              # plugin phase
+
+    elasticsearch::server               # service definition; includes announcement
+    elasticsearch::config_files         # config_files; includes discovery
+
+I'm not clear on how much the phases should be strictly broken out into one-recipe-per-phase.
+
+It's also possible to instead define a chef resource that let you defer a block to a given phase from within a callback. It would be similar to a ruby block, but have more explicit timing. This may involve jacking around inside Chef, something we've avoided to now.
+
+__________________________________________________________________________
+
+Run List is 
+
+    [role[systemwide], role[chef_client], role[ssh], role[nfs_client],
+    role[volumes], role[package_set], role[org_base], role[org_users], 
+    
+    role[hadoop],
+    role[hadoop_s3_keys], 
+    role[cassandra_server], role[zookeeper_server],
+    role[flume_master], role[flume_agent], 
+    role[ganglia_master],
+    role[ganglia_agent], role[hadoop_namenode], role[hadoop_datanode],
+    role[hadoop_jobtracker], role[hadoop_secondarynn], role[hadoop_tasktracker],
+    role[hbase_master], role[hbase_regionserver], role[hbase_stargate],
+    role[redis_server], role[mysql_client], role[redis_client],
+    role[cassandra_client], role[elasticsearch_client], role[jruby], role[pig],
+    recipe[ant], recipe[bluepill], recipe[boost], recipe[build-essential],
+    recipe[cron], recipe[git], recipe[hive], recipe[java::sun], recipe[jpackage],
+    recipe[jruby], recipe[nodejs], recipe[ntp], recipe[openssh], recipe[openssl],
+    recipe[rstats], recipe[runit], recipe[thrift], recipe[xfs], recipe[xml],
+    recipe[zabbix], recipe[zlib], recipe[apache2], recipe[nginx],
+    role[el_ridiculoso_cluster], role[el_ridiculoso_gordo], role[minidash],
+    role[org_final], recipe[hadoop_cluster::config_files], role[tuning]]
+
+Run List expands to 
+
+    build-essential, motd, zsh, emacs, ntp, nfs, nfs::client, xfs,
+    volumes::mount, volumes::resize, package_set, 
+    
+    hadoop_cluster,
+    hadoop_cluster::minidash, 
+    
+    cassandra, cassandra::install_from_release,
+    cassandra::autoconf, cassandra::server, cassandra::jna_support,
+    cassandra::config_files, zookeeper::default, zookeeper::server,
+    zookeeper::config_files, flume, flume::master, flume::agent,
+    flume::jruby_plugin, flume::hbase_sink_plugin, ganglia, ganglia::server,
+    ganglia::monitor, 
+    
+    hadoop_cluster::namenode, 
+    hadoop_cluster::datanode,
+    hadoop_cluster::jobtracker, 
+    hadoop_cluster::secondarynn,
+    hadoop_cluster::tasktracker, 
+    
+    zookeeper::client, 
+    hbase::master,
+    hbase::minidash, 
+    
+    minidash::server, 
+    hbase::regionserver, 
+    hbase::stargate,
+    redis, redis::install_from_release, redis::server, 
+    mysql, mysql::client,
+    cassandra::client, elasticsearch::default,
+    
+    elasticsearch::install_from_release, 
+    elasticsearch::plugins,
+    elasticsearch::client, 
+    
+    jruby, jruby::gems, 
+    
+    pig, 
+    pig::install_from_package,
+    pig::piggybank, 
+    pig::integration, 
+    
+    zookeeper, 
+    ant, bluepill, boost, cron,
+    git, hive, 
+    java::sun, jpackage, nodejs, openssh, openssl, rstats, runit,
+    thrift, xml, zabbix, zlib, apache2, nginx, hadoop_cluster::config_files,
+    tuning::default
+    
+
+From an actual run of el_ridiculoso-gordo:
+    
+      2 nfs::client
+      1 java::sun
+      1 aws::default
+      5 build-essential::default
+      3 motd::default
+      2 zsh::default
+      1 emacs::default
+      8 ntp::default
+      1 nfs::default
+      2 nfs::client
+      3 xfs::default
+     46 package_set::default
+      4 java::sun
+      8 tuning::ubuntu
+      6 apt::default
+      1 hadoop_cluster::add_cloudera_repo
+     44 hadoop_cluster::default
+      4 minidash::default
+      2 /srv/chef/file_store/cookbooks/minidash/providers/dashboard.rb
+      1 hadoop_cluster::minidash
+      2 /srv/chef/file_store/cookbooks/minidash/providers/dashboard.rb
+      1 boost::default
+      2 python::package
+      2 python::pip
+      1 python::virtualenv
+      2 install_from::default
+      7 thrift::default
+      9 cassandra::default
+      1 cassandra::install_from_release
+      6 /srv/chef/file_store/cookbooks/install_from/providers/release.rb
+      3 cassandra::install_from_release
+      1 cassandra::bintools
+      3 runit::default
+     11 cassandra::server
+      2 cassandra::jna_support
+      2 cassandra::config_files
+      6 zookeeper::default
+     15 zookeeper::server
+      3 zookeeper::config_files
+     18 flume::default
+      2 flume::master
+      3 flume::agent
+      2 flume::jruby_plugin
+      1 flume::hbase_sink_plugin
+     21 ganglia::server
+     20 ganglia::monitor
+     13 hadoop_cluster::namenode
+     11 hadoop_cluster::datanode
+     11 hadoop_cluster::jobtracker
+     11 hadoop_cluster::secondarynn
+     11 hadoop_cluster::tasktracker
+     14 hbase::default
+     11 hbase::master
+      1 hbase::minidash
+      2 /srv/chef/file_store/cookbooks/minidash/providers/dashboard.rb
+     13 minidash::server
+     11 hbase::regionserver
+     10 hbase::stargate
+      2 redis::default
+      2 redis::install_from_release
+      2 redis::default
+     16 redis::server
+      3 mysql::client
+      1 aws::default
+      7 elasticsearch::default
+      1 elasticsearch::install_from_release
+      6 /srv/chef/file_store/cookbooks/install_from/providers/release.rb
+      2 elasticsearch::plugins
+      3 elasticsearch::client
+      3 elasticsearch::config
+      1 jruby::default
+      9 /srv/chef/file_store/cookbooks/install_from/providers/release.rb
+      7 jruby::default
+     18 jruby::gems
+      2 pig::install_from_package
+      5 pig::piggybank
+      8 pig::integration
+      6 zookeeper::default
+      3 ant::default
+      5 bluepill::default
+      2 cron::default
+      1 git::default
+      1 hive::default
+      2 nodejs::default
+      4 openssh::default
+      7 rstats::default
+      1 xml::default
+     11 zabbix::default
+      1 zlib::default
+     10 apache2::default
+      2 apache2::mod_status
+      2 apache2::mod_alias
+      1 apache2::mod_auth_basic
+      1 apache2::mod_authn_file
+      1 apache2::mod_authz_default
+      1 apache2::mod_authz_groupfile
+      1 apache2::mod_authz_host
+      1 apache2::mod_authz_user
+      2 apache2::mod_autoindex
+      2 apache2::mod_dir
+      1 apache2::mod_env
+      2 apache2::mod_mime
+      2 apache2::mod_negotiation
+      2 apache2::mod_setenvif
+      1 apache2::default
+      8 nginx::default
+      9 hadoop_cluster::config_files