ironfan 3.1.0.rc1 → 3.1.1

data/notes/INSTALL.md

@@ -0,0 +1,135 @@
+First of all, every Chef installation needs a Chef Homebase. Chef Homebase is the place where cookbooks, roles, config files and other artifacts for managing systems with Chef will live. Store this homebase in a version control system such as Git and treat it like source code.
+
+### Conventions
+
+In all of the below,
+
+* `{homebase}`: is the directory that holds your Chef cookbooks, roles and so forth. For example, this file is in `{homebase}/README.md`.
+
+* `{username}`: identifies your personal Chef client name: the thing you use to log into the Chef WebUI.
+
+* `{organization}`: identifies the credentials set and cloud settings to use.  If your Chef server is on the Opscode platform (Try it! It's super-easy), use your organization name (the last segment of your chef_server url). If not, use an identifier you deem sensible.
+
+Ironfan Installation Instructions
+============
+
+_Before you begin, fork the repo as you'll be making changes to personalize it for your platform._
+
+1. Clone the repo. It will produce the directory we will call `homebase` from now on:
+
+        git clone https://github.com/infochimps-labs/ironfan-homebase homebase
+        cd homebase
+        git submodule foreach git checkout master
+        git submodule update --init
+
+2. Install the Ironfan gem (you may need to use `sudo`):
+
+        gem install ironfan
+
+3. Set up your [knife.rb](http://wiki.opscode.com/display/chef/Knife#Knife-ConfiguringYourSystemForKnife) file.
+
+  
+
+Go to the [Knife Plugins](http://wiki.opscode.com/display/chef/Knife+Plugins) in the Chef Wiki for additional information.
+
+
+__________________________________________________________________________
+
+
+Installing the [knife.rb](http://wiki.opscode.com/display/chef/Knife#Knife-ConfiguringYourSystemForKnife) file when using Ironfan has proven exceedingly useful -- it leaves you in good shape to avoid problems with credential management, importing 3rd-party cookbooks, and other things down the road.
+
+## <a name="knife-configuration" />Standard Knife Configuration
+
+### Set up 
+
+_Note_: If your local username differs from your Opscode Chef username, then you should `export CHEF_USER={username}` using your Opscode username (e.g. from your .bashrc) before you run any knife commands.
+
+1. So that Knife finds its configuration files, symlink the `{homebase}/knife` directory (the one holding this file) to be your `~/.chef` folder.
+
+        cd {homebase} 
+        ln -sni $CHEF_HOMEBASE/knife ~/.chef
+
+**HEY @MRFLIP MAKE SURE THAT THIS WORKS WITHOUT THE CHEF HOMEBASE VARIABLE SET**
+
+### <a name="credentials" />Credentials
+
+All the keys and settings specific to your organization are held in a directory named `credentials/`, versioned and distributed independently of the homebase. 
+
+To set up your credentials directory, visit `{homebase}/knife` and duplicate the example, naming it `credentials`:
+
+        cd $CHEF_HOMEBASE/knife 
+        rm credentials
+        cp -a example-credentials credentials
+        cd credentials
+        git init ; git add .
+        git commit -m "New credentials universe for $CHEF_ORGANIZATION" .
+        
+### User / Cloud config
+
+Edit the following places in your new `credentials`:
+
+* Organization-specific settings are in `knife/{organization}-credentials/knife-org.rb`:
+  - _organization_:             Your organization name 
+  - _chef server url_:          Edit the lines for your `chef_server_url` and `validator`. 
+
+If you are an Opscode platform user, you can skip this step -- your `chef_server_url` defaults to `https://api.opscode.com/organizations/#{organization}` and your validator to `{organization}-validator.pem`.
+  
+- Cloud-specific settings, if you are targeting a cloud provider, add account information and configuration here. 
+
+* User-specific settings are in `knife/credentials/knife-user-{username}.rb`. (You can duplicate and rename the one in `knife/example-credentials/knife-user-example.rb`)
+  - for example, if you're using Amazon EC2 you should set your access keys:
+
+          Chef::Config.knife[:aws_access_key_id]      = "XXXX"
+          Chef::Config.knife[:aws_secret_access_key]  = "XXXX"
+          Chef::Config.knife[:aws_account_id]         = "XXXX"
+        
+* Chef user key is in `{credentials_path}/{username}.pem`
+
+* Organization validator key in `{credentials_path}/{organization}-validator.pem`
+
+* If you have existing Amazon machines, place their keypairs in `{credentials_path}/ec2_keys` 
+
+__________________________________________________________________________
+
+## Try it out
+
+You should now be able to use Knife.
+
+        $ knife client list
+        chef-webui
+        cocina-chef_server-0
+        cocina-sandbox-0
+        cocina-validator
+
+        $ knife cluster list
+        +--------------------+---------------------------------------------------+ 
+        | cluster            | path                                              |
+        +--------------------+---------------------------------------------------+
+        | burninator         | /cloud/clusters/burninator.rb                     |
+        | el_ridiculoso      | /cloud/clusters/el_ridiculoso.rb                  |
+        | elasticsearch_demo | /cloud/clusters/elasticsearch_demo.rb             |
+        | hadoop_demo        | /cloud/clusters/hadoop_demo.rb                    |
+        | sandbox            | /cloud/clusters/sandbox.rb                        |
+        +--------------------+---------------------------------------------------+
+
+Launching a cluster in the cloud should now be this easy!
+
+        knife cluster launch sandbox:simple --bootstrap
+__________________________________________________________________________
+
+**SYAFLAG: Not sure where the stuff below, before getting to Next is supposed to go.  Here?**
+
+To get started with Knife and Chef, follow the [Chef Fast Start Guide](http://wiki.opscode.com/display/chef/Fast+Start+Guide). We use the hosted Chef service and are happy, we are sure you'll be happy with it too (there are instructions in the wiki to set up a Chef server, if that's what you prefer). Stop when you get to "Bootstrap the Ubuntu system" -- Ironfan makes that much easier. **SYAFLAG: "Bootstrap the Ubuntu System is the 1st step in the EC2 Bootstrap Fast Start Guide, so should we include EC2 Bootstrap Fast Start Guide?**
+
+* [Launch Cloud Instances with Knife](http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+with+Knife)
+* [EC2 Bootstrap Fast Start Guide](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide) (**Wow, we do everything in that page in _one_ command.**)
+
+
+### Next
+
+The README file in each of the subdirectories for more information about what goes in those directories. If you are bored of reading, go customize one of the files in the 'clusters/ directory'. Or, if you're a fan of ridiculous things and have ever pondered how many things you can fit in one box, launch el_[ridiculoso:](http://www.spanishdict.com/translate/ridiculoso). It contains every single recipe we have ever made stacked on top of one another. 
+
+        knife cluster launch el_ridiculoso-gordo --bootstrap
+
+
+For more information about configuring Knife, see the [Knife documentation](http://wiki.opscode.com/display/chef/knife).

data/notes/Knife-Cluster-Commands.md

@@ -0,0 +1,8 @@
+@sya please complete and fill out the descriptions + usage:
+
+
+* knife cluster kill
+* knife cluster launch
+
+(... show start stop proxy ssh boootstrap maybe some more, type 'knife cluster' and it'll list them)
+

data/notes/Silverware.md

@@ -0,0 +1,5 @@
+Major portions of the cookbooks/silverware/README file should move here
+
+@sya please make sure the true stuff is polished, the wrong stuff is deleted or fixed, the practical stuff goes in the README.md and the philosophical stuff goes here on the wiki.
+
+

data/notes/cookbook-versioning.md

@@ -0,0 +1,11 @@
+Cookbook Versioning and Tracking
+================================
+
+@temujin9 please complete and correct
+
+* git tag labels the *release* of a cookbook version: tag 'cookbooks-elasticsearch-3.1.7' denotes the *last* commit to that tag.
+* The next commit will be the one that bumps the version number: the `metadata.rb` will then read '3.1.8'.
+
+Periodically, we will release a 'gold' version set and push those to the opscode cookbook community site.
+
+* 

data/notes/declaring_volumes.md

@@ -0,0 +1,3 @@
+
+Please see the {homebase}/cookbooks/volumes/README.md
+

data/notes/design_notes-ci_testing.md

@@ -0,0 +1,169 @@
+
+  
+https://github.com/acrmp/chefspec
+
+
+pre-testing -- converge machine 
+  https://github.com/acrmp/chefspec
+
+  http://wiki.opscode.com/display/chef/Knife#Knife-test
+
+benchmarks
+
+  bonnie++ 
+  hdparm -t
+  iozone
+
+
+in-machine
+
+* x ports on x interfaces open
+* daemon is running
+* file exists and has string
+
+* log file is accumulating lines at rate X
+* script x runs successfully
+
+in-chef
+
+* runlist is X
+* chef attribute X should be Y
+
+meta
+
+* chef run was idempotent
+
+
+
+
+
+__________________________________________________________________________
+
+## Notes from around the web
+
+
+* ...
+
+  > I'm thinking that the useful thing to test is NOT did chef install
+  > some package or setup a user, but rather after chef has run can I
+  > interact with the system as I would expect from an external
+  > perspective. For example:
+  > 
+  > * Is the website accessible?
+  > * Are unused ports blocked?
+  > * When I send an email thorough the website does it end up in my inbox?
+  > 
+  > Capybara (http://github.com/jnicklas/capybara) enforces this external
+  > perspective for webapp testing:
+  > 
+  > "Access to session, request and response from the test is not
+  > possible. Maybe we’ll do response headers at some point in the future,
+  > but the others really shouldn’t be touched in an integration test
+  > anyway. "
+  > 
+  > They only let you interact with screen elements that a user could
+  > interact with. It makes sense because the things that users interact
+  > with are what provides the business value
+
+* Andrew Shafer < andrew@cloudscaling.com>
+
+  > Here's my thinking at this point... which could be wrong on every level.
+  > There is really no good way to TDD/BDD configuration management for several
+  > reasons:
+  > The recipes are already relatively declarative
+  > Mocking is useless because it may not reflect 'ground truth'
+  > The cycle times to really test convergence are relatively long
+  > Trying to test if a package is installed or not is testing the framework,
+  > not the recipe IMHO.
+  > I agree with the general sentiment that the functional service is the true
+  > test.
+  > I'm leaning towards 'testing' at that level, ideally with (a superset of?)
+  > what should be used for the production monitoring system.
+  > So the CI builds services, runs all the checks in test, green can go live
+  > and that's that.
+  
+
+* Jeremy Deininger < jeremy@rightscale.com>
+
+  > Thought I'd chime in with my experience testing system configuration code @ RightScale so far.  What we've been building are integration style cucumber tests to run a cookbook through it's paces on all platforms and OSs that we support.  
+  > First we use our API to spin up 'fresh' server clusters in EC2, one for every platform/OS (variation) that the cookbook will be supporting.  The same could be done using other cloud APIs (anyone else doing this with VMware or etc?)  Starting from scratch is important because of chef's idempotent nature.
+  > Then a cucumber test is run against every variation in parallel.  The cucumber test runs a series of recipes on the cluster then uses what we call 'spot checks' to ensure the cluster is configured and functional.  The spot checks are updated when we find a bug, to cover the bug.  An example spot check would be, sshing to every server and checking the mysql.err file for bad strings.
+  > These high level integration tests are long running but have been very useful flushing out bugs.
+  > ...
+  > If you stop by the #rightscale channel on Freenode I'd be happy to embarrass myself by giving you a sneak peak at the features etc..  Would love to bounce ideas around and collaborate if you're interested. jeremydei on Freenode IRC
+
+Ranjib Dey < ranjibd@th...s.com>
+
+  > So far, what we've done for testing is to use rspec for implementing tests.  Here's an example:
+  > 
+  >    it "should respond on port 80" do
+  >      lambda {
+  >        TCPSocket.open(@server, 'http')
+  >      }.should_not raise_error
+  >    end
+  > 
+  > Before running the tests, I have to manually bootstrap a node using knife. If my instance is the only one in its environment, the spec can find it using knife's search feature.  The bootstrap takes a few minutes, and the 20 or so tests take about half a minute to run.
+  > 
+  > While I'm iteratively developing a recipe, my work cycle is to edit source, upload a cookbook, and rerun chef-client (usually by rerunning knife boostrap, because the execution environment is different from invoking chef-client directly).  This feels a bit slower than the cycle I'm used to when coding in Ruby because of the upload and bootstrap steps.
+  > 
+  > I like rspec over other testing tools because of how it generates handy reports, such as this one, which displays an English list of covered test cases:
+  > 
+  >    $ rspec spec/ -f doc
+  > 
+  >    Foo role
+  >      should respond on port 80
+  >      should run memcached
+  >      should accept memcached connections
+  >      should have mysql account
+  >      should allow passwordless sudo to user foo as user bar
+  >      should allow passwordless sudo to root as a member of sysadmin
+  >      should allow key login as user bar
+  >      should mount homedirs on ext4, not NFS
+  >      should rotate production.log
+  >      should have baz as default vhost
+  >      ...
+  > 
+  > That sample report also gives a feel for sort of things we check.  So far, nearly all of our checks are non-intrusive enough to run on a production system.  (The exception is testing of local email delivery configurations.)
+  > 
+  > Areas I'd love to see improvement:
+  > 
+  >    * Shortening the edit-upload-bootstrap-test cycle
+  >    * Automating the bootstrap in the context of testing
+  >    * Adding rspec primitives for Chef-related testing, which might
+  >      include support for multiple platforms
+  > 
+  > As an example of rspec primitives, instead of:
+  > 
+  >    it "should respond on port 80" do
+  >      lambda {
+  >        TCPSocket.open(@server, 'http')
+  >      }.should_not raise_error
+  >    end
+  > 
+  > I'd like to write:
+  > 
+  >    it { should respond_on_port(80) }
+  > 
+  > Rspec supports the the syntactic sugar; it's just a matter of adding some "matcher" plugins.
+  > 
+  > How do other chef users verify that recipes work as expected?
+  > 
+  > I'm not sure how applicable my approach is to opscode/cookbooks because it relies on having a specific server configuration and can only test a cookbook in the context of that single server.  If we automated the boostrap step so it could be embedded into the rspec setup blocks, it would be possible to test a cookbook in several sample contexts, but the time required to setup each server instance might be prohibitive.
+  > 
+
+
+Andrew Crump < acrump@gmail.com>
+
+  > Integration tests that exercise the service you are building definitely give you the most bang for buck.
+  > 
+  > We found the feedback cycle slow as well so I wrote chefspec which builds on RSpec to support unit testing cookbooks:
+  > 
+  > https://github.com/acrmp/chefspec
+  > 
+  > This basically fakes a convergence and allows you to make assertions about the created resources. At first glance Chef's declarative nature makes this less useful, but once you start introducing conditional execution I've found this to be a time saver.
+  > 
+  > If you're looking to do CI (which you should be) converging twice goes some way to verifying that your recipes are idempotent.
+  > 
+  > knife cookbook test is a useful first gate for CI:
+  > 
+  > http://wiki.opscode.com/display/chef/Knife#Knife-test

data/notes/design_notes-cookbook_event_ordering.md

@@ -0,0 +1,212 @@
+# Cookbook event ordering
+
+
+Most cookbooks have some set of the following
+
+* base configuration
+* announce component
+  - before discovery so it can be found
+  - currently done in converge stage -- some aspects might be incompletely defined?
+
+* register apt repository if any
+* create daemon user
+  - before directories so we can set permissions
+  - before package install so uid is stable
+* install, as package, git deploy or install from release
+  - often have to halt legacy services -- config files don't exist
+* create any remaining directories 
+  - after package install so it has final say
+* install plugins
+  - after directories, before config files
+* define service
+  - before config file creation so we can notify
+  - can't start it yet because no config files
+* discover component (own or other, same or other machines)
+* write config files (notify service of changes)
+  - must follow everything so info is current
+* register a minidash dashboard
+* trigger start (or restart) of service
+
+Run List is 
+
+    [role[systemwide], role[chef_client], role[ssh], role[nfs_client],
+    role[volumes], role[package_set], role[org_base], role[org_users], 
+    
+    role[hadoop],
+    role[hadoop_s3_keys], 
+    role[cassandra_server], role[zookeeper_server],
+    role[flume_master], role[flume_agent], 
+    role[ganglia_master],
+    role[ganglia_agent], role[hadoop_namenode], role[hadoop_datanode],
+    role[hadoop_jobtracker], role[hadoop_secondarynn], role[hadoop_tasktracker],
+    role[hbase_master], role[hbase_regionserver], role[hbase_stargate],
+    role[redis_server], role[mysql_client], role[redis_client],
+    role[cassandra_client], role[elasticsearch_client], role[jruby], role[pig],
+    recipe[ant], recipe[bluepill], recipe[boost], recipe[build-essential],
+    recipe[cron], recipe[git], recipe[hive], recipe[java::sun], recipe[jpackage],
+    recipe[jruby], recipe[nodejs], recipe[ntp], recipe[openssh], recipe[openssl],
+    recipe[rstats], recipe[runit], recipe[thrift], recipe[xfs], recipe[xml],
+    recipe[zabbix], recipe[zlib], recipe[apache2], recipe[nginx],
+    role[el_ridiculoso_cluster], role[el_ridiculoso_gordo], role[minidash],
+    role[org_final], recipe[hadoop_cluster::config_files], role[tuning]]
+
+Run List expands to 
+
+    build-essential, motd, zsh, emacs, ntp, nfs, nfs::client, xfs,
+    volumes::mount, volumes::resize, package_set, 
+    
+    hadoop_cluster,
+    hadoop_cluster::minidash, 
+    
+    cassandra, cassandra::install_from_release,
+    cassandra::autoconf, cassandra::server, cassandra::jna_support,
+    cassandra::config_files, zookeeper::default, zookeeper::server,
+    zookeeper::config_files, flume, flume::master, flume::agent,
+    flume::jruby_plugin, flume::hbase_sink_plugin, ganglia, ganglia::server,
+    ganglia::monitor, 
+    
+    hadoop_cluster::namenode, 
+    hadoop_cluster::datanode,
+    hadoop_cluster::jobtracker, 
+    hadoop_cluster::secondarynn,
+    hadoop_cluster::tasktracker, 
+    
+    zookeeper::client, 
+    hbase::master,
+    hbase::minidash, 
+    
+    minidash::server, 
+    hbase::regionserver, 
+    hbase::stargate,
+    redis, redis::install_from_release, redis::server, 
+    mysql, mysql::client,
+    cassandra::client, elasticsearch::default,
+    
+    elasticsearch::install_from_release, 
+    elasticsearch::plugins,
+    elasticsearch::client, 
+    
+    jruby, jruby::gems, 
+    
+    pig, 
+    pig::install_from_package,
+    pig::piggybank, 
+    pig::integration, 
+    
+    zookeeper, 
+    ant, bluepill, boost, cron,
+    git, hive, 
+    java::sun, jpackage, nodejs, openssh, openssl, rstats, runit,
+    thrift, xml, zabbix, zlib, apache2, nginx, hadoop_cluster::config_files,
+    tuning::default
+    
+
+From an actual run of el_ridiculoso-gordo:
+    
+      2 nfs::client
+      1 java::sun
+      1 aws::default
+      5 build-essential::default
+      3 motd::default
+      2 zsh::default
+      1 emacs::default
+      8 ntp::default
+      1 nfs::default
+      2 nfs::client
+      3 xfs::default
+     46 package_set::default
+      4 java::sun
+      8 tuning::ubuntu
+      6 apt::default
+      1 hadoop_cluster::add_cloudera_repo
+     44 hadoop_cluster::default
+      4 minidash::default
+      2 /srv/chef/file_store/cookbooks/minidash/providers/dashboard.rb
+      1 hadoop_cluster::minidash
+      2 /srv/chef/file_store/cookbooks/minidash/providers/dashboard.rb
+      1 boost::default
+      2 python::package
+      2 python::pip
+      1 python::virtualenv
+      2 install_from::default
+      7 thrift::default
+      9 cassandra::default
+      1 cassandra::install_from_release
+      6 /srv/chef/file_store/cookbooks/install_from/providers/release.rb
+      3 cassandra::install_from_release
+      1 cassandra::bintools
+      3 runit::default
+     11 cassandra::server
+      2 cassandra::jna_support
+      2 cassandra::config_files
+      6 zookeeper::default
+     15 zookeeper::server
+      3 zookeeper::config_files
+     18 flume::default
+      2 flume::master
+      3 flume::agent
+      2 flume::jruby_plugin
+      1 flume::hbase_sink_plugin
+     21 ganglia::server
+     20 ganglia::monitor
+     13 hadoop_cluster::namenode
+     11 hadoop_cluster::datanode
+     11 hadoop_cluster::jobtracker
+     11 hadoop_cluster::secondarynn
+     11 hadoop_cluster::tasktracker
+     14 hbase::default
+     11 hbase::master
+      1 hbase::minidash
+      2 /srv/chef/file_store/cookbooks/minidash/providers/dashboard.rb
+     13 minidash::server
+     11 hbase::regionserver
+     10 hbase::stargate
+      2 redis::default
+      2 redis::install_from_release
+      2 redis::default
+     16 redis::server
+      3 mysql::client
+      1 aws::default
+      7 elasticsearch::default
+      1 elasticsearch::install_from_release
+      6 /srv/chef/file_store/cookbooks/install_from/providers/release.rb
+      2 elasticsearch::plugins
+      3 elasticsearch::client
+      3 elasticsearch::config
+      1 jruby::default
+      9 /srv/chef/file_store/cookbooks/install_from/providers/release.rb
+      7 jruby::default
+     18 jruby::gems
+      2 pig::install_from_package
+      5 pig::piggybank
+      8 pig::integration
+      6 zookeeper::default
+      3 ant::default
+      5 bluepill::default
+      2 cron::default
+      1 git::default
+      1 hive::default
+      2 nodejs::default
+      4 openssh::default
+      7 rstats::default
+      1 xml::default
+     11 zabbix::default
+      1 zlib::default
+     10 apache2::default
+      2 apache2::mod_status
+      2 apache2::mod_alias
+      1 apache2::mod_auth_basic
+      1 apache2::mod_authn_file
+      1 apache2::mod_authz_default
+      1 apache2::mod_authz_groupfile
+      1 apache2::mod_authz_host
+      1 apache2::mod_authz_user
+      2 apache2::mod_autoindex
+      2 apache2::mod_dir
+      1 apache2::mod_env
+      2 apache2::mod_mime
+      2 apache2::mod_negotiation
+      2 apache2::mod_setenvif
+      1 apache2::default
+      8 nginx::default
+      9 hadoop_cluster::config_files

data/notes/design_notes-dsl_object.md

@@ -0,0 +1,55 @@
+
+
+* is a subclass of extlib's `Mash` (Hash with indifferent access). 
+  - in the 
+
+* does not have to be performant. Obviously it can't suck, but we're not going to worry about say the overhead of converting values to `dsl_mash`es on assignment.
+
+## Layering
+
+This is the key thing. We don't prescribe *any* resolution semantics except the following:
+
+* layers are evaluated in order to create a composite, `dup`ing as you go.
+* while building a composite, when a later layer and the current layer collide:
+  - if layer has merge logic, hand it off. 
+  - simple + any:      clobber
+  - array + array:     layer value appended to composite value
+  - hash  + hash:      recurse
+  - otherwise:         error
+  
+###  no complicated resolution rules allowed
+
+This is the key to the whole thing.
+
+* You can very easily 
+* You can adorn an object with merge logic if it's more complicated than that
+
+### duping (?)
+
+I'm not certain of the always-`dup` rule. We'll find out
+
+__________________________________________________________________________
+
+## Interface
+
+### setter/getter/block
+
+
+* `self.foo`                -- returns value of foo
+* `self.get(:foo)`          -- sets foo to val
+* `self.foo(val)`           -- sets foo to val
+* `self.set(:foo, val)`     -- sets foo to val
+* `self.foo ||= val`        -- sets foo to val if foo is unset, `false` or `nil`
+* `self.default(:foo, val)` -- sets foo to val if foo is unset
+* `self.unset(:foo)`        -- unsets foo
+   
+### collection attributes
+
+
+    | array    | array  |     add or clobber
+    | 
+   
+### nested objects   
+
+
+### type co

data/notes/design_notes-meta_discovery.md

@@ -0,0 +1,59 @@
+@temujin9 has proposed, and it's a good propose, that there should exist such a thing as an 'integration cookbook'.
+
+The hadoop_cluster cookbook should describe the hadoop_cluster, the ganglia cookbook ganglia, and the zookeeper cookbook zookeeper. Each should provide hooks that are neighborly but not exhibitionist, but should mind its own business.
+
+The job of tying those components together should belong to a separate concern. It should know how and when to copy hbase jars into the pig home dir, or what cluster service_provide'r a redis client should reference.
+
+## Practical implications
+
+* I'm going to revert out the `node[:zookeeper][:cluster_name]` attributes -- services should always announce under their cluster.
+
+* Until we figure out how and when to separate integrations, I'm going to isolate entanglements into their own recipes within cookbooks: so, the ganglia part of hadoop will become `ganglia_integration` or somesuch.
+
+## Example integrations
+
+### Copying jars
+
+Pig needs jars from hbase and zookeeper.  They should announce they have jars; pig should announce its home directory;  the integration should decide how and where to copy the jars.
+
+### Reference a service
+
+Right now in several places we have attributes like `node[:zookeeper][:cluster_name]`, used to specify the cluster that provides_service zookeeper.
+
+* Server recipes should never use `node[:zookeeper][:cluster_name]` -- they should always announce under their native cluster. (I'd kinda like to give `provides_service` some sugar to assume the cluster name, but need to find something backwards-compatible to use)
+
+* Need to take a better survey of usage among clients to determine how to do this.
+
+* cases:
+  - hbase cookbook refs: hadoop, zookeeper, ganglia
+  - flume cookbook refs: zookeeper, ganglia. 
+  - flume agents may reference several different flume provides_service'rs
+  - API using two different elasticsearch clusters
+  
+### Logging, monitoring
+
+* tell flume you have logs to pick up
+* tell ganglia to monitor you
+
+### Service Dashboard
+
+Let everything with a dashboard say so, and then let one resource create a page that links to each.
+
+
+________________________
+
+These are still forming, ideas welcome.
+
+
+
+
+Sometimes we want 
+
+* if there are volumes marked for 'mysql-table_data', use that; otherwise, the 'persistent' datastore, if any; else the 'bulk' datastore, if any; else the 'fallback' datastore (which is guaranteed to exist).
+
+* IP addresses (or hostnames):
+  - `[:private_ip, :public_ip]`
+  - `[:private_ip]`
+  - `:primary_ip`
+  
+* .