ironfan 3.1.3 → 3.1.4

data/notes/core_concepts.md

@@ -0,0 +1,189 @@
+# Ironfan Core Concepts
+
+<a name="TOC"></a>
+* [Build your architecture from clusters of cooperating machines](#clusters)
+* [Decoupled *Components* connect](#components)
+* [Components *Announce* their capabilities](#announcements)
+* [Announcements enable *Service Discovery*](#discovery)
+* [Components announce cross-cutting *Aspects*](#aspects)
+* [Aspects enable zero-conf *Amenities*](#amenities) - 
+* [Announcements effectively define a component's *Contract*](#contract) - 
+* [Contracts enable zero-conf *specification testing*](#specs) t
+* [Specs + monitoring enable zero-conf *integration testing*](#ci) t
+* [Systems *Bind* to provisioned resources](#binding) t
+* [Binding declarations enable *Reasource Sharing*](#resource-sharing)
+
+<a name="overview"></a>
+### Overview
+
+Ironfan is your system diagram come to life. In ironfan, you use Chef to assemble and configure components on each machine. Ironfan assembles those machines into clusters -- a group of machines united to provide an important service. For example, at Infochimps one cluster of machines serves the webpages for infochimps.com; another consists only of elasticsearch machines to power our API; and yet another to run the lightweight goliath proxies that implement our API. Our data scientists are able to spin up and shut down terabyte-scale hadoop clusters in minutes. All this is supported by an Ops team of one -- who spends most of his time hacking on Ironfan. 
+
+The powerful abstractions provided by Chef and Ironfan enables an autowiring system diagram, inevitable best practices in the form of "amenities", and a readable, testable contract for each component in the stack.
+
+<a name="clusters"></a><a name="facets"></a>
+### Clusters and Facets
+
+A `cluster`, as mentioned, groups a set of machines around a common purpose. Within that cluster, you define `facet`s: a set of servers with identical components (and nearly identical configuration). 
+
+For example, a typical web stack cluster might have these facets:
+
+* `webnode`s: nginx reverse-proxies requests to a pool of unicorns running Rails
+* `mysql`: one or many MySQL servers, with attached persistent storage
+* `qmaster`s: a redis DB and resque front end to distribute batch-processing tasks
+* `qworkers`s: resque worker processes
+
+<a name="components"></a>
+### Components
+
+As you can see, the details of a machine largely follow from the list its `component`s: `mysql_server`, `resque_dashboard`, and so forth. What's a component? If you would draw it in a box on your system diagram, want to discover it from elsewhere, or it it forms part of the contract for your machine, it's a component. 
+
+Some systems have more than one component: the `ganglia` monitoring system has a component named `agent` to gather operating metrics, and a component named `master` to aggregate those metrics. 
+
+Those examples all describe daemon processes that listen on ports, but component is more general that that -- it's any isolatable piece of functionality that is interesting to an outside consumer. Here is a set of example systems we'll refer to repeatedly:
+
+* *Ganglia*, a distributed system monitoring tool. The `agent` components gather and exchange system metrics, and the `master` component aggregates them. A basic setup would run the `master` component on a single machine, and the `agent` component on many machines (including the master). In order to work, the master must discover all agents, and each agent must discover the master.
+
+* *Elasticsearch* is a powerful distributed document database. A basic setup runs a single `server` component on each machine. Elasticsearch handles discovery, but needs a stable subset of them to declare as discovery `seed`s.
+
+* *Nginx* is a fast, lightweight webserver (similar to apache). Its `server` component can proxy web requests for one or many web apps. Those apps register a `site` component, which defines the receiving address (public/private/local), how the app connects to nginx (socket, port, files).
+
+* *Pig* is a Big Data analysis tool that works with Hadoop, Elasticsearch and more. It provides an executable, and imports jars from hadoop, elasticsearch and others.
+
+<a name="announcements"></a>
+### Components *Announce* their capabilities
+
+Notice the recurring patterns: *capabilities* (serve webpages, execute script, send metrics, answer queries), *handles* (ip+port, jars, swarm), *aspects* (ports, daemons, logs, files, dashboards).
+
+The Silverware cookbook lets your services `announce` their capabilities and `discover` other resources.
+
+Chef cookbooks describe the related components that form a system. You should always have a recipe, separate from the `default` recipe, that clearly corresponds to the component: the `ganglia` cookbook has `master` and `agent` recipes; the `pig` cookbook has `install_from_package` and `install_from_release` recipes. Those recipes are grouped together into Chef roles that encapsulate the component: the `elasticsearch_server` role calls the recipes to install the software, start the daemon process, and write the config files, each in the correct order.
+
+Cookbooks do *not* bake in assumptions about their scale or about the machine their on. The same Elasticsearch cookbook can deploy a tiny little search box to sit next to a web app, or one server in a distributed terabyte scale database.
+
+<a name="discovery"></a>
+### Announcements enable *Service Discovery*
+
+The `discover` and `discover_all` connect decoupled components. Your systems
+
+* Don't care whether the discovered components are on the same machine, different machines, or a remote data center.
+* Don't care about the number of underlying machines -- the whole thing might run on your laptop while developing, across a handful of nodes in staging, and on dozens of nodes in production.
+* Don't necessarily care about the actual system -- your load balancer doesn't care whether it's nginx or apache or anything else, it just wants to discover the correct set of `webnode`s.
+
+<a name="aspects"></a>
+### Components announce cross-cutting *Aspects*
+
+Besides the component's capabilities, the announcement also describes its aspects: cross-cutting attributes common to many components.
+
+* **log**:         write data to a log file.
+* **daemon**:      long-running process. Can specify run state, resource bounds,  etc.
+* **port**:        serves data over a port. Can specify the protocol, performance expectations, etc.
+* **dashboard**:   HTML, JMX, etc -- internal component metrics and control
+* **executable**:  executes scripts
+* **export**:      libraries, `jar`s, `conf` files, etc 
+* **consumes**:    registered whenever you `discover` another component
+
+<a name="amenities"></a>
+### Aspects enable zero-conf *Amenities* 
+
+Typically, consumers discover their provider, and the provider is unconcerned with which consumers it attends to. Ironfan lets you invert this pattern: decoupled `amenities` find components they can cater to.
+
+* A log aspect would enable the following amenities
+  - `logrotated` to intelligently manage its logs
+  - `flume` to archive logs to a predictable location
+  - If the log is known to be an apache web log, a flume decorator can track rate and duration of requests and errors.
+* A port aspect would enable
+  - zeroconf configuration of firewall and security groups
+  - remote monitors to regularly pinging the port for uptime and latency 
+  - and pings the interfaces that it should *not* appear on to ensure the firewall is in place?
+
+<a name="contracts"></a>
+### Announcements effectively define a component's *Contract*
+
+The announcements that components make don’t just facilitate discovery. In a larger sense, they describe the external contract for the component.
+
+When `nginx` announces that it listens on `node[:nginx][:http_port] = 80`, it is promising a capability (namely, that http requests to that port return certain results). When elasticsearch announces that it runs the `elasticsearch` daemon, it promised that the daemon will be running, with the right privileges, and not consuming more than its fair share of resources.
+
+<a name="specs"></a>
+### Contracts enable zero-conf *Specification Testing*
+
+
+
+* A daemon aspect
+  - implies a process should be running
+  - owned by the right user
+  - with a stable PID
+  - and live within defined memory bounds
+* A log aspect
+  - should be open and receiving content from the process
+  - should contain lines showing successful startup (and not contain lines matching an error).
+* A dashboard/JMX/metrics aspect:
+  - actual configuration settings as read out of the running app should match those drawn from the node attributes. No more finding out a setting was overridden by some hidden config file.
+  - should have a healthy heartbeat and status
+
+[Ironfan-CI](http://github.com/infochimps-labs/ironfan-ci) uses the announcement  to create a suite of detailed [Cucumber](http://cukes.info) (via [Cuken](https://github.com/hedgehog/cuken)) feature tests that document and enforce the machine's contract. You're not limited to just the zeroconf tests: it's easy to drop in additional cucumber specs.
+
+Ironfan-CI is young -- it's for the tenacious zealot only -- but is the subject of current work and developing fast.
+
+<a name="ci"></a>
+### Specs + Monitoring enable zero-conf *Full-stack Testing*
+
+You can now look at monitoring as the equivalent of a full-stack continuous integration test suite. The same announcement that Ironfan-CI maps into cucumber statements can as well drive your favorite monitoring suite (or more likely, the monitoring suite you hate the least).
+
+The Ironfan Enterprise product ships with Zabbix, which is actually pretty loveable -- even moreso when you don't have to perform fiddly repeated template definitions.
+
+<a name="binding"></a>
+### Systems *Bind* to provisioned resources 
+
+Components should adapt to their machine, but be largely unaware of its defaul arrangement. One common anti-pattern we see in many cookbooks is to place data at some application-specific absolute path, to assume a certain layout of volumes.
+
+When my grandmother comes to visit, she quite reasonably asks for a room with a comfortable bed and a short climb. This means that at my apartment, she stays in the main bedroom and I use the couch. At my brother's house, she stays in the downstairs guest room, while my brother and sister-in-law stay in their bedroom.
+
+Suppose Grandmom instead always chose 'the master bedroom on the first floor' no matter how the house was set up. At my apartment, she'd find herself in the parking garage. At my brother's house, she'd find herself in a crowded bed and uninvited from returning to visit.
+
+Similarly, the well-mannered cookbook does not hard-code a large data directory onto the root partition. The root drive is the private domain of the operating system; typically, there's a large and comfortably-appointed volume just for it to use. On the other hand, hard-coding a location of `/mnt/external2` will end in tears if I'm testing the cookbook on my laptop, where no such drive exists.
+
+The solution is to request for volumes by their characteristics, and defer to the machine's best effort in meeting that request.
+
+        # Data striped across all persistent dirs
+        volume_dirs('foo.datanode.data') do
+          type          :persistent, :bulk, :fallback
+          selects       :all
+          mode          "0700"
+        end
+
+        # Scratch space for indexing, striped across all scratch dirs
+        volume_dirs('foo.indexer.journal') do
+          type          :fast, local, :bulk, :fallback
+          selects       :first
+          mode          "0755"
+        end
+
+Another example of this is binding to a network interface. Unfortunately most cookbooks choose the primary address; most of ours choose the 'private' interface if any and fall back to the primary.
+
+The right pattern here is 
+* provisioners tag resources
+* cookbooks to request the best match to their purpose
+* at the cookbook's option, if no good match is found use a fallback or raise an exception
+
+<a name="resource-sharing"></a>
+### Binding declarations enable *Resource Sharing*
+
+Resource sharing is yet another place where an assertive announcement can enable best practices.
+
+Right now, most java-based components hard-code a default JVM heap size. This can lead to a situation where a component shows up on a 16GB machine with 1GB heap allocated, or where five components show up on a 0.7GB machine each with 1GB allocated. 
+
+We instead deserve a deft but highly predictable way to apportion resources (disks, ram, etc). Nothing that gets in the way of explicit tuning, but one which gives a reasonable result in the default case.
+
+The Hadoop cookbook has an initial stab at this, but for the most part Resource Sharing is on the roadmap but not yet in place.
+
+
+__________________________________________________________________________
+
+### Learn More
+
+[Aspect-Oriented Programming](http://en.wikipedia.org/wiki/Aspect-oriented_programming): The Ironfan concept of `aspects` as cross-cutting concerns is taken from AOP. Amenities don't correspond precisely to join cuts etc., so don't take the analogy too far. (Or perhaps instead help us understand how to take the analogy the rest of the way.)
+
+
+Ironfan's primary models form a component-based approach to building a  [Service-Oriented Architecture](http://msdn.microsoft.com/en-us/library/aa480021.aspx). Model examples of a modern SOA include the [Netflix API](http://www.slideshare.net/danieljacobson/the-futureofnetflixapi) (see [also](http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html)) and [Postrank](http://www.igvita.com/2011/03/08/goliath-non-blocking-ruby-19-web-server/) (see [also](http://www.igvita.com/2010/01/28/cluster-monitoring-with-ganglia-ruby/)).
+
+

data/notes/declaring_volumes.md

@@ -1,3 +1,3 @@
 
-Please see the {homebase}/cookbooks/volumes/README.md
+Please see the [README from the volumes cookbook](https://github.com/infochimps-labs/ironfan-pantry/blob/master/cookbooks/volumes/README.md) for more information.
 

data/notes/design_notes-aspect_oriented_devops.md

@@ -0,0 +1,36 @@
+
+Examples of concerns that tend to be crosscutting include:
+
+Synchronization -- (declare an action dependency, trigger, event)
+Real-time constraints
+Feature interaction
+Memory management
+  - data checks
+  - feature checks
+* security
+  - firewall rules
+  - access control
+Logging
+Monitoring
+Business rules
+Tuning 
+Refactor pivot
+
+
+AOP:
+
+- Scattered (1:n) / Tangled (n:1) 
+- join point: hook
+- point cut: matches join points
+- advice: behavior evoked at point cut
+
+* Interception
+  - Interjection of advice, at least around methods.
+* Introduction
+  - Enhancing with new (orthogonal!) state and behavior .
+* Inspection
+  - Access to meta-information that may be exploited by pointcuts or
+advice.
+* Modularization
+  - Encapsulate as aspects.
+

data/notes/design_notes-cookbook_event_ordering.md

@@ -27,6 +27,43 @@ Most cookbooks have some set of the following
 * register a minidash dashboard
 * trigger start (or restart) of service
 
+## Proposal:
+
+kill `role`s, in favor of `stack`s.
+
+A runlist is assembled in the following phases:
+
+* `initial` configuration
+* `before_install`
+  - `account`
+* `install`
+  - `plugin` install
+  - `directories`
+* `announcement`
+  - `service` definition
+- `discovery`
+* `commit`
+  - `config_files`: write config files to disk
+* `finalize`
+* `launch`
+
+As you can see, most layers have semantic names (`plugin`, `user`, etc); if you name your recipe correctly, they will be assigned to the correct phase. Otherwise, you can attach them explicitly to a semantic or non-semantic phase.
+
+    elasticsearch_datanode component:
+    
+    elasticsearch::default              # initial
+    elasticsearch::install_from_release #install phase
+    elasticsearch::plugins              # plugin phase
+
+    elasticsearch::server               # service definition; includes announcement
+    elasticsearch::config_files         # config_files; includes discovery
+
+I'm not clear on how much the phases should be strictly broken out into one-recipe-per-phase.
+
+It's also possible to instead define a chef resource that let you defer a block to a given phase from within a callback. It would be similar to a ruby block, but have more explicit timing. This may involve jacking around inside Chef, something we've avoided to now.
+
+__________________________________________________________________________
+
 Run List is 
 
     [role[systemwide], role[chef_client], role[ssh], role[nfs_client],

data/notes/design_notes-dsl_object.md

@@ -1,4 +1,7 @@
 
+property :color, :type => Integer
+property :color, :type => String
+property :color, :type => Pathname
 
 * is a subclass of extlib's `Mash` (Hash with indifferent access). 
   - in the 
@@ -35,14 +38,65 @@ __________________________________________________________________________
 ### setter/getter/block
 
 
-* `self.foo`                -- returns value of foo
-* `self.get(:foo)`          -- sets foo to val
-* `self.foo(val)`           -- sets foo to val
-* `self.set(:foo, val)`     -- sets foo to val
-* `self.foo ||= val`        -- sets foo to val if foo is unset, `false` or `nil`
-* `self.default(:foo, val)` -- sets foo to val if foo is unset
-* `self.unset(:foo)`        -- unsets foo
-   
+* `obj.foo`                -- returns value of foo
+* `obj.foo(val)`           -- sets foo to val; val must not be `nil`
+* `obj.unset(:foo)`        -- unsets foo
+
+want:
+
+* `obj.foo ||= val`        -- sets foo to val if foo is unset, `false` or `nil`
+* `obj.default(:foo, val)` -- sets foo to val if foo is unset
+
+provisionally, these are `protected`:
+
+* `obj.get(:foo)`          -- returns value of foo
+* `obj.set(:foo, val)`     -- sets foo to val
+
+### how hash-like is this?
+
+
+
+* obj.deep_get(:foo, :bar, :baz) # 
+* obj.deep_set(:foo, :bar, :baz, val) # sets foo bar baz. 
+  - where property has a type, it uses that type
+  - otherwise it uses Mash, and calls [] on it
+
+
+* TD votes NO on magic recursive hash behavior
+
+* `obj[:foo]`
+* `obj[:foo] = val`
+* `obj[:foo][:bar][:baz]`     -- when this has a value
+* `obj[:foo][:bar][:baz]`     -- when some or all of those have never been set?
+* `obj[:foo][:bar][:baz] = 1` -- when `obj[:foo]` has never been set
+* `obj[:foo][:bar][:baz] = 1` -- when `obj[:foo]` has never been set
+* `obj[:foo][:bar][:baz] ||= 1` -- **hard** ?I think?
+
+* `obj[:foo]`       -- nil
+* `obj[:foo][:bar]` -- raise
+* `obj[:foo][:bar] = 3`  -- 
+  - now obj[:foo][:bar] is 3 
+  - and obj[:foo] is a ?dsl_object?? but 
+  
+`obj[:foo][:bar]`  
+  
+Suppose `obj[:foo]` is set to a 
+
+Seems clear these should do the right thing:
+
+* `obj.merge`
+* `obj.reverse_merge`
+* `obj.keys`
+* `obj.values`
+* ... and a few more
+
+Also:
+
+* `obj.to_a`?
+* `obj.each`?
+* other crazy Enumerable properties?
+
+
 ### collection attributes
 
 
@@ -52,4 +106,4 @@ __________________________________________________________________________
 ### nested objects   
 
 
-### type co
+### type co

data/notes/homebase-layout.txt

@@ -0,0 +1,102 @@
+Ironfan Homebase Layout
+=======================
+
+Your Chef Homebase contains several directories, and each contains a README.md file describing its purpose and use in greater detail.
+
+This directory structure came out of a *lot* of trial and error, and is working very well where many others didn't. This homebase makes it easy to pull in third-party pantries (cookbook collections) and track upstream changes without being locked in.
+
+## Main Directories
+
+The main assets you'll use are:
+
+* `clusters/`   - Clusters fully describe your machines, from its construction ('an 8-core machine on the Amazon EC2 cloud') to its roles ('install Cassandra, Ganglia for monitoring, and silverware to manage its logs and firewall').
+* `cookbooks/`  - Cookbooks you download or create. Cookbooks install components, for example `cassandra` or `java`.
+* `roles/`      - Roles organize cookbooks and attribute overrides to describe the specific composition of your system. For example, you install Cassandra attaching the `cassandra_server` role to your machine. (.rb or .json files)
+
+These folders hold supporting files. You're less likely to visit here regularly.
+
+* `knife/`        - Chef and cloud configuration, and their myriad attendant credentials.
+* `environments/` - Organization-wide attribute values. (.json or .rb files)
+* `data_bags/`    - Data bags are an occasionally-useful alternative to node metadata for distributing information to your machines. (.json files)
+* `certificates/` - SSL certificates generated by `rake ssl_cert` live here.
+* `tasks/`        - Rake tasks for common administrative tasks.
+* `vendor/`       - cookbooks are checked out to `vendor`; symlinks in the `cookbooks/` directory select which ones will be deployed to chef server. The vendor directory comes with the Ironfan, Opscode and (should you be a lucky customer) Ironfan Enterprise chef pantries.
+* `notes/`        - a submoduled copy of the [ironfan wiki](http://github.com/infochimps-labs/ironfan/wiki`
+
+## Directory setup
+
+The core structure of the homebase is as follows ("├─*" means "git submodule'd"):
+
+    homebase
+    ├── clusters                        - cluster definition files
+    │   └── ( clusters )
+    ├── cookbooks                       - symlinks to cookbooks
+    │   ├── @vendor/ironfan-pantry/cookbooks/...
+    │   ├── @vendor/opscode/cookbooks/...
+    │   └── @vendor/org-pantry/cookbooks/...
+    ├── environments                    - environment definition files
+    │   └── ( environments )
+    ├── data_bags                       - symlinks to data_bags
+    │   ├── @vendor/ironfan-pantry/roles/...
+    │   └── @vendor/org-pantry/roles/...
+    ├── roles                           - symlinks to roles
+    │   ├── @vendor/ironfan-pantry/roles/...
+    │   └── @vendor/org-pantry/roles/...
+    ├── vendor
+    │   ├─* ironfan-pantry              - git submodule of https://github.com/infochimps-labs/ironfan-pantry
+    │   │   ├── cookbooks
+    │   │   ├── data_bags
+    │   │   ├── roles
+    │   │   └── tasks
+    │   ├─* ironfan-enterprise          - git submodule of ironfan-enterprise, if you're a lucky customer
+    │   │   ├── cookbooks
+    │   │   ├── data_bags
+    │   │   ├── roles
+    │   │   └── tasks
+    │   ├── opscode
+    │   │   └─* cookbooks               - git submodule of https://github.com/infochimps-labs/opscode_cookbooks; itself a closely-tracked fork of https://github.com/opscode/cookbooks
+    │   └── org-pantry                  - organization specific roles, cookbooks, etc.
+    │       ├── cookbooks
+    │       ├── data_bags
+    │       ├── roles
+    │       └── tasks
+    ├── knife                           - credentials (see below)
+    ├── certificates
+    ├── config
+    ├─* notes
+    ├── tasks
+    └── vagrants                        - vagrant files (when using ironfan-ci)
+
+The `vendor/opscode/cookbooks` and `vendor/ironfan-pantry` directories are actually [git submodules](http://help.github.com/submodules/). This makes it easy to track upstream changes in each collection while still maintaining your own modifications.
+
+We recommend you place your cookbooks in `vendor/org-pantry`. If you have cookbooks &c from other pantries that have significant changes, you can duplicate it into this `vendor/org-pantry` and simply change the symlink in homebase/cookbooks/.
+
+## Knife dir setup
+
+We recommend you version your credentials directory separately from your homebase. You will want to place it under version control, but you should not place it into a central git repository -- this holds the keys to the kingdom.
+
+We exclude the chef user key (`(user).pem`) and user-specific knife settings (`knife-user-(user).rb`) from the repo as well. Each user has their own revokable client key, and their own cloud credentials set, and those live nowhere but their own computers.
+
+    knife/
+    ├── knife.rb
+    ├── credentials -> (organization)-credentials
+    ├── (organization)-credentials
+    │   ├── knife-org.rb                - org-specific stuff, and cloud assets (elastic IPs, AMI image ids, etc)
+    │   ├── (user).pem                  - your chef client key *GITIGNORED*
+    │   ├── knife-user-(user).rb        - your user-specific knife customizations *GITIGNORED*
+    │   ├── (organization)-validator.pem- chef validator key, used to create client keys
+    │   ├── client_keys
+    │   │   └── (transient client keys) - you can delete these at will; only useful if you're debugging a bad bootstrap
+    │   ├── ec2_keys
+    │   │   └── (transient client keys) - ssh keys for cloud machines (in EC2 parlance, the private half of your keypair)
+    │   ├── certificates
+    │   ├── data_bag_keys
+    │   └── ec2_certs
+    ├── bootstrap                       - knife cluster bootstrap scripts
+    ├── plugins
+    │   └── knife
+    │       └── (knife plugins)         - knife plugins
+    └── .gitignore                      - make sure not to version the secret/user-specific stuff (*-keypairs, *-credentials.rb, knife-user-*.rb)
+
+    
+(You can safely ignore the directories above that aren't annotated; they're useful in certain advanced contexts but not immediately)

data/notes/knife-cluster-commands.md

@@ -0,0 +1,18 @@
+# Ironfan Knife Commands
+
+Available cluster subcommands: (for details, `knife SUB-COMMAND --help`)
+
+    knife cluster list (options)                                  - show available clusters
+    knife cluster bootstrap   CLUSTER-[FACET-[INDEXES]] (options) - bootstrap all servers described by given cluster slice
+    knife cluster kick        CLUSTER-[FACET-[INDEXES]] (options) - start a run of chef-client on each server, tailing the logs and exiting when the run completes.
+    knife cluster kill        CLUSTER-[FACET-[INDEXES]] (options) - kill all servers described by given cluster slice
+    knife cluster launch      CLUSTER-[FACET-[INDEXES]] (options) - Creates chef node and chef apiclient, pre-populates chef node, and instantiates in parallel their cloud machines. With --bootstrap flag, will ssh in to machines as they become ready and launch the bootstrap process
+    knife cluster proxy       CLUSTER-[FACET-[INDEXES]] (options) - Runs the ssh command to open a SOCKS proxy to the given host, and writes a PAC (automatic proxy config) file to /tmp/ironfan_proxy-YOURNAME.pac. Only the first host is used, even if multiple match.
+    knife cluster show        CLUSTER-[FACET-[INDEXES]] (options) - a helpful display of cluster's cloud and chef state
+    knife cluster ssh         CLUSTER-[FACET-[INDEXES]] COMMAND (options) - run an interactive ssh session, or execuse the given command, across a cluster slice
+    knife cluster start       CLUSTER-[FACET-[INDEXES]] (options) - start all servers described by given cluster slice
+    knife cluster stop        CLUSTER-[FACET-[INDEXES]] (options) - stop all servers described by given cluster slice
+    knife cluster sync        CLUSTER-[FACET-[INDEXES]] (options) - Update chef server and cloud machines with current cluster definition
+    knife cluster vagrant CMD CLUSTER-[FACET-[INDEXES]] (options) - runs the given command against a vagrant environment created from your cluster definition. EARLY, use at your own risk
+
+

data/notes/philosophy.md

@@ -0,0 +1,14 @@
+
+## Philosophy
+
+Some general principles of how we use chef.
+
+* *Chef server is never the repository of truth* -- it only mirrors the truth. A file is tangible and immediate to access.
+* Specifically, we want truth to live in the git repo, and be enforced by the chef server.  This means that everything is versioned, documented and exchangeable. *There is no truth but git, and chef is its messenger*.
+* *Systems, services and significant modifications cluster should be obvious from the `clusters` file*.  I don't want to have to bounce around nine different files to find out which thing installed a redis:server. The existence of anything that opens a port should be obvious when I look at the cluster file.
+* *Roles define systems, clusters assemble systems into a machine*.
+  - For example, a resque worker queue has a redis, a webserver and some config files -- your cluster should invoke a @whatever_queue@ role, and the @whatever_queue@ role should include recipes for the component services.
+  - the existence of anything that opens a port _or_ runs as a service should be obvious when I look at the roles file.
+* *include_recipe considered harmful* Do NOT use include_recipe for anything that a) provides a service, b) launches a daemon or c) is interesting in any way. (so: @include_recipe java@ yes; @include_recipe iptables@ no.) You should note the dependency in the metadata.rb. This seems weird, but the breaking behavior is purposeful: it makes you explicitly state all dependencies.
+* It's nice when *machines are in full control of their destiny*. Their initial setup (elastic IP, attaching a drive) is often best enforced externally. However, machines should be able independently assert things like load balancer registration which may change at any point in their lifetime.
+* It's even nicer, though, to have *full idempotency from the command line*: I can at any time push truth from the git repo to the chef server and know that it will take hold.