ironfan 4.3.4 → 4.4.0

data/notes/design_notes-meta_discovery.md

@@ -0,0 +1,59 @@
+@temujin9 has proposed, and it's a good propose, that there should exist such a thing as an 'integration cookbook'.
+
+The hadoop_cluster cookbook should describe the hadoop_cluster, the ganglia cookbook ganglia, and the zookeeper cookbook zookeeper. Each should provide hooks that are neighborly but not exhibitionist, but should mind its own business.
+
+The job of tying those components together should belong to a separate concern. It should know how and when to copy hbase jars into the pig home dir, or what cluster service_provide'r a redis client should reference.
+
+## Practical implications
+
+* I'm going to revert out the `node[:zookeeper][:cluster_name]` attributes -- services should always announce under their cluster.
+
+* Until we figure out how and when to separate integrations, I'm going to isolate entanglements into their own recipes within cookbooks: so, the ganglia part of hadoop will become `ganglia_integration` or somesuch.
+
+## Example integrations
+
+### Copying jars
+
+Pig needs jars from hbase and zookeeper.  They should announce they have jars; pig should announce its home directory;  the integration should decide how and where to copy the jars.
+
+### Reference a service
+
+Right now in several places we have attributes like `node[:zookeeper][:cluster_name]`, used to specify the cluster that provides_service zookeeper.
+
+* Server recipes should never use `node[:zookeeper][:cluster_name]` -- they should always announce under their native cluster. (I'd kinda like to give `provides_service` some sugar to assume the cluster name, but need to find something backwards-compatible to use)
+
+* Need to take a better survey of usage among clients to determine how to do this.
+
+* cases:
+  - hbase cookbook refs: hadoop, zookeeper, ganglia
+  - flume cookbook refs: zookeeper, ganglia. 
+  - flume agents may reference several different flume provides_service'rs
+  - API using two different elasticsearch clusters
+  
+### Logging, monitoring
+
+* tell flume you have logs to pick up
+* tell ganglia to monitor you
+
+### Service Dashboard
+
+Let everything with a dashboard say so, and then let one resource create a page that links to each.
+
+
+________________________
+
+These are still forming, ideas welcome.
+
+
+
+
+Sometimes we want 
+
+* if there are volumes marked for 'mysql-table_data', use that; otherwise, the 'persistent' datastore, if any; else the 'bulk' datastore, if any; else the 'fallback' datastore (which is guaranteed to exist).
+
+* IP addresses (or hostnames):
+  - `[:private_ip, :public_ip]`
+  - `[:private_ip]`
+  - `:primary_ip`
+  
+* .  

data/notes/ec2-pricing_and_capacity.md

@@ -0,0 +1,69 @@
+## Compute Costs
+
+
+    code         	$/mo	 $/day	 $/hr	CPU/$	Mem/$	  mem 	  cpu 	cores	cpcore	storage	bits	IO      	type       	name
+    t1.micro       	  15	  0.48	  .02	   13	   13	  0.61	  0.25	 0.25	  1   	      0	   32	Low     	Micro      	Micro
+    m1.small       	  58	  1.92	  .08	   13	   21	  1.7 	  1   	 1   	  1   	    160	   32	Moderate	Standard   	Small
+    m1.medium      	 116	  3.84	  .165	   13	   13	  3.75	  2   	 2   	  1   	    410	   32	Moderate	Standard   	Medium
+    c1.medium      	 120	  3.96	  .17	   30	   10	  1.7 	  5   	 2   	  2.5 	    350	   32	Moderate	High-CPU   	Medium
+    m1.large       	 232	  7.68	  .32	   13	   23	  7.5 	  4   	 2   	  2   	    850	   64	High    	Standard   	Large
+    m2.xlarge      	 327	 10.80	  .45	   14	   38	 17.1 	  6.5 	 2   	  3.25	    420	   64	Moderate	High-Memory	Extra Large
+    m1.xlarge      	 465	 15.36	  .64	   13	   23	 15   	  8   	 4   	  2   	   1690	   64	High    	Standard   	Extra Large
+    c1.xlarge      	 479	 15.84	  .66	   30	   11	  7   	 20   	 8   	  2.5 	   1690	   64	High    	High-CPU   	Extra Large
+    m2.2xlarge     	 653	 21.60	  .90	   14	   38	 34.2 	 13   	 4   	  3.25	    850	   64	High    	High-Memory	Double Extra Large
+    cc1.4xlarge    	 944	 31.20	 1.30	   26	   18	 23   	 33.5 	 2   	 16.75	   1690	   64	10GB    	Compute    	Quadruple Extra Large
+    m2.4xlarge     	1307	 43.20	 1.80	   14	   38	 68.4 	 26   	 8   	  3.25	   1690	   64	High    	High-Memory	Quadruple Extra Large
+    cg1.4xlarge    	1525	 50.40	 2.10	   16	   10	 22   	 33.5 	 2   	 16.75	   1690	   64	10GB    	Cluster GPU	Quadruple Extra Large
+    cc2.8xlarge    	1742	 57.60	 2.40	   37	   25	 60.5 	 88   	 2   	 44   	   3370	   64	10GB    	Compute    	Eight Extra Large
+
+    dummy header ln	  15	  0.48	 0.02	12345	12345	  0.61	  0.25	 0.25	  1.00	6712345	32123	Low     	Micro      	Micro                
+
+
+## Storage Costs
+
+                            $/GB..mo		$/GB.mo  	$/Mio	
+    EBS Volume     			$0.10				
+    EBS I/O       			            	         	$0.10
+    EBS Snapshot S3			$0.083				
+
+                         	Std $/GB.mo		Red.Red. $/GB.mo
+    S3 1st tb            	$0.125     	 	$0.093
+    S3 next 49tb         	$0.110     	 	$0.083 
+    S3 next 450tb        	$0.095  		$0.073 
+
+### Storing 1TB data
+
+(Cost of storage, neglecting I/O costs, and assuming the ratio of EBS volume size to snapshot size is as given)
+
+* http://aws.amazon.com/ec2/instance-types/
+* http://aws.amazon.com/ec2/#pricing
+
+### How much does EBS cost?
+
+The costs of EBS will be similar to the pricing structure of data storage on S3.  There are three types of costs associated with EBS.
+
+Storage Cost + Transaction Cost + S3 Snapshot Cost = Total Cost of EBS
+
+NOTE: For current pricing information, be sure to check Amazon EC2 Pricing.
+
+#### Storage Costs
+
+The cost of an EBS Volume is $0.10/GB per month.  You are responsible for paying for the amount of disk space that you reserve, not for the amount of the disk space that you actually use.  If you reserve a 1TB volume, but only use 1GB, you will be paying for 1TB.
+* $0.10/GB per month of provisioned storage
+* $0.10/GB per 1 million I/O requests
+ 
+#### Transaction Costs
+
+In addition to the storage cost for EBS Volumes, you will also be charged for I/O transactions. The cost is $0.10 per million I/O transactions, where one transaction is equivalent to one read or write.  This number may be smaller than the actual number of transactions performed by your application because of the Linux cache for all file systems.
+$0.10 per 1 million I/O requests
+ 
+#### S3 Snapshot Costs
+
+Snapshot costs are compressed and based on altered blocks from the previous snapshot backup.  Files that have altered blocks on the disk and then been deleted will add cost to the Snapshots for example.  Remember, snapshots are at the data block level.
+$0.15 per GB-month of data stored
+$0.01 per 1,000 PUT requests (when saving a snapshot)
+$0.01 per 10,000 GET requests (when loading a snapshot)
+
+NOTE:  Payment charges stop the moment you delete a volume.  If you delete a volume and the status appears as "deleting" for an extended period of time, you will not be charged for the time needed to complete the deletion.
+
+ 

data/notes/homebase-layout.txt

@@ -0,0 +1,102 @@
+Ironfan Homebase Layout
+=======================
+
+Your Chef Homebase contains several directories, and each contains a README.md file describing its purpose and use in greater detail.
+
+This directory structure came out of a *lot* of trial and error, and is working very well where many others didn't. This homebase makes it easy to pull in third-party pantries (cookbook collections) and track upstream changes without being locked in.
+
+## Main Directories
+
+The main assets you'll use are:
+
+* `clusters/`   - Clusters fully describe your machines, from its construction ('an 8-core machine on the Amazon EC2 cloud') to its roles ('install Cassandra, Ganglia for monitoring, and silverware to manage its logs and firewall').
+* `cookbooks/`  - Cookbooks you download or create. Cookbooks install components, for example `cassandra` or `java`.
+* `roles/`      - Roles organize cookbooks and attribute overrides to describe the specific composition of your system. For example, you install Cassandra attaching the `cassandra_server` role to your machine. (.rb or .json files)
+
+These folders hold supporting files. You're less likely to visit here regularly.
+
+* `knife/`        - Chef and cloud configuration, and their myriad attendant credentials.
+* `environments/` - Organization-wide attribute values. (.json or .rb files)
+* `data_bags/`    - Data bags are an occasionally-useful alternative to node metadata for distributing information to your machines. (.json files)
+* `certificates/` - SSL certificates generated by `rake ssl_cert` live here.
+* `tasks/`        - Rake tasks for common administrative tasks.
+* `vendor/`       - cookbooks are checked out to `vendor`; symlinks in the `cookbooks/` directory select which ones will be deployed to chef server. The vendor directory comes with the Ironfan, Opscode and (should you be a lucky customer) Ironfan Enterprise chef pantries.
+* `notes/`        - a submoduled copy of the [ironfan wiki](http://github.com/infochimps-labs/ironfan/wiki`
+
+## Directory setup
+
+The core structure of the homebase is as follows ("├─*" means "git submodule'd"):
+
+    homebase
+    ├── clusters                        - cluster definition files
+    │   └── ( clusters )
+    ├── cookbooks                       - symlinks to cookbooks
+    │   ├── @vendor/ironfan-pantry/cookbooks/...
+    │   ├── @vendor/opscode/cookbooks/...
+    │   └── @vendor/org-pantry/cookbooks/...
+    ├── environments                    - environment definition files
+    │   └── ( environments )
+    ├── data_bags                       - symlinks to data_bags
+    │   ├── @vendor/ironfan-pantry/roles/...
+    │   └── @vendor/org-pantry/roles/...
+    ├── roles                           - symlinks to roles
+    │   ├── @vendor/ironfan-pantry/roles/...
+    │   └── @vendor/org-pantry/roles/...
+    ├── vendor
+    │   ├─* ironfan-pantry              - git submodule of https://github.com/infochimps-labs/ironfan-pantry
+    │   │   ├── cookbooks
+    │   │   ├── data_bags
+    │   │   ├── roles
+    │   │   └── tasks
+    │   ├─* ironfan-enterprise          - git submodule of ironfan-enterprise, if you're a lucky customer
+    │   │   ├── cookbooks
+    │   │   ├── data_bags
+    │   │   ├── roles
+    │   │   └── tasks
+    │   ├── opscode
+    │   │   └─* cookbooks               - git submodule of https://github.com/infochimps-labs/opscode_cookbooks; itself a closely-tracked fork of https://github.com/opscode/cookbooks
+    │   └── org-pantry                  - organization specific roles, cookbooks, etc.
+    │       ├── cookbooks
+    │       ├── data_bags
+    │       ├── roles
+    │       └── tasks
+    ├── knife                           - credentials (see below)
+    ├── certificates
+    ├── config
+    ├─* notes
+    ├── tasks
+    └── vagrants                        - vagrant files (when using ironfan-ci)
+
+The `vendor/opscode/cookbooks` and `vendor/ironfan-pantry` directories are actually [git submodules](http://help.github.com/submodules/). This makes it easy to track upstream changes in each collection while still maintaining your own modifications.
+
+We recommend you place your cookbooks in `vendor/org-pantry`. If you have cookbooks &c from other pantries that have significant changes, you can duplicate it into this `vendor/org-pantry` and simply change the symlink in homebase/cookbooks/.
+
+## Knife dir setup
+
+We recommend you version your credentials directory separately from your homebase. You will want to place it under version control, but you should not place it into a central git repository -- this holds the keys to the kingdom.
+
+We exclude the chef user key (`(user).pem`) and user-specific knife settings (`knife-user-(user).rb`) from the repo as well. Each user has their own revokable client key, and their own cloud credentials set, and those live nowhere but their own computers.
+
+    knife/
+    ├── knife.rb
+    ├── credentials -> (organization)-credentials
+    ├── (organization)-credentials
+    │   ├── knife-org.rb                - org-specific stuff, and cloud assets (elastic IPs, AMI image ids, etc)
+    │   ├── (user).pem                  - your chef client key *GITIGNORED*
+    │   ├── knife-user-(user).rb        - your user-specific knife customizations *GITIGNORED*
+    │   ├── (organization)-validator.pem- chef validator key, used to create client keys
+    │   ├── client_keys
+    │   │   └── (transient client keys) - you can delete these at will; only useful if you're debugging a bad bootstrap
+    │   ├── ec2_keys
+    │   │   └── (transient client keys) - ssh keys for cloud machines (in EC2 parlance, the private half of your keypair)
+    │   ├── certificates
+    │   ├── data_bag_keys
+    │   └── ec2_certs
+    ├── bootstrap                       - knife cluster bootstrap scripts
+    ├── plugins
+    │   └── knife
+    │       └── (knife plugins)         - knife plugins
+    └── .gitignore                      - make sure not to version the secret/user-specific stuff (*-keypairs, *-credentials.rb, knife-user-*.rb)
+
+    
+(You can safely ignore the directories above that aren't annotated; they're useful in certain advanced contexts but not immediately)

data/notes/knife-cluster-commands.md

@@ -0,0 +1,18 @@
+# Ironfan Knife Commands
+
+Available cluster subcommands: (for details, `knife SUB-COMMAND --help`)
+
+    knife cluster list (options)                                  - show available clusters
+    knife cluster bootstrap   CLUSTER-[FACET-[INDEXES]] (options) - bootstrap all servers described by given cluster slice
+    knife cluster kick        CLUSTER-[FACET-[INDEXES]] (options) - start a run of chef-client on each server, tailing the logs and exiting when the run completes.
+    knife cluster kill        CLUSTER-[FACET-[INDEXES]] (options) - kill all servers described by given cluster slice
+    knife cluster launch      CLUSTER-[FACET-[INDEXES]] (options) - Creates chef node and chef apiclient, pre-populates chef node, and instantiates in parallel their cloud machines. With --bootstrap flag, will ssh in to machines as they become ready and launch the bootstrap process
+    knife cluster proxy       CLUSTER-[FACET-[INDEXES]] (options) - Runs the ssh command to open a SOCKS proxy to the given host, and writes a PAC (automatic proxy config) file to /tmp/ironfan_proxy-YOURNAME.pac. Only the first host is used, even if multiple match.
+    knife cluster show        CLUSTER-[FACET-[INDEXES]] (options) - a helpful display of cluster's cloud and chef state
+    knife cluster ssh         CLUSTER-[FACET-[INDEXES]] COMMAND (options) - run an interactive ssh session, or execuse the given command, across a cluster slice
+    knife cluster start       CLUSTER-[FACET-[INDEXES]] (options) - start all servers described by given cluster slice
+    knife cluster stop        CLUSTER-[FACET-[INDEXES]] (options) - stop all servers described by given cluster slice
+    knife cluster sync        CLUSTER-[FACET-[INDEXES]] (options) - Update chef server and cloud machines with current cluster definition
+    knife cluster vagrant CMD CLUSTER-[FACET-[INDEXES]] (options) - runs the given command against a vagrant environment created from your cluster definition. EARLY, use at your own risk
+
+

data/notes/named-cloud-objects.md

@@ -0,0 +1,11 @@
+# Named Cloud Objects
+
+To add a new machine image, place this snippet:
+
+    Chef::Config[:ec2_image_info] ||= {}
+    Chef::Config[:ec2_image_info].merge!({
+      # ... lines like this:
+      # %w[ us-west-1 64-bit ebs natty ] => { :image_id => 'ami-4d580408' },
+    })
+
+in your knife.rb or whereever. ironfan will notice that it exists and add to it, rather than clobbering it.

data/notes/philosophy.md

@@ -0,0 +1,13 @@
+## Philosophy
+
+Some general principles of how we use Chef.
+
+* *Chef server is never the repository of truth* -- it only mirrors the truth. A file is tangible and immediate to access.
+* Specifically, we want truth to live in the git repo, and be enforced by the Chef server.  This means that everything is versioned, documented and exchangeable. *There is no truth but git, and Chef is its messenger*.
+* *Systems, services and significant modifications cluster should be obvious from the `clusters` file*.  I don't want to have to bounce around nine different files to find out which thing installed a redis:server. The existence of anything that opens a port should be obvious when I look at the cluster file.
+* *Roles define systems, clusters assemble systems into a machine*.
+  - For example, a resque worker queue has a redis, a webserver and some config files -- your cluster should invoke a @whatever_queue@ role, and the @whatever_queue@ role should include recipes for the component services.
+  - the existence of anything that opens a port _or_ runs as a service should be obvious when I look at the roles file.
+* *include_recipe considered harmful* Do NOT use include_recipe for anything that a) provides a service, b) launches a daemon or c) is interesting in any way. (so: @include_recipe java@ yes; @include_recipe iptables@ no.) You should note the dependency in the metadata.rb. This seems weird, but the breaking behavior is purposeful: it makes you explicitly state all dependencies.
+* It's nice when *machines are in full control of their destiny*. Their initial setup (elastic IP, attaching a drive) is often best enforced externally. However, machines should be able independently assert things like load balancer registration which may change at any point in their lifetime.
+* It's even nicer, though, to have *full idempotency from the command line*: I can at any time push truth from the git repo to the Chef server and know that it will take hold.

data/notes/rake_tasks.md

@@ -0,0 +1,24 @@
+
+Rake Tasks
+==========
+
+The homebase contains a `Rakefile` that includes tasks that are installed with the Chef libraries. To view the tasks available with in the homebase with a brief description, run `rake -T`.
+
+Besides your `~/.chef/knife.rb` file, the Rakefile loads `config/rake.rb`, which sets:
+
+* Constants used in the `ssl_cert` task for creating the certificates.
+* Constants that set the directory locations used in various tasks.
+
+If you use the `ssl_cert` task, change the values in the `config/rake.rb` file appropriately. These values were also used in the `new_cookbook` task, but that task is replaced by the `knife cookbook create` command which can be configured below.
+
+The default task (`default`) is run when executing `rake` with no arguments. It will call the task `test_cookbooks`.
+
+The following standard Chef tasks are typically accomplished using the rake file:
+
+* `bundle_cookbook[cookbook]` - Creates cookbook tarballs in the `pkgs/` dir.
+* `install` - Calls `update`, `roles` and `upload_cookbooks` Rake tasks.
+* `ssl_cert` - Create self-signed SSL certificates in `certificates/` dir.
+* `update` - Update the homebase from source control server, understands git and svn.
+* `roles` - iterates over the roles and uploads with `knife role from file`.
+
+Most other tasks use knife: run a bare `knife cluster`, `knife cookbook` (etc) to find out more.

data/notes/renamed-recipes.txt

@@ -0,0 +1,142 @@
+cassandra      	:: default                 	| 
+cassandra      	:: add_apt_repo            	| new
+cassandra      	:: install_from_git        	| 
+cassandra      	:: install_from_package    	| 
+cassandra      	:: install_from_release    	| 
+cassandra      	:: config_from_data_bag       	| autoconf
+cassandra      	:: client                  	| 
+cassandra      	:: server                  	| 
+cassandra      	:: authentication          	| not include_recipe'd -- added to role
+cassandra      	:: bintools                	| 
+cassandra      	:: ec2snitch               	| 
+cassandra      	:: jna_support             	| 
+cassandra      	:: mx4j                    	| 
+cassandra      	:: iptables                	| 
+cassandra      	:: ruby_client             	| 
+cassandra      	:: config_files            	| new
+
+elasticsearch  	:: default                 	| 
+elasticsearch  	:: install_from_git        	| 
+elasticsearch  	:: install_from_release    	| 
+elasticsearch  	:: plugins                 	| install_plugins
+elasticsearch  	:: server                  	| 
+elasticsearch  	:: client                  	| 
+elasticsearch  	:: load_balancer           	| 
+elasticsearch  	:: config_files               	| config
+
+flume          	:: default                 	| 
+flume          	:: master                  	| 
+flume          	:: agent                   	| node
+flume          	:: plugin-hbase_sink       	| hbase_sink_plugin
+flume          	:: plugin-jruby            	| jruby_plugin     
+flume          	:: test_flow               	| 
+flume          	:: test_s3_source          	| 
+flume          	:: config_files              	| config
+
+ganglia        	:: agent                   	| 
+ganglia        	:: default                 	| 
+ganglia        	:: server                  	| 
+ganglia        	:: config_files            	| new
+
+graphite       	:: default                 	| 
+graphite       	:: carbon                  	| 
+graphite       	:: ganglia                 	| 
+graphite       	:: dashboard                  	| web
+graphite       	:: whisper                 	| 
+
+hadoop_cluster 	:: default                 	| 
+hadoop_cluster 	:: add_cloudera_repo       	| 
+hadoop_cluster 	:: datanode                	| 
+hadoop_cluster 	:: doc                     	| 
+hadoop_cluster 	:: hdfs_fuse               	| 
+hadoop_cluster 	:: jobtracker              	| 
+hadoop_cluster 	:: namenode                	| 
+hadoop_cluster 	:: secondarynn             	| 
+hadoop_cluster 	:: tasktracker             	| 
+hadoop_cluster 	:: wait_on_hdfs_safemode     	| 
+hadoop_cluster 	:: fake_topology           	| 
+hadoop_cluster 	:: minidash                	| 
+hadoop_cluster 	:: config_files            	| cluster_conf
+
+hbase          	:: default                 	| 
+hbase          	:: master                  	| 
+hbase          	:: minidash                	| 
+hbase          	:: regionserver            	| 
+hbase          	:: stargate                	| 
+hbase          	:: thrift                  	| 
+hbase          	:: backup_tables           	| 
+hbase          	:: config_files              	| config
+
+jenkins        	:: default                 	| 
+jenkins        	:: server                  	| 
+jenkins        	:: user_key                	| 
+jenkins        	:: node_ssh                	| 
+jenkins        	:: osx_worker              	| 
+jenkins        	:: build_from_github       	| 
+jenkins        	:: build_ruby_rspec        	|
+jenkins        	:: auth_github_oauth       	| 
+jenkins        	:: plugins                 	| 
+#
+jenkins        	:: add_apt_repo            	| 
+jenkins        	:: iptables                	| 
+jenkins        	:: node_jnlp               	| 
+jenkins        	:: node_windows            	| 
+jenkins        	:: proxy_apache2           	| 
+jenkins        	:: proxy_nginx             	| 
+
+minidash       	:: default                 	| 
+minidash       	:: server                  	| 
+
+mongodb        	:: default                 	| 
+mongodb        	:: apt                     	| add_apt_repo
+mongodb        	:: install_from_release       	| source
+mongodb        	:: backup                  	| 
+mongodb        	:: config_server           	| fixme
+mongodb        	:: mongos                  	| fixme
+mongodb        	:: server                  	| 
+
+nfs            	:: client                  	| 
+nfs            	:: default                 	| 
+nfs            	:: server                  	| 
+
+redis          	:: default                 	| 
+redis          	:: install_from_package    	| 
+redis          	:: install_from_release    	| 
+redis          	:: client                  	| 
+redis          	:: server                  	| 
+
+resque         	:: default                 	| 
+resque         	:: dedicated_redis         	| 
+resque         	:: dashboard               	| 
+
+route53        	:: default                 	| 
+route53        	:: set_hostname                	| ec2
+
+statsd         	:: default                 	| 
+statsd         	:: server                  	| 
+
+volumes        	:: default                 	| 
+volumes        	:: build_raid              	| 
+volumes        	:: format                  	| 
+volumes        	:: mount                   	| 
+volumes        	:: resize                  	| 
+volumes_ebs    	:: default                 	| 
+volumes_ebs    	:: attach_ebs              	| 
+
+zabbix         	:: agent                   	| 
+zabbix         	:: agent_prebuild          	| 
+zabbix         	:: agent_source            	| 
+zabbix         	:: database                	| 
+zabbix         	:: database_mysql          	| 
+zabbix         	:: default                 	| 
+zabbix         	:: firewall                	| 
+zabbix         	:: server                  	| 
+zabbix         	:: server_source           	| 
+zabbix         	:: web                     	| 
+zabbix         	:: web_apache              	| 
+zabbix         	:: web_nginx               	| 
+
+zookeeper      	:: default                 	| 
+zookeeper      	:: client                  	| 
+zookeeper      	:: server                  	| 
+zookeeper      	:: config_files            	|