ironfan 4.7.6 → 4.7.7

data/CHANGELOG.md

@@ -1,3 +1,6 @@
+# v4.7.7
+* Allow per-ephemeral-disk options using :disks attribute (thanks @nickmarden)
+
 # v4.7.6
 * adding chef-client-nonce invocation to knife cluster kick
 

data/VERSION

@@ -1 +1 @@
-4.7.6
+4.7.7

data/ironfan.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = "ironfan"
-  s.version = "4.7.6"
+  s.version = "4.7.7"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Infochimps"]
-  s.date = "2013-01-25"
+  s.date = "2013-01-31"
   s.description = "Ironfan allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks."
   s.email = "coders@infochimps.com"
   s.extra_rdoc_files = [
@@ -90,10 +90,12 @@ Gem::Specification.new do |s|
     "lib/ironfan/provider/virtualbox.rb",
     "lib/ironfan/provider/virtualbox/machine.rb",
     "lib/ironfan/requirements.rb",
+    "notes/Future-development-proposals.md",
     "notes/Home.md",
     "notes/INSTALL-cloud_setup.md",
     "notes/INSTALL.md",
     "notes/Ironfan-Roadmap.md",
+    "notes/Upgrading-to-v4.md",
     "notes/advanced-superpowers.md",
     "notes/aws_servers.jpg",
     "notes/aws_user_key.png",
@@ -117,7 +119,6 @@ Gem::Specification.new do |s|
     "notes/silverware.md",
     "notes/style_guide.md",
     "notes/tips_and_troubleshooting.md",
-    "notes/version-3_2.md",
     "notes/walkthrough-hadoop.md",
     "notes/walkthrough-web.md",
     "spec/chef/cluster_bootstrap_spec.rb",

data/lib/ironfan/dsl/ec2.rb

@@ -104,7 +104,13 @@ module Ironfan
             mount_options       'defaults,noatime'
             tags({:bulk => true, :local => true, :fallback => true})
           end
-          ephemeral.receive! mount_ephemerals
+          ephemeral_attrs = mount_ephemerals.clone
+          if ephemeral_attrs.has_key?(:disks)
+            disk_attrs = mount_ephemerals[:disks][idx] || { }
+            ephemeral_attrs.delete(:disks)
+            ephemeral_attrs.merge!(disk_attrs)
+          end
+          ephemeral.receive! ephemeral_attrs
           result << ephemeral
         end
         result

data/notes/{version-3_2.md → Future-development-proposals.md}

@@ -1,16 +1,9 @@
+## Nathan
+* **Clusters from JSON** - this is theoretically quite easy, given the DSL's gorillib underpinnings.
 
-# v3.2.0 (future): Revamped undercarriage, spec coverage, standalone usage
-
-This is a Snow Leopard-style version change. No new features to speak of, but a much more solid and predictable foundation. 
-
-* **significantly cleaner DSL mixin**: uses the new, awesome `Gorillib::Builder`, giving it a much cleaner handling of fields and collections
-
-* **attributes are late-resolved**: in previous versions, the way you 'resolved' a server was to collapse the entire attribute set of cluster/facet/server hard onto the server model, a consistent source of bugs. Resolution is now done with the `Gorillib::Record::Overlay` mechanism, which means that you can set an attribute on the cluster and read it from the facet; change it later an all lower layers see the update.
-
+## Flip
 * **standalone usable**: can use ironfan-knife as a standalone library.
 
-# v3.3.x (future): Coherent universe of Servers, Components, Aspects
-
 * **spec coverage**: 
 
 * **coherent data model**: 

data/notes/Home.md

@@ -1,3 +1,8 @@
+>## **Ironfan: A Community Discussion Webinar**
+**<p>Thursday, January 31 @ 10a P, 12p C, 1p E</p>**
+Join Nathaniel Eliot, @temujin9, DevOps Engineer and lead on Ironfan, in this community discussion. Ironfan is a lightweight cluster orchestration toolset, built on top of Chef, which empowers spinning up of Hadoop clusters in under 20 minutes. Nathan has been responsible for Ironfan’s core plugin code, cookbooks, and other components to stabilize both Infochimps’ open source offerings, and internal architectures.
+[Register Now](https://www4.gotomeeting.com/register/188375087) 
+
 ## Overview
 
 Ironfan, the foundation of The Infochimps Platform, is an expressive toolset for constructing scalable, resilient architectures. It works in the cloud, in the data center, and on your laptop, and it makes your system diagram visible and inevitable. Inevitable systems coordinate automatically to interconnect, removing the hassle of manual configuration of connection points (and the associated danger of human error). For more information about Ironfan and the Infochimps Platform, visit [infochimps.com](https://www.infochimps.com).

data/notes/Upgrading-to-v4.md

@@ -0,0 +1,66 @@
+While the refactoring that lead to version 4 was intended to be as backwards compatible as possible, there have been some small but important changes to the way homebases and the DSL work.
+
+## Bundler
+Ironfan v4 uses bundler to manage its dependencies. In order to take advantage of it, the homebase's Gemfile should be updated to use ```gem 'ironfan', "~> 4.0"``` 
+
+We highly recommend that you run all your knife commands via bundle exec. This can be accomplished with an alias:
+```
+knife() {
+  bundle exec knife "$@"
+}
+```
+
+If you are comfortable with having bundle run every knife command (e.g. - you only have one homebase, or are using a Ironfan > 3.1.6 for all homebases you do use), you can add the above snippet to your .bashrc.
+
+## Vagrant
+Vagrant support has been discontinued for the time being. One of the first targets for the multicloud capabilities of Ironfan v4 will be a Virtualbox or Vagrant extension.
+
+## DSL Changes
+### Role implications removed
+In v3, certain roles could trigger further steps via role_implications.rb, which was used to add servers to corresponding EC2 Security Groups. This was deemed to be too risky and indirect, and has been removed for now. (A better mechanism for binding roles and provider-specific resources into repeatable components is being worked on.)
+
+If you used any of the roles below, you will probably want to add the following stanzas next to them in the clusters file, to replace the removed implications. **Be aware that EC2 instances can only be added to a security group at startup; if you fail to add the security groups before launch, you will have to kill and relaunch the machines to change them.**
+
+* `role :systemwide`
+```
+cloud(:ec2).security_group :systemwide
+```
+* `role :nfs_server`
+```
+cloud(:ec2).security_group(:nfs_server).authorize_group :nfs_client
+```
+* `role :nfs_client`
+```
+cloud(:ec2).security_group :nfs_client
+```
+* `role :ssh`
+```
+cloud(:ec2).security_group(:ssh).authorize_port_range 22..22
+```
+* `role :chef_server`
+```
+cloud(:ec2).security_group :chef_server do
+  authorize_port_range 4000..4000  # chef-server-api
+  authorize_port_range 4040..4040  # chef-server-webui
+end
+```
+* `role :web_server`
+```
+cloud(:ec2).security_group("#{self.cluster_name}-web_server") do
+  authorize_port_range  80..80
+  authorize_port_range 443..443
+end
+```
+* `role :redis_server`
+```
+cloud(:ec2).security_group("#{self.cluster_name}-redis_server") do
+  authorize_group("#{self.cluster_name}-redis_client")
+end
+```
+* `role :redis_client`
+```
+cloud(:ec2).security_group("#{self.cluster_name}-redis_client")
+```
+
+### Default statements removed
+Defaults should not need to be selected, and have been removed as a statement from the cluster DSL (in both cluster and volume). Although this is a non-breaking change, it has been flagged to raise a halting error, to alert people to the role_implications change above (which lacks well-defined indicators of its usage).

data/notes/ec2-pricing_and_capacity.md

@@ -1,35 +1,37 @@
 ## Compute Costs
 
 
-    code         	$/mo	 $/day	 $/hr	CPU/$	Mem/$	  mem 	  cpu 	cores	cpcore	storage	bits	IO      	type       	name
-    t1.micro       	  15	  0.48	  .02	   13	   13	  0.61	  0.25	 0.25	  1   	      0	   32	Low     	Micro      	Micro
-    m1.small       	  58	  1.92	  .08	   13	   21	  1.7 	  1   	 1   	  1   	    160	   32	Moderate	Standard   	Small
-    m1.medium      	 116	  3.84	  .165	   13	   13	  3.75	  2   	 2   	  1   	    410	   32	Moderate	Standard   	Medium
-    c1.medium      	 120	  3.96	  .17	   30	   10	  1.7 	  5   	 2   	  2.5 	    350	   32	Moderate	High-CPU   	Medium
-    m1.large       	 232	  7.68	  .32	   13	   23	  7.5 	  4   	 2   	  2   	    850	   64	High    	Standard   	Large
-    m2.xlarge      	 327	 10.80	  .45	   14	   38	 17.1 	  6.5 	 2   	  3.25	    420	   64	Moderate	High-Memory	Extra Large
-    m1.xlarge      	 465	 15.36	  .64	   13	   23	 15   	  8   	 4   	  2   	   1690	   64	High    	Standard   	Extra Large
-    c1.xlarge      	 479	 15.84	  .66	   30	   11	  7   	 20   	 8   	  2.5 	   1690	   64	High    	High-CPU   	Extra Large
-    m2.2xlarge     	 653	 21.60	  .90	   14	   38	 34.2 	 13   	 4   	  3.25	    850	   64	High    	High-Memory	Double Extra Large
-    cc1.4xlarge    	 944	 31.20	 1.30	   26	   18	 23   	 33.5 	 2   	 16.75	   1690	   64	10GB    	Compute    	Quadruple Extra Large
-    m2.4xlarge     	1307	 43.20	 1.80	   14	   38	 68.4 	 26   	 8   	  3.25	   1690	   64	High    	High-Memory	Quadruple Extra Large
-    cg1.4xlarge    	1525	 50.40	 2.10	   16	   10	 22   	 33.5 	 2   	 16.75	   1690	   64	10GB    	Cluster GPU	Quadruple Extra Large
-    cc2.8xlarge    	1742	 57.60	 2.40	   37	   25	 60.5 	 88   	 2   	 44   	   3370	   64	10GB    	Compute    	Eight Extra Large
-
-    dummy header ln	  15	  0.48	 0.02	12345	12345	  0.61	  0.25	 0.25	  1.00	6712345	32123	Low     	Micro      	Micro                
-
+    code         	$/mo	 $/day	 $/hr	Mem/$	CPU/$	  mem 	  cpu 	cores	cpcore	storage	 disks	 bits	ebs-opt	IO
+    t1.micro       	  15	  0.48	  .02	   13	   13	  0.61	  0.25	 0.25	  1   	      0	     0	   32	    -	Low
+    m1.small       	  47	  1.56	  .065	   26	   15	  1.7 	  1   	 1   	  1   	    160	     1	   32	    -	Moderate
+    m1.medium      	  95	  3.12	  .13 	   15	   15	  3.75	  2   	 2   	  1   	    410	     1	   32	    -	Moderate
+    c1.medium      	 124	  4.08	  .165	   10	   30	  1.7 	  5   	 2   	  2.5 	    350	     1	   32	    -	Moderate
+    m1.large       	 190	  6.24	  .26 	   29	   15	  7.5 	  4   	 2   	  2   	    850	     2	   64	  500	High
+    m2.xlarge      	 329	 10.80	  .45 	   38	   14	 17.1 	  6.5 	 2   	  3.25	    420	     1	   64	    -	Moderate
+    m1.xlarge      	 380	 12.48	  .52 	   29	   15	 15   	  8   	 4   	  2   	   1690	     4	   64	 1000	High
+    m3.xlarge      	 424	 13.92	  .58	   26	   22	 15   	 13   	 4   	  3.25	      0	     0	   64	    -	Moderate
+    c1.xlarge      	 482	 15.84	  .66	   11	   30	  7   	 20   	 8   	  2.5 	   1690	     4	   64	    -	High
+    m2.2xlarge     	 658	 21.60	  .90	   38	   14	 34.2 	 13   	 4   	  3.25	    850	     2	   64	    -	High
+    m3.2xlarge     	 847	 27.84	 1.16	   26	   22	 30   	 26   	 8   	  3.25	      0	     0	   64	    -	High
+    cc1.4xlarge    	 950	 31.20	 1.30	   18	   26	 23   	 33.5 	 8   	  4.2 	   1690	     4	   64	    -	10GB
+    m2.4xlarge     	1315	 43.20	 1.80	   38	   14	 68.4 	 26   	 8   	  3.25	   1690	     2	   64	 1000	High
+    cg1.4xlarge    	1534	 50.40	 2.10	   10	   16	 22   	 33.5 	 8   	  4.2 	   1690	     4	   64	    -	10GB
+    cc2.8xlarge    	1753	 57.60	 2.40	   25	   37	 60.5 	 88   	16   	  5.5 	   3370	     2	   64	    -	10GB
+    hi1.4xlarge    	2265	 74.40	 3.10	   20	   11	 60.5 	 35   	16   	  2.2 	   2048	 ssd 2	   64	    -	10GB
+    cr1.8xlarge    	2557	 84.00	 3.50	   70	   25	244   	 88   	16   	  5.5 	    240	 ssd 2	   64	    -	10GB
+    hs1.8xlarge    	3361	110.40	 4.60	   25	    8	117   	 35   	16    	  2.2 	  49152	    24	   64	    -	10GB
 
 ## Storage Costs
 
-                            $/GB..mo		$/GB.mo  	$/Mio	
-    EBS Volume     			$0.10				
+                            $/GB..mo		$/GB.mo  	$/Mio
+    EBS Volume     			$0.10
     EBS I/O       			            	         	$0.10
-    EBS Snapshot S3			$0.083				
+    EBS Snapshot S3			$0.083
 
                          	Std $/GB.mo		Red.Red. $/GB.mo
     S3 1st tb            	$0.125     	 	$0.093
-    S3 next 49tb         	$0.110     	 	$0.083 
-    S3 next 450tb        	$0.095  		$0.073 
+    S3 next 49tb         	$0.110     	 	$0.083
+    S3 next 450tb        	$0.095  		$0.073
 
 ### Storing 1TB data
 
@@ -51,12 +53,12 @@ NOTE: For current pricing information, be sure to check Amazon EC2 Pricing.
 The cost of an EBS Volume is $0.10/GB per month.  You are responsible for paying for the amount of disk space that you reserve, not for the amount of the disk space that you actually use.  If you reserve a 1TB volume, but only use 1GB, you will be paying for 1TB.
 * $0.10/GB per month of provisioned storage
 * $0.10/GB per 1 million I/O requests
- 
+
 #### Transaction Costs
 
 In addition to the storage cost for EBS Volumes, you will also be charged for I/O transactions. The cost is $0.10 per million I/O transactions, where one transaction is equivalent to one read or write.  This number may be smaller than the actual number of transactions performed by your application because of the Linux cache for all file systems.
 $0.10 per 1 million I/O requests
- 
+
 #### S3 Snapshot Costs
 
 Snapshot costs are compressed and based on altered blocks from the previous snapshot backup.  Files that have altered blocks on the disk and then been deleted will add cost to the Snapshots for example.  Remember, snapshots are at the data block level.
@@ -65,5 +67,3 @@ $0.01 per 1,000 PUT requests (when saving a snapshot)
 $0.01 per 10,000 GET requests (when loading a snapshot)
 
 NOTE:  Payment charges stop the moment you delete a volume.  If you delete a volume and the status appears as "deleting" for an extended period of time, you will not be charged for the time needed to complete the deletion.
-
- 

data/notes/knife-cluster-commands.md

@@ -1,5 +1,7 @@
 # Ironfan Knife Commands
 
+## Available Commands
+
 Available cluster subcommands: (for details, `knife SUB-COMMAND --help`)
 
     knife cluster list (options)                                  - show available clusters
@@ -15,4 +17,5 @@ Available cluster subcommands: (for details, `knife SUB-COMMAND --help`)
     knife cluster sync        CLUSTER-[FACET-[INDEXES]] (options) - Update chef server and cloud machines with current cluster definition
     knife cluster vagrant CMD CLUSTER-[FACET-[INDEXES]] (options) - runs the given command against a vagrant environment created from your cluster definition. EARLY, use at your own risk
 
+## Examples
 

data/notes/walkthrough-web.md

@@ -40,7 +40,7 @@ The dbnode facet has one server, which will be named `web_demo-dbnode-0`; the we
 
 Each server inherits the appropriate behaviors from its facet and cluster. All the servers in this cluster have the `base_role`, `chef_client` and `ssh` roles. The dbnode machines additionally house a MySQL server, while the webnodes have an nginx reverse proxy for the custom `web_demo_webapp`.
 
-As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the server, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
+As you can see, the dbnode facet asks for a different flavor of machine (`m1.large`) than the cluster default (`t1.micro`). Settings in the facet override those in the cluster, and settings in the server override those of its facet. You economically describe only what's significant about each machine.
 
 ### Cluster-level tools
 

data/spec/ironfan/ec2/cloud_provider_spec.rb

@@ -13,6 +13,10 @@ describe Ironfan::Dsl::Cluster do
 
       facet :web do
         instances 3
+        cloud(:ec2) do
+          flavor 'm1.small'
+          mount_ephemerals({ :disks => { 0 => { :mount_point => '/data' } } })
+        end
       end
 
     end
@@ -30,6 +34,10 @@ describe Ironfan::Dsl::Cluster do
     it 'should have one cloud provider, EC2' do
       @facet.servers[0].clouds.keys.should == [ :ec2 ]
     end
+
+    it 'should have its first ephemeral disk mounted at /data' do
+      @facet.servers[0].implied_volumes[1].mount_point.should == '/data'
+    end
   end
 
 end

metadata

@@ -2,7 +2,7 @@
 name: ironfan
 version: !ruby/object:Gem::Version 
   prerelease: 
-  version: 4.7.6
+  version: 4.7.7
 platform: ruby
 authors: 
 - Infochimps
@@ -10,7 +10,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2013-01-25 00:00:00 Z
+date: 2013-01-31 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: chef
@@ -216,10 +216,12 @@ files:
 - lib/ironfan/provider/virtualbox.rb
 - lib/ironfan/provider/virtualbox/machine.rb
 - lib/ironfan/requirements.rb
+- notes/Future-development-proposals.md
 - notes/Home.md
 - notes/INSTALL-cloud_setup.md
 - notes/INSTALL.md
 - notes/Ironfan-Roadmap.md
+- notes/Upgrading-to-v4.md
 - notes/advanced-superpowers.md
 - notes/aws_servers.jpg
 - notes/aws_user_key.png
@@ -243,7 +245,6 @@ files:
 - notes/silverware.md
 - notes/style_guide.md
 - notes/tips_and_troubleshooting.md
-- notes/version-3_2.md
 - notes/walkthrough-hadoop.md
 - notes/walkthrough-web.md
 - spec/chef/cluster_bootstrap_spec.rb
@@ -282,7 +283,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
-      hash: -1978687581575052392
+      hash: 1940435726309718533
       segments: 
       - 0
       version: "0"