ironfan 4.0.2 → 4.0.3

data/README.md

@@ -45,4 +45,4 @@ Ironfan consists of the following Toolset:
 
 ### Getting Help
 * Feel free to contact us at info@infochimps.com or 855-DATA-FUN
-* Also, you invited to a [private consultation](http://www.infochimps.com/free-big-data-consultation?utm_source=git&utm_medium=referral&utm_campaign=consult) with Infochimps founders on your big data project.
+* Also, you're invited to a [private consultation](http://www.infochimps.com/free-big-data-consultation?utm_source=git&utm_medium=referral&utm_campaign=consult) with Infochimps founders on your big data project.

data/TODO.md

@@ -2,57 +2,19 @@
 
 * We should use the `Fog::Compute::AWS::FLAVORS` constant that [fog defines](http://rubydoc.info/github/fog/fog/master/Fog/Compute/AWS) in the cloud code (instead of the one we put there)
 
-* All over the place there is the following construct (absolutely necessary, absolutely horrid):
-
-      foo = Mash.new().merge(node[:system]).merge(node[:system][:component])
-      
-  You might look at this and think "gee I know a much simpler way to do that". That simpler way does not work; this way does.
-
-  I propose adding a 'smush' method to `silverware/libraries/cookbook_utils`: 
-  
-      ```ruby
-      module Ironfan::CookbookUtils
-        module_function
-        
-        # Merge the given objects (node attributes, hashes, or anything
-        # else with `#to_hash`) into a combined `Mash` object. Objects 
-        # given later in the list 'win' over objects given earlier.
-        #
-        # @examples
-        #   template_vars = Ironfan::CookbookUtils.smush( node[:flume], node[:flume][:agent], :zookeeper_port => node[:zookeeper][:port] )
-        #
-        # @param [Array[#to_hash]] smushables -- any number of things that respond to `#to_hash`
-        #
-        def smush(*smushables)
-          result = Mash.new
-          smushables.compact.each do |smushable|
-            result.merge! smushable.to_hash
-          end
-          result
-        end
-        
-      end
-
-  (obviously the hard part is not writing the method, it's applying it to all the cookbooks.)
-
 
 
 ### Knife commands
-
-* reify notion of 'homebase'; cluster commands work off it
 * move away from referring to Chef::Config everywhere; 
 
 
-
-
-
 # Old Issues Triage
 From https://github.com/infochimps-labs/ironfan/issues/102 with love. Deleted ownership, and everything that was done or has its own issue.
 
 ## Must Do
 * merge volumes into silverware. merge ebs_volumes into ec2 cookbook 
 * Basic CI testing of cookbooks 
-* RSpecs for silverback (lib and knife tools) 
+* RSpecs for ironfan (lib and knife tools)
 * RSpecs for silverware are mostly in place -- ensure they are. 
 * push cookbooks to community.opscode.com  
 * refine and explain updated git workflow
@@ -60,18 +22,12 @@ From https://github.com/infochimps-labs/ironfan/issues/102 with love. Deleted ow
 ## Docco
 Use the [opscode EC2 fast start](http://wiki.opscode.com/display/chef/EC2+Bootstrap+Fast+Start+Guide) as a guide -- our getting started should start at the same place, and cover the same detail as the EC2 bootstrap guide.
 
-* Clear description of metadiscovery
+* Clear description of discovery
 * make sure README files in cookbooks aren’t wildly inaccurate
-* Carry out setup directions, ensure they work:
-  - cluster_chef if you’re using our homebase
-  - cluster_chef if you’re using opscode’s homebase
-* local vagrant environment 
 * hadoop cluster bootstrapping
 
 ## Piddly Shit
-
 * standardize the `zabbix` cookbook (no more /opt, etc -- more in the TODO)
-* volumes don't deep merge -- eg you have to mount_ephemerals in the facet if you modify htem
 * kill_old_service should disable services (may be leaving /etc/rc.d cruft).
 * kill old service doesn't go the first time. why?
 * chef client/server cookbook: set chef user UID / GID; client can set log directory
@@ -81,8 +37,6 @@ Use the [opscode EC2 fast start](http://wiki.opscode.com/display/chef/EC2+Bootst
 * style-guide alignment (prefix_root becomes prefix)
 
 ## Really Want
-
-* unify the hashlike underpinning to be same across silverware & cluster_chef. Make sure we love (or accept) all the differences between it and Gorrillib’s, and between it and Chef’s.
 * Keys are transmitted in databags, using a helper, and not in node attributes
 * easy to create a dummy node (load balancer, external resource, etc)
 * components can have arbitrary attributes (kinda. they take an `:info` param, behavior which may change later)
@@ -138,11 +92,9 @@ Use the [opscode EC2 fast start](http://wiki.opscode.com/display/chef/EC2+Bootst
 * move cluster discovery to cloud class.
 * Server#normalize! doesn’t imprint object (ie. server attributes poke through to the facet & cluster, rather than being *set* on the object)
 * The fact you can only see one cluster at a time is stupid.
-* security group pairing is sucky.
 * ubuntu home drive bullshit
 * Finer-grained security group control (eg nfs server only opens a couple ports, not all)
 * nfs recipe uses discovery right (thus allowing more than one NFS share to exist in the universe)
-* roles UGGGHHHHAERWSDFKHSBLAH
 
 ## Ponies!
 * sync cookbooks up/down to `infochimps-cookbooks/` 
@@ -152,4 +104,3 @@ Use the [opscode EC2 fast start](http://wiki.opscode.com/display/chef/EC2+Bootst
 * spot pricing
 * rackspace compatibility
 * cookbook munger reads comments in attributes file to populate metadata.rb
-* `gem install ironfan; ironfan install` checks everything out

data/VERSION

@@ -1 +1 @@
-4.0.2
+4.0.3

data/ironfan.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = "ironfan"
-  s.version = "4.0.2"
+  s.version = "4.0.3"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Infochimps"]
-  s.date = "2012-09-07"
+  s.date = "2012-09-12"
   s.description = "Ironfan allows you to orchestrate not just systems but clusters of machines. It includes a powerful layer on top of knife and a collection of cloud cookbooks."
   s.email = "coders@infochimps.com"
   s.extra_rdoc_files = [

data/lib/ironfan/broker/computer.rb

@@ -34,12 +34,6 @@ module Ironfan
       #
       def correlate
         chosen_resources.each do |res|
-          unless res.respond_to? :expected_ids
-            Chef::Log.warn("FIXME: Using correlate! instead of on_correlate in #{res}")
-            res.correlate! self
-            return
-          end
-        
           res.expected_ids(self).each do |id|
             next unless res.recall? id
 
@@ -55,9 +49,6 @@ module Ironfan
         end
       end
 
-      def correlate_with(res)
-      end
-
       def validate
         computer = self
         Ironfan.delegate_to(chosen_resources) { validate_computer! computer }

data/lib/ironfan/broker/drive.rb

@@ -8,11 +8,11 @@ module Ironfan
 
       field :name,              String
 
-      def volume=(value)
-        super
-        return unless value
-        # inscribe the cluster DSL values into chef attributes
-        volume.attributes.each_pair {|k,v| node[k.to_s] = v }
+      # A drive should include volume attributes in any node references
+      def node()
+        result = super
+        result.merge! volume.attributes unless volume.nil?
+        result
       end
 
     end

data/lib/ironfan/provider/chef/node.rb

@@ -67,7 +67,7 @@ module Ironfan
           normal[:facet_name] =         server.facet_name
           normal[:permanent] =          computer.permanent?
           normal[:volumes] =            {}
-          computer.drives.each {|v| normal[:volumes][v.name] = v.node}
+          computer.drives.each {|d| normal[:volumes][d.name] = d.node}
         end
 
         def announce_state state

data/lib/ironfan/provider/ec2/ebs_volume.rb

@@ -16,14 +16,14 @@ module Ironfan
           :to => :adaptee
         field :dsl_volume,        Ironfan::Dsl::Volume
 
-        def self.shared?()      false;  end
+        def self.shared?()      true;   end
         def self.multiple?()    true;   end
         def self.resource_type()        :ebs_volume;   end
         def self.expected_ids(computer)
           computer.server.volumes.values.map do |volume|
             saved = computer.node[:volumes][volume.name][:volume_id] rescue nil
             ebs_name = "#{computer.server.fullname}-#{volume.name}"
-            [ volume.volume_id, saved, ebs_name]
+            [ volume.volume_id, saved, ebs_name ]
           end.flatten.compact
         end
 
@@ -31,6 +31,11 @@ module Ironfan
           tags["Name"] || tags["name"] || id
         end
 
+        def drivename
+          return id unless tags.key? "Name"
+          tags["Name"].split('-').pop
+        end
+
         def ephemeral_device?
           false
         end
@@ -54,59 +59,22 @@ module Ironfan
         end
 
         def on_correlate(computer)
-          Chef::Log.warn("ebs_volume::on_correlate is incomplete")
+          drive = computer.drive(drivename)
+          drive.disk = self
+          drive.node = computer.node[:volumes][drivename].to_hash rescue {}
+          drive
         end
 
-#         def self.correlate!(computer)
-#           Chef::Log.warn("CODE SMELL: overly large method: #{caller}")
-#           return unless computer.server?
-#           computer.server.volumes.each do |volume|
-#             unless volume.attachable.to_s == 'ebs'
-#               Chef::Log.debug("Ignoring non-EBS volume = #{volume}")
-#               next
-#             end
-# 
-#             drive =             computer.drive(volume.name)
-#             drive.node =        computer.node[:volumes][volume.name].to_hash rescue {}
-#             if drive.node.has_key? :volume_id
-#               node_volume_id =  drive.node[:volume_id]
-#             end
-#             ebs_name =          "#{computer.server.fullname}-#{volume.name}"
-# 
-#             volume_id =         volume.volume_id
-#             volume_id ||=       node_volume_id
-# 
-#             # Volumes may match against name derived from the cluster definition,
-#             #   or volume_id from the cluster definition or node
-#             case
-#             when (recall? ebs_name)
-#               ebs =             recall ebs_name
-#             when (volume_id and recall? volume_id)
-#               ebs =             recall volume_id
-#             else
-#               log_message =     "Volume not found: name = #{ebs_name}"
-#               log_message +=    ", volume_id = #{volume_id}"
-#               Chef::Log.debug(log_message)
-#               next
-#             end
-# 
-#             # Make sure all the known volume_ids match
-#             [ ebs.id, volume.volume_id, node_volume_id ].compact.each do |id|
-#               id == volume_id or ebs.bogus << :volume_id_mismatch
-#             end if volume_id
-# 
-#             computer.bogus +=   ebs.bogus
-#             ebs.owner =         computer.object_id
-#             drive.volume =      volume
-#             drive.disk =        ebs
-#           end
-#         end
         def self.validate_computer!(computer)
-          Chef::Log.warn("ebs_volume::validate_computer! is incomplete")
-#             # Make sure all the known volume_ids match
-#             [ ebs.id, volume.volume_id, node_volume_id ].compact.each do |id|
-#               id == volume_id or ebs.bogus << :volume_id_mismatch
-#             end if volume_id
+          computer.drives.each do |drive|
+            next unless drive.disk.class == EbsVolume
+            [ (drive.node['volume_id'] rescue nil),
+              (drive.volume.volume_id  rescue nil)
+            ].compact.each do |id|
+              Chef::Log.debug "checking #{id} against ebs_volume id #{drive.disk.id}"
+              id == drive.disk.id or drive.disk.bogus << :volume_id_mismatch
+            end
+          end
         end
 
         #
@@ -126,8 +94,6 @@ module Ironfan
                 ebs.server =          computer.machine.adaptee
               end
             end
-            # Record the volume information in chef
-            drive.node['volume_id'] =  ebs.id
           end
         end
       end

data/lib/ironfan/provider/ec2/machine.rb

@@ -165,8 +165,11 @@ module Ironfan
             drive.disk = ebs_vol
 
             vol_name = "#{computer.name}-#{drive.volume.name}"
+            tags['server'] = computer.name
             tags['name'] = vol_name
             tags['Name'] = vol_name
+            tags['mount_point'] = drive.volume.mount_point
+            tags['device'] = drive.volume.device
             Ec2.ensure_tags(tags,ebs_vol)
           end
         end
@@ -222,8 +225,7 @@ module Ironfan
               hsh['Ebs.SnapshotId'] = volume.snapshot_id if volume.snapshot_id.present?
               hsh['Ebs.VolumeSize'] = volume.size.to_s   if volume.size.present?
               hsh['Ebs.DeleteOnTermination'] = (not volume.keep).to_s
-            else
-              next
+            else next
             end
             hsh
           end.compact

data/lib/ironfan/provider.rb

@@ -54,7 +54,6 @@ module Ironfan
       # Discovery
       #
       def self.load!(*p)                Ironfan.noop(self,__method__,*p);   end
-      def self.correlate!(*p)           Ironfan.noop(self,__method__,*p);   end
       def self.validate_computer!(*p)   Ironfan.noop(self,__method__,*p);   end
       def self.validate_resources!(*p)  Ironfan.noop(self,__method__,*p);   end
 
@@ -70,7 +69,7 @@ module Ironfan
       #
       # Utilities
       #
-      [:shared?, :multiple?, :load!,:correlate!,:validate_computer!,
+      [:shared?, :multiple?, :load!,:validate_computer!,
        :validate_resources!,:create!,:save!,:destroy!].each do |method_name|
         define_method(method_name) {|*p| self.class.send(method_name,*p) }
       end

metadata

@@ -2,7 +2,7 @@
 name: ironfan
 version: !ruby/object:Gem::Version 
   prerelease: 
-  version: 4.0.2
+  version: 4.0.3
 platform: ruby
 authors: 
 - Infochimps
@@ -10,7 +10,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2012-09-07 00:00:00 Z
+date: 2012-09-12 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: chef
@@ -232,7 +232,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements: 
   - - ">="
     - !ruby/object:Gem::Version 
-      hash: 318287528418950841
+      hash: 3321055109104960194
       segments: 
       - 0
       version: "0"