fog_tracker 0.1.1 → 0.1.2

data/README.md

@@ -28,11 +28,10 @@ Install the Fog Tracker gem (and its dependencies if necessary) from RubyGems
 ----------------
 How is it [done]? (Usage)
 ----------------
-1) Just require the gem, and create a `FogTracker::Tracker`. Pass it some account information in a hash, perhaps loaded from a YAML file:
+1) Require the gem, and create a `FogTracker::Tracker`. Pass it some account information in a hash, perhaps loaded from a YAML file:
 
     require 'fog_tracker'
     tracker = FogTracker::Tracker.new(YAML::load(File.read 'accounts.yml'))
-    tracker.start
 
   Here are the contents of a sample `accounts.yml`:
 
@@ -54,24 +53,39 @@ How is it [done]? (Usage)
         :rackspace_username: XXXXXXXXX
       :polling_time: 180
 
-2) The tracker will run asynchronously, with one thread per account. You can call `start()` and `stop()` on it, and query the resulting collections of Fog Resource objects using a filter-based query:
+2) Call `start` on the Tracker. It will run asynchronously, with one thread per account. At any time, you can call `start` or `stop` on it, and query the resulting collections of Fog Resource objects.
+
+    tracker.start
+
+3) Access the Fog object collections by calling `Tracker::query`, and passing it a filter-based query String. The query string format is: `account_name::service::provider::collection`
+
+    # get all Compute instances across all accounts and providers
+    tracker.query("*::Compute::*::servers")
+
+    # get all Amazon EC2 Resources, of all types, across all accounts
+    tracker["*::Compute::AWS::*"]	# the [] operator is the same as query()
+
+    # get all S3 objects in a given account
+    tracker["my production account::Storage::AWS::files"]
 
-	# get all Compute instances across all accounts and providers
-	tracker.query("*::Compute::*::servers")
+  If you're tired of calling `each` on the results of every query, pass a single-argument block, and it will be invoked once with each resulting resource:
 
-	# get all Amazon EC2 Resources, of all types, across all accounts
-	tracker["*::Compute::AWS::*"]	# the [] operator is the same as query()
+    tracker.query("*::*::*::*"){|r| puts "Found #{r.class} #{r.identity}"}
 
-	# get all S3 objects in a given account
-	tracker["my production account::Storage::AWS::files"]
+  You can also pass a Proc to the Tracker at initialization, which will be invoked whenever an account's Resources have been updated. It should accept a list of the updated Resources as its first argument:
 
-  The query string format is:    "`account name::service::provider::collection`"
+    FogTracker::Tracker.new(YAML::load(File.read 'accounts.yml'),
+      :callback => Proc.new do |resources|
+      	puts "Got #{resources.count} resources from account "+
+		      resources.first.tracker_account[:name]
+      end
+    ).start
 
-  You can also pass a Proc to the Tracker at initialization, which will be invoked whenever an account's Resources have been updated -- see the API docs for details.
+  The appropriate account information for each Fog resource, read from `accounts.yml` can be obtained by calling its `tracker_account` method.
 
 
 ----------------
-Who is it? (Contribution / Development)
+Who is it? (Contribution)
 ----------------
 This Gem was created by Benton Roberts _(benton@bentonroberts.com)_, but draws heavily on the work of the [Fog project](http://fog.io/). Thanks to geemus, and to all Fog contributors.
 

data/lib/fog_tracker/account_tracker.rb

@@ -1,13 +1,13 @@
 module FogTracker
 
-  # Tracks a single Fog account in an ActiveRecord database
+  # Tracks all collections in a single Fog account
   class AccountTracker
     require 'fog'
 
     attr_reader :name, :account, :log, :delay
     attr_reader :resource_trackers
 
-    # Creates an object for tracking a single Fog account
+    # Creates an object for tracking all collections in a single Fog account
     #
     # ==== Attributes
     #
@@ -41,7 +41,7 @@ module FogTracker
             while true do
               @log.info "Polling account #{@name}..."
               @resource_trackers.each {|tracker| tracker.update}
-              @callback.call @name if @callback
+              @callback.call(all_resources) if @callback
               sleep @delay
             end
           rescue Exception => e
@@ -82,6 +82,13 @@ module FogTracker
       types.map {|type| type.to_s}
     end
 
+    # Returns an Array of all this Account's currently tracked Resources
+    def all_resources
+      (@resource_trackers.collect do |tracker|
+        tracker.collection
+      end).flatten
+    end
+
     private
 
     # Creates and returns an Array of ResourceTracker objects -

data/lib/fog_tracker/model.rb

@@ -0,0 +1,20 @@
+# add an accessor and a method to decorate Fog::Model instances
+# with tracker account information
+module Fog
+  class Model
+
+    # a FogTracker::CollectionTracker
+    attr_accessor :fog_collection_tracker
+
+    # returns a clean copy of the resource's account information
+    # from the its collection tracker
+    def tracker_account
+      if fog_collection_tracker
+        fog_collection_tracker.clean_account_data
+      else
+        Hash.new
+      end
+    end
+
+  end
+end

data/lib/fog_tracker/query/query_processor.rb

@@ -22,10 +22,11 @@ module FogTracker
       # Returns an Array of Resources, filtered by +query+
       def execute(query)
         acct_pattern, svc_pattern, prov_pattern, col_pattern = parse_query(query)
-        results_by_account = get_results_by_account(acct_pattern)
-        results = filter_by_service(results_by_account, svc_pattern)
-        results = filter_by_provider(results, prov_pattern)
-        results = filter_by_collection(results, col_pattern)
+        filter_by_collection(
+          filter_by_provider(
+            filter_by_service(get_results_by_account(acct_pattern), svc_pattern),
+            prov_pattern),
+          col_pattern)
       end
 
       private

data/lib/fog_tracker/resource_tracker.rb

@@ -1,11 +1,11 @@
 module FogTracker
 
-  # Tracks a single Fog Resource type for a single Account
+  # Tracks a single Fog collection in a single account
   class ResourceTracker
 
     attr_accessor :collection
 
-    # Creates an object for tracking a single Fog account
+    # Creates an object for tracking a single Fog collection in a single account
     #
     # ==== Attributes
     #
@@ -25,8 +25,26 @@ module FogTracker
     # instances of the relevant resource type, and saves them as @collection
     def update
       @log.info "Polling #{@type} on #{@account_name}..."
-      @collection = @account_tracker.connection.send(@type)
-      @log.info "Discovered #{@collection.count} #{@type} on #{@account_name}."
+      fog_collection = @account_tracker.connection.send(@type) || Array.new
+      @log.info "Fetching #{fog_collection.count} #{@type} on #{@account_name}."
+      new_collection = Array.new
+      # Here's where most of the network overhead is actually incurred
+      fog_collection.each do |resource|
+        @log.debug "Fetching resource: #{resource.class} #{resource.identity}"
+        resource.fog_collection_tracker = self
+        new_collection << resource
+        @log.debug "Got resource: #{resource.inspect}"
+      end
+      @collection = new_collection
+    end
+
+    # Returns a Hash of account information as resource.tracker_account
+    # a :name parameter is added, and the :credentials are removed
+    def clean_account_data
+      @clean_data ||= @account  # generate this data only once per res
+      @clean_data[:name] = @account_name
+      @clean_data[:credentials] = Hash.new
+      @clean_data
     end
 
   end

data/lib/fog_tracker/tracker.rb

@@ -58,15 +58,18 @@ module FogTracker
     end
 
     # Returns an array of Resources matching the +query_string+
+    # Calls any block passed for each resulting resource
     #
     # ==== Attributes
     #
     # * +query_string+ - a String used to filter the discovered Resources
     #          it might look like: "Account Name::Compute::AWS::servers"
     def query(query_string)
-      FogTracker::Query::QueryProcessor.new(
+      results = FogTracker::Query::QueryProcessor.new(
         @trackers, :logger => @log
       ).execute(query_string)
+      (results.each {|r| yield r}) if block_given?
+      results
     end
     alias :[] :query
 

data/lib/fog_tracker/version.rb

@@ -1,3 +1,3 @@
 module FogTracker
-  VERSION = "0.1.1"
+  VERSION = "0.1.2"
 end

data/lib/fog_tracker.rb

@@ -1,6 +1,7 @@
 require 'logger'
 
-# Load all ruby files from 'fog_tracker' directory
+# Load all ruby files from the 'fog' and 'fog_tracker' directories
+Dir[File.join(File.dirname(__FILE__), "fog/**/*.rb")].each {|f| require f}
 Dir[File.join(File.dirname(__FILE__), "fog_tracker/**/*.rb")].each {|f| require f}
 
 module FogTracker

data/spec/lib/fog_tracker/resource_tracker_spec.rb

@@ -5,7 +5,9 @@ module FogTracker
     describe ResourceTracker do
 
       before(:each) do
-        @account_tracker = mock_account_tracker
+        @account_tracker = AccountTracker.new(
+          FAKE_ACCOUNT_NAME, FAKE_ACCOUNT, :logger => LOG
+        )
         @tracker = ResourceTracker.new(
           FAKE_COLLECTION, @account_tracker
         )
@@ -20,6 +22,15 @@ module FogTracker
           @account_tracker.connection.should_receive(FAKE_COLLECTION)
           @tracker.update
         end
+        it "attaches account information to all resources" do
+          account_tracker = mock_account_tracker(5,5)
+          tracker = ResourceTracker.new(FAKE_COLLECTION, account_tracker)
+          tracker.update
+          tracker.collection.each do |resource|
+            resource.tracker_account[:name].should == FAKE_ACCOUNT_NAME
+            resource.tracker_account[:credentials].should == {}
+          end
+        end
       end
     end
 

data/spec/lib/fog_tracker/tracker_spec.rb

@@ -34,5 +34,30 @@ module FogTracker
       end
     end
 
+    describe '#query' do
+      it "invokes any passed code block once per resulting Resource" do
+        receiver = double "resrouce callback object"
+        receiver.stub(:callback)
+        @tracker.start
+        sleep THREAD_STARTUP_DELAY
+        res_count = @tracker.query('*::*::*::*').count
+        receiver.should_receive(:callback).exactly(res_count).times
+        @tracker.query('*::*::*::*') {|r| receiver.callback(r)}
+      end
+    end
+
+    describe '#start' do
+      it "invokes the Tracker's callback Proc when an account is updated" do
+        receiver = double "account callback object"
+        receiver.stub(:callback)
+        tracker = Tracker.new(ACCOUNTS, :logger => LOG,
+          :callback => Proc.new {|resources| receiver.callback(resources)}
+        )
+        receiver.should_receive(:callback).exactly(ACCOUNTS.size).times
+        tracker.start
+        sleep THREAD_STARTUP_DELAY
+      end
+    end
+
   end
 end

data/spec/support/_mocks.rb

@@ -32,6 +32,10 @@ module FogTracker
   module Query
     QUERY = {}  # Used in query_processor_spec.rb
   end
+  class MockFogResource < Fog::Model
+    #attr_accessor :tracker_account
+    def identity ; "random-resource-id-#{rand 65536}" end
+  end
 end
 
 # Create some fake Fog Resource and Collection Classes
@@ -54,8 +58,9 @@ end
 def mock_account_tracker(num_collections = 1, resources_per_collection = 0)
   fake_account_tracker = double('mock_account_tracker')
   fake_account_tracker.stub(:account).and_return(Hash.new)
-  fake_account_tracker.stub(:name).and_return("fake account tracker")
+  fake_account_tracker.stub(:name).and_return(FogTracker::FAKE_ACCOUNT_NAME)
   fake_account_tracker.stub(:log).and_return(LOG)
+  fake_account_tracker.stub(:all_resources).and_return(Array.new)
   fake_account_tracker.stub(:connection).and_return(mock_fog_connection)
   # create an array of mock ResourceTrackers
   trackers = (1..num_collections).map do |class_index|
@@ -70,23 +75,10 @@ end
 
 def mock_fog_connection
   fake_fog_connection = double('mock_fog_connection')
-  fake_fog_connection.stub(FogTracker::FAKE_COLLECTION).and_return([ 1, 2, 3 ])
-  fake_fog_connection
-end
-
-def mock_resource_tracker
-  fake_resource_tracker = double('mock_resource_tracker')
-  fake_resource_tracker.stub(:update)
-  fake_resource_tracker
-end
-
-def mock_fog_resource(resource_class)
-  fake_resource = resource_class.new
-  fake_collection_class =
-  fake_resource.stub(:collection).and_return(
-    collection_class.new
+  fake_fog_connection.stub(FogTracker::FAKE_COLLECTION).and_return(
+    [ FogTracker::MockFogResource.new, FogTracker::MockFogResource.new ]
   )
-  fake_resource
+  fake_fog_connection
 end
 
 def mock_resource_tracker(resource_class, number_of_resources = 0)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fog_tracker
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-01-25 00:00:00.000000000Z
+date: 2012-01-27 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog
-  requirement: &2151921100 !ruby/object:Gem::Requirement
+  requirement: &70121482437700 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *2151921100
+  version_requirements: *70121482437700
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &2151920060 !ruby/object:Gem::Requirement
+  requirement: &70121482437100 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2151920060
+  version_requirements: *70121482437100
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &2151919280 !ruby/object:Gem::Requirement
+  requirement: &70121482436440 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2151919280
+  version_requirements: *70121482436440
 - !ruby/object:Gem::Dependency
   name: rdoc
-  requirement: &2151918500 !ruby/object:Gem::Requirement
+  requirement: &70121482435700 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2151918500
+  version_requirements: *70121482435700
 - !ruby/object:Gem::Dependency
   name: guard
-  requirement: &2151917620 !ruby/object:Gem::Requirement
+  requirement: &70121482434720 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2151917620
+  version_requirements: *70121482434720
 - !ruby/object:Gem::Dependency
   name: guard-rake
-  requirement: &2151915940 !ruby/object:Gem::Requirement
+  requirement: &70121482433280 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2151915940
+  version_requirements: *70121482433280
 - !ruby/object:Gem::Dependency
   name: ruby_gntp
-  requirement: &2151914940 !ruby/object:Gem::Requirement
+  requirement: &70121482432400 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -87,7 +87,7 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2151914940
+  version_requirements: *70121482432400
 description: This gem peridically polls mutiple cloud computing services using the
   fog gem, asynchronously updating the state of the resulting collections of Fog Resources.
 email:
@@ -107,6 +107,7 @@ files:
 - fog_tracker.gemspec
 - lib/fog_tracker.rb
 - lib/fog_tracker/account_tracker.rb
+- lib/fog_tracker/model.rb
 - lib/fog_tracker/query/query_processor.rb
 - lib/fog_tracker/resource_tracker.rb
 - lib/fog_tracker/tracker.rb
@@ -135,7 +136,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 3740439790259369910
+      hash: -2556488865982976006
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -144,10 +145,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 3740439790259369910
+      hash: -2556488865982976006
 requirements: []
 rubyforge_project: fog_tracker
-rubygems_version: 1.8.15
+rubygems_version: 1.8.10
 signing_key: 
 specification_version: 3
 summary: Tracks the state of cloud computing resources across multiple accounts with