fog_tracker 0.2.2 → 0.3.0

data/README.md

@@ -35,23 +35,27 @@ How is it [done]? (Usage)
 
   Here are the contents of a sample `accounts.yml`:
 
-    AWS EC2 development account:
-      :provider: AWS
-      :service: Compute
+    AWS EC2 production account:   # The account name - can be anything
+      :provider: AWS      # This is the Fog provider Module
+      :service: Compute   # The Fog service Module. So, Fog::Compute::AWS
       :credentials:
         :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
         :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-      :polling_time: 180
-	  :exclude_resources:
-	  - :flavors
-	  - :images
-    Rackspace development account:
-      :provider: Rackspace
-      :service: Compute
+      :delay: 120 # Wait time between successive pollings (in seconds)
+      :exclude_resources:
+      - :account  # No need to poll for accounts - those are listed here
+      - :flavors  # You may or may not want EC2 server types
+      - :images   # Takes a while to list all AMIs (works though)
+    AWS S3 development account:
+      :provider: AWS
+      :service: Storage
+      :exclude_resources:
+      #- :directories   # The S3 buckets - fetch the list
+      - :files          # "secondary" entity to buckets - does not work yet
       :credentials:
-        :rackspace_api_key: XXXXXXXXXXXXXXXXXXXX
-        :rackspace_username: XXXXXXXXX
-      :polling_time: 180
+        :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
+        :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+      :delay: 150
 
 2) Call `start` on the Tracker. It will run asynchronously, with one thread per account. At any time, you can call `start` or `stop` on it, and query the resulting collections of Fog Resource objects.
 
@@ -63,7 +67,7 @@ How is it [done]? (Usage)
     tracker.query("*::Compute::*::servers")
 
     # get all Amazon EC2 Resources, of all types, across all accounts
-    tracker["*::Compute::AWS::*"]	# the [] operator is the same as query()
+    tracker["*::Compute::AWS::*"]	# the [] operator is aliased to query()
 
     # get all S3 buckets in a given account
     tracker["my production account::Storage::AWS::directories"]
@@ -75,7 +79,7 @@ How is it [done]? (Usage)
 
         tracker.query("*::*::*::*"){|r| puts "Found #{r.class} #{r.identity}"}
 
-* You can pass a callback Proc to the Tracker at initialization, which will be invoked whenever all an account's Resources have been updated. It should accept an Array containing the updated Resources as its first parameter:
+* You can pass a callback Proc to the Tracker at initialization, which will be invoked whenever all an account's Resources have been updated. It should accept an Array as its first parameter, which will contain the all the account's Resources. Like this:
 
         FogTracker::Tracker.new(YAML::load(File.read 'accounts.yml'),
           :callback => Proc.new do |resources|
@@ -84,15 +88,27 @@ How is it [done]? (Usage)
           end
         ).start
 
-* The resources returned from a query are all Fog::Model objects, but they are "decorated" with some extra methods for fetching the account information, or for fetching more resources. This simplifies the code that consumes the query results, because it does not have to know anything about the tracker. Here are the methods added by {FogTracker::Extensions::FogModel}:
+* The resources returned from a query are all Fog::Model objects, but they are "decorated" with some extra methods for fetching the account information, or for fetching more resources. This simplifies the code that consumes the query results, because it does not have to know anything about the tracker. Here are the some of the methods added by {FogTracker::Extensions::FogModel}:
   1. `tracker_account` returns a Hash of the Resource's account information _(:name is added; :credentials are removed)_.
-  2. `tracker_query(query_string)` queries the tracker for more resources.
-  3. `account_resources(collection_query)` also returns an Array of resources, but only from the same account. (This is essentially shorthand for `tracker_query("account::service::provider::#{collection_query}")`)
-  4. `tracker_description` returns a descriptive string that is short, but unique to this resource.
+  2. `tracker_description` returns a descriptive identifier (a String) that is unique to this resource.
+  3. `tracker_query(query_string)` queries the tracker for more resources.
+  4. `account_resources(collection_query)` also returns an Array of resources, but only from the same account. (This is essentially shorthand for `tracker_query("account::service::provider::#{collection_query}")`)
 
 * Any Exceptions that occur in the Tracker's polling threads are rescued and logged. If you want to take further action, you can initialize the Tracker with an `:error_callback` Proc. This is similar to the Account update `:callback` above, except that the parameter for `:error_callback` should be an Exception instead of an Array of Resources.
 
-* The Tracker can also be used synchronously. Its `update` method polls all accounts immediately, and waits for the result (the updated Array of resource objects) in the current thread.
+* The Tracker can also be used synchronously. Its `update` method polls all accounts immediately, one at a time, and waits for the result (the updated Array of resource objects) in the current thread. In this mode, any exceptions are raised immediately.
+
+----------------
+*Known Limitations / Bugs*
+
+* Some Fog resources are not currently supported, because polling them depends on making multiple calls to the service provider, once for each primary key of some other resource. These "secondary resources" include:
+  * S3 Objects (Fog::Storage[:aws].files) - each is dependent
+    on its bucket (Fog::Storage[:aws].directories)
+  * Route 53 addresses (Fog::DNS[:aws].addresses) - each is dependent
+    on its DNS zone (Fog::DNS[:aws].zones)
+
+  Supporting these would involve hard-coding these dependencies into this gem. Currently, all Fog resource information is introspected, which allows this gem to track new Fog functionality with no update. If you really want to get information on these resources, you can traverse the Fog resource graph in your Account update `:callback`.
+
 
 ----------------
 Who is it? (Contribution)

data/config/accounts.yml.example

@@ -1,25 +1,29 @@
-AWS EC2 production account: 
-  :provider: AWS
-  :service: Compute
-  :credentials: 
+AWS EC2 production account:   # The account name - can be anything
+  :provider: AWS      # This is the Fog provider Module
+  :service: Compute   # The Fog service Module. So, Fog::Compute::AWS
+  :credentials:
     :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
     :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-  :polling_time: 120
+  :delay: 120 # Wait time between successive pollings (in seconds)
   :exclude_resources:
-AWS EC2 development account: 
+  - :account  # No need to poll for accounts - those are listed here
+  - :flavors  # You may or may not want EC2 server types
+  - :images   # Takes a while to list all AMIs (works though)
+AWS S3 development account:
   :provider: AWS
-  :service: Compute
-  :credentials: 
+  :service: Storage
+  :credentials:
     :aws_access_key_id: XXXXXXXXXXXXXXXXXXXX
     :aws_secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-  :polling_time: 180
+  :delay: 150
   :exclude_resources:
-  - :flavors
-  - :images
-Rackspace development account: 
+  #- :directories   # The S3 buckets - fetch the list
+  - :files          # "secondary" entity to buckets - does not work yet
+Rackspace development account:
   :provider: Rackspace
   :service: Compute
-  :credentials: 
+  :credentials:
     :rackspace_api_key: XXXXXXXXXXXXXXXXXXXX
     :rackspace_username: XXXXXXXXX
-  :polling_time: 180
+  :delay: 180
+  :exclude_resources: # Don't know which of these work yet...

data/lib/fog_tracker/account_tracker.rb

@@ -12,6 +12,8 @@ module FogTracker
     attr_reader :log
     # How long to wait between successive polling of this account (Integer)
     attr_reader :delay
+    # The time that the *second-to-last* successful poll finished
+    attr_reader :preceeding_update_time
 
     # Creates an object for tracking all collections in a single Fog account
     # @param [String] account_name a human-readable name for the account
@@ -28,9 +30,9 @@ module FogTracker
       @account  = account
       @callback = options[:callback]
       @log      = options[:logger] || FogTracker.default_logger
-      @delay    = options[:delay]  || account[:polling_time] ||
+      @delay    = options[:delay]  || account[:delay] ||
                               FogTracker::DEFAULT_POLLING_TIME
-      @account[:polling_time] = @delay
+      @account[:delay] = @delay
       @error_proc = options[:error_callback]
       @log.debug "Creating tracker for account #{@name}."
       create_collection_trackers
@@ -58,7 +60,7 @@ module FogTracker
     # Stops polling this tracker's account
     def stop
       if running?
-        @log.info "Stopping tracker for #{name}..."
+        @log.info "Stopping tracker for #{@name}..."
         @timer.kill
         @timer = nil
       else
@@ -71,6 +73,9 @@ module FogTracker
       begin
         @log.info "Polling account #{@name}..."
         @collection_trackers.each {|tracker| tracker.update}
+        @preceeding_update_time = @most_recent_update
+        @most_recent_update     = Time.now
+        @log.info "Polled account #{@name}"
         @callback.call(all_resources) if @callback
       rescue Exception => e
         @log.error "Exception polling account #{name}: #{e.message}"

data/lib/fog_tracker/collection_tracker.rb

@@ -24,26 +24,28 @@ module FogTracker
     # Polls the {AccountTracker}'s connection for updated info on all existing
     # instances of this tracker's resource_type
     def update
-      @log.info "Polling #{@type} on #{@account_name}..."
+      new_collection = Array.new
       fog_collection = @account_tracker.connection.send(@type) || Array.new
       @log.info "Fetching #{fog_collection.count} #{@type} on #{@account_name}."
-      new_collection = Array.new
       # Here's where most of the network overhead is actually incurred
       fog_collection.each do |resource|
         @log.debug "Fetching resource: #{resource.class} #{resource.identity}"
         resource._fog_collection_tracker = self
         new_collection << resource
-        @log.debug "Got resource: #{resource.inspect}"
+        #@log.debug "Got resource: #{resource.inspect}"
       end
+      @log.info "Fetched #{new_collection.count} #{@type} on #{@account_name}."
       @collection = new_collection
     end
 
     # @return [Hash] a Hash of account information, slighly modified:
     #    a :name parameter is added, and the :credentials are removed
+    #    :preceeding_update_time is also added
     def clean_account_data
       @clean_data ||= @account  # generate this data only once per res
       @clean_data[:name] = @account_name
       @clean_data[:credentials] = Hash.new
+      @clean_data[:preceeding_update_time] = @account_tracker.preceeding_update_time
       @clean_data
     end
 

data/lib/fog_tracker/extensions/fog_model.rb

@@ -52,8 +52,8 @@ module FogTracker
         results
       end
 
-      # Returns a short description of this resource
-      # @return [String] a short description of this resource
+      # returns a descriptive identifier unique to this resource
+      # @return [String] the type, identity, and account name of the resource
       def tracker_description
          type = (self.class.name.match(/::([^:]+)$/))[1]
         "#{type} #{self.identity} in account #{tracker_account[:name]}"

data/lib/fog_tracker/query/query_processor.rb

@@ -1,7 +1,7 @@
 module FogTracker
   module Query
-    # A class for gathering all Fog resources for a set of {AccountTracker}s,
-    # and filtering them based on a regular-expression-based query
+    # A class for filtering the Fog resources for a set of {AccountTracker}s,
+    # based on a regular-expression-based query
     class QueryProcessor
 
       # The Regular Expression which all resource queries must match
@@ -16,7 +16,7 @@ module FogTracker
         @log      = options[:logger] || FogTracker.default_logger
       end
 
-      # Uses the query string to filter this account's resources
+      # Uses the query string to filter all polled resources
       # for a desired subset
       # @param [String] query a string used to filter for matching resources
       # @return [Array <Fog::Model>] an Array of Resources, filtered by query
@@ -36,7 +36,7 @@ module FogTracker
       # Returns an Array of 4 RegEx objeccts based on the +query_string+
       # for matching [account name, service, provider, collection]
       def parse_query(query_string)
-        @log.debug "Parsing Query #{query_string}"
+        #@log.debug "Parsing Query #{query_string}"
         tokenize(query_string).map {|token| regex_from_token(token)}
       end
 

data/lib/fog_tracker/tracker.rb

@@ -92,6 +92,13 @@ module FogTracker
       results
     end
 
+    # Returns the time of given account's most recent successful update
+    # @param [String] account_name the name of the account that was polled
+    # @return [Time] the time the account finished updating
+    def preceeding_update_time(account_name)
+      @trackers[account_name].preceeding_update_time
+    end
+
     # Returns this tracker's logger, for changing logging dynamically
     def logger ; @log end
 

data/lib/fog_tracker/version.rb

@@ -1,3 +1,3 @@
 module FogTracker
-  VERSION = "0.2.2"
+  VERSION = "0.3.0"
 end

data/spec/lib/fog_tracker/account_tracker_spec.rb

@@ -27,8 +27,8 @@ module FogTracker
       it "exposes its logger" do
         @tracker.log.should_not == nil
       end
-      it "always sets a polling time" do
-        @tracker.account[:polling_time].should be_an_instance_of(Fixnum)
+      it "always has a delay" do
+        @tracker.account[:delay].should be_an_instance_of(Fixnum)
       end
 
       describe '#connection' do
@@ -76,6 +76,12 @@ module FogTracker
             (Proc.new { @tracker.update }).should raise_error
           end
         end
+        it "saves the time of it's next-to-last update as @preceeding_update_time" do
+          @tracker.update
+          @tracker.preceeding_update_time.should == nil
+          @tracker.update
+          @tracker.preceeding_update_time.should_not == nil
+        end
       end
 
       describe '#start' do

data/spec/lib/fog_tracker/collection_tracker_spec.rb

@@ -22,10 +22,17 @@ module FogTracker
           @tracker.update
         end
         it "attaches account information to all resources" do
+          fake_connection = double('mock fog connection')
+          fake_connection.stub(:servers).and_return(
+            [Fog::Compute[:aws].servers.new])
+          now = Time.now
+          @account_tracker.stub(:connection).and_return fake_connection
+          @account_tracker.stub(:preceeding_update_time).and_return now
           @tracker.update
           @tracker.collection.each do |resource|
             resource.tracker_account[:name].should == FAKE_ACCOUNT_NAME
             resource.tracker_account[:credentials].should == {}
+            resource.tracker_account[:preceeding_update_time].should == now
           end
         end
       end

data/spec/lib/fog_tracker/tracker_spec.rb

@@ -66,6 +66,17 @@ module FogTracker
       end
     end
 
+    describe '#preceeding_update_time' do
+      context "when given an account name" do
+        it "returns time of the account's next-most-recent succesful update" do
+          @tracker.update
+          @tracker.preceeding_update_time(ACCOUNTS.keys.first).should == nil
+          @tracker.update
+          @tracker.preceeding_update_time(ACCOUNTS.keys.first).should_not == nil
+        end
+      end
+    end
+
     describe '#all' do
       WILDCARD_QUERY = '*::*::*::*'
       before(:each) do

data/spec/support/_mocks.rb

@@ -8,7 +8,7 @@ module FogTracker
   FAKE_ACCOUNT = {
     :provider     => 'AWS',
     :service      => 'Compute',
-    :polling_time => 10,
+    :delay => 10,
     :credentials  => {
       :aws_access_key_id => "fake user",
       :aws_secret_access_key => 'fake password'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fog_tracker
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.3.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-02-09 00:00:00.000000000Z
+date: 2012-02-18 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fog
-  requirement: &70268295001560 !ruby/object:Gem::Requirement
+  requirement: &70238491411280 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70268295001560
+  version_requirements: *70238491411280
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70268295000400 !ruby/object:Gem::Requirement
+  requirement: &70238491410860 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70268295000400
+  version_requirements: *70238491410860
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70268294999500 !ruby/object:Gem::Requirement
+  requirement: &70238491410440 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70268294999500
+  version_requirements: *70238491410440
 - !ruby/object:Gem::Dependency
   name: yard
-  requirement: &70268294998340 !ruby/object:Gem::Requirement
+  requirement: &70238491410020 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70268294998340
+  version_requirements: *70238491410020
 - !ruby/object:Gem::Dependency
   name: guard
-  requirement: &70268294997600 !ruby/object:Gem::Requirement
+  requirement: &70238491409600 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70268294997600
+  version_requirements: *70238491409600
 - !ruby/object:Gem::Dependency
   name: guard-rspec
-  requirement: &70268294996740 !ruby/object:Gem::Requirement
+  requirement: &70238491409180 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70268294996740
+  version_requirements: *70238491409180
 - !ruby/object:Gem::Dependency
   name: ruby_gntp
-  requirement: &70268294995920 !ruby/object:Gem::Requirement
+  requirement: &70238491408740 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -87,7 +87,7 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70268294995920
+  version_requirements: *70238491408740
 description: This gem peridically polls mutiple cloud computing services using the
   fog gem, asynchronously updating the state of the resulting collections of Fog Resources.
 email:
@@ -138,7 +138,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -2942181140589294968
+      hash: -64177952162432192
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -147,7 +147,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -2942181140589294968
+      hash: -64177952162432192
 requirements: []
 rubyforge_project: fog_tracker
 rubygems_version: 1.8.10