methodmissing-scrooge 1.0.0

data/README.textile

@@ -0,0 +1,262 @@
+h1. Scrooge
+
+A Framework and ORM agnostic Model / record attribute tracker to ensure production
+Ruby applications only fetch the database content needed to minimize wire traffic
+and reduce conversion overheads to native Ruby types.
+
+This is mostly an experiment into unobtrusive tracking, respecting development workflows
+and understanding Rack internals better. 
+
+h2. Why bother ?
+
+* Object conversion and moving unnecessary data is both expensive and tax existing infrastructure in high load setups
+* Manually extracting and scoping SELECT clauses is not sustainable in a clean and painless manner with iterative development, even less so in large projects.
+
+h2. Suggested Use
+
+Scrooge *requires* a comprehensive existing functional or integration test suite for best results.If test coverage is flaky or non-existent, then a) you shouldn't worry about performance tuning and b) shouldn't be reading this - go spec.
+
+There's 2 basic modes of operation, a tracking or a scope phase.
+
+h4. Resources
+
+A resource is :
+
+* A controller and action endpoint ( inferred through framework specific routing )
+* A content type / format - a PDF representation may have different Model attribute requirements than a vanilla ERB view.
+* Request method - typically popular public facing GET requests  
+
+All Model to attribute mappings is tracked on a per Resource basis.Multiple Models per Resource is supported.
+
+h4. Tracking
+
+In tracking mode, which is the default when no scope is given and Scrooge is enabled, Scrooge installs filters ( either through Rack middleware or framework specific hooks ) that track attribute access on a per Resource basis.
+
+A Kernel#at_exit callback dumps and timestamps this profile ( or scope ) to eg. *framework_configuration_directory/config/scopes/1234147851/scope.yml* 
+
+This typically happens during functional or integration testing.
+
+The test log may look like :
+
+<pre>
+<code>
+  Processing HotelsController#index (for 0.0.0.0 at 2009-02-09 02:55:55) [GET]
+    Parameters: {"action"=>"index", "controller"=>"hotels"}
+  [Scrooge] Track with resource #< :/ ()
+  [Scrooge] Track for Resource #<GET :hotels/index (*/*)
+   - #<Hotel :from_price, :narrative, :star_rating, :latitude, :created_at, :hotel_name, :updated_at, :important_notes, :id, :apt, :location_id, :nearest_tube, :longitude, :telephone, :nearest_rail, :location_name, :distance>
+   - #<Image :thumbnail_width, :created_at, :title, :updated_at, :url, :thumbnail_height, :height, :thumbnail_url, :has_thumbnail, :hotel_id, :width>
+    Hotel Load (0.3ms)   SELECT * FROM `hotels` LIMIT 0, 15
+  Rendering template within layouts/application
+  Rendering hotels/index
+    Image Load (0.2ms)   SELECT * FROM `images` WHERE (`images`.hotel_id = 491) LIMIT 1
+  [Scrooge] read attribute updated_at
+  Rendered hotels/_hotel (2.7ms)
+  Rendered shared/_header (0.1ms)
+  Rendered shared/_navigation (0.3ms)
+  Missing template hotels/_index_sidebar.erb in view path app/views
+  Rendered shared/_sidebar (0.1ms)
+  Rendered shared/_footer (0.1ms)
+  Completed in 91ms (View: 90, DB: 1) | 200 OK [http://test.host/hotels]
+    SQL (0.3ms)   ROLLBACK
+    SQL (0.1ms)   BEGIN
+</code>
+</pre>
+
+An example scope / profile, saved to disk :
+
+<pre>
+<code>
+--- 
+- hotels_show_get: 
+    :action: show
+    :controller: hotels
+    :method: :get
+    :format: "*/*"
+    :models: 
+    - Address: 
+      - line1
+      - line2
+      - created_at
+      - postcode
+      - updated_at
+      - country_id
+      - county
+      - location_id
+      - town
+      - hotel_id
+    - Hotel: 
+      - important_notes
+      - location_id
+- locations_index_get: 
+    :action: index
+    :controller: locations
+    :method: :get
+    :format: "*/*"
+    :models: 
+    - Location: 
+      - name
+      - created_at
+      - code
+      - updated_at
+      - level
+      - id
+- countries_index_get: 
+    :action: index
+    :controller: countries
+    :method: :get
+    :format: "*/*"
+    :models: 
+    - Country: 
+      - name
+      - created_at
+      - code
+      - updated_at
+      - id
+      - location_id
+      - continent_id
+- hotels_index_get: 
+    :action: index
+    :controller: hotels
+    :method: :get
+    :format: "*/*"
+    :models: 
+    - Hotel: 
+      - from_price
+      - narrative
+      - star_rating
+      - latitude
+      - created_at
+      - hotel_name
+      - updated_at
+      - important_notes
+      - id
+      - apt
+      - location_id
+      - nearest_tube
+      - longitude
+      - telephone
+      - nearest_rail
+      - location_name
+      - distance
+    - Image: 
+      - thumbnail_width
+      - created_at
+      - title
+      - updated_at
+      - url
+      - thumbnail_height
+      - height
+      - thumbnail_url
+      - has_thumbnail
+      - hotel_id
+      - width
+</code>
+</pre>
+
+h4. Scoping
+
+A previously persisted scope / profile can be restored from disk and injected to the applicable Resources.Database content retrieved will match that of the given scope timestamp.
+
+This is typically pushed to production and adjusted for each major release or deployment.
+
+Log output may look like :
+
+<pre>
+<code>
+  Processing HotelsController#index (for 0.0.0.0 at 2009-02-09 02:59:41) [GET]
+    Parameters: {"action"=>"index", "controller"=>"hotels"}
+  [Scrooge] Scope for Model #<Image :created_at, :thumbnail_width, :title, :updated_at, :url, :id, :thumbnail_height, :height, :thumbnail_url, :has_thumbnail, :width, :hotel_id>
+  [Scrooge] Scope for Model #<Hotel :narrative, :from_price, :created_at, :latitude, :star_rating, :hotel_name, :updated_at, :important_notes, :apt, :id, :nearest_tube, :location_id, :nearest_rail, :telephone, :longitude, :distance, :location_name>
+    Hotel Load (0.4ms)   SELECT hotels.narrative, hotels.from_price, hotels.created_at, hotels.latitude, hotels.star_rating, hotels.hotel_name, hotels.updated_at, hotels.important_notes, hotels.apt, hotels.id, hotels.nearest_tube, hotels.location_id, hotels.nearest_rail, hotels.telephone, hotels.longitude, hotels.distance, hotels.location_name FROM `hotels` LIMIT 0, 15
+  Rendering template within layouts/application
+  Rendering hotels/index
+    Image Load (0.2ms)   SELECT images.created_at, images.thumbnail_width, images.title, images.updated_at, images.url, images.id, images.thumbnail_height, images.height, images.thumbnail_url, images.has_thumbnail, images.width, images.hotel_id FROM `images` WHERE (`images`.hotel_id = 491) LIMIT 1
+  Rendered hotels/_hotel (2.8ms)
+  Rendered shared/_header (0.1ms)
+  Rendered shared/_navigation (0.3ms)
+  Missing template hotels/_index_sidebar.erb in view path app/views
+  Rendered shared/_sidebar (0.1ms)
+  Rendered shared/_footer (0.1ms)
+  Completed in 90ms (View: 5, DB: 1) | 200 OK [http://test.host/hotels]
+    SQL (0.1ms)   ROLLBACK
+    SQL (0.1ms)   BEGIN
+</code>
+</pre>
+
+h2. Installation
+
+h4. As a Rails plugin ( Recommended )
+
+  ./script/plugin install git://github.com/methodmissing/scrooge.git
+
+h4. From Git
+
+  git pull git://github.com/methodmissing/scrooge.git
+
+
+h4. As a Gem
+
+  sudo gem install methodmissing-scrooge -s http://gems.github.com
+
+h2. Configuration
+
+Scrooge installs ( see recommended installation above ) a configuration file with the following format within *framework_configuration_directory/scrooge.yml ( RAILS_ROOT/config/scrooge.yml for a Rails setup ) :
+
+  production:
+    orm: :active_record
+    storage: :memory
+    scope:
+    on_missing_attribute: :reload # or :raise
+    enabled: true    
+  development:
+    orm: :active_record
+    storage: :memory
+    scope: 
+    on_missing_attribute: :reload # or :raise
+    enabled: true    
+  test:
+    orm: :active_record
+    storage: :memory  
+    scope:
+    on_missing_attribute: :reload # or :raise
+    enabled: true 
+
+h4. ORM 
+
+Scrooge is ORM agnostic and ships with an ActiveRecord layer.
+
+  orm: :active_record
+
+h4. Storage backend
+
+Tracking results can be persisted to a given backend or storage option.Ships with a memory store, but can be extended to file system, memcached etc. as all Tracker components is designed to be Marshal friendly.
+
+  storage: :memory
+
+h4. Scope
+
+A scope is a reference to a timestamped Scrooge run where access to Model attributes is tracked on a per Resource basis.
+
+  scope: 1234567891
+
+If not scope is given in the configuration, ENV['scope'] would also be considered to facilitate configuration through Capistrano etc.
+
+h4. Handling Missing Attributes
+
+When the contents for a given Model attribute has not been retrieved from the database, most ORM frameworks raise an error by default.This is configurable to reloading the model with all it's columns or raise instead.
+
+  on_missing_attribute: :reload # or :raise
+
+h4. Status
+
+Scrooge can be disabled with :
+
+  enabled: false 
+
+h2. Notes
+
+This is an initial release, has not yet been battle tested in production and is pending Ruby 1.9.1 compatibility.
+
+Initially evaluated a centralized tracker concept for multi-server environments with minimal configuration overhead
+and on the fly scope injection after a given warmup threshold but found that to be overkill for a first release. 

data/lib/scrooge/core/string.rb

@@ -0,0 +1,29 @@
+module Scrooge
+  module Core
+    module String
+      
+      # Framework agnostic String <=> Constant helpers.
+      # Perhaps not the cleanest abstraction, but also not good practice to piggy
+      # back on or use a naming convention that may clash with and uproot the API
+      # any given framework ships with.
+      
+      def to_const
+        self.gsub(/\/(.?)/) { "::#{$1.upcase}" }.gsub(/(?:^|_)(.)/) { $1.upcase }
+      end
+      
+      def to_const!( instantiate = true )
+        begin
+          const = Object.module_eval(to_const, __FILE__, __LINE__)
+          instantiate ? const.new : const
+        rescue => exception 
+          exception.to_s.match( /uninitialized constant/ ) ? self : raise
+        end
+      end
+         
+    end
+  end
+end
+
+class String
+  include Scrooge::Core::String
+end

data/lib/scrooge/core/symbol.rb

@@ -0,0 +1,21 @@
+module Scrooge
+  module Core
+    module Symbol
+      
+      # See Scrooge::Core::Symbol
+      
+      def to_const
+        to_s.to_const
+      end
+         
+      def to_const!
+        to_s.to_const!  
+      end   
+         
+    end
+  end
+end
+
+class Symbol
+  include Scrooge::Core::Symbol
+end

data/lib/scrooge/core/thread.rb

@@ -0,0 +1,26 @@
+module Scrooge
+  module Core
+    module Thread
+      
+      # Scrooge Resource tracker scoped to the current Thread for threadsafety in
+      # multi-threaded environments.
+      
+      def scrooge_resource
+        current[:scrooge_resource] ||= Scrooge::Tracker::Resource.new
+      end
+      
+      def scrooge_resource=( resource )
+        current[:scrooge_resource] = resource
+      end
+      
+      def reset_scrooge_resource!
+        current[:scrooge_resource] = nil
+      end  
+         
+    end
+  end
+end
+
+class Thread
+  extend Scrooge::Core::Thread
+end

data/lib/scrooge/framework/base.rb

@@ -0,0 +1,304 @@
+module Scrooge
+  module Framework
+    
+    # Scrooge is framework agnostic and attempts to abstract the following :
+    #
+    # * current environment
+    # * app root dir
+    # * app tmp dir
+    # * app config dir 
+    # * logging    
+    # * resource endpoints 
+    # * caching
+    # * injecting Rack MiddleWare 
+    #
+    # Framework Signatures 
+    #
+    # Scrooge will attempt to determine the current active framework it's deployed with
+    # through various framework specific hooks.
+    #
+    # module Scrooge
+    #    module Framework
+    #      module YetAnother < Base
+    #        ...
+    #        signature do
+    #          Object.const_defined?( "UnlikeAnyOther" )
+    #        end
+    #        ...
+    #      end
+    #    end
+    #  end 
+    
+    autoload :Rails, 'scrooge/framework/rails'
+    
+    class Base < Scrooge::Base
+      
+      GUARD = Mutex.new
+      
+      SCOPE_REGEX = /\d{10}/.freeze
+      
+      CONFIGURATION_FILE = 'scrooge.yml'.freeze
+      
+      SCOPE_FILE = 'scope.yml'.freeze
+      
+      class NotImplemented < StandardError
+      end
+      
+      class NoSupportedFrameworks < StandardError
+      end
+      
+      class InvalidScopeSignature < StandardError
+      end        
+   
+      class << self
+
+        # Per framework signature lookup.
+        #        
+        @@signatures = {}
+        @@signatures[self.name] = Hash.new( [] )
+        
+        # Support none by default.
+        #
+        @@frameworks = []        
+                
+        # Registers a framework signature.
+        #        
+        def signature( &block )
+          @@signatures[self.name] = signatures << block
+        end  
+        
+        # All signatures for the current klass. 
+        #
+        def signatures
+          @@signatures[self.name] || []
+        end
+        
+        # All supported frameworks.
+        #
+        def frameworks
+          @@frameworks
+        end
+        
+        # Infer the framework Scrooge attaches to in a first yield manner.
+        # A match of all defined signatures is required.
+        #
+        def which_framework?
+          iterate_frameworks() || raise( NoSupportedFrameworks )
+        end
+        
+        # Yield an instance of the current framework.
+        #
+        def instantiate
+          which_framework?().new
+        end
+        
+        private
+        
+          def inherited( subclass ) #:nodoc:
+            @@frameworks << subclass
+          end
+        
+          def iterate_frameworks #:nodoc:
+            frameworks.detect do |framework|
+              framework.signatures.all?{|sig| sig.call }
+            end
+          end  
+                        
+      end
+      
+      # The framework environment eg. test, development etc.
+      #
+      def environment
+        raise NotImplemented
+      end
+      
+      # Application root directory
+      #
+      def root
+        raise NotImplemented  
+      end
+      
+      # Application temp. directory
+      #
+      def tmp
+        raise NotImplemented
+      end      
+      
+      # Application configuration directory
+      #
+      def config
+        raise NotImplemented
+      end
+      
+      # Application logger instance.
+      # API compat with stdlib Logger assumed.
+      #
+      def logger
+        raise NotImplemented
+      end
+      
+      # Supplement the current Resource tracker with additional environment context. 
+      #
+      def resource( env, request = nil )
+        raise NotImplemented
+      end
+      
+      # Write to the framework cache.
+      #
+      def write_cache( key, value )
+        raise NotImplemented
+      end
+      
+      # Read from the framework cache.
+      #
+      def read_cache( key )
+        raise NotImplemented
+      end 
+      
+      # Access to the framework's Rack middleware stack.
+      #
+      def middleware
+        raise NotImplemented
+      end
+      
+      # Inject scoping middleware. 
+      #
+      def install_scope_middleware( tracker )
+        raise NotImplemented
+      end
+      
+      # Inject tracking middleware.
+      #
+      def install_tracking_middleware
+        raise NotImplemented
+      end
+      
+      # Register a code block to run when the host framework is fully initialized.
+      #
+      def initialized( &block )
+        raise NotImplemented
+      end
+      
+      # Yields a controller constant from a given Resource Tracker
+      #
+      def controller( resource )
+        raise NotImplemented
+      end
+      
+      # Retrieve all previously persisted scopes tracked with Scrooge. 
+      #
+      def scopes
+        ensure_scopes_path do
+          Dir.entries( scopes_path ).grep( SCOPE_REGEX )
+        end  
+      end
+      
+      # Determine if there's any previously persisted scopes.
+      #
+      def scopes?
+        !scopes().empty?
+      end
+      
+      # Return the scopes storage path for the current framework.
+      #
+      def scopes_path
+        @profiles_path ||= File.join( config, 'scrooge', 'scopes' )
+      end
+
+      # Return the scopes storage path for a given scope and optional filename.
+      #
+      def scope_path( scope, filename = nil )
+        path = File.join( scopes_path, scope.to_s )
+        filename ? File.join( path, filename ) : path
+      end
+      
+      # Log a message to the logger.
+      #
+      def log( message )
+        logger.info "[Scrooge] #{message}"
+      end
+      
+      # Persist the current tracker as scope or restore a previously persisted scope
+      # from a given signature. 
+      #
+      def scope!( scope = nil )
+        scope ? from_scope!( scope ) : to_scope!()
+      end
+      
+      # Do we have a valid scope signature ?
+      #
+      def scope?( scope )
+        scopes.include?( scope.to_s )
+      end
+      
+      # Restore a previously persisted scope to the current tracker from a given 
+      # signature.Raises Scrooge::Framework::InvalidScopeSignature if the signature
+      # could not be found.
+      #
+      def from_scope!( scope )
+        GUARD.synchronize do
+          if scope?( scope )
+            restore_scope!( scope )
+          else
+            raise InvalidScopeSignature
+          end
+        end    
+      end
+      
+      # Dump the current tracker to the filesystem.
+      #
+      def to_scope!
+        GUARD.synchronize do
+          scope = Time.now.to_i
+          dump_scope!( scope )
+          scope
+        end  
+      end      
+      
+      # Full path the scrooge configuration file.
+      #
+      def configuration_file
+        @configuration_file ||= File.join( config, CONFIGURATION_FILE )
+      end
+      
+      private
+       
+       def restore_scope!( scope ) #:nodoc:
+         tracker = Scrooge::Tracker::App.new
+         tracker.marshal_load( scope_from_yaml( scope ) )
+         tracker
+       end
+       
+       def dump_scope!( scope ) #:nodoc:
+         ensure_scope_path( scope ) do
+           File.open( scope_path( scope, SCOPE_FILE ), 'w' ) do |io|
+             scope_to_yaml( io )
+           end
+         end
+       end
+       
+       def scope_from_yaml( scope ) #:nodoc:
+         YAML.load( IO.read( scope_path( scope.to_s, SCOPE_FILE ) ) )
+       end
+       
+       def scope_to_yaml( io ) #:nodoc:
+         YAML.dump( Scrooge::Base.profile.tracker.marshal_dump, io )
+       end 
+       
+       def ensure_scope_path( scope ) #:nodoc:
+         makedir_unless_exist( scope_path( scope ) )
+         yield if block_given?  
+       end 
+       
+       def ensure_scopes_path #:nodoc:
+         makedir_unless_exist( scopes_path )
+         yield if block_given?
+       end
+           
+      def makedir_unless_exist( path ) #:nodoc:
+        FileUtils.makedirs( path ) unless File.exist?( path )
+      end     
+               
+    end
+  end
+end