slim_scrooge 1.0.0 → 1.0.1

data/README.textile

@@ -1,10 +1,10 @@
-h1. SlimScrooge - serious optimisation of mysql for activerecord
+h1. SlimScrooge - serious optimisation for activerecord
 
 h2. What is it?
 
 It's an optimization layer to ensure your application only fetches the database content needed to minimize wire traffic, excessive SQL queries and reduce conversion overheads to native Ruby types.
 
-SlimScrooge implements both lazy loading of attributes fetched from mysql, as well as inline query optimisation, automatically restricting the columns fetched based on what was used during previous passes through the same part of your code.
+SlimScrooge implements inline query optimisation, automatically restricting the columns fetched based on what was used during previous passes through the same part of your code.
 
 SlimScrooge is similar to (and is partly derived from) "Scrooge":http://github.com/methodmissing/scrooge but has many fewer lines of code and is faster.
 
@@ -12,7 +12,11 @@ h2. Benchmark
 
 SlimScrooge performs best when the database is not on the same machine as your rails app.  In this case the overhead of fetching unnecessary data from the database can become more important.
 
-I ran a benchmark that consisted of fetching 400 real urls (culled from the log file) from our complex web app.  In this test I found a consistent *12% improvement* in performance over plain active record.  Not earth-shattering, but worthwhile.  In future releases I expect further gains.
+I ran a benchmark that consisted of fetching 400 real urls (culled from the log file) from our complex web app.  In this test I found a consistent *12% improvement* in performance over plain active record.  Not earth-shattering, but worthwhile.
+
+Note that this result was for comparing the time taken for running complete requests through rails - of which database accesses are only one part.  So the result is better than it at first sounds.
+
+In future releases I expect further gains.
 
 h2. Installation
 
@@ -54,7 +58,7 @@ h2. Technical discussion
 
 SlimScrooge hooks in at just one particular place in ActiveRecord - and that place is the find_all_by_sql method.  All select queries pass through this method.
 
-SlimScrooge is able to record each call (and where it came from in your code), and to modify queries that do SELECT * FROM en-route to mysql so that they only select the rows that are actually used by that piece of code.
+SlimScrooge is able to record each call (and where it came from in your code), and to modify queries that do SELECT * FROM en-route to the database so that they only select the rows that are actually used by that piece of code.
 
 How does SlimScrooge know which columns are actually used?
 

data/Rakefile

@@ -14,14 +14,13 @@ begin
   require 'jeweler'
   Jeweler::Tasks.new do |s|
     s.name = "slim_scrooge"
-    s.summary = "Slim_scrooge - lazy instantiation of attributes and query optimisation for ActiveRecord"
+    s.summary = "Slim_scrooge - serious optimisation for ActiveRecord"
     s.email = "sdsykes@gmail.com"
     s.homepage = "http://github.com/sdsykes/slim_scrooge"
-    s.description = "Slim scrooge boosts speed in Rails/Mysql ActiveRecord Models by lazily instantiating attributes as needed, and only querying the database for what is needed."
+    s.description = "Slim scrooge boosts speed in Rails ActiveRecord Models by only querying the database for what is needed."
     s.authors = ["Stephen Sykes"]
     s.files = FileList["[A-Z]*", "{ext,lib,test}/**/*"]
     s.extensions = "ext/extconf.rb"
-    s.add_dependency('slim-attributes', '>= 0.7.0')
   end
   Jeweler::GemcutterTasks.new
 rescue LoadError

data/VERSION.yml

@@ -1,5 +1,5 @@
 --- 
+:major: 1
 :minor: 0
-:patch: 0
+:patch: 1
 :build: 
-:major: 1

data/lib/slim_scrooge/callsite.rb

@@ -1,6 +1,9 @@
 # Author: Stephen Sykes
 
 module SlimScrooge
+  # A Callsite contains the list of columns that are accessed when an SQL
+  # query is made from a particular place in the app
+  #
   class Callsite
     ScroogeComma = ",".freeze 
     ScroogeRegexJoin = /(?:LEFT|INNER|OUTER|CROSS)*\s*(?:STRAIGHT_JOIN|JOIN)/i
@@ -9,6 +12,8 @@ module SlimScrooge
     attr_reader :columns_hash, :primary_key, :model_class
 
     class << self
+      # Make a callsite if the query is of the right type for us to optimise
+      #
       def make_callsite(model_class, original_sql)
         if use_scrooge?(model_class, original_sql)
           new(model_class)
@@ -17,12 +22,17 @@ module SlimScrooge
         end
       end
 
+      # Check if query can be optimised
+      #
       def use_scrooge?(model_class, original_sql)
         original_sql =~ select_regexp(model_class.table_name) && 
         model_class.columns_hash.has_key?(model_class.primary_key) && 
         original_sql !~ ScroogeRegexJoin
       end
       
+      # The regexp that enables us to replace the * from SELECT * with
+      # the list of columns we actually need
+      #
       def select_regexp(table_name)
         %r{SELECT (`?(?:#{table_name})?`?.?\\*) FROM}
       end
@@ -38,6 +48,8 @@ module SlimScrooge
       @seen_columns = SimpleSet.new(essential_columns)
     end
 
+    # List of columns that should always be fetched no matter what
+    #
     def essential_columns
       @model_class.reflect_on_all_associations.inject([@model_class.primary_key]) do |arr, assoc|
         if assoc.options[:dependent] && assoc.macro == :belongs_to
@@ -47,20 +59,29 @@ module SlimScrooge
       end
     end
 
+    # Returns suitable sql given a list of columns and the original query
+    #
     def scrooged_sql(seen_columns, sql)
       sql.gsub(@select_regexp, "SELECT #{scrooge_select_sql(seen_columns)} FROM")
     end
     
+    # List if columns what were not fetched
+    #
     def missing_columns(fetched_columns)
       (@all_columns - SimpleSet.new(fetched_columns)) << @primary_key
     end
     
+    # Returns sql for fetching the unfetched columns for all the rows
+    # in the result set, specified by primary_keys
+    #
     def reload_sql(primary_keys, fetched_columns)
       sql_keys = primary_keys.collect{|pk| "'#{pk}'"}.join(ScroogeComma)
       cols = scrooge_select_sql(missing_columns(fetched_columns))
       "SELECT #{cols} FROM #{@quoted_table_name} WHERE #{@quoted_table_name}.#{@primary_key} IN (#{sql_keys})"
     end
 
+    # Change a set of columns into a correctly quoted comma separated list
+    #
     def scrooge_select_sql(set)
       set.collect do |name|
         "#{@quoted_table_name}.#{@model_class.connection.quote_column_name(name)}"

data/lib/slim_scrooge/callsites.rb

@@ -1,23 +1,34 @@
 # Author: Stephen Sykes
 
 module SlimScrooge
+  # Contains the complete list of callsites
+  #
   class Callsites
     CallsitesMutex = Mutex.new
     @@callsites = {}
 
     class << self
+      # Whether we have encountered a callsite before
+      #
       def has_key?(callsite_key)
         @@callsites.has_key?(callsite_key)
       end
 
+      # Return the callsite for this key
+      #
       def [](callsite_key)
         @@callsites[callsite_key]
       end
 
+      # Generate a key string - uses the portion of the query before the WHERE 
+      # together with the callsite_hash generated by callsite_hash.c
+      #
       def callsite_key(callsite_hash, sql)
         callsite_hash + sql.gsub(/WHERE.*/i, "").hash
       end
 
+      # Create a new callsite
+      # 
       def create(sql, callsite_key, name)
         begin
           model_class = name.split.first.constantize
@@ -28,12 +39,16 @@ module SlimScrooge
         end
       end
 
+      # Add a new callsite, wrap in a mutex for safety
+      #
       def add_callsite(callsite_key, callsite)
         CallsitesMutex.synchronize do
           @@callsites[callsite_key] = callsite
         end
       end
 
+      # Record that a column was accessed, wrap in a mutex for safety
+      #
       def add_seen_column(callsite, seen_column)
         CallsitesMutex.synchronize do
           callsite.seen_columns << seen_column

data/lib/slim_scrooge/monitored_hash.rb

@@ -1,9 +1,20 @@
 # Author: Stephen Sykes
 
 module SlimScrooge
+  # A MonitoredHash allows us to return only some columns into the @attributes
+  # of an ActiveRecord model object, but to notice when an attribute that
+  # wasn't fetched is accessed.
+  #
+  # Also, when a result is first fetched for a particular callsite, we monitor
+  # all the columns so that we can immediately learn which columns are needed.
+  #
   class MonitoredHash < Hash
     attr_accessor :callsite, :result_set, :monitored_columns
     
+    # Create a monitored hash.  The unmonitored_columns are accessed like a regular
+    # hash.  The monitored columns kept separately, and new_column_access is called
+    # before they are returned.
+    #
     def self.[](monitored_columns, unmonitored_columns, callsite)
       hash = MonitoredHash.new {|hash, key| hash.new_column_access(key)}
       hash.monitored_columns = monitored_columns
@@ -12,6 +23,10 @@ module SlimScrooge
       hash
     end
     
+    # Called when an unknown column is requested, through the default proc.
+    # If the column requested is valid, and the result set is not completely
+    # loaded, then we reload.  Otherwise just note the column with add_seen_column.
+    #
     def new_column_access(name)
       if @callsite.columns_hash.has_key?(name)
         @result_set.reload! if @result_set && name != @callsite.primary_key
@@ -20,6 +35,8 @@ module SlimScrooge
       @monitored_columns[name]
     end
     
+    # Reload if needed before allowing assignment
+    #
     def []=(name, value)
       if has_key?(name)
         return super
@@ -30,16 +47,22 @@ module SlimScrooge
       @monitored_columns[name] = value
     end
     
+    # Returns the column names
+    #
     def keys
       @result_set ? @callsite.columns_hash.keys : super | @monitored_columns.keys
     end
     
+    # Check for a column name
+    #
     def has_key?(name)
       @result_set ? @callsite.columns_hash.has_key?(name) : super || @monitored_columns.has_key?(name)
     end
     
     alias_method :include?, :has_key?
     
+    # Called by Hash#update when reload is called on an ActiveRecord object
+    #
     def to_hash
       @result_set.reload! if @result_set
       @monitored_columns.merge(self)
@@ -58,6 +81,14 @@ module SlimScrooge
   end
 end
 
+# We need to change the update method of Hash so that it *always* calls
+# to_hash.  This is because it normally checks if other_hash is a kind of
+# Hash, and doesn't bother calling to_hash if so.  But we need it to call
+# to_hash, because otherwise update will not get the complete columns
+# from a MonitoredHash
+#
+# This is not harmful - to_hash in a regular Hash just returns self.
+#
 class Hash
   alias_method :c_update, :update
   def update(other_hash, &block)

data/lib/slim_scrooge/result_set.rb

@@ -1,6 +1,11 @@
 # Author: Stephen Sykes
 
 module SlimScrooge
+  # A ResultSet contains all the rows found by an sql query
+  # A call to reload! will cause all the rows in the set to be fully loaded
+  # from the database - this should be called when a column access that hasn't previously
+  # been seen by SlimScrooge is encountered
+  #
   class ResultSet
     attr_reader :rows, :callsite_key
     
@@ -14,6 +19,9 @@ module SlimScrooge
       @rows.inject({}) {|hash, row| hash[row[key]] = row; hash}
     end
     
+    # Reload all the rows in the sql result at once
+    # Reloads only those columns we didn't fetch the first time
+    #
     def reload!
       callsite = Callsites[@callsite_key]
       rows_hash = rows_by_key(callsite.primary_key)

data/lib/slim_scrooge/slim_scrooge.rb

@@ -13,19 +13,8 @@ module SlimScrooge
       def find_by_sql_with_slim_scrooge(sql)
         return find_by_sql_without_slim_scrooge(sql) if sql.is_a?(Array) # don't mess with user's custom query
         callsite_key = SlimScrooge::Callsites.callsite_key(callsite_hash, sql)
-        if SlimScrooge::Callsites.has_key?(callsite_key)      # seen before
-          if callsite = SlimScrooge::Callsites[callsite_key]  # and is scroogeable
-            seen_columns = callsite.seen_columns.dup          # dup so cols aren't changed underneath us
-            rows = connection.select_all(callsite.scrooged_sql(seen_columns, sql), "#{name} Load SlimScrooged")
-            rows.collect! {|row| MonitoredHash[{}, row, callsite]}
-            result_set = SlimScrooge::ResultSet.new(rows.dup, callsite_key, seen_columns)
-            rows.collect! do |row|
-              row.result_set = result_set
-              instantiate(row)
-            end
-          else
-            find_by_sql_without_slim_scrooge(sql)
-          end
+        if SlimScrooge::Callsites.has_key?(callsite_key)
+          find_with_callsite_key(sql, callsite_key)
         elsif callsite = SlimScrooge::Callsites.create(sql, callsite_key, name)  # new site that is scroogeable
           rows = connection.select_all(sql, "#{name} Load SlimScrooged 1st time")
           rows.collect! {|row| instantiate(MonitoredHash[row, {}, callsite])}
@@ -33,6 +22,23 @@ module SlimScrooge
           find_by_sql_without_slim_scrooge(sql)
         end
       end
+
+      private
+
+      def find_with_callsite_key(sql, callsite_key)
+        if callsite = SlimScrooge::Callsites[callsite_key]
+          seen_columns = callsite.seen_columns.dup          # dup so cols aren't changed underneath us
+          rows = connection.select_all(callsite.scrooged_sql(seen_columns, sql), "#{name} Load SlimScrooged")
+          rows.collect! {|row| MonitoredHash[{}, row, callsite]}
+          result_set = SlimScrooge::ResultSet.new(rows.dup, callsite_key, seen_columns)
+          rows.collect! do |row|
+            row.result_set = result_set
+            instantiate(row)
+          end
+        else
+          find_by_sql_without_slim_scrooge(sql)
+        end
+      end
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: slim_scrooge
 version: !ruby/object:Gem::Version 
-  version: 1.0.0
+  version: 1.0.1
 platform: ruby
 authors: 
 - Stephen Sykes
@@ -9,20 +9,11 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-11-06 00:00:00 +02:00
+date: 2009-11-07 00:00:00 +02:00
 default_executable: 
-dependencies: 
-- !ruby/object:Gem::Dependency 
-  name: slim-attributes
-  type: :runtime
-  version_requirement: 
-  version_requirements: !ruby/object:Gem::Requirement 
-    requirements: 
-    - - ">="
-      - !ruby/object:Gem::Version 
-        version: 0.7.0
-    version: 
-description: Slim scrooge boosts speed in Rails/Mysql ActiveRecord Models by lazily instantiating attributes as needed, and only querying the database for what is needed.
+dependencies: []
+
+description: Slim scrooge boosts speed in Rails ActiveRecord Models by only querying the database for what is needed.
 email: sdsykes@gmail.com
 executables: []
 
@@ -75,7 +66,7 @@ rubyforge_project:
 rubygems_version: 1.3.5
 signing_key: 
 specification_version: 3
-summary: Slim_scrooge - lazy instantiation of attributes and query optimisation for ActiveRecord
+summary: Slim_scrooge - serious optimisation for ActiveRecord
 test_files: 
 - test/active_record_setup.rb
 - test/helper.rb