slim_scrooge 0.1.1

data/README.textile

@@ -0,0 +1,85 @@
+h1. SlimScrooge - serious optimisation of mysql for activerecord
+
+h2. What is it?
+
+It's an optimization layer to ensure your application only fetches the database content needed to minimize wire traffic, excessive SQL queries and reduce conversion overheads to native Ruby types.
+
+SlimScrooge implements both lazy loading of attributes fetched from mysql, as well as inline query optimisation, automatically restricting the columns fetched based on what was used during previous passes through the same part of your code.
+
+h2. Benchmark
+
+SlimScrooge performs best when the database is not on the same machine as your rails app.  In this case the overhead of fetching unnecessary data from the database can become more important.
+
+I ran a benchmark that consisted of fetching 400 real urls (culled from the log file) from our complex web app.  In this test I found a consistent 12% improvement in performance over plain active record.  Not earth-shattering, but worthwhile.  In future releases I expect further gains.
+
+h2. Installation
+
+<pre>
+  # if you haven't already, add gemcutter to your gem sources
+  sudo gem install gemcutter
+  gem tumble
+  # install slim_scrooge
+  sudo gem install slim_scrooge
+</pre>
+
+h3. Requirements
+
+* "Slim-attributes":http://github.com/sdsykes/slim-attributes
+
+h3. What it does
+
+<pre>
+  # 1st request, sql is unchanged but columns accesses are recorded
+  Brochure Load SlimScrooged 1st time (27.1ms)   SELECT * FROM `brochures` WHERE (expires_at IS NULL)
+  
+  # 2nd request, only fetch columns that were used the first time
+  Brochure Load SlimScrooged (4.5ms)   SELECT `brochures`.expires_at,`brochures`.operator_id,`brochures`.id FROM 
+  `brochures` WHERE (expires_at IS NULL)
+
+  # 2nd request, later in code we need another column which causes a reload of all remaining columns
+  Brochure Reload SlimScrooged (0.6ms) `brochures`.name,`brochures`.comment,`brochures`.image_height,`brochures`.id,
+  `brochures`.tel,`brochures`.long_comment,`brochures`.image_name,`brochures`.image_width FROM 
+  `brochures` WHERE `brochures`.id IN ('5646','5476','4562','3456','4567','7355')
+
+  # 3rd request
+  Brochure Load SlimScrooged (4.5ms)   SELECT `brochures`.expires_at,`brochures`.operator_id,`brochures`.name,
+  `brochures`.id FROM `brochures` WHERE (expires_at IS NULL)
+</pre>
+
+h3. Technical discussion
+
+SlimScrooge hooks in at just one particular place in the mysql adapter - and that place is the select_all method.  All select queries pass through this method.
+
+SlimScrooge is able to record each call (and where it came from in your code), and to modify queries that do SELECT * FROM en-route to mysql so that they only select the rows that are actually used by that piece of code.
+
+How does SlimScrooge know which columns are actually used?
+
+Enter Slim-attributes.  Slim-attributes also hooks in at one place, the all_hashes method - and in particular it returns a proxy for the hash for each row, and doesn't actually instantiate any ruby objects for the columns in each row until they are needed.
+
+Clearly slim-attributes knows when each column is accessed, and this information can be collected by SlimScrooge for future use.
+
+In fact for efficiency, column accesses are only recorded when they are for newly accessed columns.  Columns we already know about are accessed completely normally, and slim-attributes provides its usual lazy-loading benefit without any additional overhead.
+
+h3. Caveats
+
+It is possible to delete an object and then to try to use its attributes to access another object that perhaps must be also deleted (like the option :dependent=>:destroy in Rails associations).
+
+SlimScrooge particularly checks for columns used by :dependent=>:destroy, but if you are doing something similar manually in your code then you may run into problems.  Your attempt to access the key of the dependent object could cause a reload if the column is not already noted by SlimScrooge, and the reload will fail if you have already destroyed the parent object.
+
+h2. Tests
+
+SlimScrooge performs the full activerecord test suite without errors, except for a couple of tests that check the number of queries performed.
+
+To run the tests you need to set your database up with appropriate access for the rails user, and make activerecord_unittest and activerecord_unittest2 databases.  Then run:
+
+<pre>
+  rake test_with_active_record
+</pre>
+
+h2. References
+* "Scrooge":http://github.com/methodmissing/scrooge
+* "Slim-attributes":http://github.com/sdsykes/slim-attributes
+
+h2. Authors
+* Stephen Sykes (sdsykes)
+* Special thanks to Lourens Naudé (methodmissing) for the original idea, and the C implementation of callsite_hash as well as some other bits of code that I borrowed from the original project.

data/Rakefile

@@ -0,0 +1,30 @@
+require 'rake'
+require 'rake/testtask'
+require 'test/helper'
+
+Rake::TestTask.new(:test_with_active_record) do |t|
+  t.libs << SlimScrooge::ActiveRecordTest::AR_TEST_SUITE
+  t.libs << SlimScrooge::ActiveRecordTest.connection
+  t.test_files = SlimScrooge::ActiveRecordTest.test_files
+  t.ruby_opts = ["-r #{File.join(File.dirname(__FILE__), 'test', 'active_record_setup')}"]
+  t.verbose = true
+end
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = "slim_scrooge"
+    s.summary = "Slim_scrooge - lazy instantiation of attributes and query optimisation for ActiveRecord"
+    s.email = "sdsykes@gmail.com"
+    s.homepage = "http://github.com/sdsykes/slim_scrooge"
+    s.description = "Slim scrooge boosts speed in Rails/Mysql ActiveRecord Models by lazily instantiating attributes as needed, and only querying the database for what is needed."
+    s.authors = ["Stephen Sykes"]
+    s.files = FileList["[A-Z]*", "{ext,lib,test}/**/*"]
+    s.extensions = "ext/extconf.rb"
+    s.add_dependency('slim-attributes', '>= 0.7.0')
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://
+gems.github.com"
+end

data/VERSION.yml

@@ -0,0 +1,5 @@
+--- 
+:major: 0
+:minor: 1
+:build: 
+:patch: 1

data/ext/callsite_hash.c

@@ -0,0 +1,67 @@
+/* Author: Lourens Naudé */
+
+#include "ruby.h"
+#include "node.h"
+#include "env.h"
+
+static int strhash(register const char *string) {
+  register int c;
+
+  register int val = 0;
+
+  while ((c = *string++) != '\0') {
+    val = val*997 + c;
+  }
+
+  return val + (val>>5);
+}
+
+static VALUE rb_f_callsite( VALUE obj ) {
+  struct FRAME *frame = ruby_frame;
+  NODE *n;
+  int lev = -1;
+  int csite = 0;
+
+  if (frame->last_func == ID_ALLOCATOR) {
+    frame = frame->prev;
+  }
+  if (lev < 0) {
+    ruby_set_current_source();
+    if (frame->last_func) {
+      csite += strhash(ruby_sourcefile) + ruby_sourceline + frame->last_func; 
+    }
+    else if (ruby_sourceline == 0) {
+      csite += strhash(ruby_sourcefile);
+    }
+    else {
+      csite += strhash(ruby_sourcefile) + ruby_sourceline; 
+    }
+    if (lev < -1) return INT2FIX(csite);
+  }
+  else {
+    while (lev-- > 0) {
+      frame = frame->prev;
+      if (!frame) {
+        csite = 0;
+        break;
+      }
+    }
+  }
+  for (; frame && (n = frame->node); frame = frame->prev) {
+    if (frame->prev && frame->prev->last_func) {
+      if (frame->prev->node == n) {
+        if (frame->prev->last_func == frame->last_func) continue;
+      }
+      csite += strhash(n->nd_file) + nd_line(n) + frame->prev->last_func;
+    }
+    else {
+      csite += strhash(n->nd_file) + nd_line(n);
+    }
+  }
+
+  return INT2FIX(csite);
+}
+
+void Init_callsite_hash() {
+  rb_define_global_function("callsite_hash", rb_f_callsite, 0);
+}

data/ext/extconf.rb

@@ -0,0 +1,7 @@
+require 'mkmf'
+
+dir_config('callsite_hash')
+
+create_makefile('callsite_hash')
+
+$defs.push("-DRUBY18") if have_var('rb_trap_immediate', ['ruby.h', 'rubysig.h'])

data/lib/slim_scrooge/callsite.rb

@@ -0,0 +1,68 @@
+# Author: Stephen Sykes
+
+module SlimScrooge
+  class Callsite
+    ScroogeComma = ",".freeze 
+    ScroogeRegexJoin = /(?:LEFT|INNER|OUTER|CROSS)*\s*(?:STRAIGHT_JOIN|JOIN)/i
+
+    attr_accessor :seen_columns
+    attr_reader :columns_hash, :primary_key, :model_class
+
+    class << self
+      def make_callsite(model_class, original_sql)
+        if use_scrooge?(model_class, original_sql)
+          new(model_class)
+        else
+          nil
+        end
+      end
+
+      def use_scrooge?(model_class, original_sql)
+        original_sql =~ select_regexp(model_class.table_name) && 
+        model_class.columns_hash.has_key?(model_class.primary_key) && 
+        original_sql !~ ScroogeRegexJoin
+      end
+      
+      def select_regexp(table_name)
+        %r{SELECT (`?(?:#{table_name})?`?.?\\*) FROM}
+      end
+    end
+
+    def initialize(model_class)
+      @all_columns = SimpleSet.new(model_class.column_names)
+      @model_class = model_class
+      @quoted_table_name = model_class.quoted_table_name
+      @primary_key = model_class.primary_key
+      @columns_hash = model_class.columns_hash
+      @select_regexp = self.class.select_regexp(model_class.table_name)
+      @seen_columns = SimpleSet.new(essential_columns)
+    end
+
+    def essential_columns
+      @model_class.reflect_on_all_associations.inject([@model_class.primary_key]) do |arr, assoc|
+        if assoc.options[:dependent] && assoc.macro == :belongs_to
+          arr << assoc.association_foreign_key
+        end
+        arr
+      end
+    end
+
+    def scrooged_sql(seen_columns, sql)
+      sql.gsub(@select_regexp, "SELECT #{scrooge_select_sql(seen_columns)} FROM")
+    end
+    
+    def missing_columns(fetched_columns)
+      (@all_columns - SimpleSet.new(fetched_columns)) << @primary_key
+    end
+    
+    def reload_sql(primary_keys, fetched_columns)
+      sql_keys = primary_keys.collect{|pk| "'#{pk}'"}.join(ScroogeComma)
+      cols = scrooge_select_sql(missing_columns(fetched_columns))
+      "SELECT #{cols} FROM #{@quoted_table_name} WHERE #{@quoted_table_name}.#{@primary_key} IN (#{sql_keys})"
+    end
+
+    def scrooge_select_sql(set)
+      set.collect{|name| "#{@quoted_table_name}.#{name}"}.join(ScroogeComma)
+    end
+  end
+end

data/lib/slim_scrooge/callsites.rb

@@ -0,0 +1,44 @@
+# Author: Stephen Sykes
+
+module SlimScrooge
+  class Callsites
+    CallsitesMutex = Mutex.new
+    @@callsites = {}
+
+    class << self
+      def has_key?(callsite_key)
+        @@callsites.has_key?(callsite_key)
+      end
+
+      def [](callsite_key)
+        @@callsites[callsite_key]
+      end
+
+      def callsite_key(callsite_hash, sql)
+        callsite_hash + sql.gsub(/WHERE.*/i, "").hash
+      end
+
+      def create(sql, callsite_key, name)
+        begin
+          model_class = name.split.first.constantize
+        rescue NameError, NoMethodError
+          add_callsite(callsite_key, nil)
+        else
+          add_callsite(callsite_key, Callsite.make_callsite(model_class, sql))
+        end
+      end
+
+      def add_callsite(callsite_key, callsite)
+        CallsitesMutex.synchronize do
+          @@callsites[callsite_key] = callsite
+        end
+      end
+
+      def add_seen_column(callsite, seen_column)
+        CallsitesMutex.synchronize do
+          callsite.seen_columns << seen_column
+        end
+      end
+    end
+  end
+end

data/lib/slim_scrooge/monitored_hash.rb

@@ -0,0 +1,39 @@
+# Author: Stephen Sykes
+
+module SlimScrooge
+  class MonitoredHash < Hash
+    attr_accessor :callsite, :result_set
+    
+    def self.[](original_hash, callsite, result_set)
+      hash = super(original_hash)
+      hash.callsite = callsite
+      hash.result_set = result_set
+      hash
+    end
+    
+    def [](name)
+      if @callsite.columns_hash.has_key?(name)
+        @result_set.reload! if @result_set && name != @callsite.primary_key
+        Callsites.add_seen_column(@callsite, name)
+      end
+      super
+    end
+    
+    def []=(name, value)
+      if @result_set && @callsite.columns_hash.has_key?(name)
+        @result_set.reload!
+      end
+      super
+    end
+    
+    def keys
+      @result_set ? @callsite.columns_hash.keys : super
+    end
+    
+    def has_key?(name)
+      @result_set ? @callsite.columns_hash.has_key?(name) : super
+    end
+    
+    alias_method :include?, :has_key?
+  end
+end

data/lib/slim_scrooge/result_set.rb

@@ -0,0 +1,31 @@
+# Author: Stephen Sykes
+
+module SlimScrooge
+  class ResultSet
+    attr_reader :rows, :callsite_key
+    
+    def initialize(rows, callsite_key, fetched_columns)
+      @rows = rows
+      @callsite_key = callsite_key
+      @fetched_columns = fetched_columns
+    end
+    
+    def rows_by_key(key)
+      @rows.inject({}) {|hash, row| hash[row[key]] = row; hash}
+    end
+    
+    def reload!
+      callsite = Callsites[@callsite_key]
+      rows_hash = rows_by_key(callsite.primary_key)
+      sql = callsite.reload_sql(rows_hash.keys, @fetched_columns)
+      model_class = callsite.model_class
+      new_rows = model_class.connection.send(:select, sql, "#{model_class.name} Reload SlimScrooged")
+      new_rows.each do |row|
+        if old_row = rows_hash[row[callsite.primary_key]]
+          old_row.real_hash.result_set = nil
+          row.each {|col, value| old_row[col] = value}
+        end
+      end
+    end
+  end
+end

data/lib/slim_scrooge/simple_set.rb

@@ -0,0 +1,34 @@
+module SlimScrooge
+  class SimpleSet < Hash
+    class << self
+      # Creates a new set containing the given objects
+      def [](*ary)
+        new(ary)
+      end
+    end
+ 
+    # Create a new SimpleSet containing the unique members of _arr_
+    def initialize(arr = [])
+      Array(arr).each {|x| self[x] = true}
+    end
+ 
+    # Add a value to the set, and return it
+    def <<(value)
+      self[value] = true
+      self
+    end
+
+    # Invokes block once for each item in the set. Creates an array
+    # containing the values returned by the block.
+    def collect(&block)
+      keys.collect(&block)
+    end
+
+    alias_method :to_a, :keys
+
+    # Returns set after elements in other have been removed
+    def -(other)
+      SimpleSet.new(collect {|k| other[k] ? nil : k}.compact)
+    end
+  end
+end

data/lib/slim_scrooge/slim_scrooge.rb

@@ -0,0 +1,31 @@
+# Author: Stephen Sykes
+
+module SlimScrooge
+  module SelectAll
+    def self.included(base)
+      base.alias_method_chain :select_all, :slim_scrooge
+    end
+
+    def select_all_with_slim_scrooge(sql, name = nil)
+      callsite_key = SlimScrooge::Callsites.callsite_key(callsite_hash, sql)
+      if SlimScrooge::Callsites.has_key?(callsite_key)
+        if callsite = SlimScrooge::Callsites[callsite_key]
+          seen_columns = callsite.seen_columns.dup
+          rows = select_all_without_slim_scrooge(callsite.scrooged_sql(seen_columns, sql), name + " SlimScrooged")
+          result_set = SlimScrooge::ResultSet.new(rows.dup, callsite_key, seen_columns)
+          rows.each {|row| row.real_hash = MonitoredHash[{}, callsite, result_set]}
+        else
+          select_all_without_slim_scrooge(sql, name)
+        end
+      elsif callsite = SlimScrooge::Callsites.create(sql, callsite_key, name)
+        rows = select_all_without_slim_scrooge(sql, name + " SlimScrooged 1st time")
+        rows.each {|row| row.real_hash = MonitoredHash[row.to_hash, callsite, nil]}
+        rows
+      else
+        select_all_without_slim_scrooge(sql, name)
+      end
+    end
+  end
+end
+
+ActiveRecord::ConnectionAdapters::MysqlAdapter.send(:include, SlimScrooge::SelectAll)

data/lib/slim_scrooge.rb

@@ -0,0 +1,10 @@
+# Author: Stephen Sykes
+
+require 'callsite_hash'
+require 'slim_attributes'
+require 'slim_scrooge/simple_set'
+require 'slim_scrooge/callsites'
+require 'slim_scrooge/callsite'
+require 'slim_scrooge/result_set'
+require 'slim_scrooge/monitored_hash'
+require 'slim_scrooge/slim_scrooge'

data/test/active_record_setup.rb

@@ -0,0 +1,3 @@
+require File.join(File.dirname(__FILE__), 'helper')
+
+SlimScrooge::ActiveRecordTest.setup

data/test/helper.rb

@@ -0,0 +1,91 @@
+require File.join(File.dirname(__FILE__), 'setup')
+require 'active_support/test_case'
+
+module SlimScrooge
+  class Test
+    class << self
+
+      def setup
+        setup_constants
+        make_sqlite_config
+        make_sqlite_connection
+        load_models
+        load(SCHEMA_ROOT + "/schema.rb")
+        require 'test/unit'
+      end
+
+      def test_files
+        glob("#{File.dirname(__FILE__)}/**/*_test.rb")
+      end
+
+      def test_model_files
+        %w{course}
+      end
+
+      private
+
+      def setup_constants
+        set_constant('TEST_ROOT') {File.expand_path(File.dirname(__FILE__))}
+        set_constant('SCHEMA_ROOT') {TEST_ROOT + "/schema"}
+      end
+      
+      def make_sqlite_config
+        ActiveRecord::Base.configurations = {
+          'db' => {
+            :adapter => 'sqlite3',
+            :database => 'test_db',
+            :timeout => 5000
+          }
+        }
+      end
+      
+      def load_models
+        test_model_files.each {|f| require File.join(File.dirname(__FILE__), "models", f)}
+      end
+
+      def make_sqlite_connection
+        ActiveRecord::Base.establish_connection(ActiveRecord::Base.configurations['db'])
+      end
+      
+      def set_constant(constant)
+        Object.const_set(constant, yield) unless Object.const_defined?(constant)
+      end
+      
+      def glob(pattern)
+        Dir.glob(pattern)
+      end
+    end
+  end
+  
+  class ActiveRecordTest < Test
+    class << self
+      def setup
+        setup_constants
+      end
+      
+      def test_files
+        glob("#{AR_TEST_SUITE}/cases/**/*_test.rb").sort
+      end
+
+      def connection
+        File.join(AR_TEST_SUITE, 'connections', 'native_mysql')
+      end
+
+      private
+
+      def setup_constants
+        set_constant('MYSQL_DB_USER') {'rails'}
+        set_constant('AR_TEST_SUITE') {find_active_record_test_suite}
+      end
+ 
+      def find_active_record_test_suite
+        ts = ($:).grep(/activerecord/).last.split('/')
+        ts.pop
+        ts << 'test'
+        ts.join('/')
+      end
+    end
+
+    AR_TEST_SUITE = find_active_record_test_suite
+  end
+end

data/test/models/course.rb

@@ -0,0 +1,2 @@
+class Course < ActiveRecord::Base
+end

data/test/schema/schema.rb

@@ -0,0 +1,5 @@
+ActiveRecord::Schema.define do
+  create_table :courses, :force => true do |t|
+    t.column :name, :string, :null => false
+  end
+end

data/test/setup.rb

@@ -0,0 +1,5 @@
+require 'rubygems'
+require 'active_record'
+require 'active_record/connection_adapters/mysql_adapter'
+$:.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+require 'slim_scrooge'

metadata

@@ -0,0 +1,84 @@
+--- !ruby/object:Gem::Specification 
+name: slim_scrooge
+version: !ruby/object:Gem::Version 
+  version: 0.1.1
+platform: ruby
+authors: 
+- Stephen Sykes
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-10-23 00:00:00 +03:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: slim-attributes
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.7.0
+    version: 
+description: Slim scrooge boosts speed in Rails/Mysql ActiveRecord Models by lazily instantiating attributes as needed, and only querying the database for what is needed.
+email: sdsykes@gmail.com
+executables: []
+
+extensions: 
+- ext/extconf.rb
+extra_rdoc_files: 
+- README.textile
+files: 
+- README.textile
+- Rakefile
+- VERSION.yml
+- ext/callsite_hash.c
+- ext/extconf.rb
+- lib/slim_scrooge.rb
+- lib/slim_scrooge/callsite.rb
+- lib/slim_scrooge/callsites.rb
+- lib/slim_scrooge/monitored_hash.rb
+- lib/slim_scrooge/result_set.rb
+- lib/slim_scrooge/simple_set.rb
+- lib/slim_scrooge/slim_scrooge.rb
+- test/active_record_setup.rb
+- test/helper.rb
+- test/models/course.rb
+- test/schema/schema.rb
+- test/setup.rb
+has_rdoc: true
+homepage: http://github.com/sdsykes/slim_scrooge
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.5
+signing_key: 
+specification_version: 3
+summary: Slim_scrooge - lazy instantiation of attributes and query optimisation for ActiveRecord
+test_files: 
+- test/active_record_setup.rb
+- test/helper.rb
+- test/models/course.rb
+- test/schema/schema.rb
+- test/setup.rb