ferblape-query_memcached 2.2.2

data/.gitignore

@@ -0,0 +1,2 @@
+test/testing_app/log/
+.svn

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 [Fernando Blat]
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,87 @@
+# QueryMemcached
+
+<http://github.com/ferblape/query_memcached>
+
+## Install
+
+In the root directory of your project:
+
+    script/plugin install git://github.com/ferblape/query_memcached.git
+
+If you want to install it like a git module you can follow this post, adapting the URLs to this plugin:
+
+<http://woss.name/2008/04/09/using-git-submodules-to-track-vendorrails/>
+
+## Requirements
+
+  - 1.9 >= Ruby >= 1.8.4
+  
+  - 2.2.2 >= Rails >= 2.1 
+  
+  - memcache-client gem (maybe it will work also in memcached gem by fauna, but I haven't tested it yet)
+  
+  - MySQL or PostgreSQL
+
+## Description
+
+This plugin tries to replace ActiveRecord query_cache, adding a Memcache layer for persistence of the query's cache between requests. 
+
+It is, instead of saving all the SQL query's results in memory each request you save them into Memcached, so you have them all the time.
+
+For expiring this cache, each Memcached key contains a sum of all the version numbers of the tables involved in the query. If one of that tables is modified, then the version number for that table is increased.
+
+For example, the query below involves the table _items_ and the table _places_:
+
+    SELECT * from items INNER JOIN places ON places.item_id = item.id WHERE (item.created_at >= '2008-02-02 00:00:00')
+    
+So, the version for the cache of that query will be the sum of the cache version of the table _items_ and the cache version of the table _places_.
+
+The idea behind this is way of create and expire cache was inspired by the post [The Secret to Memcached](http://blog.leetsoft.com/2007/5/22/the-secret-to-memcached) by Tobias Lütke.
+
+For this plugin to work it's supposed that your ActionController cache store configured is `mem_cache_store`.
+
+I changed the behavior in one major way.  I made the memcaching of the QueryCache optional.  
+
+You need to add `enable_memache_querycache` to your AcitveRecord model, like so:
+
+    class User < ActiveRecord::Base
+      enable_memcache_querycache :expires_in => 10.minutes
+    end
+
+Additionally you can indicate in what time should expire the cache. 90 minutes is the default value.
+
+The reason for this (drastic) change is two fold:
+
+  - For starters, there are many tables where trying to cache the contents but expiring all caching on any insert/update/delete/drop/alter on the table causes unnecessary overhead.  A sessions table is a perfect example.  I also have a metrics table and a few other tables where the contents are changed _often_.  By not enabling memcache on tables that I know will constantly be changing I can save quite a number of needless memcache calls (not caching the session queries saves two reads and two writes per request).
+  
+  - The other reason is I don't quite trust the implications of having a persisted query cache. I want to carefully roll it out, starting with just the few models that rarely change, and go from there. I'm not worried about users seeing information they shouldn't be, as the key is the query; it is more about making sure things expire correctly. I didn't want to push such a large caching change into my app without a careful (and long) rollout.
+
+## Known issues
+
+  - You can get race conditions in the version keys of tables stored in Memcached
+  
+## Running plugin tests
+
+The tests of this plugin are not so standard so I decided to wrote a very explicative instructions in a file apart named RUNNING_TESTS
+
+## TODO
+
+There is a list of pending features, bugs, and so on in a file named TODO.
+
+Any comments and suggestions are welcome.
+
+## Another comments
+
+It's so easy to adapt the plugin for Rails version 2.0.2 if you change the cache variable for another instantiated with the memcache-client.
+
+Also, it is possible to run in Rails >= 1.2.4 if you change the plugin [query\_cache](http://agilewebdevelopment.com/plugins/query_cache) with the changes of query_memcached.
+
+## Special Thanks
+
+  - Raul Murciano <http://raul.murciano.net/> for helping to adapt the plugin to Rails 2.1
+  
+  - methodmissing <http://blog.methodmissing.com/> for some fresh ideas, lock library and correct some mistakes
+  
+  - skippy <http://github.com/skippy> for a lot of work deleting and cleaning methods, and also doing this plugin optional to each model
+
+Copyright (c) 2008 [Fernando Blat](http://www.inwebwetrust.net), released under the MIT license

data/RUNNING_TESTS

@@ -0,0 +1,18 @@
+= Running tests
+
+For testing this plugin I decided to adapt a the query cache tests' from ActiveRecord. 
+
+First of all you have to know that there is a Rails 2.1 application in `test/testing_app`.
+
+So you need a database as indicated in `config/database.yml`:
+
+  user: root
+  adapter: mysql
+  database: 'activerecord_unittest'
+  encoding: utf8
+  
+Then, load the schema from `db/schema.rb`, and launch a memcache server in standar port 11211.
+
+Now, you can go to the plugin root directory and perform the test task:
+
+  rake test

data/Rakefile

@@ -0,0 +1,37 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc 'Test the query_memcached plugin.'
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = true
+end
+
+desc 'Generate documentation for the query_memcached plugin.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'QueryMemcached'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "query_memcached"
+    gem.summary = "A replacement for ActiveRecord query_cache that a adds a Memcache layer for persistence of the query's cache"
+    gem.email = ["ferblape@gmail.com"]
+    gem.homepage = "http://github.com/ferblape/query_memcached"
+    gem.authors = ["Fernando Blat"]
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: sudo gem install jeweler"
+end

data/TODO

@@ -0,0 +1,6 @@
+= TODO
+
+  - prevent race conditions [1], [2]. Test de library lock in lib/
+
+[1] http://www.eu.socialtext.net/memcached/index.cgi?faq#race_conditions_and_stale_data
+[2] http://www.eu.socialtext.net/memcached/index.cgi?faq#emulating_locking_with_the_add_command

data/init.rb

@@ -0,0 +1,3 @@
+if Object.const_defined?( 'ActionController' )
+  require 'query_memcached'
+end

data/lib/extensions/lock.rb

@@ -0,0 +1,31 @@
+# Experimental LOCK support ported form http://github.com/fauna/interlock/tree/master/lib/interlock/lock.rb
+module ActiveSupport
+  module Cache
+    class MemCacheStore < Store
+
+      def lock(key, lock_expiry = 30, retries = 5)
+        retries.times do |count|
+
+          begin
+            response = @data.add("lock:#{key}", "Locked by #{Process.pid}", lock_expiry)
+            response ||= Response::STORED
+          rescue Object => e
+          end
+
+          if response == Response::STORED
+            begin
+              value = yield( @data.get(key) )
+              @data.set(key, value)
+              return value
+            ensure
+              @data.delete("lock:#{key}")
+            end
+          else
+            sleep((2**count) / 2.0)
+          end
+        end
+      end
+
+    end
+  end
+end

data/lib/query_memcached.rb

@@ -0,0 +1,191 @@
+require 'digest/md5'
+
+module ActiveRecord
+  
+  class Base
+
+    # table_names is a special attribute that contains a regular expression with all the tables of the application
+    # Its main purpose  is to detect all the tables that a query affects.
+    # It is build in a special way:
+    #   - first we get all the tables
+    #   - then we sort them from major to minor lenght, in order to detect tables which name is a composition of two
+    #     names, i.e, posts, comments and comments_posts. It is for make easier the regular expression
+    #   - and finally, the regular expression is built
+    cattr_accessor :table_names, :enableMemcacheQueryForModels
+    
+    self.table_names = /#{connection.tables.sort_by { |c| c.length }.join('|')}/i
+    self.enableMemcacheQueryForModels ||= {}
+
+    class << self
+            
+      # put this class method at the top of your AR model to enable memcache for the queryCache, 
+      # otherwise it will use standard query cache
+      def enable_memcache_querycache(options = {})
+        if ActionController::Base.perform_caching && defined?(::Rails.cache) && ::Rails.cache.is_a?(ActiveSupport::Cache::MemCacheStore)
+          options[:expires_in] ||= 90.minutes
+          self.enableMemcacheQueryForModels[ActiveRecord::Base.send(:class_name_of_active_record_descendant, self).to_s] = options
+        else
+          warning = "[Query memcached WARNING] Disabled for #{ActiveRecord::Base.send(:class_name_of_active_record_descendant, self)} -- Memcache for QueryCache is not enabled for this model because caching is not turned on, Rails.cache is not defined, or cache engine is not mem_cache_store"
+          ActiveRecord::Base.logger.error warning
+        end
+      end
+      
+      def connection_with_memcache_query_cache
+        conn = connection_without_memcache_query_cache
+        conn.memcache_query_cache_options = self.enableMemcacheQueryForModels[self.to_s]
+        conn
+      end
+      
+      alias_method_chain :connection, :memcache_query_cache
+      
+      def cache_version_key(table_name = nil)
+        "#{global_cache_version_key}/#{table_name || self.table_name}"
+      end
+
+      def global_cache_version_key; 'version' end
+
+      # Increment the class version key number
+      def increase_version!(table_name = nil)
+        key = cache_version_key(table_name)
+        if r = ::Rails.cache.read(key)
+          ::Rails.cache.write(key, r.to_i + 1)
+        else
+          ::Rails.cache.write(key, 1)
+        end
+      end
+      
+      # Given a sql query this method extract all the table names of the database affected by the query
+      # thanks to the regular expression we have generated on the load of the plugin
+      def extract_table_names(sql)
+        sql.gsub(/`/,'').scan(self.table_names).map {|t| t.strip}.uniq
+      end
+
+    end
+
+  end
+
+  module ConnectionAdapters # :nodoc:
+    
+    class AbstractAdapter
+      attr_accessor :memcache_query_cache_options
+    end
+
+    class MysqlAdapter < AbstractAdapter
+      
+      # alias_method_chain for expiring cache if necessary
+      def execute_with_clean_query_cache(*args)
+        return execute_without_clean_query_cache(*args) unless self.memcache_query_cache_options && query_cache_enabled
+        sql = args[0].strip
+        if sql =~ /^(INSERT|UPDATE|ALTER|DROP|DELETE)/i
+          # can only modify one table at a time...so stop after matching the first table name
+          table_name = ActiveRecord::Base.extract_table_names(sql).first
+          ActiveRecord::Base.increase_version!(table_name)
+        end
+        execute_without_clean_query_cache(*args)
+      end
+
+      alias_method_chain :execute, :clean_query_cache
+      
+    end
+    
+    class PostgreSQLAdapter < AbstractAdapter
+
+      def execute_with_clean_query_cache(*args)
+        return execute_without_clean_query_cache(*args) unless self.memcache_query_cache_options && query_cache_enabled
+        sql = args[0].strip
+        if sql =~ /^(INSERT|UPDATE|ALTER|DROP|DELETE)/i
+          table_name = ActiveRecord::Base.extract_table_names(sql).first
+          ActiveRecord::Base.increase_version!(table_name)
+        end
+        execute_without_clean_query_cache(*args)
+      end
+      
+      alias_method_chain :execute, :clean_query_cache
+      
+    end
+    
+    module QueryCache
+    
+      # Enable the query cache within the block
+      def cache
+        old, @query_cache_enabled = @query_cache_enabled, true
+        @query_cache ||= {}
+        @cache_version ||= {}
+        yield
+      ensure
+        @query_cache_enabled = old
+        clear_query_cache
+      end
+    
+      # Clears the query cache.
+      #
+      # One reason you may wish to call this method explicitly is between queries
+      # that ask the database to randomize results. Otherwise the cache would see
+      # the same SQL query and repeatedly return the same result each time, silently
+      # undermining the randomness you were expecting.
+      def clear_query_cache
+        @query_cache.clear if @query_cache
+        @cache_version.clear if @cache_version
+      end
+    
+      private
+    
+      def cache_sql(sql)
+        # priority order:
+        #  - if in @query_cache (memory of local app server) we prefer this
+        #  - else if exists in Memcached we prefer that
+        #  - else perform query in database and save memory caches
+        result =
+          if (query_cache_enabled || self.memcache_query_cache_options) && @query_cache.has_key?(sql)
+            log_info(sql, "CACHE", 0.0)
+            @query_cache[sql]
+          elsif self.memcache_query_cache_options && cached_result = ::Rails.cache.read(query_key(sql), self.memcache_query_cache_options)
+            log_info(sql, "MEMCACHE", 0.0)
+            @query_cache[sql] = cached_result
+          else
+            query_result = yield
+            @query_cache[sql] = query_result if query_cache_enabled || self.memcache_query_cache_options            
+            ::Rails.cache.write(query_key(sql), query_result, self.memcache_query_cache_options) if self.memcache_query_cache_options
+            query_result
+          end
+    
+        if Array === result
+          result.collect { |row| row.dup }
+        else
+          result.duplicable? ? result.dup : result
+        end
+      rescue TypeError
+        result
+      end
+    
+      # Transforms a sql query into a valid key for Memcache
+      def query_key(sql)
+        table_names = ActiveRecord::Base.extract_table_names(sql)
+        # version_number is the sum of the global version number and all 
+        # the version numbers of each table
+        version_number = get_cache_version # global version 
+        table_names.each { |table_name| version_number += get_cache_version(table_name) }
+        "#{version_number}_#{Digest::MD5.hexdigest(sql)}"
+      end
+    
+      # Returns the cache version of a table_name. If table_name is empty its the global version
+      #
+      # We prefer to search for this key first in memory and then in Memcache
+      def get_cache_version(table_name = nil)
+        key_class_version = table_name ? ActiveRecord::Base.cache_version_key(table_name) : ActiveRecord::Base.global_cache_version_key
+        if @cache_version && @cache_version[key_class_version]
+          @cache_version[key_class_version]
+        elsif version = ::Rails.cache.read(key_class_version)
+          @cache_version[key_class_version] = version if @cache_version
+          version
+        else
+          @cache_version[key_class_version] = 0 if @cache_version
+          ::Rails.cache.write(key_class_version, 0)
+          0
+        end
+      end
+    
+    end
+    
+  end
+end

data/test/query_memcached_test.rb

@@ -0,0 +1,26 @@
+require 'test/unit'
+require File.expand_path(File.join(File.dirname(__FILE__), '/testing_app/config/environment.rb'))
+
+class QueryMemcachedTest < Test::Unit::TestCase
+  
+  def test_extract_table_names
+    p  = "select * from pets"
+    q = "select * from author_favorites where (author_favorites.author_id = 36) AND (author_favorites.user_id = 11) LIMIT 1"
+    r = "SELECT * from binary_fields where (binary_fields.place_id = 3)"
+    s = "select distinct(p.id) from pets p, binary_fields u, binary_fields pu, where pu.user_id = u.id and pu.place_id = p.id and p.id != 3"
+    t = "select count(*) as count_all from pets inner join binary_fields on pets_id = binary_fields.place_id where (place_id = 3) AND (binary_fields.user_id = 11)"
+    u = "select * from categories order by created_at DESC limit 10"
+    v = "SELECT * from pets where id IN (SELECT * from pets where place_id = pets.id)"    
+    w = "SELECT categories.* FROM categories INNER JOIN binary_fields ON binary_fields.id = categories.user_id WHERE ((binary_fields.contact_id = 1))"
+    
+    assert_equal ['pets'], ActiveRecord::Base.extract_table_names(p)
+    assert_equal ['author_favorites'], ActiveRecord::Base.extract_table_names(q)
+    assert_equal ['binary_fields'], ActiveRecord::Base.extract_table_names(r)
+    assert_equal ['pets', 'binary_fields'], ActiveRecord::Base.extract_table_names(s)
+    assert_equal ['pets', 'binary_fields'], ActiveRecord::Base.extract_table_names(t)
+    assert_equal ['categories'], ActiveRecord::Base.extract_table_names(u)
+    assert_equal ['pets'], ActiveRecord::Base.extract_table_names(v)
+    assert_equal ['categories', 'binary_fields'], ActiveRecord::Base.extract_table_names(w)
+  end
+  
+end

data/test/testing_app/README

@@ -0,0 +1 @@
+# Testing application from plugin Query Memcached

data/test/testing_app/Rakefile

@@ -0,0 +1,8 @@
+require(File.join(File.dirname(__FILE__), 'config', 'boot'))
+
+require 'rubygems'
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+
+require 'tasks/rails'