acts_as_indexed 0.6.2

data/.gitignore

@@ -0,0 +1,5 @@
+rdoc
+test/test.log
+coverage
+index
+pkg

data/CHANGELOG

@@ -0,0 +1,90 @@
+===0.6.2 [11th June 2010]
+ - Now available as a Gem as well as the original plugin. [parndt - Thanks for doing most of the hard work.]
+
+===0.6.0 [10th June 2010]
+ - Now supports Rails 3.x.x as well as Rails 2.x.x.
+ - Added global configuration options.
+ - Now recommending using with_query scope for searching.
+ - Deprecated find_with_index and will_paginate_search methods.
+
+===0.5.3 [6th June 2010]
+- Now supports non-standard table names automatically. [nandalopes]
+
+===0.5.2 [3rd May 2010]
+- Fix for Errno::ERANGE error related to certain Math.log calculations. [parndt]
+- Improved index detection in a shared-directory environment. [bob-p]
+
+===0.5.1 [11 June 2009]
+- Fixed Ruby 1.8.6 compatibility.
+
+===0.5.0 [24 April 2009]
+- Ruby 1.9 and Rails 2.3 compatibility.
+- Index location can now be set. Provides Heroku compatibility.
+- Better errors on bad options.
+- ActiveRecord order argument overrides ranking returned by find_by_index.
+- Various test environment improvements
+- Various Bugfixes
+
+===0.4.6 [10 August 2008]
+- Rolled in pagination.
+
+===0.4.5 [04 February 2008]
+- Fixed a bug where the find_options :limit would be added to the :offset, which caused incorrectly sized collections to be returned.
+- Fixed an 'ambiguous column' error when using the :includes find_options key.
+
+===0.4.4 [29 November 2007]
+- Fixed a bug causing the weighting section of the code to error out.
+
+===0.4.3 [27 September 2007]
+- Fixed a bug causing records to be deleted from index during record updates.
+
+===0.4.2 [27 September 2007]
+- Fixed a bug causing identically ranked records to be lost.
+
+===0.4.1 [22 September 2007]
+- Fixed a bug in the main search method.
+
+===0.4.0 [22 September 2007]
+- Search results now ranked by relevance.
+
+===0.3.3 [20 September 2007]
+- Fixed index update bug where deleted atoms were not removed from index.
+- Improved performance of quoted queries.
+- Improved performance of index updates.
+- When building a full index, records are retrieved and indexed in batches to reduce memory consumption.
+
+===0.3.2 [19 September 2007]
+- Fixed index update bug.
+
+===0.3.1 [18 September 2007]
+- Added RDoc documentation comments.
+
+===0.3.0 [18 September 2007]
+- Minor bug fixes.
+- min_word_size now works properly, with quieries containing small words in 
+  quotes or being preceded by a '+' symbol are now searched on.
+
+===0.2.2 [06 September 2007]
+- Search now caches query results within a session. Call the search twice in an 
+  action? Only runs once!
+
+===0.2.1 [05 September 2007]
+- AR find options can now be passed to the search to allow finer control of 
+  returned Model Objects.
+
+===0.2.0 [04 September 2007]
+- Major performance improvements.
+- Index segmentation can now be tuned.
+
+===0.1.1 [31 August 2007]
+- Added a full set of tests.
+- Fixed various set-manipulation based errors.
+- Fixed a bug when searching for quoted phrases.
+
+===0.1.01 [31 August 2007]
+
+- Fixed a casting bug occurring when adding non-string fields to the index.
+
+===0.1 [31 August 2007]
+
+- Initial release.

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2007 - 2010 Douglas Shearer
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,137 @@
+= acts_as_indexed
+
+If you find this plugin useful, please consider a donation to show your 
+support!
+
+http://www.paypal.com/cgi-bin/webscr?cmd=_send-money
+  
+Paypal address: mailto:dougal.s@gmail.com
+  
+
+== Instructions
+
+This plugin allows boolean-queried fulltext search to be added to any Rails 
+app with no dependencies and minimal setup.
+
+
+== Resources
+
+=== Install
+
+== Rails 2.x.x
+  ./script/plugin install git://github.com/dougal/acts_as_indexed.git
+
+== Rails 3.x.x
+  rails plugin install git://github.com/dougal/acts_as_indexed.git
+
+=== As a Gem
+Despite this being slightly against the the original ethos of the project, 
+acts_as_indexed is now available as a Gem as several people have requested it.
+  
+  gem install acts_as_indexed
+  
+Make sure to specify the Gem in your environment.rb file (Rails 2.x.x), or the Gemfile (Rails 3.x.x).
+
+If you don't have git installed, you can download the plugin from the GitHub 
+page (http://github.com/dougal/acts_as_indexed) and unpack it into the 
+<tt>vendor/plugins</tt> directory of your rails app. 
+
+== Usage
+
+
+=== Setup
+
+Add +acts_as_indexed+ to the top of any models you want to index, along with a
+list of the fields you wish to be indexed.
+
+  class Post < ActiveRecord::Base
+    acts_as_indexed :fields => [:title, :body]
+  
+    ...
+  end
+
+The fields are not limited to model fields, but can be any instance method of
+the current model.
+
+  class User < ActiveRecord::Base
+    acts_as_indexed :fields => [:address, :fullname]
+
+    def fullname
+      self.firstname + ' ' + self.lastname
+    end
+
+    ...
+  end
+  
+Any of the configuration options in the Further Configuration section can be added as to the acts_as_indexed method call. These will override any defaults or global configuration.
+
+
+=== Searching
+
+To search, call the +with_query+ named scope on your model, passing a query as 
+an argument.
+
+  # Returns array of Post objects.
+  my_search_results = Post.with_query('my search query')
+  
+  # Chain it with any number of ActiveRecord methods and named_scopes.
+  my_search_results = Post.public.with_query('my search query').find(:all, :limit => 10) # return the first 10 matches which are public.
+  
+  
+=== Query Options
+
+The following query operators are supported:
+
+* AND:: This is the default option. 'cat dog' will find records matching 'cat' AND 'dog'.
+* NOT:: 'cat -dog' will find records matching 'cat' AND NOT 'dog'
+* INCLUDE:: 'cat +me' will find records matching 'cat' and 'me', even if 'me' is smaller than the +min_word_size+
+* "":: Quoted terms are matched as phrases. '"cat dog"' will find records matching the whole phrase. Quoted terms can be preceded by the NOT operator; 'cat -"big dog"' etc. Quoted terms can include words shorter than the +min_word_size+.
+
+=== Pagination
+
+Since +with_query+ is a named scope, WillPaginate can be used in the normal 
+fashion.
+
+  @images = Image.with_query('girl').paginate(:page => 1, :per_page => 5)
+
+=== Further Configuration
+
+A config block can be provided in your environment files or initializers.
+Example showing defaults:
+
+  ActsAsIndexed.configure do |config|
+    config.index_file = [RAILS_ROOT,'index']
+    config.index_file_depth = 3
+    config.min_word_size = 3
+  end
+
+A full rundown of the available configuration options can be found in
+<tt>lib/acts_as_indexed/configuration.rb</tt>
+
+== RDoc Documentation
+
+To generate the RDoc documentation, run the <tt>rake rdoc</tt> task in the
+acts_as_indexed plugin folder. Then point your web browser at
+<tt>vendor/plugins/acts_as_indexed/rdoc/index.html</tt>.
+
+Alternatively, you can view the rdoc documentation 
+online[http://rdoc.info/projects/dougal/acts_as_indexed/].
+
+== Problems, Comments, Suggestions?
+
+All of the above are most welcome. mailto:dougal.s@gmail.com
+
+
+== Credits
+
+Douglas F Shearer - http:douglasfshearer.com
+
+
+== Future Releases
+
+Future releases will be looking to add the following features:
+* Optional html scrubbing during indexing.
+* Ranking affected by field weightings.
+* Support for DataMapper, Sequel and the various MongoDB ORMs.
+* UTF-8 support. See the current solution here: 
+https://gist.github.com/193903bb4e0d6e5debe1

data/Rakefile

@@ -0,0 +1,50 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc 'Test the acts_as_indexed plugin.'
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = true
+end
+
+desc 'Generate documentation for the acts_as_indexed plugin.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'ActsAsIndexed'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README.rdoc')
+  rdoc.rdoc_files.include('CHANGELOG')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+namespace :rcov do
+  desc "Generate a coverage report in coverage/"
+  task :gen do
+    sh "rcov --output coverage test/*_test.rb"
+  end
+
+  desc "Remove generated coverage files."
+  task :clobber do
+    sh "rm -rdf coverage"
+  end
+end
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gemspec|
+    gemspec.name = "acts_as_indexed"
+    gemspec.summary = "Acts As Indexed is a plugin which provides a pain-free way to add fulltext search to your Ruby on Rails app"
+    gemspec.description = gemspec.summary
+    gemspec.email = "dougal.s@gmail.com"
+    gemspec.homepage = "http://github.com/dougal/acts_as_indexed"
+    gemspec.authors = ["Douglas F Shearer"]  
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end

data/VERSION

@@ -0,0 +1 @@
+0.6.2

data/acts_as_indexed.gemspec

@@ -0,0 +1,67 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{acts_as_indexed}
+  s.version = "0.6.2"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Douglas F Shearer"]
+  s.date = %q{2010-06-11}
+  s.description = %q{Acts As Indexed is a plugin which provides a pain-free way to add fulltext search to your Ruby on Rails app}
+  s.email = %q{dougal.s@gmail.com}
+  s.extra_rdoc_files = [
+    "README.rdoc"
+  ]
+  s.files = [
+    ".gitignore",
+     "CHANGELOG",
+     "MIT-LICENSE",
+     "README.rdoc",
+     "Rakefile",
+     "VERSION",
+     "acts_as_indexed.gemspec",
+     "lib/acts_as_indexed.rb",
+     "lib/acts_as_indexed/configuration.rb",
+     "lib/acts_as_indexed/search_atom.rb",
+     "lib/acts_as_indexed/search_index.rb",
+     "lib/will_paginate_search.rb",
+     "rails/init.rb",
+     "test/abstract_unit.rb",
+     "test/acts_as_indexed_test.rb",
+     "test/configuration_test.rb",
+     "test/database.yml",
+     "test/fixtures/post.rb",
+     "test/fixtures/posts.yml",
+     "test/schema.rb",
+     "test/search_atom_test.rb",
+     "test/search_index_test.rb"
+  ]
+  s.homepage = %q{http://github.com/dougal/acts_as_indexed}
+  s.rdoc_options = ["--charset=UTF-8"]
+  s.require_paths = ["lib"]
+  s.rubygems_version = %q{1.3.7}
+  s.summary = %q{Acts As Indexed is a plugin which provides a pain-free way to add fulltext search to your Ruby on Rails app}
+  s.test_files = [
+    "test/abstract_unit.rb",
+     "test/acts_as_indexed_test.rb",
+     "test/configuration_test.rb",
+     "test/fixtures/post.rb",
+     "test/schema.rb",
+     "test/search_atom_test.rb",
+     "test/search_index_test.rb"
+  ]
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+    else
+    end
+  else
+  end
+end
+

data/lib/acts_as_indexed.rb

@@ -0,0 +1,248 @@
+# ActsAsIndexed
+# Copyright (c) 2007 - 2010 Douglas F Shearer.
+# http://douglasfshearer.com
+# Distributed under the MIT license as included with this plugin.
+
+require 'active_record'
+
+require 'acts_as_indexed/configuration'
+require 'acts_as_indexed/search_index'
+require 'acts_as_indexed/search_atom'
+
+module ActsAsIndexed #:nodoc:
+  
+  # Holds the default configuration for acts_as_indexed.
+  
+  @configuration = Configuration.new
+
+  # Returns the current configuration for acts_as_indexed.
+  
+  def self.configuration
+    @configuration
+  end
+
+  # Call this method to modify defaults in your initializers.
+  #
+  # Example showing defaults:
+  #   ActsAsIndexed.configure do |config|
+  #     config.index_file = [Rails.root,'index']
+  #     config.index_file_depth = 3
+  #     config.min_word_size = 3
+  #   end
+  
+  def self.configure
+    self.configuration ||= Configuration.new
+    yield(configuration)
+  end
+
+  def self.included(mod)
+    mod.extend(ClassMethods)
+  end
+
+  module ClassMethods
+
+    # Declares a class as searchable.
+    #
+    # ====options:
+    # fields:: Names of fields to include in the index. Symbols pointing to
+    #          instance methods of your model may also be given here.
+    # index_file_depth:: Tuning value for the index partitioning. Larger
+    #                    values result in quicker searches, but slower
+    #                    indexing. Default is 3.
+    # min_word_size:: Sets the minimum length for a word in a query. Words
+    #                 shorter than this value are ignored in searches
+    #                 unless preceded by the '+' operator. Default is 3.
+    # index_file:: Sets the location for the index. By default this is 
+    #              RAILS_ROOT/index. Specify as an array. Heroku, for
+    #              example would use RAILS_ROOT/tmp/index, which would be
+    #              set as [Rails.root,'tmp','index]
+
+    def acts_as_indexed(options = {})
+      class_eval do
+        extend ActsAsIndexed::SingletonMethods
+      end
+      include ActsAsIndexed::InstanceMethods
+
+      after_create  :add_to_index
+      before_update  :update_index
+      after_destroy :remove_from_index
+
+      # scope for Rails 3.x, named_scope for Rails 2.x.
+      if self.respond_to?(:where)
+        scope :with_query, lambda { |query| where("#{table_name}.id IN (?)", search_index(query, {}, {:ids_only => true})) }
+      else
+        named_scope :with_query, lambda { |query| { :conditions => ["#{table_name}.id IN (?)", search_index(query, {}, {:ids_only => true}) ] } }
+      end
+            
+      cattr_accessor :aai_config, :aai_fields
+
+      self.aai_fields = options.delete(:fields)
+      raise(ArgumentError, 'no fields specified') if self.aai_fields.nil? || self.aai_fields.empty?
+      
+      self.aai_config = ActsAsIndexed.configuration.dup
+      options.each do |k, v|
+        self.aai_config.send("#{k}=", v)
+      end
+      self.aai_config.index_file += [Rails.env, self.name]
+    end
+
+    # Adds the passed +record+ to the index. Index is built if it does not already exist. Clears the query cache.
+
+    def index_add(record)
+      build_index if !File.exists?(File.join(aai_config.index_file))
+      index = SearchIndex.new(aai_config.index_file, aai_config.index_file_depth, aai_fields, aai_config.min_word_size)
+      index.add_record(record)
+      index.save
+      @query_cache = {}
+      true
+    end
+
+    # Removes the passed +record+ from the index. Clears the query cache.
+
+    def index_remove(record)
+      index = SearchIndex.new(aai_config.index_file, aai_config.index_file_depth, aai_fields, aai_config.min_word_size)
+      # record won't be in index if it doesn't exist. Just return true.
+      return true if !index.exists?
+      index.remove_record(record)
+      index.save
+      @query_cache = {}
+      true
+    end
+    
+    # Updates the index.
+    # 1. Removes the previous version of the record from the index
+    # 2. Adds the new version to the index.
+    
+    def index_update(record)
+      build_index if !File.exists?(File.join(aai_config.index_file))
+      index = SearchIndex.new(aai_config.index_file, aai_config.index_file_depth, aai_fields, aai_config.min_word_size)
+      #index.remove_record(find(record.id))
+      #index.add_record(record)
+      index.update_record(record,find(record.id))
+      index.save
+      @query_cache = {}
+      true
+    end
+
+    # Finds instances matching the terms passed in +query+. Terms are ANDed by
+    # default. Returns an array of model instances or, if +ids_only+ is
+    # true, an array of integer IDs.
+    #
+    # Keeps a cache of matched IDs for the current session to speed up
+    # multiple identical searches.
+    #
+    # ====find_options
+    # Same as ActiveRecord#find options hash. An :order key will override
+    # the relevance ranking 
+    #
+    # ====options
+    # ids_only:: Method returns an array of integer IDs when set to true.
+    # no_query_cache:: Turns off the query cache when set to true. Useful for testing.
+
+    def search_index(query, find_options={}, options={})
+      # Clear the query cache off  if the key is set.
+      @query_cache = {}  if (options.has_key?('no_query_cache') || options[:no_query_cache])
+      if !@query_cache || !@query_cache[query]
+        logger.debug('Query not in cache, running search.')
+        build_index if !File.exists?(File.join(aai_config.index_file))
+        index = SearchIndex.new(aai_config.index_file, aai_config.index_file_depth, aai_fields, aai_config.min_word_size)
+        @query_cache = {} if !@query_cache
+        @query_cache[query] = index.search(query)
+      else
+        logger.debug('Query held in cache.')
+      end
+      return @query_cache[query].sort.reverse.map(&:first) if options[:ids_only] || @query_cache[query].empty?
+      
+      # slice up the results by offset and limit
+      offset = find_options[:offset] || 0
+      limit = find_options.include?(:limit) ? find_options[:limit] : @query_cache[query].size
+      part_query = @query_cache[query].sort.reverse.slice(offset,limit).map(&:first)
+      
+      # Set these to nil as we are dealing with the pagination by setting
+      # exactly what records we want.
+      find_options[:offset] = nil
+      find_options[:limit] = nil
+      
+      with_scope :find => find_options do
+        # Doing the find like this eliminates the possibility of errors occuring
+        # on either missing records (out-of-sync) or an empty results array.
+        records = find(:all, :conditions => [ "#{table_name}.id IN (?)", part_query])
+        
+        if find_options.include?(:order)
+         records # Just return the records without ranking them.
+       else
+         # Results come back in random order from SQL, so order again.
+         ranked_records = {}
+         records.each do |r|
+           ranked_records[r] = @query_cache[query][r.id]
+         end
+      
+         ranked_records.to_a.sort_by{|a| a.last }.reverse.map(&:first)
+       end
+      end
+      
+    end
+
+    private
+
+    # Builds an index from scratch for the current model class.
+    def build_index
+      increment = 500
+      offset = 0
+      while (records = find(:all, :limit => increment, :offset => offset)).size > 0
+        #p "offset is #{offset}, increment is #{increment}"
+        index = SearchIndex.new(aai_config.index_file, aai_config.index_file_depth, aai_fields, aai_config.min_word_size)
+        offset += increment
+        index.add_records(records)
+        index.save
+      end
+    end
+
+  end
+
+  # Adds model class singleton methods.
+  module SingletonMethods
+
+    # DEPRECATED. Use +with_query+ scope instead.
+    # Finds instances matching the terms passed in +query+.
+    #
+    # See ActsAsIndexed::ClassMethods#search_index.
+    def find_with_index(query='', find_options = {}, options = {})
+      warn "[DEPRECATION] `find_with_index` is deprecated and will be removed in a later release. Use `with_query(query)` instead."
+      search_index(query, find_options, options)
+    end
+
+  end
+
+  # Adds model class instance methods.
+  # Methods are called automatically by ActiveRecord on +save+, +destroy+,
+  # and +update+ of model instances.
+  module InstanceMethods
+
+    # Adds the current model instance to index.
+    # Called by ActiveRecord on +save+.
+    def add_to_index
+      self.class.index_add(self)
+    end
+
+    # Removes the current model instance to index.
+    # Called by ActiveRecord on +destroy+.
+    def remove_from_index
+      self.class.index_remove(self)
+    end
+
+    # Updates current model instance index.
+    # Called by ActiveRecord on +update+.
+    def update_index
+      self.class.index_update(self)
+    end
+  end
+
+end
+
+# reopen ActiveRecord and include all the above to make
+# them available to all our models if they want it
+
+ActiveRecord::Base.class_eval do
+  include ActsAsIndexed
+end