dpickett-thinking-sphinx 1.1.4

data/LICENCE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 Pat Allan
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,107 @@
+= Thinking Sphinx
+
+== Usage
+
+First, if you haven't done so already, check out the main usage[http://ts.freelancing-gods.com/usage.html] page. Once you've done that, the next place to look for information is the specific method docs - ThinkingSphinx::Search and ThinkingSphinx::Index::Builder in particular.
+
+Keep in mind that while Thinking Sphinx works for ActiveRecord with Merb, it doesn't yet support DataMapper (although that is planned).
+
+== Contributing
+
+Fork on GitHub and after you've committed tested patches, send a pull request.
+
+To get the spec suite running, you will need to install the not-a-mock gem if you don't already have it:
+
+  git clone git://github.com/freelancing-god/not-a-mock.git
+  cd not-a-mock
+  rake gem
+  gem install pkg/not_a_mock-1.1.0.gem
+  
+Then install the ginger gem. The steps are the same, except that you might need to sudo the gem install:
+
+  git clone git://github.com/freelancing-god/ginger.git
+  cd ginger
+  rake gem
+  sudo gem install pkg/ginger-1.1.0.gem
+
+Then set up your database:
+
+  cp spec/fixtures/database.yml.default spec/fixtures/database.yml
+  mysqladmin -u root create thinking_sphinx
+  
+Make sure you don't have another Sphinx daemon (searchd) running. If you do, quit it with "rake ts:stop"
+in the app root.
+
+You should now have a passing test suite from which to build your patch on.
+
+  rake spec
+  
+If you get the message "Failed to start searchd daemon", run the spec with sudo:
+
+  sudo rake spec
+  
+If you quit the spec suite before it's completed, you may be left with data in the test
+database, causing the next run to have failures. Let that run complete and then try again.
+
+== Contributors
+
+Since I first released this library, there's been quite a few people who have submitted patches, to my immense gratitude. Others have suggested syntax changes and general improvements. So my thanks to the following people:
+
+- Joost Hietbrink
+- Jonathan Conway
+- Gregory Mirzayantz
+- Tung Nguyen
+- Sean Cribbs
+- Benoit Caccinolo
+- John Barton
+- Oliver Beddows
+- Arthur Zapparoli
+- Dusty Doris
+- Marcus Crafter
+- Patrick Lenz
+- Björn Andreasson
+- James Healy
+- Jae-Jun Hwang
+- Xavier Shay
+- Jason Rust
+- Gopal Patel
+- Chris Heald
+- Peter Vandenberk
+- Josh French
+- Andrew Bennett
+- Jordan Fowler
+- Seth Walker
+- Joe Noon
+- Wolfgang Postler
+- Rick Olson
+- Killian Murphy
+- Morten Primdahl
+- Ryan Bates
+- David Eisinger
+- Shay Arnett
+- Minh Tran
+- Jeremy Durham
+- Piotr Sarnacki
+- Matt Johnson
+- Nicolas Blanco
+- Max Lapshin
+- Josh Natanson
+- Philip Hallstrom
+- Christian Rishøj
+- Mike Flester
+- Jim Remsik
+- Kennon Ballou
+- Henrik Nyh
+- Emil Tin
+- Doug Cole
+- Ed Hickey
+- Evan Weaver
+- Thibaut Barrere
+- Kristopher Chambers
+- Dmitrij Smalko
+- Aleksey Yeschenko
+- Lachie Cox
+- Lourens Naude
+- Tom Davies
+- Dan Pickett
+- Alex Caudill

data/lib/thinking_sphinx/active_record/delta.rb

@@ -0,0 +1,74 @@
+module ThinkingSphinx
+  module ActiveRecord
+    # This module contains all the delta-related code for models. There isn't
+    # really anything you need to call manually in here - except perhaps
+    # index_delta, but not sure what reason why.
+    # 
+    module Delta
+      # Code for after_commit callback is written by Eli Miller:
+      # http://elimiller.blogspot.com/2007/06/proper-cache-expiry-with-aftercommit.html
+      # with slight modification from Joost Hietbrink.
+      #
+      def self.included(base)
+        base.class_eval do
+          class << self
+            # Temporarily disable delta indexing inside a block, then perform a single
+            # rebuild of index at the end.
+            #
+            # Useful when performing updates to batches of models to prevent
+            # the delta index being rebuilt after each individual update.
+            #
+            # In the following example, the delta index will only be rebuilt once,
+            # not 10 times.
+            #
+            #   SomeModel.suspended_delta do
+            #     10.times do
+            #       SomeModel.create( ... )
+            #     end
+            #   end
+            #
+            def suspended_delta(reindex_after = true, &block)
+              original_setting = ThinkingSphinx.deltas_enabled?
+              ThinkingSphinx.deltas_enabled = false
+              begin
+                yield
+              ensure
+                ThinkingSphinx.deltas_enabled = original_setting
+                self.index_delta if reindex_after
+              end
+            end
+
+            # Build the delta index for the related model. This won't be called
+            # if running in the test environment.
+            #
+            def index_delta(instance = nil)
+              delta_object.index(self, instance)
+            end
+            
+            def delta_object
+              self.sphinx_indexes.first.delta_object
+            end
+          end
+          
+          private
+          
+          # Set the delta value for the model to be true.
+          def toggle_delta
+            self.class.delta_object.toggle(self) if should_toggle_delta?
+          end
+          
+          # Build the delta index for the related model. This won't be called
+          # if running in the test environment.
+          # 
+          def index_delta
+            self.class.index_delta(self) if self.class.delta_object.toggled(self)
+          end
+          
+          def should_toggle_delta?
+            !self.respond_to?(:changed?) || self.changed? || self.new_record?
+          end
+        end
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/has_many_association.rb

@@ -0,0 +1,29 @@
+module ThinkingSphinx
+  module ActiveRecord
+    module HasManyAssociation
+      def search(*args)
+        foreign_key = @reflection.primary_key_name
+        stack = [@reflection.options[:through]].compact
+        
+        attribute   = nil
+        (@reflection.klass.sphinx_indexes || []).each do |index|
+          attribute = index.attributes.detect { |attrib|
+            attrib.columns.length == 1 &&
+            attrib.columns.first.__name  == foreign_key.to_sym &&
+            attrib.columns.first.__stack == stack
+          }
+          break if attribute
+        end
+        
+        raise "Missing Attribute for Foreign Key #{foreign_key}" unless attribute
+        
+        options = args.extract_options!
+        options[:with] ||= {}
+        options[:with][attribute.unique_name] = @owner.id
+        
+        args << options
+        @reflection.klass.search(*args)
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/search.rb

@@ -0,0 +1,57 @@
+module ThinkingSphinx
+  module ActiveRecord
+    # This module covers the specific model searches - but the syntax is
+    # exactly the same as the core Search class - so use that as your refence
+    # point.
+    # 
+    module Search
+      def self.included(base)
+        base.class_eval do
+          class << self
+            # Searches for results that match the parameters provided. Will only
+            # return the ids for the matching objects. See
+            # ThinkingSphinx::Search#search for syntax examples.
+            #
+            def search_for_ids(*args)
+              options = args.extract_options!
+              options[:class] = self
+              args << options
+              ThinkingSphinx::Search.search_for_ids(*args)
+            end
+
+            # Searches for results limited to a single model. See
+            # ThinkingSphinx::Search#search for syntax examples.
+            #
+            def search(*args)
+              options = args.extract_options!
+              options[:class] = self
+              args << options
+              ThinkingSphinx::Search.search(*args)
+            end
+
+            def search_count(*args)
+              options = args.extract_options!
+              options[:class] = self
+              args << options
+              ThinkingSphinx::Search.count(*args)
+            end
+
+            def search_for_id(*args)
+              options = args.extract_options!
+              options[:class] = self
+              args << options
+              ThinkingSphinx::Search.search_for_id(*args)
+            end
+            
+            def facets(*args)
+              options = args.extract_options!
+              options[:class] = self
+              args << options
+              ThinkingSphinx::Search.facets(*args)
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record.rb

@@ -0,0 +1,245 @@
+require 'thinking_sphinx/active_record/delta'
+require 'thinking_sphinx/active_record/search'
+require 'thinking_sphinx/active_record/has_many_association'
+
+module ThinkingSphinx
+  # Core additions to ActiveRecord models - define_index for creating indexes
+  # for models. If you want to interrogate the index objects created for the
+  # model, you can use the class-level accessor :sphinx_indexes.
+  #
+  module ActiveRecord
+    def self.included(base)
+      base.class_eval do
+        class_inheritable_array :sphinx_indexes, :sphinx_facets
+        class << self
+          # Allows creation of indexes for Sphinx. If you don't do this, there
+          # isn't much point trying to search (or using this plugin at all,
+          # really).
+          #
+          # An example or two:
+          #
+          #   define_index
+          #     indexes :id, :as => :model_id
+          #     indexes name
+          #   end
+          #
+          # You can also grab fields from associations - multiple levels deep
+          # if necessary.
+          #
+          #   define_index do
+          #     indexes tags.name, :as => :tag
+          #     indexes articles.content
+          #     indexes orders.line_items.product.name, :as => :product
+          #   end
+          #
+          # And it will automatically concatenate multiple fields:
+          #
+          #   define_index do
+          #     indexes [author.first_name, author.last_name], :as => :author
+          #   end
+          #
+          # The #indexes method is for fields - if you want attributes, use
+          # #has instead. All the same rules apply - but keep in mind that
+          # attributes are for sorting, grouping and filtering, not searching.
+          #
+          #   define_index do
+          #     # fields ...
+          #     
+          #     has created_at, updated_at
+          #   end
+          #
+          # One last feature is the delta index. This requires the model to
+          # have a boolean field named 'delta', and is enabled as follows:
+          #
+          #   define_index do
+          #     # fields ...
+          #     # attributes ...
+          #     
+          #     set_property :delta => true
+          #   end
+          #
+          # Check out the more detailed documentation for each of these methods
+          # at ThinkingSphinx::Index::Builder.
+          # 
+          def define_index(&block)
+            return unless ThinkingSphinx.define_indexes?
+            
+            self.sphinx_indexes ||= []
+            index = Index.new(self, &block)
+            
+            self.sphinx_indexes << index
+            unless ThinkingSphinx.indexed_models.include?(self.name)
+              ThinkingSphinx.indexed_models << self.name
+            end
+            
+            if index.delta?
+              before_save   :toggle_delta
+              after_commit  :index_delta
+            end
+            
+            after_destroy :toggle_deleted
+            
+            index
+          end
+          alias_method :sphinx_index, :define_index
+          
+          def sphinx_index_options
+            sphinx_indexes.last.options
+          end
+          
+          # Generate a unique CRC value for the model's name, to use to
+          # determine which Sphinx documents belong to which AR records.
+          # 
+          # Really only written for internal use - but hey, if it's useful to
+          # you in some other way, awesome.
+          # 
+          def to_crc32
+            self.name.to_crc32
+          end
+          
+          def to_crc32s
+            (subclasses << self).collect { |klass| klass.to_crc32 }
+          end
+          
+          def source_of_sphinx_index
+            possible_models = self.sphinx_indexes.collect { |index| index.model }
+            return self if possible_models.include?(self)
+
+            parent = self.superclass
+            while !possible_models.include?(parent) && parent != ::ActiveRecord::Base
+              parent = parent.superclass
+            end
+
+            return parent
+          end
+          
+          def to_riddle(offset)
+            sphinx_database_adapter.setup
+            
+            indexes = [to_riddle_for_core(offset)]
+            indexes << to_riddle_for_delta(offset) if sphinx_delta?
+            indexes << to_riddle_for_distributed
+          end
+          
+          def sphinx_database_adapter
+            @sphinx_database_adapter ||=
+              ThinkingSphinx::AbstractAdapter.detect(self)
+          end
+          
+          private
+          
+          def sphinx_name
+            self.name.underscore.tr(':/\\', '_')
+          end
+          
+          def sphinx_delta?
+            self.sphinx_indexes.any? { |index| index.delta? }
+          end
+          
+          def to_riddle_for_core(offset)
+            index = Riddle::Configuration::Index.new("#{sphinx_name}_core")
+            index.path = File.join(
+              ThinkingSphinx::Configuration.instance.searchd_file_path, index.name
+            )
+            
+            set_configuration_options_for_indexes index
+            set_field_settings_for_indexes        index
+            
+            self.sphinx_indexes.select { |ts_index|
+              ts_index.model == self
+            }.each_with_index do |ts_index, i|
+              index.sources << ts_index.to_riddle_for_core(offset, i)
+            end
+            
+            index
+          end
+          
+          def to_riddle_for_delta(offset)
+            index = Riddle::Configuration::Index.new("#{sphinx_name}_delta")
+            index.parent = "#{sphinx_name}_core"
+            index.path = File.join(ThinkingSphinx::Configuration.instance.searchd_file_path, index.name)
+            
+            self.sphinx_indexes.each_with_index do |ts_index, i|
+              index.sources << ts_index.to_riddle_for_delta(offset, i) if ts_index.delta?
+            end
+            
+            index
+          end
+          
+          def to_riddle_for_distributed
+            index = Riddle::Configuration::DistributedIndex.new(sphinx_name)
+            index.local_indexes << "#{sphinx_name}_core"
+            index.local_indexes.unshift "#{sphinx_name}_delta" if sphinx_delta?
+            index
+          end
+          
+          def set_configuration_options_for_indexes(index)
+            ThinkingSphinx::Configuration.instance.index_options.each do |key, value|
+              index.send("#{key}=".to_sym, value)
+            end
+            
+            self.sphinx_indexes.each do |ts_index|
+              ts_index.options.each do |key, value|
+                index.send("#{key}=".to_sym, value) if ThinkingSphinx::Configuration::IndexOptions.include?(key.to_s) && !value.nil?
+              end
+            end
+          end
+          
+          def set_field_settings_for_indexes(index)
+            field_names = lambda { |field| field.unique_name.to_s }
+            
+            self.sphinx_indexes.each do |ts_index|
+              index.prefix_field_names += ts_index.prefix_fields.collect(&field_names)
+              index.infix_field_names  += ts_index.infix_fields.collect(&field_names)
+            end
+          end
+        end
+      end
+      
+      base.send(:include, ThinkingSphinx::ActiveRecord::Delta)
+      base.send(:include, ThinkingSphinx::ActiveRecord::Search)
+      
+      ::ActiveRecord::Associations::HasManyAssociation.send(
+        :include, ThinkingSphinx::ActiveRecord::HasManyAssociation
+      )
+      ::ActiveRecord::Associations::HasManyThroughAssociation.send(
+        :include, ThinkingSphinx::ActiveRecord::HasManyAssociation
+      )
+    end
+    
+    def in_core_index?
+      self.class.search_for_id(
+        self.sphinx_document_id,
+        "#{self.class.source_of_sphinx_index.name.underscore.tr(':/\\', '_')}_core"
+      )
+    end
+    
+    def toggle_deleted
+      return unless ThinkingSphinx.updates_enabled? && ThinkingSphinx.sphinx_running?
+      
+      config = ThinkingSphinx::Configuration.instance
+      client = Riddle::Client.new config.address, config.port
+      
+      client.update(
+        "#{self.class.sphinx_indexes.first.name}_core",
+        ['sphinx_deleted'],
+        {self.sphinx_document_id => 1}
+      ) if self.in_core_index?
+      
+      client.update(
+        "#{self.class.sphinx_indexes.first.name}_delta",
+        ['sphinx_deleted'],
+        {self.sphinx_document_id => 1}
+      ) if ThinkingSphinx.deltas_enabled? &&
+        self.class.sphinx_indexes.any? { |index| index.delta? } &&
+        self.delta
+    rescue ::ThinkingSphinx::ConnectionError
+      # nothing
+    end
+    
+    def sphinx_document_id
+      (self.id * ThinkingSphinx.indexed_models.size) +
+        ThinkingSphinx.indexed_models.index(self.class.source_of_sphinx_index.name)
+    end
+  end
+end

data/lib/thinking_sphinx/adapters/abstract_adapter.rb

@@ -0,0 +1,34 @@
+module ThinkingSphinx
+  class AbstractAdapter
+    def initialize(model)
+      @model = model
+    end
+    
+    def setup
+      # Deliberately blank - subclasses should do something though. Well, if
+      # they need to.
+    end
+      
+    def self.detect(model)
+      case model.connection.class.name
+      when "ActiveRecord::ConnectionAdapters::MysqlAdapter",
+           "ActiveRecord::ConnectionAdapters::MysqlplusAdapter"
+        ThinkingSphinx::MysqlAdapter.new model
+      when "ActiveRecord::ConnectionAdapters::PostgreSQLAdapter"
+        ThinkingSphinx::PostgreSQLAdapter.new model
+      else
+        raise "Invalid Database Adapter: Sphinx only supports MySQL and PostgreSQL"
+      end
+    end
+    
+    def quote_with_table(column)
+      "#{@model.quoted_table_name}.#{@model.connection.quote_column_name(column)}"
+    end
+    
+    protected
+    
+    def connection
+      @connection ||= @model.connection
+    end
+  end
+end

data/lib/thinking_sphinx/adapters/mysql_adapter.rb

@@ -0,0 +1,53 @@
+module ThinkingSphinx
+  class MysqlAdapter < AbstractAdapter
+    def setup
+      # Does MySQL actually need to do anything?
+    end
+    
+    def sphinx_identifier
+      "mysql"
+    end
+    
+    def concatenate(clause, separator = ' ')
+      "CONCAT_WS('#{separator}', #{clause})"
+    end
+    
+    def group_concatenate(clause, separator = ' ')
+      "GROUP_CONCAT(#{clause} SEPARATOR '#{separator}')"
+    end
+    
+    def cast_to_string(clause)
+      "CAST(#{clause} AS CHAR)"
+    end
+    
+    def cast_to_datetime(clause)
+      "UNIX_TIMESTAMP(#{clause})"
+    end
+    
+    def cast_to_unsigned(clause)
+      "CAST(#{clause} AS UNSIGNED)"
+    end
+    
+    def convert_nulls(clause, default = '')
+      default = "'#{default}'" if default.is_a?(String)
+      
+      "IFNULL(#{clause}, #{default})"
+    end
+    
+    def boolean(value)
+      value ? 1 : 0
+    end
+    
+    def crc(clause)
+      "CRC32(#{clause})"
+    end
+    
+    def utf8_query_pre
+      "SET NAMES utf8"
+    end
+    
+    def time_difference(diff)
+      "DATE_SUB(NOW(), INTERVAL #{diff} SECOND)"
+    end
+  end
+end