initforthe-thinking-sphinx 1.1.21

data/LICENCE

@@ -0,0 +1,20 @@
+Copyright (c) 2008 Pat Allan
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.textile

@@ -0,0 +1,141 @@
+h1. Thinking Sphinx
+
+h2. Usage
+
+First, if you haven't done so already, check out the main "usage":http://ts.freelancing-gods.com/usage.html page. Once you've done that, the next place to look for information is the specific method docs - ThinkingSphinx::Search and ThinkingSphinx::Index::Builder in particular.
+
+Keep in mind that while Thinking Sphinx works for ActiveRecord with Merb, it doesn't yet support DataMapper (although that is planned).
+
+h2. Contributing
+
+Fork on GitHub and after you've committed tested patches, send a pull request.
+
+To quickly see if your system is ready to run the thinking sphinx specs, run the contribute.rb script found in the project root directory. Use the following instructions to install any missing requirements.
+
+To get the spec suite running, you will need to install the not-a-mock gem if you don't already have it:
+
+  git clone git://github.com/freelancing-god/not-a-mock.git
+  cd not-a-mock
+  rake gem
+  gem install pkg/not_a_mock-1.1.0.gem
+  
+Then install the ginger gem. The steps are the same, except that you might need to sudo the gem install:
+
+  git clone git://github.com/freelancing-god/ginger.git
+  cd ginger
+  rake gem
+  sudo gem install pkg/ginger-1.1.0.gem
+  
+Alternatively, install the ginger gem directly from the freelancing-god github repository
+
+  sudo gem sources -a http://gems.github.com
+  sudo gem install freelancing-god-ginger
+
+Then set up your database:
+
+  cp spec/fixtures/database.yml.default spec/fixtures/database.yml
+  mysqladmin -u root create thinking_sphinx
+  
+This last step can be done automatically by the contribute.rb script if all dependencies are met.
+
+Make sure you don't have another Sphinx daemon (searchd) running. If you do, quit it with "rake ts:stop"
+in the app root.
+
+You should now have a passing test suite from which to build your patch on.
+
+  rake spec
+
+If you get the message "Failed to start searchd daemon", run the spec with sudo:
+
+  sudo rake spec
+
+If you quit the spec suite before it's completed, you may be left with data in the test
+database, causing the next run to have failures. Let that run complete and then try again.
+
+h2. Contributors
+
+Since I first released this library, there's been quite a few people who have submitted patches, to my immense gratitude. Others have suggested syntax changes and general improvements. So my thanks to the following people:
+
+* Joost Hietbrink
+* Jonathan Conway
+* Gregory Mirzayantz
+* Tung Nguyen
+* Sean Cribbs
+* Benoit Caccinolo
+* John Barton
+* Oliver Beddows
+* Arthur Zapparoli
+* Dusty Doris
+* Marcus Crafter
+* Patrick Lenz
+* Björn Andreasson
+* James Healy
+* Jae-Jun Hwang
+* Xavier Shay
+* Jason Rust
+* Gopal Patel
+* Chris Heald
+* Peter Vandenberk
+* Josh French
+* Andrew Bennett
+* Jordan Fowler
+* Seth Walker
+* Joe Noon
+* Wolfgang Postler
+* Rick Olson
+* Killian Murphy
+* Morten Primdahl
+* Ryan Bates
+* David Eisinger
+* Shay Arnett
+* Minh Tran
+* Jeremy Durham
+* Piotr Sarnacki
+* Matt Johnson
+* Nicolas Blanco
+* Max Lapshin
+* Josh Natanson
+* Philip Hallstrom
+* Christian Rishøj
+* Mike Flester
+* Jim Remsik
+* Kennon Ballou
+* Henrik Nyh
+* Emil Tin
+* Doug Cole
+* Ed Hickey
+* Evan Weaver
+* Thibaut Barrere
+* Kristopher Chambers
+* Dmitrij Smalko
+* Aleksey Yeschenko
+* Lachie Cox
+* Lourens Naude
+* Tom Davies
+* Dan Pickett
+* Alex Caudill
+* Jim Benton
+* John Aughey
+* Keith Pitty
+* Jeff Talbot
+* Dana Contreras
+* Menno van der Sman
+* Bill Harding
+* Isaac Feliu
+* Andrei Bocan
+* László Bácsi
+* Peter Wagenet
+* Max Lapshin
+* Martin Emde
+* David Wennergren
+* Mark Lane
+* Eric Lindvall
+* Lawrence Pit
+* Mike Bailey
+* Bill Leeper
+* Michael Reinsch
+* Anderson Dias
+* Jerome Riga
+* Tien Dung
+* Johannes Kaefer
+* Paul Campbell

data/lib/thinking_sphinx.rb

@@ -0,0 +1,215 @@
+Dir[File.join(File.dirname(__FILE__), '../vendor/*/lib')].each do |path|
+  $LOAD_PATH.unshift path
+end
+
+require 'active_record'
+require 'riddle'
+require 'after_commit'
+
+require 'thinking_sphinx/core/string'
+require 'thinking_sphinx/property'
+require 'thinking_sphinx/active_record'
+require 'thinking_sphinx/association'
+require 'thinking_sphinx/attribute'
+require 'thinking_sphinx/collection'
+require 'thinking_sphinx/configuration'
+require 'thinking_sphinx/facet'
+require 'thinking_sphinx/class_facet'
+require 'thinking_sphinx/facet_collection'
+require 'thinking_sphinx/field'
+require 'thinking_sphinx/index'
+require 'thinking_sphinx/source'
+require 'thinking_sphinx/rails_additions'
+require 'thinking_sphinx/search'
+require 'thinking_sphinx/deltas'
+
+require 'thinking_sphinx/adapters/abstract_adapter'
+require 'thinking_sphinx/adapters/mysql_adapter'
+require 'thinking_sphinx/adapters/postgresql_adapter'
+
+ActiveRecord::Base.send(:include, ThinkingSphinx::ActiveRecord)
+
+Merb::Plugins.add_rakefiles(
+  File.join(File.dirname(__FILE__), "thinking_sphinx", "tasks")
+) if defined?(Merb)
+
+module ThinkingSphinx
+  module Version #:nodoc:
+    Major = 1
+    Minor = 1
+    Tiny  = 21
+    
+    String = [Major, Minor, Tiny].join('.')
+  end
+  
+  # A ConnectionError will get thrown when a connection to Sphinx can't be
+  # made.
+  class ConnectionError < StandardError
+  end
+  
+  # A StaleIdsException is thrown by Collection.instances_from_matches if there
+  # are records in Sphinx but not in the database, so the search can be retried.
+  class StaleIdsException < StandardError
+    attr_accessor :ids
+    def initialize(ids)
+      self.ids = ids
+    end
+  end
+  
+  # The collection of indexed models. Keep in mind that Rails lazily loads
+  # its classes, so this may not actually be populated with _all_ the models
+  # that have Sphinx indexes.
+  def self.indexed_models
+    @@indexed_models ||= []
+  end
+  
+  def self.unique_id_expression(offset = nil)
+    "* #{ThinkingSphinx.indexed_models.size} + #{offset || 0}"
+  end
+  
+  # Check if index definition is disabled.
+  # 
+  def self.define_indexes?
+    @@define_indexes =  true unless defined?(@@define_indexes)
+    @@define_indexes == true
+  end
+  
+  # Enable/disable indexes - you may want to do this while migrating data.
+  # 
+  #   ThinkingSphinx.define_indexes = false
+  # 
+  def self.define_indexes=(value)
+    @@define_indexes = value
+  end
+  
+  @@deltas_enabled = nil
+
+  # Check if delta indexing is enabled.
+  # 
+  def self.deltas_enabled?
+    @@deltas_enabled  = (ThinkingSphinx::Configuration.environment != 'test') if @@deltas_enabled.nil?
+    @@deltas_enabled
+  end
+  
+  # Enable/disable all delta indexing.
+  #
+  #   ThinkingSphinx.deltas_enabled = false
+  #
+  def self.deltas_enabled=(value)
+    @@deltas_enabled = value
+  end
+  
+  @@updates_enabled = nil
+  
+  # Check if updates are enabled. True by default, unless within the test
+  # environment.
+  # 
+  def self.updates_enabled?
+    @@updates_enabled  = (ThinkingSphinx::Configuration.environment != 'test') if @@updates_enabled.nil?
+    @@updates_enabled
+  end
+  
+  # Enable/disable updates to Sphinx
+  # 
+  #   ThinkingSphinx.updates_enabled = false
+  #
+  def self.updates_enabled=(value)
+    @@updates_enabled = value
+  end
+  
+  @@suppress_delta_output = false
+  
+  def self.suppress_delta_output?
+    @@suppress_delta_output
+  end
+  
+  def self.suppress_delta_output=(value)
+    @@suppress_delta_output = value
+  end
+  
+  # Checks to see if MySQL will allow simplistic GROUP BY statements. If not,
+  # or if not using MySQL, this will return false.
+  # 
+  def self.use_group_by_shortcut?
+    !!(
+      mysql? && ::ActiveRecord::Base.connection.select_all(
+        "SELECT @@global.sql_mode, @@session.sql_mode;"
+      ).all? { |key,value| value.nil? || value[/ONLY_FULL_GROUP_BY/].nil? }
+    )
+  end
+  
+  @@remote_sphinx = false
+  
+  # An indication of whether Sphinx is running on a remote machine instead of
+  # the same machine.
+  # 
+  def self.remote_sphinx?
+    @@remote_sphinx
+  end
+  
+  # Tells Thinking Sphinx that Sphinx is running on a different machine, and
+  # thus it can't reliably guess whether it is running or not (ie: the 
+  # #sphinx_running? method), and so just assumes it is.
+  # 
+  # Useful for multi-machine deployments. Set it in your production.rb file.
+  # 
+  #   ThinkingSphinx.remote_sphinx = true
+  # 
+  def self.remote_sphinx=(value)
+    @@remote_sphinx = value
+  end
+  
+  # Check if Sphinx is running. If remote_sphinx is set to true (indicating
+  # Sphinx is on a different machine), this will always return true, and you
+  # will have to handle any connection errors yourself.
+  # 
+  def self.sphinx_running?
+    remote_sphinx? || sphinx_running_by_pid?
+  end
+  
+  # Check if Sphinx is actually running, provided the pid is on the same
+  # machine as this code.
+  # 
+  def self.sphinx_running_by_pid?
+    !!sphinx_pid && pid_active?(sphinx_pid)
+  end
+  
+  def self.sphinx_pid
+    pid_file    = ThinkingSphinx::Configuration.instance.pid_file
+    cat_command = 'cat'
+    return nil unless File.exists?(pid_file)
+    
+    if microsoft?
+      pid_file.gsub!('/', '\\')
+      cat_command = 'type'
+    end
+    
+    `#{cat_command} #{pid_file}`[/\d+/]
+  end
+  
+  def self.pid_active?(pid)
+    return true if microsoft?
+    
+    begin
+      # In JRuby this returns -1 if the process doesn't exist
+      Process.getpgid(pid.to_i) != -1
+    rescue Exception => e
+      false
+    end
+  end
+  
+  def self.microsoft?
+    RUBY_PLATFORM =~ /mswin/
+  end
+  
+  def self.jruby?
+    defined?(JRUBY_VERSION)
+  end
+  
+  def self.mysql?
+    ::ActiveRecord::Base.connection.class.name.demodulize == "MysqlAdapter" ||
+    ::ActiveRecord::Base.connection.class.name.demodulize == "MysqlplusAdapter" || (
+      jruby? && ::ActiveRecord::Base.connection.config[:adapter] == "jdbcmysql"
+    )
+  end
+end

data/lib/thinking_sphinx/active_record.rb

@@ -0,0 +1,278 @@
+require 'thinking_sphinx/active_record/attribute_updates'
+require 'thinking_sphinx/active_record/delta'
+require 'thinking_sphinx/active_record/search'
+require 'thinking_sphinx/active_record/has_many_association'
+
+module ThinkingSphinx
+  # Core additions to ActiveRecord models - define_index for creating indexes
+  # for models. If you want to interrogate the index objects created for the
+  # model, you can use the class-level accessor :sphinx_indexes.
+  #
+  module ActiveRecord
+    def self.included(base)
+      base.class_eval do
+        class_inheritable_array :sphinx_indexes, :sphinx_facets
+        class << self
+          # Allows creation of indexes for Sphinx. If you don't do this, there
+          # isn't much point trying to search (or using this plugin at all,
+          # really).
+          #
+          # An example or two:
+          #
+          #   define_index
+          #     indexes :id, :as => :model_id
+          #     indexes name
+          #   end
+          #
+          # You can also grab fields from associations - multiple levels deep
+          # if necessary.
+          #
+          #   define_index do
+          #     indexes tags.name, :as => :tag
+          #     indexes articles.content
+          #     indexes orders.line_items.product.name, :as => :product
+          #   end
+          #
+          # And it will automatically concatenate multiple fields:
+          #
+          #   define_index do
+          #     indexes [author.first_name, author.last_name], :as => :author
+          #   end
+          #
+          # The #indexes method is for fields - if you want attributes, use
+          # #has instead. All the same rules apply - but keep in mind that
+          # attributes are for sorting, grouping and filtering, not searching.
+          #
+          #   define_index do
+          #     # fields ...
+          #     
+          #     has created_at, updated_at
+          #   end
+          #
+          # One last feature is the delta index. This requires the model to
+          # have a boolean field named 'delta', and is enabled as follows:
+          #
+          #   define_index do
+          #     # fields ...
+          #     # attributes ...
+          #     
+          #     set_property :delta => true
+          #   end
+          #
+          # Check out the more detailed documentation for each of these methods
+          # at ThinkingSphinx::Index::Builder.
+          # 
+          def define_index(&block)
+            return unless ThinkingSphinx.define_indexes?
+            
+            self.sphinx_indexes ||= []
+            index = ThinkingSphinx::Index::Builder.generate(self, &block)
+            
+            self.sphinx_indexes << index
+            unless ThinkingSphinx.indexed_models.include?(self.name)
+              ThinkingSphinx.indexed_models << self.name
+            end
+            
+            if index.delta?
+              before_save   :toggle_delta
+              after_commit  :index_delta
+            end
+            
+            after_destroy :toggle_deleted
+            
+            include ThinkingSphinx::ActiveRecord::AttributeUpdates
+            
+            index
+            
+            # We want to make sure that if the database doesn't exist, then Thinking
+            # Sphinx doesn't mind when running non-TS tasks (like db:create, db:drop
+            # and db:migrate). It's a bit hacky, but I can't think of a better way.
+          rescue StandardError => err
+            case err.class.name
+            when "Mysql::Error", "Java::JavaSql::SQLException", "ActiveRecord::StatementInvalid"
+              return
+            else
+              raise err
+            end
+          end
+          alias_method :sphinx_index, :define_index
+          
+          def sphinx_index_options
+            sphinx_indexes.last.options
+          end
+          
+          # Generate a unique CRC value for the model's name, to use to
+          # determine which Sphinx documents belong to which AR records.
+          # 
+          # Really only written for internal use - but hey, if it's useful to
+          # you in some other way, awesome.
+          # 
+          def to_crc32
+            self.name.to_crc32
+          end
+          
+          def to_crc32s
+            (subclasses << self).collect { |klass| klass.to_crc32 }
+          end
+          
+          def source_of_sphinx_index
+            possible_models = self.sphinx_indexes.collect { |index| index.model }
+            return self if possible_models.include?(self)
+
+            parent = self.superclass
+            while !possible_models.include?(parent) && parent != ::ActiveRecord::Base
+              parent = parent.superclass
+            end
+
+            return parent
+          end
+          
+          def to_riddle(offset)
+            sphinx_database_adapter.setup
+            
+            indexes = [to_riddle_for_core(offset)]
+            indexes << to_riddle_for_delta(offset) if sphinx_delta?
+            indexes << to_riddle_for_distributed
+          end
+          
+          def sphinx_database_adapter
+            @sphinx_database_adapter ||=
+              ThinkingSphinx::AbstractAdapter.detect(self)
+          end
+          
+          private
+          
+          def sphinx_name
+            self.name.underscore.tr(':/\\', '_')
+          end
+          
+          def sphinx_delta?
+            self.sphinx_indexes.any? { |index| index.delta? }
+          end
+          
+          def to_riddle_for_core(offset)
+            index = Riddle::Configuration::Index.new("#{sphinx_name}_core")
+            index.path = File.join(
+              ThinkingSphinx::Configuration.instance.searchd_file_path, index.name
+            )
+            
+            set_configuration_options_for_indexes index
+            set_field_settings_for_indexes        index
+            
+            self.sphinx_indexes.select { |ts_index|
+              ts_index.model == self
+            }.each_with_index do |ts_index, i|
+              index.sources += ts_index.sources.collect { |source|
+                source.to_riddle_for_core(offset, i)
+              }
+            end
+            
+            index
+          end
+          
+          def to_riddle_for_delta(offset)
+            index = Riddle::Configuration::Index.new("#{sphinx_name}_delta")
+            index.parent = "#{sphinx_name}_core"
+            index.path = File.join(ThinkingSphinx::Configuration.instance.searchd_file_path, index.name)
+            
+            self.sphinx_indexes.each_with_index do |ts_index, i|
+              index.sources += ts_index.sources.collect { |source|
+                source.to_riddle_for_delta(offset, i)
+              } if ts_index.delta?
+            end
+            
+            index
+          end
+          
+          def to_riddle_for_distributed
+            index = Riddle::Configuration::DistributedIndex.new(sphinx_name)
+            index.local_indexes << "#{sphinx_name}_core"
+            index.local_indexes.unshift "#{sphinx_name}_delta" if sphinx_delta?
+            index
+          end
+          
+          def set_configuration_options_for_indexes(index)
+            ThinkingSphinx::Configuration.instance.index_options.each do |key, value|
+              index.send("#{key}=".to_sym, value)
+            end
+            
+            self.sphinx_indexes.each do |ts_index|
+              ts_index.options.each do |key, value|
+                index.send("#{key}=".to_sym, value) if ThinkingSphinx::Configuration::IndexOptions.include?(key.to_s) && !value.nil?
+              end
+            end
+          end
+          
+          def set_field_settings_for_indexes(index)
+            field_names = lambda { |field| field.unique_name.to_s }
+            
+            self.sphinx_indexes.each do |ts_index|
+              index.prefix_field_names += ts_index.prefix_fields.collect(&field_names)
+              index.infix_field_names  += ts_index.infix_fields.collect(&field_names)
+            end
+          end
+        end
+      end
+      
+      base.send(:include, ThinkingSphinx::ActiveRecord::Delta)
+      base.send(:include, ThinkingSphinx::ActiveRecord::Search)
+      
+      ::ActiveRecord::Associations::HasManyAssociation.send(
+        :include, ThinkingSphinx::ActiveRecord::HasManyAssociation
+      )
+      ::ActiveRecord::Associations::HasManyThroughAssociation.send(
+        :include, ThinkingSphinx::ActiveRecord::HasManyAssociation
+      )
+    end
+    
+    def in_index?(suffix)
+      self.class.search_for_id self.sphinx_document_id, sphinx_index_name(suffix)
+    end
+    
+    def in_core_index?
+      in_index? "core"
+    end
+    
+    def in_delta_index?
+      in_index? "delta"
+    end
+    
+    def in_both_indexes?
+      in_core_index? && in_delta_index?
+    end
+    
+    def toggle_deleted
+      return unless ThinkingSphinx.updates_enabled? && ThinkingSphinx.sphinx_running?
+      
+      config = ThinkingSphinx::Configuration.instance
+      client = Riddle::Client.new config.address, config.port
+      
+      client.update(
+        "#{self.class.sphinx_indexes.first.name}_core",
+        ['sphinx_deleted'],
+        {self.sphinx_document_id => 1}
+      ) if self.in_core_index?
+      
+      client.update(
+        "#{self.class.sphinx_indexes.first.name}_delta",
+        ['sphinx_deleted'],
+        {self.sphinx_document_id => 1}
+      ) if ThinkingSphinx.deltas_enabled? &&
+        self.class.sphinx_indexes.any? { |index| index.delta? } &&
+        self.toggled_delta?
+    rescue ::ThinkingSphinx::ConnectionError
+      # nothing
+    end
+    
+    def sphinx_document_id
+      (self.id * ThinkingSphinx.indexed_models.size) +
+        ThinkingSphinx.indexed_models.index(self.class.source_of_sphinx_index.name)
+    end
+    
+    private
+    
+    def sphinx_index_name(suffix)
+      "#{self.class.source_of_sphinx_index.name.underscore.tr(':/\\', '_')}_#{suffix}"
+    end
+  end
+end