DrMark-thinking-sphinx 1.1.6 → 1.1.14

data/{README → README.textile}

@@ -1,12 +1,12 @@
-= Thinking Sphinx
+h1. Thinking Sphinx
 
-== Usage
+h2. Usage
 
-First, if you haven't done so already, check out the main usage[http://ts.freelancing-gods.com/usage.html] page. Once you've done that, the next place to look for information is the specific method docs - ThinkingSphinx::Search and ThinkingSphinx::Index::Builder in particular.
+First, if you haven't done so already, check out the main "usage":http://ts.freelancing-gods.com/usage.html page. Once you've done that, the next place to look for information is the specific method docs - ThinkingSphinx::Search and ThinkingSphinx::Index::Builder in particular.
 
 Keep in mind that while Thinking Sphinx works for ActiveRecord with Merb, it doesn't yet support DataMapper (although that is planned).
 
-== Contributing
+h2. Contributing
 
 Fork on GitHub and after you've committed tested patches, send a pull request.
 
@@ -26,102 +26,102 @@ Then install the ginger gem. The steps are the same, except that you might need
   rake gem
   sudo gem install pkg/ginger-1.1.0.gem
 
-Then install the faker gem:
-
-  sudo gem install faker
-
 Then set up your database:
 
   cp spec/fixtures/database.yml.default spec/fixtures/database.yml
-  cp spec/fixtures/database.yml.default features/support/db/database.yml
   mysqladmin -u root create thinking_sphinx
-  
+
 Make sure you don't have another Sphinx daemon (searchd) running. If you do, quit it with "rake ts:stop"
 in the app root.
 
 You should now have a passing test suite from which to build your patch on.
 
   rake spec
-	rake features  (Note: You will need MySQL and Postgres gems to run the full suite. You may want to tweak your rakefile)
-  
+
 If you get the message "Failed to start searchd daemon", run the spec with sudo:
 
   sudo rake spec
-  sudo rake features  
-  
+
 If you quit the spec suite before it's completed, you may be left with data in the test
 database, causing the next run to have failures. Let that run complete and then try again.
 
-== Contributors
+h2. Contributors
 
 Since I first released this library, there's been quite a few people who have submitted patches, to my immense gratitude. Others have suggested syntax changes and general improvements. So my thanks to the following people:
 
-- Joost Hietbrink
-- Jonathan Conway
-- Gregory Mirzayantz
-- Tung Nguyen
-- Sean Cribbs
-- Benoit Caccinolo
-- John Barton
-- Oliver Beddows
-- Arthur Zapparoli
-- Dusty Doris
-- Marcus Crafter
-- Patrick Lenz
-- Björn Andreasson
-- James Healy
-- Jae-Jun Hwang
-- Xavier Shay
-- Jason Rust
-- Gopal Patel
-- Chris Heald
-- Peter Vandenberk
-- Josh French
-- Andrew Bennett
-- Jordan Fowler
-- Seth Walker
-- Joe Noon
-- Wolfgang Postler
-- Rick Olson
-- Killian Murphy
-- Morten Primdahl
-- Ryan Bates
-- David Eisinger
-- Shay Arnett
-- Minh Tran
-- Jeremy Durham
-- Piotr Sarnacki
-- Matt Johnson
-- Nicolas Blanco
-- Max Lapshin
-- Josh Natanson
-- Philip Hallstrom
-- Christian Rishøj
-- Mike Flester
-- Jim Remsik
-- Kennon Ballou
-- Henrik Nyh
-- Emil Tin
-- Doug Cole
-- Ed Hickey
-- Evan Weaver
-- Thibaut Barrere
-- Kristopher Chambers
-- Dmitrij Smalko
-- Aleksey Yeschenko
-- Lachie Cox
-- Lourens Naude
-- Tom Davies
-- Dan Pickett
-- Alex Caudill
-- Jim Benton
-- John Aughey
-- Keith Pitty
-- Jeff Talbot
-- Dana Contreras
-- Menno van der Sman
-- Bill Harding
-- Isaac Feliu
-- Andrei Bocan
-- László Bácsi
-- Peter Wagenet
+* Joost Hietbrink
+* Jonathan Conway
+* Gregory Mirzayantz
+* Tung Nguyen
+* Sean Cribbs
+* Benoit Caccinolo
+* John Barton
+* Oliver Beddows
+* Arthur Zapparoli
+* Dusty Doris
+* Marcus Crafter
+* Patrick Lenz
+* Björn Andreasson
+* James Healy
+* Jae-Jun Hwang
+* Xavier Shay
+* Jason Rust
+* Gopal Patel
+* Chris Heald
+* Peter Vandenberk
+* Josh French
+* Andrew Bennett
+* Jordan Fowler
+* Seth Walker
+* Joe Noon
+* Wolfgang Postler
+* Rick Olson
+* Killian Murphy
+* Morten Primdahl
+* Ryan Bates
+* David Eisinger
+* Shay Arnett
+* Minh Tran
+* Jeremy Durham
+* Piotr Sarnacki
+* Matt Johnson
+* Nicolas Blanco
+* Max Lapshin
+* Josh Natanson
+* Philip Hallstrom
+* Christian Rishøj
+* Mike Flester
+* Jim Remsik
+* Kennon Ballou
+* Henrik Nyh
+* Emil Tin
+* Doug Cole
+* Ed Hickey
+* Evan Weaver
+* Thibaut Barrere
+* Kristopher Chambers
+* Dmitrij Smalko
+* Aleksey Yeschenko
+* Lachie Cox
+* Lourens Naude
+* Tom Davies
+* Dan Pickett
+* Alex Caudill
+* Jim Benton
+* John Aughey
+* Keith Pitty
+* Jeff Talbot
+* Dana Contreras
+* Menno van der Sman
+* Bill Harding
+* Isaac Feliu
+* Andrei Bocan
+* László Bácsi
+* Peter Wagenet
+* Max Lapshin
+* Martin Emde
+* David Wennergren
+* Mark Lane
+* Eric Lindvall
+* Lawrence Pit
+* Mike Bailey

data/lib/thinking_sphinx/active_record/attribute_updates.rb

@@ -0,0 +1,48 @@
+module ThinkingSphinx
+  module ActiveRecord
+    module AttributeUpdates
+      def self.included(base)
+        base.class_eval do
+          after_commit :update_attribute_values
+        end
+      end
+      
+      private
+            
+      def update_attribute_values
+        return unless ThinkingSphinx.updates_enabled? && ThinkingSphinx.sphinx_running?
+        
+        config = ThinkingSphinx::Configuration.instance
+        client = Riddle::Client.new config.address, config.port
+        
+        self.sphinx_indexes.each do |index|
+          attribute_pairs  = attribute_values_for_index(index)
+          attribute_names  = attribute_pairs.keys
+          attribute_values = attribute_names.collect { |key|
+            attribute_pairs[key]
+          }
+          
+          client.update "#{index.name}_core", attribute_names, {
+            sphinx_document_id => attribute_values
+          } if in_core_index?
+        end
+      end
+      
+      def updatable_attributes(index)
+        index.attributes.select { |attrib| attrib.updatable? }
+      end
+      
+      def attribute_values_for_index(index)
+        updatable_attributes(index).inject({}) { |hash, attrib|
+          if attrib.type == :datetime
+            hash[attrib.unique_name.to_s] = attrib.live_value(self).to_time.to_i
+          else
+            hash[attrib.unique_name.to_s] = attrib.live_value self
+          end
+          
+          hash
+        }
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/delta.rb

@@ -69,7 +69,16 @@ module ThinkingSphinx
           end
           
           def should_toggle_delta?
-            !self.respond_to?(:changed?) || self.changed? || self.new_record?
+            self.new_record? || indexed_data_changed?
+          end
+          
+          def indexed_data_changed?
+            sphinx_indexes.any? { |index|
+              index.fields.any? { |field| field.changed?(self) } ||
+              index.attributes.any? { |attrib|
+                attrib.public? && attrib.changed?(self) && !attrib.updatable?
+              }
+            }
           end
         end
       end

data/lib/thinking_sphinx/active_record.rb

@@ -1,3 +1,4 @@
+require 'thinking_sphinx/active_record/attribute_updates'
 require 'thinking_sphinx/active_record/delta'
 require 'thinking_sphinx/active_record/search'
 require 'thinking_sphinx/active_record/has_many_association'
@@ -65,7 +66,7 @@ module ThinkingSphinx
             return unless ThinkingSphinx.define_indexes?
             
             self.sphinx_indexes ||= []
-            index = Index.new(self, &block)
+            index = ThinkingSphinx::Index::Builder.generate(self, &block)
             
             self.sphinx_indexes << index
             unless ThinkingSphinx.indexed_models.include?(self.name)
@@ -79,6 +80,8 @@ module ThinkingSphinx
             
             after_destroy :toggle_deleted
             
+            include ThinkingSphinx::ActiveRecord::AttributeUpdates
+            
             index
           end
           alias_method :sphinx_index, :define_index
@@ -148,7 +151,9 @@ module ThinkingSphinx
             self.sphinx_indexes.select { |ts_index|
               ts_index.model == self
             }.each_with_index do |ts_index, i|
-              index.sources << ts_index.to_riddle_for_core(offset, i)
+              index.sources += ts_index.sources.collect { |source|
+                source.to_riddle_for_core(offset, i)
+              }
             end
             
             index
@@ -160,7 +165,9 @@ module ThinkingSphinx
             index.path = File.join(ThinkingSphinx::Configuration.instance.searchd_file_path, index.name)
             
             self.sphinx_indexes.each_with_index do |ts_index, i|
-              index.sources << ts_index.to_riddle_for_delta(offset, i) if ts_index.delta?
+              index.sources += ts_index.sources.collect { |source|
+                source.to_riddle_for_delta(offset, i)
+              } if ts_index.delta?
             end
             
             index

data/lib/thinking_sphinx/adapters/postgresql_adapter.rb

@@ -11,7 +11,7 @@ module ThinkingSphinx
     
     def concatenate(clause, separator = ' ')
       clause.split(', ').collect { |field|
-        "COALESCE(CAST(#{field} as varchar), '')"
+        field[/COALESCE/] ? field : "COALESCE(CAST(#{field} as varchar), '')"
       }.join(" || '#{separator}' || ")
     end
     

data/lib/thinking_sphinx/attribute.rb

@@ -8,8 +8,8 @@ module ThinkingSphinx
   # generate SQL statements, you'll need to set the base model, and all the
   # associations. Which can get messy. Use Index.link!, it really helps.
   # 
-  class Attribute
-    attr_accessor :alias, :columns, :associations, :model, :faceted, :source
+  class Attribute < ThinkingSphinx::Property
+    attr_accessor :query_source
     
     # To create a new attribute, you'll need to pass in either a single Column
     # or an array of them, and some (optional) options.
@@ -67,20 +67,17 @@ module ThinkingSphinx
     # If you're creating attributes for latitude and longitude, don't forget
     # that Sphinx expects these values to be in radians.
     #  
-    def initialize(columns, options = {})
-      @columns      = Array(columns)
-      @associations = {}
+    def initialize(source, columns, options = {})
+      super
       
-      raise "Cannot define a field with no columns. Maybe you are trying to index a field with a reserved name (id, name). You can fix this error by using a symbol rather than a bare name (:id instead of id)." if @columns.empty? || @columns.any? { |column| !column.respond_to?(:__stack) }
+      @type           = options[:type]
+      @query_source   = options[:source]
+      @crc            = options[:crc]
       
-      @alias    = options[:as]
-      @type     = options[:type]
-      @faceted  = options[:facet]
-      @source   = options[:source]
-      @crc      = options[:crc]
+      @type         ||= :multi    unless @query_source.nil?
+      @type           = :integer  if @type == :string && @crc
       
-      @type   ||= :multi    unless @source.nil?
-      @type     = :integer  if @type == :string && @crc
+      source.attributes << self
     end
     
     # Get the part of the SELECT clause related to this attribute. Don't forget
@@ -93,37 +90,20 @@ module ThinkingSphinx
       return nil unless include_as_association?
       
       clause = @columns.collect { |column|
-        column_with_prefix(column)
+        part = column_with_prefix(column)
+        type == :string ? adapter.convert_nulls(part) : part
       }.join(', ')
       
-      separator = all_ints? ? ',' : ' '
+      separator = all_ints? || @crc ? ',' : ' '
       
-      clause = adapter.concatenate(clause, separator)       if concat_ws?
-      clause = adapter.group_concatenate(clause, separator) if is_many?
       clause = adapter.cast_to_datetime(clause)             if type == :datetime
-      clause = adapter.convert_nulls(clause)                if type == :string
       clause = adapter.crc(clause)                          if @crc
+      clause = adapter.concatenate(clause, separator)       if concat_ws?
+      clause = adapter.group_concatenate(clause, separator) if is_many?
       
       "#{clause} AS #{quote_column(unique_name)}"
     end
     
-    # Get the part of the GROUP BY clause related to this attribute - if one is
-    # needed. If not, all you'll get back is nil. The latter will happen if
-    # there isn't actually a real column to get data from, or if there's
-    # multiple data values (read: a has_many or has_and_belongs_to_many
-    # association).
-    # 
-    def to_group_sql
-      case
-      when is_many?, is_string?, ThinkingSphinx.use_group_by_shortcut?
-        nil
-      else
-        @columns.collect { |column|
-          column_with_prefix(column)
-        }
-      end
-    end
-    
     def type_to_config
       {
         :multi    => :sql_attr_multi,
@@ -136,7 +116,7 @@ module ThinkingSphinx
     end
     
     def include_as_association?
-      ! (type == :multi && (source == :query || source == :ranged_query))
+      ! (type == :multi && (query_source == :query || query_source == :ranged_query))
     end
     
     # Returns the configuration value that should be used for
@@ -153,20 +133,7 @@ module ThinkingSphinx
         unique_name
       end
     end
-    
-    # Returns the unique name of the attribute - which is either the alias of
-    # the attribute, or the name of the only column - if there is only one. If
-    # there isn't, there should be an alias. Else things probably won't work.
-    # Consider yourself warned.
-    # 
-    def unique_name
-      if @columns.length == 1
-        @alias || @columns.first.__name
-      else
-        @alias
-      end
-    end
-    
+        
     # Returns the type of the column. If that's not already set, it returns
     # :multi if there's the possibility of more than one value, :string if
     # there's more than one association, otherwise it figures out what the
@@ -192,18 +159,34 @@ module ThinkingSphinx
       end
     end
     
-    def to_facet
-      return nil unless @faceted
-      
-      ThinkingSphinx::Facet.new(self)
+    def updatable?
+      [:integer, :datetime, :boolean].include?(type) && !is_string?
+    end
+    
+    def live_value(instance)
+      object = instance
+      column = @columns.first
+      column.__stack.each { |method| object = object.send(method) }
+      object.send(column.__name)
+    end
+    
+    def all_ints?
+      @columns.all? { |col|
+        klasses = @associations[col].empty? ? [@model] :
+          @associations[col].collect { |assoc| assoc.reflection.klass }
+        klasses.all? { |klass|
+          column = klass.columns.detect { |column| column.name == col.__name.to_s }
+          !column.nil? && column.type == :integer
+        }
+      }
     end
     
     private
     
     def source_value(offset)
       if is_string?
-        "#{source.to_s.dasherize}; #{columns.first.__name}"
-      elsif source == :ranged_query
+        "#{query_source.to_s.dasherize}; #{columns.first.__name}"
+      elsif query_source == :ranged_query
         "ranged-query; #{query offset} #{query_clause}; #{range_query}"
       else
         "query; #{query offset}"
@@ -249,84 +232,10 @@ FROM #{quote_table_name assoc.table}
       }
     end
     
-    def adapter
-      @adapter ||= @model.sphinx_database_adapter
-    end
-    
-    def quote_with_table(table, column)
-      "#{quote_table_name(table)}.#{quote_column(column)}"
-    end
-    
-    def quote_column(column)
-      @model.connection.quote_column_name(column)
-    end
-    
-    def quote_table_name(table_name)
-      @model.connection.quote_table_name(table_name)
-    end
-    
-    # Indication of whether the columns should be concatenated with a space
-    # between each value. True if there's either multiple sources or multiple
-    # associations.
-    # 
-    def concat_ws?
-      multiple_associations? || @columns.length > 1
-    end
-        
-    # Checks whether any column requires multiple associations (which only
-    # happens for polymorphic situations).
-    # 
-    def multiple_associations?
-      associations.any? { |col,assocs| assocs.length > 1 }
-    end
-    
-    # Builds a column reference tied to the appropriate associations. This
-    # dives into the associations hash and their corresponding joins to
-    # figure out how to correctly reference a column in SQL.
-    # 
-    def column_with_prefix(column)
-      if column.is_string?
-        column.__name
-      elsif associations[column].empty?
-        "#{@model.quoted_table_name}.#{quote_column(column.__name)}"
-      else
-        associations[column].collect { |assoc|
-          assoc.has_column?(column.__name) ?
-          "#{quote_table_name(assoc.join.aliased_table_name)}" + 
-          ".#{quote_column(column.__name)}" :
-          nil
-        }.compact.join(', ')
-      end
-    end
-    
-    # Could there be more than one value related to the parent record? If so,
-    # then this will return true. If not, false. It's that simple.
-    # 
-    def is_many?
-      associations.values.flatten.any? { |assoc| assoc.is_many? }
-    end
-    
     def is_many_ints?
       concat_ws? && all_ints?
     end
-    
-    # Returns true if any of the columns are string values, instead of database
-    # column references.
-    def is_string?
-      columns.all? { |col| col.is_string? }
-    end
-    
-    def all_ints?
-      @columns.all? { |col|
-        klasses = @associations[col].empty? ? [@model] :
-          @associations[col].collect { |assoc| assoc.reflection.klass }
-        klasses.all? { |klass|
-          column = klass.columns.detect { |column| column.name == col.__name.to_s }
-          !column.nil? && column.type == :integer
-        }
-      }
-    end
-    
+        
     def type_from_database
       klass = @associations.values.flatten.first ? 
         @associations.values.flatten.first.reflection.klass : @model
@@ -347,9 +256,10 @@ FROM #{quote_table_name assoc.table}
       else
         raise <<-MESSAGE
 
-Cannot automatically map column type #{type_from_db} to an equivalent Sphinx
-type (integer, float, boolean, datetime, string as ordinal). You could try to
-explicitly convert the column's value in your define_index block:
+Cannot automatically map attribute #{unique_name} in #{@model.name} to an
+equivalent Sphinx type (integer, float, boolean, datetime, string as ordinal).
+You could try to explicitly convert the column's value in your define_index
+block:
   has "CAST(column AS INT)", :type => :integer, :as => :column
         MESSAGE
       end

data/lib/thinking_sphinx/class_facet.rb

@@ -0,0 +1,15 @@
+module ThinkingSphinx
+  class ClassFacet < ThinkingSphinx::Facet
+    def name
+      :class
+    end
+    
+    def attribute_name
+      "class_crc"
+    end
+    
+    def value(object, attribute_value)
+      object.class.name
+    end
+  end
+end

data/lib/thinking_sphinx/collection.rb

@@ -46,6 +46,7 @@ module ThinkingSphinx
       ids = matches.collect { |match| match[:attributes]["sphinx_internal_id"] }
       instances = ids.length > 0 ? klass.find(
         :all,
+        :joins      => options[:joins],
         :conditions => {klass.primary_key.to_sym => ids},
         :include    => (options[:include] || index_options[:include]),
         :select     => (options[:select]  || index_options[:select]),

data/lib/thinking_sphinx/configuration.rb

@@ -31,8 +31,10 @@ module ThinkingSphinx
   # fashion to database.yml - using the following keys: config_file,
   # searchd_log_file, query_log_file, pid_file, searchd_file_path, port,
   # allow_star, enable_star, min_prefix_len, min_infix_len, mem_limit,
-  # max_matches, # morphology, charset_type, charset_table, ignore_chars,
-  # html_strip, # html_remove_elements. I think you've got the idea.
+  # max_matches, morphology, charset_type, charset_table, ignore_chars,
+  # html_strip, html_remove_elements, delayed_job_priority.
+  #
+  # I think you've got the idea.
   # 
   # Each setting in the YAML file is optional - so only put in the ones you
   # want to change.
@@ -55,7 +57,8 @@ module ThinkingSphinx
         
     attr_accessor :config_file, :searchd_log_file, :query_log_file,
       :pid_file, :searchd_file_path, :address, :port, :allow_star,
-      :database_yml_file, :app_root, :bin_path, :model_directories
+      :database_yml_file, :app_root, :bin_path, :model_directories,
+      :delayed_job_priority
     
     attr_accessor :source_options, :index_options
     
@@ -87,6 +90,7 @@ module ThinkingSphinx
       self.bin_path             = ""
       self.model_directories    = ["#{app_root}/app/models/"] +
         Dir.glob("#{app_root}/vendor/plugins/*/app/models/")
+      self.delayed_job_priority = 0
       
       self.source_options  = {}
       self.index_options   = {

data/lib/thinking_sphinx/deltas/datetime_delta.rb

@@ -39,7 +39,7 @@ module ThinkingSphinx
       
       def clause(model, toggled)
         if toggled
-          "#{model.quoted_table_name}.#{@index.quote_column(@column.to_s)}" +
+          "#{model.quoted_table_name}.#{model.connection.quote_column_name(@column.to_s)}" +
           " > #{adapter.time_difference(@threshold)}"
         else
           nil

data/lib/thinking_sphinx/deltas/default_delta.rb

@@ -39,11 +39,12 @@ module ThinkingSphinx
       
       def reset_query(model)
         "UPDATE #{model.quoted_table_name} SET " +
-        "#{@index.quote_column(@column.to_s)} = #{adapter.boolean(false)}"
+        "#{model.connection.quote_column_name(@column.to_s)} = #{adapter.boolean(false)} " +
+        "WHERE #{model.connection.quote_column_name(@column.to_s)} = #{adapter.boolean(true)}"
       end
       
       def clause(model, toggled)
-        "#{model.quoted_table_name}.#{@index.quote_column(@column.to_s)}" +
+        "#{model.quoted_table_name}.#{model.connection.quote_column_name(@column.to_s)}" +
         " = #{adapter.boolean(toggled)}"
       end
       

data/lib/thinking_sphinx/deltas/delayed_delta.rb

@@ -9,13 +9,15 @@ module ThinkingSphinx
     class DelayedDelta < ThinkingSphinx::Deltas::DefaultDelta
       def index(model, instance = nil)
         ThinkingSphinx::Deltas::Job.enqueue(
-          ThinkingSphinx::Deltas::DeltaJob.new(delta_index_name(model))
+          ThinkingSphinx::Deltas::DeltaJob.new(delta_index_name(model)),
+          ThinkingSphinx::Configuration.instance.delayed_job_priority
         )
         
         Delayed::Job.enqueue(
           ThinkingSphinx::Deltas::FlagAsDeletedJob.new(
             core_index_name(model), instance.sphinx_document_id
-          )
+          ),
+          ThinkingSphinx::Configuration.instance.delayed_job_priority
         ) if instance
         
         true