thinking-sphinx 3.0.0 → 3.0.1

data/.travis.yml

@@ -1,13 +1,16 @@
 rvm:
   - 1.9.2
   - 1.9.3
-before_install: gem update --system
+before_install:
+  - gem update --system
+  - curl -O http://fs-packages.s3.amazonaws.com/fs-sphinx-2.0.6_i386_12.04.deb
+  - sudo dpkg -i fs-sphinx-2.0.6_i386_12.04.deb
 before_script:
   - "mysql -e 'create database thinking_sphinx;' > /dev/null"
   - "psql -c 'create database thinking_sphinx;' -U postgres >/dev/null"
 env:
-  - DATABASE=mysql2
-  - DATABASE=postgresql
+  - DATABASE=mysql2 SPHINX_BIN=/usr/local/sphinx-2.0.6/bin/ SPHINX_VERSION=2.0.6
+  - DATABASE=postgresql SPHINX_BIN=/usr/local/sphinx-2.0.6/bin/ SPHINX_VERSION=2.0.6
 gemfile:
   - gemfiles/rails_3_1.gemfile
   - gemfiles/rails_3_2.gemfile

data/HISTORY

@@ -1,3 +1,21 @@
+2013-02-04: 3.0.1
+* [FEATURE] Provide Capistrano deployment tasks (David Celis).
+* [FEATURE] Allow specifying of Sphinx version. Is only useful for Flying Sphinx purposes at this point - has no impact on Riddle or Sphinx.
+* [FEATURE] Support new JDBC configuration style (when JDBC can be used) (Kyle Stevens).
+* [FIX] Referring to associations via polymorphic associations in an index definition now works.
+* [FEATURE] Mysql2::Errors are wrapped as ThinkingSphinx::SphinxErrors, with subclasses of SyntaxError and ParseError used appropriately. Syntax and parse errors do not prompt a retry on a new connection.
+* [CHANGE] Use connection pool for search queries. If a query fails, it will be retried on a new connection before raising if necessary.
+* [CHANGE] Glaze always passes methods through to the underlying ActiveRecord::Base object if they don't exist on any of the panes.
+* [FIX] Don't override foreign keys for polymorphic association replacements.
+* [FIX] Quote namespaced model names in class field condition.
+* [FEATURE] Polymorphic associations can be used within index definitions when the appropriate classes are set out.
+* [FEATURE] Allow custom strings for SQL joins in index definitions.
+* [FIX] New lines are maintained and escaped in custom source queries.
+* [FIX] Subclasses of indexed models fire delta callbacks properly.
+* [FIX] Thinking Sphinx can be loaded via thinking/sphinx, to satisfy Bundler.
+* [FEATURE] indexer and searchd settings are added to the appropriate objects from config/thinking_sphinx.yml (@ygelfand).
+* [FIX] New lines are maintained and escaped in sql_query values.
+
 2013-01-02: 3.0.0
 * [CHANGE] Updating Riddle dependency to 1.5.4.
 * [FIX] Respect source options as well as underlying settings via the set_property method in index definitions.

data/README.textile

@@ -1,15 +1,19 @@
 h1. Thinking Sphinx
 
-Welcome to Thinking Sphinx version 3 - a complete rewrite from past versions. Right now it is a work-in-progress, though it does include most of the more popular features from TS 2 and earlier. It's also currently built for Rails 3.1 or newer only.
-
-Still interested? Well, read on!
+Welcome to Thinking Sphinx version 3 - a complete rewrite from past versions, built for Rails 3.1 or newer only.
 
 h2. Installation
 
-There's only a pre-release gem available at this point - or you can use the git repository reference (the edge branch is what you're after). Also, you'll need to specify the Mysql2 gem as well (this is not an inbuilt dependency because JRuby, when supported, will need something different):
+It's a gem, so install it like you would any other gem. You will also need to specify the Mysql2 gem as well (this is not an inbuilt dependency because JRuby, when supported, will need something different):
 
 <pre><code>gem 'mysql2',          '0.3.12b4'
-gem 'thinking-sphinx', '3.0.0.pre'</code></pre>
+gem 'thinking-sphinx', '3.0.0'</code></pre>
+
+The mysql2 gem is required for connecting to Sphinx, so please include it even when you're using PostgreSQL for your database.
+
+As for Sphinx itself, you need to make sure it's compiled with MySQL support, even if you're using PostgreSQL as your database - which is normally the default, but Homebrew can be too smart sometimes. If that's your compiling tool of choice, make sure you install with the @--mysql@ flag:
+
+<pre><code>brew install sphinx --mysql</code></pre>
 
 h2. Usage
 
@@ -26,6 +30,11 @@ end</code></pre>
 
 You'll notice the first argument is the model name downcased and as a symbol, and we are specifying the processor - @:active_record@. Everything inside the block is just like previous versions of Thinking Sphinx. Same goes for @config/thinking_sphinx.yml@ (formerly @config/sphinx.yml@).
 
+When you're defining indices for namespaced models, use a lowercase string with /'s for namespacing as the model reference:
+
+<pre><code># For a model named Blog::Article:
+ThinkingSphinx::Index.define 'blog/article', :with => :active_record</code></pre>
+
 Other changes:
 * SphinxQL is now used instead of the old socket connections (hence the dependency on the @mysql2@ gem).
 * You'll need to include @ThinkingSphinx::Scopes@ into your models if you want to use Sphinx scopes. Default scopes can be set as follows:
@@ -65,7 +74,7 @@ end</code></pre>
   :classes => [User, AdminUser, SupportUser]</code></pre>
 
 * The option @:rank_mode@ has now become @:ranker@ - and the options (as strings or symbols) are as follows: proximity_bm25, bm25, none, wordcount, proximity, matchany, and fieldmask.
-* There are no explicit sorting modes - all sorting must be on attributes followed by ASC or DESC. For example: @:order => '@weight DESC, created_at ASC'@.
+* There are no explicit sorting modes - all sorting must be on attributes followed by ASC or DESC. For example: <code>:order => '@weight DESC, created_at ASC'</code>.
 * If you specify just an attribute name as a symbol for the @:order@ option, it will be given the ascending direction by default. So, @:order => :created_at@ is equivalent to @:order => 'created_at ASC'@.
 * If you want to use a calculated expression for sorting, you must specify the expression as a new attribute, then use that attribute in your @:order@ option. This is done using the @:select@ option to specify extra columns available in the underlying SphinxQL (_not_ ActiveRecord/SQL) query.
 
@@ -146,7 +155,11 @@ ThinkingSphinx::Index.define :article, :with => :active_record, :delta => Occasi
   # ...
 end</code></pre>
 
-* When you're defining indices for namespaced models, use a lowercase string with /'s for namespacing as the model reference: @ThinkingSphinx::Index.define 'blog/article', :with => :active_record@.
+* Polymorphic associations used within index definitions must be declared with the corresponding models. This is much better than the old approach of querying the database on the *_type column to determine what models to join against.
+
+<pre><code>indexes events.eventable.name
+
+polymorphs events.eventable, :to => %w(Page Post User)</code></pre>
 
 h2. Search Middleware
 
@@ -176,6 +189,17 @@ All panes namespaced to @ThinkingSphinx::Panes@, and the @DistancePane@ is autom
 
 If you wish to add your own panes, go ahead. The only requirement is that the initializer must accept three arguments: the search context, the underlying search result object, and a hash of the raw values from Sphinx.
 
+h2. Deployment with Capistrano
+
+Thinking Sphinx comes with several Capistrano tasks to help ease deployment of your applications. Just require the recipes:
+
+```ruby
+# config/deploy.rb
+require 'thinking_sphinx/capistrano'
+```
+
+When running `cap deploy:setup` to get your server ready for deployments, Thinking Sphinx will also set up a shared folder for your indexes. Before finalizing a deployment, these indexes will be symlinked into the release path for use. When you deploy your application for the first time using `cap deploy:cold`, your indexes will be built for you and the search daemon will be started. Run `cap -T` to see all of the deployment tasks.
+
 h2. Limitations
 
 Almost all functionality from v2 releases are implemented, though it's worth noting that some settings haven't yet been brought across, and a handful of the smaller features don't yet exist either. Some may actually not return... we'll see.
@@ -186,7 +210,6 @@ May or may not be added:
 * Bitmask weighting helper
 * Timezone support (for databases not using UTC)
 * Abstract Inheritance support (maybe - not sure this is something many of people want).
-* Capistrano Tasks
 * Facet support for arrays of strings.
 
 h3. Sphinx Versions

data/gemfiles/rails_3_1.gemfile

@@ -2,9 +2,9 @@
 
 source :rubygems
 
-gem "riddle", "1.5.3"
-gem "combustion", :git=>"git://github.com/pat/combustion.git", :ref=>"45f50e64c3"
+gem "mysql2", "~> 0.3.12b4", :platform=>:ruby
 gem "pg", "~> 0.11.0", :platform=>:ruby
+gem "activerecord-jdbcmysql-adapter", "~> 1.1.3", :platform=>:jruby
 gem "activerecord-jdbcpostgresql-adapter", "~> 1.1.3", :platform=>:jruby
 gem "rails", "~> 3.1.0"
 

data/gemfiles/rails_3_2.gemfile

@@ -2,9 +2,9 @@
 
 source :rubygems
 
-gem "riddle", "1.5.3"
-gem "combustion", :git=>"git://github.com/pat/combustion.git", :ref=>"45f50e64c3"
+gem "mysql2", "~> 0.3.12b4", :platform=>:ruby
 gem "pg", "~> 0.11.0", :platform=>:ruby
+gem "activerecord-jdbcmysql-adapter", "~> 1.1.3", :platform=>:jruby
 gem "activerecord-jdbcpostgresql-adapter", "~> 1.1.3", :platform=>:jruby
 gem "rails", "~> 3.2.0"
 

data/lib/thinking/sphinx.rb

@@ -0,0 +1 @@
+require 'thinking_sphinx'

data/lib/thinking_sphinx.rb

@@ -35,6 +35,7 @@ require 'thinking_sphinx/callbacks'
 require 'thinking_sphinx/core'
 require 'thinking_sphinx/configuration'
 require 'thinking_sphinx/connection'
+require 'thinking_sphinx/errors'
 require 'thinking_sphinx/excerpter'
 require 'thinking_sphinx/facet'
 require 'thinking_sphinx/facet_search'

data/lib/thinking_sphinx/active_record.rb

@@ -13,9 +13,11 @@ require 'thinking_sphinx/active_record/base'
 require 'thinking_sphinx/active_record/column'
 require 'thinking_sphinx/active_record/database_adapters'
 require 'thinking_sphinx/active_record/field'
+require 'thinking_sphinx/active_record/filtered_reflection'
 require 'thinking_sphinx/active_record/index'
 require 'thinking_sphinx/active_record/interpreter'
 require 'thinking_sphinx/active_record/log_subscriber'
+require 'thinking_sphinx/active_record/polymorpher'
 require 'thinking_sphinx/active_record/property_query'
 require 'thinking_sphinx/active_record/property_sql_presenter'
 require 'thinking_sphinx/active_record/sql_builder'

data/lib/thinking_sphinx/active_record/association.rb

@@ -6,4 +6,12 @@ class ThinkingSphinx::ActiveRecord::Association
   def stack
     @column.__stack + [@column.__name]
   end
+
+  def string?
+    @column.is_a?(String)
+  end
+
+  def to_s
+    @column.to_s
+  end
 end

data/lib/thinking_sphinx/active_record/associations.rb

@@ -15,7 +15,7 @@ class ThinkingSphinx::ActiveRecord::Associations
   def aggregate_for?(stack)
     return false if stack.empty?
 
-    joins_for(stack).any? { |join|
+    joins_for(stack).compact.any? { |join|
       [:has_many, :has_and_belongs_to_many].include?(
         join.reflection.macro
       )
@@ -29,7 +29,14 @@ class ThinkingSphinx::ActiveRecord::Associations
   end
 
   def join_values
-    @joins.values
+    @joins.values.compact
+  end
+
+  def model_for(stack)
+    return model if stack.empty?
+
+    join = join_for(stack)
+    join.nil? ? nil : join.reflection.klass
   end
 
   private
@@ -40,9 +47,14 @@ class ThinkingSphinx::ActiveRecord::Associations
 
   def join_for(stack)
     @joins[stack] ||= begin
-      JoinDependency::JoinAssociation.new(
-        reflection_for(stack), base, parent_join_for(stack)
-      ).tap { |join| join.join_type = Arel::OuterJoin }
+      reflection = reflection_for stack
+      reflection.nil? ? nil : JoinDependency::JoinAssociation.new(
+        reflection, base, parent_join_for(stack)
+      ).tap { |join|
+        join.join_type = Arel::OuterJoin
+
+        rewrite_conditions_for join
+      }
     end
   end
 
@@ -65,4 +77,12 @@ class ThinkingSphinx::ActiveRecord::Associations
   def reflection_for(stack)
     parent_for(stack).active_record.reflections[stack.last]
   end
+
+  def rewrite_conditions_for(join)
+    conditions = Array(join.conditions).flatten
+    conditions.each do |condition|
+      condition.gsub! /::ts_join_alias::/,
+        model.connection.quote_table_name(join.parent.aliased_table_name)
+    end
+  end
 end

data/lib/thinking_sphinx/active_record/callbacks/delta_callbacks.rb

@@ -42,14 +42,10 @@ class ThinkingSphinx::ActiveRecord::Callbacks::DeltaCallbacks <
   end
 
   def indices
-    @indices ||= config.indices_for_references reference
+    @indices ||= ThinkingSphinx::IndexSet.new [instance.class], []
   end
 
   def processors
     delta_indices.collect &:delta_processor
   end
-
-  def reference
-    instance.class.name.underscore.to_sym
-  end
 end

data/lib/thinking_sphinx/active_record/column.rb

@@ -8,6 +8,18 @@ class ThinkingSphinx::ActiveRecord::Column
     @name
   end
 
+  def __path
+    @stack + [@name]
+  end
+
+  def __replace(stack, replacements)
+    return [self] if string? || __stack[0..stack.length-1] != stack
+
+    replacements.collect { |replacement|
+      self.class.new *(replacement + __stack[stack.length..-1]), __name
+    }
+  end
+
   def __stack
     @stack
   end

data/lib/thinking_sphinx/active_record/database_adapters.rb

@@ -28,6 +28,13 @@ module ThinkingSphinx::ActiveRecord::DatabaseAdapters
         :mysql
       when "jdbcpostgresql"
         :postgresql
+      when "jdbc"
+        match = /^jdbc:(?<adapter>mysql|postgresql):\/\//.match(model.connection.config[:url])
+        if match
+          match[:adapter].to_sym
+        else
+          model.connection.config[:adapter]
+        end
       else
         model.connection.config[:adapter]
       end

data/lib/thinking_sphinx/active_record/filtered_reflection.rb

@@ -0,0 +1,49 @@
+class ThinkingSphinx::ActiveRecord::FilteredReflection <
+  ActiveRecord::Reflection::AssociationReflection
+
+  class Options
+    attr_reader :reflection, :class_name, :options
+
+    delegate :foreign_type, :active_record, :to => :reflection
+
+    def initialize(reflection, class_name)
+      @reflection, @class_name = reflection, class_name
+      @options = reflection.options.clone
+    end
+
+    def filtered
+      options.delete :polymorphic
+      options[:class_name]    = class_name
+      options[:foreign_key] ||= "#{reflection.name}_id"
+
+      case options[:conditions]
+      when nil
+        options[:conditions] = condition
+      when Array
+        options[:conditions] << condition
+      when Hash
+        options[:conditions].merge!(reflection.foreign_type => options[:class_name])
+      else
+        options[:conditions] << " AND #{condition}"
+      end
+
+      options
+    end
+
+    private
+
+    def condition
+      "::ts_join_alias::.#{quoted_foreign_type} = '#{class_name}'"
+    end
+
+    def quoted_foreign_type
+      active_record.connection.quote_column_name foreign_type
+    end
+  end
+
+  def self.clone_with_filter(reflection, name, class_name)
+    options = Options.new(reflection, class_name).filtered
+
+    new reflection.macro, name, options, reflection.active_record
+  end
+end

data/lib/thinking_sphinx/active_record/interpreter.rb

@@ -28,6 +28,12 @@ class ThinkingSphinx::ActiveRecord::Interpreter <
     }
   end
 
+  def polymorphs(column, options)
+    __source.polymorphs << ::ThinkingSphinx::ActiveRecord::Polymorpher.new(
+      __source, column, options[:to]
+    )
+  end
+
   def sanitize_sql(*arguments)
     __source.model.send :sanitize_sql, *arguments
   end

data/lib/thinking_sphinx/active_record/polymorpher.rb

@@ -0,0 +1,50 @@
+class ThinkingSphinx::ActiveRecord::Polymorpher
+  def initialize(source, column, class_names)
+    @source, @column, @class_names = source, column, class_names
+  end
+
+  def morph!
+    append_reflections
+    morph_properties
+  end
+
+  private
+
+  attr_reader :source, :column, :class_names
+
+  def append_reflections
+    mappings.each do |class_name, name|
+      klass.reflections[name] ||= ThinkingSphinx::ActiveRecord::
+        FilteredReflection.clone_with_filter(reflection, name, class_name)
+    end
+  end
+
+  def mappings
+    @mappings ||= class_names.inject({}) do |hash, class_name|
+      hash[class_name] = "#{column.__name}_#{class_name.downcase}".to_sym
+      hash
+    end
+  end
+
+  def morphed_stacks
+    @morphed_stacks ||= mappings.values.collect { |key|
+      column.__stack + [key]
+    }
+  end
+
+  def morph_properties
+    (source.fields + source.attributes).each do |property|
+      property.rebase column.__path, :to => morphed_stacks
+    end
+  end
+
+  def reflection
+    @reflection ||= klass.reflections[column.__name]
+  end
+
+  def klass
+    @klass ||= column.__stack.inject(source.model) { |parent, key|
+      parent.reflections[key].klass
+    }
+  end
+end

data/lib/thinking_sphinx/active_record/property.rb

@@ -12,6 +12,12 @@ class ThinkingSphinx::ActiveRecord::Property
     }
   end
 
+  def rebase(associations, options)
+    @columns = columns.inject([]) do |array, column|
+      array + column.__replace(associations, options[:to])
+    end
+  end
+
   def name
     (options[:as] || columns.first.__name).to_s
   end

data/lib/thinking_sphinx/active_record/property_query.rb

@@ -7,7 +7,7 @@ class ThinkingSphinx::ActiveRecord::PropertyQuery
     identifier = [type, property.name].compact.join(' ')
     queries    = []
     if column.string?
-      queries << column.__name
+      queries << column.__name.strip.gsub(/\n/, "\\\n")
     else
       queries << to_sql
       queries << range_sql if ranged?

data/lib/thinking_sphinx/active_record/property_sql_presenter.rb

@@ -38,8 +38,14 @@ class ThinkingSphinx::ActiveRecord::PropertySQLPresenter
     clause
   end
 
+  def column_exists?(column)
+    model = associations.model_for(column.__stack)
+    model && model.column_names.include?(column.__name.to_s)
+  end
+
   def column_with_table(column)
     return column.__name if column.string?
+    return nil unless column_exists?(column)
 
     "#{associations.alias_for(column.__stack)}.#{adapter.quote column.__name}"
   end
@@ -47,7 +53,7 @@ class ThinkingSphinx::ActiveRecord::PropertySQLPresenter
   def columns_with_table
     property.columns.collect { |column|
       column_with_table(column)
-    }.join(', ')
+    }.compact.join(', ')
   end
 
   def concatenating?