thinking-sphinx 3.0.0.pre → 3.0.0.rc

data/Gemfile

@@ -6,5 +6,8 @@ gem 'combustion',
   :git => 'git://github.com/pat/combustion.git',
   :ref => '45f50e64c3'
 
-gem 'pg', '~> 0.11.0', :platform => :ruby
+gem 'mysql2', '~> 0.3.12b4', :platform => :ruby
+gem 'pg',     '~> 0.11.0',   :platform => :ruby
+
+gem 'activerecord-jdbcmysql-adapter',      '~> 1.1.3', :platform => :jruby
 gem 'activerecord-jdbcpostgresql-adapter', '~> 1.1.3', :platform => :jruby

data/HISTORY

@@ -1,2 +1,18 @@
+2012-12-22: 3.0.0.rc
+* [FEATURE] Source type support (query and ranged query) for both attributes and fields. Custom SQL strings can be supplied as well.
+* [FEATURE] Wordcount attributes and fields now supported.
+* [FEATURE] Support for Sinatra and other non-Rails frameworks.
+* [FEATURE] A sphinx scope can be defined as the default.
+* [FEATURE] An index can have multiple sources, by using define_source within the index definition.
+* [FEATURE] sanitize_sql is available within an index definition.
+* [FEATURE] Providing :prefixes => true or :infixes => true as an option when declaring a field means just the noted fields have infixes/prefixes applied.
+* [FEATURE] ThinkingSphinx::Search#query_time returns the time Sphinx took to make the query.
+* [FEATURE] Namespaced model support.
+* [FEATURE] Default settings for index definition arguments can be set in config/thinking_sphinx.yml.
+* [FIX] Correctly escape nulls in inheritance column (Darcy Laycock).
+* [FIX] Use ThinkingSphinx::Configuration#render_to_file instead of ThinkingSphinx::Configuration#build in test helpers (Darcy Laycock).
+* [FIX] Suppressing delta output in test helpers now works (Darcy Laycock).
+* [FEATURE] A custom Riddle/Sphinx controller can be supplied. Useful for Flying Sphinx to have an API layer over Sphinx commands, without needing custom gems for different Thinking Sphinx/Flying Sphinx combinations.
+
 2012-10-06: 3.0.0.pre
 * First pre-release. Not quite feature complete, but the important stuff is certainly covered. See the README for more the finer details.

data/README.textile

@@ -28,7 +28,16 @@ You'll notice the first argument is the model name downcased and as a symbol, an
 
 Other changes:
 * SphinxQL is now used instead of the old socket connections (hence the dependency on the @mysql2@ gem).
-* You'll need to include @ThinkingSphinx::Scopes@ into your models if you want to use Sphinx scopes.
+* You'll need to include @ThinkingSphinx::Scopes@ into your models if you want to use Sphinx scopes. Default scopes can be set as follows:
+
+<pre><code>class Person < ActiveRecord::Base
+  include ThinkingSphinx::Scopes
+
+  sphinx_scope(:date_order) { {:order => :created_at} }
+  default_sphinx_scope :date_order
+  # ...
+end</code></pre>
+
 * The match mode is always extended - SphinxQL doesn't know any other way.
 * ActiveRecord::Base.set_sphinx_primary_key is now an option in the index definition (alongside @:with@ in the above example): @:primary_key@ - and therefore is no longer inheritable between models.
 * If you're explicitly setting a time attribute's type, instead of @:datetime@ it should now be @:timestamp@.
@@ -113,6 +122,31 @@ batch.searches #=> [[foo results], [bar results], [baz results]]</code></pre>
 * Automatic updates of non-string attributes are still limited to those from columns on the model in question, and is disabled by default. To enable it, just set attribute_updates to true in your @config/sphinx.yml@.
 * Search result helper methods are no longer injected into the actual result objects. Read the section below on search results, glazes and panes.
 * If you're using string facets, make sure they're defined as fields, not strings. There is currently no support for multi-value string facets.
+* To have fine-grained control over when deltas are invoked, create a sub-class of your chosen delta class (the standard is @ThinkingSphinx::Deltas::DefaultDelta@) and customise the @toggle@ and @toggled?@ methods, both of which accept a single parameter being the ActiveRecord instance.
+
+<pre><code>class OccasionalDeltas < ThinkingSphinx::Deltas::DefaultDelta
+  # Invoked via a before_save callback. The default behaviour is to set the
+  # delta column to true.
+  def toggle(instance)
+    super unless instance.title_changed?
+  end
+
+  # Invoked via an after_commit callback. The default behaviour is to check
+  # whether the delta column is set to true. If this method returns true, the
+  # indexer is fired.
+  def toggled?(instance)
+    return false unless instance.title_changed?
+
+    super
+  end
+end
+
+# And in your index definition:
+ThinkingSphinx::Index.define :article, :with => :active_record, :delta => OccasionalDeltas do
+  # ...
+end</code></pre>
+
+* When you're defining indices for namespaced models, use a lowercase string with /'s for namespacing as the model reference: @ThinkingSphinx::Index.define 'blog/article', :with => :active_record@.
 
 h2. Search Middleware
 
@@ -144,25 +178,7 @@ If you wish to add your own panes, go ahead. The only requirement is that the in
 
 h2. Limitations
 
-Basic indexing and searching should be fine. There's currently only limited basic delta support. Some settings haven't yet been brought across. Many of the smaller features don't yet exist either. Some may actually not return... we'll see.
-
-Lists of what still needs to be implemented, in no particular order but in groups of vague importance:
-
-Required for Release Candidate:
-
-* Query and Ranged Query sources for Attributes and Fields
-* Using :though association shortcuts in index definitions
-* Overwritable toggle_delta? method on model
-* Query times for searches
-* Namespaced models support
-* Delayed Deltas
-* Infixing and Prefixing of specific fields
-* Default Sphinx scopes
-* sanitise_sql method in index definitions
-* Multiple sources for an index
-* JRuby support
-* Sinatra support
-* Wordcount fields and attributes
+Almost all functionality from v2 releases are implemented, though it's worth noting that some settings haven't yet been brought across, and a handful of the smaller features don't yet exist either. Some may actually not return... we'll see.
 
 May or may not be added:
 
@@ -175,11 +191,11 @@ May or may not be added:
 
 h3. Sphinx Versions
 
-TS 3 is built for Sphinx 2.x only. You cannot use 1.10-beta, 0.9.9 or anything earlier than that.
+TS 3 is built for Sphinx 2.x only. You cannot use 1.10-beta, 0.9.9 or anything earlier than that. 2.0.5 or newer is recommended.
 
 h3. Rails Versions
 
-Currently TS 3 is built to support Rails/ActiveRecord 3.1 or newer. Sinatra is not yet supported, but it will be.
+Currently TS 3 is built to support Rails/ActiveRecord 3.1 or newer. If you're using Sinatra and ActiveRecord instead of Rails, that's fine - just make sure you add the @:require => 'thinking_sphinx/sinatra'@ option when listing @thinking-sphinx@ in your Gemfile.
 
 TS 3 does not support Rails 3.0, Rails 2.x or earlier, or Merb - please refer to the TS 1.x and 2.x releases in those situations.
 
@@ -187,6 +203,8 @@ h3. Ruby Versions
 
 Built on MRI 1.9.3 and tested against MRI 1.9.2 as well. No plans to support MRI 1.8, but would like to support Rubinius and JRuby (the one catch with the latter is the different MySQL interfaces).
 
+There's also the complication that Sphinx 2.0.x releases don't work with JDBC (as JDBC sends several MySQL-specific comamnds through when initializing a connection). So, JRuby support won't appear until there's a stable Sphinx release that can interact with JDBC.
+
 h3. Database Versions
 
 MySQL 5.x and Postgres 8.4 or better are supported.
@@ -201,4 +219,4 @@ For some ideas behind my current approach, have a look through @sketchpad.rb@ in
 
 h2. Licence
 
-Copyright (c) 2007-2012, Thinking Sphinx is developed and maintained by Pat Allan, and is released under the open MIT Licence. Many thanks to "all who have contributed patches":https://github.com/freelancing-god/thinking-sphinx/contributors.
+Copyright (c) 2007-2012, Thinking Sphinx is developed and maintained by Pat Allan, and is released under the open MIT Licence. Many thanks to "all who have contributed patches":https://github.com/pat/thinking-sphinx/contributors.

data/lib/thinking_sphinx.rb

@@ -1,3 +1,10 @@
+if RUBY_PLATFORM == 'java'
+  require 'java'
+  require 'jdbc/mysql'
+else
+  require 'mysql2'
+end
+
 require 'riddle'
 require 'middleware'
 require 'active_record'
@@ -26,9 +33,11 @@ require 'thinking_sphinx/batched_search'
 require 'thinking_sphinx/callbacks'
 require 'thinking_sphinx/core'
 require 'thinking_sphinx/configuration'
+require 'thinking_sphinx/connection'
 require 'thinking_sphinx/excerpter'
 require 'thinking_sphinx/facet'
 require 'thinking_sphinx/facet_search'
+require 'thinking_sphinx/frameworks'
 require 'thinking_sphinx/index'
 require 'thinking_sphinx/index_set'
 require 'thinking_sphinx/masks'

data/lib/thinking_sphinx/active_record.rb

@@ -16,6 +16,7 @@ require 'thinking_sphinx/active_record/field'
 require 'thinking_sphinx/active_record/index'
 require 'thinking_sphinx/active_record/interpreter'
 require 'thinking_sphinx/active_record/log_subscriber'
+require 'thinking_sphinx/active_record/property_query'
 require 'thinking_sphinx/active_record/property_sql_presenter'
 require 'thinking_sphinx/active_record/sql_builder'
 require 'thinking_sphinx/active_record/sql_source'

data/lib/thinking_sphinx/active_record/attribute/sphinx_presenter.rb

@@ -10,8 +10,8 @@ class ThinkingSphinx::ActiveRecord::Attribute::SphinxPresenter
     :wordcount => :str2wordcount
   }
 
-  def initialize(attribute)
-    @attribute = attribute
+  def initialize(attribute, source)
+    @attribute, @source = attribute, source
   end
 
   def collection_type
@@ -20,7 +20,7 @@ class ThinkingSphinx::ActiveRecord::Attribute::SphinxPresenter
 
   def declaration
     if @attribute.multi?
-      "#{sphinx_type} #{@attribute.name} from field"
+      multi_declaration
     else
       @attribute.name
     end
@@ -29,4 +29,21 @@ class ThinkingSphinx::ActiveRecord::Attribute::SphinxPresenter
   def sphinx_type
     SPHINX_TYPES[@attribute.type]
   end
+
+  private
+
+  def multi_declaration
+    case @attribute.source_type
+    when :query, :ranged_query
+      query
+    else
+      "#{sphinx_type} #{@attribute.name} from field"
+    end
+  end
+
+  def query
+    ThinkingSphinx::ActiveRecord::PropertyQuery.new(
+      @attribute, @source, sphinx_type
+    ).to_s
+  end
 end

data/lib/thinking_sphinx/active_record/base.rb

@@ -20,8 +20,11 @@ module ThinkingSphinx::ActiveRecord::Base
     end
 
     def search(query = nil, options = {})
-      search = ThinkingSphinx.search query, options
-      ThinkingSphinx::Search::Merger.new(search).merge! nil, :classes => [self]
+      merger = ThinkingSphinx::Search::Merger.new ThinkingSphinx.search
+
+      merger.merge! *default_sphinx_scope_response if default_sphinx_scope?
+      merger.merge! query, options
+      merger.merge! nil, :classes => [self]
     end
 
     def search_count(query = nil, options = {})
@@ -32,5 +35,15 @@ module ThinkingSphinx::ActiveRecord::Base
       search = search query, options
       ThinkingSphinx::Search::Merger.new(search).merge! nil, :ids_only => true
     end
+
+    private
+
+    def default_sphinx_scope?
+      respond_to?(:default_sphinx_scope) && default_sphinx_scope
+    end
+
+    def default_sphinx_scope_response
+      [sphinx_scopes[default_sphinx_scope].call].flatten
+    end
   end
 end

data/lib/thinking_sphinx/active_record/callbacks/delete_callbacks.rb

@@ -5,7 +5,7 @@ class ThinkingSphinx::ActiveRecord::Callbacks::DeleteCallbacks <
 
   def after_destroy
     indices.each do |index|
-      connection.query Riddle::Query.update(
+      connection.execute Riddle::Query.update(
         index.name, index.document_id_for_key(instance.id),
         :sphinx_deleted => true
       )
@@ -21,7 +21,7 @@ class ThinkingSphinx::ActiveRecord::Callbacks::DeleteCallbacks <
   end
 
   def connection
-    @connection ||= config.connection
+    @connection ||= ThinkingSphinx::Connection.new
   end
 
   def indices

data/lib/thinking_sphinx/active_record/callbacks/update_callbacks.rb

@@ -42,7 +42,7 @@ class ThinkingSphinx::ActiveRecord::Callbacks::UpdateCallbacks <
     sphinxql = Riddle::Query.update(
       index.name, index.document_id_for_key(instance.id), attributes
     )
-    configuration.connection.query(sphinxql)
+    ThinkingSphinx::Connection.new.execute(sphinxql)
   rescue Mysql2::Error => error
     # This isn't vital, so don't raise the error.
   end

data/lib/thinking_sphinx/active_record/field.rb

@@ -1,5 +1,6 @@
 class ThinkingSphinx::ActiveRecord::Field <
   ThinkingSphinx::ActiveRecord::Property
+  include ThinkingSphinx::Core::Field
 
   def file?
     options[:file]
@@ -8,4 +9,8 @@ class ThinkingSphinx::ActiveRecord::Field <
   def with_attribute?
     options[:sortable] || options[:facet]
   end
+
+  def wordcount?
+    options[:wordcount]
+  end
 end

data/lib/thinking_sphinx/active_record/index.rb

@@ -6,7 +6,7 @@ class ThinkingSphinx::ActiveRecord::Index < Riddle::Configuration::Index
 
   def append_source
     ThinkingSphinx::ActiveRecord::SQLSource.new(
-      model, source_options
+      model, source_options.merge(:position => sources.length)
     ).tap do |source|
       sources << source
     end
@@ -25,7 +25,7 @@ class ThinkingSphinx::ActiveRecord::Index < Riddle::Configuration::Index
   end
 
   def unique_attribute_names
-    sources.collect(&:attributes).flatten.collect(&:name)
+    attributes.collect(&:name)
   end
 
   private
@@ -35,6 +35,14 @@ class ThinkingSphinx::ActiveRecord::Index < Riddle::Configuration::Index
       adapter_for(model)
   end
 
+  def attributes
+    sources.collect(&:attributes).flatten
+  end
+
+  def fields
+    sources.collect(&:fields).flatten
+  end
+
   def interpreter
     ThinkingSphinx::ActiveRecord::Interpreter
   end

data/lib/thinking_sphinx/active_record/interpreter.rb

@@ -1,6 +1,11 @@
 class ThinkingSphinx::ActiveRecord::Interpreter <
   ::ThinkingSphinx::Core::Interpreter
 
+  def define_source(&block)
+    @source = @index.append_source
+    instance_eval &block
+  end
+
   def group_by(*columns)
     __source.groupings += columns
   end
@@ -23,6 +28,10 @@ class ThinkingSphinx::ActiveRecord::Interpreter <
     }
   end
 
+  def sanitize_sql(*arguments)
+    __source.model.send :sanitize_sql, *arguments
+  end
+
   def set_property(properties)
     properties.each do |key, value|
       @index.send("#{key}=", value)   if @index.class.settings.include?(key)

data/lib/thinking_sphinx/active_record/property.rb

@@ -22,6 +22,10 @@ class ThinkingSphinx::ActiveRecord::Property
     (options[:as] || columns.first.__name).to_s
   end
 
+  def source_type
+    options[:source]
+  end
+
   def type
     nil
   end

data/lib/thinking_sphinx/active_record/property_query.rb

@@ -0,0 +1,112 @@
+class ThinkingSphinx::ActiveRecord::PropertyQuery
+  def initialize(property, source, type = nil)
+    @property, @source, @type = property, source, type
+  end
+
+  def to_s
+    identifier = [type, property.name].compact.join(' ')
+    queries    = []
+    if column.string?
+      queries << column.__name
+    else
+      queries << to_sql
+      queries << range_sql if ranged?
+    end
+
+    "#{identifier} from #{source_type}; #{queries.join('; ')}"
+  end
+
+  private
+
+  attr_reader :property, :source, :type
+
+  def base_association
+    reflections.first
+  end
+
+  def column
+    @column ||= property.columns.first
+  end
+
+  def extend_reflection(reflection)
+    return [reflection] unless reflection.through_reflection
+
+    [reflection.through_reflection, reflection.source_reflection]
+  end
+
+  def reflections
+    @reflections ||= begin
+      base = source.model
+
+      column.__stack.collect { |key|
+        reflection = base.reflections[key]
+        base       = reflection.klass
+
+        extend_reflection reflection
+      }.flatten
+    end
+  end
+
+  def joins
+    @joins ||= begin
+      remainder = reflections.collect(&:name)[1..-1]
+      return nil             if remainder.empty?
+      return remainder.first if remainder.length == 1
+
+      remainder[0..-2].reverse.inject(remainder.last) { |value, key|
+        {key => value}
+      }
+    end
+  end
+
+  def offset
+    "* #{ThinkingSphinx::Configuration.instance.indices.count} + #{source.offset}"
+  end
+
+  def quoted_foreign_key
+    quote_with_table base_association.klass.table_name,
+      base_association.foreign_key
+  end
+
+  def quoted_primary_key
+    quote_with_table reflections.last.klass.table_name, column.__name
+  end
+
+  def quote_with_table(table, column)
+    "#{quote_column(table)}.#{quote_column(column)}"
+  end
+
+  def quote_column(column)
+    ActiveRecord::Base.connection.quote_column_name(column)
+  end
+
+  def ranged?
+    property.source_type == :ranged_query
+  end
+
+  def range_sql
+    base_association.klass.unscoped.
+      select("MIN(#{quoted_foreign_key}), MAX(#{quoted_foreign_key})").to_sql
+  end
+
+  def source_type
+    property.source_type.to_s.dasherize
+  end
+
+  def to_sql
+    raise "Could not determine SQL for MVA" if reflections.empty?
+
+    relation = base_association.klass.unscoped
+    relation = relation.joins joins unless joins.blank?
+    relation = relation.select "#{quoted_foreign_key} #{offset} AS #{quote_column('id')}, #{quoted_primary_key} AS #{quote_column(property.name)}"
+
+    if ranged?
+      relation = relation.where("#{quoted_foreign_key} >= $start")
+      relation = relation.where("#{quoted_foreign_key} <= $end")
+    end
+
+    relation = relation.order("#{quoted_foreign_key} ASC") if type.nil?
+
+    relation.to_sql
+  end
+end