thinking-sphinx 2.0.6 → 2.0.7

data/HISTORY

@@ -0,0 +1,157 @@
+2.0.6 - August 28th 2011
+* Don't change superclass index definitions (otherwise sibling subclasses can end up with each others' index definitions).
+* Improved scope support (Andrew White).
+* Fixed association-filtered searches for Rails 3.1 (Andrew White).
+* Fixed polymorphic association support for Rails 3.1 (Jonathan Viney).
+* 1.4.7 changes.
+
+2.0.5 - May 25th 2011
+* Railtie equivalent for Sinatra.
+* Rails 3.1 support (Florent Piteau).
+* 1.4.6 changes.
+
+2.0.4 - May 12th 2011
+* Only determine Sphinx version when initializing Thinking Sphinx, instead of upon require.
+* Still execute logged blocks for searching even when logging isn't enabled.
+* 1.4.5 changes.
+
+2.0.3 - April 3rd 2011
+* 1.4.4 changes.
+
+2.0.2 - January 13th 2011
+* Improvement to loading the environment via Rake (Brenton Fletcher).
+* 1.4.2 changes.
+* 1.4.1 changes.
+
+2.0.1 - November 19th 2011
+* Intermittent as_json bug fix.
+* Using Rails 3 logging system (Stephen Celis).
+* Fixes for Rails 3.0.2 (Ivan Ukhov).
+
+2.0.0 - November 14th 2011
+* Handle has_many conditions defined as a hash (Artem Orlov).
+* Deprecation fixes for logging (Anton Sozontov).
+* Only load Thinking Sphinx if ActiveRecord is loaded (Matthew Higgins).
+* Rails 3 support.
+* 1.4.0 changes.
+
+1.4.7 - August 28th 2011
+* Don't search for objects when updating attributes on every save - if the object isn't there, then the error will be caught anyway.
+* Extra flexibility for association attribute references (Andrew White).
+* Setting up test suite for Travis, and fixing tests across many rubies.
+* Removing test for filtering on wordcount attribute - seems Sphinx doesn't like that.
+* If many indices are searched on, only use the first for excerpts.
+* Allow MVA facets from SQL strings that are all integers to skip object translation.
+* Adding total_count method for better Kaminari support (Dan Pickett).
+* Switching from Jeweler to Bundler for gem management.
+* Cleaning up Context - re-using camelized class names.
+* Simplified auto-star regex for 1.9 - no need to set the unicode flag.
+* Don't update sphinx_internal_id attribute when updating a record's attributes.
+* Ignore empty arrays for :without_ids (Jason Rust).
+* Now catching Riddle::ResponseError when we aren't fussed about the responses from Sphinx.
+* Distinguishing between Ruby 1.8 and 1.9 for auto-star regex (Andrew White).
+* Don't re-index at the end of suspended_delta calls if requested.
+* Adding stop_timeout setting to configure how long ts:stop should wait while checking whether Sphinx has stopped (Justin Tanner).
+* Speed improvement for types_to_crcs (Josh Goebel).
+* Adding a touched_reindex_file option, for a file that will be touched after indices are rotated (Rémi Prévost).
+* Better support for the Mysql2 adapter (Andrew White).
+* More pagination methods - next_page? and first_page?.
+* Adding use_64_bit option to ensure all timestamp attributes are treated as 64 bit integers (Andrey Deryabin).
+* Simpler, less brittle support for all Sphinx settings via Riddle (Clemens Kofler).
+* Optimised start/stop behaviour in Capistrano recipe (Lars Weiler).
+* Catch and ignore timeout errors for low priority requests.
+* Don't double up on de-polymorphised associations for generated Sphinx source SQL queries.
+
+1.4.6 - May 25th 2011
+* Require Riddle 1.3.3 or better.
+* Cast Sphinx document ids to 32bit integers to keep PostgreSQL happy when Sphinx is compiled for id64 support (Bruno Santschi).
+* Making sure suspended_delta always exists on indexed models, whether indices have been defined or not.
+* Workaround for ActsAsTaggableOn to ensure association joins work.
+* Use Sphinx string attributes instead of class_crc integers to determine which class each search result is from (Sphinx 2.x or newer).
+* String attribute support - and so, using this instead of str2ordinal for string sorting.
+* Support for fields with paired string attributes and wordcount attributes (Sphinx 1.10-beta or newer).
+* Support for file fields (Sphinx 1.10-beta or newer).
+* Wordcount attribute support (Sphinx 1.10-beta or newer).
+* Sinatra support is only used when Rails isn't loaded.
+* Sphinx 2.0.x (including 2.0.2-dev) support.
+* :without_any filtering option (the reverse of :with_all).
+
+1.4.5 - May 12th 2011
+* Require Riddle 1.3.2 or better.
+* Improved 1.9.2 support
+* Don't hack Array - make ThinkingSphinx::Search a subclass of Array instead.
+* Distinguishing between suspended deltas and disabled deltas.
+* Removing most Thread references, opting for Mutexes for better multi-threading support.
+* Added support for client_key to allow for authenticated Sphinx communication.
+* Sphinx 2.0.1 support.
+* Don't try to translate nil values (Alex Chee).
+* Performance fix for facets (Clemens Kofler).
+* Use Sphinx 0.9.9 in Capistrano recipe (Konstantin Shabanov).
+* Sinatra support (Patrick Tulskie).
+* Offer sanitize_sql method within define_index blocks (Matt Todd).
+* Performance fix: caching primary_key_for_sphinx value.
+
+1.4.4 - April 3 2011
+* Being consistent with === comparisons for Array monkeypatch.
+* Kaminari pagination support.
+* Ensure deltas are supported across multiple indices for a given model.
+* Use custom index names for excerpts calls if that's what exists (Javier Ramírez).
+* Allow for Sphinx versions compiled from source (Greg Weber).
+* Ensuring thread is being required to allow Mutexes to work.
+* Ensure primary_key_for_sphinx is inherited to subclasses (Robert Stern).
+* Don't complain if there's an error when checking for documents in a given index.
+
+1.4.3 - January 29th 2011
+* Don't memoize the database adapter.
+* Allow queries to just return specific attributes via the :only option, instead of ActiveRecord objects (Hans Hasselberg).
+* Performance fix: Only add sphinx scopes to has_many associations if there are scopes in play (Kirill Maximov).
+* Fixing CRC32 function for PostgreSQL to handle UTF characters that use more than one byte.
+
+1.4.2 - January 13th 2011
+* Ignore Sphinx errors when updating attributes (it's a low priority request).
+* Don't use the CRC32 function within the generated SQL queries for class types.
+* Direct facets to use a method to translate values using :value.
+* Ensure that if one sphinx_internal_id attribute is a bigint, they all are.
+* Ignore Sphinx errors when marking documents as deleted (it's a low priority request).
+* Don't select columns from polymorphic joins if they're not available.
+* Don't query Sphinx for a facet request if zero facets are requested.
+* Ensure we're only using local_options when it's available (Paul Schyska).
+* Don't presume the environment for Capistrano is production (Robert Glaser).
+* Allow for custom database adapters.
+* Fix for custom delta columns (Marcin Stecki).
+
+1.4.1 - December 21st 2010
+* No longer looking for attributes in :conditions option when searching.
+* Copy and then modify sort arguments.
+* Allow the generation of Sphinx configuration objects without necessarily writing to the configuration file.
+* Allow Sphinx 1.10-beta versions that have been compiled with id64 support.
+* Raise Sphinx errors when searching - which can be ignored using the :ignore_errors option (Matt Todd).
+* Handle complex :include arguments, including hashes within arrays (Paco Guzmán).
+* Handle hashes passed through to :include when searching.
+* Can now require either 'thinking_sphinx' or 'thinking-sphinx'.
+* Excerpts are less fussy about inputs.
+* No longer hard-coding RSpec colour setting - that's up to the developer.
+* Require Riddle 1.2.0 or better.
+* Use searchd to stop Sphinx (via --stop or --stopwait arguments) (Matt Todd).
+
+1.4.0 - November 14th 2010
+* No longer supporting attributes in :conditions option.
+* Case insensitive field sorting if :sortable is set to :insensitive.
+* Using Bundler for development
+* Can use make facet requests on existing searches and Sphinx scopes.
+* Don't star field markers in search queries.
+* Can use search_for_ids on Sphinx scopes.
+* Ensure the CRC32 function for PostgreSQL handles empty strings and NULLs.
+* Allow custom determination for database adapters.
+* :include search option now limits itself to the relevant classes when searching across more than one class.
+* Don't memoize primary_key_for_sphinx, which was causing trouble when objects are deleted or frozen.
+* Allow for excerpts options when searching (using :excerpt_options) (Lee Capps).
+* Wrap UTC in quotes when setting the timezone in PostgreSQL (Keith Pitt).
+* Don't colourize logs unless ActiveRecord allows for it.
+* Consistent contribution documentation (Sam Goldstein).
+* Allow the setting of Riddle's timeout value via sphinx.yml (Sam Goldstein).
+* 1.10-beta support.
+* Can set the riddle client for the search request using :client.
+* Fixing default sphinx scope support.
+* Make a reasonable attempt to check that Sphinx has stopped (Matt Todd).
+* Be more verbose if a model can't be loaded by Thinking Sphinx.

data/lib/cucumber/thinking_sphinx/external_world.rb

@@ -0,0 +1,12 @@
+require 'thinking_sphinx/test'
+
+module Cucumber
+  module ThinkingSphinx
+    class ExternalWorld
+      def initialize(suppress_delta_output = true)
+        ::ThinkingSphinx::Test.init
+        ::ThinkingSphinx::Test.start_with_autostop
+      end
+    end
+  end
+end

data/lib/cucumber/thinking_sphinx/internal_world.rb

@@ -0,0 +1,127 @@
+require 'cucumber/thinking_sphinx/sql_logger'
+
+module Cucumber
+  module ThinkingSphinx
+    class InternalWorld
+      attr_accessor :temporary_directory, :migrations_directory,
+        :models_directory, :fixtures_directory, :database_file
+      attr_accessor :adapter, :database, :username,
+        :password, :host
+
+      def initialize
+        pwd = Dir.pwd
+        @temporary_directory  = "#{pwd}/tmp"
+        @migrations_directory = "#{pwd}/features/thinking_sphinx/db/migrations"
+        @models_directory     = "#{pwd}/features/thinking_sphinx/models"
+        @fixtures_directory   = "#{pwd}/features/thinking_sphinx/db/fixtures"
+        @database_file        = "#{pwd}/features/thinking_sphinx/database.yml"
+
+        @adapter  = (ENV['DATABASE'] || 'mysql').gsub /^mysql$/, 'mysql2'
+        @database = 'thinking_sphinx'
+        @username = @adapter[/mysql/] ? 'root' : 'postgres'
+        # @password = 'thinking_sphinx'
+        @host     = 'localhost'
+      end
+
+      def setup
+        make_temporary_directory
+
+        configure_cleanup
+        configure_thinking_sphinx
+        configure_active_record
+
+        prepare_data
+        setup_sphinx
+
+        self
+      end
+
+      def configure_database
+        ActiveRecord::Base.establish_connection database_settings
+        self
+      end
+
+      private
+
+      def config
+        @config ||= ::ThinkingSphinx::Configuration.instance
+      end
+
+      def make_temporary_directory
+        FileUtils.mkdir_p temporary_directory
+        Dir["#{temporary_directory}/*"].each do |file|
+          FileUtils.rm_rf file
+        end
+      end
+
+      def configure_thinking_sphinx
+        config.config_file        = "#{temporary_directory}/sphinx.conf"
+        config.searchd_log_file   = "#{temporary_directory}/searchd.log"
+        config.query_log_file     = "#{temporary_directory}/searchd.query.log"
+        config.pid_file           = "#{temporary_directory}/searchd.pid"
+        config.searchd_file_path  = "#{temporary_directory}/indexes/"
+
+        ::ThinkingSphinx.suppress_delta_output = true
+      end
+
+      def configure_cleanup
+        Kernel.at_exit do
+          ::ThinkingSphinx::Configuration.instance.controller.stop
+          sleep(0.5) # Ensure Sphinx has shut down completely
+          ::ThinkingSphinx::ActiveRecord::LogSubscriber.logger.close
+        end
+      end
+
+      def yaml_database_settings
+        return {} unless File.exist?(@database_file)
+
+        YAML.load open(@database_file)
+      end
+
+      def database_settings
+        {
+          'adapter'  => @adapter,
+          'database' => @database,
+          'username' => @username,
+          'password' => @password,
+          'host'     => @host
+        }.merge yaml_database_settings
+      end
+
+      def configure_active_record
+        ::ThinkingSphinx::ActiveRecord::LogSubscriber.logger = Logger.new(
+          open("#{temporary_directory}/active_record.log", "a")
+        )
+
+        ActiveRecord::Base.connection.class.send(
+          :include, Cucumber::ThinkingSphinx::SqlLogger
+        )
+      end
+
+      def prepare_data
+        ::ThinkingSphinx.deltas_enabled = false
+
+        load_files migrations_directory
+        load_files models_directory
+        load_files fixtures_directory
+
+        ::ThinkingSphinx.deltas_enabled = true
+      end
+
+      def load_files(path)
+        files = Dir["#{path}/*.rb"].sort!
+        files.each do |file|
+          require file.gsub(/\.rb$/, '')
+        end
+      end
+
+      def setup_sphinx
+        FileUtils.mkdir_p config.searchd_file_path
+
+        config.build
+        config.controller.index
+        config.controller.start
+      end
+    end
+  end
+end

data/lib/cucumber/thinking_sphinx/sql_logger.rb

@@ -0,0 +1,20 @@
+module Cucumber
+  module ThinkingSphinx
+    module SqlLogger
+      def self.included(base)
+        base.send :alias_method_chain, :execute, :query_record
+      end
+      
+      IGNORED_SQL = [
+        /^PRAGMA/, /^SELECT currval/, /^SELECT CAST/, /^SELECT @@IDENTITY/,
+        /^SELECT @@ROWCOUNT/, /^SHOW FIELDS/
+      ]
+      
+      def execute_with_query_record(sql, name = nil, &block)
+        $queries_executed ||= []
+        $queries_executed << sql unless IGNORED_SQL.any? { |r| sql =~ r }
+        execute_without_query_record(sql, name, &block)
+      end
+    end
+  end
+end

data/lib/thinking-sphinx.rb

@@ -0,0 +1 @@
+require 'thinking_sphinx'

data/lib/thinking_sphinx/action_controller.rb

@@ -0,0 +1,31 @@
+module ThinkingSphinx
+  module ActionController
+    extend ActiveSupport::Concern
+
+    protected
+
+    attr_internal :query_runtime
+
+    def cleanup_view_runtime
+      log_subscriber = ThinkingSphinx::ActiveRecord::LogSubscriber
+      query_runtime_pre_render = log_subscriber.reset_runtime
+      runtime = super
+      query_runtime_post_render = log_subscriber.reset_runtime
+      self.query_runtime = query_runtime_pre_render + query_runtime_post_render
+      runtime - query_runtime_post_render
+    end
+
+    def append_info_to_payload(payload)
+      super
+      payload[:query_runtime] = query_runtime
+    end
+
+    module ClassMethods
+      def log_process_action(payload)
+        messages, query_runtime = super, payload[:query_runtime]
+        messages << ("Sphinx: %.1fms" % query_runtime.to_f) if query_runtime
+        messages
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/attribute_updates.rb

@@ -0,0 +1,53 @@
+module ThinkingSphinx
+  module ActiveRecord
+    module AttributeUpdates
+      def self.included(base)
+        base.class_eval do
+          after_save :update_attribute_values
+        end
+      end
+
+      private
+
+      def update_attribute_values
+        return true unless ThinkingSphinx.updates_enabled? &&
+          ThinkingSphinx.sphinx_running?
+
+        self.class.sphinx_indexes.each do |index|
+          attribute_pairs  = attribute_values_for_index(index)
+          attribute_names  = attribute_pairs.keys
+          attribute_values = attribute_names.collect { |key|
+            attribute_pairs[key]
+          }
+
+          update_index index.core_name, attribute_names, attribute_values
+          next unless index.delta?
+          update_index index.delta_name, attribute_names, attribute_values
+        end
+
+        true
+      end
+
+      def updatable_attributes(index)
+        index.attributes.select { |attrib| attrib.updatable? }
+      end
+
+      def attribute_values_for_index(index)
+        updatable_attributes(index).inject({}) { |hash, attrib|
+          hash[attrib.unique_name.to_s] = attrib.live_value self
+          hash
+        }
+      end
+
+      def update_index(index_name, attribute_names, attribute_values)
+        config = ThinkingSphinx::Configuration.instance
+        config.client.update index_name, attribute_names, {
+          sphinx_document_id => attribute_values
+        }
+      rescue Riddle::ConnectionError, Riddle::ResponseError,
+        ThinkingSphinx::SphinxError, Errno::ETIMEDOUT
+        # Not the end of the world if Sphinx isn't running.
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/collection_proxy.rb

@@ -0,0 +1,40 @@
+module ThinkingSphinx
+  module ActiveRecord
+    module CollectionProxy
+      def search(*args)
+        options   = args.extract_options!
+        options[:with] ||= {}
+        options[:with].merge! default_filter
+
+        args << options
+        proxy_association.klass.search(*args)
+      end
+
+      private
+
+      def attribute_for_foreign_key
+        if proxy_association.reflection.through_reflection
+          foreign_key = proxy_association.reflection.through_reflection.foreign_key
+        else
+          foreign_key = proxy_association.reflection.foreign_key
+        end
+
+        proxy_association.klass.define_indexes
+        (proxy_association.klass.sphinx_indexes || []).each do |index|
+          attribute = index.attributes.detect { |attrib|
+            attrib.columns.length == 1 &&
+            attrib.columns.first.__name  == foreign_key.to_sym ||
+            attrib.alias == foreign_key.to_sym
+          }
+          return attribute unless attribute.nil?
+        end
+
+        raise "Missing Attribute for Foreign Key #{foreign_key}"
+      end
+
+      def default_filter
+        {attribute_for_foreign_key.unique_name => proxy_association.owner.id}
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/collection_proxy_with_scopes.rb

@@ -0,0 +1,27 @@
+module ThinkingSphinx
+  module ActiveRecord
+    module CollectionProxyWithScopes
+      def self.included(base)
+        base.class_eval do
+          alias_method_chain :method_missing, :sphinx_scopes
+          alias_method_chain :respond_to?, :sphinx_scopes
+        end
+      end
+
+      def method_missing_with_sphinx_scopes(method, *args, &block)
+        klass = proxy_association.klass
+        if klass.respond_to?(:sphinx_scopes) && klass.sphinx_scopes.include?(method)
+          klass.search(:with => default_filter).send(method, *args, &block)
+        else
+          method_missing_without_sphinx_scopes(method, *args, &block)
+        end
+      end
+
+      def respond_to_with_sphinx_scopes?(method)
+        proxy_association.klass.respond_to?(:sphinx_scopes) &&
+        proxy_association.klass.sphinx_scopes.include?(scope) ||
+        respond_to_without_sphinx_scopes?(method)
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/delta.rb

@@ -0,0 +1,65 @@
+module ThinkingSphinx
+  module ActiveRecord
+    # This module contains all the delta-related code for models. There isn't
+    # really anything you need to call manually in here - except perhaps
+    # index_delta, but not sure what reason why.
+    # 
+    module Delta
+      # Code for after_commit callback is written by Eli Miller:
+      # http://elimiller.blogspot.com/2007/06/proper-cache-expiry-with-aftercommit.html
+      # with slight modification from Joost Hietbrink.
+      #
+      def self.included(base)
+        base.class_eval do
+          class << self
+            # Build the delta index for the related model. This won't be called
+            # if running in the test environment.
+            #
+            def index_delta(instance = nil)
+              delta_objects.each { |obj| obj.index(self, instance) }
+            end
+            
+            def delta_objects
+              self.sphinx_indexes.collect(&:delta_object).compact
+            end
+          end
+          
+          def toggled_delta?
+            self.class.delta_objects.any? { |obj| obj.toggled(self) }
+          end
+          
+          private
+          
+          # Set the delta value for the model to be true.
+          def toggle_delta
+            self.class.delta_objects.each { |obj|
+              obj.toggle(self)
+            } if should_toggle_delta?
+          end
+          
+          # Build the delta index for the related model. This won't be called
+          # if running in the test environment.
+          # 
+          def index_delta
+            self.class.index_delta(self) if self.class.delta_objects.any? { |obj|
+              obj.toggled(self)
+            }
+          end
+          
+          def should_toggle_delta?
+            self.new_record? || indexed_data_changed?
+          end
+          
+          def indexed_data_changed?
+            sphinx_indexes.any? { |index|
+              index.fields.any? { |field| field.changed?(self) } ||
+              index.attributes.any? { |attrib|
+                attrib.public? && attrib.changed?(self) && !attrib.updatable?
+              }
+            }
+          end
+        end
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/has_many_association.rb

@@ -0,0 +1,37 @@
+module ThinkingSphinx
+  module ActiveRecord
+    module HasManyAssociation
+      def search(*args)
+        options   = args.extract_options!
+        options[:with] ||= {}
+        options[:with].merge! default_filter
+        
+        args << options
+        @reflection.klass.search(*args)
+      end
+
+      private
+      
+      def attribute_for_foreign_key
+        foreign_key = @reflection.primary_key_name
+        stack = [@reflection.options[:through]].compact
+        
+        @reflection.klass.define_indexes
+        (@reflection.klass.sphinx_indexes || []).each do |index|
+          attribute = index.attributes.detect { |attrib|
+            attrib.columns.length == 1 &&
+            attrib.columns.first.__name  == foreign_key.to_sym ||
+            attrib.alias == foreign_key.to_sym
+          }
+          return attribute unless attribute.nil?
+        end
+        
+        raise "Missing Attribute for Foreign Key #{foreign_key}"
+      end
+      
+      def default_filter
+        {attribute_for_foreign_key.unique_name => @owner.id}
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/has_many_association_with_scopes.rb

@@ -0,0 +1,21 @@
+module ThinkingSphinx
+  module ActiveRecord
+    module HasManyAssociationWithScopes
+      def method_missing(method, *args, &block)
+        if responds_to_scope(method)
+          @reflection.klass.
+            search(:with => default_filter).
+            send(method, *args, &block)
+        else
+          super
+        end
+      end
+      
+      private
+      def responds_to_scope(scope)
+        @reflection.klass.respond_to?(:sphinx_scopes)   &&
+        @reflection.klass.sphinx_scopes.include?(scope)
+      end
+    end
+  end
+end

data/lib/thinking_sphinx/active_record/log_subscriber.rb

@@ -0,0 +1,61 @@
+require 'active_support/log_subscriber'
+
+module ThinkingSphinx
+  module ActiveRecord
+    class LogSubscriber < ActiveSupport::LogSubscriber
+      def self.runtime=(value)
+        Thread.current['thinking_sphinx_query_runtime'] = value
+      end
+
+      def self.runtime
+        Thread.current['thinking_sphinx_query_runtime'] ||= 0
+      end
+
+      def self.reset_runtime
+        rt, self.runtime = runtime, 0
+        rt
+      end
+
+      def initialize
+        super
+        @odd_or_even = false
+      end
+
+      def query(event)
+        self.class.runtime += event.duration
+        return unless logger.debug?
+
+        identifier = color('Sphinx Query (%.1fms)' % event.duration, GREEN, true)
+        query = event.payload[:query]
+        query = color query, nil, true if odd?
+
+        debug "  #{identifier}  #{query}"
+      end
+
+      def message(event)
+        return unless logger.debug?
+
+        identifier = color 'Sphinx', GREEN, true
+        message = event.payload[:message]
+        message = color message, nil, true if odd?
+
+        debug "  #{identifier}  #{message}"
+      end
+
+      def odd?
+        @odd_or_even = !@odd_or_even
+      end
+
+      def logger
+        return @logger if defined? @logger
+        self.logger = ::ActiveRecord::Base.logger
+      end
+
+      def logger=(logger)
+        @logger = logger
+      end
+
+      attach_to :thinking_sphinx
+    end
+  end
+end