pager-ultrasphinx 1.0.20080510

data/lib/ultrasphinx/search.rb

@@ -0,0 +1,456 @@
+
+module Ultrasphinx
+
+=begin rdoc
+Command-interface Search object.
+
+== Basic usage
+  
+To set up a search, instantiate an Ultrasphinx::Search object with a hash of parameters. Only the <tt>:query</tt> key is mandatory.
+  @search = Ultrasphinx::Search.new(
+    :query => @query, 
+    :sort_mode => 'descending', 
+    :sort_by => 'created_at'
+  )
+    
+Now, to run the query, call its <tt>run</tt> method. Your results will be available as ActiveRecord instances via the <tt>results</tt> method. Example:  
+  @search.run
+  @search.results
+
+= Options
+
+== Query format
+
+The query string supports boolean operation, parentheses, phrases, and field-specific search. Query words are stemmed and joined by an implicit <tt>AND</tt> by default.
+
+* Valid boolean operators are <tt>AND</tt>, <tt>OR</tt>, and <tt>NOT</tt>.
+* Field-specific searches should be formatted as <tt>fieldname:contents</tt>. (This will only work for text fields. For numeric and date fields, see the <tt>:filters</tt> parameter, below.)
+* Phrases must be enclosed in double quotes.
+    
+A Sphinx::SphinxInternalError will be raised on invalid queries. In general, queries can only be nested to one level. 
+  @query = 'dog OR cat OR "white tigers" NOT (lions OR bears) AND title:animals'
+
+== Hash parameters
+
+The hash lets you customize internal aspects of the search.
+
+<tt>:per_page</tt>:: An integer. How many results per page.
+<tt>:page</tt>:: An integer. Which page of the results to return.
+<tt>:class_names</tt>:: An array or string. The class name of the model you want to search, an array of model names to search, or <tt>nil</tt> for all available models.    
+<tt>:sort_mode</tt>:: <tt>'relevance'</tt> or <tt>'ascending'</tt> or <tt>'descending'</tt>. How to order the result set. Note that <tt>'time'</tt> and <tt>'extended'</tt> modes are available, but not tested.  
+<tt>:sort_by</tt>:: A field name. What field to order by for <tt>'ascending'</tt> or <tt>'descending'</tt> mode. Has no effect for <tt>'relevance'</tt>.
+<tt>:weights</tt>:: A hash. Text-field names and associated query weighting. The default weight for every field is 1.0. Example: <tt>:weights => {'title' => 2.0}</tt>
+<tt>:filters</tt>:: A hash. Names of numeric or date fields and associated values. You can use a single value, an array of values, or a range. (See the bottom of the ActiveRecord::Base page for an example.)
+<tt>:facets</tt>:: An array of fields for grouping/faceting. You can access the returned facet values and their result counts with the <tt>facets</tt> method.
+<tt>:location</tt>:: A hash. Specify the names of your latititude and longitude attributes as declared in your is_indexed calls. To sort the results by distance, set <tt>:sort_mode => 'extended'</tt> and <tt>:sort_by => 'distance asc'.</tt>
+<tt>:indexes</tt>:: An array of indexes to search. Currently only <tt>Ultrasphinx::MAIN_INDEX</tt> and <tt>Ultrasphinx::DELTA_INDEX</tt> are available. Defaults to both; changing this is rarely needed.
+
+== Query Defaults
+
+Note that you can set up your own query defaults in <tt>environment.rb</tt>: 
+  
+  self.class.query_defaults = HashWithIndifferentAccess.new({
+    :per_page => 10,
+    :sort_mode => 'relevance',
+    :weights => {'title' => 2.0}
+  })
+
+= Advanced features
+
+== Geographic distance
+
+If you pass a <tt>:location</tt> Hash, distance from the location in meters will be available in your result records via the <tt>distance</tt> accessor:
+
+  @search = Ultrasphinx::Search.new(:class_names => 'Point', 
+            :query => 'pizza',
+            :sort_mode => 'extended',
+            :sort_by => 'distance',
+            :location => {
+              :lat => 40.3,
+              :long => -73.6
+            })
+            
+   @search.run.first.distance #=> 1402.4
+
+Note that Sphinx expects lat/long to be indexed as radians. If you have degrees in your database, do the conversion in the <tt>is_indexed</tt> as so:
+  
+    is_indexed 'fields' => [
+        'name', 
+        'description',
+        {:field => 'lat', :function_sql => "RADIANS(?)"}, 
+        {:field => 'lng', :function_sql => "RADIANS(?)"}
+      ]
+
+Then, set <tt>Ultrasphinx::Search.client_options[:location][:units] = 'degrees'</tt>.
+
+The MySQL <tt>:double</tt> column type is recommended for storing location data. For Postgres, use <tt>:float</tt.
+
+== Interlock integration
+  
+Ultrasphinx uses the <tt>find_all_by_id</tt> method to instantiate records. If you set <tt>with_finders: true</tt> in {Interlock's}[http://blog.evanweaver.com/files/doc/fauna/interlock] <tt>config/memcached.yml</tt>, Interlock overrides <tt>find_all_by_id</tt> with a caching version.
+
+== Will_paginate integration
+
+The Search instance responds to the same methods as a WillPaginate::Collection object, so once you have called <tt>run</tt> or <tt>excerpt</tt> you can use it directly in your views:
+
+  will_paginate(@search)
+
+== Excerpt mode
+
+You can have Sphinx excerpt and highlight the matched sections in the associated fields. Instead of calling <tt>run</tt>, call <tt>excerpt</tt>. 
+  
+  @search.excerpt
+
+The returned models will be frozen and have their field contents temporarily changed to the excerpted and highlighted results. 
+  
+You need to set the <tt>content_methods</tt> key on Ultrasphinx::Search.excerpting_options to whatever groups of methods you need the excerpter to try to excerpt. The first responding method in each group for each record will be excerpted. This way Ruby-only methods are supported (for example, a metadata method which combines various model fields, or an aliased field so that the original record contents are still available).
+  
+There are some other keys you can set, such as excerpt size, HTML tags to highlight with, and number of words on either side of each excerpt chunk. Example (in <tt>environment.rb</tt>):
+  
+  Ultrasphinx::Search.excerpting_options = HashWithIndifferentAccess.new({
+    :before_match => '<strong>', 
+    :after_match => '</strong>',
+    :chunk_separator => "...",
+    :limit => 256,
+    :around => 3,
+    :content_methods => [['title'], ['body', 'description', 'content'], ['metadata']] 
+  })
+  
+Note that your database is never changed by anything Ultrasphinx does.
+
+=end    
+
+  class Search  
+  
+    include Internals
+    include Parser
+    
+    cattr_accessor :query_defaults  
+    self.query_defaults ||= HashWithIndifferentAccess.new({
+      :query => nil,
+      :page => 1,
+      :per_page => 20,
+      :sort_by => nil,
+      :sort_mode => 'relevance',
+      :indexes => [
+          MAIN_INDEX, 
+          (DELTA_INDEX if Ultrasphinx.delta_index_present?)
+        ].compact,
+      :weights => {},
+      :class_names => [],
+      :filters => {},
+      :facets => [],
+      :location => HashWithIndifferentAccess.new({
+        :lat_attribute_name  => 'lat',
+        :long_attribute_name => 'lng',
+        :units => 'radians'
+      })
+    })
+    
+    cattr_accessor :excerpting_options
+    self.excerpting_options ||= HashWithIndifferentAccess.new({
+      :before_match => "<strong>", :after_match => "</strong>",
+      :chunk_separator => "...",
+      :limit => 256,
+      :around => 3,
+      # Results should respond to one in each group of these, in precedence order, for the 
+      # excerpting to fire
+      :content_methods => [['title', 'name'], ['body', 'description', 'content'], ['metadata']] 
+    })
+    
+    cattr_accessor :client_options
+    self.client_options ||= HashWithIndifferentAccess.new({ 
+      :with_subtotals => false, 
+      :ignore_missing_records => false,
+      # Has no effect if :ignore_missing_records => false
+      :max_missing_records => 5, 
+      :max_retries => 4,
+      :retry_sleep_time => 0.5,
+      :max_facets => 1000,
+      :max_matches_offset => 1000,
+      # Whether to add an accessor to each returned result that specifies its global rank in 
+      # the search.
+      :with_global_rank => false,
+      # Which method names to try to use for loading records. You can define your own (for 
+      # example, with :includes) and then attach it here. Each method must accept an Array 
+      # of ids, but do not have to preserve order. If the class does not respond_to? any 
+      # method name in the array, :find_all_by_id will be used.
+      :finder_methods => [] 
+    })
+    
+    # Friendly sort mode mappings    
+    SPHINX_CLIENT_PARAMS = { 
+      'sort_mode' => {
+        'relevance' => :relevance,
+        'descending' => :attr_desc, 
+        'ascending' => :attr_asc, 
+        'time' => :time_segments,
+        'extended' => :extended,
+      }
+    }
+    
+    INTERNAL_KEYS = ['parsed_query'] #:nodoc:
+
+    MODELS_TO_IDS = Ultrasphinx.get_models_to_class_ids || {} 
+
+    IDS_TO_MODELS = MODELS_TO_IDS.invert #:nodoc:
+    
+    MAX_MATCHES = DAEMON_SETTINGS["max_matches"].to_i 
+
+    FACET_CACHE = {} #:nodoc: 
+    
+    # Returns the options hash.
+    def options
+      @options
+    end
+    
+    #  Returns the query string used.
+    def query
+      # Redundant with method_missing
+      @options['query']
+    end
+    
+    def parsed_query #:nodoc:
+      # Redundant with method_missing
+      @options['parsed_query']
+    end
+    
+    # Returns an array of result objects.
+    def results
+      require_run
+      @results
+    end
+    
+    # Returns the facet map for this query, if facets were used.
+    def facets
+      raise UsageError, "No facet field was configured" unless @options['facets']
+      require_run
+      @facets
+    end      
+          
+    # Returns the raw response from the Sphinx client.
+    def response
+      require_run
+      @response
+    end
+    
+    # Returns a hash of total result counts, scoped to each available model. Set <tt>Ultrasphinx::Search.client_options[:with_subtotals] = true</tt> to enable.
+    # 
+    # The subtotals are implemented as a special type of facet.
+    def subtotals
+      raise UsageError, "Subtotals are not enabled" unless self.class.client_options['with_subtotals']
+      require_run
+      @subtotals
+    end
+
+    # Returns the total result count.
+    def total_entries
+      require_run
+      [response[:total_found] || 0, MAX_MATCHES].min
+    end  
+  
+    # Returns the response time of the query, in milliseconds.
+    def time
+      require_run
+      response[:time]
+    end
+
+    # Returns whether the query has been run.  
+    def run?
+      !@response.blank?
+    end
+ 
+    # Returns the current page number of the result set. (Page indexes begin at 1.) 
+    def current_page
+      @options['page']
+    end
+  
+    # Returns the number of records per page.
+    def per_page
+      @options['per_page']
+    end
+        
+    # Returns the last available page number in the result set.  
+    def page_count
+      require_run    
+      (total_entries / per_page.to_f).ceil
+    end
+            
+    # Returns the previous page number.
+    def previous_page 
+      current_page > 1 ? (current_page - 1) : nil
+    end
+
+    # Returns the next page number.
+    def next_page
+      current_page < page_count ? (current_page + 1) : nil
+    end
+    
+    # Returns the global index position of the first result on this page.
+    def offset 
+      (current_page - 1) * per_page
+    end
+    
+    # Builds a new command-interface Search object.
+    def initialize opts = {} 
+
+      # Change to normal hashes with String keys for speed
+      opts = Hash[HashWithIndifferentAccess.new(opts._deep_dup._coerce_basic_types)]
+      unless self.class.query_defaults.instance_of? Hash
+        self.class.query_defaults = Hash[self.class.query_defaults]
+        self.class.query_defaults['location'] = Hash[self.class.query_defaults['location']]
+        
+        self.class.client_options = Hash[self.class.client_options]
+        self.class.excerpting_options = Hash[self.class.excerpting_options]
+        self.class.excerpting_options['content_methods'].map! {|ary| ary.map {|m| m.to_s}}
+      end    
+
+      # We need an annoying deep merge on the :location parameter
+      opts['location'].reverse_merge!(self.class.query_defaults['location']) if opts['location']
+
+      # Merge the rest of the defaults      
+      @options = self.class.query_defaults.merge(opts)
+      
+      @options['query'] = @options['query'].to_s
+      @options['class_names'] = Array(@options['class_names'])
+      @options['facets'] = Array(@options['facets'])
+      @options['indexes'] = Array(@options['indexes']).join(" ")
+            
+      raise UsageError, "Weights must be a Hash" unless @options['weights'].is_a? Hash
+      raise UsageError, "Filters must be a Hash" unless @options['filters'].is_a? Hash
+      
+      @options['parsed_query'] = parse(query)
+  
+      @results, @subtotals, @facets, @response = [], {}, {}, {}
+        
+      extra_keys = @options.keys - (self.class.query_defaults.keys + INTERNAL_KEYS)
+      log "discarded invalid keys: #{extra_keys * ', '}" if extra_keys.any? and RAILS_ENV != "test" 
+    end
+    
+    # Run the search, filling results with an array of ActiveRecord objects. Set the parameter to false 
+    # if you only want the ids returned.
+    def run(reify = true)
+      @request = build_request_with_options(@options)
+
+      log "searching for #{@options.inspect}"
+
+      perform_action_with_retries do
+        @response = @request.query(parsed_query, @options['indexes'])
+        log "search returned #{total_entries}/#{response[:total_found].to_i} in #{time.to_f} seconds."
+          
+        if self.class.client_options['with_subtotals']        
+          @subtotals = get_subtotals(@request, parsed_query) 
+          
+          # If the original query has a filter on this class, we will use its more accurate total rather the facet's 
+          # less accurate total.
+          if @options['class_names'].size == 1
+            @subtotals[@options['class_names'].first] = response[:total_found]
+          end
+          
+        end
+        
+        Array(@options['facets']).each do |facet|
+          @facets[facet] = get_facets(@request, parsed_query, facet)
+        end        
+        
+        @results = convert_sphinx_ids(response[:matches])
+        @results = reify_results(@results) if reify
+        
+        say "warning; #{response[:warning]}" if response[:warning]
+        raise UsageError, response[:error] if response[:error]
+        
+      end      
+      self
+    end
+  
+  
+    # Overwrite the configured content attributes with excerpted and highlighted versions of themselves.
+    # Runs run if it hasn't already been done.
+    def excerpt
+    
+      require_run         
+      return if results.empty?
+    
+      # See what fields in each result might respond to our excerptable methods
+      results_with_content_methods = results.map do |result|
+        [result, 
+          self.class.excerpting_options['content_methods'].map do |methods|
+            methods.detect do |this| 
+              result.respond_to? this
+            end
+          end
+        ]
+      end
+  
+      # Fetch the actual field contents
+      docs = results_with_content_methods.map do |result, methods|
+        methods.map do |method| 
+          method and strip_bogus_characters(result.send(method)) or ""
+        end
+      end.flatten
+      
+      excerpting_options = {
+        :docs => docs,         
+        :index => MAIN_INDEX, # http://www.sphinxsearch.com/forum/view.html?id=100
+        :words => strip_query_commands(parsed_query)
+      }
+      self.class.excerpting_options.except('content_methods').each do |key, value|
+        # Riddle only wants symbols
+        excerpting_options[key.to_sym] ||= value
+      end
+      
+      responses = perform_action_with_retries do 
+        # Ship to Sphinx to highlight and excerpt
+        @request.excerpts(excerpting_options)
+      end
+      
+      responses = responses.in_groups_of(self.class.excerpting_options['content_methods'].size)
+      
+      results_with_content_methods.each_with_index do |result_and_methods, i|
+        # Override the individual model accessors with the excerpted data
+        result, methods = result_and_methods
+        methods.each_with_index do |method, j|
+          data = responses[i][j]
+          if method
+            result._metaclass.send('define_method', method) { data }
+            attributes = result.instance_variable_get('@attributes')
+            attributes[method] = data if attributes[method]
+          end
+        end
+      end
+  
+      @results = results_with_content_methods.map do |result_and_content_method| 
+        result_and_content_method.first.freeze
+      end
+      
+      self
+    end  
+    
+            
+    # Delegates enumerable methods to @results, if possible. This allows us to behave directly like a WillPaginate::Collection. Failing that, we delegate to the options hash if a key is set. This lets us use <tt>self</tt> directly in view helpers.
+    def method_missing(*args, &block)
+      if @results.respond_to? args.first
+        @results.send(*args, &block)
+      elsif options.has_key? args.first.to_s
+        @options[args.first.to_s]
+      else
+        super
+      end
+    end
+  
+    def log msg #:nodoc:
+      Ultrasphinx.log msg
+    end
+
+    def say msg #:nodoc:
+      Ultrasphinx.say msg
+    end
+    
+    private
+    
+    def require_run
+      run unless run?
+    end
+    
+  end
+end

data/lib/ultrasphinx/spell.rb

@@ -0,0 +1,57 @@
+
+
+module Ultrasphinx
+
+=begin rdoc
+
+In order to spellcheck your user's query, Ultrasphinx bundles a small spelling module. 
+
+== Setup
+
+Make sure Aspell and the Rubygem <tt>raspell</tt> are installed. See http://blog.evanweaver.com/files/doc/fauna/raspell/ for detailed instructions.
+  
+Copy the <tt>examples/ap.multi</tt> file into your Aspell dictionary folder (<tt>/opt/local/share/aspell/</tt> on Mac, <tt>/usr/lib/aspell-0.60/</tt> on Linux). This file lets Aspell load a custom wordlist generated by Sphinx from your app data (you can configure its filename in the <tt>config/ultrasphinx/*.base</tt> files). Modify the file if you don't want to also use the default American English dictionary.
+  
+Finally, to build the custom wordlist, run:  
+  sudo rake ultrasphinx:spelling:build  
+
+You need to use <tt>sudo</tt> because Ultrasphinx needs to write to the Aspell dictionary folder. Also note that Aspell, <tt>raspell</tt>, and the custom dictionary must be available on each application server, not on the Sphinx daemon server.
+
+
+== Usage
+
+Now you can see if a query is correctly spelled as so:
+  @correction = Ultrasphinx::Spell.correct(@search.query)
+
+If <tt>@correction</tt> is not <tt>nil</tt>, go ahead and suggest it to the user. 
+
+=end
+
+  module Spell  
+  
+    begin    
+      SP = Aspell.new(Ultrasphinx::DICTIONARY)   
+      SP.suggestion_mode = Aspell::NORMAL
+      SP.set_option("ignore-case", "true")
+      Ultrasphinx.say "spelling support enabled"
+    rescue Object => e      
+      SP = nil
+      Ultrasphinx.say "spelling support not available (raspell configuration raised \"#{e}\")"
+    end    
+    
+    def self.correct string
+      return nil unless SP
+      correction = string.gsub(/[\w\']+/) do |word| 
+        unless SP.check(word)
+          SP.suggest(word).first
+        else
+          word
+        end
+      end
+      
+      correction if correction != string
+    end    
+    
+  end    
+end
+