sunspot 0.9.8 → 0.10.0

data/History.txt

@@ -1,3 +1,35 @@
+== 0.10.0 2009-10-08
+* Support for geographical search using LocalSolr
+* Support for keyword highlighting, with custom deferred formatting
+* New fulltext DSL for specifying advanced dismax options
+* Support boost queries
+* Support for search-time field boost
+* Support for phrase fields
+* Support for prefix queries
+* Set default search-time field boost in setup
+* Restrict field facet to a set of desired values
+* Query facets support all facet options
+* Allow scoping by text fields
+* Support executing searches repeatedly
+* Allow setting of Solr URL for integration tests in environment variable
+* Add support for master/slave configurations
+* Added logging options to sunspot-solr executable
+* Added default solr config file location to Sunspot::Configuration
+* Informative, non-firehose Search#inspect
+* No longer require arguments to #paginate
+* Silently ignore keyword calls with nil/blank/whitespace keywords
+* Don't require that all searched types have a referenced field
+* Correct backwards ranges
+* Raise descriptive error if no types passed to search
+* Handle empty query facets, query facet rows, and connectives
+* Quote values in range restrictions if they contain spaces
+* Fix bug in Sunspot::Util.full_const_get
+* Remove support for :other option in time faceting
+* Remove order_by_random() method
+* Removed options and Query from public API
+* Use built-in optparse instead of optiflag in bin/sunspot-solr
+* Remove dependency on haml and use erb since it is only used to generate one file
+
 == 0.9.0 2009-07-21
 * Use Dismax parser for keyword search
 * Field and document boosting

data/README.rdoc

@@ -10,6 +10,11 @@ features for indexing objects and searching for them.
 Sunspot is designed to be easily plugged in to any ORM, or even non-database-backed
 objects such as the filesystem.
 
+This README is intended as a quick primer on what Sunspot is capable of; for
+detailed treatment of Sunspot's full feature range, check out the wiki:
+http://wiki.github.com/outoftime/sunspot
+
+
 === Features:
 
 * Define indexing strategy for each searchable class using intuitive block-based API
@@ -123,9 +128,7 @@ me so I can rectify the situation!
 
 1. RSolr
 2. Daemons
-3. OptiFlag
-4. Haml
-5. Java
+4. Java
 
 Sunspot has been tested with MRI 1.8.6 and 1.8.7, REE 1.8.6, YARV 1.9.1, and
 JRuby 1.2.0
@@ -135,6 +138,35 @@ JRuby 1.2.0
 Please submit bug reports to
 http://outoftime.lighthouseapp.com/projects/20339-sunspot
 
+== Contribution Guidelines
+
+Contributions are very welcome - both new features, enhancements, and bug fixes.
+Bug reports with a failing regression test are also lovely. In order to keep the
+contribution process as organized and smooth as possible, please follow these
+guidelines:
+
+* Contributions should be submitted via Sunspot's Lighthouse account, with an
+  attached git patch. See below for how to create a git patch.
+* Patches should not make any changes in your patch to the gemspec task other
+  than adding/removing dependencies (e.g., changing the name, version, email,
+  description, etc.)
+* Patches should not include any changes to the gemspec itself.
+* Document any new methods, options, arguments, etc.
+* Write tests.
+* As much as possible, follow the coding and testing styles you see in existing
+  code. One could accuse me of being nitpicky about this, but consistent code is
+  easier to read, maintain, and enhance.
+* Don't make any massive changes to the structure of library or test code. If
+  you think something needs a huge refactor or rearrangement, shoot me a
+  message; trying to apply that kind of patch without warning opens the door to
+  a world of conflict hurt.
+
+Here's how to create a Git patch - assuming you're pulling from the canonical
+Sunspot repository at `upstream`:
+
+  git fetch upstream
+  git format-patch upstre
+
 == Further Reading
 
 * Sunspot Discussion: http://groups.google.com/group/ruby-sunspot
@@ -148,6 +180,11 @@ http://outoftime.lighthouseapp.com/projects/20339-sunspot
 * Peer Allan (peer.allan@gmail.com)
 * Dmitriy Dzema (dima@dzema.name)
 * Benjamin Krause (bk@benjaminkrause.com)
+* Marcel de Graaf (marcel@slashdev.nl)
+* Brandon Keepers (brandon@opensoul.org)
+* Peter Berkenbosch (peterberkenbosch@me.com)
+* Brian Atkinson
+* Tom Coleman (tom@thesnail.org)
 
 == License
 

data/TODO

@@ -1,9 +1,11 @@
-=== 0.9.X ===
-* Deal with empty facet queries
-* Passing an integer into the second argument of dynamic_facet() when multiple facets are requested gives the wrong value
 === 0.10 ===
-* Highlighting
-* LocalSolr
-* Text field restrictions
-* Prefixes
-* Intelligently decide whether to instantiate all facet rows at once
+* Wrap everything into :q parameter when local search performed
+* Allow boosting without field constraints
+* Allow coordinates to be specified with block in setup
+=== 0.11 ===
+* Support all operations in batches. Make it smart.
+  * Don't use more than one commits when one is equivalent
+  * Preserve adds/deletes that are done after last commit
+  * Don't do adds and deletes for the same document out of order
+  * Don't do more than one add for the same document
+  * Do use as few requests as possible within those constraints

data/VERSION.yml

@@ -1,4 +1,4 @@
 --- 
-:minor: 9
-:patch: 8
+:minor: 10
+:patch: 0
 :major: 0

data/bin/sunspot-configure-solr

@@ -1,39 +1,33 @@
 #!/usr/bin/env ruby
 
-using_gems = false
-begin
-  require 'fileutils'
-  require 'optiflag'
-  require File.join(File.dirname(__FILE__), '..', 'lib', 'sunspot', 'schema')
-rescue LoadError => e
-  if using_gems
-    raise(e)
-  else
-    using_gems = true
-    require 'rubygems'
-    retry
+require 'fileutils'
+require 'optparse'
+require File.join(File.dirname(__FILE__), '..', 'lib', 'sunspot', 'schema')
+
+solr_directory = FileUtils.pwd
+
+schema = Sunspot::Schema.new
+
+OptionParser.new do |opts|
+  opts.banner = 'Usage: sunspot-configure-solr [options]'
+
+  opts.on '--tokenizer=TOKENIZER', 'Tokenizer class to use' do |tokenizer|
+    schema.tokenizer = tokenizer
   end
-end
 
-module ConfigureSolrFlags extend OptiFlagSet
-  optional_flag 'tokenizer'
-  optional_flag 'extra_filters'
-  optional_flag 'dir'
-  and_process!
-end
+  opts.on '--extra-filter=EXTRA_FILTER', 'Extra filters for tokenized text (you may specify this multiple times)' do |extra_filter|
+    schema.add_filter(extra_filter)
+  end
+
+  opts.on '--dir=CONF_DIR', 'Directory containing Solr configuration and schema' do |dir|
+    solr_directory = File.expand_path(dir)
+  end
+end.parse!
 
-solr_directory = ARGV.flags.dir || FileUtils.pwd
 conf_directory = File.join(solr_directory, 'conf')
 schema_file = File.join(conf_directory, 'schema.xml')
-FileUtils.mkdir_p(conf_directory)
 
-schema = Sunspot::Schema.new
-schema.tokenizer = ARGV.flags.tokenizer if ARGV.flags.tokenizer
-if ARGV.flags.extra_filters
-  for filter in ARGV.flags.extra_filters.split(',')
-    schema.add_filter(filter)
-  end
-end
+FileUtils.mkdir_p(conf_directory)
 
 if File.exist?(schema_file)
   backup_file = File.join(conf_directory, "schema-#{File.mtime(schema_file).strftime('%Y%m%d%H%M%S')}.xml")

data/bin/sunspot-solr

@@ -2,9 +2,10 @@
 using_gems = false
 begin
   require 'fileutils'
+  require 'tempfile'
   require 'tmpdir'
+  require 'optparse'
   require 'daemons'
-  require 'optiflag'
 rescue LoadError => e
   if using_gems
     raise(e)
@@ -16,47 +17,67 @@ rescue LoadError => e
 end
 
 working_directory = FileUtils.pwd
-solr_home = File.join(File.dirname(__FILE__), '..', 'solr')
+solr_home = File.expand_path(File.join(File.dirname(__FILE__), '..', 'solr'))
 
-module SolrFlags extend OptiFlagSet
-  optional_flag 'p' do
-    description 'Port on which to run Solr (default 8983)'
-    long_form 'port'
+port      = '8983'
+data_dir  = File.expand_path(File.join(Dir.tmpdir, 'solr_data'))
+home      = nil
+pid_dir   = working_directory
+log_file  = nil
+log_level = 'OFF'
+
+OptionParser.new do |opts|
+  opts.banner = "Usage: sunspot-solr start [options]"
+
+  opts.on '-p', '--port=PORT', 'Port on which to run Solr (default 8983)' do |p|
+    port = p
   end
 
-  optional_flag 'd' do
-    description 'Solr data directory'
+  opts.on '-d', '--data-directory=DIRECTORY', 'Solr data directory' do |d|
+    data_dir = File.expand_path(d)
   end
 
-  optional_flag 's' do
-    description 'Solr home (should contain conf/ directory)'
+  opts.on '-s', '--solr-home=HOME', 'Solr home directory (should contain conf/ directory)' do |s|
+    solr_home = File.expand_path(s)
   end
 
-  optional_flag 'pd' do
-    long_form 'pid-dir'
-    description 'Directory for pid files'
+  opts.on '--pid-dir=PID_DIR', 'Directory for pid files' do |pd|
+    pid_dir = File.expand_path(pd)
   end
 
-  and_process!
-end
+  opts.on '-l', '--log-level=LOG_LEVEL', 'Solr logging level' do |l|
+    log_level = l
+  end
 
-port     = ARGV.flags.p || '8983'
-data_dir = File.expand_path(ARGV.flags.d || File.join(Dir.tmpdir, 'solr_data'))
-home     = File.expand_path(ARGV.flags.s) if ARGV.flags.s
-pid_dir  = File.expand_path(ARGV.flags.pd || working_directory)
+  opts.on '--log-file=LOG_FILE', 'Path to Solr log file' do |lf|
+    log_file = File.expand_path(lf)
+  end
+end.parse!
 
 options = { :dir_mode => :normal, :dir => pid_dir }
 
+def logging_properties( log_file, log_level )
+  temp_file = Tempfile.new 'logging.properties'
+  temp_file << <<PROPERTIES
+.level = #{log_level}
+handlers = java.util.logging.FileHandler
+java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
+java.util.logging.FileHandler.pattern   = #{log_file}
+PROPERTIES
+  temp_file.flush
+  temp_file.close
+  temp_file.path
+end
+
 Daemons.run_proc('sunspot-solr', options) do
-  FileUtils.cd(working_directory) do
-    FileUtils.cd(solr_home) do
-      args = ['java']
-      args << "-Djetty.port=#{port}" if port
-      args << "-Dsolr.data.dir=#{data_dir}" if data_dir
-      args << "-Dsolr.solr.home=#{home}" if home
-      args << '-jar' << 'start.jar'
-      STDERR.puts(args * ' ')
-      Kernel.exec(*args)
-    end
+  FileUtils.cd(solr_home) do
+    args = ['java']
+    args << "-Djetty.port=#{port}" if port
+    args << "-Dsolr.data.dir=#{data_dir}" if data_dir
+    args << "-Dsolr.solr.home=#{home}" if home
+    args << "-Djava.util.logging.config.file=#{logging_properties(log_file, log_level)}" if log_file and log_level != 'OFF'
+    args << '-jar' << 'start.jar'
+    STDERR.puts(args * ' ')
+    Kernel.exec(*args)
   end
 end

data/lib/sunspot/adapters.rb

@@ -127,7 +127,7 @@ module Sunspot
                 "No adapter is configured for #{original_class_name} or its superclasses. See the documentation for Sunspot::Adapters")
         end
 
-        def index_id_for(class_name, id)
+        def index_id_for(class_name, id) #:nodoc:
           "#{class_name} #{id}"
         end
 

data/lib/sunspot/composite_setup.rb

@@ -48,7 +48,7 @@ module Sunspot
     # UnrecognizedFieldError::
     #   If no field with that name is configured for any of the enclosed types.
     #
-    def text_field(field_name)
+    def text_fields(field_name)
       text_fields_hash[field_name.to_sym] || raise(
         UnrecognizedFieldError,
         "No text field configured for #{@types * ', '} with name '#{field_name}'"
@@ -104,8 +104,8 @@ module Sunspot
     #
     # Array:: Text fields configured for the enclosed types
     #
-    def text_fields
-      @text_fields ||= text_fields_hash.values
+    def all_text_fields
+      @text_fields ||= text_fields_hash.values.map { |set| set.to_a }.flatten
     end
 
     private
@@ -121,8 +121,8 @@ module Sunspot
     def text_fields_hash
       @text_fields_hash ||=
         setups.inject({}) do |hash, setup|
-          setup.text_fields.each do |text_field|
-            hash[text_field.name] ||= text_field
+          setup.all_text_fields.each do |text_field|
+            (hash[text_field.name] ||= Set.new) << text_field
           end
           hash
         end
@@ -141,21 +141,19 @@ module Sunspot
     def fields_hash
       @fields_hash ||=
         begin
-          fields_hash = @types.inject({}) do |hash, type|
+          field_sets_hash = Hash.new { |h, k| h[k] = Set.new }
+          @types.each do |type|
             Setup.for(type).fields.each do |field|
-              (hash[field.name.to_sym] ||= {})[type.name] = field
+              field_sets_hash[field.name.to_sym] << field
             end
-            hash
           end
-          fields_hash.each_pair do |field_name, field_configurations_hash|
-            if @types.any? { |type| field_configurations_hash[type.name].nil? } # at least one type doesn't have this field configured
-              fields_hash.delete(field_name)
-            elsif field_configurations_hash.values.map { |configuration| configuration.indexed_name }.uniq.length != 1 # fields with this name have different configs
-              fields_hash.delete(field_name)
-            else
-              fields_hash[field_name] = field_configurations_hash.values.first
+          fields_hash = {}
+          field_sets_hash.each_pair do |field_name, set|
+            if set.length == 1
+              fields_hash[field_name] = set.to_a.first
             end
           end
+          fields_hash
         end
     end
 

data/lib/sunspot/configuration.rb

@@ -28,11 +28,25 @@ module Sunspot
           solr do
             url 'http://127.0.0.1:8983/solr'
           end
+          master_solr do
+            url nil
+          end
           pagination do
             default_per_page 30
           end
         end
       end
+      
+      # Location for the default solr configuration files,
+      # required for bootstrapping a new solr installation
+      #
+      # ==== Returns
+      #
+      # String:: Directory with default solr config files
+      #
+      def solr_default_configuration_location
+        File.join( File.dirname(__FILE__), '../../solr/solr/conf' )
+      end
     end
   end
 end

data/lib/sunspot/data_extractor.rb

@@ -34,6 +34,9 @@ module Sunspot
       end
     end
 
+    # 
+    # Constant data extractors simply return the same value for every object.
+    #
     class Constant
       def initialize(value)
         @value = value

data/lib/sunspot/dsl/field_query.rb

@@ -6,6 +6,11 @@ module Sunspot
     # query DSL.
     #
     class FieldQuery < Scope
+      def initialize(query, setup)
+        @query = query
+        super(query.scope, setup)
+      end
+
       # Specify the order that results should be returned in. This method can
       # be called multiple times; precedence will be in the order given.
       #
@@ -15,15 +20,22 @@ module Sunspot
       # direction<Symbol>:: :asc or :desc (default :asc)
       #
       def order_by(field_name, direction = nil)
-        @query.order_by(field_name, direction)
+        sort =
+          if special = Sunspot::Query::Sort.special(field_name)
+            special.new(direction)
+          else
+            Sunspot::Query::Sort::FieldSort.new(
+              @setup.field(field_name), direction
+            )
+          end
+        @query.add_sort(sort)
       end
 
       # 
-      # Order results randomly. This will (generally) return the results in a
-      # different order each time a search is called.
+      # DEPRECATED Use <code>order_by(:random)</code>
       #
       def order_by_random
-        @query.order_by_random
+        order_by(:random)
       end
 
       # Request facets on the given field names. If the last argument is a hash,
@@ -49,6 +61,12 @@ module Sunspot
       #
       def facet(*field_names, &block)
         if block
+          options =
+            if field_names.last.is_a?(Hash)
+              field_names.pop
+            else
+              {}
+            end
           if field_names.length != 1
             raise(
               ArgumentError,
@@ -56,17 +74,26 @@ module Sunspot
             )
           end
           name = field_names.first
-          DSL::QueryFacet.new(@query.add_query_facet(name)).instance_eval(&block)
+          DSL::QueryFacet.new(@query.add_query_facet(name, options), @setup).instance_eval(&block)
         else
           options = 
             if field_names.last.is_a?(Hash)
               field_names.pop
+            else
+              {}
             end
           for field_name in field_names
-            @query.add_field_facet(field_name, options)
+            @query.add_field_facet(@setup.field(field_name), options)
           end
         end
       end
+
+      def dynamic(base_name, &block)
+        Sunspot::Util.instance_eval_or_call(
+          FieldQuery.new(@query, @setup.dynamic_field_factory(base_name)),
+          &block
+        )
+      end
     end
   end
 end

data/lib/sunspot/dsl/fields.rb

@@ -26,7 +26,11 @@ module Sunspot
       # ==== Options
       #
       # :boost<Float>::
-      #   Boost that should be applied to this field for keyword search
+      #   Index-time boost that should be applied to this field for keyword search
+      # :default_boost<Float>::
+      #   Default search-time boost to apply to this field during keyword
+      #   search. Can be overriden with DSL::Fulltext#fields or
+      #   DSL::Fulltext#boost_fields method.
       #
       def text(*names, &block)
         options = names.pop if names.last.is_a?(Hash)
@@ -39,6 +43,15 @@ module Sunspot
         end
       end
 
+      # 
+      # Specify a method that returns the geographical coordinates associated
+      # with the document. The object returned must respond to #first and #last
+      # (e.g., a two-element Array); or to #lat and one of #lng, #lon, or #long
+      #
+      def coordinates(name)
+        @setup.set_coordinates_field(name)
+      end
+
       # 
       # Specify a document-level boost. As with fields, you have the option of
       # passing an attribute name which will be called on each model, or a block