kuahyeow-sunspot 0.9.8 → 0.10.3

data/History.txt

@@ -1,3 +1,41 @@
+== 0.10.2 2009-10-09
+* Add Sunspot.config.xml_builder option, which is passed to RSolr
+
+== 0.10.1 2009-10-08
+* Fix directory bugs in sunspot-solr executable
+
+== 0.10.0 2009-10-08
+* Support for geographical search using LocalSolr
+* Support for keyword highlighting, with custom deferred formatting
+* New fulltext DSL for specifying advanced dismax options
+* Support boost queries
+* Support for search-time field boost
+* Support for phrase fields
+* Support for prefix queries
+* Set default search-time field boost in setup
+* Restrict field facet to a set of desired values
+* Query facets support all facet options
+* Allow scoping by text fields
+* Support executing searches repeatedly
+* Allow setting of Solr URL for integration tests in environment variable
+* Add support for master/slave configurations
+* Added logging options to sunspot-solr executable
+* Added default solr config file location to Sunspot::Configuration
+* Informative, non-firehose Search#inspect
+* No longer require arguments to #paginate
+* Silently ignore keyword calls with nil/blank/whitespace keywords
+* Don't require that all searched types have a referenced field
+* Correct backwards ranges
+* Raise descriptive error if no types passed to search
+* Handle empty query facets, query facet rows, and connectives
+* Quote values in range restrictions if they contain spaces
+* Fix bug in Sunspot::Util.full_const_get
+* Remove support for :other option in time faceting
+* Remove order_by_random() method
+* Removed options and Query from public API
+* Use built-in optparse instead of optiflag in bin/sunspot-solr
+* Remove dependency on haml and use erb since it is only used to generate one file
+
 == 0.9.0 2009-07-21
 * Use Dismax parser for keyword search
 * Field and document boosting

data/README.rdoc

@@ -10,6 +10,11 @@ features for indexing objects and searching for them.
 Sunspot is designed to be easily plugged in to any ORM, or even non-database-backed
 objects such as the filesystem.
 
+This README is intended as a quick primer on what Sunspot is capable of; for
+detailed treatment of Sunspot's full feature range, check out the wiki:
+http://wiki.github.com/outoftime/sunspot
+
+
 === Features:
 
 * Define indexing strategy for each searchable class using intuitive block-based API
@@ -123,9 +128,7 @@ me so I can rectify the situation!
 
 1. RSolr
 2. Daemons
-3. OptiFlag
-4. Haml
-5. Java
+4. Java
 
 Sunspot has been tested with MRI 1.8.6 and 1.8.7, REE 1.8.6, YARV 1.9.1, and
 JRuby 1.2.0
@@ -135,6 +138,35 @@ JRuby 1.2.0
 Please submit bug reports to
 http://outoftime.lighthouseapp.com/projects/20339-sunspot
 
+== Contribution Guidelines
+
+Contributions are very welcome - both new features, enhancements, and bug fixes.
+Bug reports with a failing regression test are also lovely. In order to keep the
+contribution process as organized and smooth as possible, please follow these
+guidelines:
+
+* Contributions should be submitted via Sunspot's Lighthouse account, with an
+  attached git patch. See below for how to create a git patch.
+* Patches should not make any changes in your patch to the gemspec task other
+  than adding/removing dependencies (e.g., changing the name, version, email,
+  description, etc.)
+* Patches should not include any changes to the gemspec itself.
+* Document any new methods, options, arguments, etc.
+* Write tests.
+* As much as possible, follow the coding and testing styles you see in existing
+  code. One could accuse me of being nitpicky about this, but consistent code is
+  easier to read, maintain, and enhance.
+* Don't make any massive changes to the structure of library or test code. If
+  you think something needs a huge refactor or rearrangement, shoot me a
+  message; trying to apply that kind of patch without warning opens the door to
+  a world of conflict hurt.
+
+Here's how to create a Git patch - assuming you're pulling from the canonical
+Sunspot repository at `upstream`:
+
+  git fetch upstream
+  git format-patch upstre
+
 == Further Reading
 
 * Sunspot Discussion: http://groups.google.com/group/ruby-sunspot
@@ -148,6 +180,11 @@ http://outoftime.lighthouseapp.com/projects/20339-sunspot
 * Peer Allan (peer.allan@gmail.com)
 * Dmitriy Dzema (dima@dzema.name)
 * Benjamin Krause (bk@benjaminkrause.com)
+* Marcel de Graaf (marcel@slashdev.nl)
+* Brandon Keepers (brandon@opensoul.org)
+* Peter Berkenbosch (peterberkenbosch@me.com)
+* Brian Atkinson
+* Tom Coleman (tom@thesnail.org)
 
 == License
 

data/TODO

@@ -1,9 +1,11 @@
-=== 0.9.X ===
-* Deal with empty facet queries
-* Passing an integer into the second argument of dynamic_facet() when multiple facets are requested gives the wrong value
 === 0.10 ===
-* Highlighting
-* LocalSolr
-* Text field restrictions
-* Prefixes
-* Intelligently decide whether to instantiate all facet rows at once
+* Wrap everything into :q parameter when local search performed
+* Allow boosting without field constraints
+* Allow coordinates to be specified with block in setup
+=== 0.11 ===
+* Support all operations in batches. Make it smart.
+  * Don't use more than one commits when one is equivalent
+  * Preserve adds/deletes that are done after last commit
+  * Don't do adds and deletes for the same document out of order
+  * Don't do more than one add for the same document
+  * Do use as few requests as possible within those constraints

data/VERSION.yml

@@ -1,4 +1,4 @@
 --- 
-:patch: 8
+:minor: 10
+:patch: 3
 :major: 0
-:minor: 9

data/bin/sunspot-configure-solr

@@ -1,39 +1,33 @@
 #!/usr/bin/env ruby
 
-using_gems = false
-begin
-  require 'fileutils'
-  require 'optiflag'
-  require File.join(File.dirname(__FILE__), '..', 'lib', 'sunspot', 'schema')
-rescue LoadError => e
-  if using_gems
-    raise(e)
-  else
-    using_gems = true
-    require 'rubygems'
-    retry
+require 'fileutils'
+require 'optparse'
+require File.join(File.dirname(__FILE__), '..', 'lib', 'sunspot', 'schema')
+
+solr_directory = FileUtils.pwd
+
+schema = Sunspot::Schema.new
+
+OptionParser.new do |opts|
+  opts.banner = 'Usage: sunspot-configure-solr [options]'
+
+  opts.on '--tokenizer=TOKENIZER', 'Tokenizer class to use' do |tokenizer|
+    schema.tokenizer = tokenizer
   end
-end
 
-module ConfigureSolrFlags extend OptiFlagSet
-  optional_flag 'tokenizer'
-  optional_flag 'extra_filters'
-  optional_flag 'dir'
-  and_process!
-end
+  opts.on '--extra-filter=EXTRA_FILTER', 'Extra filters for tokenized text (you may specify this multiple times)' do |extra_filter|
+    schema.add_filter(extra_filter)
+  end
+
+  opts.on '--dir=CONF_DIR', 'Directory containing Solr configuration and schema' do |dir|
+    solr_directory = File.expand_path(dir)
+  end
+end.parse!
 
-solr_directory = ARGV.flags.dir || FileUtils.pwd
 conf_directory = File.join(solr_directory, 'conf')
 schema_file = File.join(conf_directory, 'schema.xml')
-FileUtils.mkdir_p(conf_directory)
 
-schema = Sunspot::Schema.new
-schema.tokenizer = ARGV.flags.tokenizer if ARGV.flags.tokenizer
-if ARGV.flags.extra_filters
-  for filter in ARGV.flags.extra_filters.split(',')
-    schema.add_filter(filter)
-  end
-end
+FileUtils.mkdir_p(conf_directory)
 
 if File.exist?(schema_file)
   backup_file = File.join(conf_directory, "schema-#{File.mtime(schema_file).strftime('%Y%m%d%H%M%S')}.xml")

data/bin/sunspot-solr

@@ -2,9 +2,10 @@
 using_gems = false
 begin
   require 'fileutils'
+  require 'tempfile'
   require 'tmpdir'
+  require 'optparse'
   require 'daemons'
-  require 'optiflag'
 rescue LoadError => e
   if using_gems
     raise(e)
@@ -16,47 +17,67 @@ rescue LoadError => e
 end
 
 working_directory = FileUtils.pwd
-solr_home = File.join(File.dirname(__FILE__), '..', 'solr')
+solr_install = File.expand_path(File.join(File.dirname(__FILE__), '..', 'solr'))
 
-module SolrFlags extend OptiFlagSet
-  optional_flag 'p' do
-    description 'Port on which to run Solr (default 8983)'
-    long_form 'port'
+port      = '8983'
+data_dir  = File.expand_path(File.join(Dir.tmpdir, 'solr_data'))
+solr_home = File.join(solr_install, 'solr')
+pid_dir   = working_directory
+log_file  = nil
+log_level = 'OFF'
+
+OptionParser.new do |opts|
+  opts.banner = "Usage: sunspot-solr start [options]"
+
+  opts.on '-p', '--port=PORT', 'Port on which to run Solr (default 8983)' do |p|
+    port = p
   end
 
-  optional_flag 'd' do
-    description 'Solr data directory'
+  opts.on '-d', '--data-directory=DIRECTORY', 'Solr data directory' do |d|
+    data_dir = File.expand_path(d)
   end
 
-  optional_flag 's' do
-    description 'Solr home (should contain conf/ directory)'
+  opts.on '-s', '--solr-home=HOME', 'Solr home directory (should contain conf/ directory)' do |s|
+    solr_home = File.expand_path(s)
   end
 
-  optional_flag 'pd' do
-    long_form 'pid-dir'
-    description 'Directory for pid files'
+  opts.on '--pid-dir=PID_DIR', 'Directory for pid files' do |pd|
+    pid_dir = File.expand_path(pd)
   end
 
-  and_process!
-end
+  opts.on '-l', '--log-level=LOG_LEVEL', 'Solr logging level' do |l|
+    log_level = l
+  end
 
-port     = ARGV.flags.p || '8983'
-data_dir = File.expand_path(ARGV.flags.d || File.join(Dir.tmpdir, 'solr_data'))
-home     = File.expand_path(ARGV.flags.s) if ARGV.flags.s
-pid_dir  = File.expand_path(ARGV.flags.pd || working_directory)
+  opts.on '--log-file=LOG_FILE', 'Path to Solr log file' do |lf|
+    log_file = File.expand_path(lf)
+  end
+end.parse!
 
 options = { :dir_mode => :normal, :dir => pid_dir }
 
+def logging_properties( log_file, log_level )
+  temp_file = Tempfile.new 'logging.properties'
+  temp_file << <<PROPERTIES
+.level = #{log_level}
+handlers = java.util.logging.FileHandler
+java.util.logging.FileHandler.formatter = java.util.logging.SimpleFormatter
+java.util.logging.FileHandler.pattern   = #{log_file}
+PROPERTIES
+  temp_file.flush
+  temp_file.close
+  temp_file.path
+end
+
 Daemons.run_proc('sunspot-solr', options) do
-  FileUtils.cd(working_directory) do
-    FileUtils.cd(solr_home) do
-      args = ['java']
-      args << "-Djetty.port=#{port}" if port
-      args << "-Dsolr.data.dir=#{data_dir}" if data_dir
-      args << "-Dsolr.solr.home=#{home}" if home
-      args << '-jar' << 'start.jar'
-      STDERR.puts(args * ' ')
-      Kernel.exec(*args)
-    end
+  FileUtils.cd(solr_install) do
+    args = ['java']
+    args << "-Djetty.port=#{port}" if port
+    args << "-Dsolr.data.dir=#{data_dir}" if data_dir
+    args << "-Dsolr.solr.home=#{solr_home}" if solr_home
+    args << "-Djava.util.logging.config.file=#{logging_properties(log_file, log_level)}" if log_file and log_level != 'OFF'
+    args << '-jar' << File.join(solr_install, 'start.jar')
+    STDERR.puts(args * ' ')
+    Kernel.exec(*args)
   end
 end

data/lib/sunspot.rb

@@ -9,10 +9,10 @@ end
 
 require File.join(File.dirname(__FILE__), 'light_config')
 
-%w(util adapters configuration setup composite_setup field field_factory
-   data_extractor indexer query search facet facet_row instantiated_facet
-   instantiated_facet_row date_facet date_facet_row query_facet query_facet_row
-   session type dsl).each do |filename|
+%w(util adapters configuration setup composite_setup text_field_setup field
+   field_factory data_extractor indexer query search facet facet_row
+   instantiated_facet instantiated_facet_row facet_data session type
+   dsl).each do |filename|
   require File.join(File.dirname(__FILE__), 'sunspot', filename)
 end
 
@@ -36,6 +36,7 @@ module Sunspot
   UnrecognizedRestrictionError = Class.new(Exception)
   NoAdapterError = Class.new(Exception)
   NoSetupError = Class.new(Exception)
+  IllegalSearchError = Class.new(Exception)
 
   class <<self
     # Configures indexing and search for a given class.
@@ -222,18 +223,6 @@ module Sunspot
     #   Zero, one, or more types to search for. If no types are passed, all
     #   configured types will be searched.
     #
-    # ==== Options (last argument, optional)
-    #
-    # :keywords<String>:: Fulltext search string
-    # :conditions<Hash>::
-    #   Hash of key-value pairs to be used as restrictions. Keys are field
-    #   names. Scalar values are used as equality restrictions; arrays are used
-    #   as "any of" restrictions; and Ranges are used as range restrictions.
-    # :order<String>:: order field and direction (e.g., 'updated_at desc')
-    # :page<Integer>:: Page to start on for pagination
-    # :per_page<Integer>::
-    #   Number of results to use per page. Ignored if :page is not specified.
-    #
     # ==== Returns
     #
     # Sunspot::Search:: Object containing results, facets, count, etc.
@@ -242,9 +231,10 @@ module Sunspot
     # the following criteria:
     #
     # * They are not of type +text+.
-    # * They are defined for all of the classes being searched
-    # * They have the same data type for all of the classes being searched
+    # * They are defined for at least one of the classes being searched
+    # * They have the same data type for all of the classes being searched.
     # * They have the same multiple flag for all of the classes being searched.
+    # * They have the same stored flag for all of the classes being searched.
     #
     # The restrictions available are the constants defined in the
     # Sunspot::Restriction class. The standard restrictions are:

data/lib/sunspot/adapters.rb

@@ -127,7 +127,7 @@ module Sunspot
                 "No adapter is configured for #{original_class_name} or its superclasses. See the documentation for Sunspot::Adapters")
         end
 
-        def index_id_for(class_name, id)
+        def index_id_for(class_name, id) #:nodoc:
           "#{class_name} #{id}"
         end
 

data/lib/sunspot/composite_setup.rb

@@ -48,7 +48,7 @@ module Sunspot
     # UnrecognizedFieldError::
     #   If no field with that name is configured for any of the enclosed types.
     #
-    def text_field(field_name)
+    def text_fields(field_name)
       text_fields_hash[field_name.to_sym] || raise(
         UnrecognizedFieldError,
         "No text field configured for #{@types * ', '} with name '#{field_name}'"
@@ -104,8 +104,8 @@ module Sunspot
     #
     # Array:: Text fields configured for the enclosed types
     #
-    def text_fields
-      @text_fields ||= text_fields_hash.values
+    def all_text_fields
+      @text_fields ||= text_fields_hash.values.map { |set| set.to_a }.flatten
     end
 
     private
@@ -121,8 +121,8 @@ module Sunspot
     def text_fields_hash
       @text_fields_hash ||=
         setups.inject({}) do |hash, setup|
-          setup.text_fields.each do |text_field|
-            hash[text_field.name] ||= text_field
+          setup.all_text_fields.each do |text_field|
+            (hash[text_field.name] ||= Set.new) << text_field
           end
           hash
         end
@@ -141,21 +141,19 @@ module Sunspot
     def fields_hash
       @fields_hash ||=
         begin
-          fields_hash = @types.inject({}) do |hash, type|
+          field_sets_hash = Hash.new { |h, k| h[k] = Set.new }
+          @types.each do |type|
             Setup.for(type).fields.each do |field|
-              (hash[field.name.to_sym] ||= {})[type.name] = field
+              field_sets_hash[field.name.to_sym] << field
             end
-            hash
           end
-          fields_hash.each_pair do |field_name, field_configurations_hash|
-            if @types.any? { |type| field_configurations_hash[type.name].nil? } # at least one type doesn't have this field configured
-              fields_hash.delete(field_name)
-            elsif field_configurations_hash.values.map { |configuration| configuration.indexed_name }.uniq.length != 1 # fields with this name have different configs
-              fields_hash.delete(field_name)
-            else
-              fields_hash[field_name] = field_configurations_hash.values.first
+          fields_hash = {}
+          field_sets_hash.each_pair do |field_name, set|
+            if set.length == 1
+              fields_hash[field_name] = set.to_a.first
             end
           end
+          fields_hash
         end
     end
 

data/lib/sunspot/configuration.rb

@@ -7,6 +7,11 @@ module Sunspot
   #   :net_http, which is the default and uses Ruby's built-in pure-Ruby HTTP
   #   library; and :curb, which uses Ruby's libcurl bindings and requires
   #   installation of the 'curb' gem.
+  # Sunspot.config.xml_builder::
+  #   The library use to build XML messages sent to Solr. As of this writing the
+  #   options are :builder and :libxml - the latter is faster but less portable,
+  #   as it is native code. Check the documentation for RSolr::Message::Adapter
+  #   for more information.
   # Sunspot.config.solr.url::
   #   The URL at which to connect to Solr
   #   (default: 'http://localhost:8983/solr')
@@ -25,14 +30,30 @@ module Sunspot
       def build #:nodoc:
         LightConfig.build do
           http_client :net_http
+          xml_builder :builder
+
           solr do
             url 'http://127.0.0.1:8983/solr'
           end
+          master_solr do
+            url nil
+          end
           pagination do
             default_per_page 30
           end
         end
       end
+      
+      # Location for the default solr configuration files,
+      # required for bootstrapping a new solr installation
+      #
+      # ==== Returns
+      #
+      # String:: Directory with default solr config files
+      #
+      def solr_default_configuration_location
+        File.join( File.dirname(__FILE__), '../../solr/solr/conf' )
+      end
     end
   end
 end