kuahyeow-sunspot 0.9.7

data/History.txt

@@ -0,0 +1,83 @@
+== 0.9.0 2009-07-21
+* Use Dismax parser for keyword search
+* Field and document boosting
+* Specify which fields to search in keyword search
+* Allow indexing of multiple values in text fields
+* Access keyword relevance score in Hit objects
+* Allow stored fields, retrieve stored values from Hit objects
+* Support more values in shorthand restrictions
+* Disjunctions and conjunctions
+* Random ordering
+* Control all options for field facets
+* Time range facets
+* Get referenced objects from facets on foreign keys
+* Facet by class
+* Batch indexing
+* New Date field type
+* Direct access to data accessors
+* Executable to configure production Solr instances
+* Replace solr-ruby with RSolr
+* Remove accidental ActiveSupport dependency
+
+== 0.8.9 2009-06-23
+* Fix OrderedHash bug in older versions of ActiveSupport
+
+== 0.8.8 2009-06-15
+* Escape type names to support namespaced classes
+* Fix bug with anonymous modules in Ruby 1.9
+
+== 0.8.7 2009-06-10
+* Add --pid-dir option for sunspot-solr executable
+
+== 0.8.5 2009-06-09
+* Added dependencies for sunspot-solr executable to gem dependencies
+* Search for adapters using class ancestors rather than superclasses
+
+== 0.8.3 2009-06-03
+* Index objects passed as a collection in a single HTTP request
+
+== 0.8.2 2009-05-27
+* Allow specification of Solr home when using sunspot-solr
+
+== 0.8.1 2009-05-26
+* Add Search#execute! to public API
+
+== 0.8.0 2009-05-22
+* Access query API directly; instantiate search without running it
+* Dynamic fields
+* Search blocks can be evaluated in calling context
+
+== 0.7.3 2009-05-06
+* Better exception handling when class doesn't have adapter/setup
+
+== 0.7.2 2009-04-29
+* Dirty sessions
+
+== 0.7.1 2009-04-29
+* Removed extlib dependency from gemspec
+
+== 0.7.0 2009-04-28
+* Less magic in the DSL
+* Restrict by empty values
+* Negative scoping using without() method
+* Exclusion by object identity using without(instance)
+* Support for faceting
+* Explicit commits
+* Boolean field type
+* Attribute field flexibility
+* Virtual field blocks can be evaluated in calling context
+* Order available by multiple fields
+* New adapter API
+* Got rid of builder API
+* Full documentation
+
+== 0.0.2 2009-02-14
+* Run sunspot's built-in Solr instance using
+  sunspot-solr executable
+* Search hash interpretation delegated to
+  Builder object
+
+== 0.0.1 2008-12-11
+* Initial release
+* Define indexing for any class using DSL
+* Search indexed classes using DSL

data/LICENSE

@@ -0,0 +1,18 @@
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,154 @@
+= Sunspot
+
+http://outoftime.github.com/sunspot
+
+Sunspot is a Ruby library for expressive, powerful interaction with the Solr search engine.
+Sunspot is built on top of the RSolr gem, which provides a low-level interface for Solr
+interaction; Sunspot provides a simple, intuitive, expressive DSL backed by powerful
+features for indexing objects and searching for them.
+
+Sunspot is designed to be easily plugged in to any ORM, or even non-database-backed
+objects such as the filesystem.
+
+=== Features:
+
+* Define indexing strategy for each searchable class using intuitive block-based API
+* Clean separation between keyword-searchable fields and fields for scoping/ordering
+* Define fields based on existing attributes or "virtual fields" for custom indexing
+* Indexes each object's entire superclass hierarchy, for easy searching for all objects inheriting from a parent class
+* Intuitive DSL for scoping searches, with all the usual boolean operators available
+* Intuitive interface for requesting facets on indexed fields
+* Extensible adapter architecture for easy integration of other ORMs or non-model classes
+* Refine search using field facets, date range facets, or ultra-powerful query facets
+* Full compatibility with will_paginate
+* Ordering
+
+== Installation
+
+  gem sources -a http://gems.github.com
+  gem install outoftime-sunspot
+
+In order to start the packaged Solr installation, run:
+
+  sunspot-solr start -- [-d /path/to/data/directory] [-p port] [-s path/to/solr/home] [--pid-dir=path/to/pid/dir]
+
+If you don't specify a data directory, your Solr index will be stored in your operating system's temporary directory.
+
+If you specify a solr home, the directory must contain a <code>conf</code>
+directory, which should contain at least <code>schema.xml</code> and
+<code>solrconfig.xml</code>. Be sure to copy the <code>schema.xml</code> out of
+the Sunspot gem's <code>solr/solr/conf</code> directory. Sunspot relies on the
+field name patterns defined in the packaged <code>schema.xml</code>, so those
+cannot be modified.
+
+You can also run your own instance of Solr wherever you'd like; just copy the solr/config/schema.xml file out of the gem's solr into your installation.
+You can change the URL at which Sunspot accesses Solr with:
+
+  Sunspot.config.solr.url = 'http://solr.my.host:9818/solr'
+
+== Rails Integration
+
+The {Sunspot::Rails}[http://github.com/outoftime/sunspot_rails] plugin makes
+integrating Sunspot into Rails drop-in easy.
+
+== Using Sunspot
+
+=== Define an index:
+
+  class Post
+    #...
+  end
+
+  Sunspot.setup(Post) do
+    text :title, :body
+    string :author_name
+    integer :blog_id
+    integer :category_ids
+    float :average_rating, :using => :ratings_average
+    time :published_at
+    string :sort_title do
+      title.downcase.sub(/^(an?|the)\W+/, ''/) if title = self.title
+    end
+  end
+
+See Sunspot.setup for more information.
+
+Note that in order for a class to be searchable, it must have an adapter
+registered for itself or one of its subclasses. Adapters allow Sunspot to load
+objects out of persistent storage, and to determine their primary key for
+indexing. {Sunspot::Rails}[http://github.com/outoftime/sunspot_rails] comes with
+an adapter for ActiveRecord objects, but for other types of models you will need
+to define your own. See Sunspot::Adapters for more information.
+
+=== Search for objects:
+  
+  search = Sunspot.search Post do
+    keywords 'great pizza'
+    with :author_name, 'Mark Twain'
+    with(:blog_id).any_of [2, 14]
+    with(:category_ids).all_of [4, 10]
+    with(:published_at).less_than Time.now
+    any_of do
+      with(:expired_at).greater_than(Time.now)
+      with(:expired_at, nil)
+    end
+    without :title, 'Bad Title'
+    without bad_instance # specifically exclude this instance from results
+
+    paginate :page => 3, :per_page => 15
+    order_by :average_rating, :desc
+
+    facet :blog_id
+  end
+
+See Sunspot.search for more information.
+
+=== Get data from search:
+
+  search.results
+  search.total
+  search.page
+  search.per_page
+  search.facet(:blog_id)
+
+== About the API documentation
+
+All of the methods documented in the RDoc are considered part of Sunspot's
+public API. Methods that are not part of the public API are documented in the
+code, but excluded from the RDoc. If you find yourself needing to access methods
+that are not part of the public API in order to do what you need, please contact
+me so I can rectify the situation!
+
+== Dependencies
+
+1. RSolr
+2. Daemons
+3. OptiFlag
+4. Haml
+5. Java
+
+Sunspot has been tested with MRI 1.8.6 and 1.8.7, REE 1.8.6, YARV 1.9.1, and
+JRuby 1.2.0
+
+== Bugs
+
+Please submit bug reports to
+http://outoftime.lighthouseapp.com/projects/20339-sunspot
+
+== Further Reading
+
+* Sunspot Discussion: http://groups.google.com/group/ruby-sunspot
+* IRC: #sunspot-ruby @ Freenode
+* Posts about Sunspot from my tumblog: http://outofti.me/tagged/sunspot
+* Read about it on Linux Magazine: http://www.linux-mag.com/id/7341
+
+== Contributors
+
+* Mat Brown (mat@patch.com)
+* Peer Allan (peer.allan@gmail.com)
+* Dmitriy Dzema (dima@dzema.name)
+* Benjamin Krause (bk@benjaminkrause.com)
+
+== License
+
+Sunspot is distributed under the MIT License, copyright (c) 2008-2009 Mat Brown

data/Rakefile

@@ -0,0 +1,9 @@
+ENV['RUBYOPT'] = '-W1'
+
+task :environment do
+  require File.dirname(__FILE__) + '/lib/sunspot'
+end
+
+Dir['tasks/**/*.rake'].each { |t| load t }
+
+task :default => 'spec:api'

data/TODO

@@ -0,0 +1,9 @@
+=== 0.9.X ===
+* Deal with empty facet queries
+* Passing an integer into the second argument of dynamic_facet() when multiple facets are requested gives the wrong value
+=== 0.10 ===
+* Highlighting
+* LocalSolr
+* Text field restrictions
+* Prefixes
+* Intelligently decide whether to instantiate all facet rows at once

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:patch: 7
+:major: 0
+:minor: 9

data/bin/sunspot-configure-solr

@@ -0,0 +1,46 @@
+#!/usr/bin/env ruby
+
+using_gems = false
+begin
+  require 'fileutils'
+  require 'optiflag'
+  require File.join(File.dirname(__FILE__), '..', 'lib', 'sunspot', 'schema')
+rescue LoadError => e
+  if using_gems
+    raise(e)
+  else
+    using_gems = true
+    require 'rubygems'
+    retry
+  end
+end
+
+module ConfigureSolrFlags extend OptiFlagSet
+  optional_flag 'tokenizer'
+  optional_flag 'extra_filters'
+  optional_flag 'dir'
+  and_process!
+end
+
+solr_directory = ARGV.flags.dir || FileUtils.pwd
+conf_directory = File.join(solr_directory, 'conf')
+schema_file = File.join(conf_directory, 'schema.xml')
+FileUtils.mkdir_p(conf_directory)
+
+schema = Sunspot::Schema.new
+schema.tokenizer = ARGV.flags.tokenizer if ARGV.flags.tokenizer
+if ARGV.flags.extra_filters
+  for filter in ARGV.flags.extra_filters.split(',')
+    schema.add_filter(filter)
+  end
+end
+
+if File.exist?(schema_file)
+  backup_file = File.join(conf_directory, "schema-#{File.mtime(schema_file).strftime('%Y%m%d%H%M%S')}.xml")
+  STDERR.puts("Backing up current schema file to #{File.expand_path(backup_file)}")
+  FileUtils.mv(schema_file, backup_file)
+end
+
+File.open(File.join(conf_directory, 'schema.xml'), 'w') do |file|
+  file << schema.to_xml
+end

data/bin/sunspot-solr

@@ -0,0 +1,62 @@
+#!/usr/bin/env ruby
+using_gems = false
+begin
+  require 'fileutils'
+  require 'tmpdir'
+  require 'daemons'
+  require 'optiflag'
+rescue LoadError => e
+  if using_gems
+    raise(e)
+  else
+    using_gems = true
+    require 'rubygems'
+    retry
+  end
+end
+
+working_directory = FileUtils.pwd
+solr_home = File.join(File.dirname(__FILE__), '..', 'solr')
+
+module SolrFlags extend OptiFlagSet
+  optional_flag 'p' do
+    description 'Port on which to run Solr (default 8983)'
+    long_form 'port'
+  end
+
+  optional_flag 'd' do
+    description 'Solr data directory'
+  end
+
+  optional_flag 's' do
+    description 'Solr home (should contain conf/ directory)'
+  end
+
+  optional_flag 'pd' do
+    long_form 'pid-dir'
+    description 'Directory for pid files'
+  end
+
+  and_process!
+end
+
+port     = ARGV.flags.p || '8983'
+data_dir = File.expand_path(ARGV.flags.d || File.join(Dir.tmpdir, 'solr_data'))
+home     = File.expand_path(ARGV.flags.s) if ARGV.flags.s
+pid_dir  = File.expand_path(ARGV.flags.pd || working_directory)
+
+options = { :dir_mode => :normal, :dir => pid_dir }
+
+Daemons.run_proc('sunspot-solr', options) do
+  FileUtils.cd(working_directory) do
+    FileUtils.cd(solr_home) do
+      args = ['java']
+      args << "-Djetty.port=#{port}" if port
+      args << "-Dsolr.data.dir=#{data_dir}" if data_dir
+      args << "-Dsolr.solr.home=#{home}" if home
+      args << '-jar' << 'start.jar'
+      STDERR.puts(args * ' ')
+      Kernel.exec(*args)
+    end
+  end
+end

data/lib/light_config.rb

@@ -0,0 +1,40 @@
+module LightConfig
+  class Configuration
+    def initialize(&block)
+      @properties = {}
+      ::LightConfig::Builder.new(self).instance_eval(&block)
+      singleton = (class <<self; self; end)
+      @properties.keys.each do |property|
+        singleton.module_eval do
+          define_method property do
+            @properties[property]
+          end
+
+          define_method "#{property}=" do |value|
+            @properties[property] = value
+          end
+        end
+      end
+    end
+  end
+
+  class Builder
+    def initialize(configuration)
+      @configuration = configuration
+    end
+
+    def method_missing(method, *args, &block)
+      raise ArgumentError("wrong number of arguments(#{args.length} for 1)") unless args.length < 2
+      value = if block then ::LightConfig::Configuration.new(&block)
+              else args.first
+              end
+      @configuration.instance_variable_get(:@properties)[method] = value
+    end
+  end
+
+  class <<self
+    def build(&block)
+      LightConfig::Configuration.new(&block)
+    end
+  end
+end

data/lib/sunspot/adapters.rb

@@ -0,0 +1,265 @@
+module Sunspot
+  # 
+  # Sunspot works by saving references to the primary key (or natural ID) of
+  # each indexed object, and then retrieving the objects from persistent storage
+  # when their IDs are referenced in search results. In order for Sunspot to
+  # know what an object's primary key is, and how to retrieve objects from
+  # persistent storage given a primary key, an adapter must be registered for
+  # that object's class or one of its superclasses (for instance, an adapter
+  # registered for ActiveRecord::Base would be used for all ActiveRecord
+  # models).
+  #
+  # To provide Sunspot with this ability, adapters must have two roles:
+  #
+  # Data accessor::
+  #   A subclass of Sunspot::Adapters::DataAccessor, this object is instantiated
+  #   with a particular class and must respond to the #load() method, which
+  #   returns an object from persistent storage given that object's primary key.
+  #   It can also optionally implement the #load_all() method, which returns
+  #   a collection of objects given a collection of primary keys, if that can be
+  #   done more efficiently than calling #load() on each key.
+  # Instance adapter::
+  #   A subclass of Sunspot::Adapters::InstanceAdapter, this object is
+  #   instantiated with a particular instance. Its only job is to tell Sunspot
+  #   what the object's primary key is, by implementing the #id() method.
+  #
+  # Adapters are registered by registering their two components, telling Sunspot
+  # that they are available for one or more classes, and all of their
+  # subclasses. See Sunspot::Adapters::DataAccessor.register and
+  # Sunspot::Adapters::InstanceAdapter.register for the details.
+  #
+  # See spec/mocks/mock_adapter.rb for an example of how adapter classes should
+  # be implemented.
+  #
+  module Adapters
+    # Subclasses of the InstanceAdapter class should implement the #id method,
+    # which returns the primary key of the instance stored in the @instance
+    # variable. The primary key must be unique within the scope of the
+    # instance's class.
+    #
+    # ==== Example:
+    #
+    #   class FileAdapter < Sunspot::Adapters::InstanceAdapter
+    #     def id
+    #       File.expand_path(@instance.path)
+    #     end
+    #   end
+    #
+    #   # then in your initializer
+    #   Sunspot::Adapters::InstanceAdapter.register(MyAdapter, File)
+    #
+    class InstanceAdapter
+      def initialize(instance) #:nodoc:
+        @instance = instance
+      end
+
+      # 
+      # The universally-unique ID for this instance that will be stored in solr
+      #
+      # ==== Returns
+      #
+      # String:: ID for use in Solr
+      #
+      def index_id #:nodoc:
+        InstanceAdapter.index_id_for(@instance.class.name, id)
+      end
+
+      class <<self
+        # Instantiate an InstanceAdapter for the given object, searching for
+        # registered adapters for the object's class.
+        #
+        # ==== Parameters
+        #
+        # instance<Object>:: The instance to adapt
+        #
+        # ==== Returns
+        #
+        # InstanceAdapter::
+        #   An instance of an InstanceAdapter implementation that
+        #   wraps the given instance
+        #
+        def adapt(instance) #:nodoc:
+          self.for(instance.class).new(instance)
+        end
+
+        # Register an instance adapter for a set of classes. When searching for
+        # an adapter for a given instance, Sunspot starts with the instance's
+        # class, and then searches for registered adapters up the class's
+        # ancestor chain.
+        #
+        # ==== Parameters
+        #
+        # instance_adapter<Class>:: The instance adapter class to register
+        # classes...<Class>::
+        #   One or more classes that this instance adapter adapts
+        #
+        def register(instance_adapter, *classes)
+          for clazz in classes
+            instance_adapters[clazz.name.to_sym] = instance_adapter
+          end
+        end
+
+        # Find the best InstanceAdapter implementation that adapts the given
+        # class.  Starting with the class and then moving up the ancestor chain,
+        # looks for registered InstanceAdapter implementations.
+        #
+        # ==== Parameters
+        #
+        # clazz<Class>:: The class to find an InstanceAdapter for
+        #
+        # ==== Returns
+        #
+        # Class:: Subclass of InstanceAdapter, or nil if none found
+        #
+        # ==== Raises
+        #
+        # Sunspot::NoAdapterError:: If no adapter is registered for this class
+        #
+        def for(clazz) #:nodoc:
+          original_class_name = clazz.name
+          clazz.ancestors.each do |ancestor_class|
+            next if ancestor_class.name.nil? || ancestor_class.name.empty?
+            class_name = ancestor_class.name.to_sym
+            return instance_adapters[class_name] if instance_adapters[class_name]
+          end
+
+          raise(Sunspot::NoAdapterError,
+                "No adapter is configured for #{original_class_name} or its superclasses. See the documentation for Sunspot::Adapters")
+        end
+
+        def index_id_for(class_name, id)
+          "#{class_name} #{id}"
+        end
+
+        protected
+
+        # Lazy-initialize the hash of registered instance adapters
+        #
+        # ==== Returns
+        #
+        # Hash:: Hash containing class names keyed to instance adapter classes
+        #
+        def instance_adapters #:nodoc:
+          @instance_adapters ||= {}
+        end
+      end
+    end
+
+    # Subclasses of the DataAccessor class take care of retreiving instances of
+    # the adapted class from (usually persistent) storage. Subclasses must
+    # implement the #load method, which takes an id (the value returned by
+    # InstanceAdapter#id, as a string), and returns the instance referenced by
+    # that ID. Optionally, it can also override the #load_all method, which
+    # takes an array of IDs and returns an array of instances in the order
+    # given. #load_all need only be implemented if it can be done more
+    # efficiently than simply iterating over the IDs and calling #load on each
+    # individually.
+    #
+    # ==== Example
+    #
+    #   class FileAccessor < Sunspot::Adapters::InstanceAdapter
+    #     def load(id)
+    #       @clazz.open(id)
+    #     end
+    #   end
+    #
+    #   Sunspot::Adapters::DataAccessor.register(FileAccessor, File)
+    #
+    class DataAccessor
+      def initialize(clazz) #:nodoc:
+        @clazz = clazz
+      end
+
+      # Subclasses can override this class to provide more efficient bulk
+      # loading of instances. Instances must be returned in the same order
+      # that the IDs were given.
+      #
+      # ==== Parameters
+      #
+      # ids<Array>:: collection of IDs
+      #
+      # ==== Returns
+      #
+      # Array:: collection of instances, in order of IDs given
+      #
+      def load_all(ids)
+        ids.map { |id| self.load(id) }
+      end
+
+      class <<self
+        # Create a DataAccessor for the given class, searching registered
+        # adapters for the best match. See InstanceAdapter#adapt for discussion
+        # of inheritence.
+        #
+        # ==== Parameters
+        #
+        # clazz<Class>:: Class to create DataAccessor for
+        #
+        # ==== Returns
+        #
+        # DataAccessor::
+        #   DataAccessor implementation which provides access to given class
+        #
+        def create(clazz) #:nodoc:
+          self.for(clazz).new(clazz)
+        end
+
+        # Register data accessor for a set of classes. When searching for
+        # an accessor for a given class, Sunspot starts with the class,
+        # and then searches for registered adapters up the class's ancestor
+        # chain.
+        #
+        # ==== Parameters
+        #
+        # data_accessor<Class>:: The data accessor class to register
+        # classes...<Class>::
+        #   One or more classes that this data accessor providess access to
+        #
+        def register(data_accessor, *classes)
+          for clazz in classes
+            data_accessors[clazz.name.to_sym] = data_accessor
+          end
+        end
+
+        # Find the best DataAccessor implementation that adapts the given class.
+        # Starting with the class and then moving up the ancestor chain, looks
+        # for registered DataAccessor implementations.
+        #
+        # ==== Parameters
+        #
+        # clazz<Class>:: The class to find a DataAccessor for
+        #
+        # ==== Returns
+        #
+        # Class:: Implementation of DataAccessor
+        #
+        # ==== Raises
+        #
+        # Sunspot::NoAdapterError:: If no data accessor exists for the given class
+        #
+        def for(clazz) #:nodoc:
+          original_class_name = clazz.name
+          clazz.ancestors.each do |ancestor_class|
+            next if ancestor_class.name.nil? || ancestor_class.name.empty?
+            class_name = ancestor_class.name.to_sym
+            return data_accessors[class_name] if data_accessors[class_name]
+          end
+          raise(Sunspot::NoAdapterError,
+                "No data accessor is configured for #{original_class_name} or its superclasses. See the documentation for Sunspot::Adapters")
+        end
+
+        protected
+
+        # Lazy-initialize the hash of registered data accessors
+        #
+        # ==== Returns
+        #
+        # Hash:: Hash containing class names keyed to data accessor classes
+        #
+        def data_accessors #:nodoc:
+          @adapters ||= {}
+        end
+      end
+    end
+  end
+end