mwmitchell-rsolr 0.6.2 → 0.6.3

data/CHANGES.txt

@@ -1,3 +1,9 @@
+0.6.3 - January 21, 2009
+	Added a new param mapping module: RSolr::Connection::ParamMapping
+	Mapping only for fields that need conversion/escaping or nested (facet.*) etc.
+	This new module can be activated by using the #search method
+	New tests for ParamMapping
+
 0.6.2 - January 14, 2009
 	Removed mapping and indexer modules -- seems to me that a general purpose mapping library
 	would be more valuable than an embedded module in RSolr ( RMapper ?)

data/README.rdoc

@@ -45,7 +45,23 @@ Use the #search method to take advantage of some of the param mapping (currently
 
 Thanks to a little Ruby magic, we can chain symbols to create Solr "dot" syntax: :facet.field=>'cat'
 
-====Pagination
+==== Search Params
+The #search method can accept the following params:
+===== When :qt is :standard
+  :page
+  :per_page
+  :queries
+  :filters
+  :phrase_queries
+  :phrase_filters
+  :facets
+===== When :qt is :dismax (also includes the :standard params)
+  :alternate_query
+  :query_fields
+  :phrase_fields
+  :boost_query
+
+==== Pagination
 Pagination is simplified by using the :page and :per_page params when using the #search method:
 
   response = solr.search(:page=>1, :per_page=>10, :q=>'*:*')
@@ -56,14 +72,11 @@ Pagination is simplified by using the :page and :per_page params when using the
   response.next_page
 
 If you use WillPaginate, just pass-in the response to the #will_paginate view helper:
-  
+
   <%= will_paginate(@response) %>
 
 The #search method automatically figures out the :start and :rows values, based on the values of :page and :per_page. The will_paginate view helper uses the methods: #current_page, #previous_page, #next_page and #total_pages to create the pagination view widget.
 
-The #search method will be providing more param mapping, but not until RSolr gets more use. If you have ideas for a param mapping interface, let me know! I'll be throwing some stuff together as well.
-
-
 === Updating Solr
 Updating is done using native Ruby structures. Hashes are used for single documents and arrays are used for a collection of documents (hashes). These structures get turned into simple XML "messages".
 
@@ -96,91 +109,13 @@ Commit & Optimize
   solr.optimize
 
 
-==Response Formats
+== Response Formats
 The default response format is Ruby. When the :wt param is set to :ruby, the response is eval'd and wrapped up in a nice RSolr::Response class. You can get an unwrapped response by setting the :wt to "ruby" - notice, the string -- not a symbol. All other response formats are available as expected, :wt=>'xml' etc.. Currently, the only response format that gets eval'd and wrapped is :ruby.
 
 You can access the original request context (path, params, url etc.) from response.request. The response.request is a hash that contains the generated params, url, path, post data, headers etc., very useful for debugging and testing.
 
-==Data Mapping
-The RSolr::Mapper::Base class provides some nice ways of mapping data. You provide a hash mapping and a "data source". The keys of the hash mapping become the Solr field names. The values of the hash mapping get processed differently based on the value. The data source must be an Enumerable type object. The hash mapping is processed for each item in the data source.
-
-===Hash Map Processing
-If the value is a string, the value of the String is used as the final Solr field value. If the value is a Symbol, the Symbol is used as a key on the data source. An Enumerable type does the same as the Symbol, but for each item in the set. The most interesting and flexible processing occurs when the value is a Proc. When a Proc is used as a hash mapping value, the mapper executes the Proc's #call method, passing in the current data source item.
 
-===Examples
-
-  mapping = {
-    :id=>:id,
-    :title=>:title,
-    :source=>'Example',
-    :meta=>[:author, :sub_title],
-    :web_id=>proc {|item|
-      WebService.fetch_item_id_by_name(item[:name])
-    }
-  }
-  
-  data_source = [
-    {
-      :id=>100,
-      :title=>'Doc One',
-      :author=>'Mr. X',
-      :sub_title=>'A first class document.',
-      :name=>'doc_1'
-    },
-    {
-      :id=>200,
-      :title=>'Doc Two',
-      :author=>'Mr. XYZ',
-      :sub_title=>'A second class document.',
-      :name=>'doc_2'
-    }
-  ]
-  
-  mapper = RSolr::Mapper::Base(mapping)
-  mapped_data = mapper.map(data_source)
-  
-  # the following would be true...
-  mapped_data == [
-    {
-      :id=>100,
-      :title=>'Doc One',
-      :source=>'Example',
-      :meta=>['Mr. X', 'A first class document'],
-      :web_id=>'web_id_for_doc_1_for_example'
-    },
-    {
-      :id=>200,
-      :title=>'Doc Two',
-      :source=>'Example',
-      :meta=>['Mr. XYZ', 'A second class document'],
-      :web_id=>'web_id_for_doc_2_for_example'
-    }
-  ]
-
-===RSS Mapper
-There is currently one built in mapper, RSolr::Mapper::RSS. Here's an example usage:
-  
-  mapper = RSolr::Mapper::RSS.new
-  mapping = {
-    :channel=>:'channel.title',
-    :url=>:'channel.link',
-    :total=>:'items.size',
-    :title=>proc {|item,m| item.title },
-    :link=>proc {|item,m| item.link },
-    :published=>proc {|item,m| item.date },
-    :description=>proc {|item,m| item.description }
-  }
-  mapped_data = m.map('http://site.com/feed.rss')
-
-==Indexing
-RSolr comes with a simple indexer that makes use of the Solr mapper. Here's an example, using the "mapping" and "mapped_data" variables above (RSS mapper):
-
-  solr = RSolr.connect(:http)
-  i = RSolr::Indexer.new(solr, mapping)
-  i.index(mapped_data)
-
-
-==HTTP Client Adapter
+== HTTP Client Adapter
 You can specify the http client adapter to use by setting RSolr::Connection::Adapter::HTTP.client_adapter to one of:
   :net_http     uses the standard Net::HTTP library
   :curb         uses the Ruby "curl" bindings
@@ -194,6 +129,6 @@ Example of using the HTTP client only:
   hclient = RSolr::HTTPClient.connect(url, :curb)
   hclient = RSolr::HTTPClient.connect(url, :net_http)
 
-After reading this http://apocryph.org/2008/11/09/more_indepth_analysis_ruby_http_client_performance/ - I would recommend using the :curb adapter. NOTE: You can't use the :curb adapter under jRuby. To install curb:
+After reading this http://apocryph.org/2008/11/09/more_indepth_analysis_ruby_http_client_performance - I would recommend using the :curb adapter. NOTE: You can't use the :curb adapter under jRuby. To install curb:
 
   sudo gem install curb

data/lib/rsolr/connection/base.rb

@@ -5,11 +5,17 @@ class RSolr::Connection::Base
   
   attr_reader :adapter, :opts
   
+  attr_accessor :param_mappers
+  
   # "adapter" is instance of:
   #   RSolr::Adapter::HTTP
   #   RSolr::Adapter::Direct (jRuby only)
   def initialize(adapter, opts={})
     @adapter=adapter
+    @param_mappers = {
+      :standard=>RSolr::Connection::ParamMapping::Standard,
+      :dismax=>RSolr::Connection::ParamMapping::Dismax
+    }
     opts[:global_params]||={}
     default_global_params = {
       :wt=>:ruby,
@@ -39,12 +45,12 @@ class RSolr::Connection::Base
     p[:wt]==:ruby ? RSolr::Response::Query::Base.new(response) : response
   end
   
-  # same as the #query method, but with additional param mapping
-  # currently only :page and :per_page
-  # TODO: need to get some nice and friendly param mapping here:
-  # search(:fields=>'', :facet_fields=>[], :filters=>{})
-  def search(params)
-    self.query(modify_params_for_pagination(params))
+  # register your own mapper?
+  def search(params,&blk)
+    qt = params[:qt] ? params[:qt].to_sym : :dismax
+    mapper_class = @param_mappers[qt]
+    mapper = mapper_class.new(params)
+    query(mapper.map(&blk))
   end
   
   # Finds a document by its id
@@ -110,22 +116,4 @@ class RSolr::Connection::Base
     RSolr::Message
   end
   
-  # given a hash, this method will attempt to produce the
-  # correct :start and :rows values for Solr
-  # -- if the hash contains a :per_page value, it's used for the rows
-  # if the :per_page value doesn't exist (nil), the :rows value is
-  # used, and if :rows is not set, the default value is 10
-  # -- if the hash contains a :page value (the current page)
-  # it is used to calculate the :start value
-  # returns a hash with the :rows and :start values
-  # :per_page and :page are deleted
-  def modify_params_for_pagination(p)
-    per_page = p.delete(:per_page).to_s.to_i
-    p[:rows] = per_page==0 ? (p[:rows] || 10) : per_page
-    page = p.delete(:page).to_s.to_i
-    page = page > 0 ? page : 1
-    p[:start] = (page - 1) * (p[:rows] || 0)
-    p
-  end
-  
 end

data/lib/rsolr/connection/param_mapping/dismax.rb

@@ -0,0 +1,41 @@
+class RSolr::Connection::ParamMapping::Dismax < RSolr::Connection::ParamMapping::Standard
+  
+  def setup_mappings
+    super
+    
+    mapping_for :alternate_query, :q.alt do |val|
+      format_query(val)
+    end
+    
+    mapping_for :query_fields, :qf do |val|
+      create_boost_query(val)
+    end
+    
+    mapping_for :phrase_fields, :pf do |val|
+      create_boost_query(val)
+    end
+    
+    mapping_for :boost_query, :bq do |val|
+      format_query(val)
+    end
+    
+  end
+  
+  protected
+  
+  def create_boost_query(input)
+    case input
+    when Hash
+      qf = []
+      input.each_pair do |k,v|
+        qf << (v.to_s.empty? ? k : "#{k}^#{v}")
+      end
+      qf.join(' ')
+    when Array
+      input.join(' ')
+    when String
+      input
+    end
+  end
+  
+end

data/lib/rsolr/connection/param_mapping/standard.rb

@@ -0,0 +1,115 @@
+class RSolr::Connection::ParamMapping::Standard
+  
+  include RSolr::Connection::ParamMapping::MappingMethods
+  
+  attr_reader :input, :output
+  
+  def initialize(input)
+    @output = {}
+    @input = input
+    setup_mappings
+  end
+  
+  def setup_mappings
+    
+    mapping_for :per_page, :rows do |val|
+      val = val.to_s.to_i
+      val < 0 ? 0 : val
+    end
+    
+    mapping_for :page, :start do |val|
+      val = val.to_s.to_i
+      page = val > 0 ? val : 1
+      ((page - 1) * (@output[:rows] || 0))
+    end
+    
+    mapping_for :queries, :q do |val|
+      format_query(val)
+    end
+    
+    mapping_for :phrase_queries, :q do |val|
+      [@output[:q], format_query(val, true)].reject{|v|v.to_s.empty?}.join(' ')
+    end
+    
+    mapping_for :filters, :fq do |val|
+      format_query(val)
+    end
+    
+    # this must come after the :filter/:fq mapper
+    mapping_for :phrase_filters, :fq do |val|
+      [@output[:fq], format_query(val, true)].reject{|v|v.to_s.empty?}.join(' ')
+    end
+    
+    mapping_for :facets do |input|
+      next if input.to_s.empty?
+      @output[:facet] = true
+      @output[:facet.field] = []
+      if input[:queries]
+        # convert to an array if needed
+        input[:queries] = [input[:queries]] unless input[:queries].is_a?(Array)
+        @output[:facet.query] = input[:queries].map{|q|format_query(q)}
+      end
+      common_sub_fields = [:sort, :limit, :missing, :mincount, :prefix, :offset, :method, :enum.cache.minDf]
+      (common_sub_fields).each do |subfield|
+        next unless input[subfield]
+        @output["facet.#{subfield}"] = input[subfield]
+      end
+      if input[:fields]
+        input[:fields].each do |f|
+          if f.kind_of? Hash
+            key = f.keys[0]
+            value = f[key]
+            @output[:facet.field] << key
+            common_sub_fields.each do |subfield|
+              next unless value[subfield]
+              @output["f.#{key}.facet.#{subfield}"] = input[subfield]
+            end
+          else
+            @output[:facet.field] << f
+          end
+        end
+      end
+    end
+  end
+  
+  def format_query(input, quote=false)
+    case input
+    when Array
+      format_array_query(input, quote)
+    when Hash
+      format_hash_query(input, quote)
+    else
+      prep_value(input, quote)
+    end
+  end
+  
+  def format_array_query(input, quote)
+    input.collect do |v|
+      v.is_a?(Hash) ? format_hash_query(v, quote) : prep_value(v, quote)
+    end.join(' ')
+  end
+  
+  # groups values to a single field: title:(value1 value2) instead of title:value1 title:value2
+  # a value can be a range or a string
+  def format_hash_query(input, quote=false)
+    q = []
+    input.each_pair do |field,value|
+      next if value.to_s.empty? # skip blank values!
+      # create the field plus the delimiter if the field is not blank
+      value = [value] unless value.is_a?(Array)
+      fielded_queries = value.collect do |vv|
+        vv.is_a?(Range) ? "[#{vv.min} TO #{vv.max}]" : prep_value(vv, quote)
+      end
+      field = field.to_s.empty? ? '' : "#{field}:"
+      if fielded_queries.size > 0
+        q << (fielded_queries.size > 1 ? "#{field}(#{fielded_queries.join(' ')})" : "#{field}#{fielded_queries.to_s}")
+      end
+    end
+    q.join(' ')
+  end
+  
+  def prep_value(val, quote=false)
+    quote ? %(\"#{val}\") : val.to_s
+  end
+  
+end

data/lib/rsolr/connection/param_mapping.rb

@@ -0,0 +1,39 @@
+module RSolr::Connection::ParamMapping
+  
+  autoload :Standard, 'rsolr/connection/param_mapping/standard'
+  autoload :Dismax, 'rsolr/connection/param_mapping/dismax'
+  
+  module MappingMethods
+    
+    def mappers
+      @mappers ||= []
+    end
+    
+    def mapping_for(user_param_name, solr_param_name=nil, &block)
+      return unless @input[user_param_name]
+      if (m = self.mappers.detect{|m|m[:input_name] == user_param_name})
+        self.mappers.delete m
+      end
+      self.mappers << {:input_name=>user_param_name, :output_name=>solr_param_name, :block=>block}
+    end
+    
+    def map(&blk)
+      input = @input.dup
+      mappers.each do |m|
+      input_value = input[m[:input_name]]
+        input.delete m[:input_name]
+        if m[:block]
+          value = m[:block].call(input_value)
+        else
+          value = input_value
+        end
+        if m[:output_name]
+          @output[m[:output_name]] = value
+        end
+      end
+      @output.merge(input)
+    end
+    
+  end
+  
+end

data/lib/rsolr/connection.rb

@@ -2,5 +2,6 @@ module RSolr::Connection
   
   autoload :Base, 'rsolr/connection/base'
   autoload :Adapter, 'rsolr/connection/adapter'
+  autoload :ParamMapping, 'rsolr/connection/param_mapping'
   
 end

data/lib/rsolr.rb

@@ -7,7 +7,7 @@ proc {|base, files|
 
 module RSolr
   
-  VERSION = '0.6.2'
+  VERSION = '0.6.3'
   
   autoload :Message, 'rsolr/message'
   autoload :Response, 'rsolr/response'

data/test/connection/param_mapping_test.rb

@@ -0,0 +1,61 @@
+require File.join(File.dirname(__FILE__), '..', 'test_helpers')
+
+class ParamMappingTest < RSolrBaseTest
+  
+  include RSolr::Connection::ParamMapping
+  
+  def test_standard_simple
+    input = {
+      :queries=>'a query',
+      :filters=>'a filter',
+      :page=>1,
+      :per_page=>10,
+      :phrase_queries=>'a phrase query',
+      :phrase_filters=>'a phrase filter',
+      :facets=>{
+        :fields=>[:one,:two]
+      }
+    }
+    mapper = Standard.new(input)
+    output = mapper.map
+    
+    assert_equal "a query \"a phrase query\"", output[:q]
+    assert_equal "a filter \"a phrase filter\"", output[:fq]
+    assert_equal 0, output[:start]
+    assert_equal 10, output[:rows]
+    # facet.field can be specified multiple times, so we need an array
+    # the url builder automatically adds multiple params for arrays
+    assert_equal [:one, :two], output[:facet.field]
+  end
+  
+  def test_standard_complex
+    input = {
+      :queries=>['a query', {:field=>'value'}, 'blah'],
+      :filters=>['a filter', {:filter=>'field'}, 'blah'],
+      :phrase_queries=>['a phrase', {:phrase_field=>'phrase value'}],
+      :phrase_filters=>{:can_also_be_a=>'hash'}
+    }
+    mapper = Standard.new(input)
+    output = mapper.map
+    
+    assert_equal "a query field:value blah \"a phrase\" phrase_field:\"phrase value\"", output[:q]
+    assert_equal "a filter filter:field blah can_also_be_a:\"hash\"", output[:fq]
+  end
+  
+  def test_dismax
+    input = {
+      :alternate_query=>{:can_be_a_string_hash_or_array=>'OK'},
+      :query_fields=>{:a_field_to_boost=>20, :another_field_to_boost=>200},
+      :phrase_fields=>{:phrase_field=>20},
+      :boost_query=>[{:field_to_use_for_boost_query=>'a'}, 'test']
+    }
+    mapper = Dismax.new(input)
+    output = mapper.map
+    assert_equal 'can_be_a_string_hash_or_array:OK', output[:q.alt]
+    assert output[:qf]=~/another_field_to_boost\^200/
+    assert output[:qf]=~/a_field_to_boost\^20/
+    assert_equal 'phrase_field^20', output[:pf]
+    assert_equal 'field_to_use_for_boost_query:a test', output[:bq]
+  end
+  
+end

data/test/connection/test_methods.rb

@@ -5,6 +5,7 @@
 
 module ConnectionTestMethods
   
+  
   #def teardown
   #  @solr.delete_by_query('id:[* TO *]')
   #  @solr.commit
@@ -80,10 +81,11 @@ module ConnectionTestMethods
   
   def test_add
     assert_equal 0, @solr.query(:q=>'*:*').total
-    response = @solr.add(:id=>100)
+    update_response = @solr.add(:id=>100)
+    assert update_response.is_a?(RSolr::Response::Update)
+    #
     @solr.commit
     assert_equal 1, @solr.query(:q=>'*:*').total
-    assert response.is_a?(RSolr::Response::Update)
   end
   
   def test_delete_by_id

data/test/response/pagination_test.rb

@@ -16,17 +16,6 @@ class ResponsePaginationTest < RSolrBaseTest
     #assert_equal 0, dummy_connection.send(:calculate_start, 0, 50)
   end
   
-  def test_connection_modify_params_for_pagination
-    dummy_connection = RSolr::Connection::Base.new(nil)
-    p = dummy_connection.send(:modify_params_for_pagination, {:page=>1})
-    assert_equal 0, p[:start]
-    assert_equal 10, p[:rows]
-    #
-    p = dummy_connection.send(:modify_params_for_pagination, {:page=>10, :per_page=>100})
-    assert_equal 900, p[:start]
-    assert_equal 100, p[:rows]
-  end
-  
   def test_math
     response = create_response({'rows'=>5})
     assert_equal response.params['rows'], response.per_page

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: mwmitchell-rsolr
 version: !ruby/object:Gem::Version 
-  version: 0.6.2
+  version: 0.6.3
 platform: ruby
 authors: 
 - Matt Mitchell
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-01-14 00:00:00 -08:00
+date: 2009-01-21 00:00:00 -08:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -43,6 +43,9 @@ files:
 - lib/rsolr/connection/adapter/http.rb
 - lib/rsolr/connection/adapter.rb
 - lib/rsolr/connection/base.rb
+- lib/rsolr/connection/param_mapping.rb
+- lib/rsolr/connection/param_mapping/dismax.rb
+- lib/rsolr/connection/param_mapping/standard.rb
 - lib/rsolr/connection.rb
 - lib/rsolr/http_client/adapter/curb.rb
 - lib/rsolr/http_client/adapter/net_http.rb
@@ -89,6 +92,7 @@ summary: A Ruby client for Apache Solr
 test_files: 
 - test/connection/direct_test.rb
 - test/connection/http_test.rb
+- test/connection/param_mapping_test.rb
 - test/connection/test_methods.rb
 - test/core_ext_test
 - test/http_client/curb_test.rb