mwmitchell-solr 0.5.3 → 0.5.4

data/CHANGES.txt

@@ -0,0 +1,28 @@
+0.5.4 - December 29, 2008
+	
+	Re-organized the main Solr adapters, they're now in Solr::Connection::Adapter instead of Solr::Adapter
+	
+	All responses from HTTPClient and Connection::Adapter::Direct return a hash with the following keys:
+		
+		:status_code
+		:body
+		:params
+		:url
+		:path
+		:headers
+		:data
+	
+	This hash is now available in the solr response objects as #source - this will be useful in testing and debugging by allowing you to see the generated params and queries... example:
+		
+		response = Solr.query(:q=>'*:*')
+		response.source[:params]
+		response.source[:body]
+		response.source[:url]
+	
+	Added MultiValue field support in Solr::Message, thanks to Fouad Mardini
+	
+	Bug in Solr::Connection::SearchExt where the :q params was not getting generated - fixed by Fouad Mardini
+	
+	Organized tests a bit, moved connection tests into test/connection
+	
+	Fixed a bug in Solr::Connection::Adapter::HTTP where invalid HTTP POST headers were being generated

data/README.rdoc

@@ -21,25 +21,21 @@ You can set Solr params that will be sent for every request:
   
   solr = Solr.connect(:http, :global_params=>{:wt=>:ruby, :echoParams=>'EXPLICIT'})
 
-Solr.connect also yields the adapter instance if a block is supplied:
-  
-  solr = Solr.connect(:http) do |net_http|
-    net_http.class == Net::HTTP
-    # set ssl options etc..
-  end
-
 
 == Requests
 Once you have a connection, you can execute queries, updates etc..
 
 
 === Querying
-  response = solr.query(:q=>'washington', :facet=>true, :facet.limit=>-1, :facet.field=cat, :facet.field=>inStock)
+  response = solr.query(:q=>'washington', :facet=>true, :facet.limit=>-1, :facet.field=>cat, :facet.field=>inStock)
   response = solr.find_by_id(1)
 
 Using the #search method makes building more complex Solr queries easier:
   
   response = solr.search 'my search', :filters=>{:price=>(0.00..10.00)}
+  response.docs.each do |doc|
+    doc.price
+  end
 
 ====Pagination
 Pagination is simplified by using the :page and :per_page params:
@@ -55,8 +51,10 @@ If you use WillPaginate, just pass-in the response to the #will_paginate view he
   
   <%= will_paginate(@response) %>
 
+The #query and #search methods automatically figure out the :start and :rows value, based on the values of :page and :per_page. The will_paginate view helper just needs the right methods (#current_page, #previous_page, #next_page and #total_pages) to create the pagination view widget.
+
 
-=== Updating
+=== Updating Solr
 Updating is done using native Ruby structures. Hashes are used for single documents and arrays are used for a collection of documents (hashes). These structures get turned into simple XML "messages".
 
 Single document
@@ -89,13 +87,15 @@ Commit & Optimize
 
 
 ==Response Formats
-The default response format is Ruby. When the :wt param is set to :ruby, the response is eval'd and wrapped up in a nice Solr::Response class. You can get raw ruby by setting the :wt to "ruby" - notice, the string -- not a symbol. All other response formats are available as expected, :wt=>'xml' etc.. Currently, the only response format that gets eval'd and wrapped is :ruby.
+The default response format is Ruby. When the :wt param is set to :ruby, the response is eval'd and wrapped up in a nice Solr::Response class. You can get an unwrapped response by setting the :wt to "ruby" - notice, the string -- not a symbol. All other response formats are available as expected, :wt=>'xml' etc.. Currently, the only response format that gets eval'd and wrapped is :ruby.
+
+You can access the original request context (path, params, url etc.) from response.source. The response.source is a hash that contains the generated params, url, path, post data, headers etc.. This could be useful for debugging and testing.
 
 ==Data Mapping
-The Solr::Mapper::Base class provides some nice ways of mapping data. You provide a hash mapping and a "data source". The keys of the hash mapping become the Solr field names. The values of the hash mapping get processed differently based on the type of value. The data source must be an Enumerable type object. The hash mapping is called for each item in the data source.
+The Solr::Mapper::Base class provides some nice ways of mapping data. You provide a hash mapping and a "data source". The keys of the hash mapping become the Solr field names. The values of the hash mapping get processed differently based on the value. The data source must be an Enumerable type object. The hash mapping is processed for each item in the data source.
 
 ===Hash Map Processing
-If the value is a string, the String is used as the final Solr field value. If the value is a Symbol, the Symbol is used as a key on the data source. An Enumerable type does the same as the Symbol, but for each item. The most interesting processing occurs when the value is a Proc. When a Proc is used as a hash mapping value, the Solr::Mapper::Base executes the Proc's #call method, passing in the current data source item.
+If the value is a string, the value of the String is used as the final Solr field value. If the value is a Symbol, the Symbol is used as a key on the data source. An Enumerable type does the same as the Symbol, but for each item in the set. The most interesting and flexible processing occurs when the value is a Proc. When a Proc is used as a hash mapping value, the mapper executes the Proc's #call method, passing in the current data source item.
 
 ===Examples
 
@@ -145,4 +145,41 @@ If the value is a string, the String is used as the final Solr field value. If t
       :meta=>['Mr. XYZ', 'A second class document'],
       :web_id=>'web_id_for_doc_2_for_example'
     }
-  ]
+  ]
+
+===RSS Mapper
+There is currently one built in mapper, Solr::Mapper::RSS. Here's an example usage:
+  
+  mapper = Solr::Mapper::RSS.new
+  mapping = {
+    :channel=>:'channel.title',
+    :url=>:'channel.link',
+    :total=>:'items.size',
+    :title=>proc {|item,m| item.title },
+    :link=>proc {|item,m| item.link },
+    :published=>proc {|item,m| item.date },
+    :description=>proc {|item,m| item.description }
+  }
+  mapped_data = m.map('http://site.com/feed.rss')
+
+==Indexing
+Solr (ruby) comes with a simple indexer that makes use of the Solr mapper. Here's an example, using the "mapping" and "mapped_data" variables above (RSS mapper):
+
+  solr = Solr.connect(:http)
+  i = Solr::Indexer.new(solr, mapping)
+  i.index(data_source)
+
+
+==HTTP Client Adapter
+You can specify the Ruby http client to use by setting Solr::Adapter::HTTP.client_adapter to one of:
+  :net_http     uses the standard Net::HTTP library
+  :curb         uses the Ruby "curl" bindings
+
+Example:
+	
+	hclient = Solr::HTTPClient.connect(url, :curb)
+	hclient = Solr::HTTPClient.connect(url, :net_http)
+
+After reading this http://apocryph.org/2008/11/09/more_indepth_analysis_ruby_http_client_performance/ - I would recommend using the :curb adapter. NOTE: You can't use the :curb adapter under jRuby. To install curb:
+
+  sudo gem install curb

data/Rakefile

@@ -17,7 +17,7 @@ task :default => [:test_units]
 
 desc "Run basic tests"
 Rake::TestTask.new("test_units") { |t|
-  t.pattern = 'test/*_test.rb'
+  t.pattern = 'test/**/*_test.rb'
   t.verbose = true
   t.warning = true
 }

data/examples/direct.rb

@@ -1,14 +1,16 @@
 # Must be executed using jruby
 require File.join(File.dirname(__FILE__), '..', 'lib', 'solr')
 
-path_to_solr_dist=''
-
 base = File.expand_path( File.dirname(__FILE__) )
 dist = File.join(base, '..', 'apache-solr')
 home = File.join(dist, 'example', 'solr')
 
 solr = Solr.connect(:direct, :home_dir=>home, :dist_dir=>dist)
 
-#`cd ../apache-solr-1.3.0/example/exampledocs && ./post.sh ./*.xml`
+`cd ../apache-solr/example/exampledocs && ./post.sh ./*.xml`
+
+response = solr.search 'ipod', :filters=>{:price=>(0..50)}, :per_page=>2, :page=>1
+
+solr.delete_by_query('*:*')
 
-solr.search 'ipod', :filters=>{:price=>(0..50)}, :per_page=>2, :page=>1
+puts response.inspect

data/examples/http.rb

@@ -1,7 +1,12 @@
+# Must be executed using jruby
 require File.join(File.dirname(__FILE__), '..', 'lib', 'solr')
 
 solr = Solr.connect(:http)
 
-#`cd ../apache-solr-1.3.0/example/exampledocs && ./post.sh ./*.xml`
+`cd ../apache-solr/example/exampledocs && ./post.sh ./*.xml`
 
-r = solr.paginate 'ipod', :filters=>{:price=>(0..50)}, :per_page=>2, :page=>1
+response = solr.search 'ipod', :filters=>{:price=>(0..50)}, :per_page=>2, :page=>1
+
+solr.delete_by_query('*:*')
+
+puts response.inspect

data/lib/solr/{adapter → connection/adapter}/common_methods.rb

@@ -7,9 +7,7 @@
 #   request_path is a string to a handler (/select)
 #   params is a hash for query string params
 #   data is optional string of xml
-#
-#
-module Solr::Adapter::CommonMethods
+module Solr::Connection::Adapter::CommonMethods
   
   # send a request to the "select" handler
   def query(params)
@@ -45,45 +43,4 @@ module Solr::Adapter::CommonMethods
     @adapter.send_request(handler_path, params, data)
   end
   
-  # escapes a query key/value for http
-  def escape(s)
-    s.to_s.gsub(/([^ a-zA-Z0-9_.-]+)/n) {
-      '%'+$1.unpack('H2'*$1.size).join('%').upcase
-    }.tr(' ', '+') 
-  end
-  
-  def build_param(k,v)
-    "#{escape(k)}=#{escape(v)}"
-  end
-  
-  # takes a path and a hash of query params, returns an escaped url with query params
-  def build_url(path, params_hash=nil)
-    query = hash_to_params(params_hash)
-    query ? path + '?' + query : path
-  end
-  
-  #
-  # converts hash into URL query string, keys get an alpha sort
-  # if a value is an array, the array values get mapped to the same key:
-  #   hash_to_params(:q=>'blah', 'facet.field'=>['location_facet', 'format_facet'])
-  # returns:
-  #   ?q=blah&facet.field=location_facet&facet.field=format.facet
-  #
-  # if a value is empty/nil etc., the key is not added
-  def hash_to_params(params)
-    return unless params.is_a?(Hash)
-    # copy params and convert keys to strings
-    params = params.inject({}){|acc,(k,v)| acc.merge({k.to_s, v}) }
-    # get sorted keys
-    params.keys.sort.inject([]) do |acc,k|
-      v = params[k]
-      if v.is_a?(Array)
-        acc << v.reject{|i|i.to_s.empty?}.collect{|vv|build_param(k, vv)}
-      elsif ! v.to_s.empty?
-        acc.push(build_param(k, v))
-      end
-      acc
-    end.join('&')
-  end
-  
 end

data/lib/solr/{adapter → connection/adapter}/direct.rb

@@ -5,11 +5,12 @@ require 'java'
 #
 # Connection for JRuby + DirectSolrConnection
 #
-class Solr::Adapter::Direct
+class Solr::Connection::Adapter::Direct
   
-  include Solr::Adapter::CommonMethods
+  include Solr::HTTPClient::Util
+  include Solr::Connection::Adapter::CommonMethods
   
-  attr_accessor :opts, :connection, :home_dir
+  attr_accessor :opts, :home_dir
   
   # required: opts[:home_dir] is absolute path to solr home (the directory with "data", "config" etc.)
   # opts must also contain either
@@ -19,31 +20,45 @@ class Solr::Adapter::Direct
   # OTHER OPTS:
   #   :select_path => 'the/select/handler'
   #   :update_path => 'the/update/handler'
-  # If a block is given, the @connection instance (DirectSolrConnection) is yielded
   def initialize(opts, &block)
-    @home_dir = opts[:home_dir]
-    opts[:data_dir] ||= File.join(@home_dir , 'data')
+    @home_dir = opts[:home_dir].to_s
+    opts[:data_dir] ||= File.join(@home_dir, 'data')
     if opts[:dist_dir]
       # add the standard lib and dist directories to the :jar_paths
       opts[:jar_paths] = [File.join(opts[:dist_dir], 'lib'), File.join(opts[:dist_dir], 'dist')]
     end
     @opts = default_options.merge(opts)
-    require_jars(@opts[:jar_paths])
-    import_dependencies
-    @connection = DirectSolrConnection.new(@home_dir, @opts[:data_dir], nil)
-    yield @connection if block_given?
+  end
+  
+  # loads/imports the java dependencies
+  # sets the @connection instance variable
+  def connection
+    @connection ||= (
+      require_jars(@opts[:jar_paths]) if @opts[:jar_paths]
+      import_dependencies
+      DirectSolrConnection.new(@home_dir, @opts[:data_dir], nil)
+    )
   end
   
   # send a request to the connection
   # request '/update', :wt=>:xml, '</commit>'
-  def send_request(request_url_path, params={}, data=nil)
+  def send_request(path, params={}, data=nil)
     data = data.to_xml if data.respond_to?(:to_xml)
-    full_path = build_url(request_url_path, params)
+    url = build_url(path, params)
     begin
-      @connection.request(full_path, data)
+      body = connection.request(url, data)
     rescue
       raise Solr::RequestError.new($!.message)
     end
+    {
+      :status_code=>'',
+      :body=>body,
+      :url=>url,
+      :path=>path,
+      :params=>params,
+      :data=>data,
+      :headers=>{}
+    }
   end
   
   protected

data/lib/solr/connection/adapter/http.rb

@@ -0,0 +1,51 @@
+#
+# Connection for standard HTTP Solr server
+#
+class Solr::Connection::Adapter::HTTP
+  
+  class << self
+    attr_accessor :client_adapter
+  end
+  
+  @client_adapter = :net_http
+  
+  include Solr::Connection::Adapter::CommonMethods
+  
+  attr_reader :opts
+  
+  # opts can have:
+  #   :url => 'http://localhost:8080/solr'
+  #   :select_path => '/the/url/path/to/the/select/handler'
+  #   :update_path => '/the/url/path/to/the/update/handler'
+  #   :luke_path => '/admin/luke'
+  #
+  def initialize(opts={}, &block)
+    opts[:url]||='http://127.0.0.1:8983/solr'
+    @opts = default_options.merge(opts)
+  end
+  
+  def connection
+    @connection ||= Solr::HTTPClient.connect(@opts[:url], self.class.client_adapter)
+  end
+  
+  # send a request to the connection
+  # request '/update', :wt=>:xml, '</commit>'
+  def send_request(path, params={}, data=nil)
+    data = data.to_xml if data.respond_to?(:to_xml)
+    if data
+      http_context = connection.post(path, data, params, post_headers)
+    else
+      http_context = connection.get(path, params)
+    end
+    raise Solr::RequestError.new(http_context[:body]) unless http_context[:status_code] == 200
+    http_context
+  end
+  
+  protected
+  
+  # The standard post headers
+  def post_headers
+    {"Content-Type" => 'text/xml; charset=utf-8'}
+  end
+  
+end

data/lib/solr/connection/adapter.rb

@@ -0,0 +1,7 @@
+module Solr::Connection::Adapter
+  
+  autoload :Direct, 'solr/connection/adapter/direct'
+  autoload :HTTP, 'solr/connection/adapter/http'
+  autoload :CommonMethods, 'solr/connection/adapter/common_methods'
+  
+end

data/lib/solr/connection/search_ext.rb

@@ -1,32 +1,29 @@
 module Solr::Connection::SearchExt
   
-  def search(query, params={})
-    if params[:fields].is_a?(Array)
-      params[:fl] = params.delete(:fields).join(' ')
-    else
-      params[:fl] = params.delete :fields
-    end
-    fq = build_filters(params.delete(:filters)).join(' ') if params[:filters]
-    if params[:fq] and fq
-      params[:fq] += " AND #{fq}"
-    else
-      params[:fq] = fq
+  def search(q_param, params={})
+    if params[:fields]
+      fields = params.delete :fields
+      params[:fl] = fields.is_a?(Array) ? fields.join(' ') : fields
     end
+    
+    params[:fq] = build_filters(params.delete(:filters)) if params[:filters]
     facets = params.delete(:facets) if params[:facets]
+    
     if facets
       if facets.is_a?(Array)
-        params << {:facet => true}
-        params += build_facets(facets)          
+        params.merge!({:facet => true})
+        params.merge! build_facets(facets)          
       elsif facets.is_a?(Hash)
-        params << {:facet => true}
-        params += build_facet(facets)
+        params.mrege!({:facet => true})
+        #params += build_facet(facets)
       elsif facets.is_a?(String)
-        params += facets
+        #params += facets
       else
         raise 'facets must either be a Hash or an Array'
       end
     end
-    params[:qt] ||= :dismax
+    #params[:qt] ||= :dismax
+    params[:q] = build_query(q_param)
     self.query params
   end
   
@@ -81,10 +78,11 @@ module Solr::Connection::SearchExt
     end
     params
   end
-
+  
   def build_facets(facet_array)
-    facet_array.inject([]) do |params, facet_hash|
-      params.push build_facet(facet_hash)
+    facet_array.inject({}) do |p, facet_hash|
+      build_facet(facet_hash).each {|k| p.merge!(k) }
+      p
     end
   end
 
@@ -96,10 +94,12 @@ module Solr::Connection::SearchExt
       if 'field' == k.to_s
         params << {"facet.field" => v}
       elsif 'query' == k.to_s
-        q = build_query("facet.query", v)
-        params << q
-      elsif ['name', :name].include?(k.to_s)
-        # do nothing
+        q = build_query(v)
+        params << {"facet.query"=>q}
+        if facet_name
+          # keep track of names => facet_queries
+          name_to_facet_query[facet_name] = q['facet.query']
+        end
       else
         params << {"f.#{facet_hash[:field]}.facet.#{k}" => v}
       end
@@ -107,4 +107,8 @@ module Solr::Connection::SearchExt
     params
   end
   
+  def name_to_facet_query
+    @name_to_facet_query ||= {}
+  end
+
 end