rsolr 1.0.8 → 1.1.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: e401f0dcdd04d8215eec9b67a1e7c775c41ac7f4
+  data.tar.gz: 6e466531f33da1560869ce39586101dfc4bbb08e
+SHA512:
+  metadata.gz: 29a5ac0f898a33e2d8efc2a0b0b7ed618c169e63597dcc283573ae9302183f0ee983ec18e30d3c9dc100bb13e06ff86358d02ad0df91515101efc8a26f82b7f8
+  data.tar.gz: 1a509a26c086ffbad2543154106584989d7c40d42b747a422a622ba77fe1dd87202373ae1cd8d10727a0b61b89ad16a5cfcdc5655375ebc7d4b5733d9fa7dcb0

data/.travis.yml

@@ -0,0 +1,21 @@
+addons:
+  apt:
+    packages:
+    - libgmp-dev
+language: ruby
+sudo: false
+rvm:
+  - 2.3.1
+  - 2.2.5
+  - jruby-9.0.5.0
+
+notifications:
+  irc: "irc.freenode.org#blacklight"
+  email:
+      - blacklight-commits@googlegroups.com
+
+env:
+  global:
+    - JRUBY_OPTS="-J-Xms512m -J-Xmx1024m"
+    - NOKOGIRI_USE_SYSTEM_LIBRARIES=true
+jdk: oraclejdk8

data/CHANGES.txt

@@ -1,5 +1,24 @@
+1.0.12
+   - Fix bug where specifying the wt property as a string, would add the supplied value to the default ('ruby') rather than overriding it.
+
+1.0.11
+   - add RSolr.solr_escape method and add deprecation messages to RSolr.escape (ndushay)
+   - use stdlib URI.escape methods instead of homegrown in RSolr::URI (ndushay)
+   - fix bug with Rsolr::Uri.create adding trailing slash if query params (ndushay)
+   - update rake tasks  (cbeer)
+   - add Ruby 2.2.0 to travis ci build (ndushay)
+   - Housekeeping (badges to README, license in gemspec, correct url in gemspec ...)  (ndushay)
+   - Improve rdoc styling (udaykadaboina)
+   - Support setting default_wt per connection via its options. (jcoleman)
+   - eliminates the usage of per-instance `extend` (jcoleman)
+   - Upgrade to RSpec 3 (blackwinter, adamjonas, cbeer)
+   - Fixed RSolr::Error to_s  (PofMagicfingers)
+1.0.10 xxx
+
 1.0.8
    -  Fix connection refused errors in specs + add basic auth support (Denis Goeury)
+   - Ability to set :retry_503, the number of retry attempts for a 503 response
+     with a Retry-After header.
 1.0.7
    - Response body encoding is set to response charset in Ruby >= 1.9
    - Ability to set :read_timeout and :open_timeout when creating new instance of RSolr
@@ -14,4 +33,4 @@
    - Jeweler is no longer used for building the gemspec
 1.0.3
    - Proper encodings in Ruby 1.9
-   - Applied pull request from https://github.com/mwmitchell/rsolr/pull/20
+   - Applied pull request from https://github.com/mwmitchell/rsolr/pull/20

data/Gemfile

@@ -1,15 +1,9 @@
-source "http://rubygems.org"
+source "https://rubygems.org"
 
 gemspec
 
 gem "builder", ">= 2.1.2"
 
-group :development do
-  gem "rake", "~> 0.9.2"
-  gem "rdoc", "~> 3.9.4"
-end
-
-group :test do
-  gem "rake", "~> 0.9.2"
-  gem "rspec", "~> 2.6.0"
+if defined? RUBY_VERSION and RUBY_VERSION < "1.9"
+  gem 'nokogiri', "< 1.6"
 end

data/README.rdoc

@@ -1,12 +1,14 @@
 =RSolr
+{<img src="https://travis-ci.org/rsolr/rsolr.svg?branch=master" alt="Build Status" />}[https://travis-ci.org/rsolr/rsolr] {<img src="https://badge.fury.io/rb/rsolr.svg" alt="Gem Version" />}[http://badge.fury.io/rb/rsolr]
+
 
 A simple, extensible Ruby client for Apache Solr.
 
 ==Documentation
-The code docs for the last *release* can be viewed here: http://rubydoc.info/gems/rsolr/1.0.6/frames
+The code docs http://www.rubydoc.info/gems/rsolr
 
 == Installation:
-  sudo gem install rsolr
+  gem install rsolr
 
 == Example:
   require 'rubygems'
@@ -24,18 +26,25 @@ The code docs for the last *release* can be viewed here: http://rubydoc.info/gem
   # send a request to /catalog
   response = solr.get 'catalog', :params => {:q => '*:*'}
 
-When the Solr :wt is :ruby, then the response will be a Hash. This Hash is the same object returned by Solr, but evaluated as Ruby. If the :wt is not :ruby, then the response will be a String.
+When the Solr +:wt+ is +:ruby+, then the response will be a Hash. This Hash is the same object returned by Solr, but evaluated as Ruby. If the +:wt+ is not +:ruby+, then the response will be a String.
 
-The response also exposes 2 attribute readers (for any :wt value), :request and :response. Both are Hash objects with symbolized keys.
+The response also exposes 2 attribute readers (for any +:wt+ value), +:request+ and +:response+. Both are Hash objects with symbolized keys.
 
-The :request attribute contains the original request context. You can use this for debugging or logging. Some of the keys this object contains are :uri, :query, :method etc..
+The +:request+ attribute contains the original request context. You can use this for debugging or logging. Some of the keys this object contains are +:uri+, +:query+, +:method+ etc..
 
-The :response attribute contains the original response. This object contains the :status, :body and :headers keys.
+The +:response+ attribute contains the original response. This object contains the +:status+, +:body+ and +:headers+ keys.
 
 == Timeouts
 The read and connect timeout settings can be set when creating a new instance of RSolr:
   solr = RSolr.connect(:read_timeout => 120, :open_timeout => 120)
 
+== Retry 503s
+A 503 is usually a temporary error which RSolr may retry if requested. You may specify the number of retry attempts with the +:retry_503+ option.
+
+Only requests which specify a Retry-After header will be retried, after waiting the indicated retry interval, otherwise RSolr will treat the request as a 500. You may specify a maximum Retry-After interval to wait with the +:retry_after_limit+ option (default: one second).
+  solr = RSolr.connect(:retry_503 => 1, :retry_after_limit => 1)
+
+
 == Querying
 Use the #get / #post method to send search requests to the /select handler:
   response = solr.get 'select', :params => {
@@ -45,7 +54,7 @@ Use the #get / #post method to send search requests to the /select handler:
   }
   response["response"]["docs"].each{|doc| puts doc["id"] }
 
-The :params sent into the method are sent to Solr as-is, which is to say they are converted to Solr url style, but no special mapping is used.
+The +:params+ sent into the method are sent to Solr as-is, which is to say they are converted to Solr url style, but no special mapping is used.
 When an array is used, multiple parameters *with the same name* are generated for the Solr query. Example:
   
   solr.get 'select', :params => {:q=>'roses', :fq=>['red', 'violet']}
@@ -64,7 +73,7 @@ The paginate method returns WillPaginate ready "docs" objects, so for example in
   <%= will_paginate @solr_response["response"]["docs"] %>
 
 ===Method Missing
-The RSolr::Client class also uses method_missing for setting the request handler/path:
+The +RSolr::Client+ class also uses +method_missing+ for setting the request handler/path:
   
   solr.paintings :params => {:q=>'roses', :fq=>['red', 'violet']}
   
@@ -79,7 +88,7 @@ This works with pagination as well:
 There may be cases where the query string is too long for a GET request. RSolr solves this issue by converting hash objects into form-encoded strings:
   response = solr.music :data => {:q => "*:*"}
 
-The :data hash is serialized as a form-encoded query string, and the correct content-type headers are sent along to Solr.
+The +:data+ hash is serialized as a form-encoded query string, and the correct content-type headers are sent along to Solr.
 
 ===Sending HEAD Requests
 There may be cases where you'd like to send a HEAD request to Solr:
@@ -87,11 +96,11 @@ There may be cases where you'd like to send a HEAD request to Solr:
 
 ==Sending HTTP Headers
 Solr responds to the request headers listed here: http://wiki.apache.org/solr/SolrAndHTTPCaches
-To send header information to Solr using RSolr, just use the :headers option:
+To send header information to Solr using RSolr, just use the +:headers+ option:
   response = solr.head "admin/ping", :headers => {"Cache-Control" => "If-None-Match"}
 
 ===Building a Request
-RSolr::Client provides a method for building a request context, which can be useful for debugging or logging etc.:
++RSolr::Client+ provides a method for building a request context, which can be useful for debugging or logging etc.:
   request_context = solr.build_request "select", :data => {:q => "*:*"}, :method => :post, :headers => {}
   
 To build a paginated request use build_paginated_request:
@@ -107,7 +116,7 @@ Multiple documents via #add
   documents = [{:id=>1, :price=>1.00}, {:id=>2, :price=>10.50}]
   solr.add documents
 
-The optional :add_attributes hash can also be used to set Solr "add" document attributes:
+The optional +:add_attributes+ hash can also be used to set Solr "add" document attributes:
   solr.add documents, :add_attributes => {:commitWithin => 10}
 
 Raw XML via  #update
@@ -144,7 +153,7 @@ Delete by array of queries
   solr.optimize, :optimize_attributes => {}
 
 == Response Formats
-The default response format is Ruby. When the :wt param is set to :ruby, the response is eval'd resulting in a Hash. You can get a raw response by setting the :wt to "ruby" - notice, the string -- not a symbol. RSolr will eval the Ruby string ONLY if the :wt value is :ruby. All other response formats are available as expected, :wt=>'xml' etc..
+The default response format is Ruby. When the +:wt+ param is set to +:ruby+, the response is eval'd resulting in a Hash. You can get a raw response by setting the +:wt+ to +"ruby"+ - notice, the string -- not a symbol. RSolr will eval the Ruby string ONLY if the :wt value is :ruby. All other response formats are available as expected, +:wt=>'xml'+ etc..
 
 ===Evaluated Ruby (default)
   solr.get 'select', :params => {:wt => :ruby} # notice :ruby is a Symbol
@@ -176,27 +185,21 @@ The default response format is Ruby. When the :wt param is set to :ruby, the res
 * Send me a pull request. Bonus points for topic branches.
 
 ==Contributors
-* Antoine Latter
-* Dmitry Lihachev
-* Lucas Souza
-* Peter Kieltyka
-* Rob Di Marco
+* Nathan Witmer
 * Magnus Bergmark
-* Jonathan Rochkind
-* Chris Beer
-* Craig Smith
+* shima
 * Randy Souza
-* Colin Steele
-* Peter Kieltyka
-* Lorenzo Riccucci
-* Mike Perham
 * Mat Brown
-* Shairon Toledo
-* Matthew Rudy
-* Fouad Mardini
 * Jeremy Hinegardner
-* Nathan Witmer
-* "shima"
+* Denis Goeury
+* shairon toledo
+* Rob Di Marco
+* Peter Kieltyka
+* Mike Perham
+* Lucas Souza
+* Dmitry Lihachev
+* Antoine Latter
+* Naomi Dushay
 
 ==Author
 
@@ -204,4 +207,4 @@ Matt Mitchell <mailto:goodieboy@gmail.com>
 
 ==Copyright
 
-Copyright (c) 2008-2010 Matt Mitchell. See LICENSE for details.
+Copyright (c) 2008-2010 Matt Mitchell. See LICENSE for details.

data/Rakefile

@@ -1,15 +1,6 @@
 require 'rake'
-require 'rake/testtask'
 require 'bundler/gem_tasks'
 
-require 'rubygems/package_task'
-
-ENV['RUBYOPT'] = '-W1'
- 
-task :environment do
-  require File.dirname(__FILE__) + '/lib/rsolr'
-end
- 
 Dir['tasks/**/*.rake'].each { |t| load t }
 
-task :default => ['spec:api']
+task :default => ['spec']

data/lib/rsolr.rb

@@ -1,14 +1,6 @@
-$: << "#{File.dirname(__FILE__)}" unless $:.include? File.dirname(__FILE__)
-
-require 'rubygems'
-
 module RSolr
   
-  %W(Response Char Client Error Connection Uri Xml).each{|n|autoload n.to_sym, "rsolr/#{n.downcase}"}
-  
-  def self.version; "1.0.8" end
-  
-  VERSION = self.version
+  Dir.glob(File.expand_path("../rsolr/*.rb", __FILE__)).each{|rb_file| require(rb_file)}
   
   def self.connect *args
     driver = Class === args[0] ? args[0] : RSolr::Connection
@@ -16,7 +8,19 @@ module RSolr
     Client.new driver.new, opts
   end
   
-  # RSolr.escape
+  # RSolr.escape, which is deprecated as of 2015-02
   extend Char
   
+  # backslash escape characters that have special meaning to Solr query parser
+  # per http://lucene.apache.org/core/4_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#Escaping_Special_Characters
+  #  + - & | ! ( ) { } [ ] ^ " ~ * ? : \ /
+  # see also http://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_4_9_1/solr/solrj/src/java/org/apache/solr/client/solrj/util/ClientUtils.java
+  #   escapeQueryChars method
+  # @return [String] str with special chars preceded by a backslash
+  def self.solr_escape(str)
+    # note that the gsub will parse the escaped backslashes, as will the ruby code sending the query to Solr 
+    # so the result sent to Solr is ultimately a single backslash in front of the particular character 
+    str.gsub(/([+\-&|!\(\)\{\}\[\]\^"~\*\?:\\\/])/, '\\\\\1')
+  end
+  
 end

data/lib/rsolr/char.rb

@@ -1,9 +1,12 @@
 # A module that contains (1) string related methods
+# @deprecated remove this module when we remove the method (duh)
 module RSolr::Char
   
   # backslash everything
   # that isn't a word character
+  # @deprecated - this is incorrect Solr escaping
   def escape value
+    warn "[DEPRECATION] `RSolr.escape` is deprecated (and incorrect).  Use `RSolr.solr_escape` instead."
     value.gsub(/(\W)/, '\\\\\1')
   end
   
@@ -18,4 +21,4 @@ module RSolr::Char
   #   }.join(delim)
   # end
   
-end
+end

data/lib/rsolr/client.rb

@@ -1,7 +1,22 @@
+begin
+  require 'json'
+rescue LoadError
+end
+
 class RSolr::Client
-  
-  attr_reader :connection, :uri, :proxy, :options
-  
+
+  class << self
+    def default_wt
+      @default_wt || :ruby
+    end
+
+    def default_wt= value
+      @default_wt = value
+    end
+  end
+
+  attr_reader :connection, :uri, :proxy, :options, :update_path
+
   def initialize connection, options = {}
     @proxy = @uri = nil
     @connection = connection
@@ -13,22 +28,24 @@ class RSolr::Client
         proxy_url = options[:proxy].dup
         proxy_url << "/" unless proxy_url.nil? or proxy_url[-1] == ?/
         @proxy = RSolr::Uri.create proxy_url if proxy_url
+      elsif options[:proxy] == false
+        @proxy = false  # used to avoid setting the proxy from the environment.
       end
     end
+    @update_path = options.fetch(:update_path, 'update')
     @options = options
   end
-  
+
   # returns the request uri object.
   def base_request_uri
     base_uri.request_uri if base_uri
   end
-  
-  # returns the uri proxy if present,
-  # otherwise just the uri object.
+
+  # returns the RSolr::URI uri object.
   def base_uri
-    @proxy ? @proxy : @uri
+    @uri
   end
-  
+
   # Create the get, post, and head methods
   %W(get post head).each do |meth|
     class_eval <<-RUBY
@@ -37,7 +54,7 @@ class RSolr::Client
     end
     RUBY
   end
-  
+
   # A paginated request method.
   # Converts the page and per_page
   # arguments into "rows" and "start".
@@ -47,9 +64,9 @@ class RSolr::Client
     raise "'rows' or 'start' params should not be set when using +paginate+" if ["start", "rows"].include?(opts[:params].keys)
     execute build_paginated_request(page, per_page, path, opts)
   end
-  
+
   # POST XML messages to /update with optional params.
-  # 
+  #
   # http://wiki.apache.org/solr/UpdateXmlMessages#add.2BAC8-update
   #
   # If not set, opts[:headers] will be set to a hash with the key
@@ -64,24 +81,24 @@ class RSolr::Client
   def update opts = {}
     opts[:headers] ||= {}
     opts[:headers]['Content-Type'] ||= 'text/xml'
-    post 'update', opts
+    post opts.fetch(:path, update_path), opts
   end
-  
-  # 
+
+  #
   # +add+ creates xml "add" documents and sends the xml data to the +update+ method
-  # 
+  #
   # http://wiki.apache.org/solr/UpdateXmlMessages#add.2BAC8-update
-  # 
+  #
   # single record:
-  # solr.update(:id=>1, :name=>'one')
+  # solr.add(:id=>1, :name=>'one')
+  #
+  # add using an array
   #
-  # update using an array
-  # 
-  # solr.update(
+  # solr.add(
   #   [{:id=>1, :name=>'one'}, {:id=>2, :name=>'two'}],
   #   :add_attributes => {:boost=>5.0, :commitWithin=>10}
   # )
-  # 
+  #
   def add doc, opts = {}
     add_attributes = opts.delete :add_attributes
     update opts.merge(:data => xml.add(doc, add_attributes))
@@ -104,16 +121,16 @@ class RSolr::Client
     optimize_attributes = opts.delete :optimize_attributes
     update opts.merge(:data => xml.optimize(optimize_attributes))
   end
-  
+
   # send </rollback>
-  # 
+  #
   # http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22
-  # 
+  #
   # NOTE: solr 1.4 only
   def rollback opts = {}
     update opts.merge(:data => xml.rollback)
   end
-  
+
   # Delete one or many documents by id
   #   solr.delete_by_id 10
   #   solr.delete_by_id([12, 41, 199])
@@ -122,22 +139,22 @@ class RSolr::Client
   end
 
   # delete one or many documents by query.
-  # 
+  #
   # http://wiki.apache.org/solr/UpdateXmlMessages#A.22delete.22_by_ID_and_by_Query
-  # 
+  #
   #   solr.delete_by_query 'available:0'
   #   solr.delete_by_query ['quantity:0', 'manu:"FQ"']
   def delete_by_query query, opts = {}
     update opts.merge(:data => xml.delete_by_query(query))
   end
-  
+
   # shortcut to RSolr::Xml::Generator
   def xml
     @xml ||= RSolr::Xml::Generator.new
   end
-  
+
   # +send_and_receive+ is the main request method responsible for sending requests to the +connection+ object.
-  # 
+  #
   # "path" : A string value that usually represents a solr request handler
   # "opts" : A hash, which can contain the following keys:
   #   :method : required - the http method (:get, :post or :head)
@@ -145,7 +162,7 @@ class RSolr::Client
   #   :data : optional - post data -- if a hash is given, it's sent as "application/x-www-form-urlencoded; charset=UTF-8"
   #   :headers : optional - hash of request headers
   # All other options are passed right along to the connection's +send_and_receive+ method (:get, :post, or :head)
-  # 
+  #
   # +send_and_receive+ returns either a string or hash on a successful ruby request.
   # When the :params[:wt] => :ruby, the response will be a hash, else a string.
   #
@@ -155,18 +172,53 @@ class RSolr::Client
   # then passes the request/response into +adapt_response+.
   def send_and_receive path, opts
     request_context = build_request path, opts
-    [:open_timeout, :read_timeout].each do |k|
-      request_context[k] = @options[k]
-    end
     execute request_context
   end
-  
-  # 
+
+  #
   def execute request_context
+
     raw_response = connection.execute self, request_context
+
+    while retry_503?(request_context, raw_response)
+      request_context[:retry_503] -= 1
+      sleep retry_after(raw_response)
+      raw_response = connection.execute self, request_context
+    end
+
     adapt_response(request_context, raw_response) unless raw_response.nil?
   end
-  
+
+  def retry_503?(request_context, response)
+    return false if response.nil?
+    status = response[:status] && response[:status].to_i
+    return false unless status == 503
+    retry_503 = request_context[:retry_503]
+    return false unless retry_503 && retry_503 > 0
+    retry_after_limit = request_context[:retry_after_limit] || 1
+    retry_after = retry_after(response)
+    return false unless retry_after && retry_after <= retry_after_limit
+    true
+  end
+
+  # Retry-After can be a relative number of seconds from now, or an RFC 1123 Date.
+  # If the latter, attempt to convert it to a relative time in seconds.
+  def retry_after(response)
+    retry_after = Array(response[:headers]['Retry-After'] || response[:headers]['retry-after']).flatten.first.to_s
+    if retry_after =~ /\A[0-9]+\Z/
+      retry_after = retry_after.to_i
+    else
+      begin
+        retry_after_date = DateTime.parse(retry_after)
+        retry_after = retry_after_date.to_time - Time.now
+        retry_after = nil if retry_after < 0
+      rescue ArgumentError
+        retry_after = retry_after.to_i
+      end
+    end
+    retry_after
+  end
+
   # +build_request+ accepts a path and options hash,
   # then prepares a normalized hash to return for sending
   # to a solr connection driver.
@@ -187,7 +239,7 @@ class RSolr::Client
     opts[:proxy] = proxy unless proxy.nil?
     opts[:method] ||= :get
     raise "The :data option can only be used if :method => :post" if opts[:method] != :post and opts[:data]
-    opts[:params] = opts[:params].nil? ? {:wt => :ruby} : {:wt => :ruby}.merge(opts[:params])
+    opts[:params] = params_with_wt(opts[:params])
     query = RSolr::Uri.params_to_solr(opts[:params]) unless opts[:params].empty?
     opts[:query] = query
     if opts[:data].is_a? Hash
@@ -197,9 +249,20 @@ class RSolr::Client
     end
     opts[:path] = path
     opts[:uri] = base_uri.merge(path.to_s + (query ? "?#{query}" : "")) if base_uri
+
+    [:open_timeout, :read_timeout, :retry_503, :retry_after_limit].each do |k|
+      opts[k] = @options[k]
+    end
+
     opts
   end
-  
+
+  def params_with_wt(params)
+    return { wt: default_wt } if params.nil?
+    return params if params.key?(:wt) || params.key?('wt')
+    { wt: default_wt }.merge(params)
+  end
+
   def build_paginated_request page, per_page, path, opts
     per_page = per_page.to_s.to_i
     page = page.to_s.to_i-1
@@ -208,12 +271,7 @@ class RSolr::Client
     opts[:params]["rows"] = per_page
     build_request path, opts
   end
-  
-  #  A mixin for used by #adapt_response
-  module Context
-    attr_accessor :request, :response
-  end
-  
+
   # This method will evaluate the :body value
   # if the params[:uri].params[:wt] == :ruby
   # ... otherwise, the body is returned as is.
@@ -228,14 +286,22 @@ class RSolr::Client
     raise "The response does not have the correct keys => :body, :headers, :status" unless
       %W(body headers status) == response.keys.map{|k|k.to_s}.sort
     raise RSolr::Error::Http.new request, response unless [200,302].include? response[:status]
-    result = request[:params][:wt] == :ruby ? evaluate_ruby_response(request, response) : response[:body]
-    result.extend Context
-    result.request, result.response = request, response
-    result.is_a?(Hash) ? result.extend(RSolr::Response) : result
+
+    result = if respond_to? "evaluate_#{request[:params][:wt]}_response", true
+      send "evaluate_#{request[:params][:wt]}_response", request, response
+    else
+      response[:body]
+    end
+
+    if result.is_a?(Hash) || request[:method] == :head
+      result = RSolr::HashWithResponse.new(request, response, result)
+    end
+
+    result
   end
-  
+
   protected
-  
+
   # converts the method name for the solr request handler path.
   def method_missing name, *args
     if name.to_s =~ /^paginated?_(.+)$/
@@ -244,7 +310,7 @@ class RSolr::Client
       send_and_receive name, *args
     end
   end
-  
+
   # evaluates the response[:body],
   # attempts to bring the ruby string to life.
   # If a SyntaxError is raised, then
@@ -259,5 +325,18 @@ class RSolr::Client
       raise RSolr::Error::InvalidRubyResponse.new request, response
     end
   end
-  
+
+  def evaluate_json_response request, response
+    return response[:body] unless defined? JSON
+
+    begin
+      JSON.parse response[:body].to_s
+    rescue JSON::ParserError
+      raise RSolr::Error::InvalidJsonResponse.new request, response
+    end
+  end
+
+  def default_wt
+    self.options[:default_wt] || self.class.default_wt
+  end
 end