rsolr 0.13.0.pre → 1.0.0.beta

data/README.rdoc

@@ -1,5 +1,7 @@
 =RSolr
 
+Notice: This document is only for the the 1.0 (pre-release) in the master branch. The last stable gem release documentation can be found here: http://github.com/mwmitchell/rsolr/tree/v0.12.1
+
 A simple, extensible Ruby client for Apache Solr.
 
 == Installation:
@@ -11,23 +13,20 @@ A simple, extensible Ruby client for Apache Solr.
   require 'rsolr'
   
   # Direct connection
-  solr = RSolr.connect :url=>'http://solrserver.com'
+  solr = RSolr.connect 'http://solrserver.com'
   
   # Connecting over a proxy server
-  solr = RSolr.connect :url=>'http://solrserver.com', :proxy=>'http://user:pass@proxy.example.com:8080'
+  solr = RSolr.connect 'http://solrserver.com', :proxy=>'http://user:pass@proxy.example.com:8080'
   
   # send a request to /select
-  response = solr.select :q=>'*:*'
+  response = solr.get 'select', :q=>'*:*'
   
   # send a request to a custom request handler; /catalog
-  response = solr.request '/catalog', :q=>'*:*'
+  response = solr.get 'catalog', :q=>'*:*'
   
-  # alternative to above:
-  response = solr.catalog :q=>'*:*'
-
 == Querying
 Use the #select method to send requests to the /select handler:
-  response = solr.select({
+  response = solr.get('select', {
     :q=>'washington',
     :start=>0,
     :rows=>10
@@ -35,25 +34,23 @@ Use the #select method to send requests to the /select handler:
 
 The params sent into the method are sent to Solr as-is. The one exception is if a value is an array. When an array is used, multiple parameters *with the same name* are generated for the Solr query. Example:
   
-  solr.select :q=>'roses', :fq=>['red', 'violet']
+  solr.get 'select', :q=>'roses', :fq=>['red', 'violet']
 
 The above statement generates this Solr query:
   
-  ?q=roses&fq=red&fq=violet
+  select?q=roses&fq=red&fq=violet
 
-Use the #request method for a custom request handler path:
-  response = solr.request '/documents', :q=>'test'
-
-A shortcut for the above example use a method call instead:
-  response = solr.documents :q=>'test'
+There may be cases where the query string is too long for a GET request. RSolr solves this issue by providing a simple way to POST a query to Solr:
+  response = solr.post "select", nil, enormous_params_hash
 
+nil is passed in as the query string data. The enormous_params_hash variable ends up serialized as a form-encoded query string, and the correct content-type headers are sent along to Solr.
 
 == Updating Solr
-Updating uses native Ruby structures. Hashes are used for single documents and arrays are used for a collection of documents (hashes). These structures get turned into simple XML "messages". Raw XML strings can also be used.
+Updating us done using native Ruby objects. Hashes are used for single documents and arrays are used for a collection of documents (hashes). These objects get turned into simple XML "messages". Raw XML strings can also be used.
 
 Raw XML via  #update
-  solr.update '</commit>'
-  solr.update '</optimize>'
+  solr.update '<commit/>'
+  solr.update '<optimize/>'
 
 Single document via #add
   solr.add :id=>1, :price=>1.00
@@ -91,22 +88,26 @@ Commit & optimize shortcuts
 The default response format is Ruby. When the :wt param is set to :ruby, the response is eval'd resulting in a Hash. You can get a raw response by setting the :wt to "ruby" - notice, the string -- not a symbol. RSolr will eval the Ruby string ONLY if the :wt value is :ruby. All other response formats are available as expected, :wt=>'xml' etc..
 
 ===Evaluated Ruby (default)
-  solr.select(:wt=>:ruby) # notice :ruby is a Symbol
+  solr.get 'select', :wt=>:ruby # notice :ruby is a Symbol
 ===Raw Ruby
-  solr.select(:wt=>'ruby') # notice 'ruby' is a String
+  solr.get 'select', :wt=>'ruby' # notice 'ruby' is a String
 
 ===XML:
-  solr.select(:wt=>:xml)
+  solr.get 'select', :wt=>:xml
 ===JSON:
-  solr.select(:wt=>:json)
+  solr.get 'select', :wt=>:json
 
-You can access the original request context (path, params, url etc.) by calling the #raw method:
-  response = solr.select :q=>'*:*'
-  response.raw[:status_code]
-  response.raw[:body]
-  response.raw[:url]
+You can access the original request context (path, params, url etc.) by calling the #request method:
+  result = solr.get 'select', :q=>'*:*'
+  result.request[:uri]
+  result.request[:params]
+  etc..
 
-The raw is a hash that contains the generated params, url, path, post data, headers etc., very useful for debugging and testing.
+Similarly, the object returned has a response object. This contains any headers that Solr returned, along with the raw response body:
+  result = solr.get 'select', :q=>'*:*'
+  result.response[:headers]
+  result.response[:status]
+  result.response[:body]
 
 ==Related Resources & Projects
 * {RSolr Google Group}[http://groups.google.com/group/rsolr] -- The RSolr discussion group
@@ -127,6 +128,16 @@ The raw is a hash that contains the generated params, url, path, post data, head
   (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
 * Send me a pull request. Bonus points for topic branches.
 
+== Note on Patches/Pull Requests
+ 
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
 ==Contributors
 * Lorenzo Riccucci
 * Mike Perham
@@ -136,4 +147,12 @@ The raw is a hash that contains the generated params, url, path, post data, head
 * Fouad Mardini
 * Jeremy Hinegardner
 * Nathan Witmer
-* Craig Smith
+* Craig Smith
+
+==Author
+
+Matt Mitchell <mailto:goodieboy@gmail.com>
+
+==Copyright
+
+Copyright (c) 2008-2010 mwmitchell. See LICENSE for details.

data/VERSION

@@ -1 +1 @@
-0.13.0.pre
+1.0.0.beta

data/lib/rsolr/char.rb

@@ -0,0 +1,9 @@
+# A module that contains (1) string related methods
+module RSolr::Char
+  
+  # backslash everything that isn't a word character
+  def escape value
+    value.gsub /(\W)/, '\\\\\1'
+  end
+  
+end

data/lib/rsolr/client.rb

@@ -2,37 +2,31 @@ class RSolr::Client
   
   attr_reader :connection
   
-  # "connection" is instance of:
-  #   RSolr::Adapter::HTTP
-  #   RSolr::Adapter::Direct (jRuby only)
-  # or any other class that uses the connection "interface"
-  def initialize(connection)
+  def initialize connection
     @connection = connection
   end
   
-  # Send a request to a request handler using the method name.
-  # Also proxies to the #paginate method if the method starts with "paginate_"
-  def method_missing(method_name, *args, &blk)
-    request("/#{method_name}", *args, &blk)
+  # GET request
+  def get path = '', params = nil, headers = nil
+    send_request :get, path, params, nil, headers
   end
   
-  # sends data to the update handler
-  # data can be a string of xml, or an object that returns xml from its #to_xml method
-  def update(data, params={})
-    request '/update', params, data
+  # essentially a GET, but no response body
+  def head path = '', params = nil, headers = nil
+    send_request :head, path, params, nil, headers
   end
   
-  # send request solr
-  # params is hash with valid solr request params (:q, :fl, :qf etc..)
-  #   if params[:wt] is not set, the default is :ruby
-  #   if :wt is something other than :ruby, the raw response body is used
-  #   otherwise, a simple Hash is returned
-  #   NOTE: to get raw ruby, use :wt=>'ruby' <- a string, not a symbol like :ruby  
-  #
-  #
-  def request(path, params={}, *extra)
-    response = @connection.request(path, map_params(params), *extra)
-    adapt_response(response)
+  # A path is required for a POST since, well...
+  # the / resource doesn't do anything with a POST.
+  # Also, Solr doesn't do headers with a POST
+  def post path, data = nil, params = nil, headers = nil
+    send_request :post, path, params, data, headers
+  end
+  
+  # POST XML messages to /update with optional params
+  def update data, params = {}, headers = {}
+    headers['Content-Type'] ||= 'text/xml'
+    post 'update', data, params, headers
   end
   
   # 
@@ -43,10 +37,10 @@ class RSolr::Client
   # solr.update([{:id=>1, :name=>'one'}, {:id=>2, :name=>'two'}])
   #
   def add(doc, params={}, &block)
-    update message.add(doc, params, &block)
+    update xml.add(doc, params, &block)
   end
 
-  # send "commit" message with options
+  # send "commit" xml with options
   #
   # Options recognized by solr
   #
@@ -58,10 +52,10 @@ class RSolr::Client
   # *NOTE* :expungeDeletes is Solr 1.4 only
   #
   def commit( options = {} )
-    update message.commit( options )
+    update xml.commit( options )
   end
 
-  # send "optimize" message with options.
+  # send "optimize" xml with options.
   #
   # Options recognized by solr
   #
@@ -73,61 +67,89 @@ class RSolr::Client
   # *NOTE* :expungeDeletes is Solr 1.4 only
   #
   def optimize( options = {} )
-    update message.optimize( options )
+    update xml.optimize( options )
   end
 
   # send </rollback>
   # NOTE: solr 1.4 only
   def rollback
-    update message.rollback
+    update xml.rollback
   end
 
   # Delete one or many documents by id
   #   solr.delete_by_id 10
   #   solr.delete_by_id([12, 41, 199])
   def delete_by_id(id)
-    update message.delete_by_id(id)
+    update xml.delete_by_id(id)
   end
 
   # delete one or many documents by query
   #   solr.delete_by_query 'available:0'
   #   solr.delete_by_query ['quantity:0', 'manu:"FQ"']
   def delete_by_query(query)
-    update message.delete_by_query(query)
+    update xml.delete_by_query(query)
   end
   
   # shortcut to RSolr::Message::Generator
-  def message
-    @message ||= RSolr::Message::Generator.new
+  def xml
+    @xml ||= RSolr::Xml::Generator.new
   end
   
-  protected
+  def send_request method, path, params, data, headers
+    params = map_params params
+    uri, data, headers = build_request path, params, data, headers
+    request_context = {:connection=>connection, :method => method, :uri => uri, :data => data, :headers => headers, :params => params}
+    begin
+      response = data ? connection.send(method, uri, data, headers) : connection.send(method, uri, headers)
+    rescue
+     $!.extend(RSolr::Error::SolrContext).request = request_context
+     raise $!
+    end
+    raise "The connection adapter returned an unexpected object" unless response.is_a?(Hash)
+    raise RSolr::Error::Http.new request_context, response unless [200,302].include?(response[:status])
+    adapt_response request_context, response
+  end
   
-  # sets default params etc.. - could be used as a mapping hook
-  # type of request should be passed in here? -> map_params(:query, {})
-  def map_params(params)
-    params||={}
-    {:wt=>:ruby}.merge(params)
+  def map_params params
+    params = params.nil? ? {} : params.dup
+    params[:wt] ||= :ruby
+    params
   end
-
-  # "connection_response" must be a hash with the following keys:
-  #   :params - a sub hash of standard solr params
-  #   : body - the raw response body from the solr server
-  # This method will evaluate the :body value if the params[:wt] == :ruby
-  # otherwise, the body is returned
-  # The return object has a special method attached called #raw
-  # This method gives you access to the original response from the connection,
-  # so you can access things like the actual :url sent to solr,
-  # the raw :body, original :params and original :data
-  def adapt_response(connection_response)
-    data = connection_response[:body]
-    # if the wt is :ruby, evaluate the ruby string response
-    if connection_response[:params][:wt] == :ruby
-      data = Kernel.eval(data)
+  
+  def build_request path, params, data, headers
+    params ||= {}
+    headers ||= {}
+    request_uri = params.any? ? "#{path}?#{RSolr::Uri.params_to_solr params}" : path
+    if data
+      if data.is_a? Hash
+        data = RSolr::Uri.params_to_solr data
+        headers['Content-Type'] ||= 'application/x-www-form-urlencoded'
+      end
+    end
+    [request_uri, data, headers]
+  end
+  
+  # This method will evaluate the :body value
+  # if the params[:uri].params[:wt] == :ruby
+  # ... otherwise, the body is returned as is.
+  # The return object has a special method attached called #context.
+  # This method gives you access to the original
+  # request and response from the connection.
+  # This method will raise an InvalidRubyResponse
+  # if the :wt => :ruby and the body
+  # couldn't be evaluated.
+  def adapt_response request, response
+    data = response[:body]
+    if request[:params][:wt] == :ruby
+      begin
+        data = Kernel.eval data.to_s
+      rescue SyntaxError
+        raise RSolr::Error::InvalidRubyResponse.new request, response
+      end
     end
-    # attach a method called #raw that returns the original connection response value
-    def data.raw; @raw end
-    data.send(:instance_variable_set, '@raw', connection_response)
+    data.extend Module.new.instance_eval{attr_accessor :request, :response; self}
+    data.request = request
+    data.response = response
     data
   end
   

data/lib/rsolr/error.rb

@@ -0,0 +1,122 @@
+module RSolr::Error
+  
+  module SolrContext
+    
+    attr_accessor :request, :response
+    
+    def to_s
+      m = "#{super.to_s}"
+      
+      if response
+        m << " - #{response[:status]} #{Http::STATUS_CODES[response[:status].to_i]}"
+        details = parse_solr_error_response response[:body]
+        m << "Error: #{details}\n" if details
+      end
+      
+      m << "\n" + self.backtrace[0..10].join("\n")
+      m << "\n\nSolr Request:"
+      m << "\n  Method: #{request[:method].to_s.upcase}"
+      m << "\n  Base URL: #{request[:connection].uri.to_s}"
+      m << "\n  URL: #{request[:uri]}"
+      m << "\n  Params: #{request[:params].inspect}"
+      m << "\n  Data: #{request[:data].inspect}" if request[:data]
+      m << "\n  Headers: #{request[:headers].inspect}"
+      if response
+        m << "\n\nSolr Response:"
+        m << "\n  Code: #{response[:status]}"
+        m << "\n  Headers: #{response[:headers].inspect}"
+      end
+      m
+    end
+    
+    protected
+    
+    def parse_solr_error_response body
+      begin
+        info = body.scan(/<pre>(.*)<\/pre>/mi)[0]
+        partial = info.to_s.split("\n")[0..10]
+        partial.join("\n").gsub("&gt;", ">").gsub("&lt;", "<")
+      rescue
+        nil
+      end
+    end
+    
+  end
+  
+  class Http < RuntimeError
+    
+    include SolrContext
+    
+    # ripped right from ActionPack
+    # Defines the standard HTTP status codes, by integer, with their
+    # corresponding default message texts.
+    # Source: http://www.iana.org/assignments/http-status-codes
+    STATUS_CODES = {
+      100 => "Continue",
+      101 => "Switching Protocols",
+      102 => "Processing",
+
+      200 => "OK",
+      201 => "Created",
+      202 => "Accepted",
+      203 => "Non-Authoritative Information",
+      204 => "No Content",
+      205 => "Reset Content",
+      206 => "Partial Content",
+      207 => "Multi-Status",
+      226 => "IM Used",
+
+      300 => "Multiple Choices",
+      301 => "Moved Permanently",
+      302 => "Found",
+      303 => "See Other",
+      304 => "Not Modified",
+      305 => "Use Proxy",
+      307 => "Temporary Redirect",
+
+      400 => "Bad Request",
+      401 => "Unauthorized",
+      402 => "Payment Required",
+      403 => "Forbidden",
+      404 => "Not Found",
+      405 => "Method Not Allowed",
+      406 => "Not Acceptable",
+      407 => "Proxy Authentication Required",
+      408 => "Request Timeout",
+      409 => "Conflict",
+      410 => "Gone",
+      411 => "Length Required",
+      412 => "Precondition Failed",
+      413 => "Request Entity Too Large",
+      414 => "Request-URI Too Long",
+      415 => "Unsupported Media Type",
+      416 => "Requested Range Not Satisfiable",
+      417 => "Expectation Failed",
+      422 => "Unprocessable Entity",
+      423 => "Locked",
+      424 => "Failed Dependency",
+      426 => "Upgrade Required",
+
+      500 => "Internal Server Error",
+      501 => "Not Implemented",
+      502 => "Bad Gateway",
+      503 => "Service Unavailable",
+      504 => "Gateway Timeout",
+      505 => "HTTP Version Not Supported",
+      507 => "Insufficient Storage",
+      510 => "Not Extended"
+    }
+    
+    def initialize request, response
+      @request, @response = request, response
+    end
+    
+  end
+  
+  # Thrown if the :wt is :ruby
+  # but the body wasn't succesfully parsed/evaluated
+  class InvalidRubyResponse < Http
+    
+  end
+  
+end

data/lib/rsolr/http.rb

@@ -0,0 +1,106 @@
+module RSolr
+  
+  class Http
+    
+    attr_reader :uri, :proxy, :options
+    
+    def initialize base_url, options = {}
+      @options = options
+      @uri = base_url
+      @proxy = options[:proxy]
+    end
+    
+    def base_uri
+      @proxy ? @proxy.request_uri : @uri.request_uri
+    end
+    
+    def http
+      @http ||= (
+        http = if proxy
+          proxy_user, proxy_pass = proxy.userinfo.split(/:/) if proxy.userinfo
+          Net::HTTP.Proxy(proxy.host, proxy.port, proxy_user, proxy_pass).new uri.host, uri.port
+        else
+          Net::HTTP.new uri.host, uri.port
+        end
+      
+        http.use_ssl = uri.port == 443 || uri.instance_of?(URI::HTTPS)
+  
+        if options[:timeout] && options[:timeout].is_a?(Integer)
+          http.open_timeout = options[:timeout]
+          http.read_timeout = options[:timeout]
+        end
+  
+        if options[:pem] && http.use_ssl?
+          http.cert = OpenSSL::X509::Certificate.new(options[:pem])
+          http.key = OpenSSL::PKey::RSA.new(options[:pem])
+          http.verify_mode = OpenSSL::SSL::VERIFY_PEER
+        else
+          http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+        end
+
+        if options[:debug_output]
+          http.set_debug_output(options[:debug_output])
+        end
+
+        http
+      )
+    end
+  
+    # send in path w/query
+    def get request_uri, headers = {}
+      req = setup_raw_request Net::HTTP::Get, request_uri, headers
+      perform_request http, req
+    end
+  
+    # send in path w/query
+    def head request_uri, headers = {}
+      req = setup_raw_request Net::HTTP::Head, request_uri, headers
+      perform_request http, req
+    end
+  
+    # send in path w/query
+    def post request_uri, data, headers = {}
+      req = setup_raw_request Net::HTTP::Post, request_uri, headers
+      req.body = data if data
+      perform_request http, req
+    end
+    
+    def perform_request http, request
+      begin
+        response = http.request request
+        {:status => response.code.to_i, :headers => response.to_hash, :body => response.body}
+      rescue NoMethodError
+        $!.message == "undefined method `closed?' for nil:NilClass" ?
+          raise(Errno::ECONNREFUSED.new) :
+          raise($!)
+      end
+    end
+    
+    def setup_raw_request http_method, request_uri, headers = {}
+      raw_request = http_method.new "#{base_uri}#{request_uri}"
+      raw_request.initialize_http_header headers
+      raw_request.basic_auth username, password if options[:basic_auth]
+      if options[:digest_auth]
+        res = http.head(request_uri, headers)
+        if res['www-authenticate'] != nil && res['www-authenticate'].length > 0
+          raw_request.digest_auth username, password, res
+        end
+      end
+      raw_request
+    end
+  
+    def credentials
+      options[:basic_auth] || options[:digest_auth]
+    end
+
+    def username
+      credentials[:username]
+    end
+
+    def password
+      credentials[:password]
+    end
+  
+  end
+  
+end

data/lib/rsolr/uri.rb

@@ -0,0 +1,55 @@
+module RSolr::Uri
+  
+  def self.create url
+    ::URI.parse url[-1] == ?/ ? url : "#{url}/"
+  end
+  
+  # Returns a query string param pair as a string.
+  # Both key and value are escaped.
+  def build_param(k,v)
+    "#{escape_query_value(k)}=#{escape_query_value(v)}"
+  end
+
+  # Return the bytesize of String; uses String#size under Ruby 1.8 and
+  # String#bytesize under 1.9.
+  if ''.respond_to?(:bytesize)
+    def bytesize(string)
+      string.bytesize
+    end
+  else
+    def bytesize(string)
+      string.size
+    end
+  end
+
+  # Creates a Solr based query string.
+  # Keys that have arrays values are set multiple times:
+  #   params_to_solr(:q => 'query', :fq => ['a', 'b'])
+  # is converted to:
+  #   ?q=query&fq=a&fq=b
+  def params_to_solr(params)
+    mapped = params.map do |k, v|
+      next if v.to_s.empty?
+      if v.class == Array
+        params_to_solr(v.map { |x| [k, x] })
+      else
+        build_param k, v
+      end
+    end
+    mapped.compact.join("&")
+  end
+  
+  # Performs URI escaping so that you can construct proper
+  # query strings faster.  Use this rather than the cgi.rb
+  # version since it's faster.
+  # (Stolen from Rack).
+  def escape_query_value(s)
+    s.to_s.gsub(/([^ a-zA-Z0-9_.-]+)/n) {
+      #'%'+$1.unpack('H2'*$1.size).join('%').upcase
+      '%'+$1.unpack('H2'*bytesize($1)).join('%').upcase
+    }.tr(' ', '+')
+  end
+  
+  extend self
+  
+end