wku-ruby-rets 2.0.7

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7334380eaedc3dd2482334b030f6c3f9c561351f
+  data.tar.gz: 2d959fe4c4c2077cd73f4593ac89131d81fa45a4
+SHA512:
+  metadata.gz: 802e0d0fe5d4b17305bea448083ea16fde66ed90ab4d92a3381ae30b1a3dbb22afe9484bc5fdf6bd01c0bce9834505b70292fd56722a4dc769c787890f5b47ed
+  data.tar.gz: ea43b713311a72b218819f1ea93ffa89af2ebb5eabd5c224f46143e232e84be310878b1e10272269b94af2f93893bbfe9ce57b893e55db954bb24ed8e8ec63d1

data/CHANGELOG.md

@@ -0,0 +1,104 @@
+# Overview
+
+## 2.0.8 [In Progress]
+
+### Features
+  * Added support for using a HTTP Proxy
+  * Streaming can be disabled for `get_metadata` calls
+
+### Fixes
+  * Disabled gzip while streaming is enabled to fix parsing issues, it's still enabled when streaming is disabled (`:disable_stream => true`)
+  * No longer errors if Set-Cookie is returned with an invalid value (RAPRETS)
+
+## 2.0.7
+
+Confirmed to work with Ruby 2.0.0-p0
+
+### Fixes
+  * `RETS-Version` is no longer set to `RETS/1.7` unless an error `20037` is returned and no other one is specified
+  * `RETS-Version` can be set based on a HTTP 200 response for systems which only pass it after authentication (Innovia)
+
+## 2.0.6
+
+### Features
+  * System name added to `rets_data[:system_id]` when pulling metadata out
+  * `client.request_size` and `client.request_hash` can be used when streaming is disabled in `client.search`
+  * Added `client.request_time` which gives the time taken in seconds for a `client.search` or `client.get_object` to run
+
+### Fixes
+  * Fixed an authentication issue when using Retsiq where it would return 20041 in place of HTTP 401
+  * Fixed errors when streaming with an unknown Content-Length and chunked encoding
+  * Fixed a block being required when making a `client.search` call with `:count_mode => :only`
+  * Fixed character encoding that was causing issues on some RETS servers (Paul Trippett)
+  * Fixed an exception when Rapattoni RETS servers returned a "RETS-STATUS" XML tag on a multipart `client.get_object` call
+  * Fixed a `RETS::APIError` code 0 error under some RETS systems when making a call to `client.get_object` with `:location`
+
+## 2.0.5
+
+### Features
+  * Make `client.rets_data` available immediately when calling `client.search` rather than having to wait until it finishes (Paul Trippett)
+  * `:rets_version` can be passed to `client.login` without the User-Agent fields being set, for RETS servers that require the version to be passed initially
+
+### Fixes
+  * Default to HTTP Digest authentication when Digest and Basic are provided (Paul Trippett)
+  * Fixed authentication header parser if a server returns Basic/Digest and the Digest data goes stale (Paul Trippett)
+  * Fixed query strings from the initial login request not being saved when discovery service URLs (Paul Trippett)
+
+## 2.0.4
+
+### Features
+  * Added support for RETS servers that use digest authentication without the quality of protection flag (MRIS)
+  * Added SSL support (Paul Trippett)
+
+### Fixes
+  * Fixed metadata parsing breaking if a field wasn't filled out (Paul Trippett)
+  * Fixed multipart parsing for `client.get_object` if a part is blank
+
+## 2.0.3
+
+### Fixes
+  * Fixed a stack overflow due to how Interealty handles User-Agent authentication errors
+
+## 2.0.2
+
+### Features
+  * Dropped support for TimeoutSeconds, instead if an HTTP 401 is received after a successful request then a reauthentication is forced. Provides better compatibility with how some RETS implementations handle sessions
+
+### Fixes
+  * Client methods no longer return the HTTP request
+  * Requests will correctly be called after a HTTP digest becomes stale
+
+## 2.0.1
+
+### API Changes
+  * `client.login` will now raise `ResponseError` errors if the RETS tag cannot be found in the response
+  * `client.login` added the ability to pass `:rets_version` to force the RETS Version used in HTTP requests. Provides a small speedup as it can skip one HTTP request depending on the RETS implementation
+  * `client.get_object` can return both Content-Description or Description rather than just Description. Also will return Preferred
+
+### Features
+  * Added support for TimeoutSeconds, after the timeout passes the gem seamlessly reauthenticates
+  * Improved the edge case handling for authentication requests to greatly increase compatability with logging into any RETS based system
+
+### Fixes
+  * Object multipart parsing no longer fails if the boundary is wrapped in quotes
+  * Response parsing won't fail if the RETS server uses odd casing for the "ReplyText" and "ReplyCode" args in RETS
+
+## 2.0.0
+
+### API Changes
+  * `client.logout` will now raise `CapabilityNotFound` errors if it's unsupported
+  * `client.get_object` now requires a block which is yielded to rather than returning an array of the content
+  * `client.get_object` headers are now returned in lowercase form ("content-id" not "Content-ID" and so on)
+  * `RETS::Client.login` now uses `:useragent => {:name => "Foo", :password => "Bar"}` to pass User Agent data
+  * `RETS::Client.login` no longer implies the User-Agent username or password by the primary username and password
+
+### Features
+  * Added support for Count, Offset, Select and RestrictedIndicators in `client.search`
+  * Added support for Location in `client.get_object`
+  * RETS reply code, text and other data such as count or delimiter can be gotten through `client.rets_data` after the call is finished
+
+### Fixes
+  * Redid how authentication is handled, no longer implies HTTP Basic auth when using RETS-UA-Authorization
+  * RETS-Version is now used for RETS-UA-Authorization when available with "RETS/1.7" as a fallback
+  * Exceptions are now raised consistently and have been simplifed to `APIError`, `HTTPError`, `Unauthorized` and `CapabilityNotFound`
+  * `HTTPError` and `APIError` now include the reply text and code in `reply_code` and `reply_text`

data/MIT-LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2011 Placester, Inc
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,43 @@
+Overview
+===
+Simplifies the process of dealing with RETS servers. Without having to worry about the various authentication methods or edge cases associated with dealing with RETS. Should work against all 1.x implementations.
+
+Compability
+-
+Tested against Ruby 1.9.3, 2.0.0, RBX and JRuby, build history is available [here](http://travis-ci.org/Placester/ruby-rets).
+
+<img src="https://secure.travis-ci.org/Placester/ruby-rets.png?branch=master&.png"/>
+
+Documentation
+-
+See http://rubydoc.info/github/Placester/ruby-rets/master/frames for full documentation.
+
+Examples
+-
+
+    client = RETS::Client.login(:url => "http://foobar.com/rets/Login", :username => "foo", :password => "bar")
+    client.search(:search_type => :Property, :class => :RES, :query => "(ListPrice=50000-)") do |data|
+      # RETS data in key/value format, as COMPACT-DECODED
+    end
+
+    client.get_object(:resource => :Property, :type => :Photo, :location => false, :id => "1:0:*") do |headers, content|
+      puts "Object-ID #{headers"object-id"]}, Content-ID #{headers["content-id"]}, Description #{["description"]}"
+      puts "Data"
+      puts content
+    end
+
+VCR / WebMock
+-
+Due to the streaming parser, the search features won't work with a library like VCR or Ephemeral Response. For WebMock, you can use the below patch to enable support for saving the HTTP requests to speed up your own tests.
+
+    module Net
+        module WebMockHTTPResponse
+            def self.extended(response)
+                response.instance_variable_set(:@socket, StringIO.new(response.body))
+            end
+        end
+    end
+
+License
+-
+Licensed under MIT

data/Rakefile

@@ -0,0 +1,12 @@
+require "bundler"
+Bundler.setup
+
+require "rake"
+require "rspec"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new("spec") do |spec|
+  spec.pattern = "spec/**/*_spec.rb"
+end
+
+task :default => :spec

data/lib/rets/base/core.rb

@@ -0,0 +1,303 @@
+# For more information on what the possible values of fields that are passed to the RETS server can be, see {http://www.rets.org/documentation}.
+module RETS
+  module Base
+    class Core
+      GET_OBJECT_DATA = ["object-id", "description", "content-id", "content-description", "location", "content-type", "preferred"]
+
+      # Can be called after any {RETS::Base::Core} call that hits the RETS Server.
+      # @return [String] How big the request was
+      attr_reader :request_size
+
+      # Can be called after any {RETS::Base::Core} call that hits the RETS Server.
+      # @return [String] SHA1 hash of the request
+      attr_reader :request_hash
+
+      # Can be called after any {#get_object} or {#search} call that hits the RETS Server.
+      # @return [Float] How long the request took
+      attr_reader :request_time
+
+      # Can be called after any {RETS::Base::Core} call that hits the RETS Server.
+      # @return [Hash]
+      #   Gives access to the miscellaneous RETS data, such as reply text, code, delimiter, count and so on depending on the API call made.
+      #   * *text* (String) - Reply text from the server
+      #   * *code* (String) - Reply code from the server
+      attr_reader :rets_data
+
+      def initialize(http, urls)
+        @http = http
+        @urls = urls
+      end
+
+      ##
+      # Attempts to logout of the RETS server.
+      #
+      # @raise [RETS::CapabilityNotFound]
+      # @raise [RETS::APIError]
+      # @raise [RETS::HTTPError]
+      def logout
+        unless @urls[:logout]
+          raise RETS::CapabilityNotFound.new("No Logout capability found for given user.")
+        end
+
+        @http.request(:url => @urls[:logout])
+
+        nil
+      end
+
+      ##
+      # Whether the RETS server has the requested capability.
+      #
+      # @param [Symbol] type Lowercase of the capability, "getmetadata", "getobject" and so on
+      # @return [Boolean]
+      def has_capability?(type)
+        @urls.has_key?(type)
+      end
+
+      ##
+      # Requests metadata from the RETS server.
+      #
+      # @param [Hash] args
+      # @option args [String] :type Metadata to request, the same value if you were manually making the request, "METADATA-SYSTEM", "METADATA-CLASS" and so on
+      # @option args [String] :id Filter the data returned by ID, "*" would return all available data
+      # @option args [Integer, Optional] :read_timeout How many seconds to wait before giving up
+      # @option args [Boolean, Optional] :disable_stream Disables the streaming setup for data and instead loads it all and then parses
+      #
+      # @yield For every group of metadata downloaded
+      # @yieldparam [String] :type Type of data that was parsed with "METADATA-" stripped out, for "METADATA-SYSTEM" this will be "SYSTEM"
+      # @yieldparam [Hash] :attrs Attributes of the data, generally *Version*, *Date* and *Resource* but can vary depending on what metadata you requested
+      # @yieldparam [Array] :metadata Array of hashes with metadata info
+      #
+      # @raise [RETS::CapabilityNotFound]
+      # @raise [RETS::APIError]
+      # @raise [RETS::HTTPError]
+      # @see #rets_data
+      # @see #request_size
+      # @see #request_hash
+      def get_metadata(args, &block)
+        raise ArgumentError, "No block passed" unless block_given?
+
+        unless @urls[:getmetadata]
+          raise RETS::CapabilityNotFound.new("No GetMetadata capability found for given user.")
+        end
+
+        @request_size, @request_hash, @request_time, @rets_data = nil, nil, nil, nil
+
+        start = Time.now.utc.to_f
+        @http.request(:url => @urls[:getmetadata], :read_timeout => args[:read_timeout], :disable_compression => !args[:disable_stream], :params => {:Format => :COMPACT, :Type => args[:type], :ID => args[:id]}) do |response|
+          if args[:disable_stream]
+            stream = StringIO.new(response.body)
+            @request_time = Time.now.utc.to_f - start
+          else
+            stream = RETS::StreamHTTP.new(response)
+          end
+
+          sax = RETS::Base::SAXMetadata.new(block)
+          Nokogiri::XML::SAX::Parser.new(sax).parse_io(stream)
+
+          if args[:disable_stream]
+            @request_size, @request_hash = response.body.length, Digest::SHA1.hexdigest(response.body)
+          else
+            @request_size, @request_hash = stream.size, stream.hash
+          end
+
+          @rets_data = sax.rets_data
+        end
+
+        nil
+      end
+
+      ##
+      # Requests an object from the RETS server.
+      #
+      # @param [Hash] args
+      # @option args [String] :resource Resource to load, typically *Property*
+      # @option args [String] :type Type of object you want, usually *Photo*
+      # @option args [String] :id What objects to return
+      # @option args [Array, Optional] :accept Array of MIME types to accept, by default this is *image/png*, *image/gif* and *image/jpeg*
+      # @option args [Boolean, Optional] :location Return the location of the object rather than the contents of it
+      # @option args [Integer, Optional] :read_timeout How many seconds to wait before timing out
+      #
+      # @yield For every object downloaded
+      # @yieldparam [Hash] :headers Object headers
+      #     * *object-id* (String) - Objects ID
+      #     * *content-id* (String) - Content ID
+      #     * *content-type* (String) - MIME type of the content
+      #     * *description* (String, Optional) - A description of the object
+      #     * *location* (String, Optional) - Where the file is located, only returned is *location* is true
+      # @yieldparam [String, Optional] :content Content for the object, not called when *location* is set
+      #
+      # @raise [RETS::CapabilityNotFound]
+      # @raise [RETS::APIError]
+      # @raise [RETS::HTTPError]
+      # @see #rets_data
+      # @see #request_size
+      # @see #request_hash
+      def get_object(args, &block)
+        raise ArgumentError, "No block passed" unless block_given?
+
+        unless @urls[:getobject]
+          raise RETS::CapabilityNotFound.new("No GetObject capability found for given user.")
+        end
+
+        req = {:url => @urls[:getobject], :read_timeout => args[:read_timeout], :headers => {}}
+        req[:params] = {:Resource => args[:resource], :Type => args[:type], :Location => (args[:location] ? 1 : 0), :ID => args[:id]}
+        if args[:accept].is_a?(Array)
+          req[:headers]["Accept"] = args[:accept].join(",")
+        else
+          req[:headers]["Accept"] = "image/png,image/gif,image/jpeg"
+        end
+
+        # Will get swapped to a streaming call rather than a download-and-parse later, easy to do as it's called with a block now
+        start = Time.now.utc.to_f
+
+        @request_size, @request_hash, @request_time, @rets_data = nil, nil, nil, nil
+        @http.request(req) do |response|
+          body = response.read_body
+
+          @request_time = Time.now.utc.to_f - start
+          @request_size, @request_hash = body.length, Digest::SHA1.hexdigest(body)
+
+          # Make sure we aren't erroring
+          if body =~ /(<RETS (.+)\>)/
+            # RETSIQ (and probably others) return a <RETS> tag on a location request without any error inside
+            # since parsing errors out of full image data calls is a tricky pain. We're going to keep the
+            # loose error checking, but will confirm that it has an actual error code
+            code, text = @http.get_rets_response(Nokogiri::XML($1).xpath("//RETS").first)
+            unless code == "0"
+              @rets_data = {:code => code, :text => text}
+
+              if code == "20403"
+                return
+              else
+                raise RETS::APIError.new("#{code}: #{text}", code, text)
+              end
+            end
+          end
+
+          # Using a wildcard somewhere
+          if response.content_type == "multipart/parallel"
+            boundary = response.type_params["boundary"]
+            boundary.gsub!(/^"|"$/, "")
+
+            parts = body.split("--#{boundary}\r\n")
+            parts.last.gsub!("\r\n--#{boundary}--", "")
+            parts.each do |part|
+              part.strip!
+              next if part == ""
+
+              headers, content = part.split("\r\n\r\n", 2)
+
+              parsed_headers = {}
+              headers.split("\r\n").each do |line|
+                name, value = line.split(":", 2)
+                next unless value and value != ""
+
+                parsed_headers[name.downcase] = value.strip
+              end
+
+              # Check off the first children because some Rap Rets seems to use RETS-Status
+              # and it will include it with an object while returning actual data.
+              # It only does this for multipart requests, single image pulls will use <RETS> like it should.
+              if parsed_headers["content-type"] == "text/xml"
+                code, text = @http.get_rets_response(Nokogiri::XML(content).children.first)
+                next if code == "20403"
+              end
+
+              if block.arity == 1
+                yield parsed_headers
+              else
+                yield parsed_headers, content
+              end
+
+            end
+
+          # Either text (error) or an image of some sorts, which is irrelevant for this
+          else
+            headers = {}
+            GET_OBJECT_DATA.each do |field|
+              next unless response.header[field] and response.header[field] != ""
+              headers[field] = response.header[field].strip
+            end
+
+            if block.arity == 1
+              yield headers
+            else
+              yield headers, body
+            end
+          end
+        end
+
+        nil
+      end
+
+      ##
+      # Searches the RETS server for data.
+      #
+      # @param [Hash] args
+      # @option args [String] :search_type What to search on, typically *Property*, *Office* or *Agent*
+      # @option args [String] :class What class of data to return, varies between RETS implementations and can be anything from *1* to *ResidentialProperty*
+      # @option args [String] :query How to filter data, should be unescaped as CGI::escape will be called on the string
+      # @option args [Symbol, Optional] :count_mode Either *:only* to return just the total records found or *:both* to get count and records returned
+      # @option args [Integer, Optional] :limit Limit total records returned
+      # @option args [Integer, Optional] :offset Offset to start returning records from
+      # @option args [Array, Optional] :select Restrict the fields the RETS server returns
+      # @option args [Boolean, Optional] :standard_names Whether to use standard names for all fields
+      # @option args [String, Optional] :restricted String to show in place of a field value for any restricted fields the user cannot see
+      # @option args [Integer, Optional] :read_timeout How long to wait for data from the socket before giving up
+      # @option args [Boolean, Optional] :disable_stream Disables the streaming setup for data and instead loads it all and then parses
+      #
+      # @yield Called for every <DATA></DATA> group from the RETS server
+      # @yieldparam [Hash] :data One record of data from the RETS server
+      #
+      # @raise [RETS::CapabilityNotFound]
+      # @raise [RETS::APIError]
+      # @raise [RETS::HTTPError]
+      # @see #rets_data
+      # @see #request_size
+      # @see #request_hash
+      def search(args, &block)
+        if !block_given? and args[:count_mode] != :only
+          raise ArgumentError, "No block found"
+        end
+
+        unless @urls[:search]
+          raise RETS::CapabilityNotFound.new("Cannot find URL for Search call")
+        end
+
+        req = {:url => @urls[:search], :read_timeout => args[:read_timeout], :disable_compression => !args[:disable_stream]}
+        req[:params] = {:Format => "COMPACT-DECODED", :SearchType => args[:search_type], :QueryType => "DMQL2", :Query => args[:query], :Class => args[:class], :Limit => args[:limit], :Offset => args[:offset], :RestrictedIndicator => args[:restricted]}
+        req[:params][:Select] = args[:select].join(",") if args[:select].is_a?(Array)
+        req[:params][:StandardNames] = 1 if args[:standard_names]
+
+        if args[:count_mode] == :only
+          req[:params][:Count] = 2
+        elsif args[:count_mode] == :both
+          req[:params][:Count] = 1
+        end
+
+        @request_size, @request_hash, @request_time, @rets_data = nil, nil, nil, {}
+
+        start = Time.now.utc.to_f
+        @http.request(req) do |response|
+          if args[:disable_stream]
+            stream = StringIO.new(response.body)
+            @request_time = Time.now.utc.to_f - start
+          else
+            stream = RETS::StreamHTTP.new(response)
+          end
+
+          sax = RETS::Base::SAXSearch.new(@rets_data, block)
+          Nokogiri::XML::SAX::Parser.new(sax).parse_io(stream)
+
+          if args[:disable_stream]
+            @request_size, @request_hash = response.body.length, Digest::SHA1.hexdigest(response.body)
+          else
+            @request_size, @request_hash = stream.size, stream.hash
+          end
+        end
+
+        nil
+      end
+    end
+  end
+end