down 3.0.0 → 3.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c16a220821aeb2a11910334c3598ccb5b823473d
-  data.tar.gz: 21b1426c6169e82627fb445cbe0526456e9c9f19
+  metadata.gz: 29dd2b9e7c612e2be576964dc0b35203d4e6b038
+  data.tar.gz: 3f9146bd292d11b4bcd82b196b8c59a4125cff2d
 SHA512:
-  metadata.gz: 97447da54fba5009a0dca7b3013906dc5c0f707620b2ab98bd031f85e0b02180829188bb9df31daeacdcc49138aba024f630a2d9a3c5170d6f4119add90ab2c8
-  data.tar.gz: '09498dc3f7ad7256700f42fb536b76293d6f3825e2b686eb091772b22e366116a346130832ea46d28b739fe76e99f477575e3d003efa852696c8fb492128243e'
+  metadata.gz: 5dba71a205bb2f2e0353774e8801cf8692e1c0e5605bdb62810f9bd1ea16bc0076e8d17f2edaa7986ad019675a723c91b693bd8c8f9f19a8637f6aa2630ad040
+  data.tar.gz: d1d810b2defabd1a460dcd614e65d578b9c27f2ac6008efb5fd3d56c3a080cb12d721a4196383b9eb1349c7939cce9ff07716692a6c803f13eb095d136fd9aa4

data/README.md

@@ -57,29 +57,6 @@ Down.download("http://user:password@example.org")
 Down.open("http://user:password@example.org")
 ```
 
-### Download errors
-
-There are a lot of ways in which a download can fail:
-
-* Response status was 4xx or 5xx
-* Domain was not found
-* Timeout occurred
-* URL is invalid
-* ...
-
-Down attempts to unify all of these exceptions into one `Down::NotFound` error
-(because this is what actually happened from the outside perspective). If you
-want to retrieve the original error raised, in Ruby 2.1+ you can use
-`Exception#cause`:
-
-```rb
-begin
-  Down.download("http://example.com")
-rescue Down::Error => exception
-  exception.cause #=> #<Timeout::Error>
-end
-```
-
 ## Streaming
 
 Down has the ability to retrieve content of the remote file *as it is being
@@ -162,7 +139,7 @@ Down::ChunkedIO.new(...)
 * `:on_close` – called when streaming finishes or IO is closed
 * `:data` - custom data that you want to store (returned by `#data`)
 * `:rewindable` - whether to cache retrieved data into a file (defaults to `true`)
-* `:encoding` - force content to be returned in specified encoding (defaults to ASCII-8BIT)
+* `:encoding` - force content to be returned in specified encoding (defaults to `Encoding::BINARY`)
 
 Here is an example of wrapping streaming MongoDB files:
 
@@ -182,30 +159,78 @@ io = Down::ChunkedIO.new(
 )
 ```
 
-## open-uri + Net::HTTP
+### Exceptions
+
+Down tries to recognize various types of exceptions and re-raise them as one of
+the `Down::Error` subclasses. This is Down's exception hierarchy:
+
+* `Down::Error`
+  * `Down::TooLarge`
+  * `Down::NotFound`
+    * `Down::InvalidUrl`
+    * `Down::TooManyRedirects`
+    * `Down::ResponseError`
+      * `Down::ClientError`
+      * `Down::ServerError`
+    * `Down::ConnectionError`
+    * `Down::TimeoutError`
+    * `Down::SSLError`
+
+## Backends
+
+By default Down implements `Down.download` and `Down.open` using the built-in
+[open-uri] + [Net::HTTP] Ruby standard libraries. However, there are other
+backends as well:
+
+```rb
+require "down/net_http" # uses open-uri + Net::HTTP
+require "down/http"     # uses HTTP.rb gem
+```
+
+When a backend is loaded, is overrides `Down.download` and `Down.open` methods,
+but it's recommended you always use the backends explicitly:
 
-Then [open-uri] + Net::HTTP is the default backend, loaded by requiring `down`
-or `down/net_http`:
+```rb
+# not recommended
+Down.download("...")
+Down.open("...")
+
+# recommended
+Down::NetHttp.download("...")
+Down::NetHttp.open("...")
+```
+
+### open-uri + Net::HTTP
 
 ```rb
-require "down"
-# or
+gem "down", ">= 3.0"
+```
+```rb
 require "down/net_http"
+
+tempfile = Down::NetHttp.download("http://nature.com/forest.jpg")
+tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz.jpg>
+
+io = Down::NetHttp.open("http://nature.com/forest.jpg")
+io #=> #<Down::ChunkedIO ...>
 ```
 
-`Down.download` is implemented as a wrapper around open-uri, and fixes some of
-open-uri's undesired behaviours:
+`Down::NetHttp.download` is implemented as a wrapper around open-uri, and fixes
+some of open-uri's undesired behaviours:
+
+* uses `URI::HTTP#open` or `URI::HTTPS#open` directly for [security](https://sakurity.com/blog/2015/02/28/openuri.html)
+* always returns a `Tempfile` object, whereas open-uri returns `StringIO`
+  when file is smaller than 10KB
+* gives the extension to the `Tempfile` object from the URL
+* allows you to limit maximum number of redirects
 
-* open-uri returns `StringIO` for files smaller than 10KB, and `Tempfile`
-  otherwise, but `Down.download` always returns a `Tempfile`
-* open-uri doesn't give any extension to the returned `Tempfile`, but
-  `Down.download` adds the extension from the URL
-* ...
+On the other hand `Down::NetHttp.open` is implemented using Net::HTTP directly,
+as open-uri
 
-Since open-uri doesn't expose support for partial downloads, `Down.open` is
-implemented using `Net::HTTP` directly.
+Since open-uri doesn't expose support for partial downloads,
+`Down::NetHttp.open` is implemented using `Net::HTTP` directly.
 
-### Redirects
+#### Redirects
 
 `Down.download` turns off open-uri's following redirects, as open-uri doesn't
 have a way to limit the maximum number of hops, and implements its own. By
@@ -213,28 +238,38 @@ default maximum of 2 redirects will be followed, but you can change it via the
 `:max_redirects` option:
 
 ```rb
-Down.download("http://example.com/image.jpg")                   # 2 redirects allowed
-Down.download("http://example.com/image.jpg", max_redirects: 5) # 5 redirects allowed
-Down.download("http://example.com/image.jpg", max_redirects: 0) # 0 redirects allowed
+Down::NetHttp.download("http://example.com/image.jpg")                   # 2 redirects allowed
+Down::NetHttp.download("http://example.com/image.jpg", max_redirects: 5) # 5 redirects allowed
+Down::NetHttp.download("http://example.com/image.jpg", max_redirects: 0) # 0 redirects allowed
 ```
 
-### Proxy
+#### Proxy
 
 Both `Down.download` and `Down.open` support a `:proxy` option, where you can
 specify a URL to an HTTP proxy which should be used when downloading.
 
 ```rb
-Down.download("http://example.com/image.jpg", proxy: "http://proxy.org")
-Down.open("http://example.com/image.jpg",     proxy: "http://user:password@proxy.org")
+Down::NetHttp.download("http://example.com/image.jpg", proxy: "http://proxy.org")
+Down::NetHttp.open("http://example.com/image.jpg",     proxy: "http://user:password@proxy.org")
 ```
 
-### Additional options
+#### Timeouts
+
+Both `Down.download` and `Down.open` support `:read_timeout` and `:open_timeout`
+options, which are forwarded to `Net::HTTP`:
+
+```rb
+Down::NetHttp.download("http://example.com/image.jpg", open_timeout: 5)
+Down::NetHttp.open("http://example.com/image.jpg", read_timeout: 10)
+```
+
+#### Additional options
 
 Any additional options passed to `Down.download` will be forwarded to
 [open-uri], so you can for example add basic authentication or a timeout:
 
 ```rb
-Down.download "http://example.com/image.jpg",
+Down::NetHttp.download "http://example.com/image.jpg",
   http_basic_authentication: ['john', 'secret'],
   read_timeout: 5
 ```
@@ -244,19 +279,23 @@ semantics as in open-uri, and any options with String keys will be interpreted
 as request headers, like with open-uri.
 
 ```rb
-Down.open("http://example.com/image.jpg", {"Authorization" => "..."})
+Down::NetHttp.open("http://example.com/image.jpg", {"Authorization" => "..."})
 ```
 
-## HTTP.rb
-
-The [HTTP.rb] backend can be used by requiring `down/http`:
+### HTTP.rb
 
 ```rb
+gem "down", "~> 3.0"
 gem "http", "~> 2.1"
-gem "down"
 ```
 ```rb
 require "down/http"
+
+tempfile = Down::Http.download("http://nature.com/forest.jpg")
+tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz.jpg>
+
+io = Down::Http.open("http://nature.com/forest.jpg")
+io #=> #<Down::ChunkedIO ...>
 ```
 
 Some features that give the HTTP.rb backend an advantage over open-uri +
@@ -266,9 +305,7 @@ Net::HTTP include:
 * Proper support for streaming downloads (`#download` and now reuse `#open`)
 * Proper support for SSL
 * Chaninable HTTP client builder API for setting default options
-* Persistent connections
-* Auto-inflating compressed response bodies
-* ...
+* Support for persistent connections
 
 ### Default client
 
@@ -290,13 +327,13 @@ All additional options passed to `Down::Download` and `Down.open` will be
 forwarded to `HTTP::Client#request`:
 
 ```rb
-Down.download("http://example.org/image.jpg", headers: {"Accept-Encoding" => "gzip"})
+Down::Http.download("http://example.org/image.jpg", headers: {"Accept-Encoding" => "gzip"})
 ```
 
 If you prefer to add options using the chainable API, you can pass a block:
 
 ```rb
-Down.open("http://example.org/image.jpg") do |client|
+Down::Http.open("http://example.org/image.jpg") do |client|
   client.timeout(read: 3)
 end
 ```
@@ -315,23 +352,21 @@ backend is thread safe.
 
 ## Development
 
-The test suite runs the http://httpbin.org/ server locally, and uses it to test
-downloads. Httpbin is a Python package which is run with GUnicorn:
-
-```
-$ pip install gunicorn httpbin
-```
-
-Afterwards you can run tests with
+You can run tests with
 
 ```
 $ rake test
 ```
 
+The test suite pulls and runs [kennethreitz/httpbin] as a Docker container, so
+you'll need to have Docker installed and running.
+
 ## License
 
 [MIT](LICENSE.txt)
 
 [open-uri]: http://ruby-doc.org/stdlib-2.3.0/libdoc/open-uri/rdoc/OpenURI.html
+[Net::HTTP]: https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html
 [HTTP.rb]: https://github.com/httprb/http
 [Addressable::URI]: https://github.com/sporkmonger/addressable
+[kennethreitz/httpbin]: https://github.com/kennethreitz/httpbin

data/down.gemspec

@@ -18,4 +18,5 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "minitest", "~> 5.8"
   spec.add_development_dependency "mocha"
   spec.add_development_dependency "http", "~> 2.1"
+  spec.add_development_dependency "docker-api"
 end

data/lib/down/errors.rb

@@ -1,16 +1,41 @@
 module Down
-  class Error < StandardError
-  end
+  # generic error which is a superclass to all other errors
+  class Error < StandardError; end
 
-  class TooLarge < Error
-  end
+  # raised when the file is larger than the specified maximum size
+  class TooLarge < Error; end
+
+  # raised when the file failed to be retrieved for whatever reason
+  class NotFound < Error; end
+
+  # raised when the given URL couldn't be parsed
+  class InvalidUrl < NotFound; end
+
+  # raised when the number of redirects was larger than the specified maximum
+  class TooManyRedirects < NotFound; end
 
-  class NotFound < Error
+  # raised when response returned 4xx or 5xx response
+  class ResponseError < NotFound
     attr_reader :response
 
-    def initialize(message, response: nil)
+    def initialize(message, response:)
       super(message)
       @response = response
     end
   end
+
+  # raised when response returned 4xx response
+  class ClientError < ResponseError; end
+
+  # raised when response returned 5xx response
+  class ServerError < ResponseError; end
+
+  # raised when there was an error connecting to the server
+  class ConnectionError < NotFound; end
+
+  # raised when connecting to the server too longer than the specified timeout
+  class TimeoutError < ConnectionError; end
+
+  # raised when an SSL error was raised
+  class SSLError < NotFound; end
 end

data/lib/down/http.rb

@@ -65,28 +65,23 @@ module Down
     def open(url, **options, &block)
       rewindable = options.delete(:rewindable)
 
-      response = get(url, **options, &block)
-
-      if response.code.between?(400, 599)
-        raise Down::NotFound.new("file not found", response: response)
+      begin
+        response = get(url, **options, &block)
+        response_error!(response) if !response.status.success?
+      rescue => exception
+        request_error!(exception)
       end
 
       down_options = {
-        chunks:   response.body.enum_for(:each),
-        size:     response.content_length,
-        data:     { status: response.status, headers: response.headers.to_h, response: response },
+        chunks: response.body.enum_for(:each),
+        size:   response.content_length,
+        data:   { status: response.code, headers: response.headers.to_h, response: response },
       }
       down_options[:encoding]   = response.content_type.charset if response.content_type.charset
       down_options[:on_close]   = -> { response.connection.close } unless client.persistent?
       down_options[:rewindable] = rewindable if rewindable != nil
 
       Down::ChunkedIO.new(down_options)
-    rescue HTTP::ConnectionError,
-           HTTP::Request::UnsupportedSchemeError,
-           HTTP::TimeoutError
-      raise Down::NotFound, "file not found"
-    rescue HTTP::Redirector::TooManyRedirectsError
-      raise Down::NotFound, "too many redirects"
     end
 
     def get(url, **options, &block)
@@ -112,6 +107,42 @@ module Down
       Thread.current[:down_client] = value
     end
 
+    def response_error!(response)
+      args = [response.status.to_s, response: response]
+
+      case response.code
+      when 400..499 then raise Down::ClientError.new(*args)
+      when 500..599 then raise Down::ServerError.new(*args)
+      else               raise Down::ResponseError.new(*args)
+      end
+    end
+
+    def request_error!(exception)
+      case exception
+      when HTTP::Request::UnsupportedSchemeError
+        raise Down::InvalidUrl, exception.message
+      when Errno::ECONNREFUSED
+        raise Down::ConnectionError, "connection was refused"
+      when HTTP::ConnectionError,
+           Errno::ECONNABORTED,
+           Errno::ECONNRESET,
+           Errno::EPIPE,
+           Errno::EINVAL,
+           Errno::EHOSTUNREACH
+        raise Down::ConnectionError, exception.message
+      when SocketError
+        raise Down::ConnectionError, "domain name could not be resolved"
+      when HTTP::TimeoutError
+        raise Down::TimeoutError, exception.message
+      when HTTP::Redirector::TooManyRedirectsError
+        raise Down::TooManyRedirects, exception.message
+      when defined?(OpenSSL) && OpenSSL::SSL::SSLError
+        raise Down::SSLError, exception.message
+      else
+        raise exception
+      end
+    end
+
     module DownloadedFile
       attr_accessor :url, :headers
 

data/lib/down/net_http.rb

@@ -73,7 +73,7 @@ module Down
         uri = URI(uri)
 
         if uri.class != URI::HTTP && uri.class != URI::HTTPS
-          raise URI::InvalidURIError, "url is not http nor https"
+          raise Down::InvalidUrl, "URL scheme needs to be http or https"
         end
 
         if uri.user || uri.password
@@ -83,24 +83,29 @@ module Down
         end
 
         downloaded_file = uri.open(open_uri_options)
-      rescue OpenURI::HTTPRedirect => redirect
+      rescue OpenURI::HTTPRedirect => exception
         if (tries -= 1) > 0
-          uri = redirect.uri
+          uri = exception.uri
 
-          if !redirect.io.meta["set-cookie"].to_s.empty?
-            open_uri_options["Cookie"] = redirect.io.meta["set-cookie"]
+          if !exception.io.meta["set-cookie"].to_s.empty?
+            open_uri_options["Cookie"] = exception.io.meta["set-cookie"]
           end
 
           retry
         else
-          raise Down::NotFound, "too many redirects"
+          raise Down::TooManyRedirects, "too many redirects"
         end
-      rescue OpenURI::HTTPError,
-             URI::InvalidURIError,
-             Errno::ECONNREFUSED,
-             SocketError,
-             Timeout::Error
-        raise Down::NotFound, "file not found"
+      rescue OpenURI::HTTPError => exception
+        code, message = exception.io.status
+        response_class = Net::HTTPResponse::CODE_TO_OBJ.fetch(code)
+        response = response_class.new(nil, code, message)
+        exception.io.metas.each do |name, values|
+          values.each { |value| response.add_field(name, value) }
+        end
+
+        response_error!(response)
+      rescue => exception
+        request_error!(exception)
       end
 
       # open-uri will return a StringIO instead of a Tempfile if the filesize is
@@ -117,7 +122,15 @@ module Down
     end
 
     def open(uri, options = {})
-      uri = URI(uri)
+      begin
+        uri = URI(uri)
+        if uri.class != URI::HTTP && uri.class != URI::HTTPS
+          raise Down::InvalidUrl, "URL scheme needs to be http or https"
+        end
+      rescue URI::InvalidURIError
+        raise Down::InvalidUrl, "URL was invalid"
+      end
+
       http_class = Net::HTTP
 
       if options[:proxy]
@@ -143,6 +156,9 @@ module Down
         http.cert_store = store
       end
 
+      http.read_timeout = options[:read_timeout] if options.key?(:read_timeout)
+      http.open_timeout = options[:open_timeout] if options.key?(:open_timeout)
+
       request_headers = options.select { |key, value| key.is_a?(String) }
       get = Net::HTTP::Get.new(uri.request_uri, request_headers)
       get.basic_auth(uri.user, uri.password) if uri.user || uri.password
@@ -156,9 +172,13 @@ module Down
         end
       end
 
-      response = request.resume
+      begin
+        response = request.resume
 
-      raise Down::NotFound, "request returned status #{response.code} and body:\n#{response.body}" if response.code.to_i.between?(400, 599)
+        response_error!(response) unless (200..299).cover?(response.code.to_i)
+      rescue => exception
+        request_error!(exception)
+      end
 
       down_params = {
         chunks: response.enum_for(:read_body),
@@ -193,6 +213,47 @@ module Down
       tempfile
     end
 
+    def response_error!(response)
+      code    = response.code.to_i
+      message = response.message.split(" ").map(&:capitalize).join(" ")
+
+      args = ["#{code} #{message}", response: response]
+
+      case response.code.to_i
+      when 400..499 then raise Down::ClientError.new(*args)
+      when 500..599 then raise Down::ServerError.new(*args)
+      else               raise Down::ResponseError.new(*args)
+      end
+    end
+
+    def request_error!(exception)
+      case exception
+      when URI::InvalidURIError
+        raise Down::InvalidUrl, "URL was invalid"
+      when Errno::ECONNREFUSED
+        raise Down::ConnectionError, "connection was refused"
+      when EOFError,
+           IOError,
+           Errno::ECONNABORTED,
+           Errno::ECONNRESET,
+           Errno::EPIPE,
+           Errno::EINVAL,
+           Errno::EHOSTUNREACH
+        raise Down::ConnectionError, exception.message
+      when SocketError
+        raise Down::ConnectionError, "domain name could not be resolved"
+      when Errno::ETIMEDOUT,
+           Timeout::Error,
+           Net::OpenTimeout,
+           Net::ReadTimeout
+        raise Down::TimeoutError, "request timed out"
+      when defined?(OpenSSL) && OpenSSL::SSL::SSLError
+        raise Down::SSLError, exception.message
+      else
+        raise exception
+      end
+    end
+
     module DownloadedFile
       def original_filename
         filename_from_content_disposition || filename_from_uri

data/lib/down/version.rb

@@ -1,3 +1,3 @@
 module Down
-  VERSION = "3.0.0"
+  VERSION = "3.1.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: down
 version: !ruby/object:Gem::Version
-  version: 3.0.0
+  version: 3.1.0
 platform: ruby
 authors:
 - Janko Marohnić
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-05-24 00:00:00.000000000 Z
+date: 2017-06-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest
@@ -52,6 +52,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.1'
+- !ruby/object:Gem::Dependency
+  name: docker-api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: 
 email:
 - janko.marohnic@gmail.com