down 4.8.1 → 5.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ab86935222e38ea1981821660b1ae90ff71a4a739706468d2c7bfaadd48b4956
-  data.tar.gz: 8c8b1245a41f426e4ecd4c378c1bd3ed2ac967b92b58f421c3ea6dc9bee78c2e
+  metadata.gz: d48d573f542ac195a462c3bab0ebe1546c8dd78b0a5210eed1d1fed1f66b674f
+  data.tar.gz: 492b997d3e889475544267d753df5fb900d28728131602e6950a57dfacb7842c
 SHA512:
-  metadata.gz: 73bd4f1f52e1791171b9ed3d68b50d4ac294e12de2bb947df04d3fffb2bb960904ff35c5f89d158add762b137652818720c0ea05ac41ebc119d7d34d57e5cbdb
-  data.tar.gz: d0b0f7c9e3ef8e602beeb7ef8b3216c090c863ea1434c8cf40096da6833546aa525cc1b99b3b448ed92d70494d0e25f6d0df72dcf286b59c0b335c9fcd6885ac
+  metadata.gz: e0fa81667368033a51588f37ae8a5fca2198ce341268e5c6fc24a8241be1af7ce88cd4bc7e8076f69af41b3e0318a4c8733c28a9d173dcdae750b59a9e2401e6
+  data.tar.gz: 568632e1c75daa84a838675ae49ceb844a4500e781b8303cf3c1259974b1d01a26e3d2420e08d0d544c81ef312dc790cafdd47fcdc2ed6f3a853b91f8a99bd48

data/CHANGELOG.md

@@ -1,3 +1,47 @@
+## 5.2.0 (2020-09-20)
+
+* Add `:uri_normalizer` option to `Down::NetHttp` (@janko)
+
+* Add `:http_basic_authentication` option to `Down::NetHttp#download` (@janko)
+
+* Fix uninitialized instance variables warnings in `Down::ChunkedIO` (@janko)
+
+* Handle unknown HTTP error codes in `Down::NetHttp` (@darndt)
+
+## 5.1.1 (2020-02-04)
+
+* Fix keyword arguments warnings on Ruby 2.7 in `Down.download` and `Down.open` (@janko)
+
+## 5.1.0 (2020-01-09)
+
+* Fix keyword arguments warnings on Ruby 2.7 (@janko)
+
+* Fix `FrozenError` exception in `Down::ChunkedIO#readpartial` (@janko)
+
+* Deprecate passing headers as top-level options in `Down::NetHttp` (@janko)
+
+## 5.0.1 (2019-12-20)
+
+* In `Down::NetHttp` only use Addressable normalization if `URI.parse` fails (@coding-chimp)
+
+## 5.0.0 (2019-09-26)
+
+* Change `ChunkedIO#each_chunk` to return chunks in original encoding (@janko)
+
+* Always return binary strings in `ChunkedIO#readpartial` (@janko)
+
+* Handle frozen chunks in `Down::ChunkedIO` (@janko)
+
+* Change `ChunkedIO#gets` to return lines in specified encoding (@janko)
+
+* Halve memory allocation for `ChunkedIO#gets` (@janko)
+
+* Halve memory allocation for `ChunkedIO#read` without arguments (@janko)
+
+* Drop support for `HTTP::Client` argument in `Down::HTTP.new` (@janko)
+
+* Repurpose `Down::NotFound` to be raised on `404 Not Found` response (@janko)
+
 ## 4.8.1 (2019-05-01)
 
 * Make `ChunkedIO#read`/`#readpartial` with length always return strings in binary encoding (@janko)

data/README.md

@@ -1,13 +1,13 @@
 # Down
 
 Down is a utility tool for streaming, flexible and safe downloading of remote
-files. It can use [open-uri] + `Net::HTTP`, [HTTP.rb] or `wget` as the backend
+files. It can use [open-uri] + `Net::HTTP`, [http.rb] or `wget` as the backend
 HTTP library.
 
 ## Installation
 
 ```rb
-gem "down", "~> 4.4"
+gem "down", "~> 5.0"
 ```
 
 ## Downloading
@@ -57,8 +57,12 @@ specific location on disk, you can specify the `:destination` option:
 
 ```rb
 Down.download("http://example.com/image.jpg", destination: "/path/to/destination")
+#=> nil
 ```
 
+In this case `Down.download` won't have any return value, so if you need a File
+object you'll have to create it manually.
+
 ### Basic authentication
 
 `Down.download` and `Down.open` will automatically detect and apply HTTP basic
@@ -103,6 +107,16 @@ remote_file.eof? #=> true
 remote_file.close # closes the HTTP connection and deletes the internal Tempfile
 ```
 
+The following IO methods are implemented:
+
+* `#read` & `#readpartial`
+* `#gets`
+* `#seek`
+* `#pos` & `#tell`
+* `#eof?`
+* `#rewind`
+* `#close`
+
 ### Caching
 
 By default the downloaded content is internally cached into a `Tempfile`, so
@@ -147,10 +161,10 @@ remote_file.data[:headers]  #=> { ... }
 remote_file.data[:response] # returns the response object
 ```
 
-Note that `Down::NotFound` error will automatically be raised if response
-status was 4xx or 5xx.
+Note that a `Down::ResponseError` exception will automatically be raised if
+response status was 4xx or 5xx.
 
-### `Down::ChunkedIO`
+### Down::ChunkedIO
 
 The `Down.open` performs HTTP logic and returns an instance of
 `Down::ChunkedIO`. However, `Down::ChunkedIO` is a generic class that can wrap
@@ -196,21 +210,23 @@ the `Down::Error` subclasses. This is Down's exception hierarchy:
 
 * `Down::Error`
   * `Down::TooLarge`
-  * `Down::NotFound`
-    * `Down::InvalidUrl`
-    * `Down::TooManyRedirects`
-    * `Down::ResponseError`
-      * `Down::ClientError`
-      * `Down::ServerError`
-    * `Down::ConnectionError`
-    * `Down::TimeoutError`
-    * `Down::SSLError`
+  * `Down::InvalidUrl`
+  * `Down::TooManyRedirects`
+  * `Down::ResponseError`
+    * `Down::ClientError`
+      * `Down::NotFound`
+    * `Down::ServerError`
+  * `Down::ConnectionError`
+  * `Down::TimeoutError`
+  * `Down::SSLError`
 
 ## Backends
 
-By default Down implements `Down.download` and `Down.open` using the built-in
-[open-uri] + [Net::HTTP] Ruby standard libraries. However, there are other
-backends as well, see the sections below.
+The following backends are available:
+
+* [Down::NetHttp](#downnethttp) (default)
+* [Down::Http](#downhttp)
+* [Down::Wget](#downwget)
 
 You can use the backend directly:
 
@@ -232,7 +248,10 @@ Down.download("...")
 Down.open("...")
 ```
 
-### open-uri + Net::HTTP
+### Down::NetHttp
+
+The `Down::NetHttp` backend implements downloads using [open-uri] and
+[Net::HTTP].
 
 ```rb
 gem "down", "~> 4.4"
@@ -314,6 +333,18 @@ Down::NetHttp.open("http://example.com/image.jpg",
   ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER)
 ```
 
+#### URI normalization
+
+If the URL isn't parseable by `URI.parse`, `Down::NetHttp` will
+attempt to normalize the URL using [Addressable::URI], URI-escaping
+any potentially unescaped characters. You can change the normalizer
+via the `:uri_normalizer` option:
+
+```rb
+# this skips URL normalization
+Down::NetHttp.download("http://example.com/image.jpg", uri_normalizer: -> (url) { url })
+```
+
 #### Additional options
 
 Any additional options passed to `Down.download` will be forwarded to
@@ -334,7 +365,9 @@ net_http.download("http://example.com/image.jpg")
 net_http.open("http://example.com/image.jpg")
 ```
 
-### HTTP.rb
+### Down::Http
+
+The `Down::Http` backend implements downloads using the [http.rb] gem.
 
 ```rb
 gem "down", "~> 4.4"
@@ -350,7 +383,7 @@ io = Down::Http.open("http://nature.com/forest.jpg")
 io #=> #<Down::ChunkedIO ...>
 ```
 
-Some features that give the HTTP.rb backend an advantage over `open-uri` +
+Some features that give the http.rb backend an advantage over `open-uri` and
 `Net::HTTP` include:
 
 * Low memory usage (**10x less** than `open-uri`/`Net::HTTP`)
@@ -401,7 +434,10 @@ down = Down::Http.new(method: :post)
 down.download("http://example.org/image.jpg")
 ```
 
-### Wget (experimental)
+### Down::Wget (experimental)
+
+The `Down::Wget` backend implements downloads using the `wget` command line
+utility.
 
 ```rb
 gem "down", "~> 4.4"
@@ -418,9 +454,8 @@ io = Down::Wget.open("http://nature.com/forest.jpg")
 io #=> #<Down::ChunkedIO ...>
 ```
 
-The Wget backend uses the `wget` command line utility for downloading. One
-major advantage of `wget` is that it automatically resumes downloads that were
-interrupted due to network failures, which is very useful when you're
+One major advantage of `wget` is that it automatically resumes downloads that
+were interrupted due to network failures, which is very useful when you're
 downloading large files.
 
 However, the Wget backend should still be considered experimental, as it wasn't
@@ -447,10 +482,12 @@ wget.open("http://nature.com/forest.jpg")
 
 ## Supported Ruby versions
 
-* MRI 2.2
 * MRI 2.3
 * MRI 2.4
-* JRuby
+* MRI 2.5
+* MRI 2.6
+* MRI 2.7
+* JRuby 9.2
 
 ## Development
 
@@ -469,6 +506,6 @@ you'll need to have Docker installed and running.
 
 [open-uri]: http://ruby-doc.org/stdlib-2.3.0/libdoc/open-uri/rdoc/OpenURI.html
 [Net::HTTP]: https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html
-[HTTP.rb]: https://github.com/httprb/http
+[http.rb]: https://github.com/httprb/http
 [Addressable::URI]: https://github.com/sporkmonger/addressable
 [kennethreitz/httpbin]: https://github.com/kennethreitz/httpbin

data/down.gemspec

@@ -4,7 +4,7 @@ Gem::Specification.new do |spec|
   spec.name         = "down"
   spec.version      = Down::VERSION
 
-  spec.required_ruby_version = ">= 2.1"
+  spec.required_ruby_version = ">= 2.3"
 
   spec.summary      = "Robust streaming downloads using Net::HTTP, HTTP.rb or wget."
   spec.homepage     = "https://github.com/janko/down"
@@ -20,8 +20,9 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "minitest", "~> 5.8"
   spec.add_development_dependency "mocha", "~> 1.5"
   spec.add_development_dependency "rake"
-  spec.add_development_dependency "http", "~> 4.0"
+  spec.add_development_dependency "http", "~> 4.3"
   spec.add_development_dependency "posix-spawn" unless RUBY_ENGINE == "jruby"
   spec.add_development_dependency "http_parser.rb"
   spec.add_development_dependency "docker-api"
+  spec.add_development_dependency "warning" if RUBY_VERSION >= "2.4"
 end

data/lib/down.rb

@@ -6,12 +6,12 @@ require "down/net_http"
 module Down
   module_function
 
-  def download(*args, &block)
-    backend.download(*args, &block)
+  def download(*args, **options, &block)
+    backend.download(*args, **options, &block)
   end
 
-  def open(*args, &block)
-    backend.open(*args, &block)
+  def open(*args, **options, &block)
+    backend.open(*args, **options, &block)
   end
 
   # Allows setting a backend via a symbol or a downloader object.

data/lib/down/backend.rb

@@ -9,12 +9,12 @@ require "fileutils"
 
 module Down
   class Backend
-    def self.download(*args, &block)
-      new.download(*args, &block)
+    def self.download(*args, **options, &block)
+      new.download(*args, **options, &block)
     end
 
-    def self.open(*args, &block)
-      new.open(*args, &block)
+    def self.open(*args, **options, &block)
+      new.open(*args, **options, &block)
     end
 
     private

data/lib/down/chunked_io.rb

@@ -36,6 +36,8 @@ module Down
       @rewindable = rewindable
       @buffer     = nil
       @position   = 0
+      @next_chunk = nil
+      @closed     = false
 
       retrieve_chunk # fetch first chunk so that we know whether the file is empty
     end
@@ -63,21 +65,20 @@ module Down
     def read(length = nil, outbuf = nil)
       fail IOError, "closed stream" if closed?
 
-      remaining_length = length
+      data   = outbuf.clear.force_encoding(Encoding::BINARY) if outbuf
+      data ||= "".b
 
-      begin
-        data = readpartial(remaining_length, outbuf)
-        data = data.dup unless outbuf
-        remaining_length = length - data.bytesize if length
-      rescue EOFError
-      end
+      remaining_length = length
 
       until remaining_length == 0 || eof?
-        data << readpartial(remaining_length)
+        data << readpartial(remaining_length, buffer ||= String.new)
         remaining_length = length - data.bytesize if length
       end
 
-      data.to_s unless length && length > 0 && (data.nil? || data.empty?)
+      buffer.clear if buffer # deallocate string
+
+      data.force_encoding(@encoding) unless length
+      data unless data.empty? && length && length > 0
     end
 
     # Implements IO#gets semantics. Without arguments it retrieves lines of
@@ -108,27 +109,33 @@ module Down
 
       separator = "\n\n" if separator.empty?
 
-      begin
-        data = readpartial(limit)
+      data = String.new
 
-        until data.include?(separator) || data.bytesize == limit || eof?
-          remaining_length = limit - data.bytesize if limit
-          data << readpartial(remaining_length, outbuf ||= String.new)
-        end
+      until data.include?(separator) || data.bytesize == limit || eof?
+        remaining_length = limit - data.bytesize if limit
+        data << readpartial(remaining_length, buffer ||= String.new)
+      end
+
+      buffer.clear if buffer # deallocate buffer
+
+      line, extra = data.split(separator, 2)
+      line << separator if data.include?(separator)
 
-        line, extra = data.split(separator, 2)
-        line << separator if data.include?(separator)
+      data.clear # deallocate data
 
+      if extra
         if cache
-          cache.pos -= extra.to_s.bytesize
+          cache.pos -= extra.bytesize
         else
-          @buffer = @buffer.to_s.prepend(extra.to_s)
+          if @buffer
+            @buffer.prepend(extra)
+          else
+            @buffer = extra
+          end
         end
-      rescue EOFError
-        line = nil
       end
 
-      line
+      line.force_encoding(@encoding) if line
     end
 
     # Implements IO#readpartial semantics. If there is any content readily
@@ -139,33 +146,33 @@ module Down
     # or the next chunk. This is useful when you don't care about the size of
     # chunks and you want to minimize string allocations.
     #
-    # With `length` argument returns maximum of that amount of bytes.
+    # With `maxlen` argument returns maximum of that amount of bytes (default
+    # is 16KB).
     #
     # With `outbuf` argument each call will return that same string object,
     # where the value is replaced with retrieved content.
     #
     # Raises EOFError if end of file is reached. Raises IOError if closed.
-    def readpartial(length = nil, outbuf = nil)
+    def readpartial(maxlen = nil, outbuf = nil)
       fail IOError, "closed stream" if closed?
 
-      data = outbuf.clear.force_encoding(@encoding) if outbuf
+      maxlen ||= 16*1024
 
-      return data.to_s if length == 0
+      data   = cache.read(maxlen, outbuf) if cache && !cache.eof?
+      data ||= outbuf.clear.force_encoding(Encoding::BINARY) if outbuf
+      data ||= "".b
 
-      if cache && !cache.eof?
-        data = cache.read(length, outbuf)
-        data.force_encoding(@encoding)
-      end
+      return data if maxlen == 0
 
-      if @buffer.nil? && (data.nil? || data.empty?)
+      if @buffer.nil? && data.empty?
         fail EOFError, "end of file reached" if chunks_depleted?
         @buffer = retrieve_chunk
       end
 
-      remaining_length = data && length ? length - data.bytesize : length
+      remaining_length = maxlen - data.bytesize
 
       unless @buffer.nil? || remaining_length == 0
-        if remaining_length && remaining_length < @buffer.bytesize
+        if remaining_length < @buffer.bytesize
           buffered_data = @buffer.byteslice(0, remaining_length)
           @buffer       = @buffer.byteslice(remaining_length..-1)
         else
@@ -173,21 +180,46 @@ module Down
           @buffer       = nil
         end
 
-        if data
-          data << buffered_data
-        else
-          data = buffered_data
-        end
+        data << buffered_data
 
         cache.write(buffered_data) if cache
 
-        buffered_data.clear unless buffered_data.equal?(data)
+        buffered_data.clear unless buffered_data.frozen?
       end
 
       @position += data.bytesize
 
-      data.force_encoding(Encoding::BINARY) if length
-      data
+      data.force_encoding(Encoding::BINARY)
+    end
+
+    # Implements IO#seek semantics.
+    def seek(amount, whence = IO::SEEK_SET)
+      fail Errno::ESPIPE, "Illegal seek" if cache.nil?
+
+      case whence
+      when IO::SEEK_SET, :SET
+        target_pos = amount
+      when IO::SEEK_CUR, :CUR
+        target_pos = @position + amount
+      when IO::SEEK_END, :END
+        unless chunks_depleted?
+          cache.seek(0, IO::SEEK_END)
+          IO.copy_stream(self, File::NULL)
+        end
+
+        target_pos = cache.size + amount
+      else
+        fail ArgumentError, "invalid whence: #{whence.inspect}"
+      end
+
+      if target_pos <= cache.size
+        cache.seek(target_pos)
+      else
+        cache.seek(0, IO::SEEK_END)
+        IO.copy_stream(self, File::NULL, target_pos - cache.size)
+      end
+
+      @position = cache.pos
     end
 
     # Implements IO#pos semantics. Returns the current position of the
@@ -195,6 +227,7 @@ module Down
     def pos
       @position
     end
+    alias tell pos
 
     # Implements IO#eof? semantics. Returns whether we've reached end of file.
     # It returns true if cache is at the end and there is no more content to
@@ -272,7 +305,7 @@ module Down
     def retrieve_chunk
       chunk = @next_chunk
       @next_chunk = chunks_fiber.resume
-      chunk.force_encoding(@encoding) if chunk
+      chunk
     end
 
     # Returns whether there is any content left to retrieve.

data/lib/down/errors.rb

@@ -7,20 +7,17 @@ module Down
   # raised when the file is larger than the specified maximum size
   class TooLarge < Error; end
 
-  # raised when the file failed to be retrieved for whatever reason
-  class NotFound < Error; end
-
   # raised when the given URL couldn't be parsed
-  class InvalidUrl < NotFound; end
+  class InvalidUrl < Error; end
 
   # raised when the number of redirects was larger than the specified maximum
-  class TooManyRedirects < NotFound; end
+  class TooManyRedirects < Error; end
 
   # raised when response returned 4xx or 5xx response
-  class ResponseError < NotFound
+  class ResponseError < Error
     attr_reader :response
 
-    def initialize(message, response: nil)
+    def initialize(message, response = nil)
       super(message)
       @response = response
     end
@@ -29,15 +26,18 @@ module Down
   # raised when response returned 4xx response
   class ClientError < ResponseError; end
 
+  # raised when response returned 404 response
+  class NotFound < ClientError; end
+
   # raised when response returned 5xx response
   class ServerError < ResponseError; end
 
   # raised when there was an error connecting to the server
-  class ConnectionError < NotFound; end
+  class ConnectionError < Error; end
 
   # raised when connecting to the server too longer than the specified timeout
   class TimeoutError < ConnectionError; end
 
   # raised when an SSL error was raised
-  class SSLError < NotFound; end
+  class SSLError < Error; end
 end

data/lib/down/http.rb

@@ -12,12 +12,7 @@ module Down
   # Provides streaming downloads implemented with HTTP.rb.
   class Http < Backend
     # Initializes the backend with common defaults.
-    def initialize(options = {}, &block)
-      if options.is_a?(HTTP::Client)
-        warn "[Down] Passing an HTTP::Client object to Down::Http#initialize is deprecated and won't be supported in Down 5. Use the block initialization instead."
-        options = options.default_options.to_hash
-      end
-
+    def initialize(**options, &block)
       @method = options.delete(:method) || :get
       @client = HTTP
         .headers("User-Agent" => "Down/#{Down::VERSION}")
@@ -111,9 +106,10 @@ module Down
 
     # Raises non-sucessful response as a Down::ResponseError.
     def response_error!(response)
-      args = [response.status.to_s, response: response]
+      args = [response.status.to_s, response]
 
       case response.code
+      when 404      then raise Down::NotFound.new(*args)
       when 400..499 then raise Down::ClientError.new(*args)
       when 500..599 then raise Down::ServerError.new(*args)
       else               raise Down::ResponseError.new(*args)

data/lib/down/net_http.rb

@@ -12,27 +12,34 @@ require "fileutils"
 module Down
   # Provides streaming downloads implemented with Net::HTTP and open-uri.
   class NetHttp < Backend
+    URI_NORMALIZER = -> (url) do
+      addressable_uri = Addressable::URI.parse(url)
+      addressable_uri.normalize.to_s
+    end
+
     # Initializes the backend with common defaults.
-    def initialize(options = {})
-      @options = {
-        "User-Agent" => "Down/#{Down::VERSION}",
-        max_redirects: 2,
-        open_timeout:  30,
-        read_timeout:  30,
-      }.merge(options)
+    def initialize(*args, **options)
+      @options = merge_options({
+        headers:        { "User-Agent" => "Down/#{Down::VERSION}" },
+        max_redirects:  2,
+        open_timeout:   30,
+        read_timeout:   30,
+        uri_normalizer: URI_NORMALIZER,
+      }, *args, **options)
     end
 
     # Downloads a remote file to disk using open-uri. Accepts any open-uri
     # options, and a few more.
-    def download(url, options = {})
-      options = @options.merge(options)
+    def download(url, *args, **options)
+      options = merge_options(@options, *args, **options)
 
       max_size            = options.delete(:max_size)
       max_redirects       = options.delete(:max_redirects)
       progress_proc       = options.delete(:progress_proc)
       content_length_proc = options.delete(:content_length_proc)
       destination         = options.delete(:destination)
-      headers             = options.delete(:headers) || {}
+      headers             = options.delete(:headers)
+      uri_normalizer      = options.delete(:uri_normalizer)
 
       # Use open-uri's :content_lenth_proc or :progress_proc to raise an
       # exception early if the file is too large.
@@ -74,7 +81,7 @@ module Down
       open_uri_options.merge!(options)
       open_uri_options.merge!(headers)
 
-      uri = ensure_uri(addressable_normalize(url))
+      uri = ensure_uri(normalize_uri(url, uri_normalizer: uri_normalizer))
 
       # Handle basic authentication in the remote URL.
       if uri.user || uri.password
@@ -95,13 +102,17 @@ module Down
 
     # Starts retrieving the remote file using Net::HTTP and returns an IO-like
     # object which downloads the response body on-demand.
-    def open(url, options = {})
-      uri     = ensure_uri(addressable_normalize(url))
-      options = @options.merge(options)
+    def open(url, *args, **options)
+      options = merge_options(@options, *args, **options)
+
+      max_redirects  = options.delete(:max_redirects)
+      uri_normalizer = options.delete(:uri_normalizer)
+
+      uri = ensure_uri(normalize_uri(url, uri_normalizer: uri_normalizer))
 
       # Create a Fiber that halts when response headers are received.
       request = Fiber.new do
-        net_http_request(uri, options) do |response|
+        net_http_request(uri, options, follows_remaining: max_redirects) do |response|
           Fiber.yield response
         end
       end
@@ -131,7 +142,7 @@ module Down
     private
 
     # Calls open-uri's URI::HTTP#open method. Additionally handles redirects.
-    def open_uri(uri, options, follows_remaining: 0)
+    def open_uri(uri, options, follows_remaining:)
       uri.open(options)
     rescue OpenURI::HTTPRedirect => exception
       raise Down::TooManyRedirects, "too many redirects" if follows_remaining == 0
@@ -186,7 +197,7 @@ module Down
     end
 
     # Makes a Net::HTTP request and follows redirects.
-    def net_http_request(uri, options, follows_remaining: options.fetch(:max_redirects, 2), &block)
+    def net_http_request(uri, options, follows_remaining:, &block)
       http, request = create_net_http(uri, options)
 
       begin
@@ -251,12 +262,13 @@ module Down
       http.read_timeout = options[:read_timeout] if options.key?(:read_timeout)
       http.open_timeout = options[:open_timeout] if options.key?(:open_timeout)
 
-      headers = options.select { |key, value| key.is_a?(String) }
-      headers.merge!(options[:headers]) if options[:headers]
+      headers = options[:headers].to_h
       headers["Accept-Encoding"] = "" # Net::HTTP's inflater causes FiberErrors
 
       get = Net::HTTP::Get.new(uri.request_uri, headers)
-      get.basic_auth(uri.user, uri.password) if uri.user || uri.password
+
+      user, password = options[:http_basic_authentication] || [uri.user, uri.password]
+      get.basic_auth(user, password) if user || password
 
       [http, get]
     end
@@ -284,9 +296,10 @@ module Down
     end
 
     # Makes sure that the URL is properly encoded.
-    def addressable_normalize(url)
-      addressable_uri = Addressable::URI.parse(url)
-      addressable_uri.normalize.to_s
+    def normalize_uri(url, uri_normalizer:)
+      URI(url)
+    rescue URI::InvalidURIError
+      uri_normalizer.call(url)
     end
 
     # When open-uri raises an exception, it doesn't expose the response object.
@@ -295,7 +308,11 @@ module Down
     def rebuild_response_from_open_uri_exception(exception)
       code, message = exception.io.status
 
-      response_class = Net::HTTPResponse::CODE_TO_OBJ.fetch(code)
+      response_class = Net::HTTPResponse::CODE_TO_OBJ.fetch(code) do |code|
+        Net::HTTPResponse::CODE_CLASS_TO_OBJ.fetch(code[0]) do
+          Net::HTTPUnknownResponse
+        end
+      end
       response       = response_class.new(nil, code, message)
 
       exception.io.metas.each do |name, values|
@@ -310,9 +327,10 @@ module Down
       code    = response.code.to_i
       message = response.message.split(" ").map(&:capitalize).join(" ")
 
-      args = ["#{code} #{message}", response: response]
+      args = ["#{code} #{message}", response]
 
       case response.code.to_i
+      when 404      then raise Down::NotFound.new(*args)
       when 400..499 then raise Down::ClientError.new(*args)
       when 500..599 then raise Down::ServerError.new(*args)
       else               raise Down::ResponseError.new(*args)
@@ -335,6 +353,24 @@ module Down
       end
     end
 
+    # Merge default and ad-hoc options, merging nested headers.
+    def merge_options(options, headers = {}, **new_options)
+      # Deprecate passing headers as top-level options, taking into account
+      # that Ruby 2.7+ accepts kwargs with string keys.
+      if headers.any?
+        warn %([Down::NetHttp] Passing headers as top-level options has been deprecated, use the :headers option instead, e.g: `Down::NetHttp.download(headers: { "Key" => "Value", ... }, ...)`)
+        new_options[:headers] = headers
+      elsif new_options.any? { |key, value| key.is_a?(String) }
+        warn %([Down::NetHttp] Passing headers as top-level options has been deprecated, use the :headers option instead, e.g: `Down::NetHttp.download(headers: { "Key" => "Value", ... }, ...)`)
+        new_options[:headers] = new_options.select { |key, value| key.is_a?(String) }
+        new_options.reject! { |key, value| key.is_a?(String) }
+      end
+
+      options.merge(new_options) do |key, value1, value2|
+        key == :headers ? value1.merge(value2) : value2
+      end
+    end
+
     # Defines some additional attributes for the returned Tempfile (on top of what
     # OpenURI::Meta already defines).
     module DownloadedFile

data/lib/down/version.rb

@@ -1,5 +1,5 @@
 # frozen-string-literal: true
 
 module Down
-  VERSION = "4.8.1"
+  VERSION = "5.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: down
 version: !ruby/object:Gem::Version
-  version: 4.8.1
+  version: 5.2.0
 platform: ruby
 authors:
 - Janko Marohnić
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-05-01 00:00:00.000000000 Z
+date: 2020-09-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: addressable
@@ -72,14 +72,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '4.0'
+        version: '4.3'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '4.0'
+        version: '4.3'
 - !ruby/object:Gem::Dependency
   name: posix-spawn
   requirement: !ruby/object:Gem::Requirement
@@ -122,6 +122,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: warning
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: 
 email:
 - janko.marohnic@gmail.com
@@ -154,14 +168,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.1'
+      version: '2.3'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.1
 signing_key: 
 specification_version: 4
 summary: Robust streaming downloads using Net::HTTP, HTTP.rb or wget.