down 4.8.1 → 5.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ab86935222e38ea1981821660b1ae90ff71a4a739706468d2c7bfaadd48b4956
-  data.tar.gz: 8c8b1245a41f426e4ecd4c378c1bd3ed2ac967b92b58f421c3ea6dc9bee78c2e
+  metadata.gz: e0544a70de3afab2a00c68df6923572a5533f05a0964b8c5186728019d04e55b
+  data.tar.gz: 3474da6a6c7a182aa02deb72139002ebf97ac98bee9c4a833de2b0413e373924
 SHA512:
-  metadata.gz: 73bd4f1f52e1791171b9ed3d68b50d4ac294e12de2bb947df04d3fffb2bb960904ff35c5f89d158add762b137652818720c0ea05ac41ebc119d7d34d57e5cbdb
-  data.tar.gz: d0b0f7c9e3ef8e602beeb7ef8b3216c090c863ea1434c8cf40096da6833546aa525cc1b99b3b448ed92d70494d0e25f6d0df72dcf286b59c0b335c9fcd6885ac
+  metadata.gz: 64d22c0a25ddf60f2dea37cb03264ec5770a8c7877fb7a843052abca4b043ba9d2d0c8eed830fc2cec97bb852338b114e2907c357aaf4b68aad26a6860459f67
+  data.tar.gz: 992209a7e4201cab464958bac0960ba4a6dc0344cd410b7ceaec2a02b89ebc1bd876396349204bb0bfe175a279fbdb41900c9fee85eee7e98d74eec7afa2e7a0

data/CHANGELOG.md

@@ -1,3 +1,79 @@
+## 5.4.0 (2022-12-26)
+
+* Add new HTTPX backend, which supports HTTP/2 protocol among other features (@HoneyryderChuck)
+
+## 5.3.1 (2022-03-25)
+
+* Correctly split cookie headers on `;` instead of `,` when forwarding them on redirects (@ermolaev)
+
+## 5.3.0 (2022-02-20)
+
+* Add `:extension` argument to `Down.download` for overriding tempfile extension (@razum2um)
+
+* Normalize response header names for http.rb and wget backends (@zarqman)
+
+## 5.2.4 (2021-09-12)
+
+* Keep original cookies between redirections (@antprt)
+
+## 5.2.3 (2021-08-03)
+
+* Bump addressable version requirement to 2.8+ to remediate vulnerability (@aldodelgado)
+
+## 5.2.2 (2021-05-27)
+
+* Add info about received content length in `Down::TooLarge` error (@evheny0)
+
+* Relax http.rb constraint to allow versions 5.x (@mgrunberg)
+
+## 5.2.1 (2021-04-26)
+
+* Raise `Down::NotModified` on 304 response status in `Down::NetHttp#open` (@ellafeldmann)
+
+## 5.2.0 (2020-09-20)
+
+* Add `:uri_normalizer` option to `Down::NetHttp` (@janko)
+
+* Add `:http_basic_authentication` option to `Down::NetHttp#open` (@janko)
+
+* Fix uninitialized instance variables warnings in `Down::ChunkedIO` (@janko)
+
+* Handle unknown HTTP error codes in `Down::NetHttp` (@darndt)
+
+## 5.1.1 (2020-02-04)
+
+* Fix keyword arguments warnings on Ruby 2.7 in `Down.download` and `Down.open` (@janko)
+
+## 5.1.0 (2020-01-09)
+
+* Fix keyword arguments warnings on Ruby 2.7 (@janko)
+
+* Fix `FrozenError` exception in `Down::ChunkedIO#readpartial` (@janko)
+
+* Deprecate passing headers as top-level options in `Down::NetHttp` (@janko)
+
+## 5.0.1 (2019-12-20)
+
+* In `Down::NetHttp` only use Addressable normalization if `URI.parse` fails (@coding-chimp)
+
+## 5.0.0 (2019-09-26)
+
+* Change `ChunkedIO#each_chunk` to return chunks in original encoding (@janko)
+
+* Always return binary strings in `ChunkedIO#readpartial` (@janko)
+
+* Handle frozen chunks in `Down::ChunkedIO` (@janko)
+
+* Change `ChunkedIO#gets` to return lines in specified encoding (@janko)
+
+* Halve memory allocation for `ChunkedIO#gets` (@janko)
+
+* Halve memory allocation for `ChunkedIO#read` without arguments (@janko)
+
+* Drop support for `HTTP::Client` argument in `Down::HTTP.new` (@janko)
+
+* Repurpose `Down::NotFound` to be raised on `404 Not Found` response (@janko)
+
 ## 4.8.1 (2019-05-01)
 
 * Make `ChunkedIO#read`/`#readpartial` with length always return strings in binary encoding (@janko)

data/README.md

@@ -1,13 +1,13 @@
 # Down
 
 Down is a utility tool for streaming, flexible and safe downloading of remote
-files. It can use [open-uri] + `Net::HTTP`, [HTTP.rb] or `wget` as the backend
-HTTP library.
+files. It can use [open-uri] + `Net::HTTP`, [http.rb], [HTTPX], or `wget` as
+the backend HTTP library.
 
 ## Installation
 
 ```rb
-gem "down", "~> 4.4"
+gem "down", "~> 5.0"
 ```
 
 ## Downloading
@@ -57,6 +57,17 @@ specific location on disk, you can specify the `:destination` option:
 
 ```rb
 Down.download("http://example.com/image.jpg", destination: "/path/to/destination")
+#=> nil
+```
+
+In this case `Down.download` won't have any return value, so if you need a File
+object you'll have to create it manually.
+
+You can also keep the tempfile, but override the extension:
+
+```rb
+tempfile = Down.download("http://example.com/some/file", extension: "txt")
+File.extname(tempfile.path) #=> ".txt"
 ```
 
 ### Basic authentication
@@ -103,6 +114,16 @@ remote_file.eof? #=> true
 remote_file.close # closes the HTTP connection and deletes the internal Tempfile
 ```
 
+The following IO methods are implemented:
+
+* `#read` & `#readpartial`
+* `#gets`
+* `#seek`
+* `#pos` & `#tell`
+* `#eof?`
+* `#rewind`
+* `#close`
+
 ### Caching
 
 By default the downloaded content is internally cached into a `Tempfile`, so
@@ -143,14 +164,14 @@ You can access the response status and headers of the HTTP request that was made
 ```rb
 remote_file = Down.open("http://example.com/image.jpg")
 remote_file.data[:status]   #=> 200
-remote_file.data[:headers]  #=> { ... }
+remote_file.data[:headers]  #=> { "Content-Type" => "image/jpeg", ... } (header names are normalized)
 remote_file.data[:response] # returns the response object
 ```
 
-Note that `Down::NotFound` error will automatically be raised if response
-status was 4xx or 5xx.
+Note that a `Down::ResponseError` exception will automatically be raised if
+response status was 4xx or 5xx.
 
-### `Down::ChunkedIO`
+### Down::ChunkedIO
 
 The `Down.open` performs HTTP logic and returns an instance of
 `Down::ChunkedIO`. However, `Down::ChunkedIO` is a generic class that can wrap
@@ -196,21 +217,25 @@ the `Down::Error` subclasses. This is Down's exception hierarchy:
 
 * `Down::Error`
   * `Down::TooLarge`
-  * `Down::NotFound`
-    * `Down::InvalidUrl`
-    * `Down::TooManyRedirects`
-    * `Down::ResponseError`
-      * `Down::ClientError`
-      * `Down::ServerError`
-    * `Down::ConnectionError`
-    * `Down::TimeoutError`
-    * `Down::SSLError`
+  * `Down::InvalidUrl`
+  * `Down::TooManyRedirects`
+  * `Down::NotModified`
+  * `Down::ResponseError`
+    * `Down::ClientError`
+      * `Down::NotFound`
+    * `Down::ServerError`
+  * `Down::ConnectionError`
+  * `Down::TimeoutError`
+  * `Down::SSLError`
 
 ## Backends
 
-By default Down implements `Down.download` and `Down.open` using the built-in
-[open-uri] + [Net::HTTP] Ruby standard libraries. However, there are other
-backends as well, see the sections below.
+The following backends are available:
+
+* [Down::NetHttp](#downnethttp) (default)
+* [Down::Http](#downhttp)
+* [Down::Httpx](#downhttpx)
+* [Down::Wget](#downwget)
 
 You can use the backend directly:
 
@@ -232,10 +257,13 @@ Down.download("...")
 Down.open("...")
 ```
 
-### open-uri + Net::HTTP
+### Down::NetHttp
+
+The `Down::NetHttp` backend implements downloads using [open-uri] and
+[Net::HTTP] standard libraries.
 
 ```rb
-gem "down", "~> 4.4"
+gem "down", "~> 5.0"
 ```
 ```rb
 require "down/net_http"
@@ -314,6 +342,18 @@ Down::NetHttp.open("http://example.com/image.jpg",
   ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER)
 ```
 
+#### URI normalization
+
+If the URL isn't parseable by `URI.parse`, `Down::NetHttp` will
+attempt to normalize the URL using [Addressable::URI], URI-escaping
+any potentially unescaped characters. You can change the normalizer
+via the `:uri_normalizer` option:
+
+```rb
+# this skips URL normalization
+Down::NetHttp.download("http://example.com/image.jpg", uri_normalizer: -> (url) { url })
+```
+
 #### Additional options
 
 Any additional options passed to `Down.download` will be forwarded to
@@ -334,11 +374,13 @@ net_http.download("http://example.com/image.jpg")
 net_http.open("http://example.com/image.jpg")
 ```
 
-### HTTP.rb
+### Down::Http
+
+The `Down::Http` backend implements downloads using the [http.rb] gem.
 
 ```rb
-gem "down", "~> 4.4"
-gem "http", "~> 4.0"
+gem "down", "~> 5.0"
+gem "http", "~> 5.0"
 ```
 ```rb
 require "down/http"
@@ -350,7 +392,7 @@ io = Down::Http.open("http://nature.com/forest.jpg")
 io #=> #<Down::ChunkedIO ...>
 ```
 
-Some features that give the HTTP.rb backend an advantage over `open-uri` +
+Some features that give the http.rb backend an advantage over `open-uri` and
 `Net::HTTP` include:
 
 * Low memory usage (**10x less** than `open-uri`/`Net::HTTP`)
@@ -401,10 +443,35 @@ down = Down::Http.new(method: :post)
 down.download("http://example.org/image.jpg")
 ```
 
-### Wget (experimental)
+### Down::Httpx
+
+The `Down::Httpx` backend implements downloads using the [HTTPX] gem, which
+supports the HTTP/2 protocol, in addition to many other features.
+
+```rb
+gem "down", "~> 5.0"
+gem "httpx", "~> 0.22"
+```
+```rb
+require "down/httpx"
+
+tempfile = Down::Httpx.download("http://nature.com/forest.jpg")
+tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz.jpg>
+
+io = Down::Httpx.open("http://nature.com/forest.jpg")
+io #=> #<Down::ChunkedIO ...>
+```
+
+It's implemented in much of the same way as `Down::Http`, so be sure to check
+its docs for ways to pass additional options.
+
+### Down::Wget (experimental)
+
+The `Down::Wget` backend implements downloads using the `wget` command line
+utility.
 
 ```rb
-gem "down", "~> 4.4"
+gem "down", "~> 5.0"
 gem "posix-spawn" # omit if on JRuby
 gem "http_parser.rb"
 ```
@@ -418,9 +485,8 @@ io = Down::Wget.open("http://nature.com/forest.jpg")
 io #=> #<Down::ChunkedIO ...>
 ```
 
-The Wget backend uses the `wget` command line utility for downloading. One
-major advantage of `wget` is that it automatically resumes downloads that were
-interrupted due to network failures, which is very useful when you're
+One major advantage of `wget` is that it automatically resumes downloads that
+were interrupted due to network failures, which is very useful when you're
 downloading large files.
 
 However, the Wget backend should still be considered experimental, as it wasn't
@@ -447,28 +513,37 @@ wget.open("http://nature.com/forest.jpg")
 
 ## Supported Ruby versions
 
-* MRI 2.2
 * MRI 2.3
 * MRI 2.4
-* JRuby
+* MRI 2.5
+* MRI 2.6
+* MRI 2.7
+* MRI 3.0
+* MRI 3.1
+* JRuby 9.3
 
 ## Development
 
-You can run tests with
+Tests require that a [httpbin] server is running locally, which you can do via Docker:
+
+```sh
+$ docker pull kennethreitz/httpbin
+$ docker run -p 80:80 kennethreitz/httpbin
+```
+
+Then you can run tests:
 
 ```
 $ bundle exec rake test
 ```
 
-The test suite pulls and runs [kennethreitz/httpbin] as a Docker container, so
-you'll need to have Docker installed and running.
-
 ## License
 
 [MIT](LICENSE.txt)
 
 [open-uri]: http://ruby-doc.org/stdlib-2.3.0/libdoc/open-uri/rdoc/OpenURI.html
 [Net::HTTP]: https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html
-[HTTP.rb]: https://github.com/httprb/http
+[http.rb]: https://github.com/httprb/http
+[HTTPX]: https://github.com/HoneyryderChuck/httpx
 [Addressable::URI]: https://github.com/sporkmonger/addressable
-[kennethreitz/httpbin]: https://github.com/kennethreitz/httpbin
+[httpbin]: https://github.com/postmanlabs/httpbin

data/down.gemspec

@@ -4,7 +4,7 @@ Gem::Specification.new do |spec|
   spec.name         = "down"
   spec.version      = Down::VERSION
 
-  spec.required_ruby_version = ">= 2.1"
+  spec.required_ruby_version = ">= 2.3"
 
   spec.summary      = "Robust streaming downloads using Net::HTTP, HTTP.rb or wget."
   spec.homepage     = "https://github.com/janko/down"
@@ -15,13 +15,19 @@ Gem::Specification.new do |spec|
   spec.files        = Dir["README.md", "LICENSE.txt", "CHANGELOG.md", "*.gemspec", "lib/**/*.rb"]
   spec.require_path = "lib"
 
-  spec.add_dependency "addressable", "~> 2.5"
+  spec.add_dependency "addressable", "~> 2.8"
 
   spec.add_development_dependency "minitest", "~> 5.8"
   spec.add_development_dependency "mocha", "~> 1.5"
   spec.add_development_dependency "rake"
-  spec.add_development_dependency "http", "~> 4.0"
+  spec.add_development_dependency "httpx", "~> 0.22", ">= 0.22.2"
+  # http 5.0 drop support of ruby 2.3 and 2.4. We still support those versions.
+  if RUBY_VERSION >= "2.5"
+    spec.add_development_dependency "http", "~> 5.0"
+  else
+    spec.add_development_dependency "http", "~> 4.3"
+  end
   spec.add_development_dependency "posix-spawn" unless RUBY_ENGINE == "jruby"
-  spec.add_development_dependency "http_parser.rb"
-  spec.add_development_dependency "docker-api"
+  spec.add_development_dependency "http_parser.rb" unless RUBY_ENGINE == "jruby"
+  spec.add_development_dependency "warning" if RUBY_VERSION >= "2.4"
 end

data/lib/down/backend.rb

@@ -9,12 +9,12 @@ require "fileutils"
 
 module Down
   class Backend
-    def self.download(*args, &block)
-      new.download(*args, &block)
+    def self.download(*args, **options, &block)
+      new.download(*args, **options, &block)
     end
 
-    def self.open(*args, &block)
-      new.open(*args, &block)
+    def self.open(*args, **options, &block)
+      new.open(*args, **options, &block)
     end
 
     private
@@ -29,5 +29,12 @@ module Down
 
       nil
     end
+
+    def normalize_headers(response_headers)
+      response_headers.inject({}) do |headers, (downcased_name, value)|
+        name = downcased_name.split("-").map(&:capitalize).join("-")
+        headers.merge!(name => value)
+      end
+    end
   end
 end

data/lib/down/chunked_io.rb

@@ -36,6 +36,8 @@ module Down
       @rewindable = rewindable
       @buffer     = nil
       @position   = 0
+      @next_chunk = nil
+      @closed     = false
 
       retrieve_chunk # fetch first chunk so that we know whether the file is empty
     end
@@ -63,21 +65,20 @@ module Down
     def read(length = nil, outbuf = nil)
       fail IOError, "closed stream" if closed?
 
-      remaining_length = length
+      data   = outbuf.clear.force_encoding(Encoding::BINARY) if outbuf
+      data ||= "".b
 
-      begin
-        data = readpartial(remaining_length, outbuf)
-        data = data.dup unless outbuf
-        remaining_length = length - data.bytesize if length
-      rescue EOFError
-      end
+      remaining_length = length
 
       until remaining_length == 0 || eof?
-        data << readpartial(remaining_length)
+        data << readpartial(remaining_length, buffer ||= String.new)
         remaining_length = length - data.bytesize if length
       end
 
-      data.to_s unless length && length > 0 && (data.nil? || data.empty?)
+      buffer.clear if buffer # deallocate string
+
+      data.force_encoding(@encoding) unless length
+      data unless data.empty? && length && length > 0
     end
 
     # Implements IO#gets semantics. Without arguments it retrieves lines of
@@ -108,27 +109,33 @@ module Down
 
       separator = "\n\n" if separator.empty?
 
-      begin
-        data = readpartial(limit)
+      data = String.new
 
-        until data.include?(separator) || data.bytesize == limit || eof?
-          remaining_length = limit - data.bytesize if limit
-          data << readpartial(remaining_length, outbuf ||= String.new)
-        end
+      until data.include?(separator) || data.bytesize == limit || eof?
+        remaining_length = limit - data.bytesize if limit
+        data << readpartial(remaining_length, buffer ||= String.new)
+      end
+
+      buffer.clear if buffer # deallocate buffer
+
+      line, extra = data.split(separator, 2)
+      line << separator if data.include?(separator)
 
-        line, extra = data.split(separator, 2)
-        line << separator if data.include?(separator)
+      data.clear # deallocate data
 
+      if extra
         if cache
-          cache.pos -= extra.to_s.bytesize
+          cache.pos -= extra.bytesize
         else
-          @buffer = @buffer.to_s.prepend(extra.to_s)
+          if @buffer
+            @buffer.prepend(extra)
+          else
+            @buffer = extra
+          end
         end
-      rescue EOFError
-        line = nil
       end
 
-      line
+      line.force_encoding(@encoding) if line
     end
 
     # Implements IO#readpartial semantics. If there is any content readily
@@ -139,33 +146,33 @@ module Down
     # or the next chunk. This is useful when you don't care about the size of
     # chunks and you want to minimize string allocations.
     #
-    # With `length` argument returns maximum of that amount of bytes.
+    # With `maxlen` argument returns maximum of that amount of bytes (default
+    # is 16KB).
     #
     # With `outbuf` argument each call will return that same string object,
     # where the value is replaced with retrieved content.
     #
     # Raises EOFError if end of file is reached. Raises IOError if closed.
-    def readpartial(length = nil, outbuf = nil)
+    def readpartial(maxlen = nil, outbuf = nil)
       fail IOError, "closed stream" if closed?
 
-      data = outbuf.clear.force_encoding(@encoding) if outbuf
+      maxlen ||= 16*1024
 
-      return data.to_s if length == 0
+      data   = cache.read(maxlen, outbuf) if cache && !cache.eof?
+      data ||= outbuf.clear.force_encoding(Encoding::BINARY) if outbuf
+      data ||= "".b
 
-      if cache && !cache.eof?
-        data = cache.read(length, outbuf)
-        data.force_encoding(@encoding)
-      end
+      return data if maxlen == 0
 
-      if @buffer.nil? && (data.nil? || data.empty?)
+      if @buffer.nil? && data.empty?
         fail EOFError, "end of file reached" if chunks_depleted?
         @buffer = retrieve_chunk
       end
 
-      remaining_length = data && length ? length - data.bytesize : length
+      remaining_length = maxlen - data.bytesize
 
       unless @buffer.nil? || remaining_length == 0
-        if remaining_length && remaining_length < @buffer.bytesize
+        if remaining_length < @buffer.bytesize
           buffered_data = @buffer.byteslice(0, remaining_length)
           @buffer       = @buffer.byteslice(remaining_length..-1)
         else
@@ -173,21 +180,46 @@ module Down
           @buffer       = nil
         end
 
-        if data
-          data << buffered_data
-        else
-          data = buffered_data
-        end
+        data << buffered_data
 
         cache.write(buffered_data) if cache
 
-        buffered_data.clear unless buffered_data.equal?(data)
+        buffered_data.clear unless buffered_data.frozen?
       end
 
       @position += data.bytesize
 
-      data.force_encoding(Encoding::BINARY) if length
-      data
+      data.force_encoding(Encoding::BINARY)
+    end
+
+    # Implements IO#seek semantics.
+    def seek(amount, whence = IO::SEEK_SET)
+      fail Errno::ESPIPE, "Illegal seek" if cache.nil?
+
+      case whence
+      when IO::SEEK_SET, :SET
+        target_pos = amount
+      when IO::SEEK_CUR, :CUR
+        target_pos = @position + amount
+      when IO::SEEK_END, :END
+        unless chunks_depleted?
+          cache.seek(0, IO::SEEK_END)
+          IO.copy_stream(self, File::NULL)
+        end
+
+        target_pos = cache.size + amount
+      else
+        fail ArgumentError, "invalid whence: #{whence.inspect}"
+      end
+
+      if target_pos <= cache.size
+        cache.seek(target_pos)
+      else
+        cache.seek(0, IO::SEEK_END)
+        IO.copy_stream(self, File::NULL, target_pos - cache.size)
+      end
+
+      @position = cache.pos
     end
 
     # Implements IO#pos semantics. Returns the current position of the
@@ -195,6 +227,7 @@ module Down
     def pos
       @position
     end
+    alias tell pos
 
     # Implements IO#eof? semantics. Returns whether we've reached end of file.
     # It returns true if cache is at the end and there is no more content to
@@ -272,7 +305,7 @@ module Down
     def retrieve_chunk
       chunk = @next_chunk
       @next_chunk = chunks_fiber.resume
-      chunk.force_encoding(@encoding) if chunk
+      chunk
     end
 
     # Returns whether there is any content left to retrieve.

data/lib/down/errors.rb

@@ -7,20 +7,20 @@ module Down
   # raised when the file is larger than the specified maximum size
   class TooLarge < Error; end
 
-  # raised when the file failed to be retrieved for whatever reason
-  class NotFound < Error; end
-
   # raised when the given URL couldn't be parsed
-  class InvalidUrl < NotFound; end
+  class InvalidUrl < Error; end
 
   # raised when the number of redirects was larger than the specified maximum
-  class TooManyRedirects < NotFound; end
+  class TooManyRedirects < Error; end
+
+  # raised when the requested resource has not been modified
+  class NotModified < Error; end
 
   # raised when response returned 4xx or 5xx response
-  class ResponseError < NotFound
+  class ResponseError < Error
     attr_reader :response
 
-    def initialize(message, response: nil)
+    def initialize(message, response = nil)
       super(message)
       @response = response
     end
@@ -29,15 +29,18 @@ module Down
   # raised when response returned 4xx response
   class ClientError < ResponseError; end
 
+  # raised when response returned 404 response
+  class NotFound < ClientError; end
+
   # raised when response returned 5xx response
   class ServerError < ResponseError; end
 
   # raised when there was an error connecting to the server
-  class ConnectionError < NotFound; end
+  class ConnectionError < Error; end
 
   # raised when connecting to the server too longer than the specified timeout
   class TimeoutError < ConnectionError; end
 
   # raised when an SSL error was raised
-  class SSLError < NotFound; end
+  class SSLError < Error; end
 end