down 4.8.1 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ab86935222e38ea1981821660b1ae90ff71a4a739706468d2c7bfaadd48b4956
-  data.tar.gz: 8c8b1245a41f426e4ecd4c378c1bd3ed2ac967b92b58f421c3ea6dc9bee78c2e
+  metadata.gz: a1f7a1532b638ed92acdb3bdf74211dbd022e7a0de37d1fdbc25f665337d4bf1
+  data.tar.gz: 7bbc2684d53e278376981b4dd4741c49bc0d8da0990ece28fb03776f39aea6fe
 SHA512:
-  metadata.gz: 73bd4f1f52e1791171b9ed3d68b50d4ac294e12de2bb947df04d3fffb2bb960904ff35c5f89d158add762b137652818720c0ea05ac41ebc119d7d34d57e5cbdb
-  data.tar.gz: d0b0f7c9e3ef8e602beeb7ef8b3216c090c863ea1434c8cf40096da6833546aa525cc1b99b3b448ed92d70494d0e25f6d0df72dcf286b59c0b335c9fcd6885ac
+  metadata.gz: 6a3e62293c5fa1e5b43a5c835af3804ab8307babfb573179057ec279239b72a16ff74b2b2e96df9da033a3e4e13be4af6adeb8ec00dd73a27f75942b646419de
+  data.tar.gz: d267672bb2468c8998e271ff3849908c7beead00d8b9777799e6bc0abd1ab51ccf14d5c567c16f3bb30a41eb3c57b66ffbc2755d333055fba972d327a865acb4

data/CHANGELOG.md

@@ -1,3 +1,21 @@
+## 5.0.0 (2019-09-26)
+
+* Change `ChunkedIO#each_chunk` to return chunks in original encoding (@janko)
+
+* Always return binary strings in `ChunkedIO#readpartial` (@janko)
+
+* Handle frozen chunks in `Down::ChunkedIO` (@janko)
+
+* Change `ChunkedIO#gets` to return lines in specified encoding (@janko)
+
+* Halve memory allocation for `ChunkedIO#gets` (@janko)
+
+* Halve memory allocation for `ChunkedIO#read` without arguments (@janko)
+
+* Drop support for `HTTP::Client` argument in `Down::HTTP.new` (@janko)
+
+* Repurpose `Down::NotFound` to be raised on `404 Not Found` response (@janko)
+
 ## 4.8.1 (2019-05-01)
 
 * Make `ChunkedIO#read`/`#readpartial` with length always return strings in binary encoding (@janko)

data/README.md

@@ -1,7 +1,7 @@
 # Down
 
 Down is a utility tool for streaming, flexible and safe downloading of remote
-files. It can use [open-uri] + `Net::HTTP`, [HTTP.rb] or `wget` as the backend
+files. It can use [open-uri] + `Net::HTTP`, [http.rb] or `wget` as the backend
 HTTP library.
 
 ## Installation
@@ -57,8 +57,12 @@ specific location on disk, you can specify the `:destination` option:
 
 ```rb
 Down.download("http://example.com/image.jpg", destination: "/path/to/destination")
+#=> nil
 ```
 
+In this case `Down.download` won't have any return value, so if you need a File
+object you'll have to create it manually.
+
 ### Basic authentication
 
 `Down.download` and `Down.open` will automatically detect and apply HTTP basic
@@ -103,6 +107,16 @@ remote_file.eof? #=> true
 remote_file.close # closes the HTTP connection and deletes the internal Tempfile
 ```
 
+The following IO methods are implemented:
+
+* `#read` & `#readpartial`
+* `#gets`
+* `#seek`
+* `#pos` & `#tell`
+* `#eof?`
+* `#rewind`
+* `#close`
+
 ### Caching
 
 By default the downloaded content is internally cached into a `Tempfile`, so
@@ -147,10 +161,10 @@ remote_file.data[:headers]  #=> { ... }
 remote_file.data[:response] # returns the response object
 ```
 
-Note that `Down::NotFound` error will automatically be raised if response
-status was 4xx or 5xx.
+Note that a `Down::ResponseError` exception will automatically be raised if
+response status was 4xx or 5xx.
 
-### `Down::ChunkedIO`
+### Down::ChunkedIO
 
 The `Down.open` performs HTTP logic and returns an instance of
 `Down::ChunkedIO`. However, `Down::ChunkedIO` is a generic class that can wrap
@@ -196,21 +210,23 @@ the `Down::Error` subclasses. This is Down's exception hierarchy:
 
 * `Down::Error`
   * `Down::TooLarge`
-  * `Down::NotFound`
-    * `Down::InvalidUrl`
-    * `Down::TooManyRedirects`
-    * `Down::ResponseError`
-      * `Down::ClientError`
-      * `Down::ServerError`
-    * `Down::ConnectionError`
-    * `Down::TimeoutError`
-    * `Down::SSLError`
+  * `Down::InvalidUrl`
+  * `Down::TooManyRedirects`
+  * `Down::ResponseError`
+    * `Down::ClientError`
+      * `Down::NotFound`
+    * `Down::ServerError`
+  * `Down::ConnectionError`
+  * `Down::TimeoutError`
+  * `Down::SSLError`
 
 ## Backends
 
-By default Down implements `Down.download` and `Down.open` using the built-in
-[open-uri] + [Net::HTTP] Ruby standard libraries. However, there are other
-backends as well, see the sections below.
+The following backends are available:
+
+* [Down::NetHttp](#downnethttp) (default)
+* [Down::Http](#downhttp)
+* [Down::Wget](#downwget)
 
 You can use the backend directly:
 
@@ -232,7 +248,10 @@ Down.download("...")
 Down.open("...")
 ```
 
-### open-uri + Net::HTTP
+### Down::NetHttp
+
+The `Down::NetHttp` backend implements downloads using [open-uri] and
+[Net::HTTP].
 
 ```rb
 gem "down", "~> 4.4"
@@ -334,7 +353,9 @@ net_http.download("http://example.com/image.jpg")
 net_http.open("http://example.com/image.jpg")
 ```
 
-### HTTP.rb
+### Down::Http
+
+The `Down::Http` backend implements downloads using the [http.rb] gem.
 
 ```rb
 gem "down", "~> 4.4"
@@ -350,7 +371,7 @@ io = Down::Http.open("http://nature.com/forest.jpg")
 io #=> #<Down::ChunkedIO ...>
 ```
 
-Some features that give the HTTP.rb backend an advantage over `open-uri` +
+Some features that give the http.rb backend an advantage over `open-uri` and
 `Net::HTTP` include:
 
 * Low memory usage (**10x less** than `open-uri`/`Net::HTTP`)
@@ -401,7 +422,10 @@ down = Down::Http.new(method: :post)
 down.download("http://example.org/image.jpg")
 ```
 
-### Wget (experimental)
+### Down::Wget (experimental)
+
+The `Down::Wget` backend implements downloads using the `wget` command line
+utility.
 
 ```rb
 gem "down", "~> 4.4"
@@ -418,9 +442,8 @@ io = Down::Wget.open("http://nature.com/forest.jpg")
 io #=> #<Down::ChunkedIO ...>
 ```
 
-The Wget backend uses the `wget` command line utility for downloading. One
-major advantage of `wget` is that it automatically resumes downloads that were
-interrupted due to network failures, which is very useful when you're
+One major advantage of `wget` is that it automatically resumes downloads that
+were interrupted due to network failures, which is very useful when you're
 downloading large files.
 
 However, the Wget backend should still be considered experimental, as it wasn't
@@ -450,6 +473,8 @@ wget.open("http://nature.com/forest.jpg")
 * MRI 2.2
 * MRI 2.3
 * MRI 2.4
+* MRI 2.5
+* MRI 2.6
 * JRuby
 
 ## Development
@@ -469,6 +494,6 @@ you'll need to have Docker installed and running.
 
 [open-uri]: http://ruby-doc.org/stdlib-2.3.0/libdoc/open-uri/rdoc/OpenURI.html
 [Net::HTTP]: https://ruby-doc.org/stdlib-2.4.1/libdoc/net/http/rdoc/Net/HTTP.html
-[HTTP.rb]: https://github.com/httprb/http
+[http.rb]: https://github.com/httprb/http
 [Addressable::URI]: https://github.com/sporkmonger/addressable
 [kennethreitz/httpbin]: https://github.com/kennethreitz/httpbin

data/lib/down/chunked_io.rb

@@ -63,21 +63,18 @@ module Down
     def read(length = nil, outbuf = nil)
       fail IOError, "closed stream" if closed?
 
+      data             = outbuf.to_s.clear.force_encoding(Encoding::BINARY)
       remaining_length = length
 
-      begin
-        data = readpartial(remaining_length, outbuf)
-        data = data.dup unless outbuf
-        remaining_length = length - data.bytesize if length
-      rescue EOFError
-      end
-
       until remaining_length == 0 || eof?
-        data << readpartial(remaining_length)
+        data << readpartial(remaining_length, buffer ||= String.new)
         remaining_length = length - data.bytesize if length
       end
 
-      data.to_s unless length && length > 0 && (data.nil? || data.empty?)
+      buffer.clear if buffer # deallocate string
+
+      data.force_encoding(@encoding) unless length
+      data unless data.empty? && length && length > 0
     end
 
     # Implements IO#gets semantics. Without arguments it retrieves lines of
@@ -108,27 +105,33 @@ module Down
 
       separator = "\n\n" if separator.empty?
 
-      begin
-        data = readpartial(limit)
+      data = String.new
 
-        until data.include?(separator) || data.bytesize == limit || eof?
-          remaining_length = limit - data.bytesize if limit
-          data << readpartial(remaining_length, outbuf ||= String.new)
-        end
+      until data.include?(separator) || data.bytesize == limit || eof?
+        remaining_length = limit - data.bytesize if limit
+        data << readpartial(remaining_length, buffer ||= String.new)
+      end
 
-        line, extra = data.split(separator, 2)
-        line << separator if data.include?(separator)
+      buffer.clear if buffer # deallocate buffer
 
+      line, extra = data.split(separator, 2)
+      line << separator if data.include?(separator)
+
+      data.clear # deallocate data
+
+      if extra
         if cache
-          cache.pos -= extra.to_s.bytesize
+          cache.pos -= extra.bytesize
         else
-          @buffer = @buffer.to_s.prepend(extra.to_s)
+          if @buffer
+            @buffer.prepend(extra)
+          else
+            @buffer = extra
+          end
         end
-      rescue EOFError
-        line = nil
       end
 
-      line
+      line.force_encoding(@encoding) if line
     end
 
     # Implements IO#readpartial semantics. If there is any content readily
@@ -145,27 +148,25 @@ module Down
     # where the value is replaced with retrieved content.
     #
     # Raises EOFError if end of file is reached. Raises IOError if closed.
-    def readpartial(length = nil, outbuf = nil)
+    def readpartial(maxlen = nil, outbuf = nil)
       fail IOError, "closed stream" if closed?
 
-      data = outbuf.clear.force_encoding(@encoding) if outbuf
+      maxlen ||= 16*1024
 
-      return data.to_s if length == 0
+      data   = cache.read(maxlen, outbuf) if cache && !cache.eof?
+      data ||= outbuf.to_s.clear
 
-      if cache && !cache.eof?
-        data = cache.read(length, outbuf)
-        data.force_encoding(@encoding)
-      end
+      return data if maxlen == 0
 
-      if @buffer.nil? && (data.nil? || data.empty?)
+      if @buffer.nil? && data.empty?
         fail EOFError, "end of file reached" if chunks_depleted?
         @buffer = retrieve_chunk
       end
 
-      remaining_length = data && length ? length - data.bytesize : length
+      remaining_length = maxlen - data.bytesize
 
       unless @buffer.nil? || remaining_length == 0
-        if remaining_length && remaining_length < @buffer.bytesize
+        if remaining_length < @buffer.bytesize
           buffered_data = @buffer.byteslice(0, remaining_length)
           @buffer       = @buffer.byteslice(remaining_length..-1)
         else
@@ -173,21 +174,46 @@ module Down
           @buffer       = nil
         end
 
-        if data
-          data << buffered_data
-        else
-          data = buffered_data
-        end
+        data << buffered_data
 
         cache.write(buffered_data) if cache
 
-        buffered_data.clear unless buffered_data.equal?(data)
+        buffered_data.clear unless buffered_data.frozen?
       end
 
       @position += data.bytesize
 
-      data.force_encoding(Encoding::BINARY) if length
-      data
+      data.force_encoding(Encoding::BINARY)
+    end
+
+    # Implements IO#seek semantics.
+    def seek(amount, whence = IO::SEEK_SET)
+      fail Errno::ESPIPE, "Illegal seek" if cache.nil?
+
+      case whence
+      when IO::SEEK_SET, :SET
+        target_pos = amount
+      when IO::SEEK_CUR, :CUR
+        target_pos = @position + amount
+      when IO::SEEK_END, :END
+        unless chunks_depleted?
+          cache.seek(0, IO::SEEK_END)
+          IO.copy_stream(self, File::NULL)
+        end
+
+        target_pos = cache.size + amount
+      else
+        fail ArgumentError, "invalid whence: #{whence.inspect}"
+      end
+
+      if target_pos <= cache.size
+        cache.seek(target_pos)
+      else
+        cache.seek(0, IO::SEEK_END)
+        IO.copy_stream(self, File::NULL, target_pos - cache.size)
+      end
+
+      @position = cache.pos
     end
 
     # Implements IO#pos semantics. Returns the current position of the
@@ -195,6 +221,7 @@ module Down
     def pos
       @position
     end
+    alias tell pos
 
     # Implements IO#eof? semantics. Returns whether we've reached end of file.
     # It returns true if cache is at the end and there is no more content to
@@ -272,7 +299,7 @@ module Down
     def retrieve_chunk
       chunk = @next_chunk
       @next_chunk = chunks_fiber.resume
-      chunk.force_encoding(@encoding) if chunk
+      chunk
     end
 
     # Returns whether there is any content left to retrieve.

data/lib/down/errors.rb

@@ -7,17 +7,14 @@ module Down
   # raised when the file is larger than the specified maximum size
   class TooLarge < Error; end
 
-  # raised when the file failed to be retrieved for whatever reason
-  class NotFound < Error; end
-
   # raised when the given URL couldn't be parsed
-  class InvalidUrl < NotFound; end
+  class InvalidUrl < Error; end
 
   # raised when the number of redirects was larger than the specified maximum
-  class TooManyRedirects < NotFound; end
+  class TooManyRedirects < Error; end
 
   # raised when response returned 4xx or 5xx response
-  class ResponseError < NotFound
+  class ResponseError < Error
     attr_reader :response
 
     def initialize(message, response: nil)
@@ -29,15 +26,18 @@ module Down
   # raised when response returned 4xx response
   class ClientError < ResponseError; end
 
+  # raised when response returned 404 response
+  class NotFound < ClientError; end
+
   # raised when response returned 5xx response
   class ServerError < ResponseError; end
 
   # raised when there was an error connecting to the server
-  class ConnectionError < NotFound; end
+  class ConnectionError < Error; end
 
   # raised when connecting to the server too longer than the specified timeout
   class TimeoutError < ConnectionError; end
 
   # raised when an SSL error was raised
-  class SSLError < NotFound; end
+  class SSLError < Error; end
 end

data/lib/down/http.rb

@@ -13,11 +13,6 @@ module Down
   class Http < Backend
     # Initializes the backend with common defaults.
     def initialize(options = {}, &block)
-      if options.is_a?(HTTP::Client)
-        warn "[Down] Passing an HTTP::Client object to Down::Http#initialize is deprecated and won't be supported in Down 5. Use the block initialization instead."
-        options = options.default_options.to_hash
-      end
-
       @method = options.delete(:method) || :get
       @client = HTTP
         .headers("User-Agent" => "Down/#{Down::VERSION}")
@@ -114,6 +109,7 @@ module Down
       args = [response.status.to_s, response: response]
 
       case response.code
+      when 404      then raise Down::NotFound.new(*args)
       when 400..499 then raise Down::ClientError.new(*args)
       when 500..599 then raise Down::ServerError.new(*args)
       else               raise Down::ResponseError.new(*args)

data/lib/down/net_http.rb

@@ -313,6 +313,7 @@ module Down
       args = ["#{code} #{message}", response: response]
 
       case response.code.to_i
+      when 404      then raise Down::NotFound.new(*args)
       when 400..499 then raise Down::ClientError.new(*args)
       when 500..599 then raise Down::ServerError.new(*args)
       else               raise Down::ResponseError.new(*args)

data/lib/down/version.rb

@@ -1,5 +1,5 @@
 # frozen-string-literal: true
 
 module Down
-  VERSION = "4.8.1"
+  VERSION = "5.0.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: down
 version: !ruby/object:Gem::Version
-  version: 4.8.1
+  version: 5.0.0
 platform: ruby
 authors:
 - Janko Marohnić
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2019-05-01 00:00:00.000000000 Z
+date: 2019-09-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: addressable