down 5.4.2 → 5.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a8a87deb869d776663b7fa4a32a87fcf73ffe2434e2fbe4fdef1ce7d6e48cf55
-  data.tar.gz: 294efe7598f57309b0cbc3a3190685663384c9b3374c9afa621e728205bc367e
+  metadata.gz: 04d70935026de1e61cbda54578c360164e4ab252658c2b7a5531701e8195f55d
+  data.tar.gz: 047bbd998f10b26ad7865f4dc445a5ddc4ad811c4639ba51b8691eb1436cb6af
 SHA512:
-  metadata.gz: 1d7d3f067b9de755a9976d572e668cad8d39e38968da71f209a5044debd3b0db14c7f70bf96b4237eae1da73178cdcafd454f28cc1bba34e6d92f02194ee94e0
-  data.tar.gz: 9638590a3e4ee48ccd9c6409aadd140182315a0a648896b52a14fd5419c67bba1a4266cf00e51ec9dca6b17e77520373373ef7693d8155bfeb5409078c3ca9f6
+  metadata.gz: '0924393fcdee5795751393b9965e3cb2221da29baa5797d30bdd19115edb63972e8c6119310e1f65f77feeae7aae61ef5f03452394c77b57a57aaa1d8663a0da'
+  data.tar.gz: 808f8ac45f19f7a54e708b9b51ca4e659474eebe2bd14d894157a9454d4a2726d1260d323c61445218d8cc00f4b25497fb1c680c527e0fbc51a9cf1f374d605c

data/CHANGELOG.md

@@ -1,3 +1,37 @@
+## 5.6.1 (2026-10-18)
+
+* Avoid warnings on requiring `cgi` on Ruby 4.0+ (@janko)
+
+## 5.6.0 (2026-04-25)
+
+* Require Ruby 3.2+ (@janko)
+
+* Return `Down::Chunked#gets(nil, n)` in specified encoding for `IO` compatibility (@janko)
+
+* Allow net/http to transparently decompress response bodies again (@janko)
+
+* Handle `:auth_on_redirect` for absolute redirects in net/http backend (@janko)
+
+* Fix `#pos` not always being updated after `Down::ChunkedIO#gets` (@janko)
+
+* Add `#eof` alias to `Down::ChunkedIO#eof?` for `IO` compatibility
+
+* Add `#length` alias for `Down::ChunkedIO#size` for `multipart-post` gem compatibility (@janko)
+
+## 5.5.0 (2026-03-18)
+
+* Add support for http.rb 6.0 (@sferik)
+
+* Fix `Down::TooManyRedirects` error with `Down::Httpx` happening on first redirect (@artrybalko)
+
+* Add `:auth_on_redirect` option to `Down::NetHttp` backend for skipping authentication on redirects (@makrsmark)
+
+* Add `:tempfile_name` keyword argument to `.download` for overriding tempfile prefix (@softwaregravy)
+
+* Drop support for Ruby 2.6 and older (@janko)
+
+* Deprecate wget backend (@janko)
+
 ## 5.4.2 (2024-04-19)
 
 * Add support for HTTPX 1.x (@HoneyryderChuck)

data/README.md

@@ -1,8 +1,8 @@
 # Down
 
 Down is a utility tool for streaming, flexible and safe downloading of remote
-files. It can use [open-uri] + `Net::HTTP`, [http.rb], [HTTPX], or `wget` as
-the backend HTTP library.
+files. It can use [open-uri] + `Net::HTTP`, [http.rb] or [HTTPX] as the backend
+HTTP library.
 
 ## Installation
 
@@ -70,6 +70,13 @@ tempfile = Down.download("http://example.com/some/file", extension: "txt")
 File.extname(tempfile.path) #=> ".txt"
 ```
 
+You can also override the default tempfile prefix:
+
+```rb
+tempfile = Down.download("http://example.com/image.jpg", tempfile_name: "custom-prefix")
+File.basename(tempfile.path) #=> "custom-prefix20150925-55456-z7vxqz.jpg"
+```
+
 ### Basic authentication
 
 `Down.download` and `Down.open` will automatically detect and apply HTTP basic
@@ -235,7 +242,6 @@ The following backends are available:
 * [Down::NetHttp](#downnethttp) (default)
 * [Down::Http](#downhttp)
 * [Down::Httpx](#downhttpx)
-* [Down::Wget](#downwget)
 
 You can use the backend directly:
 
@@ -450,7 +456,7 @@ supports the HTTP/2 protocol, in addition to many other features.
 
 ```rb
 gem "down", "~> 5.0"
-gem "httpx", "~> 0.22"
+gem "httpx", "~> 1.0"
 ```
 ```rb
 require "down/httpx"
@@ -465,61 +471,8 @@ io #=> #<Down::ChunkedIO ...>
 It's implemented in much of the same way as `Down::Http`, so be sure to check
 its docs for ways to pass additional options.
 
-### Down::Wget (experimental)
-
-The `Down::Wget` backend implements downloads using the `wget` command line
-utility.
-
-```rb
-gem "down", "~> 5.0"
-gem "posix-spawn" # omit if on JRuby
-gem "http_parser.rb"
-```
-```rb
-require "down/wget"
-
-tempfile = Down::Wget.download("http://nature.com/forest.jpg")
-tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz.jpg>
-
-io = Down::Wget.open("http://nature.com/forest.jpg")
-io #=> #<Down::ChunkedIO ...>
-```
-
-One major advantage of `wget` is that it automatically resumes downloads that
-were interrupted due to network failures, which is very useful when you're
-downloading large files.
-
-However, the Wget backend should still be considered experimental, as it wasn't
-easy to implement a CLI wrapper that streams output, so it's possible that I've
-made mistakes. Let me know how it's working out for you 😉.
-
-#### Additional arguments
-
-You can pass additional arguments to the underlying `wget` commmand via symbols:
-
-```rb
-Down::Wget.download("http://nature.com/forest.jpg", :no_proxy, connect_timeout: 3)
-Down::Wget.open("http://nature.com/forest.jpg", user: "janko", password: "secret")
-```
-
-You can also initialize the backend with default arguments:
-
-```rb
-wget = Down::Wget.new(:no_proxy, connect_timeout: 3)
-
-wget.download("http://nature.com/forest.jpg")
-wget.open("http://nature.com/forest.jpg")
-```
-
 ## Development
 
-Tests require that a [httpbin] server is running locally, which you can do via Docker:
-
-```sh
-$ docker pull kennethreitz/httpbin
-$ docker run -p 80:80 kennethreitz/httpbin
-```
-
 Then you can run tests:
 
 ```
@@ -535,4 +488,3 @@ $ bundle exec rake test
 [http.rb]: https://github.com/httprb/http
 [HTTPX]: https://github.com/HoneyryderChuck/httpx
 [Addressable::URI]: https://github.com/sporkmonger/addressable
-[httpbin]: https://github.com/postmanlabs/httpbin

data/down.gemspec

@@ -4,9 +4,9 @@ Gem::Specification.new do |spec|
   spec.name         = "down"
   spec.version      = Down::VERSION
 
-  spec.required_ruby_version = ">= 2.3"
+  spec.required_ruby_version = ">= 3.2"
 
-  spec.summary      = "Robust streaming downloads using Net::HTTP, HTTP.rb or wget."
+  spec.summary      = "Robust streaming downloads using Net::HTTP, http.rb or HTTPX."
   spec.homepage     = "https://github.com/janko/down"
   spec.authors      = ["Janko Marohnić"]
   spec.email        = ["janko.marohnic@gmail.com"]
@@ -16,18 +16,13 @@ Gem::Specification.new do |spec|
   spec.require_path = "lib"
 
   spec.add_dependency "addressable", "~> 2.8"
+  spec.add_dependency "base64", "~> 0.3"
 
-  spec.add_development_dependency "minitest", "~> 5.8"
+  spec.add_development_dependency "minitest", "~> 6.0"
   spec.add_development_dependency "mocha", "~> 1.5"
   spec.add_development_dependency "rake"
-  spec.add_development_dependency "httpx", "~> 0", "< 2.0.0"
-  # http 5.0 drop support of ruby 2.3 and 2.4. We still support those versions.
-  if RUBY_VERSION >= "2.5"
-    spec.add_development_dependency "http", "~> 5.0"
-  else
-    spec.add_development_dependency "http", "~> 4.3"
-  end
-  spec.add_development_dependency "posix-spawn" unless RUBY_ENGINE == "jruby"
-  spec.add_development_dependency "http_parser.rb" unless RUBY_ENGINE == "jruby"
-  spec.add_development_dependency "warning" if RUBY_VERSION >= "2.4"
+  spec.add_development_dependency "httpx", "~> 1.0", "< 1.4.4"
+  spec.add_development_dependency "http", "~> 6.0"
+  spec.add_development_dependency "warning"
+  spec.add_development_dependency "csv"
 end

data/lib/down/chunked_io.rb

@@ -42,6 +42,11 @@ module Down
       retrieve_chunk # fetch first chunk so that we know whether the file is empty
     end
 
+    # For compatibility with multipart-post gem.
+    def length
+      @size
+    end
+
     # Yields elements of the underlying enumerator.
     def each_chunk
       fail IOError, "closed stream" if closed?
@@ -105,7 +110,7 @@ module Down
         separator = separator_or_limit
       end
 
-      return read(limit) if separator.nil?
+      return read(limit)&.force_encoding(@encoding) if separator.nil?
 
       separator = "\n\n" if separator.empty?
 
@@ -124,6 +129,8 @@ module Down
       data.clear # deallocate data
 
       if extra
+        @position -= extra.bytesize
+
         if cache
           cache.pos -= extra.bytesize
         else
@@ -238,6 +245,7 @@ module Down
       return false if cache && !cache.eof?
       @buffer.nil? && chunks_depleted?
     end
+    alias eof eof?
 
     # Implements IO#rewind semantics. Rewinds the Down::ChunkedIO by rewinding
     # the cache and setting the position to the beginning of the file. Raises

data/lib/down/http.rb

@@ -1,8 +1,9 @@
 # frozen-string-literal: true
 
-gem "http", ">= 2.1.0", "< 6"
+gem "http", ">= 2.1.0", "< 7"
 
 require "http"
+require "addressable/uri"
 
 require "down/backend"
 
@@ -12,36 +13,39 @@ module Down
   # Provides streaming downloads implemented with HTTP.rb.
   class Http < Backend
     # Initializes the backend with common defaults.
-    def initialize(**options, &block)
-      @method = options.delete(:method) || :get
+    def initialize(method: :get, **options, &block)
+      @method = method
       @client = HTTP
         .headers("User-Agent" => "Down/#{Down::VERSION}")
         .follow(max_hops: 2)
         .timeout(connect: 30, write: 30, read: 30)
 
-      @client = HTTP::Client.new(@client.default_options.merge(options)) if options.any?
+      if options.any?
+        client_class = defined?(HTTP::Session) ? HTTP::Session : HTTP::Client
+        @client = client_class.new(@client.default_options.merge(options))
+      end
       @client = block.call(@client) if block
     end
 
     # Downlods the remote file to disk. Accepts HTTP.rb options via a hash or a
     # block, and some additional options as well.
-    def download(url, max_size: nil, progress_proc: nil, content_length_proc: nil, destination: nil, extension: nil, **options, &block)
-      response = request(url, **options, &block)
+    def download(url, max_size: nil, progress_proc: nil, content_length_proc: nil, destination: nil, extension: nil, tempfile_name: nil, **, &)
+      response = request(url, **, &)
 
-      content_length_proc.call(response.content_length) if content_length_proc && response.content_length
+      content_length_proc&.call(response.content_length) if response.content_length
 
       if max_size && response.content_length && response.content_length > max_size
         raise Down::TooLarge, "file is too large (#{response.content_length/1024/1024}MB, max is #{max_size/1024/1024}MB)"
       end
 
       extname  = extension ? ".#{extension}" : File.extname(response.uri.path)
-      tempfile = Tempfile.new(["down-http", extname], binmode: true)
+      tempfile = Tempfile.new([tempfile_name || "down-http", extname], binmode: true)
 
       stream_body(response) do |chunk|
         tempfile.write(chunk)
         chunk.clear # deallocate string
 
-        progress_proc.call(tempfile.size) if progress_proc
+        progress_proc&.call(tempfile.size)
 
         if max_size && tempfile.size > max_size
           raise Down::TooLarge, "file is too large (#{tempfile.size/1024/1024}MB, max is #{max_size/1024/1024}MB)"
@@ -56,15 +60,15 @@ module Down
 
       download_result(tempfile, destination)
     rescue
-      tempfile.close! if tempfile
+      tempfile&.close!
       raise
     end
 
     # Starts retrieving the remote file and returns an IO-like object which
     # downloads the response body on-demand. Accepts HTTP.rb options via a hash
     # or a block.
-    def open(url, rewindable: true, **options, &block)
-      response = request(url, **options, &block)
+    def open(url, rewindable: true, **, &)
+      response = request(url, **, &)
 
       Down::ChunkedIO.new(
         chunks:     enum_for(:stream_body, response),
@@ -81,27 +85,27 @@ module Down
 
     private
 
-    def request(url, method: @method, **options, &block)
-      response = send_request(method, url, **options, &block)
+    def request(url, method: @method, **, &)
+      response = send_request(method, url, **, &)
       response_error!(response) unless response.status.success?
       response
     end
 
-    def send_request(method, url, **options, &block)
+    def send_request(method, url, **, &block)
       uri = HTTP::URI.parse(url)
 
       client = @client
       client = client.basic_auth(user: uri.user, pass: uri.password) if uri.user || uri.password
       client = block.call(client) if block
 
-      client.request(method, url, options)
+      client.request(method, url, **)
     rescue => exception
       request_error!(exception)
     end
 
     # Yields chunks of the response body to the block.
-    def stream_body(response, &block)
-      response.body.each(&block)
+    def stream_body(response, &)
+      response.body.each(&)
     rescue => exception
       request_error!(exception)
     ensure
@@ -123,7 +127,7 @@ module Down
     # Re-raise HTTP.rb exceptions as Down::Error exceptions.
     def request_error!(exception)
       case exception
-      when HTTP::Request::UnsupportedSchemeError, Addressable::URI::InvalidURIError
+      when HTTP::Request::UnsupportedSchemeError, Addressable::URI::InvalidURIError, *invalid_url_errors
         raise Down::InvalidUrl, exception.message
       when HTTP::ConnectionError
         raise Down::ConnectionError, exception.message
@@ -138,6 +142,10 @@ module Down
       end
     end
 
+    def invalid_url_errors
+      defined?(HTTP::URI::InvalidError) ? [HTTP::URI::InvalidError] : []
+    end
+
     # Defines some additional attributes for the returned Tempfile.
     module DownloadedFile
       attr_accessor :url, :headers

data/lib/down/httpx.rb

@@ -6,19 +6,16 @@ require "httpx"
 
 require "down/backend"
 
-
 module Down
   # Provides streaming downloads implemented with HTTPX.
   class Httpx < Backend
-    # Initializes the backend
-
     USER_AGENT = "Down/#{Down::VERSION}"
 
     BASIC_AUTH = HTTPX::VERSION >= "1.0.0" ? :basic_auth : :basic_authentication
 
-    def initialize(**options, &block)
-      @method = options.delete(:method) || :get
-      headers = options.delete(:headers) || {}
+    # Initializes the backend with common defaults.
+    def initialize(method: :get, headers: {}, **options, &block)
+      @method = method
       @client = HTTPX
           .plugin(:follow_redirects, max_redirects: 2)
           .plugin(BASIC_AUTH)
@@ -32,20 +29,15 @@ module Down
       @client = block.call(@client) if block
     end
 
-
-    # Downlods the remote file to disk. Accepts HTTPX options via a hash or a
+    # Downloads the remote file to disk. Accepts HTTPX options via a hash or a
     # block, and some additional options as well.
-    def download(url, max_size: nil, progress_proc: nil, content_length_proc: nil, destination: nil, extension: nil, **options, &block)
-      client = @client
-
-      response = request(client, url, **options, &block)
-
-      content_length = nil
+    def download(url, max_size: nil, progress_proc: nil, content_length_proc: nil, destination: nil, extension: nil, tempfile_name: nil, **, &)
+      response = request(url, **, &)
 
       if response.headers.key?("content-length")
         content_length = response.headers["content-length"].to_i
 
-        content_length_proc.call(content_length) if content_length_proc
+        content_length_proc&.call(content_length)
 
         if max_size && content_length > max_size
           response.close
@@ -54,13 +46,13 @@ module Down
       end
 
       extname  = extension ? ".#{extension}" : File.extname(response.uri.path)
-      tempfile = Tempfile.new(["down-http", extname], binmode: true)
+      tempfile = Tempfile.new([tempfile_name || "down-http", extname], binmode: true)
 
       stream_body(response) do |chunk|
         tempfile.write(chunk)
         chunk.clear # deallocate string
 
-        progress_proc.call(tempfile.size) if progress_proc
+        progress_proc&.call(tempfile.size)
 
         if max_size && tempfile.size > max_size
           raise Down::TooLarge, "file is too large (#{tempfile.size/1024/1024}MB, max is #{max_size/1024/1024}MB)"
@@ -77,20 +69,19 @@ module Down
 
       download_result(tempfile, destination)
     rescue
-      tempfile.close! if tempfile
+      tempfile&.close!
       raise
     end
 
     # Starts retrieving the remote file and returns an IO-like object which
-    # downloads the response body on-demand. Accepts HTTP.rb options via a hash
+    # downloads the response body on-demand. Accepts HTTPX options via a hash
     # or a block.
-    def open(url, rewindable: true, **options, &block)
-      response = request(@client, url, stream: true, **options, &block)
-      size = response.headers["content-length"]
-      size = size.to_i if size
+    def open(url, rewindable: true, **, &)
+      response = request(url, stream: true, **, &)
+
       Down::ChunkedIO.new(
         chunks:     enum_for(:stream_body, response),
-        size:       size,
+        size:       response.headers["content-length"]&.to_i,
         encoding:   response.body.encoding,
         rewindable: rewindable,
         data:       {
@@ -105,15 +96,23 @@ module Down
 
     # Yields chunks of the response body to the block.
     def stream_body(response, &block)
-      response.each(&block)
+      response.each do |chunk|
+        next if (300..399).include?(response.status)
+
+        block.call(chunk)
+      end
+
+      if (300..399).include?(response.status)
+        raise Down::TooManyRedirects, "too many redirects"
+      end
     rescue => exception
       request_error!(exception)
     end
 
-    def request(client, url, method: @method, **options, &block)
-      response = send_request(client, method, url, **options, &block)
+    def request(url, method: @method, **, &)
+      response = send_request(method, url, **, &)
       response.raise_for_status
-      response_error!(response) unless (200..299).include?(response.status)
+      response_error!(response) unless (200..399).include?(response.status)
       response
     rescue HTTPX::HTTPError
       response_error!(response)
@@ -121,7 +120,7 @@ module Down
       request_error!(error)
     end
 
-    def send_request(client, method, url, **options, &block)
+    def send_request(method, url, **, &block)
       uri = URI(url)
       client = @client
       if uri.user || uri.password
@@ -130,7 +129,7 @@ module Down
       end
       client = block.call(client) if block
 
-      client.request(method.to_s.upcase, uri, stream: true, **options)
+      client.request(method.to_s.upcase, uri, stream: true, **)
     rescue => exception
       request_error!(exception)
     end
@@ -140,7 +139,6 @@ module Down
       args = [response.status.to_s, response]
 
       case response.status
-      when 300..399 then raise Down::TooManyRedirects, "too many redirects"
       when 404      then raise Down::NotFound.new(*args)
       when 400..499 then raise Down::ClientError.new(*args)
       when 500..599 then raise Down::ServerError.new(*args)

data/lib/down/net_http.rb

@@ -17,30 +17,33 @@ module Down
       addressable_uri.normalize.to_s
     end
 
+    DEFAULT_OPTIONS = {
+      headers:        { "User-Agent" => "Down/#{Down::VERSION}" }.freeze,
+      max_redirects:  2,
+      open_timeout:   30,
+      read_timeout:   30,
+      uri_normalizer: URI_NORMALIZER,
+    }.freeze
+
     # Initializes the backend with common defaults.
-    def initialize(*args, **options)
-      @options = merge_options({
-        headers:        { "User-Agent" => "Down/#{Down::VERSION}" },
-        max_redirects:  2,
-        open_timeout:   30,
-        read_timeout:   30,
-        uri_normalizer: URI_NORMALIZER,
-      }, *args, **options)
+    def initialize(...)
+      @options = merge_options(DEFAULT_OPTIONS, ...)
     end
 
     # Downloads a remote file to disk using open-uri. Accepts any open-uri
     # options, and a few more.
-    def download(url, *args, **options)
-      options = merge_options(@options, *args, **options)
+    def download(url, ...)
+      options = merge_options(@options, ...)
 
       max_size            = options.delete(:max_size)
-      max_redirects       = options.delete(:max_redirects)
       progress_proc       = options.delete(:progress_proc)
       content_length_proc = options.delete(:content_length_proc)
+      proxy               = options.delete(:proxy)
       destination         = options.delete(:destination)
       headers             = options.delete(:headers)
       uri_normalizer      = options.delete(:uri_normalizer)
       extension           = options.delete(:extension)
+      tempfile_name       = options.delete(:tempfile_name)
 
       # Use open-uri's :content_lenth_proc or :progress_proc to raise an
       # exception early if the file is too large.
@@ -52,20 +55,20 @@ module Down
           if size && max_size && size > max_size
             raise Down::TooLarge, "file is too large (#{size/1024/1024}MB, max is #{max_size/1024/1024}MB)"
           end
-          content_length_proc.call(size) if content_length_proc
+          content_length_proc&.call(size)
         },
         progress_proc: proc { |current_size|
           if max_size && current_size > max_size
             raise Down::TooLarge, "file is too large (#{current_size/1024/1024}MB, max is #{max_size/1024/1024}MB)"
           end
-          progress_proc.call(current_size) if progress_proc
+          progress_proc&.call(current_size)
         },
         redirect: false,
       }
 
       # Handle basic authentication in the :proxy option.
-      if options[:proxy]
-        proxy    = URI(options.delete(:proxy))
+      if proxy
+        proxy    = URI(proxy)
         user     = proxy.user
         password = proxy.password
 
@@ -79,10 +82,9 @@ module Down
         end
       end
 
-      open_uri_options.merge!(options)
-      open_uri_options.merge!(headers)
+      open_uri_options.merge!(options, headers)
 
-      uri = ensure_uri(normalize_uri(url, uri_normalizer: uri_normalizer))
+      uri = ensure_uri(normalize_uri(url, uri_normalizer:))
 
       # Handle basic authentication in the remote URL.
       if uri.user || uri.password
@@ -91,11 +93,11 @@ module Down
         uri.password = nil
       end
 
-      open_uri_file = open_uri(uri, open_uri_options, follows_remaining: max_redirects)
+      open_uri_file = open_uri(uri, **open_uri_options)
 
       # Handle the fact that open-uri returns StringIOs for small files.
       extname = extension ? ".#{extension}" : File.extname(open_uri_file.base_uri.path)
-      tempfile = ensure_tempfile(open_uri_file, extname)
+      tempfile = ensure_tempfile(open_uri_file, extname, tempfile_name)
       OpenURI::Meta.init tempfile, open_uri_file # add back open-uri methods
       tempfile.extend Down::NetHttp::DownloadedFile
 
@@ -104,17 +106,16 @@ module Down
 
     # Starts retrieving the remote file using Net::HTTP and returns an IO-like
     # object which downloads the response body on-demand.
-    def open(url, *args, **options)
-      options = merge_options(@options, *args, **options)
+    def open(url, ...)
+      options = merge_options(@options, ...)
 
-      max_redirects  = options.delete(:max_redirects)
       uri_normalizer = options.delete(:uri_normalizer)
 
-      uri = ensure_uri(normalize_uri(url, uri_normalizer: uri_normalizer))
+      uri = ensure_uri(normalize_uri(url, uri_normalizer:))
 
       # Create a Fiber that halts when response headers are received.
       request = Fiber.new do
-        net_http_request(uri, options, follows_remaining: max_redirects) do |response|
+        net_http_request(uri, **options) do |response|
           Fiber.yield response
         end
       end
@@ -126,7 +127,7 @@ module Down
       # Build an IO-like object that will retrieve response body on-demand.
       Down::ChunkedIO.new(
         chunks:     enum_for(:stream_body, response),
-        size:       response["Content-Length"] && response["Content-Length"].to_i,
+        size:       response["Content-Length"]&.to_i,
         encoding:   response.type_params["charset"],
         rewindable: options.fetch(:rewindable, true),
         on_close:   -> { request.resume }, # close HTTP connnection
@@ -141,10 +142,10 @@ module Down
     private
 
     # Calls open-uri's URI::HTTP#open method. Additionally handles redirects.
-    def open_uri(uri, options, follows_remaining:)
+    def open_uri(uri, max_redirects:, auth_on_redirect: false, **options)
       uri.open(options)
     rescue OpenURI::HTTPRedirect => exception
-      raise Down::TooManyRedirects, "too many redirects" if follows_remaining == 0
+      raise Down::TooManyRedirects, "too many redirects" if max_redirects == 0
 
       # fail if redirect URI scheme is not http or https
       begin
@@ -152,9 +153,13 @@ module Down
       rescue Down::InvalidUrl
         response = rebuild_response_from_open_uri_exception(exception)
 
-        raise ResponseError.new("Invalid Redirect URI: #{exception.uri}", response: response)
+        raise ResponseError.new("Invalid Redirect URI: #{exception.uri}", response:)
       end
 
+      # do not leak credentials on redirect
+      options.delete("Authorization") unless auth_on_redirect
+      options.delete(:http_basic_authentication) unless auth_on_redirect
+
       # forward cookies on the redirect
       if !exception.io.meta["set-cookie"].to_s.empty?
         options["Cookie"] ||= ''
@@ -164,14 +169,14 @@ module Down
         options["Cookie"] = (old_cookies | new_cookies).join(';')
       end
 
-      follows_remaining -= 1
+      max_redirects -= 1
       retry
     rescue OpenURI::HTTPError => exception
       response = rebuild_response_from_open_uri_exception(exception)
 
       # open-uri attempts to parse the redirect URI, so we re-raise that exception
       if exception.message.include?("(Invalid Location URI)")
-        raise ResponseError.new("Invalid Redirect URI: #{response["Location"]}", response: response)
+        raise ResponseError.new("Invalid Redirect URI: #{response["Location"]}", response:)
       end
 
       response_error!(response)
@@ -182,8 +187,8 @@ module Down
     # Converts the given IO into a Tempfile if it isn't one already (open-uri
     # returns a StringIO when there is less than 10KB of content), and gives
     # it the specified file extension.
-    def ensure_tempfile(io, extension)
-      tempfile = Tempfile.new(["down-net_http", extension], binmode: true)
+    def ensure_tempfile(io, extension, tempfile_name = nil)
+      tempfile = Tempfile.new([tempfile_name || "down-net_http", extension], binmode: true)
 
       if io.is_a?(Tempfile)
         # Windows requires file descriptors to be closed before files are moved
@@ -200,8 +205,8 @@ module Down
     end
 
     # Makes a Net::HTTP request and follows redirects.
-    def net_http_request(uri, options, follows_remaining:, &block)
-      http, request = create_net_http(uri, options)
+    def net_http_request(uri, max_redirects:, auth_on_redirect: false, **options, &)
+      http, request = create_net_http(uri, **options)
 
       begin
         response = http.start do
@@ -222,28 +227,38 @@ module Down
       if response.is_a?(Net::HTTPNotModified)
         raise Down::NotModified
       elsif response.is_a?(Net::HTTPRedirection)
-        raise Down::TooManyRedirects if follows_remaining == 0
+        raise Down::TooManyRedirects if max_redirects == 0
 
         # fail if redirect URI is not a valid http or https URL
         begin
           location = ensure_uri(response["Location"], allow_relative: true)
         rescue Down::InvalidUrl
-          raise ResponseError.new("Invalid Redirect URI: #{response["Location"]}", response: response)
+          raise ResponseError.new("Invalid Redirect URI: #{response["Location"]}", response:)
         end
 
+        # do not leak credentials on redirect
+        options[:headers].delete("Authorization") unless auth_on_redirect
+
         # handle relative redirects
-        location = uri + location if location.relative?
+        if location.relative?
+          location = uri + location
+          uri.user = nil unless auth_on_redirect
+          uri.password = nil unless auth_on_redirect
+        elsif auth_on_redirect && (uri.user || uri.password)
+          # for absolute redirects, carry over URI credentials since the new location won't have them
+          options[:http_basic_authentication] ||= [uri.user, uri.password]
+        end
 
-        net_http_request(location, options, follows_remaining: follows_remaining - 1, &block)
+        net_http_request(location, **options, max_redirects: max_redirects - 1, auth_on_redirect:, &)
       end
     end
 
     # Build a Net::HTTP object for making a request.
-    def create_net_http(uri, options)
+    def create_net_http(uri, proxy: nil, ssl_verify_mode: nil, ssl_ca_cert: nil, read_timeout: nil, open_timeout: nil, headers: {}, http_basic_authentication: nil, **)
       http_class = Net::HTTP
 
-      if options[:proxy]
-        proxy = URI(options[:proxy])
+      if proxy
+        proxy = URI(proxy)
         http_class = Net::HTTP::Proxy(proxy.hostname, proxy.port, proxy.user, proxy.password)
       end
 
@@ -252,10 +267,10 @@ module Down
       # Handle SSL parameters (taken from the open-uri implementation).
       if uri.is_a?(URI::HTTPS)
         http.use_ssl = true
-        http.verify_mode = options[:ssl_verify_mode] || OpenSSL::SSL::VERIFY_PEER
+        http.verify_mode = ssl_verify_mode || OpenSSL::SSL::VERIFY_PEER
         store = OpenSSL::X509::Store.new
-        if options[:ssl_ca_cert]
-          Array(options[:ssl_ca_cert]).each do |cert|
+        if ssl_ca_cert
+          Array(ssl_ca_cert).each do |cert|
             File.directory?(cert) ? store.add_path(cert) : store.add_file(cert)
           end
         else
@@ -264,23 +279,20 @@ module Down
         http.cert_store = store
       end
 
-      http.read_timeout = options[:read_timeout] if options.key?(:read_timeout)
-      http.open_timeout = options[:open_timeout] if options.key?(:open_timeout)
-
-      headers = options[:headers].to_h
-      headers["Accept-Encoding"] = "" # Net::HTTP's inflater causes FiberErrors
+      http.read_timeout = read_timeout
+      http.open_timeout = open_timeout
 
-      get = Net::HTTP::Get.new(uri, headers)
+      get = Net::HTTP::Get.new(uri, headers.to_h)
 
-      user, password = options[:http_basic_authentication] || [uri.user, uri.password]
+      user, password = http_basic_authentication || [uri.user, uri.password]
       get.basic_auth(user, password) if user || password
 
       [http, get]
     end
 
     # Yields chunks of the response body to the block.
-    def stream_body(response, &block)
-      response.read_body(&block)
+    def stream_body(response, &)
+      response.read_body(&)
     rescue => exception
       request_error!(exception)
     end
@@ -370,6 +382,8 @@ module Down
         warn %([Down::NetHttp] Passing headers as top-level options has been deprecated, use the :headers option instead, e.g: `Down::NetHttp.download(headers: { "Key" => "Value", ... }, ...)`)
         new_options[:headers] = new_options.select { |key, value| key.is_a?(String) }
         new_options.reject! { |key, value| key.is_a?(String) }
+      else
+        new_options[:headers] ||= {}
       end
 
       options.merge(new_options) do |key, value1, value2|

data/lib/down/utils.rb

@@ -1,4 +1,8 @@
-require "cgi"
+if RUBY_VERSION >= "4.0"
+  require "cgi/escape"
+else
+  require "cgi/util"
+end
 
 module Down
   module Utils

data/lib/down/version.rb

@@ -1,5 +1,5 @@
 # frozen-string-literal: true
 
 module Down
-  VERSION = "5.4.2"
+  VERSION = "5.6.1"
 end

data/lib/down/wget.rb

@@ -1,5 +1,7 @@
 # frozen-string-literal: true
 
+warn "The Down::Wget backend has been deprecated and will be removed in the next major version. Please use the Net::HTTP/open-uri, http.rb or HTTPX backend instead."
+
 begin
   require "posix-spawn"
 rescue LoadError

data/lib/down.rb

@@ -6,12 +6,12 @@ require "down/net_http"
 module Down
   module_function
 
-  def download(*args, **options, &block)
-    backend.download(*args, **options, &block)
+  def download(...)
+    backend.download(...)
   end
 
-  def open(*args, **options, &block)
-    backend.open(*args, **options, &block)
+  def open(...)
+    backend.open(...)
   end
 
   # Allows setting a backend via a symbol or a downloader object.

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: down
 version: !ruby/object:Gem::Version
-  version: 5.4.2
+  version: 5.6.1
 platform: ruby
 authors:
 - Janko Marohnić
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-04-19 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: addressable
@@ -24,20 +23,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.8'
+- !ruby/object:Gem::Dependency
+  name: base64
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.8'
+        version: '6.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.8'
+        version: '6.0'
 - !ruby/object:Gem::Dependency
   name: mocha
   requirement: !ruby/object:Gem::Requirement
@@ -72,50 +85,36 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.0'
     - - "<"
       - !ruby/object:Gem::Version
-        version: 2.0.0
+        version: 1.4.4
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.0'
     - - "<"
       - !ruby/object:Gem::Version
-        version: 2.0.0
+        version: 1.4.4
 - !ruby/object:Gem::Dependency
   name: http
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.0'
+        version: '6.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.0'
-- !ruby/object:Gem::Dependency
-  name: posix-spawn
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
+        version: '6.0'
 - !ruby/object:Gem::Dependency
-  name: http_parser.rb
+  name: warning
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -129,7 +128,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: warning
+  name: csv
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -142,7 +141,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description:
 email:
 - janko.marohnic@gmail.com
 executables: []
@@ -167,7 +165,6 @@ homepage: https://github.com/janko/down
 licenses:
 - MIT
 metadata: {}
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -175,15 +172,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.3'
+      version: '3.2'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.3
-signing_key:
+rubygems_version: 4.0.13
 specification_version: 4
-summary: Robust streaming downloads using Net::HTTP, HTTP.rb or wget.
+summary: Robust streaming downloads using Net::HTTP, http.rb or HTTPX.
 test_files: []