down 5.4.1 → 5.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5e25eaeebd866fe407bcb39282937568f1c5489feb0b3ac479d92c2b19d8920c
-  data.tar.gz: 151e9d28039d7698a32e25ff7b1dae39c32e871ed295809f11a586c5f6221bcc
+  metadata.gz: 9658c84910478f3dfce33c56625fe4e60ad4099afdceeef9288355d8bc43ba26
+  data.tar.gz: 4009abef1c18006798565de452a7472a418cdda49d5c736fc262bb7cc058855c
 SHA512:
-  metadata.gz: 3ee68adf94333adcb92f9453ec0c65e23437458e6586f0078471ba8b5691d3ca0b5e377656a16fb6750ee585f255cd5af38218dbf873fd6b7e5e9ec1997a8997
-  data.tar.gz: 49f80f1b039ba67e86e1515aac927382249f2bdca33cb8d71e8be8753d82d6f69c56d010be4521c2e661a51662adb698478348949fc01ac22e1419212cfc640c
+  metadata.gz: 3181b47973db1b65ea4c9c3a7f6d39beed4d4cdd2e37bdf3218a3924204f64a1c98b0df70e4c6b7176a99d32396840f094f89b26c50d39e263277d11f7472840
+  data.tar.gz: 906c4e18bb4576aa5985e2da38ddbbcc0dff228255773fc3a2f3c9ced301f65bcf4dacd6529d1c293eec1502b719ca4ce99b7b07cfab3d8cf70012b97fc6ce97

data/CHANGELOG.md

@@ -1,3 +1,21 @@
+## 5.5.0 (2026-03-18)
+
+* Add support for http.rb 6.0 (@sferik)
+
+* Fix `Down::TooManyRedirects` error with `Down::Httpx` happening on first redirect (@artrybalko)
+
+* Add `:auth_on_redirect` option to `Down::NetHttp` backend for skipping authentication on redirects (@makrsmark)
+
+* Add `:tempfile_name` keyword argument to `.download` for overriding tempfile prefix (@softwaregravy)
+
+* Drop support for Ruby 2.6 and older (@janko)
+
+* Deprecate wget backend (@janko)
+
+## 5.4.2 (2024-04-19)
+
+* Add support for HTTPX 1.x (@HoneyryderChuck)
+
 ## 5.4.1 (2023-05-20)
 
 * Handle additional params in `Content-Disposition` header (@janko)

data/README.md

@@ -1,8 +1,8 @@
 # Down
 
 Down is a utility tool for streaming, flexible and safe downloading of remote
-files. It can use [open-uri] + `Net::HTTP`, [http.rb], [HTTPX], or `wget` as
-the backend HTTP library.
+files. It can use [open-uri] + `Net::HTTP`, [http.rb] or [HTTPX] as the backend
+HTTP library.
 
 ## Installation
 
@@ -70,6 +70,13 @@ tempfile = Down.download("http://example.com/some/file", extension: "txt")
 File.extname(tempfile.path) #=> ".txt"
 ```
 
+You can also override the default tempfile prefix:
+
+```rb
+tempfile = Down.download("http://example.com/image.jpg", tempfile_name: "custom-prefix")
+File.basename(tempfile.path) #=> "custom-prefix20150925-55456-z7vxqz.jpg"
+```
+
 ### Basic authentication
 
 `Down.download` and `Down.open` will automatically detect and apply HTTP basic
@@ -235,7 +242,6 @@ The following backends are available:
 * [Down::NetHttp](#downnethttp) (default)
 * [Down::Http](#downhttp)
 * [Down::Httpx](#downhttpx)
-* [Down::Wget](#downwget)
 
 You can use the backend directly:
 
@@ -450,7 +456,7 @@ supports the HTTP/2 protocol, in addition to many other features.
 
 ```rb
 gem "down", "~> 5.0"
-gem "httpx", "~> 0.22"
+gem "httpx", "~> 1.0"
 ```
 ```rb
 require "down/httpx"
@@ -465,52 +471,6 @@ io #=> #<Down::ChunkedIO ...>
 It's implemented in much of the same way as `Down::Http`, so be sure to check
 its docs for ways to pass additional options.
 
-### Down::Wget (experimental)
-
-The `Down::Wget` backend implements downloads using the `wget` command line
-utility.
-
-```rb
-gem "down", "~> 5.0"
-gem "posix-spawn" # omit if on JRuby
-gem "http_parser.rb"
-```
-```rb
-require "down/wget"
-
-tempfile = Down::Wget.download("http://nature.com/forest.jpg")
-tempfile #=> #<Tempfile:/var/folders/k7/6zx6dx6x7ys3rv3srh0nyfj00000gn/T/20150925-55456-z7vxqz.jpg>
-
-io = Down::Wget.open("http://nature.com/forest.jpg")
-io #=> #<Down::ChunkedIO ...>
-```
-
-One major advantage of `wget` is that it automatically resumes downloads that
-were interrupted due to network failures, which is very useful when you're
-downloading large files.
-
-However, the Wget backend should still be considered experimental, as it wasn't
-easy to implement a CLI wrapper that streams output, so it's possible that I've
-made mistakes. Let me know how it's working out for you 😉.
-
-#### Additional arguments
-
-You can pass additional arguments to the underlying `wget` commmand via symbols:
-
-```rb
-Down::Wget.download("http://nature.com/forest.jpg", :no_proxy, connect_timeout: 3)
-Down::Wget.open("http://nature.com/forest.jpg", user: "janko", password: "secret")
-```
-
-You can also initialize the backend with default arguments:
-
-```rb
-wget = Down::Wget.new(:no_proxy, connect_timeout: 3)
-
-wget.download("http://nature.com/forest.jpg")
-wget.open("http://nature.com/forest.jpg")
-```
-
 ## Development
 
 Tests require that a [httpbin] server is running locally, which you can do via Docker:

data/down.gemspec

@@ -4,9 +4,9 @@ Gem::Specification.new do |spec|
   spec.name         = "down"
   spec.version      = Down::VERSION
 
-  spec.required_ruby_version = ">= 2.3"
+  spec.required_ruby_version = ">= 2.7"
 
-  spec.summary      = "Robust streaming downloads using Net::HTTP, HTTP.rb or wget."
+  spec.summary      = "Robust streaming downloads using Net::HTTP, http.rb or HTTPX."
   spec.homepage     = "https://github.com/janko/down"
   spec.authors      = ["Janko Marohnić"]
   spec.email        = ["janko.marohnic@gmail.com"]
@@ -16,18 +16,13 @@ Gem::Specification.new do |spec|
   spec.require_path = "lib"
 
   spec.add_dependency "addressable", "~> 2.8"
+  spec.add_dependency "base64", "~> 0.3"
 
-  spec.add_development_dependency "minitest", "~> 5.8"
+  spec.add_development_dependency "minitest", "~> 6.0"
   spec.add_development_dependency "mocha", "~> 1.5"
   spec.add_development_dependency "rake"
-  spec.add_development_dependency "httpx", "~> 0.22", ">= 0.22.2"
-  # http 5.0 drop support of ruby 2.3 and 2.4. We still support those versions.
-  if RUBY_VERSION >= "2.5"
-    spec.add_development_dependency "http", "~> 5.0"
-  else
-    spec.add_development_dependency "http", "~> 4.3"
-  end
-  spec.add_development_dependency "posix-spawn" unless RUBY_ENGINE == "jruby"
-  spec.add_development_dependency "http_parser.rb" unless RUBY_ENGINE == "jruby"
-  spec.add_development_dependency "warning" if RUBY_VERSION >= "2.4"
+  spec.add_development_dependency "httpx", "~> 1.0", "< 1.4.4"
+  spec.add_development_dependency "http", RUBY_VERSION >= "3.2" ? "~> 6.0" : "~> 5.0"
+  spec.add_development_dependency "warning"
+  spec.add_development_dependency "csv"
 end

data/lib/down/http.rb

@@ -1,8 +1,9 @@
 # frozen-string-literal: true
 
-gem "http", ">= 2.1.0", "< 6"
+gem "http", ">= 2.1.0", "< 7"
 
 require "http"
+require "addressable/uri"
 
 require "down/backend"
 
@@ -19,13 +20,16 @@ module Down
         .follow(max_hops: 2)
         .timeout(connect: 30, write: 30, read: 30)
 
-      @client = HTTP::Client.new(@client.default_options.merge(options)) if options.any?
+      if options.any?
+        client_class = defined?(HTTP::Session) ? HTTP::Session : HTTP::Client
+        @client = client_class.new(@client.default_options.merge(options))
+      end
       @client = block.call(@client) if block
     end
 
     # Downlods the remote file to disk. Accepts HTTP.rb options via a hash or a
     # block, and some additional options as well.
-    def download(url, max_size: nil, progress_proc: nil, content_length_proc: nil, destination: nil, extension: nil, **options, &block)
+    def download(url, max_size: nil, progress_proc: nil, content_length_proc: nil, destination: nil, extension: nil, tempfile_name: nil, **options, &block)
       response = request(url, **options, &block)
 
       content_length_proc.call(response.content_length) if content_length_proc && response.content_length
@@ -35,7 +39,7 @@ module Down
       end
 
       extname  = extension ? ".#{extension}" : File.extname(response.uri.path)
-      tempfile = Tempfile.new(["down-http", extname], binmode: true)
+      tempfile = Tempfile.new([tempfile_name || "down-http", extname], binmode: true)
 
       stream_body(response) do |chunk|
         tempfile.write(chunk)
@@ -94,7 +98,7 @@ module Down
       client = client.basic_auth(user: uri.user, pass: uri.password) if uri.user || uri.password
       client = block.call(client) if block
 
-      client.request(method, url, options)
+      client.request(method, url, **options)
     rescue => exception
       request_error!(exception)
     end
@@ -123,7 +127,7 @@ module Down
     # Re-raise HTTP.rb exceptions as Down::Error exceptions.
     def request_error!(exception)
       case exception
-      when HTTP::Request::UnsupportedSchemeError, Addressable::URI::InvalidURIError
+      when HTTP::Request::UnsupportedSchemeError, Addressable::URI::InvalidURIError, *invalid_url_errors
         raise Down::InvalidUrl, exception.message
       when HTTP::ConnectionError
         raise Down::ConnectionError, exception.message
@@ -138,6 +142,10 @@ module Down
       end
     end
 
+    def invalid_url_errors
+      defined?(HTTP::URI::InvalidError) ? [HTTP::URI::InvalidError] : []
+    end
+
     # Defines some additional attributes for the returned Tempfile.
     module DownloadedFile
       attr_accessor :url, :headers

data/lib/down/httpx.rb

@@ -14,12 +14,14 @@ module Down
 
     USER_AGENT = "Down/#{Down::VERSION}"
 
+    BASIC_AUTH = HTTPX::VERSION >= "1.0.0" ? :basic_auth : :basic_authentication
+
     def initialize(**options, &block)
       @method = options.delete(:method) || :get
       headers = options.delete(:headers) || {}
       @client = HTTPX
           .plugin(:follow_redirects, max_redirects: 2)
-          .plugin(:basic_authentication)
+          .plugin(BASIC_AUTH)
           .plugin(:stream)
           .with(
             headers: { "user-agent": USER_AGENT }.merge(headers),
@@ -31,9 +33,9 @@ module Down
     end
 
 
-    # Downlods the remote file to disk. Accepts HTTPX options via a hash or a
+    # Downloads the remote file to disk. Accepts HTTPX options via a hash or a
     # block, and some additional options as well.
-    def download(url, max_size: nil, progress_proc: nil, content_length_proc: nil, destination: nil, extension: nil, **options, &block)
+    def download(url, max_size: nil, progress_proc: nil, content_length_proc: nil, destination: nil, extension: nil, tempfile_name: nil, **options, &block)
       client = @client
 
       response = request(client, url, **options, &block)
@@ -52,7 +54,7 @@ module Down
       end
 
       extname  = extension ? ".#{extension}" : File.extname(response.uri.path)
-      tempfile = Tempfile.new(["down-http", extname], binmode: true)
+      tempfile = Tempfile.new([tempfile_name || "down-http", extname], binmode: true)
 
       stream_body(response) do |chunk|
         tempfile.write(chunk)
@@ -80,7 +82,7 @@ module Down
     end
 
     # Starts retrieving the remote file and returns an IO-like object which
-    # downloads the response body on-demand. Accepts HTTP.rb options via a hash
+    # downloads the response body on-demand. Accepts HTTPX options via a hash
     # or a block.
     def open(url, rewindable: true, **options, &block)
       response = request(@client, url, stream: true, **options, &block)
@@ -103,7 +105,15 @@ module Down
 
     # Yields chunks of the response body to the block.
     def stream_body(response, &block)
-      response.each(&block)
+      response.each do |chunk|
+        next if (300..399).include?(response.status)
+
+        block.call(chunk)
+      end
+
+      if (300..399).include?(response.status)
+        raise Down::TooManyRedirects, "too many redirects"
+      end
     rescue => exception
       request_error!(exception)
     end
@@ -111,7 +121,7 @@ module Down
     def request(client, url, method: @method, **options, &block)
       response = send_request(client, method, url, **options, &block)
       response.raise_for_status
-      response_error!(response) unless (200..299).include?(response.status)
+      response_error!(response) unless (200..399).include?(response.status)
       response
     rescue HTTPX::HTTPError
       response_error!(response)
@@ -138,7 +148,6 @@ module Down
       args = [response.status.to_s, response]
 
       case response.status
-      when 300..399 then raise Down::TooManyRedirects, "too many redirects"
       when 404      then raise Down::NotFound.new(*args)
       when 400..499 then raise Down::ClientError.new(*args)
       when 500..599 then raise Down::ServerError.new(*args)

data/lib/down/net_http.rb

@@ -41,6 +41,8 @@ module Down
       headers             = options.delete(:headers)
       uri_normalizer      = options.delete(:uri_normalizer)
       extension           = options.delete(:extension)
+      auth_on_redirect    = options.delete(:auth_on_redirect)
+      tempfile_name       = options.delete(:tempfile_name)
 
       # Use open-uri's :content_lenth_proc or :progress_proc to raise an
       # exception early if the file is too large.
@@ -91,11 +93,11 @@ module Down
         uri.password = nil
       end
 
-      open_uri_file = open_uri(uri, open_uri_options, follows_remaining: max_redirects)
+      open_uri_file = open_uri(uri, open_uri_options, follows_remaining: max_redirects, auth_on_redirect: auth_on_redirect)
 
       # Handle the fact that open-uri returns StringIOs for small files.
       extname = extension ? ".#{extension}" : File.extname(open_uri_file.base_uri.path)
-      tempfile = ensure_tempfile(open_uri_file, extname)
+      tempfile = ensure_tempfile(open_uri_file, extname, tempfile_name)
       OpenURI::Meta.init tempfile, open_uri_file # add back open-uri methods
       tempfile.extend Down::NetHttp::DownloadedFile
 
@@ -107,14 +109,15 @@ module Down
     def open(url, *args, **options)
       options = merge_options(@options, *args, **options)
 
-      max_redirects  = options.delete(:max_redirects)
-      uri_normalizer = options.delete(:uri_normalizer)
+      max_redirects    = options.delete(:max_redirects)
+      uri_normalizer   = options.delete(:uri_normalizer)
+      auth_on_redirect = options.delete(:auth_on_redirect)
 
       uri = ensure_uri(normalize_uri(url, uri_normalizer: uri_normalizer))
 
       # Create a Fiber that halts when response headers are received.
       request = Fiber.new do
-        net_http_request(uri, options, follows_remaining: max_redirects) do |response|
+        net_http_request(uri, options, follows_remaining: max_redirects, auth_on_redirect: auth_on_redirect) do |response|
           Fiber.yield response
         end
       end
@@ -141,7 +144,7 @@ module Down
     private
 
     # Calls open-uri's URI::HTTP#open method. Additionally handles redirects.
-    def open_uri(uri, options, follows_remaining:)
+    def open_uri(uri, options, follows_remaining:, auth_on_redirect:)
       uri.open(options)
     rescue OpenURI::HTTPRedirect => exception
       raise Down::TooManyRedirects, "too many redirects" if follows_remaining == 0
@@ -155,6 +158,10 @@ module Down
         raise ResponseError.new("Invalid Redirect URI: #{exception.uri}", response: response)
       end
 
+      # do not leak credentials on redirect
+      options.delete("Authorization") unless auth_on_redirect
+      options.delete(:http_basic_authentication) unless auth_on_redirect
+
       # forward cookies on the redirect
       if !exception.io.meta["set-cookie"].to_s.empty?
         options["Cookie"] ||= ''
@@ -182,8 +189,8 @@ module Down
     # Converts the given IO into a Tempfile if it isn't one already (open-uri
     # returns a StringIO when there is less than 10KB of content), and gives
     # it the specified file extension.
-    def ensure_tempfile(io, extension)
-      tempfile = Tempfile.new(["down-net_http", extension], binmode: true)
+    def ensure_tempfile(io, extension, tempfile_name = nil)
+      tempfile = Tempfile.new([tempfile_name || "down-net_http", extension], binmode: true)
 
       if io.is_a?(Tempfile)
         # Windows requires file descriptors to be closed before files are moved
@@ -200,7 +207,7 @@ module Down
     end
 
     # Makes a Net::HTTP request and follows redirects.
-    def net_http_request(uri, options, follows_remaining:, &block)
+    def net_http_request(uri, options, follows_remaining:, auth_on_redirect:, &block)
       http, request = create_net_http(uri, options)
 
       begin
@@ -231,10 +238,17 @@ module Down
           raise ResponseError.new("Invalid Redirect URI: #{response["Location"]}", response: response)
         end
 
+        # do not leak credentials on redirect
+        options[:headers].delete("Authorization") unless auth_on_redirect
+
         # handle relative redirects
-        location = uri + location if location.relative?
+        if location.relative?
+          location = uri + location
+          uri.user = nil unless auth_on_redirect
+          uri.password = nil unless auth_on_redirect
+        end
 
-        net_http_request(location, options, follows_remaining: follows_remaining - 1, &block)
+        net_http_request(location, options, follows_remaining: follows_remaining - 1, auth_on_redirect: auth_on_redirect, &block)
       end
     end
 

data/lib/down/version.rb

@@ -1,5 +1,5 @@
 # frozen-string-literal: true
 
 module Down
-  VERSION = "5.4.1"
+  VERSION = "5.5.0"
 end

data/lib/down/wget.rb

@@ -1,5 +1,7 @@
 # frozen-string-literal: true
 
+warn "The Down::Wget backend has been deprecated and will be removed in the next major version. Please use the Net::HTTP/open-uri, http.rb or HTTPX backend instead."
+
 begin
   require "posix-spawn"
 rescue LoadError

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: down
 version: !ruby/object:Gem::Version
-  version: 5.4.1
+  version: 5.5.0
 platform: ruby
 authors:
 - Janko Marohnić
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-05-20 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: addressable
@@ -24,20 +23,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.8'
+- !ruby/object:Gem::Dependency
+  name: base64
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.8'
+        version: '6.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.8'
+        version: '6.0'
 - !ruby/object:Gem::Dependency
   name: mocha
   requirement: !ruby/object:Gem::Requirement
@@ -72,50 +85,36 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.22'
-    - - ">="
+        version: '1.0'
+    - - "<"
       - !ruby/object:Gem::Version
-        version: 0.22.2
+        version: 1.4.4
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.22'
-    - - ">="
+        version: '1.0'
+    - - "<"
       - !ruby/object:Gem::Version
-        version: 0.22.2
+        version: 1.4.4
 - !ruby/object:Gem::Dependency
   name: http
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.0'
+        version: '6.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '5.0'
-- !ruby/object:Gem::Dependency
-  name: posix-spawn
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
+        version: '6.0'
 - !ruby/object:Gem::Dependency
-  name: http_parser.rb
+  name: warning
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -129,7 +128,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: warning
+  name: csv
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -142,7 +141,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description:
 email:
 - janko.marohnic@gmail.com
 executables: []
@@ -167,7 +165,6 @@ homepage: https://github.com/janko/down
 licenses:
 - MIT
 metadata: {}
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -175,15 +172,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.3'
+      version: '2.7'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.12
-signing_key:
+rubygems_version: 3.6.7
 specification_version: 4
-summary: Robust streaming downloads using Net::HTTP, HTTP.rb or wget.
+summary: Robust streaming downloads using Net::HTTP, http.rb or HTTPX.
 test_files: []