httpx 0.20.5 → 0.21.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 47766fc5b4fb9cb70e7b0feb8333656b987a25aea262c645c36de53e7cc3f464
-  data.tar.gz: 2f49417d21841b803d9f44b76c199785a54b7983c7712cf8a39b4837900e892b
+  metadata.gz: 1195773336c2fb0b92ed10abfafa8a99c7cfd19af31b932dbfff72d073ed3f47
+  data.tar.gz: 634e9e62edcd8184d014b1870cbc303cdfcc8035e0c76431b10c1e9143bdbb23
 SHA512:
-  metadata.gz: f74fd61a16e738fbe668312e14d7c658af062f4f3dbcf65360b33892e27f31a524e7b675136de5e7693e327ba9a388c74c995e43b4e49a4593ec63fa385d07df
-  data.tar.gz: b04ae701b05663bd8c94d74416fd6a124d3a700e5860065d373f452383cc6f48a5569e8696dfac5a7f398824c163b3ddb62553a56c465022dc35ac34e627a479
+  metadata.gz: bee27d21eef64771e054b4517e99371462cc28fb1c5cad4353137e111e96a4d22131d573aa5c31fe6b86c7007466feec5d1181ae6787a3a6ef87459468d911c8
+  data.tar.gz: 15fa97c4456ce2d8f875590075c9ceb778096aca70d27a793c5cafa026f038331439ac62fd823210290013c0466c072a2eff5c1a0a06b1c880254e542d3f9e16

data/README.md

@@ -19,13 +19,17 @@ And also:
 
 * Compression (gzip, deflate, brotli)
 * Streaming Requests
-* Authentication (Basic Auth, Digest Auth)
+* Authentication (Basic Auth, Digest Auth, NTLM)
 * Expect 100-continue
 * Multipart Requests
-* Cookies
+* Advanced Cookie handling
 * HTTP/2 Server Push
-* H2C Upgrade
+* HTTP/1.1 Upgrade (support for "h2c", "h2")
 * Automatic follow redirects
+* GRPC
+* WebDAV
+* Circuit Breaker
+* HTTP-based response cache
 * International Domain Names
 
 ## How
@@ -36,7 +40,14 @@ Here are some simple examples:
 HTTPX.get("https://nghttp2.org").to_s #=> "<!DOCT...."
 ```
 
-And that's the simplest one there is.
+And that's the simplest one there is. But you can also do:
+
+```ruby
+HTTPX.post("http://example.com", form: { user: "john", password: "pass" })
+
+http = HTTPX.with(headers: { "x-my-name" => "joe" })
+http.patch(("http://example.com/file", body: File.open("path/to/file")) # request body is streamed
+```
 
 If you want to do some more things with the response, you can get an `HTTPX::Response`:
 
@@ -50,7 +61,7 @@ puts body #=> #<HTTPX::Response ...
 You can also send as many requests as you want simultaneously:
 
 ```ruby
-page1, page2, page3 = HTTPX.get("https://news.ycombinator.com/news", "https://news.ycombinator.com/news?p=2", "https://news.ycombinator.com/news?p=3")
+page1, page2, page3 =`HTTPX.get("https://news.ycombinator.com/news", "https://news.ycombinator.com/news?p=2", "https://news.ycombinator.com/news?p=3")
 ```
 
 ## Installation
@@ -73,43 +84,53 @@ and then just require it in your program:
 require "httpx"
 ```
 
-## Why Should I care?
+## What makes it the best ruby HTTP client
 
-In Ruby, HTTP client implementations are a known cheap commodity. Why this one?
 
-### Concurrency
+### Concurrency, HTTP/2 support
 
-This library supports HTTP/2 seamlessly (which means, if the request is secure, and the server support ALPN negotiation AND HTTP/2, the request will be made through HTTP/2). If you pass multiple URIs, and they can utilize the same connection, they will run concurrently in it.
+`httpx` supports HTTP/2 (for "https" requests, it'll automatically do ALPN negotiation). However if the server supports HTTP/1.1, it will use HTTP pipelining, falling back to 1 request at a time if the server doesn't support it either (and it'll use Keep-Alive connections, unless the server does not support).
 
-However if the server supports HTTP/1.1, it will try to use HTTP pipelining, falling back to 1 request at a time if the server doesn't support it (if the server support Keep-Alive connections, it will reuse the same connection).
+If you passed multiple URIs, it'll perform all of the requests concurrently, by mulitplexing on the necessary sockets (and it'll batch requests to the same socket when the origin is the same):
+
+```ruby
+HTTPX.get(
+  "https://news.ycombinator.com/news",
+  "https://news.ycombinator.com/news?p=2",
+  "https://google.com/q=me"
+) # first two requests will be multiplexed on the same socket.
+```
 
 ### Clean API
 
 `httpx` builds all functions around the `HTTPX` module, so that all calls can compose of each other. Here are a few examples:
 
 ```ruby
-response = HTTPX.get("https://www.google.com")
-response = HTTPX.post("https://www.nghttp2.org/httpbin/post", params: {name: "John", age: "22"})
+response = HTTPX.get("https://www.google.com", params: { q: "me" })
+response = HTTPX.post("https://www.nghttp2.org/httpbin/post", form: {name: "John", age: "22"})
 response = HTTPX.plugin(:basic_authentication)
                 .basic_authentication("user", "pass")
                 .get("https://www.google.com")
+
+# more complex client objects can be cached, and are thread-safe
+http = HTTPX.plugin(:compression).plugin(:expect).with(headers: { "x-pvt-token" => "TOKEN"})
+http.get("https://example.com") # the above options will apply
+http.post("https://example2.com",  form: {name: "John", age: "22"}) # same, plus the form POST body
 ```
 
 ### Lightweight
 
-It ships with a plugin system similar to the ones used by [sequel](https://github.com/jeremyevans/sequel), [roda](https://github.com/jeremyevans/roda) or [shrine](https://github.com/janko-m/shrine).
-
-It means that it loads the bare minimum to perform requests, and the user has to explicitly load the plugins, in order to get the features he/she needs.
+It ships with most features published as a plugin, making vanilla `httpx` lightweight and dependency-free, while allowing you to "pay for what you use"
 
-It also means that it ships with the minimum amount of dependencies.
+The plugin system is similar to the ones used by [sequel](https://github.com/jeremyevans/sequel), [roda](https://github.com/jeremyevans/roda) or [shrine](https://github.com/janko-m/shrine).
 
-### DNS-over-HTTPS
+### Advanced DNS features
 
-`HTTPX` ships with custom DNS resolver implementations, including a DNS-over-HTTPS resolver.
+`HTTPX` ships with custom DNS resolver implementations, including a native Happy Eyeballs resolver immplementation, and a DNS-over-HTTPS resolver.
 
-## Easy to test
+## User-driven test suite
 
-The test suite runs against [httpbin proxied over nghttp2](https://nghttp2.org/httpbin/), so there are no mocking/stubbing false positives. The test suite uses [minitest](https://github.com/seattlerb/minitest), but its matchers usage is (almost) limited to `#assert` (`assert` is all you need).
+The test suite runs against [httpbin proxied over nghttp2](https://nghttp2.org/httpbin/), so actual requests are performed during tests.
 
 ## Supported Rubies
 
@@ -122,18 +143,15 @@ All Rubies greater or equal to 2.1, and always latest JRuby and Truffleruby.
 | ------------- | --------------------------------------------------- |
 | Website       | https://honeyryderchuck.gitlab.io/httpx/            |
 | Documentation | https://honeyryderchuck.gitlab.io/httpx/rdoc/       |
-| Wiki          | https://gitlab.com/honeyryderchuck/httpx/wikis/home |
+| Wiki          | https://honeyryderchuck.gitlab.io/httpx/wiki/home.html |
 | CI            | https://gitlab.com/honeyryderchuck/httpx/pipelines  |
+| Rubygems      | https://rubygems.org/gems/httpx                     |
 
 ## Caveats
 
 ### ALPN support
 
-`HTTPS` TLS backend is ruby's own `openssl` gem.
-
-If your requirement is to run requests over HTTP/2 and TLS, make sure you run a version of the gem which compiles OpenSSL 1.0.2 (Ruby 2.3 and higher are guaranteed to).
-
-In order to use HTTP/2 under JRuby, [check this link](https://gitlab.com/honeyryderchuck/httpx/-/wikis/JRuby-Truffleruby-Other-Rubies) to know what to do.
+ALPN negotiation is required for "auto" HTTP/2 "https" requests. This is available in ruby since version 2.3 .
 
 ### Known bugs
 

data/doc/release_notes/0_13_0.md

@@ -34,7 +34,7 @@ HTTPX.get("http://example.com", addresses: %w[172.5.3.1 172.5.3.2]))
 You should also use it to connect to HTTP servers bound to a UNIX socket, in which case you'll have to provide a path:
 
 ```ruby
-HTTPX.get("http://example.com", addresses: %w[/path/to/usocket]))
+HTTPX.get("http://example.com", transport: "unix", addresses: %w[/path/to/usocket]))
 ```
 
 The `:transport_options` are therefore deprecated, and will be moved in a major version.

data/doc/release_notes/0_21_0.md

@@ -0,0 +1,96 @@
+# 0.21.0
+
+## Features
+
+### `:write_timeout`, `:read_timeout` and `:request_timeout`
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Timeouts
+
+The following timeouts are now supported:
+
+* `:write_timeout`: total time (in seconds) to write a request to the server;
+* `:read_timeout`: total time (in seconds) to read a response from the server;
+* `:request_timeout`: tracks both of the above (time to write the request and read a response);
+
+```ruby
+HTTPX.with(timeout: { request_timeout: 60}).get(...
+```
+
+Just like `:connect_timeout`, the new timeouts are deadline-oriented, rather than op-oriented, meaning that they do not reset on each socket operation (as most ruby HTTP clients do).
+
+None of them has a default value, in order not to break integrations, but that'll change in a future v1, where they'll become the default timeouts.
+
+### Circuit Breaker plugin
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Circuit-Breaker
+
+The `:circuit_breaker` plugin wraps around errors happening when performing HTTP requests, and support options for setting maximum number of attempts before circuit opens (`:circuit_breaker_max_attempts`), period after which attempts should be reset (`:circuit_breaker_reset_attempts_in`), timespan until circuit half-opens (`circuit_breaker_break_in`), respective half-open drip rate (`:circuit_breaker_half_open_drip_rate`), and a callback to do your own check on whether a response has failed, in case you want HTTP level errors to be marked as failed attempts (`:circuit_breaker_break_on`).
+
+Read the wiki for more info about the defaults.
+
+```ruby
+http = HTTPX.plugin(:circuit_breaker)
+# that's it!
+http.get(...
+```
+
+### WebDAV plugin
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/WebDav
+
+The `:webdav` introduces some "convenience" methods to perform common WebDAV operations.
+
+```ruby
+webdav = HTTPX.plugin(:webdav, origin: "http://webdav-server")
+              .plugin(:digest_authentication).digest_auth("user", "pass")
+
+res = webdav.put("/file.html", body: "this is the file body")
+res = webdav.copy("/file.html", "/newdir/copy.html")
+# ...
+```
+
+### XML transcoder, `:xml` option and `response.xml`
+
+A new transcoder was added fot the XML mime type, which requires `"nokogiri"` to be installed. It can both serialize Nokogiri nodes in a request, and parse response content into nokogiri nodes:
+
+```ruby
+response = HTTPX.post("https://xml-server.com", xml: Nokogiri::XML("<xml ..."))
+response.xml #=> #(Document:0x16e4 { name = "document", children = ...
+```
+
+## Improvements
+
+### `:proxy` plugin: `:no_proxy` option
+
+Support was added, in the `:proxy` plugin, to declare domains, either via regexp patterns, or strings, for which requests should bypass the proxy.
+
+```ruby
+http = HTTPX.plugin(:proxy).with_proxy(
+    uri: "http://10.10.0.1:51432",
+    no_proxy: ["gitlab.local", /*.google.com/]
+)
+http.get("https://duckduckgo.com/?q=httpx") #=> proxied
+http.get("https://google.com/?q=httpx") #=> not proxied
+http.get("https://gitlab.com") #=> proxied
+http.get("https://gitlab.local") #=> not proxied
+```
+
+### OOTB support for other JSON libraries
+
+If one of `multi_json`, `oj` or `yajl` is available, all `httpx` operations doing JSON parsing or dumping will use it (the `json` standard library will be used otherwise).
+
+```ruby
+require "oj"
+require "httpx"
+
+response = HTTPX.post("https://somedomain.json", json: { "foo" => "bar" }) # will use "oj"
+puts response.json # will use "oj"
+```
+
+## Bugfixes
+
+* `:expect` plugin: `:expect_timeout` can accept floats (not just integers).
+
+## Chore
+
+* DoH `:https` resolver: support was removed for the "application/dns-json" mime-type (it was only supported in practice by the Google DoH resolver, which has since added support for the standardized "application/dns-message").

data/doc/release_notes/0_21_1.md

@@ -0,0 +1,12 @@
+# 0.21.1
+
+## Bugfixes
+
+* fix: protecting tcp connect phase against low-level syscall errors
+  * such as network unreachable, which can happen if connectivity is lost meanwhile.
+* native resolver: fix for nameserver switch not happening in case of DNS timeout.
+  * when more than a nameserver was advertised by the system.
+
+## Chore
+
+* Removing usage of deprecated `Random::DEFAULT.rand` (using `Random.rand` instead)-

data/lib/httpx/connection/http1.rb

@@ -118,7 +118,7 @@ module HTTPX
       log(color: :yellow) { response.headers.each.map { |f, v| "-> HEADER: #{f}: #{v}" }.join("\n") }
 
       @request.response = response
-      on_complete if response.complete?
+      on_complete if response.finished?
     end
 
     def on_trailers(h)
@@ -158,6 +158,7 @@ module HTTPX
       @request = nil
       @requests.shift
       response = request.response
+      response.finish!
       emit(:response, request, response)
 
       if @parser.upgrade?

data/lib/httpx/connection.rb

@@ -34,6 +34,7 @@ module HTTPX
     include Callbacks
 
     using URIExtensions
+    using NumericExtensions
 
     require "httpx/connection/http2"
     require "httpx/connection/http1"
@@ -233,6 +234,7 @@ module HTTPX
           # when pushing a request into an existing connection, we have to check whether there
           # is the possibility that the connection might have extended the keep alive timeout.
           # for such cases, we want to ping for availability before deciding to shovel requests.
+          log(level: 3) { "keep alive timeout expired, pinging connection..." }
           @pending << request
           parser.ping
           transition(:active) if @state == :inactive
@@ -430,6 +432,8 @@ module HTTPX
       @inflight += 1
       parser.send(request)
 
+      set_request_timeouts(request)
+
       return unless @state == :inactive
 
       transition(:active)
@@ -508,9 +512,14 @@ module HTTPX
 
     def transition(nextstate)
       handle_transition(nextstate)
-    rescue Errno::ECONNREFUSED,
+    rescue Errno::ECONNABORTED,
+           Errno::ECONNREFUSED,
+           Errno::ECONNRESET,
            Errno::EADDRNOTAVAIL,
            Errno::EHOSTUNREACH,
+           Errno::EINVAL,
+           Errno::ENETUNREACH,
+           Errno::EPIPE,
            TLSError => e
       # connect errors, exit gracefully
       handle_error(e)
@@ -573,7 +582,7 @@ module HTTPX
           error = ex
         else
           # inactive connections do not contribute to the select loop, therefore
-          # they should fail due to such errors.
+          # they should not fail due to such errors.
           return if @state == :inactive
 
           if @timeout
@@ -591,10 +600,45 @@ module HTTPX
     def handle_error(error)
       parser.handle_error(error) if @parser && parser.respond_to?(:handle_error)
       while (request = @pending.shift)
-        response = ErrorResponse.new(request, error, @options)
+        response = ErrorResponse.new(request, error, request.options)
         request.response = response
         request.emit(:response, response)
       end
     end
+
+    def set_request_timeouts(request)
+      write_timeout = request.write_timeout
+      request.once(:headers) do
+        @timers.after(write_timeout) { write_timeout_callback(request, write_timeout) }
+      end unless write_timeout.nil? || write_timeout.infinite?
+
+      read_timeout = request.read_timeout
+      request.once(:done) do
+        @timers.after(read_timeout) { read_timeout_callback(request, read_timeout) }
+      end unless read_timeout.nil? || read_timeout.infinite?
+
+      request_timeout = request.request_timeout
+      request.once(:headers) do
+        @timers.after(request_timeout) { read_timeout_callback(request, request_timeout, RequestTimeoutError) }
+      end unless request_timeout.nil? || request_timeout.infinite?
+    end
+
+    def write_timeout_callback(request, write_timeout)
+      return if request.state == :done
+
+      @write_buffer.clear
+      error = WriteTimeoutError.new(request, nil, write_timeout)
+      on_error(error)
+    end
+
+    def read_timeout_callback(request, read_timeout, error_type = ReadTimeoutError)
+      response = request.response
+
+      return if response && response.finished?
+
+      @write_buffer.clear
+      error = error_type.new(request, request.response, read_timeout)
+      on_error(error)
+    end
   end
 end

data/lib/httpx/errors.rb

@@ -24,6 +24,24 @@ module HTTPX
 
   class ConnectTimeoutError < TimeoutError; end
 
+  class RequestTimeoutError < TimeoutError
+    attr_reader :request
+
+    def initialize(request, response, timeout)
+      @request = request
+      @response = response
+      super(timeout, "Timed out after #{timeout} seconds")
+    end
+
+    def marshal_dump
+      [message]
+    end
+  end
+
+  class ReadTimeoutError < RequestTimeoutError; end
+
+  class WriteTimeoutError < RequestTimeoutError; end
+
   class SettingsTimeoutError < TimeoutError; end
 
   class ResolveTimeoutError < TimeoutError; end

data/lib/httpx/extensions.rb

@@ -54,6 +54,14 @@ module HTTPX
     Numeric.__send__(:include, NegMethods)
   end
 
+  module NumericExtensions
+    refine Numeric do
+      def infinite?
+        self == Float::INFINITY
+      end unless Numeric.method_defined?(:infinite?)
+    end
+  end
+
   module StringExtensions
     refine String do
       def delete_suffix!(suffix)
@@ -135,10 +143,6 @@ module HTTPX
   end
 
   module RegexpExtensions
-    # If you wonder why this is there: the oauth feature uses a refinement to enhance the
-    # Regexp class locally with #match? , but this is never tested, because ActiveSupport
-    # monkey-patches the same method... Please ActiveSupport, stop being so intrusive!
-    # :nocov:
     refine(Regexp) do
       def match?(*args)
         !match(*args).nil?

data/lib/httpx/io/unix.rb

@@ -33,7 +33,7 @@ module HTTPX
       else
         if @options.transport_options
           # :nocov:
-          warn ":#{__method__} is deprecated, use :addresses instead"
+          warn ":transport_options is deprecated, use :addresses instead"
           @path = @options.transport_options[:path]
           # :nocov:
         else

data/lib/httpx/options.rb

@@ -10,6 +10,7 @@ module HTTPX
     OPERATION_TIMEOUT = 60
     KEEP_ALIVE_TIMEOUT = 20
     SETTINGS_TIMEOUT = 10
+    READ_TIMEOUT = WRITE_TIMEOUT = REQUEST_TIMEOUT = Float::INFINITY
 
     # https://github.com/ruby/resolv/blob/095f1c003f6073730500f02acbdbc55f83d70987/lib/resolv.rb#L408
     ip_address_families = begin
@@ -34,6 +35,9 @@ module HTTPX
         settings_timeout: SETTINGS_TIMEOUT,
         operation_timeout: OPERATION_TIMEOUT,
         keep_alive_timeout: KEEP_ALIVE_TIMEOUT,
+        read_timeout: READ_TIMEOUT,
+        write_timeout: WRITE_TIMEOUT,
+        request_timeout: REQUEST_TIMEOUT,
       },
       :headers => {},
       :window_size => WINDOW_SIZE,
@@ -197,7 +201,7 @@ module HTTPX
     end
 
     %i[
-      params form json body ssl http2_settings
+      params form json xml body ssl http2_settings
       request_class response_class headers_class request_body_class
       response_body_class connection_class options_class
       io fallback_protocol debug debug_level transport_options resolver_class resolver_options
@@ -206,7 +210,7 @@ module HTTPX
       def_option(method_name)
     end
 
-    REQUEST_IVARS = %i[@params @form @json @body].freeze
+    REQUEST_IVARS = %i[@params @form @xml @json @body].freeze
     private_constant :REQUEST_IVARS
 
     def ==(other)
@@ -263,7 +267,7 @@ module HTTPX
         instance_variables.each do |ivar|
           value = other.instance_variable_get(ivar)
           value = case value
-                  when Symbol, Fixnum, TrueClass, FalseClass # rubocop:disable Lint/UnifiedInteger
+                  when Symbol, Numeric, TrueClass, FalseClass
                     value
                   else
                     value.dup

data/lib/httpx/plugins/circuit_breaker/circuit.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+module HTTPX
+  module Plugins::CircuitBreaker
+    #
+    # A circuit is assigned to a given absoolute url or origin.
+    #
+    # It sets +max_attempts+, the number of attempts the circuit allows, before it is opened.
+    # It sets +reset_attempts_in+, the time a circuit stays open at most, before it resets.
+    # It sets +break_in+, the time that must elapse before an open circuit can transit to the half-open state.
+    # It sets +circuit_breaker_half_open_drip_rate+, the rate of requests a circuit allows to be performed when in an half-open state.
+    #
+    class Circuit
+      def initialize(max_attempts, reset_attempts_in, break_in, circuit_breaker_half_open_drip_rate)
+        @max_attempts = max_attempts
+        @reset_attempts_in = reset_attempts_in
+        @break_in = break_in
+        @circuit_breaker_half_open_drip_rate = 1 - circuit_breaker_half_open_drip_rate
+        @attempts = 0
+        @state = :closed
+      end
+
+      def respond
+        try_close
+
+        case @state
+        when :closed
+          nil
+        when :half_open
+          # return nothing or smth based on ratio
+          return if Random.rand >= @circuit_breaker_half_open_drip_rate
+
+          @response
+        when :open
+
+          @response
+        end
+      end
+
+      def try_open(response)
+        return unless @state == :closed
+
+        now = Utils.now
+
+        if @attempts.positive?
+          @attempts = 0 if now - @attempted_at > @reset_attempts_in
+        else
+          @attempted_at = now
+        end
+
+        @attempts += 1
+
+        return unless @attempts >= @max_attempts
+
+        @state = :open
+        @opened_at = now
+        @response = response
+      end
+
+      def try_close
+        case @state
+        when :closed
+          nil
+        when :half_open
+          # reset!
+          @attempts = 0
+          @opened_at = @attempted_at = @response = nil
+          @state = :closed
+
+        when :open
+          @state = :half_open if Utils.elapsed_time(@opened_at) > @break_in
+        end
+      end
+    end
+  end
+end

data/lib/httpx/plugins/circuit_breaker/circuit_store.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+module HTTPX::Plugins::CircuitBreaker
+  using HTTPX::URIExtensions
+
+  class CircuitStore
+    def initialize(options)
+      @circuits = Hash.new do |h, k|
+        h[k] = Circuit.new(
+          options.circuit_breaker_max_attempts,
+          options.circuit_breaker_reset_attempts_in,
+          options.circuit_breaker_break_in,
+          options.circuit_breaker_half_open_drip_rate
+        )
+      end
+    end
+
+    def try_open(uri, response)
+      circuit = get_circuit_for_uri(uri)
+
+      circuit.try_open(response)
+    end
+
+    # if circuit is open, it'll respond with the stored response.
+    # if not, nil.
+    def try_respond(request)
+      circuit = get_circuit_for_uri(request.uri)
+
+      circuit.respond
+    end
+
+    private
+
+    def get_circuit_for_uri(uri)
+      uri = URI(uri)
+
+      if @circuits.key?(uri.origin)
+        @circuits[uri.origin]
+      else
+        @circuits[uri.to_s]
+      end
+    end
+  end
+end