httpx 0.16.1 → 0.18.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 005e6856a12b549480cb2fcf8becd893e041026e9df634fa102fc410ab3527ef
-  data.tar.gz: 00722d3ffe0822eb6f434df84db78e9ffb9b53cbd103220546dc611f3763117c
+  metadata.gz: e2f0ac68d969d4462c03c28fa818fa055e4e3aabad5a599197df6cc912274673
+  data.tar.gz: '069b5100a14183e79a968f4bb69363304ce046d098d1283f934568ea0fc23e06'
 SHA512:
-  metadata.gz: e8c2c8c1ff44c845a9773e79b3564ed7491cbbcbfd6aa20ea8e93a7a4a029f5afe177c53693173b493efe54122324448ee14e34807aa37db77d3699cc7bb75e2
-  data.tar.gz: bad8e79d228c37dd3efafcf530ba7188e4759e5aca807c7b693a7102b6a2bc339663bade32d78c80a62e691f8a8378731f864639965554ab78f4f031994f0626
+  metadata.gz: bddcf91c4cd851204ff789798bc8745e6e7f2b39face3ae77c83de4aa8be4de278f24a0ce1405f5e142f2f358fbf78472a5a816ae315ac43574c81adb6b82838
+  data.tar.gz: f9d992536ae6e464833dbe05c6a884ba373f84d90e42657bd18bab9fc69288561361041563dbe26dee92e9b9f337ee57d7eb7b9f7586abe705905184cd57f9e3

data/README.md

@@ -45,7 +45,7 @@ response = HTTPX.get("https://nghttp2.org")
 puts response.status #=> 200
 body = response.body
 puts body #=> #<HTTPX::Response ...
-``` 
+```
 
 You can also send as many requests as you want simultaneously:
 
@@ -79,7 +79,7 @@ In Ruby, HTTP client implementations are a known cheap commodity. Why this one?
 
 ### Concurrency
 
-This library supports HTTP/2 seamlessly (which means, if the request is secure, and the server support ALPN negotiation AND HTTP/2, the request will be made through HTTP/2). If you pass multiple URIs, and they can utilize the same connection, they will run concurrently in it. 
+This library supports HTTP/2 seamlessly (which means, if the request is secure, and the server support ALPN negotiation AND HTTP/2, the request will be made through HTTP/2). If you pass multiple URIs, and they can utilize the same connection, they will run concurrently in it.
 
 However if the server supports HTTP/1.1, it will try to use HTTP pipelining, falling back to 1 request at a time if the server doesn't support it (if the server support Keep-Alive connections, it will reuse the same connection).
 
@@ -137,7 +137,8 @@ In order to use HTTP/2 under JRuby, [check this link](https://gitlab.com/honeyry
 
 ### Known bugs
 
-Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryderchuck/httpx/issues/36)).
+* Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryderchuck/httpx/issues/36)).
+* Using `total_timeout` along with the `:persistent` plugin [does not work as you might expect](https://gitlab.com/honeyryderchuck/httpx/-/wikis/Timeouts#total_timeout).
 
 ## Contributing
 

data/doc/release_notes/0_17_0.md

@@ -0,0 +1,49 @@
+# 0.17.0
+
+## Features
+
+### Response mime type decoders (#json, #form)
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Response-Handling#response-decoding
+
+Two new methods, `#json` and `#form`, were added to `HTTPX::Response`. As the name implies, they'll decode the raw payload into ruby objects you can work with.
+
+```ruby
+# after HTTPX.get("https://api.smth/endpoint-returning-json")
+response.json # same as JSON.dump(response.to_s)
+```
+
+Although not yet documented, integrating custom decoders is also possible (i.e. parsing HTML with `nokogiri` or something similar).
+
+## Improvements
+
+### Connection: reduce interest calculations
+
+Due to it being an intensive task, internal interest calculation in connections was reduce to the bare minimum.
+
+### Immutable Options, internal recycling of instances, improves memory usage in the happy path
+
+A lot of effort went into avoiding generating options objects internally whenever necessary. This means, when sending several requests with the same set of options (the most common case in `httpx` usage), internally only one object is passed around. For that, the following improvements were done:
+
+* `Options#merge` returns the same options the the options being merged are a subset of the current set of options (b126938a6547e09b726dd64298fb488891d938e9).
+* `Session#build_request` bypasses instantiation of options if it receives an `Options` object (which happens internally in the happy path, if users don't call `#build_request` directly) (3d549817cb41d4b904102fdc61afe3ecd9170893).
+* Improving internal `Session` APIs to not pass around options, and instead rely on accessing request options.
+* `Options#to_hash` does not build internal garbage arrays anymore (cc02679b804f63798f5d2136a039be1624e96ab6).
+
+### Reduce regexp operations in the HTTP/1 parser
+
+Some code paths in the HTTP/1 parser still using regular expressions were replaced by string operations accomplishing the same.
+
+### HTTP/1 improvements on the complexity of connection accounting calculations
+
+Managing open HTTP/1 connections relies on operations calculating whether there are requests waiting for completion. This relied on traversing all requests for that connectionn (O(n)); it now only checks the completion state of the first and last request of that connection, given that all requests in HTTP/1 are sequential (O(1)); this optimization brings a big improvement to persistent and pipelined requests (65261217b1270913e4bb93717e8b8dcfa775565a).
+
+## Bugfixes
+
+* fixing HTTP/1 protocol uncompliant exposing multiple values for the "Host" header (e435dd0534314508262184fb03d83124d89d2079).
+
+* Custom response finalizer introduced in 0.16.0 has been reverted. It was brought to my attention that `Tempfile` implementation already takes care of the file on GC (and `httpx` was duplicating), and the approach taken in `httpx` was buggy in several ways (not tolerant to forks, never recycled finalizers...) (aa3be21c890f92a41afcc7931f01dd24cc801f7c).
+
+## Chore
+
+RBS Typing improvements based on latest stdlib signatures additions, such as `openssl`, `digest`, `socket` and others.

data/doc/release_notes/0_18_0.md

@@ -0,0 +1,69 @@
+# 0.18.0
+
+## Features
+
+### Response Cache
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Response-Cache
+
+The `:response_cache` plugin handles transparent usage of HTTP caching and conditional requests to improve performance and bandwidth usage.
+
+```ruby
+client = HTTPX.plugin(:response_cache)
+r1 = client.get("https://nghttp2.org/httpbin/cache")
+r2 = client.get("https://nghttp2.org/httpbin/cache")
+
+r1.status #=> 200
+r2.status #=> 304
+r1.body == r2.body #=> true
+```
+
+### jitter on "retry-after"
+
+On the `:retries` plugin, jitter calculation is now applied to the value in seconds defined by user after which a request should be retried (i.e. if `:retry_after` option is set to `2`, the retry interval may be `1.5422312` seconds, for example). This is important to avoid cases of synchronized "thundering herd", where server rejects requests, but they all get retried at the same time because the retry interval is exactly the same.
+
+You can override the jitter calculation function by using the [:retry_jitter](https://gitlab.com/honeyryderchuck/httpx/-/wikis/Retries#retry_jitter) option:
+
+```ruby
+HTTPX.plugin(:retries, retry_after: 2, retry_jitter: ->(interval) { interval + rand }) # interval is 3
+```
+
+You can opt out of this by setting `HTTPX_NO_JITTER=1` environment variable.
+
+### Response#error
+
+`HTTPX::Response#error` was added, to match `HTTPX::Response#error`. It returns an exception for 4xx/5xx responses (`HTTPX::HTTPError`), `nil` otherwise. It allows for end users to write such code:
+
+```ruby
+if (response = HTTPX.get(uri)).error
+  # success
+else
+  #error
+end
+```
+
+## Improvements
+
+* `webmock` adapter: added support for "stub_http_request#to_timeout" (https://gitlab.com/honeyryderchuck/httpx/-/merge_requests/165).
+
+## timers not a dependency
+
+The functionality provided by the `timers` gem was replaced by a simpler custom implementation. Although powerful, its complexity was somewhat unnecessary for `httpx`'s simpler event loop, where user-defined timeouts are usually the same for a given batch of requests. The removal of `timers` reduces the number of dependencies to 1, which is `http-2-next` and is maintained by me.
+
+## AWS plugins
+
+* `aws_sdk_authentication` plugin: removed implementation relying on  `aws-sdk-s3`, replacing it with an `aws-sdk-core` relying implementation, which only uses credentials strategies and region discovery (the whole point of this SDK is to use a minimal subset of AWS SDK).
+
+
+## Bugfixes
+
+* Fixed Error class declaration on response decoders when mime type is invalid (https://gitlab.com/honeyryderchuck/httpx/-/merge_requests/166).
+* `ErrorResponse#to_s` now removes ANSI escape sequences from error backtraces.
+* Persistent connections were kept around both in the pool and in the selector; the first is necessary, but the second caused busy loop scenarios all over; they are now removed when no requests are being handled.
+* Connections which failed connection handshake were removed from the pool, but not from the selector list, causing busy loop scenarios in a few cases; this has been fixed.
+* Fixed issue where HTTP/2 streams were being closed twice (and signaling it also twice), messing connection accounting in the pool.
+* DoH resolver was always subscribed to the default thread "connection pool", which broke scenarios were session was patched to use its custom pool; it now ensures that it subscribes to the pool it was created in.
+* `:aws_sigv4` plugin: removed require of `aws-sdk-s3`, left there by mistake (the whole point of the plugin is to run without the AWS SDK).
+## Chore
+
+* `HTTPX::ErrorResponse#status` is now deprecated.

data/doc/release_notes/0_18_1.md

@@ -0,0 +1,12 @@
+# 0.18.1
+
+## Bugfixes
+
+* HTTP/1.1 pipelining logs were logging the previously-buffered requests all together for each triggered request, which created some confusion for users when reporting errors. This has been fixed.
+* HTTP/2 coalescing is now skipped when performing TLS connections with VERIFY_NONE.
+* HTTP/2 peer GOAWAY frames will now result in a (retryable) connection error, instead of being ignored and leaving a "ghost" connection behind.
+* fixed total timeout call which was not raising the exception.
+
+## Chore
+
+This gem now requires MFA-based gem releases.

data/doc/release_notes/0_18_2.md

@@ -0,0 +1,10 @@
+# 0.18.2
+
+## Bugfixes
+
+* A bug was reported and fixed, whereby a persistent connection with a `:total_timeout` set was triggering the timeout and leaving the process looping indefinitely.
+
+
+## Chore
+
+The quirk of using the `:persistent` plugin with `:total_timeout` has been documented: https://gitlab.com/honeyryderchuck/httpx/-/wikis/Timeouts#total_timeout.

data/lib/httpx/adapters/datadog.rb

@@ -64,7 +64,7 @@ module Datadog
           def finish(response)
             return unless @span
 
-            if response.respond_to?(:error)
+            if response.is_a?(::HTTPX::ErrorResponse)
               @span.set_error(response.error)
             else
               @span.set_tag(Datadog::Ext::HTTP::STATUS_CODE, response.status.to_s)

data/lib/httpx/adapters/faraday.rb

@@ -22,6 +22,8 @@ module Faraday
       # :nocov:
 
       module RequestMixin
+        using ::HTTPX::HashExtensions
+
         private
 
         def build_request(env)
@@ -38,7 +40,7 @@ module Faraday
           timeout_options = {
             connect_timeout: env.request.open_timeout,
             operation_timeout: env.request.timeout,
-          }.reject { |_, v| v.nil? }
+          }.compact
 
           options = {
             ssl: {},
@@ -101,7 +103,7 @@ module Faraday
           end
 
           def on_response(&blk)
-            if block_given?
+            if blk
               @on_response = lambda do |response|
                 blk.call(response)
               end
@@ -112,7 +114,7 @@ module Faraday
           end
 
           def on_complete(&blk)
-            if block_given?
+            if blk
               @on_complete = blk
               self
             else

data/lib/httpx/adapters/webmock.rb

@@ -19,7 +19,7 @@ module WebMock
       module InstanceMethods
         private
 
-        def send_requests(*requests, options)
+        def send_requests(*requests)
           request_signatures = requests.map do |request|
             request_signature = _build_webmock_request_signature(request)
             WebMock::RequestRegistry.instance.requested_signatures.put(request_signature)
@@ -47,7 +47,7 @@ module WebMock
 
           unless real_requests.empty?
             reqs = real_requests.keys
-            reqs.zip(super(*reqs, options)).each do |req, res|
+            reqs.zip(super(*reqs)).each do |req, res|
               idx = real_requests[req]
 
               if WebMock::CallbackRegistry.any_callbacks?
@@ -86,7 +86,9 @@ module WebMock
         end
 
         def _build_from_webmock_response(request, webmock_response)
-          return ErrorResponse.new(request, webmock_response.exception, request.options) if webmock_response.exception
+          return _build_error_response(request, HTTPX::TimeoutError.new(1, "Timed out")) if webmock_response.should_timeout
+
+          return _build_error_response(request, webmock_response.exception) if webmock_response.exception
 
           response = request.options.response_class.new(request,
                                                         webmock_response.status[0],
@@ -95,6 +97,10 @@ module WebMock
           response << webmock_response.body.dup
           response
         end
+
+        def _build_error_response(request, exception)
+          HTTPX::ErrorResponse.new(request, exception, request.options)
+        end
       end
     end
 

data/lib/httpx/altsvc.rb

@@ -10,14 +10,14 @@ module HTTPX
     module_function
 
     def cached_altsvc(origin)
-      now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      now = Utils.now
       @altsvc_mutex.synchronize do
         lookup(origin, now)
       end
     end
 
     def cached_altsvc_set(origin, entry)
-      now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      now = Utils.now
       @altsvc_mutex.synchronize do
         return if @altsvcs[origin].any? { |altsvc| altsvc["origin"] == entry["origin"] }
 

data/lib/httpx/chainable.rb

@@ -4,9 +4,9 @@ module HTTPX
   module Chainable
     %i[head get post put delete trace options connect patch].each do |meth|
       class_eval(<<-MOD, __FILE__, __LINE__ + 1)
-        def #{meth}(*uri, **options)
-          request(:#{meth}, uri, **options)
-        end
+        def #{meth}(*uri, **options)        # def get(*uri, **options)
+          request(:#{meth}, uri, **options) #   request(:get, uri, **options)
+        end                                 # end
       MOD
     end
 
@@ -78,7 +78,7 @@ module HTTPX
       with(option.to_sym => (args.first || options))
     end
 
-    def respond_to_missing?(meth)
+    def respond_to_missing?(meth, *)
       return super unless meth =~ /\Awith_(.+)/
 
       option = Regexp.last_match(1)

data/lib/httpx/connection/http1.rb

@@ -36,6 +36,8 @@ module HTTPX
 
       request = @requests.first
 
+      return unless request
+
       return :w if request.interests == :w || !@buffer.empty?
 
       :r
@@ -59,7 +61,12 @@ module HTTPX
     def empty?
       # this means that for every request there's an available
       # partial response, so there are no in-flight requests waiting.
-      @requests.empty? || @requests.all? { |request| !request.response.nil? }
+      @requests.empty? || (
+        # checking all responses can be time-consuming. Alas, as in HTTP/1, responses
+        # do not come out of order, we can get away with checking first and last.
+        !@requests.first.response.nil? &&
+        (@requests.size == 1 || !@requests.last.response.nil?)
+      )
     end
 
     def <<(data)
@@ -260,7 +267,7 @@ module HTTPX
     def set_protocol_headers(request)
       if !request.headers.key?("content-length") &&
          request.body.bytesize == Float::INFINITY
-        request.chunk!
+        request.body.chunk!
       end
 
       connection = request.headers["connection"]
@@ -278,17 +285,16 @@ module HTTPX
         # on the last request of the possible batch (either allowed max requests,
         # or if smaller, the size of the batch itself)
         requests_limit = [@max_requests, @requests.size].min
-        if request != @requests[requests_limit - 1]
-          "keep-alive"
-        else
+        if request == @requests[requests_limit - 1]
           "close"
+        else
+          "keep-alive"
         end
       end
 
-      {
-        "host" => (request.headers["host"] || request.authority),
-        "connection" => connection,
-      }
+      extra_headers = { "connection" => connection }
+      extra_headers["host"] = request.authority unless request.headers.key?("host")
+      extra_headers
     end
 
     def headline_uri(request)
@@ -309,8 +315,9 @@ module HTTPX
     end
 
     def join_headers(request)
-      @buffer << "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}" << CRLF
-      log(color: :yellow) { "<- HEADLINE: #{@buffer.to_s.chomp.inspect}" }
+      headline = "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}"
+      @buffer << headline << CRLF
+      log(color: :yellow) { "<- HEADLINE: #{headline.chomp.inspect}" }
       extra_headers = set_protocol_headers(request)
       join_headers2(request.headers.each(extra_headers))
       log { "<- " }
@@ -318,7 +325,7 @@ module HTTPX
     end
 
     def join_body(request)
-      return if request.empty?
+      return if request.body.empty?
 
       while (chunk = request.drain_body)
         log(color: :green) { "<- DATA: #{chunk.bytesize} bytes..." }
@@ -327,7 +334,9 @@ module HTTPX
         throw(:buffer_full, request) if @buffer.full?
       end
 
-      raise request.drain_error if request.drain_error
+      return unless (error = request.drain_error)
+
+      raise error
     end
 
     def join_trailers(request)
@@ -354,7 +363,7 @@ module HTTPX
     }.freeze
 
     def capitalized(field)
-      UPCASED[field] || field.to_s.split("-").map(&:capitalize).join("-")
+      UPCASED[field] || field.split("-").map(&:capitalize).join("-")
     end
   end
   Connection.register "http/1.1", Connection::HTTP1

data/lib/httpx/connection/http2.rb

@@ -1,7 +1,6 @@
 # frozen_string_literal: true
 
 require "securerandom"
-require "io/wait"
 require "http/2/next"
 
 module HTTPX
@@ -17,6 +16,12 @@ module HTTPX
       end
     end
 
+    class GoawayError < Error
+      def initialize
+        super(0, :no_error)
+      end
+    end
+
     attr_reader :streams, :pending
 
     def initialize(buffer, options)
@@ -56,7 +61,7 @@ module HTTPX
 
       return :w if !@pending.empty? && can_buffer_more_requests?
 
-      return :w if @streams.each_key.any? { |r| r.interests == :w }
+      return :w unless @drains.empty?
 
       if @buffer.empty?
         return if @streams.empty? && @pings.empty?
@@ -218,7 +223,7 @@ module HTTPX
       log(level: 1, color: :yellow) do
         request.headers.merge(extra_headers).each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{v}" }.join("\n")
       end
-      stream.headers(request.headers.each(extra_headers), end_stream: request.empty?)
+      stream.headers(request.headers.each(extra_headers), end_stream: request.body.empty?)
     end
 
     def join_trailers(stream, request)
@@ -234,7 +239,7 @@ module HTTPX
     end
 
     def join_body(stream, request)
-      return if request.empty?
+      return if request.body.empty?
 
       chunk = @drains.delete(request) || request.drain_body
       while chunk
@@ -249,7 +254,9 @@ module HTTPX
         chunk = next_chunk
       end
 
-      on_stream_refuse(stream, request, request.drain_error) if request.drain_error
+      return unless (error = request.drain_error)
+
+      on_stream_refuse(stream, request, error)
     end
 
     ######
@@ -257,8 +264,10 @@ module HTTPX
     ######
 
     def on_stream_headers(stream, request, h)
-      if request.response && request.response.version == "2.0"
-        on_stream_trailers(stream, request, h)
+      response = request.response
+
+      if response.is_a?(Response) && response.version == "2.0"
+        on_stream_trailers(stream, response, h)
         return
       end
 
@@ -274,11 +283,11 @@ module HTTPX
       handle(request, stream) if request.expects?
     end
 
-    def on_stream_trailers(stream, request, h)
+    def on_stream_trailers(stream, response, h)
       log(color: :yellow) do
         h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{v}" }.join("\n")
       end
-      request.response.merge_headers(h)
+      response.merge_headers(h)
     end
 
     def on_stream_data(stream, request, data)
@@ -288,23 +297,25 @@ module HTTPX
     end
 
     def on_stream_refuse(stream, request, error)
-      stream.close
       on_stream_close(stream, request, error)
+      stream.close
     end
 
     def on_stream_close(stream, request, error)
+      return if error == :stream_closed && !@streams.key?(request)
+
       log(level: 2) { "#{stream.id}: closing stream" }
       @drains.delete(request)
       @streams.delete(request)
 
-      if error && error != :no_error
+      if error
         ex = Error.new(stream.id, error)
         ex.set_backtrace(caller)
         response = ErrorResponse.new(request, ex, request.options)
         emit(:response, request, response)
       else
         response = request.response
-        if response.status == 421
+        if response && response.status == 421
           ex = MisdirectedRequestError.new(response)
           ex.set_backtrace(caller)
           emit(:error, request, ex)
@@ -339,9 +350,16 @@ module HTTPX
 
     def on_close(_last_frame, error, _payload)
       is_connection_closed = @connection.state == :closed
-      if error && error != :no_error
+      if error
         @buffer.clear if is_connection_closed
-        ex = Error.new(0, error)
+        if error == :no_error
+          ex = GoawayError.new
+          @pending.unshift(*@streams.keys)
+          @drains.clear
+          @streams.clear
+        else
+          ex = Error.new(0, error)
+        end
         ex.set_backtrace(caller)
         handle_error(ex)
       end
@@ -385,10 +403,10 @@ module HTTPX
     end
 
     def on_pong(ping)
-      if !@pings.delete(ping.to_s)
-        close(:protocol_error, "ping payload did not match")
-      else
+      if @pings.delete(ping.to_s)
         emit(:pong)
+      else
+        close(:protocol_error, "ping payload did not match")
       end
     end
   end