httpx 1.8.3 → 1.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5faeb9a08adff992ddd2661b4ccd5176cb2c49d17dbb6a0ea593e1a00c10f35b
-  data.tar.gz: 04f81429bbe03762c641531f0886c9b133f8fa04f8607943534ae36736f705e7
+  metadata.gz: e272ec707fd07e29993d6f5eff50f84f40ad4da61503552f1de2698a8b56b72b
+  data.tar.gz: 10ebe441bd62cc1fec49c3caedb7a564dd998d2dd49d0740b3224f583f23e506
 SHA512:
-  metadata.gz: e750874b2e206ff8604e804fa526847cc40c5511608d818c3a4a7c3136b383f5ca7c8baffbc856bd080ca0b77becc9ebc10d88e7743b5a757c7dd00b49a7e57c
-  data.tar.gz: 0e6e2f5aaf85be60d2fc044cad8ad7e215ae9a7768c11681f0ea5a5bcb264f3f7b8fb3aaf5a21bb2c94a8b03445ac26e0972e637ebc3a30d4f204238d79dbb96
+  metadata.gz: 860445bf67dc3bb69f9724d1dcc13f5a7c21a936420addd0923aa4d955be929afd95b87fb032f5a226d245d65b4e07654b6df431992e755341583a5f0e5c26f9
+  data.tar.gz: 4f11cd526c5099fa92d47fbd6b08a4366af6a01f4075f3df96f861cf7f3ce33e17a34e69f436b453e51d8afa07535b2bc62c04e1c1f260942782b0f432cc7383

data/doc/release_notes/1_8_4.md

@@ -0,0 +1,13 @@
+# 1.8.4
+
+## Improvements
+
+* send requests which were not sent, or were not processed by the server, when the server sends back a GOAWAY frame (which is deemed recoverable, like when there's no error), after forcing a reconnection (instead of bubbling up the GOAWAY error for all of them).
+
+## Bugfixes
+
+* `:retries` plugin: also retry requests which failed due to a RST_STREAM frame with REFUSED_STREAM error code.
+* `:retries` plugin: resend requests which were piped to a closed connection internally.
+* `:proxy` plugin: fix issue related with reconnecting inactive connections shut down by the peer, when used with the `:persistent` plugin, where the CONNECT request wasn't being sent to establish the tunnel.
+* fixed compatibility issue starting with the `json` gem 3.0.0, when calling `HTTPX::Response#json`.
+* avoid bookkepping issue in session, where the `on_response_arrived` on a request may be called more than once (which it can happen in the `:follow_redirects` plugin, where a request may have completed due to an error, while the request is waiting to be resent due to `retry-after`).

data/doc/release_notes/1_9_0.md

@@ -0,0 +1,75 @@
+# 1.9.0
+
+## Features
+
+### `:hedging` plugin
+
+The `:hedging` implements a strategy called "request hedging", whereby a request is sent a request two times, over different connections, and picking up whichever arrives first.
+
+```ruby
+http = HTTP.plugin(:hedging)
+http.get("http://www.example.com")
+# sent two times, only the fastest one comes back...
+```
+
+ou can read more about it in https://gitlab.com/os85/httpx/wikis/Hedging .
+
+### `:proxy/http` plugin: new options `:proxy_headers` and `:enable_proxy_tunnel`
+
+The `:proxy/http` plugin (which gets automatically loaded by the `:proxy` plugin) gains two new options.
+
+#### `:proxy_headers`
+
+Can be passed a separate set of HTTP headers, which are going to be sent to the proxy when establishing the the CONNECT proxy tunnel.
+
+```ruby
+http = HTTPX.plugin(:proxy).with(proxy_headers: { "x-proxy-1" => "foo" }, headers: { "x-header-1" => "bar" }).get("https://example.com")
+# -> CONNECT example.com HTTP/1.1
+# -> Host: example.com
+# -> X-Proxy-1: foo
+# ...
+# <- 200 OK
+# ...
+# -> GET / HTTP/1.1
+# -> X-Headers-1: bar
+# ...
+```
+
+#### `:enable_proxy_tunnel`
+
+When set to `true`, it forces the establishment of a CONNECT proxy tunnel before sending the request. Bear in mind that this will be the norm for `https://` requests, so only use this in cases where the requests are to be sent to an `http://` URL and a CONNECT proxy tunnel is required.
+
+## Improvements
+
+* `:request_timeout` will also observe the connect phase, in case a request is sent to an idle connection. This means that, if you're using `:request_timeout`, you no longer need to specify `:connect_timeout`.
+  * The same will happen if you're using `:total_request_timeout`.
+* do not require "io/wait" anymore for ruby 3.2 or higher.
+* `:proxy/http` plugin: raise a `HTTPX::ConnectTimeoutError` if there was a timeout error while waiting for the CONNECT request to complete.
+
+## Bugfixes
+
+* `:auth` plugin: fixed clause inverted check, which caused caused a new token to always be generated on retry.
+* `:auth` plugin: take auth_header_expires_at into account when deciding to regenerate on retry (token may have been expired by the time the request will be retried).
+* `:tracing` plugin: fixed request initial time calculation in order to take into account the time to wait for a PING correctly.
+* `:tracing` plugin: fixed request initial time calculation for pending requests which were waiting for the connect phase and errored meanwhie.
+* `:proxy/http` plugin: `:headers` will not be sent anymore to the proxy when establishing the CONNECT proxy tunnel.
+* socket connect timeout bookkeeping is fixed, where in a multi request scenario, connect timeout errors shall not be raised too early if an error happens before the remainder of connections elapses.
+* `:proxy` plugin: allow proxy params to be frozen, which also fixes an issue where proxies identified as unhealthy by a previous request, would still be retried on for different hostnames.
+* `:callbacks` plugin: fix broken visibility of session `#close` method.
+* native resolver: retry dns queries on not-imp and refused errors (treats them as transient resolver errors).
+* `webmock` adapter: fix emission of `:response` callback so that this only happens after the body is fully buffered (which makes the `:callbacks` plugin `.on_response_completed` callback works the same way when and when not stubbed).
+* `:addresses`: fix support for the documented variant where a hostname/ips hash is passed.
+* `http2`: fix interests calculation, so that:
+  * a connection expects to read from connections which still have in-transit streams.
+  * an HTTP/2 with no available flow control window in its streams will wait for incoming WINDOW_UPDATE frames.
+* `http2`: pause buffering the request body if there's no available control window in the corresponding stream or connection.
+* `http2`: reset the ping timer too if a connection receives a GOAWAY frame while waiting for a PING to be ack'ed.
+* wait for read or write for any operation, considering that `OpenSSL::SSL::SSLSocket` objects may return both `.wait_readable` and `:wait_writable` for reads and writes.
+* when using `:max_requests`, also account for inflight requests when deciding to send it to the parser or the pending queue.
+* fixed `HTTPX::Options#merge` which was not freezing the resulting options object, something which happens when enabling plugins, which made resulting sessions unusable across Ractors.
+* fixed the `Alt-Svc` header parser to be able to parse values with multiple authorities.
+* fixed request params handling so that they are passed to all requests invoked (instead of only the first of them).
+
+## Chore
+
+* Emit warning message whenever connection is deactivated with pending requests; the message contains internal state which should help debug it.

data/lib/httpx/adapters/webmock.rb

@@ -133,10 +133,10 @@ module WebMock
             request.transition(:done)
             response.finish!
             request.response = response
+            response << mock_response.body.dup unless response.is_a?(HTTPX::ErrorResponse)
             request.emit_response(response)
             request_signature.headers = request.headers.to_h
 
-            response << mock_response.body.dup unless response.is_a?(HTTPX::ErrorResponse)
           elsif WebMock.net_connect_allowed?(request_signature.uri)
             if WebMock::CallbackRegistry.any_callbacks?
               request.on(:response) do |resp|

data/lib/httpx/altsvc.rb

@@ -10,7 +10,7 @@ module HTTPX
 
       H2_ALTSVC_SCHEMES = %w[https h2].freeze
 
-      ALTSVC_IGNORE_IVARS = %i[@ssl].freeze
+      ALTSVC_IGNORE_IVARS = %i[ssl].freeze
 
       def send(request)
         request.headers["alt-used"] = @origin.authority if @parser && !@write_buffer.full? && match_altsvcs?(request.uri)
@@ -30,7 +30,7 @@ module HTTPX
       # +uri+
       def match_altsvcs?(uri)
         @origins.any? { |origin| altsvc_match?(uri, origin) } ||
-          AltSvc.cached_altsvc(@origin).any? do |altsvc|
+          AltSvc.cached_altsvc(@origin.to_s).any? do |altsvc|
             origin = altsvc["origin"]
             altsvc_match?(origin, uri.origin)
           end
@@ -115,33 +115,45 @@ module HTTPX
         return
       end
 
-      parse(altsvc) do |alt_origin, alt_params|
+      parse(altsvc, host: host) do |alt_origin, alt_params|
         alt_origin.host ||= host
         yield(alt_origin, origin, alt_params)
       end
     end
 
-    def parse(altsvc)
-      return enum_for(__method__, altsvc) unless block_given?
+    def parse(altsvc, host: nil)
+      return enum_for(__method__, altsvc, host: host) unless block_given?
 
       scanner = StringScanner.new(altsvc)
       until scanner.eos?
-        alt_service = scanner.scan(/[^=]+=("[^"]+"|[^;,]+)/)
+        alt_services = [] #: Array[String]
 
-        alt_params = []
+        while (alt_service = scanner.scan(/[^=]+=("[^"]+")/))
+          alt_services << alt_service
+          scanner.skip(/[, ]+/)
+        end
+
+        scanner.skip(/;/)
+
+        alt_params = [] #: Array[String]
         loop do
           alt_param = scanner.scan(/[^=]+=("[^"]+"|[^;,]+)/)
           alt_params << alt_param.strip if alt_param
           scanner.skip(/;/)
           break if scanner.eos? || scanner.scan(/ *, */)
         end
-        alt_params = Hash[alt_params.map { |field| field.split("=", 2) }]
+        alt_params = Hash[alt_params.map do |field|
+          field.split("=", 2)
+        end]
 
-        alt_proto, alt_authority = alt_service.split("=", 2)
-        alt_origin = parse_altsvc_origin(alt_proto, alt_authority)
-        return unless alt_origin
+        alt_services.each do |alt_service|
+          alt_proto, alt_authority = alt_service.split("=", 2)
+          alt_origin = parse_altsvc_origin(alt_proto, alt_authority, host: host)
 
-        yield(alt_origin, alt_params.merge("proto" => alt_proto))
+          next unless alt_origin
+
+          yield(alt_origin, alt_params.merge("proto" => alt_proto))
+        end
       end
     end
 
@@ -149,19 +161,21 @@ module HTTPX
       case alt_proto
       when "h2c"
         "http"
-      when "h2"
+      when "h2", "quic", /^h3/
         "https"
       end
     end
 
-    def parse_altsvc_origin(alt_proto, alt_origin)
+    def parse_altsvc_origin(alt_proto, alt_origin, host: nil)
       alt_scheme = parse_altsvc_scheme(alt_proto)
 
       return unless alt_scheme
 
       alt_origin = alt_origin[1..-2] if alt_origin.start_with?("\"") && alt_origin.end_with?("\"")
 
-      URI.parse("#{alt_scheme}://#{alt_origin}")
+      uri = URI.parse("#{alt_scheme}://#{alt_origin}")
+      uri.host = host if uri.host.nil? || uri.host.empty?
+      uri
     end
   end
 end

data/lib/httpx/callbacks.rb

@@ -14,9 +14,9 @@ module HTTPX
       end
     end
 
-    def emit(type, *args)
+    def emit(type, ...)
       log { "emit #{type.inspect} callbacks" } if respond_to?(:log)
-      callbacks(type).delete_if { |pr| :delete == pr.call(*args) } # rubocop:disable Style/YodaCondition
+      callbacks(type).delete_if { |pr| :delete == pr.call(...) } # rubocop:disable Style/YodaCondition
     end
 
     def callbacks_for?(type)

data/lib/httpx/chainable.rb

@@ -5,18 +5,21 @@ module HTTPX
   # delegates to a default session when extended.
   module Chainable
     %w[head get post put delete trace options connect patch].each do |meth|
+      http_verb = meth.upcase
       class_eval(<<-MOD, __FILE__, __LINE__ + 1)
+        # sends a #{http_verb} request for the given +uri+, using +options+ for each.
         def #{meth}(*uri, **options)                # def get(*uri, **options)
-          request("#{meth.upcase}", uri, **options) #   request("GET", uri, **options)
+          request("#{http_verb}", uri, **options)   #   request("GET", uri, **options)
         end                                         # end
       MOD
     end
 
     # delegates to the default session (see HTTPX::Session#request).
-    def request(*args, **options)
-      branch(default_options).request(*args, **options)
+    def request(...)
+      branch(default_options).request(...)
     end
 
+    # returns a new session with the "Accept" header set to +type+.
     def accept(type)
       with(headers: { "accept" => String(type) })
     end

data/lib/httpx/connection/http1.rb

@@ -34,7 +34,7 @@ module HTTPX
     end
 
     def timeout
-      @options.timeout[:operation_timeout]
+      @options.timeout.operation_timeout
     end
 
     def interests
@@ -48,10 +48,6 @@ module HTTPX
     end
 
     def reset
-      if @ping_timer
-        @ping_timer.cancel
-        @ping_timer = nil
-      end
       @max_requests = @options.max_requests || MAX_REQUESTS
       @parser.reset!
       @handshake_completed = false
@@ -92,14 +88,12 @@ module HTTPX
       @parser << data
     end
 
-    def send(request)
-      unless @max_requests.positive?
-        @pending << request
+    def send(request, head = false)
+      unless @max_requests.positive? && @max_requests > @requests.size
+        head ? @pending.unshift(request) : @pending << request
         return
       end
 
-      return if @requests.include?(request)
-
       @requests << request
       @pipelining = @max_concurrent_requests > 1 && @requests.size > 1
     end
@@ -132,9 +126,9 @@ module HTTPX
       headers = request.options.headers_class.new(h)
       response = request.options.response_class.new(request,
                                                     @parser.status_code,
-                                                    @parser.http_version.join("."),
+                                                    @parser.http_version,
                                                     headers)
-      request.log(color: :yellow) { "-> HEADLINE: #{response.status} HTTP/#{@parser.http_version.join(".")}" }
+      request.log(color: :yellow) { "-> HEADLINE: #{response.status} HTTP/#{@parser.http_version}" }
       request.log(color: :yellow) { response.headers.each.map { |f, v| "-> HEADER: #{f}: #{log_redact_headers(v)}" }.join("\n") }
 
       if response.content_length && response.content_length > request.options.max_response_body_size
@@ -187,7 +181,7 @@ module HTTPX
 
       @request = nil
       @requests.shift
-      response = request.response
+      response = request.response #: Response
       emit(:response, request, response)
 
       if @parser.upgrade?
@@ -198,19 +192,17 @@ module HTTPX
 
       @parser.reset!
       @max_requests -= 1
-      if response.is_a?(ErrorResponse)
-        disable
-      else
-        manage_connection(request, response)
-      end
+      manage_connection(request, response)
 
       if exhausted?
         @pending.unshift(*@requests)
         @requests.clear
 
         emit(:exhausted)
+      elsif @requests.any?
+        consume
       else
-        send(@pending.shift) unless @pending.empty?
+        send(@pending.shift, true) unless @pending.empty?
       end
     end
 

data/lib/httpx/connection/http2.rb

@@ -23,9 +23,29 @@ module HTTPX
     end
 
     class GoawayError < Error
-      def initialize(code = :no_error)
+      UNRECOVERABLE_ERRORS = %i[settings_timeout inadequate_security].freeze
+
+      def initialize(code, last_stream_id)
+        @code = code
+        @last_stream_id = last_stream_id
         super(0, code)
       end
+
+      def last_stream_id
+        return Float::INFINITY if unrecoverable?
+
+        @last_stream_id
+      end
+
+      def unrecoverable?
+        UNRECOVERABLE_ERRORS.include?(@code)
+      end
+    end
+
+    class RstStreamError < Error
+    end
+
+    class RefusedStreamError < RstStreamError
     end
 
     attr_reader :streams, :pending
@@ -48,9 +68,9 @@ module HTTPX
     end
 
     def timeout
-      return @options.timeout[:operation_timeout] if @handshake_completed
+      return @options.timeout.operation_timeout if @handshake_completed
 
-      @options.timeout[:settings_timeout]
+      @options.timeout.settings_timeout
     end
 
     def interests
@@ -77,27 +97,45 @@ module HTTPX
         return :r
       end
 
+      # only wait for writable if we're not waiting on more streams,
+      # or waiting on ping ACKs.
+      nothing_to_wait_for = @streams.empty? && @pings.empty?
+
       # there are pending bufferable requests
-      return :w if !@pending.empty? && can_buffer_more_requests?
+      if !@pending.empty? && can_buffer_more_requests?
+        # only wait for writable if we're not waiting on more streams,
+        # or waiting on ping ACKs.
+        return nothing_to_wait_for ? :w : :rw
+      end
 
-      # there are pending frames from the last run
-      return :w unless @drains.empty?
+      unless @drains.empty?
+        return :w unless @buffer.empty?
+
+        streams_with_no_window = @drains.none? do |stream, _|
+          # if an existing stream has buffered bytes, it must wait for
+          # WINDOW_UPDATE frames
+          stream.send_buffer.empty?
+        end
+
+        return streams_with_no_window ? :r : :rw
+
+      end
 
       if @buffer.empty?
         # skip if no more requests or pings to process
-        return if @streams.empty? && @pings.empty?
+        return if nothing_to_wait_for
 
         :r
       else
         # buffered frames
-        :w
+        nothing_to_wait_for ? :w : :rw
       end
     end
 
     def close
       unless @connection.closed?
         @connection.goaway
-        emit(:timeout, @options.timeout[:close_handshake_timeout])
+        emit(:timeout, @options.timeout.close_handshake_timeout)
       end
       emit(:close)
     end
@@ -149,18 +187,45 @@ module HTTPX
     end
 
     def handle_error(ex, request = nil)
-      if ex.is_a?(OperationTimeoutError) && !@handshake_completed && @connection.state != :closed
-        @connection.goaway(:settings_timeout, "closing due to settings timeout")
-        emit(:close_handshake)
-        settings_ex = SettingsTimeoutError.new(ex.timeout, ex.message)
-        settings_ex.set_backtrace(ex.backtrace)
-        ex = settings_ex
-      end
-      while (req, _ = @streams.shift)
+      last_stream_id = Float::INFINITY
+      case ex
+      when OperationTimeoutError
+        if !@handshake_completed && @connection.state != :closed
+          @connection.goaway(:settings_timeout, "closing due to settings timeout")
+          emit(:close_handshake)
+          settings_ex = SettingsTimeoutError.new(ex.timeout, ex.message)
+          settings_ex.set_backtrace(ex.backtrace)
+          ex = settings_ex
+        end
+      when GoawayError
+        last_stream_id = ex.last_stream_id
+      end
+
+      inflight_unprocessed_requests = [] #: Array[Request]
+
+      while (req, stream = @streams.shift)
+        @drains.delete(stream)
+
         next if request && request == req
 
+        if stream.id > last_stream_id
+          req.transition(:idle)
+          # unprocessed request
+          inflight_unprocessed_requests << req
+
+          next
+        end
+
         emit(:error, req, ex)
       end
+
+      if ex.is_a?(GoawayError) && !ex.unrecoverable?
+        # resend unprocessed requests on a different connection
+        @pending.unshift(*inflight_unprocessed_requests) if inflight_unprocessed_requests.any?
+        emit(:exhausted, ex) if @pending.any?
+        return
+      end
+
       while (req = @pending.shift)
         next if request && request == req
 
@@ -277,13 +342,23 @@ module HTTPX
     def join_body(stream, request)
       return if request.body.empty?
 
-      chunk = @drains.delete(request) || request.drain_body
+      # the next chunk waits until the peer grants more window.
+      throw(:buffer_full) if @drains.key?(stream) && (!stream.send_buffer.empty? || !@connection.send_buffer.empty?)
+
+      chunk = @drains.delete(stream) || request.drain_body
       while chunk
         next_chunk = request.drain_body
         send_chunk(request, stream, chunk, next_chunk)
 
-        if next_chunk && (@buffer.full? || request.body.unbounded_body?)
-          @drains[request] = next_chunk
+        if next_chunk && (
+          # send window is full
+          @buffer.full? ||
+          # stream / chunked request, must write a chunk at a time
+          request.body.unbounded_body? ||
+          # peer window is full, must wait for WINDOW_UPDATE frame(s)
+          !stream.send_buffer.empty? || !@connection.send_buffer.empty?
+        )
+          @drains[stream] = next_chunk
           throw(:buffer_full)
         end
 
@@ -395,7 +470,14 @@ module HTTPX
         when :http_1_1_required
           emit(:error, request, error)
         else
-          ex = Error.new(stream.id, error)
+          ex =
+            case error
+            when :refused_stream
+              RefusedStreamError.new(stream.id, error)
+            else
+              RstStreamError.new(stream.id, error)
+            end
+
           ex.set_backtrace(caller)
           response = ErrorResponse.new(request, ex)
           request.response = response
@@ -410,7 +492,10 @@ module HTTPX
           end
         end
       end
-      send(@pending.shift) unless @pending.empty?
+
+      consume
+
+      send(@pending.shift, true) unless @pending.empty?
 
       return unless @streams.empty? && exhausted?
 
@@ -432,7 +517,7 @@ module HTTPX
       send_pending
     end
 
-    def on_close(_last_frame, error, _payload)
+    def on_close(last_stream_id, error, _payload)
       is_connection_closed = @connection.closed?
       if error
         @buffer.clear if is_connection_closed
@@ -442,12 +527,11 @@ module HTTPX
             emit(:error, request, error)
           end
         else
-          ex = GoawayError.new(error)
+          ex = GoawayError.new(error, last_stream_id)
           ex.set_backtrace(caller)
 
           handle_error(ex)
           teardown
-
         end
       end
       return unless is_connection_closed && @streams.empty?
@@ -539,8 +623,9 @@ module HTTPX
 
     def teardown(request = nil)
       if request
-        @drains.delete(request)
-        @streams.delete(request)
+        if (stream = @streams.delete(request))
+          @drains.delete(stream)
+        end
       else
         @drains.clear
         @streams.clear