httpx 1.8.0 → 1.8.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 74104ce6c4d05053e2a005974de5f0cd66346a1cb4558b7c0887d3a2d691d226
-  data.tar.gz: ca0ce3e30f509d44e4c0daac30a0b5d9949c118c19a715561c276f1debaf5dbf
+  metadata.gz: c34100ea496b94c4dbcfcd95330a5c8fa21dd5997d49dff54f62ced363589f7f
+  data.tar.gz: 24ef436a40b17cf4df3b2a9722c36437704c0c2ed1f5a1afc59b16801c36dbb4
 SHA512:
-  metadata.gz: 0de3c6baf56c86272cc45bdc06e1187655e759ddf329414d71fcbf4603c9d0e03791641e05f5c07ef84cd578a3bba1a855fde77b5b0f1f2e11df1cf60ea0815a
-  data.tar.gz: a276c6ed937c8d04103382d517d56d9df805886f23b05a38f6e654fcd6935e17407101e1eedc2b6605b4f347a37262116024a61633ebdd660131b06fc1f3fde9
+  metadata.gz: 7796301721da3f362b4ec13bf5bffcdec864819dddf404dfb7c7ffadde69508de32e52b85e163b144d4ade6b3579aa256107680332a264b27cbe67fdd9e23875
+  data.tar.gz: 63f47f3e7c9e50b720c8637dabc506fe30e79fb403e6d658646ff6fb894dc02774c8e1c2491d938392f1184d8f26f681361f9ffd1dec6675a622c03ba0441342

data/doc/release_notes/1_7_1.md

@@ -17,5 +17,4 @@
 * `:stream_bidi` plugin: fix internal state preventing bidi requests from being retried.
 * `:stream_bidi` plugin: will only allow initial request body being passed using `:body` param (others, like `:json`, will raise an exception)
 * https resolver: return `:idle` on `#state` calls when no connection is available (sometimes called in internal log messages).
-* selector loop fix: when there are no selectables and an interval is passed, sleep instead of returning (thereby avoiding potential busy loop).
-*
+* selector loop fix: when there are no selectables and an interval is passed, sleep instead of returning (thereby avoiding potential busy loop).

data/doc/release_notes/1_8_1.md

@@ -0,0 +1,27 @@
+# 1.8.1
+
+## Improvements
+
+* the session will now eagerly write initial data to connections instead of waiting for I/O the first time (saves a syscall).
+* removed all rescue blocks rescuing `StandardError` internally; this could interfere with usages of `httpx` inside `Timeout.timeout` blocks, which may emit an exception at any point, including in blocks where that exception could be rescued, leaving the session in a potentially irrecoverable state and looping.
+* debug logs change the display of ids from 'tid:123, fid:123' to 'tid=123 fid:123'.
+  * and when using the `datadog` adapter, all related correlation ids (trace id, span id, etc) will also be added to the identifiers in the same format
+* plugins built on top of the `:retries` plugin will skip the `retry_after` interval when certain conditions are met.
+  * `:persistent` plugin: request will be retried immediately, when the error happens after the keep alive interval expired.
+  * `:auth` plugin: request will be retried immediately if it failed due to a "401 Unauthorized Error".
+* `:ssrf_filter` plugin: removed public/private references of exception message.
+
+## Bugfixes
+
+* `:auth` plugin: handle edge case where `:auth_header_expires_at` returns no value.
+* `:digest_auth` plugin: do not retry if there was an issue building building the authorization header after the probe.
+* `webmock` adapter: prevent mocked requests from opening a connection to the peer server that will not be used.
+* clear and cancel request timers on completion or reset, ensuring that they won't be retained way past the point where the request is gone.
+* connection: cancel and discard ping timer before closing (removes timer reference)
+* connection: fix logic where purging-while-resending requests to a coalesced connection was leaving references in the old connection.
+* moved an `on_body_chunk` callback to the `:callbacks` plugin, where it's used and expected to be defined.
+* `http2`: avoid closing the connection on internal errors which are supposed to be specific to a request, such as the max headers or max response body size checks.
+* system resolver will use an internal exception when using `Timeout.timeout` on resolutions to avoid conflicts with the default timeout exception class.
+* prevent edge case where `#alpn_protocol` was being called on a `TCPSocket`.
+* fixed v1.8.0 regression causing connection to loop when an error happens while connecting.
+* resolver cache: fixed cache eviction logic, where alias answers weren't being evicted, causing resolutions to stall after a certain number of resolutions.

data/doc/release_notes/1_8_2.md

@@ -0,0 +1,30 @@
+# 1.8.2
+
+## Improvements
+
+* `:proxy` plugin: allow setting `:no_proxy` option to `"*"` (or `["*"]`) to allow bypassing proxy for all requests.
+* Extending sessions with unknown options will now raise an error, instead of silently ignoring them (consistent with the initialization case now):
+
+```ruby
+session = HTTPX.with(unknown_option: 1) # will raise a HTTPX::Error: unknown option: unknown_option
+```
+
+* Use `HTTP2::Connection#closed?` internally (instead of `#state`) to check for connection state; this will, in a future release of `http-2`, seamlessly support the new `:closing` state.
+* Add missing `response` attribute reader to `HTTPX::RequestTimeoutError`.
+* Improve correctness of calling `#dup` on requests and connections.
+
+## Bugfixes
+
+* `:auth` plugin: generate token first, then set the "expires_at" timestamp (bearer tokens, specially if JWT, may carry information about when it expires).
+* `:auth` plugin: do not check for token freshness on a retry, instead force token regeneration.
+* `:retries` plugin: do not apply "retry after" delay after a ping timeout error (the ping timeout already implies a delay, so retry on a new connection should be immediate).
+* `:retries` plugin: do not retry on any HTTP/2 frame error (only on GOAWAY or PING frame errors).
+* `:persistent` plugin: do not decrement request remaining retries on ping or GOAWAY frame errors.
+* `:tracing` plugin: set connection initialization time to nil no termination (so that it doesn't get reused on persistent connections after reuse).
+* `:follow_redirects` plugin: fixed issue when a timeout error would be raised but only bubble up to the root request, not properly closing the in-flight follow up request, and leaving the request handling phase in a broken state.
+* datadog adapter: the connection initialization time is now set only on connect or first request received, the reason, making it more accurate.
+* webmock adapter: `.to_timeout` helper will make the request raise a `HTTPX::RequestTimeoutError` instead of a plain `TimeoutError` (otherwise, if using persistent sessions, it'd trigger a retry instead and mess with webmock request count bookkeeping).
+* native resolver: do not transition to `:open` if the socket was closed during resolution (which happens when an error is raised while the query bytes are buffered).
+* fix for when setting a new session callback on an ssl context object which has been frozen after the first connection attempt (done by binding the ssl context callback to an internal ivar which can be changed).
+* connection: protect against `Errno::ETIMEDOUT` errors on socket close by forcing transition to closed.
+* do not set a ping timer for HTTP/1.1 on keep alive flow, as there's no ping and the connection is reestablished.

data/lib/httpx/adapters/datadog.rb

@@ -185,6 +185,13 @@ module Datadog::Tracing
         end
       end
 
+      # patches httpx debug logs to include datadog correlation ids
+      module LoggablePatch
+        def log_identifiers
+          "#{super} #{Datadog::Tracing.log_correlation}"
+        end
+      end
+
       module Configuration
         # Default settings for httpx
         #
@@ -297,6 +304,8 @@ module Datadog::Tracing
 
           ::HTTPX.send(:remove_const, :Session)
           ::HTTPX.send(:const_set, :Session, datadog_session.class)
+
+          ::HTTPX::Loggable.singleton_class.prepend(LoggablePatch)
         end
       end
 

data/lib/httpx/adapters/faraday.rb

@@ -115,11 +115,11 @@ module Faraday
             ssl_options
           end
         else
-          # :nocov:
+          # simplecov:disable
           def ssl_options_from_env(*)
             {}
           end
-          # :nocov:
+          # simplecov:enable
         end
       end
 

data/lib/httpx/adapters/webmock.rb

@@ -34,7 +34,8 @@ module WebMock
         end
 
         def build_from_webmock_response(request, webmock_response)
-          return build_error_response(request, HTTPX::TimeoutError.new(1, "Timed out")) if webmock_response.should_timeout
+          return build_error_response(request,
+                                      HTTPX::RequestTimeoutError.new(request, webmock_response, 1)) if webmock_response.should_timeout
 
           return build_error_response(request, webmock_response.exception) if webmock_response.exception
 
@@ -155,6 +156,12 @@ module WebMock
             raise WebMock::NetConnectNotAllowedError, request_signature
           end
         end
+
+        private
+
+        def connect
+          super unless @mocked
+        end
       end
     end
 

data/lib/httpx/buffer.rb

@@ -46,6 +46,11 @@ module HTTPX
       def_delegator :@buffer, :<<
     end
 
+    def initialize_dup(other)
+      super
+      @buffer = other.instance_variable_get(:@buffer).dup
+    end
+
     def full?
       @buffer.bytesize >= @limit
     end

data/lib/httpx/connection/http1.rb

@@ -29,7 +29,7 @@ module HTTPX
       @version = [1, 1]
       @pending = []
       @requests = []
-      @request = nil
+      @callbacks = @request = nil
       @handshake_completed = @pipelining = false
     end
 
@@ -162,18 +162,12 @@ module HTTPX
 
       return unless request
 
-      begin
-        request.log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
-        request.log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
+      request.log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
+      request.log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
 
-        response = request.response
+      response = request.response
 
-        response << chunk
-      rescue StandardError => e
-        error_response = ErrorResponse.new(request, e)
-        request.response = error_response
-        dispatch(request)
-      end
+      response << chunk
     end
 
     def on_complete
@@ -416,7 +410,7 @@ module HTTPX
     end
 
     def capitalized(field)
-      UPCASED[field] || field.split("-").map(&:capitalize).join("-")
+      UPCASED.fetch(field) { field.split("-").map(&:capitalize).join("-") }
     end
   end
 end

data/lib/httpx/connection/http2.rb

@@ -31,12 +31,14 @@ module HTTPX
     attr_reader :streams, :pending
 
     def initialize(buffer, options)
+      @callbacks = nil
       @options = options
       @settings = @options.http2_settings
       @pending = []
       @streams = {}
       @drains = {}
       @pings = []
+      @streams_to_close_after_receive = []
       @buffer = buffer
       @handshake_completed = false
       @wait_for_handshake = @settings.key?(:wait_for_handshake) ? @settings.delete(:wait_for_handshake) : true
@@ -52,7 +54,7 @@ module HTTPX
     end
 
     def interests
-      if @connection.state == :closed
+      if @connection.closed?
         return unless @handshake_completed
 
         return if @buffer.empty?
@@ -93,7 +95,7 @@ module HTTPX
     end
 
     def close
-      unless @connection.state == :closed
+      unless @connection.closed?
         @connection.goaway
         emit(:timeout, @options.timeout[:close_handshake_timeout])
       end
@@ -101,7 +103,7 @@ module HTTPX
     end
 
     def empty?
-      @connection.state == :closed || @streams.empty?
+      @connection.closed? || @streams.empty?
     end
 
     def exhausted?
@@ -110,6 +112,12 @@ module HTTPX
 
     def <<(data)
       @connection << data
+
+      while (stream, request, error = @streams_to_close_after_receive.shift)
+        # these streams were marked for cancellation due to errors found while processing the
+        # data received by the peer.
+        emit_stream_error(stream, request, error)
+      end
     end
 
     def send(request, head = false)
@@ -128,7 +136,7 @@ module HTTPX
     rescue ::HTTP2::Error::StreamLimitExceeded
       @pending.unshift(request)
       false
-    rescue StandardError => e
+    rescue ::HTTP2::Error::Error, ArgumentError => e
       emit(:error, request, e)
     end
 
@@ -333,6 +341,8 @@ module HTTPX
       @streams[request] = stream
 
       handle(request, stream) if request.expects?
+    rescue HTTPX::Error => e
+      @streams_to_close_after_receive << [stream, request, e]
     end
 
     def on_stream_trailers(stream, request, response, h)
@@ -345,7 +355,18 @@ module HTTPX
     def on_stream_data(stream, request, data)
       request.log(level: 1, color: :green) { "#{stream.id}: <- DATA: #{data.bytesize} bytes..." }
       request.log(level: 2, color: :green) { "#{stream.id}: <- #{log_redact_body(data.inspect)}" }
+
+      return unless request.response
+
       request.response << data
+    rescue HTTPX::Error => e
+      if stream.state == :closing
+        # there won't be any more chunks to process after this one.
+        emit_stream_error(stream, request, e)
+      else
+        # defer until the last chunk from the payload is processed.
+        @streams_to_close_after_receive << [stream, request, e]
+      end
     end
 
     def on_stream_refuse(stream, request, error)
@@ -381,11 +402,12 @@ module HTTPX
           emit(:response, request, response)
         end
       else
-        response = request.response
-        if response && response.is_a?(Response) && response.status == 421
-          emit(:error, request, :http_1_1_required)
-        else
-          emit(:response, request, response)
+        if (response = request.response)
+          if response.is_a?(Response) && response.status == 421
+            emit(:error, request, :http_1_1_required)
+          else
+            emit(:response, request, response)
+          end
         end
       end
       send(@pending.shift) unless @pending.empty?
@@ -411,7 +433,7 @@ module HTTPX
     end
 
     def on_close(_last_frame, error, _payload)
-      is_connection_closed = @connection.state == :closed
+      is_connection_closed = @connection.closed?
       if error
         @buffer.clear if is_connection_closed
         case error
@@ -509,6 +531,12 @@ module HTTPX
       emit(:pong)
     end
 
+    def emit_stream_error(stream, request, error)
+      teardown(request)
+      stream.close
+      emit(:error, request, error)
+    end
+
     def teardown(request = nil)
       if request
         @drains.delete(request)

data/lib/httpx/connection.rb

@@ -36,19 +36,20 @@ module HTTPX
 
     def_delegator :@write_buffer, :empty?
 
-    attr_reader :type, :io, :origin, :origins, :state, :pending, :options, :ssl_session, :sibling
+    attr_reader :type, :io, :origin, :origins, :state, :pending, :options, :ssl_session, :sibling,
+                :read_buffer, :write_buffer
 
     attr_writer :current_selector
 
     attr_accessor :current_session, :family
 
-    protected :ssl_session, :sibling
+    protected :ssl_session, :sibling, :read_buffer, :write_buffer
 
     def initialize(uri, options)
       @current_session = @current_selector = @max_concurrent_requests =
                            @parser = @sibling = @coalesced_connection = @altsvc_connection =
-                                                  @ping_timer = @family = @io = @ssl_session =
-                                                                            @timeout = @connected_at = @response_received_at = nil
+                                                  @callbacks = @ping_timer = @family =
+                                                                 @io = @ssl_session = @timeout = @connected_at = @response_received_at = nil
 
       @exhausted = @cloned = @main_sibling = false
 
@@ -76,6 +77,29 @@ module HTTPX
       self.addresses = @options.addresses if @options.addresses
     end
 
+    # dupped initialization
+    def initialize_dup(orig)
+      super
+      @callbacks = @parser = @sibling = @coalesced_connection = @altsvc_connection = nil
+      @origins = orig.origins.dup
+      @read_buffer = orig.read_buffer.dup
+      @write_buffer = orig.write_buffer.dup
+      @inflight = 0
+      @pending = []
+      transition(:idle)
+
+      if @io
+        # initialize new IO object with the same set of addresses
+        addresses = @io.addresses
+        @io = nil
+        self.addresses = addresses
+      end
+
+      return unless @current_session && @current_selector
+
+      @current_session.pin(self, @current_selector)
+    end
+
     def peer
       @origin
     end
@@ -148,11 +172,11 @@ module HTTPX
 
     def merge(connection)
       @origins |= connection.instance_variable_get(:@origins)
-      if @ssl_session.nil? && connection.ssl_session
-        @ssl_session = connection.ssl_session
-        @io.session_new_cb do |sess|
-          @ssl_session = sess
-        end if @io
+      if @ssl_session.nil? && (ssl_session = connection.ssl_session)
+        @ssl_session = ssl_session
+        # the socket only needs the merged session if it can still resume it,
+        # i.e. if TLS hasn't been negotiated yet.
+        @io.ssl_session = ssl_session if @io.is_a?(SSL) && !@io.connected?
       end
       connection.purge_pending do |req|
         req.transition(:idle)
@@ -161,15 +185,17 @@ module HTTPX
     end
 
     def purge_pending(&block)
-      pendings = []
       if @parser
         pending = @parser.pending
         @inflight -= pending.size
-        pendings << pending
+        pending.reject! do |req|
+          block.call(req)
+          true
+        end
       end
-      pendings << @pending
-      pendings.each do |pending|
-        pending.reject!(&block)
+      @pending.reject! do |req|
+        block.call(req)
+        true
       end
     end
 
@@ -203,7 +229,7 @@ module HTTPX
       return @parser.interests if @parser
 
       nil
-    rescue StandardError => e
+    rescue Error => e
       on_error(e)
       nil
     end
@@ -215,6 +241,8 @@ module HTTPX
     def call
       case @state
       when :idle
+        return if no_more_requests?
+
         connect
 
         # when opening the tcp or ssl socket fails
@@ -222,7 +250,7 @@ module HTTPX
 
         consume
       when :closed
-        return if @pending.empty?
+        return if no_more_requests?
 
         # there are pending requests to send, restart the state machine.
         idling
@@ -247,10 +275,14 @@ module HTTPX
         consume
       end
       nil
-    rescue IOError => e
+    rescue Errno::ECONNRESET,
+           Errno::EINVAL,
+           SocketError,
+           IOError,
+           TLSError => e
       @write_buffer.clear
       on_io_error(e)
-    rescue StandardError => e
+    rescue Error => e
       @write_buffer.clear
       on_error(e)
     rescue Exception => e # rubocop:disable Lint/RescueException
@@ -258,6 +290,10 @@ module HTTPX
       raise e
     end
 
+    def initial_call
+      call
+    end
+
     def close
       transition(:active) if @state == :inactive
 
@@ -314,6 +350,11 @@ module HTTPX
       # (example: HTTP/1 parser disabling pipelining)
       return if @state == :idle && @pending.any?
 
+      if @ping_timer
+        @ping_timer.cancel
+        @ping_timer = nil
+      end
+
       parser = @parser
 
       if parser && parser.respond_to?(:max_concurrent_requests)
@@ -324,7 +365,6 @@ module HTTPX
       end
 
       transition(:closing)
-
       transition(:closed)
     end
 
@@ -433,6 +473,11 @@ module HTTPX
 
     def on_io_error(e)
       on_error(e)
+
+      # do not force close if parser resets the connection.
+      # can happen i.e. when HTTP/1.1 pipelining is disabled.
+      return if @state == :idle && @pending.any?
+
       force_close(true)
     end
 
@@ -456,7 +501,7 @@ module HTTPX
       reset
     end
 
-    # :nocov:
+    # simplecov:disable
     def inspect
       "#<#{self.class}:#{object_id} " \
         "@origin=#{@origin} " \
@@ -464,7 +509,7 @@ module HTTPX
         "@pending=#{@pending.size} " \
         "@io=#{@io}>"
     end
-    # :nocov:
+    # simplecov:enable
 
     private
 
@@ -490,7 +535,7 @@ module HTTPX
           # * the number of pending requests
           # * the number of inflight requests
           # * whether the write buffer has bytes (i.e. for close handshake)
-          if @pending.empty? && @inflight.zero? && @write_buffer.empty?
+          if no_more_requests? && @write_buffer.empty?
             no_more_requests_loop_check if @parser && @parser.pending.any?
 
             # terminate if an altsvc connection has been established
@@ -540,7 +585,7 @@ module HTTPX
             break if @state == :closing || @state == :closed
 
             # exit #consume altogether if all outstanding requests have been dealt with
-            if @pending.empty? && @inflight.zero? && @write_buffer.empty? # rubocop:disable Style/Next
+            if no_more_requests? && @write_buffer.empty? # rubocop:disable Style/Next
               no_more_requests_loop_check if @parser && @parser.pending.any?
 
               # terminate if an altsvc connection has been established
@@ -753,14 +798,19 @@ module HTTPX
            Errno::ECONNRESET,
            Errno::EADDRNOTAVAIL,
            Errno::EHOSTUNREACH,
-           Errno::EINVAL,
            Errno::ENETUNREACH,
+           Errno::EHOSTDOWN,
+           Errno::ENETDOWN,
+           Errno::EINVAL,
            Errno::EPIPE,
            Errno::ENOENT,
            SocketError,
            IOError => e
       on_connect_error(e)
-    rescue TLSError, ::HTTP2::Error::ProtocolError, ::HTTP2::Error::HandshakeError => e
+    rescue TLSError,
+           Errno::ETIMEDOUT,
+           ::HTTP2::Error::ProtocolError,
+           ::HTTP2::Error::HandshakeError => e
       # connect errors, exit gracefully
       handle_error(e)
       handle_connect_error(e) if connecting?
@@ -957,9 +1007,12 @@ module HTTPX
 
       parser.ping
 
+      return unless parser.waiting_for_ping?
+
       ping_timeout = @options.timeout[:ping_timeout]
 
       @ping_timer = @current_selector.after(ping_timeout) do
+        log(level: 3) { "ping timeout expired..." }
         error = PingTimeoutError.new(ping_timeout, "Timed out after #{ping_timeout} seconds")
         on_error(error)
       end
@@ -986,6 +1039,11 @@ module HTTPX
                    "along with debug logs"
     end
 
+    # true when there are no more pending nor inflight (in parser) requests
+    def no_more_requests?
+      @pending.empty? && @inflight.zero?
+    end
+
     # recover internal state and emit all relevant error responses when +error+ was raised.
     # this takes an optiona +request+ which may have already been handled and can be opted out
     # in the state recovery process.
@@ -1099,13 +1157,14 @@ module HTTPX
         end
 
         timer = selector.after(timeout, callback)
-        request.active_timeouts << label
+        timer.label = label
+        request.active_timeouts << timer
 
         Array(finish_events).each do |event|
           # clean up request timeouts if the connection errors out
           request.set_timeout_callback(event) do
             timer.cancel
-            request.active_timeouts.delete(label)
+            request.active_timeouts.delete(timer)
           end
         end
       end

data/lib/httpx/errors.rb

@@ -43,6 +43,9 @@ module HTTPX
     # The HTTPX::Request request object this exception refers to.
     attr_reader :request
 
+    # The response object this exception refers to.
+    attr_reader :response
+
     # initializes the exception with the +request+ and +response+ it refers to, and the
     # +timeout+ causing the error, and the
     def initialize(request, response, timeout)

data/lib/httpx/headers.rb

@@ -140,12 +140,12 @@ module HTTPX
       @headers.to_s
     end
 
-    # :nocov:
+    # simplecov:disable
     def inspect
       "#<#{self.class}:#{object_id} " \
         "#{to_hash.inspect}>"
     end
-    # :nocov:
+    # simplecov:enable
 
     # this is internal API and doesn't abide to other public API
     # guarantees, like downcasing strings.