httpx 1.7.8 → 1.8.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b725c719f0e33c27285f1f228dc9c9b19869d65f43d75c9e7b7f95dc8f262278
-  data.tar.gz: 92efa77ade2bae77b3d983a08038305a557953653f5e37236254b12060b0a494
+  metadata.gz: eece37bd091a7fea17ccc62eca63acaee714c08658d081a938fc38495a4c8813
+  data.tar.gz: 72dcd038803106f08a0317c8d7bc97d22d6cdf5c469ba8f8f9d380d00e8f5ea7
 SHA512:
-  metadata.gz: 2c52b06329f4d13d95242ec6043cf615e077a6dcdef6ad3cfe06d8893eb27555df4d5cdbcd95b621f9d43b4c40281becfbdb4005a074ea34463ca6e951537363
-  data.tar.gz: 4060403c67050b2d9603ec801330d0e32d24c86bf10094ce396e646ab2649870981df0eb66f042a5da1835ba349ef69024148458b6fa3e23657ae2c1a8bc7cfc
+  metadata.gz: 5af43571f7da30f709883474fb61ef3661b31fa836d249c423726f4815fb7b2bf5c1281e22ca7ad15c00ed26b2ffc950d521b14a9f5c02e7d95d4198b4b986d0
+  data.tar.gz: bd73537be9a610b590d7897cd96f7b6213ece24ea7079ce995a3e05cbee363dcf0f575405ea8d2e612a64737e3bbc70dbdc25ab9a24f5b7e154b9a5187974ffe

data/doc/release_notes/1_7_1.md

@@ -17,5 +17,4 @@
 * `:stream_bidi` plugin: fix internal state preventing bidi requests from being retried.
 * `:stream_bidi` plugin: will only allow initial request body being passed using `:body` param (others, like `:json`, will raise an exception)
 * https resolver: return `:idle` on `#state` calls when no connection is available (sometimes called in internal log messages).
-* selector loop fix: when there are no selectables and an interval is passed, sleep instead of returning (thereby avoiding potential busy loop).
-*
+* selector loop fix: when there are no selectables and an interval is passed, sleep instead of returning (thereby avoiding potential busy loop).

data/doc/release_notes/1_8_0.md

@@ -0,0 +1,100 @@
+# 1.8.0
+
+## Features
+
+### New plugins
+
+#### `:server_sent_events` plugin
+
+The `:server_sent_events` plugin provides a convenience API to deal with `text/event-stream` requests, on top of the `:stream` plugin.
+
+```ruby
+session = HTTPX.plugin(:server_sent_events)
+
+sse_response = session.get("https://example.com/event-stream", event_stream: true)
+
+sse_response.each_message do |message|
+  puts message.id
+  puts message.event
+  puts message.data
+end
+```
+
+You can read more about it in https://gitlab.com/os85/httpx/wikis/Server-Sent-Events .
+
+#### `:cache` plugin
+
+The `:cache` plugin allows caching responses. It exposes some options to determine some of its functionality, i.e. whether a request can use a cached response, whether a response can be cached, whether a cached response is still valid, etc.
+
+This functionality was extracted from the `:response_cache` plugin, which now uses it under the hood.
+
+You can read more about it in https://gitlab.com/os85/httpx/wikis/Cache .
+
+#### `:ntlm_v2_auth` plugin
+
+The `:ntlm_v2_auth` plugin is now available. It implements the most recent version of the NTLM authentication scheme supported by Microsoft products.
+
+You can read more about it in https://gitlab.com/os85/httpx/wikis/Auth#ntlm-v2-auth .
+
+### New timeouts
+
+#### `:total_request_timeout`
+
+You can use the `:total_request_timeout` to time the time it takes a request to get its final response. This includes when your requests follows redirects (via the `:follow_redirects` plugin) or is retried multiple times (via the `:retries´ plugin).
+
+#### `:ping_timeout`
+
+Defines the number of seconds a connection waits to receive a ping response when probing for a connection for liveness.
+
+Defaults to 2 seconds.
+
+### `:ssrf_filter` plugin new options
+
+#### `:extra_unsafe_ranges`
+
+A list of extra unsafe IPs or IP ranges to the default deny list.
+
+#### `:safe_private_ranges`
+
+A list of IPs or IP ranges which are allowed and would otherwise be denied.
+
+### `:auth` plugin new option
+
+#### `:reset_auth_header_expires_in/at`
+
+The `:reset_auth_header_expires_in` and `:reset_auth_header_expires_at`  options enable discarding an authorization token an X number of seconds after it has been generated, or at a particuar point in time, respectively. This is useful when the token is dynamically generated, so that you can preemptively renegotiate a new one.
+
+The `:oauth` plugin makes use of these fields, alongside the `expires_in` claim from token responses, to refresh the token as soon as the it expires.
+
+### `:max_response_body_size`
+
+Can be set to the maximum number of bytes a response may have, after which it'll return an `HTTPX::ErrorResponse`. The limit is enforced based on the `content-length` header and as bytes are received.
+
+### `:max_response_headers`
+
+Can be set to the maximum number of headers a response may have.
+
+### `:max_response_header_value_size`
+
+Can be set to the maximum number of bytes a header value may have. In cases where a header field may be spread across multiple entries (ex. `"cookie"`), the limit is enforced on the aggregate byte size.
+
+### resolver file cache
+
+By specifying `:file` as the resolver `:cache` option, the DNS entries will be cached in a known location in your file system. This will allow sharing entries across processes within the same machine to reduce overall DNS traffic. You can use it via:
+
+```ruby
+session = HTTPX.with(resolver_options: { cache: :file })
+session.get("https://example.com")
+```
+
+## Improvements
+
+* `http-2` minimum version is now 1.2.0, which brings performance benefits around frame parsing and other common operations.
+
+## Bugfixes
+
+* several fixes to make `httpx` usable inside a fiber scheduler in ruby 4.
+* `:proxy` plugin: unescape user/password from proxy options before reusing it (to avoid using percent-encoded values when p.ex. generating base64-encoding for basic auth).
+* `:retries` plugin: fixed the polynomial and exponential backoff `:retry_after` strategies calculation.
+* `:tracing` plugin: fixing span start time set up when request is sent to a closed connection, or when a long-lived connection will be probed for liveness.
+* datadog adapter: fixed integration with the `datadog` gem v2.34 or higher.

data/doc/release_notes/1_8_1.md

@@ -0,0 +1,27 @@
+# 1.8.1
+
+## Improvements
+
+* the session will now eagerly write initial data to connections instead of waiting for I/O the first time (saves a syscall).
+* removed all rescue blocks rescuing `StandardError` internally; this could interfere with usages of `httpx` inside `Timeout.timeout` blocks, which may emit an exception at any point, including in blocks where that exception could be rescued, leaving the session in a potentially irrecoverable state and looping.
+* debug logs change the display of ids from 'tid:123, fid:123' to 'tid=123 fid:123'.
+  * and when using the `datadog` adapter, all related correlation ids (trace id, span id, etc) will also be added to the identifiers in the same format
+* plugins built on top of the `:retries` plugin will skip the `retry_after` interval when certain conditions are met.
+  * `:persistent` plugin: request will be retried immediately, when the error happens after the keep alive interval expired.
+  * `:auth` plugin: request will be retried immediately if it failed due to a "401 Unauthorized Error".
+* `:ssrf_filter` plugin: removed public/private references of exception message.
+
+## Bugfixes
+
+* `:auth` plugin: handle edge case where `:auth_header_expires_at` returns no value.
+* `:digest_auth` plugin: do not retry if there was an issue building building the authorization header after the probe.
+* `webmock` adapter: prevent mocked requests from opening a connection to the peer server that will not be used.
+* clear and cancel request timers on completion or reset, ensuring that they won't be retained way past the point where the request is gone.
+* connection: cancel and discard ping timer before closing (removes timer reference)
+* connection: fix logic where purging-while-resending requests to a coalesced connection was leaving references in the old connection.
+* moved an `on_body_chunk` callback to the `:callbacks` plugin, where it's used and expected to be defined.
+* `http2`: avoid closing the connection on internal errors which are supposed to be specific to a request, such as the max headers or max response body size checks.
+* system resolver will use an internal exception when using `Timeout.timeout` on resolutions to avoid conflicts with the default timeout exception class.
+* prevent edge case where `#alpn_protocol` was being called on a `TCPSocket`.
+* fixed v1.8.0 regression causing connection to loop when an error happens while connecting.
+* resolver cache: fixed cache eviction logic, where alias answers weren't being evicted, causing resolutions to stall after a certain number of resolutions.

data/lib/httpx/adapters/datadog.rb

@@ -15,8 +15,10 @@ module Datadog::Tracing
 
       TAG_BASE_SERVICE = if Gem::Version.new(DATADOG_VERSION::STRING) < Gem::Version.new("1.15.0")
         "_dd.base_service"
-      else
+      elsif Gem::Version.new(DATADOG_VERSION::STRING) < Gem::Version.new("2.34.0")
         Datadog::Tracing::Contrib::Ext::Metadata::TAG_BASE_SERVICE
+      else
+        Datadog::Tracing::Metadata::Ext::TAG_BASE_SERVICE
       end
       TAG_PEER_HOSTNAME = Datadog::Tracing::Metadata::Ext::TAG_PEER_HOSTNAME
       TAG_PEER_SERVICE = Datadog::Tracing::Metadata::Ext::TAG_PEER_SERVICE
@@ -183,6 +185,13 @@ module Datadog::Tracing
         end
       end
 
+      # patches httpx debug logs to include datadog correlation ids
+      module LoggablePatch
+        def log_identifiers
+          "#{super} #{Datadog::Tracing.log_correlation}"
+        end
+      end
+
       module Configuration
         # Default settings for httpx
         #
@@ -295,6 +304,8 @@ module Datadog::Tracing
 
           ::HTTPX.send(:remove_const, :Session)
           ::HTTPX.send(:const_set, :Session, datadog_session.class)
+
+          ::HTTPX::Loggable.singleton_class.prepend(LoggablePatch)
         end
       end
 

data/lib/httpx/adapters/faraday.rb

@@ -115,11 +115,11 @@ module Faraday
             ssl_options
           end
         else
-          # :nocov:
+          # simplecov:disable
           def ssl_options_from_env(*)
             {}
           end
-          # :nocov:
+          # simplecov:enable
         end
       end
 

data/lib/httpx/adapters/webmock.rb

@@ -155,6 +155,12 @@ module WebMock
             raise WebMock::NetConnectNotAllowedError, request_signature
           end
         end
+
+        private
+
+        def connect
+          super unless @mocked
+        end
       end
     end
 

data/lib/httpx/connection/http1.rb

@@ -14,6 +14,7 @@ module HTTPX
       "www-authenticate" => "WWW-Authenticate",
       "http2-settings" => "HTTP2-Settings",
       "content-md5" => "Content-MD5",
+      "last-event-id" => "Last-Event-ID",
     }.freeze
     attr_reader :pending, :requests
 
@@ -23,7 +24,7 @@ module HTTPX
       @options = options
       @max_concurrent_requests = @options.max_concurrent_requests || MAX_REQUESTS
       @max_requests = @options.max_requests
-      @parser = Parser::HTTP1.new(self)
+      @parser = Parser::HTTP1.new(self, options.max_response_headers, options.max_response_header_value_size)
       @buffer = buffer
       @version = [1, 1]
       @pending = []
@@ -47,6 +48,10 @@ module HTTPX
     end
 
     def reset
+      if @ping_timer
+        @ping_timer.cancel
+        @ping_timer = nil
+      end
       @max_requests = @options.max_requests || MAX_REQUESTS
       @parser.reset!
       @handshake_completed = false
@@ -132,6 +137,10 @@ module HTTPX
       request.log(color: :yellow) { "-> HEADLINE: #{response.status} HTTP/#{@parser.http_version.join(".")}" }
       request.log(color: :yellow) { response.headers.each.map { |f, v| "-> HEADER: #{f}: #{log_redact_headers(v)}" }.join("\n") }
 
+      if response.content_length && response.content_length > request.options.max_response_body_size
+        raise HTTPX::Error, "maximum response body size exceeded"
+      end
+
       request.response = response
       on_complete if response.finished?
     end
@@ -153,18 +162,12 @@ module HTTPX
 
       return unless request
 
-      begin
-        request.log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
-        request.log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
+      request.log(color: :green) { "-> DATA: #{chunk.bytesize} bytes..." }
+      request.log(level: 2, color: :green) { "-> #{log_redact_body(chunk.inspect)}" }
 
-        response = request.response
+      response = request.response
 
-        response << chunk
-      rescue StandardError => e
-        error_response = ErrorResponse.new(request, e)
-        request.response = error_response
-        dispatch(request)
-      end
+      response << chunk
     end
 
     def on_complete
@@ -407,7 +410,7 @@ module HTTPX
     end
 
     def capitalized(field)
-      UPCASED[field] || field.split("-").map(&:capitalize).join("-")
+      UPCASED.fetch(field) { field.split("-").map(&:capitalize).join("-") }
     end
   end
 end

data/lib/httpx/connection/http2.rb

@@ -37,6 +37,7 @@ module HTTPX
       @streams = {}
       @drains = {}
       @pings = []
+      @streams_to_close_after_receive = []
       @buffer = buffer
       @handshake_completed = false
       @wait_for_handshake = @settings.key?(:wait_for_handshake) ? @settings.delete(:wait_for_handshake) : true
@@ -110,6 +111,12 @@ module HTTPX
 
     def <<(data)
       @connection << data
+
+      while (stream, request, error = @streams_to_close_after_receive.shift)
+        # these streams were marked for cancellation due to errors found while processing the
+        # data received by the peer.
+        emit_stream_error(stream, request, error)
+      end
     end
 
     def send(request, head = false)
@@ -118,7 +125,7 @@ module HTTPX
         return false
       end
       unless (stream = @streams[request])
-        stream = @connection.new_stream
+        stream = @connection.new_stream(**request.http2_stream_options)
         handle_stream(stream, request)
         @streams[request] = stream
         @max_requests -= 1
@@ -128,6 +135,8 @@ module HTTPX
     rescue ::HTTP2::Error::StreamLimitExceeded
       @pending.unshift(request)
       false
+    rescue ::HTTP2::Error::Error, ArgumentError => e
+      emit(:error, request, e)
     end
 
     def consume
@@ -312,11 +321,27 @@ module HTTPX
       end
       _, status = h.shift
       headers = request.options.headers_class.new(h)
+
+      raise HTTPX::Error, "maximum number of response headers exceeded" if h.size > @options.max_response_headers
+
+      if (max_header_value_size = @options.max_response_header_value_size)
+        headers.each do |_, v| # rubocop:disable Style/HashEachMethods
+          raise HTTPX::Error, "maximum header value size exceeded" if v.size > max_header_value_size
+        end
+      end
+
       response = request.options.response_class.new(request, status, "2.0", headers)
+
+      if response.content_length && response.content_length > request.options.max_response_body_size
+        raise HTTPX::Error.new, "maximum response body size exceeded"
+      end
+
       request.response = response
       @streams[request] = stream
 
       handle(request, stream) if request.expects?
+    rescue HTTPX::Error => e
+      @streams_to_close_after_receive << [stream, request, e]
     end
 
     def on_stream_trailers(stream, request, response, h)
@@ -329,7 +354,18 @@ module HTTPX
     def on_stream_data(stream, request, data)
       request.log(level: 1, color: :green) { "#{stream.id}: <- DATA: #{data.bytesize} bytes..." }
       request.log(level: 2, color: :green) { "#{stream.id}: <- #{log_redact_body(data.inspect)}" }
+
+      return unless request.response
+
       request.response << data
+    rescue HTTPX::Error => e
+      if stream.state == :closing
+        # there won't be any more chunks to process after this one.
+        emit_stream_error(stream, request, e)
+      else
+        # defer until the last chunk from the payload is processed.
+        @streams_to_close_after_receive << [stream, request, e]
+      end
     end
 
     def on_stream_refuse(stream, request, error)
@@ -365,11 +401,12 @@ module HTTPX
           emit(:response, request, response)
         end
       else
-        response = request.response
-        if response && response.is_a?(Response) && response.status == 421
-          emit(:error, request, :http_1_1_required)
-        else
-          emit(:response, request, response)
+        if (response = request.response)
+          if response.is_a?(Response) && response.status == 421
+            emit(:error, request, :http_1_1_required)
+          else
+            emit(:response, request, response)
+          end
         end
       end
       send(@pending.shift) unless @pending.empty?
@@ -428,11 +465,28 @@ module HTTPX
     end
 
     def frame_with_extra_info(frame)
+      flags_bits = frame.fetch(:flags, 0)
       case frame[:type]
       when :data
-        frame.merge(payload: frame[:payload].bytesize)
-      when :headers, :ping
-        frame.merge(payload: log_redact_headers(frame[:payload]))
+        flags = [] #: Array[Symbol]
+        flags << :end_stream if flags_bits.anybits?(0b0001)
+        flags << :padded if flags_bits.anybits?(0b1000)
+        frame.merge(payload: frame[:payload].bytesize, flags: flags)
+      when :push_promise, :headers
+        flags = [] #: Array[Symbol]
+        flags << :end_stream if flags_bits.anybits?(0b0001)
+        flags << :priority if flags_bits.anybits?(0b0010)
+        flags << :end_headers if flags_bits.anybits?(0b0100)
+        flags << :padded if flags_bits.anybits?(0b1000)
+        frame.merge(payload: log_redact_headers(frame[:payload]), flags: flags)
+      when :ping
+        flags = [] #: Array[Symbol]
+        flags << :ack if flags_bits.anybits?(0b0001)
+        frame.merge(payload: log_redact_headers(frame[:payload]), flags: flags)
+      when :settings
+        flags = [] #: Array[Symbol]
+        flags << :ack if flags_bits.anybits?(0b0001)
+        frame.merge(flags: flags)
       when :window_update
         connection_or_stream = if (id = frame[:stream]).zero?
           @connection
@@ -476,6 +530,12 @@ module HTTPX
       emit(:pong)
     end
 
+    def emit_stream_error(stream, request, error)
+      teardown(request)
+      stream.close
+      emit(:error, request, error)
+    end
+
     def teardown(request = nil)
       if request
         @drains.delete(request)