httpx 1.1.5 → 1.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d95b9f470645015a3308ade4ab2349375eddd0598f6919fbf063d4feba61926c
-  data.tar.gz: 95518fc5601eb0ba9a22e13814f9f0a339addca71e94c33122ed211780049ca4
+  metadata.gz: 7e5ee54988be76a44ae512da359d83c502f8a43073f244a116a8fdc45fa7b87d
+  data.tar.gz: e3c08652a8d08eadbd1ef14b20005f93ff4131a712f8234c3eab94c7fce07167
 SHA512:
-  metadata.gz: 729bfc7fc5888f6872ecf9dfdcaefa6df68ba96b705cf1ccd1e8b9b866d25bf2c3081577b3c1828b99556009fc686c0705eb30dcb6fd5f2a3fb951128e0c621e
-  data.tar.gz: 1173ae3cd242caf251c099e14db090efc3ac3524d8e3f63e04c331f2938a6d747c9874999d79dbc4790894fc8c447c900740235a62cb0376f2b124dad3674acc
+  metadata.gz: 58f16e523d23215d89a8c873a5ce12491b52726073c334c4d9800e17e40dca8111ad43ab2467c37939a19bebbaa000c8db144fdbba87cc3d22cb699687df6699
+  data.tar.gz: 64dd9bb70af4173339019b62fa9f54fdf3f22c4bf593b51fb3de624844ac8a599a51b49cae10623aee8b8cff24af4dee69e39bedfec7da3d1ae229d733632e9c

data/README.md

@@ -61,7 +61,7 @@ puts body #=> #<HTTPX::Response ...
 You can also send as many requests as you want simultaneously:
 
 ```ruby
-page1, page2, page3 =`HTTPX.get("https://news.ycombinator.com/news", "https://news.ycombinator.com/news?p=2", "https://news.ycombinator.com/news?p=3")
+page1, page2, page3 = HTTPX.get("https://news.ycombinator.com/news", "https://news.ycombinator.com/news?p=2", "https://news.ycombinator.com/news?p=3")
 ```
 
 ## Installation
@@ -107,22 +107,22 @@ HTTPX.get(
 
 ```ruby
 response = HTTPX.get("https://www.google.com", params: { q: "me" })
-response = HTTPX.post("https://www.nghttp2.org/httpbin/post", form: {name: "John", age: "22"})
+response = HTTPX.post("https://www.nghttp2.org/httpbin/post", form: { name: "John", age: "22" })
 response = HTTPX.plugin(:basic_auth)
                 .basic_auth("user", "pass")
                 .get("https://www.google.com")
 
 # more complex client objects can be cached, and are thread-safe
-http = HTTPX.plugin(:expect).with(headers: { "x-pvt-token" => "TOKEN"})
+http = HTTPX.plugin(:expect).with(headers: { "x-pvt-token" => "TOKEN" })
 http.get("https://example.com") # the above options will apply
-http.post("https://example2.com",  form: {name: "John", age: "22"}) # same, plus the form POST body
+http.post("https://example2.com", form: { name: "John", age: "22" }) # same, plus the form POST body
 ```
 
 ### Lightweight
 
 It ships with most features published as a plugin, making vanilla `httpx` lightweight and dependency-free, while allowing you to "pay for what you use"
 
-The plugin system is similar to the ones used by [sequel](https://github.com/jeremyevans/sequel), [roda](https://github.com/jeremyevans/roda) or [shrine](https://github.com/janko-m/shrine).
+The plugin system is similar to the ones used by [sequel](https://github.com/jeremyevans/sequel), [roda](https://github.com/jeremyevans/roda) or [shrine](https://github.com/shrinerb/shrine).
 
 ### Advanced DNS features
 
@@ -136,7 +136,7 @@ The test suite runs against [httpbin proxied over nghttp2](https://nghttp2.org/h
 
 All Rubies greater or equal to 2.7, and always latest JRuby and Truffleruby.
 
-**Note**: This gem is tested against all latest patch versions, i.e. if you're using 3.2.0 and you experience some issue, please test it against 3.2.$latest before creating an issue.
+**Note**: This gem is tested against all latest patch versions, i.e. if you're using 3.3.0 and you experience some issue, please test it against 3.3.$latest before creating an issue.
 
 ## Resources
 |               |                                                        |

data/doc/release_notes/1_1_1.md

@@ -2,13 +2,13 @@
 
 ## improvements
 
-* (Re-)enabling default retries in DNS name queries; this had been disabled as a result of revamping timouts, and resulted in queries only being sent once, which is very little for UDP-related traffic, and breaks if using DNs rate-limiting software. Retries the query just once, for now.
+* (Re-)enabling default retries in DNS name queries; this had been disabled as a result of revamping timeouts, and resulted in queries only being sent once, which is very little for UDP-related traffic, and breaks if using DNs rate-limiting software. Retries the query just once, for now.
 
 ## bugfixes
 
 * reset timers when adding new intervals, as these may be added as a result on after-select connection handling, and must wait for the next tick cycle (before the patch, they were triggering too soon).
 * fixed "on close" callback leak on connection reuse, which caused linear performance regression in benchmarks performing one request per connection.
-* fixed hanging connection whan an HTTP/1.1 emitted a "connection: close" header but the server would not emit one (it closes the connection now).
+* fixed hanging connection when an HTTP/1.1 emitted a "connection: close" header but the server would not emit one (it closes the connection now).
 * fixed recursive dns cached lookups which may have already expired, and created nil entries in the returned address list.
 * dns system resolver is now able to retry on failure.
 

data/doc/release_notes/1_2_0.md

@@ -0,0 +1,49 @@
+# 1.2.0
+
+## Features
+
+### `:ssrf_filter` plugin
+
+The `:ssrf_filter` plugin prevents server-side request forgery attacks, by blocking requests to the internal network. This is useful when the URLs used to perform requests aren’t under the developer control (such as when they are inserted via a web application form).
+
+```ruby
+http = HTTPX.plugin(:ssrf_filter)
+
+# this works
+response = http.get("https://example.com")
+
+# this doesn't
+response = http.get("http://localhost:3002")
+response = http.get("http://[::1]:3002")
+response = http.get("http://169.254.169.254/latest/meta-data/")
+```
+
+More info under https://honeyryderchuck.gitlab.io/httpx/wiki/SSRF-Filter
+
+### `:callbacks` plugin
+
+The session callbacks introduced in v0.24.0 are in its own plugin. Older code will still work and emit a deprecation warning.
+
+More info under https://honeyryderchuck.gitlab.io/httpx/wiki/Callbacks
+
+### `:redirect_on` option for `:follow_redirects` plugin
+
+This option allows passing a callback which, when returning `false`, can interrupt the redirect loop.
+
+```ruby
+http = HTTPX.plugin(:follow_redirects).with(redirect_on: ->(location_uri) { BLACKLIST_HOSTS.include?(location_uri.host) })
+```
+
+### `:close_on_handshake_timeout` timeout
+
+A new `:timeout` option, `:close_handshake_timeout`, is added, which monitors connection readiness when performing HTTP/2 connection termination handshake.
+
+## Improvements
+
+* Internal "eden connections" concept was removed, and connection objects are now kept-and-reused during the lifetime of a session, even when closed. This simplified connectio pool implementation and improved performance.
+* request using `:proxy` and `:retries` plugin enabled sessions will now retry on proxy connection establishment related errors.
+
+## Bugfixes
+
+* webmock adapter: mocked responses storing decoded payloads won't try to decode them again (fixes vcr/webmock integrations).
+* webmock adapter: fix issue related with making real requests over webmock-enabled connection.

data/doc/release_notes/1_2_1.md

@@ -0,0 +1,6 @@
+# 1.2.1
+
+## Bugfixes
+
+* DoH resolver: try resolving other candidates on "domain not found" error (same behaviour as with native resolver).
+* Allow HTTP/2 connections to exit cleanly when TLS session gets corrupted and termination handshake can't be performed.

data/lib/httpx/adapters/webmock.rb

@@ -41,7 +41,9 @@ module WebMock
           request.options.response_class.new(request,
                                              webmock_response.status[0],
                                              "2.0",
-                                             webmock_response.headers)
+                                             webmock_response.headers).tap do |res|
+            res.mocked = true
+          end
         end
 
         def build_error_response(request, exception)
@@ -50,16 +52,36 @@ module WebMock
       end
 
       module InstanceMethods
-        def build_connection(*)
+        def init_connection(*)
           connection = super
           connection.once(:unmock_connection) do
+            unless connection.addresses
+              connection.__send__(:callbacks)[:connect_error].clear
+              pool.__send__(:unregister_connection, connection)
+            end
             pool.__send__(:resolve_connection, connection)
-            pool.__send__(:unregister_connection, connection) unless connection.addresses
           end
           connection
         end
       end
 
+      module ResponseMethods
+        attr_accessor :mocked
+
+        def initialize(*)
+          super
+          @mocked = false
+        end
+      end
+
+      module ResponseBodyMethods
+        def decode_chunk(chunk)
+          return chunk if @response.mocked
+
+          super
+        end
+      end
+
       module ConnectionMethods
         def initialize(*)
           super

data/lib/httpx/altsvc.rb

@@ -4,6 +4,58 @@ require "strscan"
 
 module HTTPX
   module AltSvc
+    # makes connections able to accept requests destined to primary service.
+    module ConnectionMixin
+      using URIExtensions
+
+      def send(request)
+        request.headers["alt-used"] = @origin.authority if @parser && !@write_buffer.full? && match_altsvcs?(request.uri)
+
+        super
+      end
+
+      def match?(uri, options)
+        return false if !used? && (@state == :closing || @state == :closed)
+
+        match_altsvcs?(uri) && match_altsvc_options?(uri, options)
+      end
+
+      private
+
+      # checks if this is connection is an alternative service of
+      # +uri+
+      def match_altsvcs?(uri)
+        @origins.any? { |origin| altsvc_match?(uri, origin) } ||
+          AltSvc.cached_altsvc(@origin).any? do |altsvc|
+            origin = altsvc["origin"]
+            altsvc_match?(origin, uri.origin)
+          end
+      end
+
+      def match_altsvc_options?(uri, options)
+        return @options == options unless @options.ssl.all? do |k, v|
+          v == (k == :hostname ? uri.host : options.ssl[k])
+        end
+
+        @options.options_equals?(options, Options::REQUEST_BODY_IVARS + %i[@ssl])
+      end
+
+      def altsvc_match?(uri, other_uri)
+        other_uri = URI(other_uri)
+
+        uri.origin == other_uri.origin || begin
+          case uri.scheme
+          when "h2"
+            (other_uri.scheme == "https" || other_uri.scheme == "h2") &&
+              uri.host == other_uri.host &&
+              uri.port == other_uri.port
+          else
+            false
+          end
+        end
+      end
+    end
+
     @altsvc_mutex = Thread::Mutex.new
     @altsvcs = Hash.new { |h, k| h[k] = [] }
 
@@ -46,7 +98,7 @@ module HTTPX
 
       altsvc = response.headers["alt-svc"]
 
-      # https://tools.ietf.org/html/rfc7838#section-3
+      # https://datatracker.ietf.org/doc/html/rfc7838#section-3
       # A field value containing the special value "clear" indicates that the
       # origin requests all alternatives for that origin to be invalidated
       # (including those specified in the same response, in case of an
@@ -99,7 +151,10 @@ module HTTPX
     end
 
     def parse_altsvc_origin(alt_proto, alt_origin)
-      alt_scheme = parse_altsvc_scheme(alt_proto) or return
+      alt_scheme = parse_altsvc_scheme(alt_proto)
+
+      return unless alt_scheme
+
       alt_origin = alt_origin[1..-2] if alt_origin.start_with?("\"") && alt_origin.end_with?("\"")
 
       URI.parse("#{alt_scheme}://#{alt_origin}")

data/lib/httpx/buffer.rb

@@ -3,6 +3,14 @@
 require "forwardable"
 
 module HTTPX
+  # Internal class to abstract a string buffer, by wrapping a string and providing the
+  # minimum possible API and functionality required.
+  #
+  #     buffer = Buffer.new(640)
+  #     buffer.full? #=> false
+  #     buffer << "aa"
+  #     buffer.capacity #=> 638
+  #
   class Buffer
     extend Forwardable
 

data/lib/httpx/chainable.rb

@@ -1,6 +1,8 @@
 # frozen_string_literal: true
 
 module HTTPX
+  # Session mixin, implements most of the APIs that the users call.
+  # delegates to a default session when extended.
   module Chainable
     %w[head get post put delete trace options connect patch].each do |meth|
       class_eval(<<-MOD, __FILE__, __LINE__ + 1)
@@ -10,19 +12,7 @@ module HTTPX
       MOD
     end
 
-    %i[
-      connection_opened connection_closed
-      request_error
-      request_started request_body_chunk request_completed
-      response_started response_body_chunk response_completed
-    ].each do |meth|
-      class_eval(<<-MOD, __FILE__, __LINE__ + 1)
-        def on_#{meth}(&blk)   # def on_connection_opened(&blk)
-          on(:#{meth}, &blk)   #   on(:connection_opened, &blk)
-        end                    # end
-      MOD
-    end
-
+    # delegates to the default session (see HTTPX::Session#request).
     def request(*args, **options)
       branch(default_options).request(*args, **options)
     end
@@ -31,10 +21,12 @@ module HTTPX
       with(headers: { "accept" => String(type) })
     end
 
+    # delegates to the default session (see HTTPX::Session#wrap).
     def wrap(&blk)
       branch(default_options).wrap(&blk)
     end
 
+    # returns a new instance loaded with the +pl+ plugin and +options+.
     def plugin(pl, options = nil, &blk)
       klass = is_a?(S) ? self.class : Session
       klass = Class.new(klass)
@@ -42,44 +34,71 @@ module HTTPX
       klass.plugin(pl, options, &blk).new
     end
 
+    # returns a new instance loaded with +options+.
     def with(options, &blk)
       branch(default_options.merge(options), &blk)
     end
 
-    protected
-
-    def on(*args, &blk)
-      branch(default_options).on(*args, &blk)
-    end
-
     private
 
+    # returns default instance of HTTPX::Options.
     def default_options
       @options || Session.default_options
     end
 
+    # returns a default instance of HTTPX::Session.
     def branch(options, &blk)
       return self.class.new(options, &blk) if is_a?(S)
 
       Session.new(options, &blk)
     end
 
-    def method_missing(meth, *args, **options)
-      return super unless meth =~ /\Awith_(.+)/
+    def method_missing(meth, *args, **options, &blk)
+      case meth
+      when /\Awith_(.+)/
 
-      option = Regexp.last_match(1)
+        option = Regexp.last_match(1)
 
-      return super unless option
+        return super unless option
 
-      with(option.to_sym => (args.first || options))
-    end
+        with(option.to_sym => args.first || options)
+      when /\Aon_(.+)/
+        callback = Regexp.last_match(1)
 
-    def respond_to_missing?(meth, *)
-      return super unless meth =~ /\Awith_(.+)/
+        return super unless %w[
+          connection_opened connection_closed
+          request_error
+          request_started request_body_chunk request_completed
+          response_started response_body_chunk response_completed
+        ].include?(callback)
 
-      option = Regexp.last_match(1)
+        warn "DEPRECATION WARNING: calling `.#{meth}` on plain HTTPX sessions is deprecated. " \
+             "Use HTTPX.plugin(:callbacks).#{meth} instead."
 
-      default_options.respond_to?(option) || super
+        plugin(:callbacks).__send__(meth, *args, **options, &blk)
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(meth, *)
+      case meth
+      when /\Awith_(.+)/
+        option = Regexp.last_match(1)
+
+        default_options.respond_to?(option) || super
+      when /\Aon_(.+)/
+        callback = Regexp.last_match(1)
+
+        %w[
+          connection_opened connection_closed
+          request_error
+          request_started request_body_chunk request_completed
+          response_started response_body_chunk response_completed
+        ].include?(callback) || super
+      else
+        super
+      end
     end
   end
 end

data/lib/httpx/connection/http1.rb

@@ -12,6 +12,8 @@ module HTTPX
 
     attr_reader :pending, :requests
 
+    attr_accessor :max_concurrent_requests
+
     def initialize(buffer, options)
       @options = Options.new(options)
       @max_concurrent_requests = @options.max_concurrent_requests || MAX_REQUESTS
@@ -47,6 +49,7 @@ module HTTPX
       @max_requests = @options.max_requests || MAX_REQUESTS
       @parser.reset!
       @handshake_completed = false
+      @pending.concat(@requests) unless @requests.empty?
     end
 
     def close
@@ -218,6 +221,7 @@ module HTTPX
     end
 
     def ping
+      reset
       emit(:reset)
       emit(:exhausted)
     end
@@ -262,6 +266,7 @@ module HTTPX
 
     def disable
       disable_pipelining
+      reset
       emit(:reset)
       throw(:called)
     end
@@ -292,29 +297,31 @@ module HTTPX
         request.body.chunk!
       end
 
-      connection = request.headers["connection"]
+      extra_headers = {}
 
-      connection ||= if request.persistent?
-        # when in a persistent connection, the request can't be at
-        # the edge of a renegotiation
-        if @requests.index(request) + 1 < @max_requests
-          "keep-alive"
-        else
-          "close"
-        end
-      else
-        # when it's not a persistent connection, it sets "Connection: close" always
-        # on the last request of the possible batch (either allowed max requests,
-        # or if smaller, the size of the batch itself)
-        requests_limit = [@max_requests, @requests.size].min
-        if request == @requests[requests_limit - 1]
-          "close"
+      unless request.headers.key?("connection")
+        connection_value = if request.persistent?
+          # when in a persistent connection, the request can't be at
+          # the edge of a renegotiation
+          if @requests.index(request) + 1 < @max_requests
+            "keep-alive"
+          else
+            "close"
+          end
         else
-          "keep-alive"
+          # when it's not a persistent connection, it sets "Connection: close" always
+          # on the last request of the possible batch (either allowed max requests,
+          # or if smaller, the size of the batch itself)
+          requests_limit = [@max_requests, @requests.size].min
+          if request == @requests[requests_limit - 1]
+            "close"
+          else
+            "keep-alive"
+          end
         end
-      end
 
-      extra_headers = { "connection" => connection }
+        extra_headers["connection"] = connection_value
+      end
       extra_headers["host"] = request.authority unless request.headers.key?("host")
       extra_headers
     end
@@ -370,12 +377,10 @@ module HTTPX
     end
 
     def join_headers2(headers)
-      buffer = "".b
       headers.each do |field, value|
-        buffer << "#{capitalized(field)}: #{value}" << CRLF
+        buffer = "#{capitalized(field)}: #{value}#{CRLF}"
         log(color: :yellow) { "<- HEADER: #{buffer.chomp}" }
         @buffer << buffer
-        buffer.clear
       end
     end
 

data/lib/httpx/connection/http2.rb

@@ -55,7 +55,7 @@ module HTTPX
         return :w
       end
 
-      unless (@connection.state == :connected && @handshake_completed)
+      unless @connection.state == :connected && @handshake_completed
         return @buffer.empty? ? :r : :rw
       end
 
@@ -73,8 +73,11 @@ module HTTPX
     end
 
     def close
-      @connection.goaway unless @connection.state == :closed
-      emit(:close)
+      unless @connection.state == :closed
+        @connection.goaway
+        emit(:timeout, @options.timeout[:close_handshake_timeout])
+      end
+      emit(:close, true)
     end
 
     def empty?
@@ -147,6 +150,7 @@ module HTTPX
 
     def send_pending
       while (request = @pending.shift)
+        # TODO: this request should go back to top of stack
         break unless send(request)
       end
     end