httpx 0.18.6 → 0.19.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b6784fbd393b5caed888e6c4d1a421cde9c7aca49f35baaafbf13f10a68e9d02
-  data.tar.gz: ddd5bb8793fa72b19e462b411f5eaaaf553f30d589a68dfdde64224c49d902f6
+  metadata.gz: b9e79c1b196931855e7af542bd611050a95648d2888d539cb20de5026c4a3ccf
+  data.tar.gz: 4cc919ede2c58d6bcf86433ddf4ceba9fdca5416186d5984fa2f29a92707668d
 SHA512:
-  metadata.gz: 949f18741f10eaa57463b8764569e69cdfb1ca6f4eefa30bbdcf142cbbd8b61b360d9bd78898a816a4ebd9e15d4ffd1c0d8282033a459826919ce8132d5b105f
-  data.tar.gz: e4430eea967c9f5628d51e6150bc4f63e704eaed2bd61fda191f3539a3f102e00932cc8f63ddb735c33bbf21e2694bb23fed5eaea588159e71fa6f484016c150
+  metadata.gz: 5a35e9f740d6df65190a8e28cc2d2d8adfe9c208842290ceb0c8f6c9dd273e19066c7cc851d917f6daac663c205401e7df26916a0f41c01653e53488d404f669
+  data.tar.gz: 67203d583847abee46ac940ff82c11624298907e191ff0c6c5408b1cddf0c909ac8820de24f11ef1cc9ac7030e5d11e807668e9f991967fc4bb4654f92ddd3c8

data/README.md

@@ -140,11 +140,15 @@ In order to use HTTP/2 under JRuby, [check this link](https://gitlab.com/honeyry
 * Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryderchuck/httpx/issues/36)).
 * Using `total_timeout` along with the `:persistent` plugin [does not work as you might expect](https://gitlab.com/honeyryderchuck/httpx/-/wikis/Timeouts#total_timeout).
 
+## Versioning Policy
+
+Although 0.x software, `httpx` is considered API-stable and production-ready, i.e. current API or options may be subject to deprecation and emit log warnings, but can only effectively be removed in a major version change.
+
 ## Contributing
 
 * Discuss your contribution in an issue
 * Fork it
 * Make your changes, add some tests
-* Ensure all tests pass (`bundle exec rake test`)
+* Ensure all tests pass (`docker-compose -f docker-compose.yml -f docker-compose-ruby-{RUBY_VERSION}.yml run httpx bundle exec rake test`)
 * Open a Merge Request (that's Pull Request in Github-ish)
 * Wait for feedback

data/doc/release_notes/0_18_5.md

@@ -3,8 +3,8 @@
 ## Improvements
 
 * ruby 3.1 is now officially supported.
-* when user sets a `Host` header for an HTTP/2 request, this will be used in the `:authority` HTTP/2 pseudo-header, instead of silently ignored (mimicking what "curl" does).
+* when a user sets a `Host` header for an HTTP/2 request, this will be used in the `:authority` HTTP/2 pseudo-header, instead of silently ignored (mimicking what "curl" does).
 
 ## Bugfixes
 
-* fixed "throw outside of catch block" error happening when pipelining requests on an HTTP/1 connnection and resulting in a timeout.
+* fixed "throw outside of catch block" error happening when pipelining requests on an HTTP/1 connection and resulting in a timeout.

data/doc/release_notes/0_18_7.md

@@ -0,0 +1,5 @@
+# 0.18.6
+
+## Bugfixes
+
+* multipart plugin: fixed `filemagic` integration by rewinding the file after mime-type detection.

data/doc/release_notes/0_19_0.md

@@ -0,0 +1,39 @@
+# 0.19.0
+
+## Features
+
+### Happy Eyeballs v2
+
+When the system supports dual-stack networking, `httpx` implements the Happy Eyeballs v2 algorithm (RFC 8305) to resolve hostnames to both IPv6 and IPv4 addresses while privileging IPv6 connectivity. This is implemented by `httpx` both for the `:native` as well as the `:https` (DoH) resolver (which do not perform address sorting, thereby being "DNS-based load-balancing" friendly), and "outsourced" to `getaddrinfo` when using the `:system` resolver.
+
+IPv6 connectivity will also be privileged for `/etc/hosts` local DNS (i.e. `localhost` connections will connec to `::1`).
+
+A new option, `:ip_families`, will also be available (`[Socket::AF_INET6, Socket::AF_INET]` in dual-stack systems). If you'd like to i.e. force IPv4 connectivity, you can do use it (`client = HTTPX.with(ip_families: [Socket::AF_INET])`).
+
+## Improvements
+
+### DNS: :system resolver uses getaddrinfo (instead of the resolver lib)
+
+The `:system` resolver switched to using the `getaddinfo` system function to perform DNS requests. Not only is this call **not** blocking the session event loop anymore (unlike pre-0.19.0 `:system` resolver), it adds a lot of functionality that the stdlib `resolv` library just doesn't support at the moment (such as SRV records).
+
+### HTTP/2 proxy support
+
+The `:proxy` plugin handles "prior-knowledge" HTTP/2 proxies.
+
+```ruby
+HTTPX.plugin(:proxy, fallback_protocol: "h2").with_proxy(uri: "http://http2-proxy:3128").get(...
+```
+
+Connection coalescing has also been enabled for proxied connections (also `CONNECT`-tunneled connections).
+
+### curl-to-httpx
+
+widget in [project website](https://honeyryderchuck.gitlab.io/httpx/) to turn curl commands into the equivalent `httpx` code.
+
+## Bugfixes
+
+* faraday adapter now supports passing session options.
+* proxy: several fixes which enabled env-var (`HTTP(S)_PROXY`) defined proxy support.
+* proxy: fixed graceful recovery from proxy tcp connect errors.
+* several fixes around CNAMEs timeouts with the native resolver.
+* https resolver is now closed when wrapping session closes (it was left open).

data/doc/release_notes/0_19_1.md

@@ -0,0 +1,5 @@
+# 0.19.1
+
+## Bugfixes
+
+Fixing a DNS dual-stack case where one the resolvers may have finished way before the previous one and will therefore return no timeout.

data/doc/release_notes/0_19_2.md

@@ -0,0 +1,7 @@
+# 0.19.2
+
+## Bugfixes
+
+* skip resolution delay path for early resolve cases
+
+when the early resolve path (using IP, /etc/hosts IP, IP from cache) is followed, emit_addresses is called, and in a particular case (dual-stack network but using an IPv4 address), the happy eyeballs resolution delay path was activated when it shouldn't (it's meant to be used only for DNS network requests), and resulted in @pool being called before it was ever set. This simple check ensures that it doesn't happen before it must.

data/lib/httpx/adapters/faraday.rb

@@ -81,6 +81,9 @@ module Faraday
 
           def response=(response)
             super
+
+            return if response.is_a?(::HTTPX::ErrorResponse)
+
             response.body.on_data = @response_on_data
           end
         end
@@ -136,7 +139,7 @@ module Faraday
 
           def on_response(&blk)
             if blk
-              @on_response = lambda do |response|
+              @on_response = ->(response) do
                 blk.call(response)
               end
               self
@@ -200,9 +203,9 @@ module Faraday
         end
       end
 
-      def initialize(app)
+      def initialize(app, options = {})
         super(app)
-        @session = Session.new
+        @session = Session.new(options)
       end
 
       def call(env)
@@ -210,6 +213,7 @@ module Faraday
         if parallel?(env)
           handler = env[:parallel_manager].enqueue(env)
           handler.on_response do |response|
+            response.raise_for_status
             save_response(env, response.status, response.body.to_s, response.headers, response.reason) do |response_headers|
               response_headers.merge!(response.headers)
             end

data/lib/httpx/connection/http1.rb

@@ -297,10 +297,6 @@ module HTTPX
       extra_headers
     end
 
-    def headline_uri(request)
-      request.path
-    end
-
     def handle(request)
       catch(:buffer_full) do
         request.transition(:headers)
@@ -314,8 +310,12 @@ module HTTPX
       end
     end
 
+    def join_headline(request)
+      "#{request.verb.to_s.upcase} #{request.path} HTTP/#{@version.join(".")}"
+    end
+
     def join_headers(request)
-      headline = "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}"
+      headline = join_headline(request)
       @buffer << headline << CRLF
       log(color: :yellow) { "<- HEADLINE: #{headline.chomp.inspect}" }
       extra_headers = set_protocol_headers(request)

data/lib/httpx/connection/http2.rb

@@ -158,10 +158,6 @@ module HTTPX
       end
     end
 
-    def headline_uri(request)
-      request.path
-    end
-
     def handle(request, stream)
       catch(:buffer_full) do
         request.transition(:headers)
@@ -213,7 +209,7 @@ module HTTPX
       {
         ":scheme" => request.scheme,
         ":method" => request.verb.to_s.upcase,
-        ":path" => headline_uri(request),
+        ":path" => request.path,
         ":authority" => request.authority,
       }
     end

data/lib/httpx/connection.rb

@@ -44,7 +44,7 @@ module HTTPX
 
     def_delegator :@write_buffer, :empty?
 
-    attr_reader :origin, :state, :pending, :options
+    attr_reader :io, :origin, :origins, :state, :pending, :options
 
     attr_writer :timers
 
@@ -78,7 +78,11 @@ module HTTPX
     # this is a semi-private method, to be used by the resolver
     # to initiate the io object.
     def addresses=(addrs)
-      @io ||= IO.registry(@type).new(@origin, addrs, @options) # rubocop:disable Naming/MemoizedInstanceVariableName
+      if @io
+        @io.add_addresses(addrs)
+      else
+        @io = IO.registry(@type).new(@origin, addrs, @options)
+      end
     end
 
     def addresses
@@ -489,6 +493,18 @@ module HTTPX
     end
 
     def transition(nextstate)
+      handle_transition(nextstate)
+    rescue Errno::ECONNREFUSED,
+           Errno::EADDRNOTAVAIL,
+           Errno::EHOSTUNREACH,
+           TLSError => e
+      # connect errors, exit gracefully
+      handle_error(e)
+      @state = :closed
+      emit(:close)
+    end
+
+    def handle_transition(nextstate)
       case nextstate
       when :idle
         @timeout = @current_timeout = @options.timeout[:connect_timeout]
@@ -525,14 +541,6 @@ module HTTPX
         emit(:activate)
       end
       @state = nextstate
-    rescue Errno::ECONNREFUSED,
-           Errno::EADDRNOTAVAIL,
-           Errno::EHOSTUNREACH,
-           TLSError => e
-      # connect errors, exit gracefully
-      handle_error(e)
-      @state = :closed
-      emit(:close)
     end
 
     def purge_after_closed
@@ -550,6 +558,10 @@ module HTTPX
           ex.set_backtrace(error.backtrace)
           error = ex
         else
+          # inactive connections do not contribute to the select loop, therefore
+          # they should fail due to such errors.
+          return if @state == :inactive
+
           if @timeout
             @timeout -= error.timeout
             return unless @timeout <= 0

data/lib/httpx/extensions.rb

@@ -54,6 +54,22 @@ module HTTPX
     Numeric.__send__(:include, NegMethods)
   end
 
+  module StringExtensions
+    refine String do
+      def delete_suffix!(suffix)
+        suffix = Backports.coerce_to_str(suffix)
+        chomp! if frozen?
+        len = suffix.length
+        if len > 0 && index(suffix, -len)
+          self[-len..-1] = ''
+          self
+        else
+          nil
+        end
+      end unless String.method_defined?(:delete_suffix!)
+    end
+  end
+
   module HashExtensions
     refine Hash do
       def compact

data/lib/httpx/headers.rb

@@ -2,8 +2,6 @@
 
 module HTTPX
   class Headers
-    EMPTY = [].freeze
-
     class << self
       def new(headers = nil)
         return headers if headers.is_a?(self)

data/lib/httpx/io/tcp.rb

@@ -15,6 +15,7 @@ module HTTPX
 
     def initialize(origin, addresses, options)
       @state = :idle
+      @addresses = []
       @hostname = origin.host
       @options = Options.new(options)
       @fallback_protocol = @options.fallback_protocol
@@ -30,15 +31,29 @@ module HTTPX
         raise Error, "Given IO objects do not match the request authority" unless @io
 
         _, _, _, @ip = @io.addr
-        @addresses ||= [@ip]
-        @ip_index = @addresses.size - 1
+        @addresses << @ip
         @keep_open = true
         @state = :connected
       else
-        @addresses = addresses.map { |addr| addr.is_a?(IPAddr) ? addr : IPAddr.new(addr) }
+        add_addresses(addresses)
       end
       @ip_index = @addresses.size - 1
-      @io ||= build_socket
+      # @io ||= build_socket
+    end
+
+    def add_addresses(addrs)
+      return if addrs.empty?
+
+      addrs = addrs.map { |addr| addr.is_a?(IPAddr) ? addr : IPAddr.new(addr) }
+
+      ip_index = @ip_index || (@addresses.size - 1)
+      if addrs.first.ipv6?
+        # should be the next in line
+        @addresses = [*@addresses[0, ip_index], *addrs, *@addresses[ip_index..-1]]
+      else
+        @addresses.unshift(*addrs)
+        @ip_index += addrs.size if @ip_index
+      end
     end
 
     def to_io
@@ -52,20 +67,26 @@ module HTTPX
     def connect
       return unless closed?
 
-      if @io.closed?
+      if !@io || @io.closed?
         transition(:idle)
         @io = build_socket
       end
       try_connect
-    rescue Errno::EHOSTUNREACH => e
+    rescue Errno::ECONNREFUSED,
+           Errno::EADDRNOTAVAIL,
+           Errno::EHOSTUNREACH => e
       raise e if @ip_index <= 0
 
+      log { "failed connecting to #{@ip} (#{e.message}), trying next..." }
       @ip_index -= 1
+      @io = build_socket
       retry
     rescue Errno::ETIMEDOUT => e
       raise ConnectTimeoutError.new(@options.timeout[:connect_timeout], e.message) if @ip_index <= 0
 
+      log { "failed connecting to #{@ip} (#{e.message}), trying next..." }
       @ip_index -= 1
+      @io = build_socket
       retry
     end
 

data/lib/httpx/options.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "socket"
+
 module HTTPX
   class Options
     WINDOW_SIZE = 1 << 14 # 16K
@@ -9,6 +11,18 @@ module HTTPX
     KEEP_ALIVE_TIMEOUT = 20
     SETTINGS_TIMEOUT = 10
 
+    # https://github.com/ruby/resolv/blob/095f1c003f6073730500f02acbdbc55f83d70987/lib/resolv.rb#L408
+    ip_address_families = begin
+      list = Socket.ip_address_list
+      if list.any? { |a| a.ipv6? && !a.ipv6_loopback? && !a.ipv6_linklocal? }
+        [Socket::AF_INET6, Socket::AF_INET]
+      else
+        [Socket::AF_INET]
+      end
+    rescue NotImplementedError
+      [Socket::AF_INET]
+    end
+
     DEFAULT_OPTIONS = {
       :debug => ENV.key?("HTTPX_DEBUG") ? $stderr : nil,
       :debug_level => (ENV["HTTPX_DEBUG"] || 1).to_i,
@@ -37,6 +51,7 @@ module HTTPX
       :persistent => false,
       :resolver_class => (ENV["HTTPX_RESOLVER"] || :native).to_sym,
       :resolver_options => { cache: true },
+      :ip_families => ip_address_families,
     }.freeze
 
     begin
@@ -110,20 +125,18 @@ module HTTPX
     end
 
     def initialize(options = {})
-      defaults = DEFAULT_OPTIONS.merge(options)
-      defaults.each do |k, v|
-        next if v.nil?
-
-        begin
-          value = __send__(:"option_#{k}", v)
-          instance_variable_set(:"@#{k}", value)
-        rescue NoMethodError
-          raise Error, "unknown option: #{k}"
-        end
-      end
+      __initialize__(options)
       freeze
     end
 
+    def freeze
+      super
+      @origin.freeze
+      @timeout.freeze
+      @headers.freeze
+      @addresses.freeze
+    end
+
     def option_origin(value)
       URI(value)
     end
@@ -174,6 +187,10 @@ module HTTPX
       Array(value)
     end
 
+    def option_ip_families(value)
+      Array(value)
+    end
+
     %i[
       params form json body ssl http2_settings
       request_class response_class headers_class request_body_class
@@ -249,5 +266,21 @@ module HTTPX
         end
       end
     end
+
+    private
+
+    def __initialize__(options = {})
+      defaults = DEFAULT_OPTIONS.merge(options)
+      defaults.each do |k, v|
+        next if v.nil?
+
+        begin
+          value = __send__(:"option_#{k}", v)
+          instance_variable_set(:"@#{k}", value)
+        rescue NoMethodError
+          raise Error, "unknown option: #{k}"
+        end
+      end
+    end
   end
 end

data/lib/httpx/plugins/cookies.rb

@@ -18,12 +18,6 @@ module HTTPX
         require "httpx/plugins/cookies/set_cookie_parser"
       end
 
-      module OptionsMethods
-        def option_cookies(value)
-          value.is_a?(Jar) ? value : Jar.new(value)
-        end
-      end
-
       module InstanceMethods
         extend Forwardable
 
@@ -77,7 +71,7 @@ module HTTPX
       end
 
       module OptionsMethods
-        def initialize(*)
+        def __initialize__(*)
           super
 
           return unless @headers.key?("cookie")
@@ -89,6 +83,10 @@ module HTTPX
             end
           end
         end
+
+        def option_cookies(value)
+          value.is_a?(Jar) ? value : Jar.new(value)
+        end
       end
     end
     register_plugin :cookies, Cookies

data/lib/httpx/plugins/internal_telemetry.rb

@@ -81,7 +81,7 @@ module HTTPX
           super
         end
 
-        def transition(nextstate)
+        def handle_transition(nextstate)
           state = @state
           super
           meter_elapsed_time("Connection##{object_id}[#{@origin}]: #{state} -> #{nextstate}") if nextstate == @state

data/lib/httpx/plugins/multipart/mime_type_detector.rb

@@ -14,9 +14,13 @@ module HTTPX
         def call(file, _)
           return nil if file.eof? # FileMagic returns "application/x-empty" for empty files
 
-          FileMagic.open(FileMagic::MAGIC_MIME_TYPE) do |filemagic|
+          mime = FileMagic.open(FileMagic::MAGIC_MIME_TYPE) do |filemagic|
             filemagic.buffer(file.read(MAGIC_NUMBER))
           end
+
+          file.rewind
+
+          mime
         end
       elsif defined?(Marcel)
         def call(file, filename)

data/lib/httpx/plugins/proxy/http.rb

@@ -13,7 +13,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && @options.proxy.uri.scheme == "http"
 
             case nextstate
@@ -23,7 +23,8 @@ module HTTPX
               @io.connect
               return unless @io.connected?
 
-              @parser = ConnectProxyParser.new(@write_buffer, @options.merge(max_concurrent_requests: 1))
+              @parser = registry(@io.protocol).new(@write_buffer, @options.merge(max_concurrent_requests: 1))
+              @parser.extend(ProxyParser)
               @parser.once(:response, &method(:__http_on_connect))
               @parser.on(:close) { transition(:closing) }
               __http_proxy_connect
@@ -36,7 +37,7 @@ module HTTPX
                 @parser.close
                 @parser = nil
               when :idle
-                @parser = ProxyParser.new(@write_buffer, @options)
+                @parser.callbacks.clear
                 set_parser_callbacks(@parser)
               end
             end
@@ -54,7 +55,7 @@ module HTTPX
               @inflight += 1
               parser.send(connect_request)
             else
-              transition(:connected)
+              handle_transition(:connected)
             end
           end
 
@@ -76,9 +77,11 @@ module HTTPX
           end
         end
 
-        class ProxyParser < Connection::HTTP1
-          def headline_uri(request)
-            request.uri.to_s
+        module ProxyParser
+          def join_headline(request)
+            return super if request.verb == :connect
+
+            "#{request.verb.to_s.upcase} #{request.uri} HTTP/#{@version.join(".")}"
           end
 
           def set_protocol_headers(request)
@@ -91,22 +94,6 @@ module HTTPX
           end
         end
 
-        class ConnectProxyParser < ProxyParser
-          attr_reader :pending
-
-          def headline_uri(request)
-            return super unless request.verb == :connect
-
-            tunnel = request.path
-            log { "establishing HTTP proxy tunnel to #{tunnel}" }
-            tunnel
-          end
-
-          def empty?
-            @requests.reject { |r| r.verb == :connect }.empty? || @requests.all? { |request| !request.response.nil? }
-          end
-        end
-
         class ConnectRequest < Request
           def initialize(uri, _options)
             super(:connect, uri, {})

data/lib/httpx/plugins/proxy/socks4.rb

@@ -27,7 +27,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && PROTOCOLS.include?(@options.proxy.uri.scheme)
 
             case nextstate

data/lib/httpx/plugins/proxy/socks5.rb

@@ -46,7 +46,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && @options.proxy.uri.scheme == "socks5"
 
             case nextstate