httpx 0.18.4 → 0.19.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4c6e05e5bda153614ac00aac76cdaff0d5d727c58a99fb2297c545aa54b27c0e
-  data.tar.gz: 9d682e4136c3e8a3d769e02f96209a72dadbb5722a01d4ec8267cec06b5f1a3a
+  metadata.gz: 1d2af1a18388c76620d8a5b28e0f1d2dd90a4d2d0ee2c514be85ea7ba1b05c45
+  data.tar.gz: ca44331c601c46d51813cd5ccab8b258e94cf517913e26b78716885234e6cf86
 SHA512:
-  metadata.gz: f2549dd8e2b9786dfb6916f88bd9979e06fc32806a09ead664f97d67612eb3d4b182e4a48d1b23f037e899ecda50ad36b343e85e97258fd3ecb730f63238417e
-  data.tar.gz: f0388924dfba2717069b9cea8463e658d6262c6bfb874185c60baaab1650055fe4a98acd9fa39a6a060f00aa0aa05f1345d627bac138ecf5e17ad4d8aacd92b2
+  metadata.gz: eac8b6a85fdd44dda2bcaf000cc7dc5dfa19b075165b52fa709dd8b575e9b4a919293e724242d2449794b74e28cf34c00687c6f99e23baee563c45626429b3c7
+  data.tar.gz: 564f2e8f6d580070eaffa3cda012f0efb032948ce4851b96f947cd5a004d1e8fde9496f29031b341709ccc44040ea3473304697e5d5c7c659b7c2cd286702b78

data/doc/release_notes/0_18_3.md

@@ -4,4 +4,4 @@
 
 * request bodies eager-loaded from enumerables yield duped partial chunks.
 
-An error was observed while looking at webmock integration, where requests formed via the multipart plugin where returning an empty string as body. The issue was caused by an optimization on multipart encoder, which reuses the same buffer when reading chunks. Unfortunately, these cannot be yielded the same way via IO.copy_stream, as the same (cleared) buffer will be used to generate the eager-loaded body chunks.
+An error was observed while looking at webmock integration, where requests formed via the multipart plugin were returning an empty string as body. The issue was caused by an optimization on multipart encoder, which reuses the same buffer when reading chunks. Unfortunately, these cannot be yielded the same way via IO.copy_stream, as the same (cleared) buffer will be used to generate the eager-loaded body chunks.

data/doc/release_notes/0_18_4.md

@@ -4,11 +4,11 @@
 
 * faraday adapter: added support for `#on_data` callback in order to support [faraday streaming](https://lostisland.github.io/faraday/usage/streaming).
 
-* multipart plugin: removed support for file mime type detection using `mime-types`. The reasoning behind it was that `mime-types` uses the filename, which is a very innacurate detection strategy (ex: an mp4 video will be identified as `application/mp4`, instead of the correct `video/mp4`).
+* multipart plugin: removed support for file mime type detection using `mime-types`. The reasoning behind it was that `mime-types` uses the filename, which is a very inaccurate detection strategy (ex: an mp4 video will be identified as `application/mp4`, instead of the correct `video/mp4`).
 * multipart plugin: supported for file mime type detection using `marcel` and `filemagic` was added. Both use the magic header bytes, which is a more accurate strategy for file type detection.
 
 ## Bugfixes
 
 * webmock adapter has been reimplemented to work with `httpx` plugins (such as the `:retries` plugin). Some other fixes were applied to make it work better under `vcr` (a common `webmock` extension).
 
-* fixed the URI-related bug which was making requests stall under ruby 3.1 (still not officially testinng against it).
+* fixed the URI-related bug which was making requests stall under ruby 3.1 (still not officially testing against it).

data/doc/release_notes/0_18_5.md

@@ -0,0 +1,10 @@
+# 0.18.5
+
+## Improvements
+
+* ruby 3.1 is now officially supported.
+* when a user sets a `Host` header for an HTTP/2 request, this will be used in the `:authority` HTTP/2 pseudo-header, instead of silently ignored (mimicking what "curl" does).
+
+## Bugfixes
+
+* fixed "throw outside of catch block" error happening when pipelining requests on an HTTP/1 connection and resulting in a timeout.

data/doc/release_notes/0_18_6.md

@@ -0,0 +1,5 @@
+# 0.18.6
+
+## Bugfixes
+
+* multipart plugin: fixed missing constant in `filemagic` integration.

data/doc/release_notes/0_18_7.md

@@ -0,0 +1,5 @@
+# 0.18.6
+
+## Bugfixes
+
+* multipart plugin: fixed `filemagic` integration by rewinding the file after mime-type detection.

data/doc/release_notes/0_19_0.md

@@ -0,0 +1,39 @@
+# 0.19.0
+
+## Features
+
+### Happy Eyeballs v2
+
+When the system supports dual-stack networking, `httpx` implements the Happy Eyeballs v2 algorithm (RFC 8305) to resolve hostnames to both IPv6 and IPv4 addresses while privileging IPv6 connectivity. This is implemented by `httpx` both for the `:native` as well as the `:https` (DoH) resolver (which do not perform address sorting, thereby being "DNS-based load-balancing" friendly), and "outsourced" to `getaddrinfo` when using the `:system` resolver.
+
+IPv6 connectivity will also be privileged for `/etc/hosts` local DNS (i.e. `localhost` connections will connec to `::1`).
+
+A new option, `:ip_families`, will also be available (`[Socket::AF_INET6, Socket::AF_INET]` in dual-stack systems). If you'd like to i.e. force IPv4 connectivity, you can do use it (`client = HTTPX.with(ip_families: [Socket::AF_INET])`).
+
+## Improvements
+
+### DNS: :system resolver uses getaddrinfo (instead of the resolver lib)
+
+The `:system` resolver switched to using the `getaddinfo` system function to perform DNS requests. Not only is this call **not** blocking the session event loop anymore (unlike pre-0.19.0 `:system` resolver), it adds a lot of functionality that the stdlib `resolv` library just doesn't support at the moment (such as SRV records).
+
+### HTTP/2 proxy support
+
+The `:proxy` plugin handles "prior-knowledge" HTTP/2 proxies.
+
+```ruby
+HTTPX.plugin(:proxy, fallback_protocol: "h2").with_proxy(uri: "http://http2-proxy:3128").get(...
+```
+
+Connection coalescing has also been enabled for proxied connections (also `CONNECT`-tunneled connections).
+
+### curl-to-httpx
+
+widget in [project website](https://honeyryderchuck.gitlab.io/httpx/) to turn curl commands into the equivalent `httpx` code.
+
+## Bugfixes
+
+* faraday adapter now supports passing session options.
+* proxy: several fixes which enabled env-var (`HTTP(S)_PROXY`) defined proxy support.
+* proxy: fixed graceful recovery from proxy tcp connect errors.
+* several fixes around CNAMEs timeouts with the native resolver.
+* https resolver is now closed when wrapping session closes (it was left open).

data/lib/httpx/adapters/faraday.rb

@@ -81,6 +81,9 @@ module Faraday
 
           def response=(response)
             super
+
+            return if response.is_a?(::HTTPX::ErrorResponse)
+
             response.body.on_data = @response_on_data
           end
         end
@@ -136,7 +139,7 @@ module Faraday
 
           def on_response(&blk)
             if blk
-              @on_response = lambda do |response|
+              @on_response = ->(response) do
                 blk.call(response)
               end
               self
@@ -200,9 +203,9 @@ module Faraday
         end
       end
 
-      def initialize(app)
+      def initialize(app, options = {})
         super(app)
-        @session = Session.new
+        @session = Session.new(options)
       end
 
       def call(env)
@@ -210,6 +213,7 @@ module Faraday
         if parallel?(env)
           handler = env[:parallel_manager].enqueue(env)
           handler.on_response do |response|
+            response.raise_for_status
             save_response(env, response.status, response.body.to_s, response.headers, response.reason) do |response_headers|
               response_headers.merge!(response.headers)
             end

data/lib/httpx/connection/http1.rb

@@ -184,7 +184,7 @@ module HTTPX
       end
 
       if @pipelining
-        disable
+        catch(:called) { disable }
       else
         @requests.each do |request|
           emit(:error, request, ex)
@@ -297,10 +297,6 @@ module HTTPX
       extra_headers
     end
 
-    def headline_uri(request)
-      request.path
-    end
-
     def handle(request)
       catch(:buffer_full) do
         request.transition(:headers)
@@ -314,8 +310,12 @@ module HTTPX
       end
     end
 
+    def join_headline(request)
+      "#{request.verb.to_s.upcase} #{request.path} HTTP/#{@version.join(".")}"
+    end
+
     def join_headers(request)
-      headline = "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}"
+      headline = join_headline(request)
       @buffer << headline << CRLF
       log(color: :yellow) { "<- HEADLINE: #{headline.chomp.inspect}" }
       extra_headers = set_protocol_headers(request)

data/lib/httpx/connection/http2.rb

@@ -158,10 +158,6 @@ module HTTPX
       end
     end
 
-    def headline_uri(request)
-      request.path
-    end
-
     def handle(request, stream)
       catch(:buffer_full) do
         request.transition(:headers)
@@ -213,13 +209,19 @@ module HTTPX
       {
         ":scheme" => request.scheme,
         ":method" => request.verb.to_s.upcase,
-        ":path" => headline_uri(request),
+        ":path" => request.path,
         ":authority" => request.authority,
       }
     end
 
     def join_headers(stream, request)
       extra_headers = set_protocol_headers(request)
+
+      if request.headers.key?("host")
+        log { "forbidden \"host\" header found (#{request.headers["host"]}), will use it as authority..." }
+        extra_headers[":authority"] = request.headers["host"]
+      end
+
       log(level: 1, color: :yellow) do
         request.headers.merge(extra_headers).each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{v}" }.join("\n")
       end

data/lib/httpx/connection.rb

@@ -44,7 +44,7 @@ module HTTPX
 
     def_delegator :@write_buffer, :empty?
 
-    attr_reader :origin, :state, :pending, :options
+    attr_reader :io, :origin, :origins, :state, :pending, :options
 
     attr_writer :timers
 
@@ -78,7 +78,11 @@ module HTTPX
     # this is a semi-private method, to be used by the resolver
     # to initiate the io object.
     def addresses=(addrs)
-      @io ||= IO.registry(@type).new(@origin, addrs, @options) # rubocop:disable Naming/MemoizedInstanceVariableName
+      if @io
+        @io.add_addresses(addrs)
+      else
+        @io = IO.registry(@type).new(@origin, addrs, @options)
+      end
     end
 
     def addresses
@@ -489,6 +493,18 @@ module HTTPX
     end
 
     def transition(nextstate)
+      handle_transition(nextstate)
+    rescue Errno::ECONNREFUSED,
+           Errno::EADDRNOTAVAIL,
+           Errno::EHOSTUNREACH,
+           TLSError => e
+      # connect errors, exit gracefully
+      handle_error(e)
+      @state = :closed
+      emit(:close)
+    end
+
+    def handle_transition(nextstate)
       case nextstate
       when :idle
         @timeout = @current_timeout = @options.timeout[:connect_timeout]
@@ -525,14 +541,6 @@ module HTTPX
         emit(:activate)
       end
       @state = nextstate
-    rescue Errno::ECONNREFUSED,
-           Errno::EADDRNOTAVAIL,
-           Errno::EHOSTUNREACH,
-           TLSError => e
-      # connect errors, exit gracefully
-      handle_error(e)
-      @state = :closed
-      emit(:close)
     end
 
     def purge_after_closed
@@ -550,6 +558,10 @@ module HTTPX
           ex.set_backtrace(error.backtrace)
           error = ex
         else
+          # inactive connections do not contribute to the select loop, therefore
+          # they should fail due to such errors.
+          return if @state == :inactive
+
           if @timeout
             @timeout -= error.timeout
             return unless @timeout <= 0

data/lib/httpx/extensions.rb

@@ -54,6 +54,22 @@ module HTTPX
     Numeric.__send__(:include, NegMethods)
   end
 
+  module StringExtensions
+    refine String do
+      def delete_suffix!(suffix)
+        suffix = Backports.coerce_to_str(suffix)
+        chomp! if frozen?
+        len = suffix.length
+        if len > 0 && index(suffix, -len)
+          self[-len..-1] = ''
+          self
+        else
+          nil
+        end
+      end unless String.method_defined?(:delete_suffix!)
+    end
+  end
+
   module HashExtensions
     refine Hash do
       def compact

data/lib/httpx/headers.rb

@@ -2,8 +2,6 @@
 
 module HTTPX
   class Headers
-    EMPTY = [].freeze
-
     class << self
       def new(headers = nil)
         return headers if headers.is_a?(self)

data/lib/httpx/io/tcp.rb

@@ -15,6 +15,7 @@ module HTTPX
 
     def initialize(origin, addresses, options)
       @state = :idle
+      @addresses = []
       @hostname = origin.host
       @options = Options.new(options)
       @fallback_protocol = @options.fallback_protocol
@@ -30,15 +31,29 @@ module HTTPX
         raise Error, "Given IO objects do not match the request authority" unless @io
 
         _, _, _, @ip = @io.addr
-        @addresses ||= [@ip]
-        @ip_index = @addresses.size - 1
+        @addresses << @ip
         @keep_open = true
         @state = :connected
       else
-        @addresses = addresses.map { |addr| addr.is_a?(IPAddr) ? addr : IPAddr.new(addr) }
+        add_addresses(addresses)
       end
       @ip_index = @addresses.size - 1
-      @io ||= build_socket
+      # @io ||= build_socket
+    end
+
+    def add_addresses(addrs)
+      return if addrs.empty?
+
+      addrs = addrs.map { |addr| addr.is_a?(IPAddr) ? addr : IPAddr.new(addr) }
+
+      ip_index = @ip_index || (@addresses.size - 1)
+      if addrs.first.ipv6?
+        # should be the next in line
+        @addresses = [*@addresses[0, ip_index], *addrs, *@addresses[ip_index..-1]]
+      else
+        @addresses.unshift(*addrs)
+        @ip_index += addrs.size if @ip_index
+      end
     end
 
     def to_io
@@ -52,20 +67,26 @@ module HTTPX
     def connect
       return unless closed?
 
-      if @io.closed?
+      if !@io || @io.closed?
         transition(:idle)
         @io = build_socket
       end
       try_connect
-    rescue Errno::EHOSTUNREACH => e
+    rescue Errno::ECONNREFUSED,
+           Errno::EADDRNOTAVAIL,
+           Errno::EHOSTUNREACH => e
       raise e if @ip_index <= 0
 
+      log { "failed connecting to #{@ip} (#{e.message}), trying next..." }
       @ip_index -= 1
+      @io = build_socket
       retry
     rescue Errno::ETIMEDOUT => e
       raise ConnectTimeoutError.new(@options.timeout[:connect_timeout], e.message) if @ip_index <= 0
 
+      log { "failed connecting to #{@ip} (#{e.message}), trying next..." }
       @ip_index -= 1
+      @io = build_socket
       retry
     end
 

data/lib/httpx/options.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "socket"
+
 module HTTPX
   class Options
     WINDOW_SIZE = 1 << 14 # 16K
@@ -9,6 +11,18 @@ module HTTPX
     KEEP_ALIVE_TIMEOUT = 20
     SETTINGS_TIMEOUT = 10
 
+    # https://github.com/ruby/resolv/blob/095f1c003f6073730500f02acbdbc55f83d70987/lib/resolv.rb#L408
+    ip_address_families = begin
+      list = Socket.ip_address_list
+      if list.any? { |a| a.ipv6? && !a.ipv6_loopback? && !a.ipv6_linklocal? }
+        [Socket::AF_INET6, Socket::AF_INET]
+      else
+        [Socket::AF_INET]
+      end
+    rescue NotImplementedError
+      [Socket::AF_INET]
+    end
+
     DEFAULT_OPTIONS = {
       :debug => ENV.key?("HTTPX_DEBUG") ? $stderr : nil,
       :debug_level => (ENV["HTTPX_DEBUG"] || 1).to_i,
@@ -37,6 +51,7 @@ module HTTPX
       :persistent => false,
       :resolver_class => (ENV["HTTPX_RESOLVER"] || :native).to_sym,
       :resolver_options => { cache: true },
+      :ip_families => ip_address_families,
     }.freeze
 
     begin
@@ -110,20 +125,18 @@ module HTTPX
     end
 
     def initialize(options = {})
-      defaults = DEFAULT_OPTIONS.merge(options)
-      defaults.each do |k, v|
-        next if v.nil?
-
-        begin
-          value = __send__(:"option_#{k}", v)
-          instance_variable_set(:"@#{k}", value)
-        rescue NoMethodError
-          raise Error, "unknown option: #{k}"
-        end
-      end
+      __initialize__(options)
       freeze
     end
 
+    def freeze
+      super
+      @origin.freeze
+      @timeout.freeze
+      @headers.freeze
+      @addresses.freeze
+    end
+
     def option_origin(value)
       URI(value)
     end
@@ -174,6 +187,10 @@ module HTTPX
       Array(value)
     end
 
+    def option_ip_families(value)
+      Array(value)
+    end
+
     %i[
       params form json body ssl http2_settings
       request_class response_class headers_class request_body_class
@@ -249,5 +266,21 @@ module HTTPX
         end
       end
     end
+
+    private
+
+    def __initialize__(options = {})
+      defaults = DEFAULT_OPTIONS.merge(options)
+      defaults.each do |k, v|
+        next if v.nil?
+
+        begin
+          value = __send__(:"option_#{k}", v)
+          instance_variable_set(:"@#{k}", value)
+        rescue NoMethodError
+          raise Error, "unknown option: #{k}"
+        end
+      end
+    end
   end
 end

data/lib/httpx/plugins/cookies.rb

@@ -18,12 +18,6 @@ module HTTPX
         require "httpx/plugins/cookies/set_cookie_parser"
       end
 
-      module OptionsMethods
-        def option_cookies(value)
-          value.is_a?(Jar) ? value : Jar.new(value)
-        end
-      end
-
       module InstanceMethods
         extend Forwardable
 
@@ -77,7 +71,7 @@ module HTTPX
       end
 
       module OptionsMethods
-        def initialize(*)
+        def __initialize__(*)
           super
 
           return unless @headers.key?("cookie")
@@ -89,6 +83,10 @@ module HTTPX
             end
           end
         end
+
+        def option_cookies(value)
+          value.is_a?(Jar) ? value : Jar.new(value)
+        end
       end
     end
     register_plugin :cookies, Cookies

data/lib/httpx/plugins/internal_telemetry.rb

@@ -81,7 +81,7 @@ module HTTPX
           super
         end
 
-        def transition(nextstate)
+        def handle_transition(nextstate)
           state = @state
           super
           meter_elapsed_time("Connection##{object_id}[#{@origin}]: #{state} -> #{nextstate}") if nextstate == @state

data/lib/httpx/plugins/multipart/mime_type_detector.rb

@@ -9,12 +9,18 @@ module HTTPX
 
       # inspired by https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/determine_mime_type.rb
       if defined?(FileMagic)
+        MAGIC_NUMBER = 256 * 1024
+
         def call(file, _)
           return nil if file.eof? # FileMagic returns "application/x-empty" for empty files
 
-          FileMagic.open(FileMagic::MAGIC_MIME_TYPE) do |filemagic|
+          mime = FileMagic.open(FileMagic::MAGIC_MIME_TYPE) do |filemagic|
             filemagic.buffer(file.read(MAGIC_NUMBER))
           end
+
+          file.rewind
+
+          mime
         end
       elsif defined?(Marcel)
         def call(file, filename)

data/lib/httpx/plugins/proxy/http.rb

@@ -13,7 +13,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && @options.proxy.uri.scheme == "http"
 
             case nextstate
@@ -23,7 +23,8 @@ module HTTPX
               @io.connect
               return unless @io.connected?
 
-              @parser = ConnectProxyParser.new(@write_buffer, @options.merge(max_concurrent_requests: 1))
+              @parser = registry(@io.protocol).new(@write_buffer, @options.merge(max_concurrent_requests: 1))
+              @parser.extend(ProxyParser)
               @parser.once(:response, &method(:__http_on_connect))
               @parser.on(:close) { transition(:closing) }
               __http_proxy_connect
@@ -36,7 +37,7 @@ module HTTPX
                 @parser.close
                 @parser = nil
               when :idle
-                @parser = ProxyParser.new(@write_buffer, @options)
+                @parser.callbacks.clear
                 set_parser_callbacks(@parser)
               end
             end
@@ -54,7 +55,7 @@ module HTTPX
               @inflight += 1
               parser.send(connect_request)
             else
-              transition(:connected)
+              handle_transition(:connected)
             end
           end
 
@@ -76,9 +77,11 @@ module HTTPX
           end
         end
 
-        class ProxyParser < Connection::HTTP1
-          def headline_uri(request)
-            request.uri.to_s
+        module ProxyParser
+          def join_headline(request)
+            return super if request.verb == :connect
+
+            "#{request.verb.to_s.upcase} #{request.uri} HTTP/#{@version.join(".")}"
           end
 
           def set_protocol_headers(request)
@@ -91,22 +94,6 @@ module HTTPX
           end
         end
 
-        class ConnectProxyParser < ProxyParser
-          attr_reader :pending
-
-          def headline_uri(request)
-            return super unless request.verb == :connect
-
-            tunnel = request.path
-            log { "establishing HTTP proxy tunnel to #{tunnel}" }
-            tunnel
-          end
-
-          def empty?
-            @requests.reject { |r| r.verb == :connect }.empty? || @requests.all? { |request| !request.response.nil? }
-          end
-        end
-
         class ConnectRequest < Request
           def initialize(uri, _options)
             super(:connect, uri, {})

data/lib/httpx/plugins/proxy/socks4.rb

@@ -27,7 +27,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && PROTOCOLS.include?(@options.proxy.uri.scheme)
 
             case nextstate

data/lib/httpx/plugins/proxy/socks5.rb

@@ -46,7 +46,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && @options.proxy.uri.scheme == "socks5"
 
             case nextstate