httpx 0.18.5 → 0.19.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 843e16ae32c900167850fe3a864446af2cd6b6a9d3ca6368f64b999964a1a5b2
-  data.tar.gz: 533464195c08ab7e0ba2e5aa7871fdf8b59e59643122ca48ce5e7cf8a6ebb351
+  metadata.gz: 56e51d7951de9e5964eb1106d8829504e1d0fbac2ca7906b64f6139467159403
+  data.tar.gz: 6b7726c50101f8356cf04628ee9ff32cbd6babd3bf98f426d768ae52d835f4a0
 SHA512:
-  metadata.gz: 6e170cf6a47bce0a1eeeea25c53870615958f77e2fe9eb9b4176783b860bc26c8d45712e0ad758ba2481bd2712a21bb14d10a32fb5f17871a9421413edf162ac
-  data.tar.gz: ae5a0bc77a54cf66528dfd89c7127b5ce64730be6fc0fe8be6d79966dfa36e2e04b348f39e1e91d3b457e34c35f8b3a33e18bc656f7823874ffbaedc72e3b8ad
+  metadata.gz: d29f216b4176e5085d7495d3ac015eb4b9f7e76f4dd064ae1f822890cc44d1718ec14fe9a3935515af6c7882762ebd875a0e15983d5707a2c9f68d069bfd084c
+  data.tar.gz: 30537730f11c0bfffd43882abd430c8a006a2a863b7ef247286f085440781d1b44e048d031df2c3af95fffc9d5ce04fcaad6bd0d05d039fc77023a0dc95a7508

data/README.md

@@ -140,11 +140,15 @@ In order to use HTTP/2 under JRuby, [check this link](https://gitlab.com/honeyry
 * Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryderchuck/httpx/issues/36)).
 * Using `total_timeout` along with the `:persistent` plugin [does not work as you might expect](https://gitlab.com/honeyryderchuck/httpx/-/wikis/Timeouts#total_timeout).
 
+## Versioning Policy
+
+Although 0.x software, `httpx` is considered API-stable and production-ready, i.e. current API or options may be subject to deprecation and emit log warnings, but can only effectively be removed in a major version change.
+
 ## Contributing
 
 * Discuss your contribution in an issue
 * Fork it
 * Make your changes, add some tests
-* Ensure all tests pass (`bundle exec rake test`)
+* Ensure all tests pass (`docker-compose -f docker-compose.yml -f docker-compose-ruby-{RUBY_VERSION}.yml run httpx bundle exec rake test`)
 * Open a Merge Request (that's Pull Request in Github-ish)
 * Wait for feedback

data/doc/release_notes/0_18_5.md

@@ -3,8 +3,8 @@
 ## Improvements
 
 * ruby 3.1 is now officially supported.
-* when user sets a `Host` header for an HTTP/2 request, this will be used in the `:authority` HTTP/2 pseudo-header, instead of silently ignored (mimicking what "curl" does).
+* when a user sets a `Host` header for an HTTP/2 request, this will be used in the `:authority` HTTP/2 pseudo-header, instead of silently ignored (mimicking what "curl" does).
 
 ## Bugfixes
 
-* fixed "throw outside of catch block" error happening when pipelining requests on an HTTP/1 connnection and resulting in a timeout.
+* fixed "throw outside of catch block" error happening when pipelining requests on an HTTP/1 connection and resulting in a timeout.

data/doc/release_notes/0_18_6.md

@@ -0,0 +1,5 @@
+# 0.18.6
+
+## Bugfixes
+
+* multipart plugin: fixed missing constant in `filemagic` integration.

data/doc/release_notes/0_18_7.md

@@ -0,0 +1,5 @@
+# 0.18.6
+
+## Bugfixes
+
+* multipart plugin: fixed `filemagic` integration by rewinding the file after mime-type detection.

data/doc/release_notes/0_19_0.md

@@ -0,0 +1,39 @@
+# 0.19.0
+
+## Features
+
+### Happy Eyeballs v2
+
+When the system supports dual-stack networking, `httpx` implements the Happy Eyeballs v2 algorithm (RFC 8305) to resolve hostnames to both IPv6 and IPv4 addresses while privileging IPv6 connectivity. This is implemented by `httpx` both for the `:native` as well as the `:https` (DoH) resolver (which do not perform address sorting, thereby being "DNS-based load-balancing" friendly), and "outsourced" to `getaddrinfo` when using the `:system` resolver.
+
+IPv6 connectivity will also be privileged for `/etc/hosts` local DNS (i.e. `localhost` connections will connec to `::1`).
+
+A new option, `:ip_families`, will also be available (`[Socket::AF_INET6, Socket::AF_INET]` in dual-stack systems). If you'd like to i.e. force IPv4 connectivity, you can do use it (`client = HTTPX.with(ip_families: [Socket::AF_INET])`).
+
+## Improvements
+
+### DNS: :system resolver uses getaddrinfo (instead of the resolver lib)
+
+The `:system` resolver switched to using the `getaddinfo` system function to perform DNS requests. Not only is this call **not** blocking the session event loop anymore (unlike pre-0.19.0 `:system` resolver), it adds a lot of functionality that the stdlib `resolv` library just doesn't support at the moment (such as SRV records).
+
+### HTTP/2 proxy support
+
+The `:proxy` plugin handles "prior-knowledge" HTTP/2 proxies.
+
+```ruby
+HTTPX.plugin(:proxy, fallback_protocol: "h2").with_proxy(uri: "http://http2-proxy:3128").get(...
+```
+
+Connection coalescing has also been enabled for proxied connections (also `CONNECT`-tunneled connections).
+
+### curl-to-httpx
+
+widget in [project website](https://honeyryderchuck.gitlab.io/httpx/) to turn curl commands into the equivalent `httpx` code.
+
+## Bugfixes
+
+* faraday adapter now supports passing session options.
+* proxy: several fixes which enabled env-var (`HTTP(S)_PROXY`) defined proxy support.
+* proxy: fixed graceful recovery from proxy tcp connect errors.
+* several fixes around CNAMEs timeouts with the native resolver.
+* https resolver is now closed when wrapping session closes (it was left open).

data/doc/release_notes/0_19_1.md

@@ -0,0 +1,5 @@
+# 0.19.1
+
+## Bugfixes
+
+Fixing a DNS dual-stack case where one the resolvers may have finished way before the previous one and will therefore return no timeout.

data/lib/httpx/adapters/faraday.rb

@@ -81,6 +81,9 @@ module Faraday
 
           def response=(response)
             super
+
+            return if response.is_a?(::HTTPX::ErrorResponse)
+
             response.body.on_data = @response_on_data
           end
         end
@@ -136,7 +139,7 @@ module Faraday
 
           def on_response(&blk)
             if blk
-              @on_response = lambda do |response|
+              @on_response = ->(response) do
                 blk.call(response)
               end
               self
@@ -200,9 +203,9 @@ module Faraday
         end
       end
 
-      def initialize(app)
+      def initialize(app, options = {})
         super(app)
-        @session = Session.new
+        @session = Session.new(options)
       end
 
       def call(env)
@@ -210,6 +213,7 @@ module Faraday
         if parallel?(env)
           handler = env[:parallel_manager].enqueue(env)
           handler.on_response do |response|
+            response.raise_for_status
             save_response(env, response.status, response.body.to_s, response.headers, response.reason) do |response_headers|
               response_headers.merge!(response.headers)
             end

data/lib/httpx/connection/http1.rb

@@ -297,10 +297,6 @@ module HTTPX
       extra_headers
     end
 
-    def headline_uri(request)
-      request.path
-    end
-
     def handle(request)
       catch(:buffer_full) do
         request.transition(:headers)
@@ -314,8 +310,12 @@ module HTTPX
       end
     end
 
+    def join_headline(request)
+      "#{request.verb.to_s.upcase} #{request.path} HTTP/#{@version.join(".")}"
+    end
+
     def join_headers(request)
-      headline = "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}"
+      headline = join_headline(request)
       @buffer << headline << CRLF
       log(color: :yellow) { "<- HEADLINE: #{headline.chomp.inspect}" }
       extra_headers = set_protocol_headers(request)

data/lib/httpx/connection/http2.rb

@@ -158,10 +158,6 @@ module HTTPX
       end
     end
 
-    def headline_uri(request)
-      request.path
-    end
-
     def handle(request, stream)
       catch(:buffer_full) do
         request.transition(:headers)
@@ -213,7 +209,7 @@ module HTTPX
       {
         ":scheme" => request.scheme,
         ":method" => request.verb.to_s.upcase,
-        ":path" => headline_uri(request),
+        ":path" => request.path,
         ":authority" => request.authority,
       }
     end

data/lib/httpx/connection.rb

@@ -44,7 +44,7 @@ module HTTPX
 
     def_delegator :@write_buffer, :empty?
 
-    attr_reader :origin, :state, :pending, :options
+    attr_reader :io, :origin, :origins, :state, :pending, :options
 
     attr_writer :timers
 
@@ -78,7 +78,11 @@ module HTTPX
     # this is a semi-private method, to be used by the resolver
     # to initiate the io object.
     def addresses=(addrs)
-      @io ||= IO.registry(@type).new(@origin, addrs, @options) # rubocop:disable Naming/MemoizedInstanceVariableName
+      if @io
+        @io.add_addresses(addrs)
+      else
+        @io = IO.registry(@type).new(@origin, addrs, @options)
+      end
     end
 
     def addresses
@@ -489,6 +493,18 @@ module HTTPX
     end
 
     def transition(nextstate)
+      handle_transition(nextstate)
+    rescue Errno::ECONNREFUSED,
+           Errno::EADDRNOTAVAIL,
+           Errno::EHOSTUNREACH,
+           TLSError => e
+      # connect errors, exit gracefully
+      handle_error(e)
+      @state = :closed
+      emit(:close)
+    end
+
+    def handle_transition(nextstate)
       case nextstate
       when :idle
         @timeout = @current_timeout = @options.timeout[:connect_timeout]
@@ -525,14 +541,6 @@ module HTTPX
         emit(:activate)
       end
       @state = nextstate
-    rescue Errno::ECONNREFUSED,
-           Errno::EADDRNOTAVAIL,
-           Errno::EHOSTUNREACH,
-           TLSError => e
-      # connect errors, exit gracefully
-      handle_error(e)
-      @state = :closed
-      emit(:close)
     end
 
     def purge_after_closed
@@ -550,6 +558,10 @@ module HTTPX
           ex.set_backtrace(error.backtrace)
           error = ex
         else
+          # inactive connections do not contribute to the select loop, therefore
+          # they should fail due to such errors.
+          return if @state == :inactive
+
           if @timeout
             @timeout -= error.timeout
             return unless @timeout <= 0

data/lib/httpx/extensions.rb

@@ -54,6 +54,22 @@ module HTTPX
     Numeric.__send__(:include, NegMethods)
   end
 
+  module StringExtensions
+    refine String do
+      def delete_suffix!(suffix)
+        suffix = Backports.coerce_to_str(suffix)
+        chomp! if frozen?
+        len = suffix.length
+        if len > 0 && index(suffix, -len)
+          self[-len..-1] = ''
+          self
+        else
+          nil
+        end
+      end unless String.method_defined?(:delete_suffix!)
+    end
+  end
+
   module HashExtensions
     refine Hash do
       def compact

data/lib/httpx/headers.rb

@@ -2,8 +2,6 @@
 
 module HTTPX
   class Headers
-    EMPTY = [].freeze
-
     class << self
       def new(headers = nil)
         return headers if headers.is_a?(self)

data/lib/httpx/io/tcp.rb

@@ -15,6 +15,7 @@ module HTTPX
 
     def initialize(origin, addresses, options)
       @state = :idle
+      @addresses = []
       @hostname = origin.host
       @options = Options.new(options)
       @fallback_protocol = @options.fallback_protocol
@@ -30,15 +31,29 @@ module HTTPX
         raise Error, "Given IO objects do not match the request authority" unless @io
 
         _, _, _, @ip = @io.addr
-        @addresses ||= [@ip]
-        @ip_index = @addresses.size - 1
+        @addresses << @ip
         @keep_open = true
         @state = :connected
       else
-        @addresses = addresses.map { |addr| addr.is_a?(IPAddr) ? addr : IPAddr.new(addr) }
+        add_addresses(addresses)
       end
       @ip_index = @addresses.size - 1
-      @io ||= build_socket
+      # @io ||= build_socket
+    end
+
+    def add_addresses(addrs)
+      return if addrs.empty?
+
+      addrs = addrs.map { |addr| addr.is_a?(IPAddr) ? addr : IPAddr.new(addr) }
+
+      ip_index = @ip_index || (@addresses.size - 1)
+      if addrs.first.ipv6?
+        # should be the next in line
+        @addresses = [*@addresses[0, ip_index], *addrs, *@addresses[ip_index..-1]]
+      else
+        @addresses.unshift(*addrs)
+        @ip_index += addrs.size if @ip_index
+      end
     end
 
     def to_io
@@ -52,20 +67,26 @@ module HTTPX
     def connect
       return unless closed?
 
-      if @io.closed?
+      if !@io || @io.closed?
         transition(:idle)
         @io = build_socket
       end
       try_connect
-    rescue Errno::EHOSTUNREACH => e
+    rescue Errno::ECONNREFUSED,
+           Errno::EADDRNOTAVAIL,
+           Errno::EHOSTUNREACH => e
       raise e if @ip_index <= 0
 
+      log { "failed connecting to #{@ip} (#{e.message}), trying next..." }
       @ip_index -= 1
+      @io = build_socket
       retry
     rescue Errno::ETIMEDOUT => e
       raise ConnectTimeoutError.new(@options.timeout[:connect_timeout], e.message) if @ip_index <= 0
 
+      log { "failed connecting to #{@ip} (#{e.message}), trying next..." }
       @ip_index -= 1
+      @io = build_socket
       retry
     end
 

data/lib/httpx/options.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "socket"
+
 module HTTPX
   class Options
     WINDOW_SIZE = 1 << 14 # 16K
@@ -9,6 +11,18 @@ module HTTPX
     KEEP_ALIVE_TIMEOUT = 20
     SETTINGS_TIMEOUT = 10
 
+    # https://github.com/ruby/resolv/blob/095f1c003f6073730500f02acbdbc55f83d70987/lib/resolv.rb#L408
+    ip_address_families = begin
+      list = Socket.ip_address_list
+      if list.any? { |a| a.ipv6? && !a.ipv6_loopback? && !a.ipv6_linklocal? }
+        [Socket::AF_INET6, Socket::AF_INET]
+      else
+        [Socket::AF_INET]
+      end
+    rescue NotImplementedError
+      [Socket::AF_INET]
+    end
+
     DEFAULT_OPTIONS = {
       :debug => ENV.key?("HTTPX_DEBUG") ? $stderr : nil,
       :debug_level => (ENV["HTTPX_DEBUG"] || 1).to_i,
@@ -37,6 +51,7 @@ module HTTPX
       :persistent => false,
       :resolver_class => (ENV["HTTPX_RESOLVER"] || :native).to_sym,
       :resolver_options => { cache: true },
+      :ip_families => ip_address_families,
     }.freeze
 
     begin
@@ -110,20 +125,18 @@ module HTTPX
     end
 
     def initialize(options = {})
-      defaults = DEFAULT_OPTIONS.merge(options)
-      defaults.each do |k, v|
-        next if v.nil?
-
-        begin
-          value = __send__(:"option_#{k}", v)
-          instance_variable_set(:"@#{k}", value)
-        rescue NoMethodError
-          raise Error, "unknown option: #{k}"
-        end
-      end
+      __initialize__(options)
       freeze
     end
 
+    def freeze
+      super
+      @origin.freeze
+      @timeout.freeze
+      @headers.freeze
+      @addresses.freeze
+    end
+
     def option_origin(value)
       URI(value)
     end
@@ -174,6 +187,10 @@ module HTTPX
       Array(value)
     end
 
+    def option_ip_families(value)
+      Array(value)
+    end
+
     %i[
       params form json body ssl http2_settings
       request_class response_class headers_class request_body_class
@@ -249,5 +266,21 @@ module HTTPX
         end
       end
     end
+
+    private
+
+    def __initialize__(options = {})
+      defaults = DEFAULT_OPTIONS.merge(options)
+      defaults.each do |k, v|
+        next if v.nil?
+
+        begin
+          value = __send__(:"option_#{k}", v)
+          instance_variable_set(:"@#{k}", value)
+        rescue NoMethodError
+          raise Error, "unknown option: #{k}"
+        end
+      end
+    end
   end
 end

data/lib/httpx/plugins/cookies.rb

@@ -18,12 +18,6 @@ module HTTPX
         require "httpx/plugins/cookies/set_cookie_parser"
       end
 
-      module OptionsMethods
-        def option_cookies(value)
-          value.is_a?(Jar) ? value : Jar.new(value)
-        end
-      end
-
       module InstanceMethods
         extend Forwardable
 
@@ -77,7 +71,7 @@ module HTTPX
       end
 
       module OptionsMethods
-        def initialize(*)
+        def __initialize__(*)
           super
 
           return unless @headers.key?("cookie")
@@ -89,6 +83,10 @@ module HTTPX
             end
           end
         end
+
+        def option_cookies(value)
+          value.is_a?(Jar) ? value : Jar.new(value)
+        end
       end
     end
     register_plugin :cookies, Cookies

data/lib/httpx/plugins/internal_telemetry.rb

@@ -81,7 +81,7 @@ module HTTPX
           super
         end
 
-        def transition(nextstate)
+        def handle_transition(nextstate)
           state = @state
           super
           meter_elapsed_time("Connection##{object_id}[#{@origin}]: #{state} -> #{nextstate}") if nextstate == @state

data/lib/httpx/plugins/multipart/mime_type_detector.rb

@@ -9,12 +9,18 @@ module HTTPX
 
       # inspired by https://github.com/shrinerb/shrine/blob/master/lib/shrine/plugins/determine_mime_type.rb
       if defined?(FileMagic)
+        MAGIC_NUMBER = 256 * 1024
+
         def call(file, _)
           return nil if file.eof? # FileMagic returns "application/x-empty" for empty files
 
-          FileMagic.open(FileMagic::MAGIC_MIME_TYPE) do |filemagic|
+          mime = FileMagic.open(FileMagic::MAGIC_MIME_TYPE) do |filemagic|
             filemagic.buffer(file.read(MAGIC_NUMBER))
           end
+
+          file.rewind
+
+          mime
         end
       elsif defined?(Marcel)
         def call(file, filename)

data/lib/httpx/plugins/proxy/http.rb

@@ -13,7 +13,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && @options.proxy.uri.scheme == "http"
 
             case nextstate
@@ -23,7 +23,8 @@ module HTTPX
               @io.connect
               return unless @io.connected?
 
-              @parser = ConnectProxyParser.new(@write_buffer, @options.merge(max_concurrent_requests: 1))
+              @parser = registry(@io.protocol).new(@write_buffer, @options.merge(max_concurrent_requests: 1))
+              @parser.extend(ProxyParser)
               @parser.once(:response, &method(:__http_on_connect))
               @parser.on(:close) { transition(:closing) }
               __http_proxy_connect
@@ -36,7 +37,7 @@ module HTTPX
                 @parser.close
                 @parser = nil
               when :idle
-                @parser = ProxyParser.new(@write_buffer, @options)
+                @parser.callbacks.clear
                 set_parser_callbacks(@parser)
               end
             end
@@ -54,7 +55,7 @@ module HTTPX
               @inflight += 1
               parser.send(connect_request)
             else
-              transition(:connected)
+              handle_transition(:connected)
             end
           end
 
@@ -76,9 +77,11 @@ module HTTPX
           end
         end
 
-        class ProxyParser < Connection::HTTP1
-          def headline_uri(request)
-            request.uri.to_s
+        module ProxyParser
+          def join_headline(request)
+            return super if request.verb == :connect
+
+            "#{request.verb.to_s.upcase} #{request.uri} HTTP/#{@version.join(".")}"
           end
 
           def set_protocol_headers(request)
@@ -91,22 +94,6 @@ module HTTPX
           end
         end
 
-        class ConnectProxyParser < ProxyParser
-          attr_reader :pending
-
-          def headline_uri(request)
-            return super unless request.verb == :connect
-
-            tunnel = request.path
-            log { "establishing HTTP proxy tunnel to #{tunnel}" }
-            tunnel
-          end
-
-          def empty?
-            @requests.reject { |r| r.verb == :connect }.empty? || @requests.all? { |request| !request.response.nil? }
-          end
-        end
-
         class ConnectRequest < Request
           def initialize(uri, _options)
             super(:connect, uri, {})

data/lib/httpx/plugins/proxy/socks4.rb

@@ -27,7 +27,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && PROTOCOLS.include?(@options.proxy.uri.scheme)
 
             case nextstate

data/lib/httpx/plugins/proxy/socks5.rb

@@ -46,7 +46,7 @@ module HTTPX
 
           private
 
-          def transition(nextstate)
+          def handle_transition(nextstate)
             return super unless @options.proxy && @options.proxy.uri.scheme == "socks5"
 
             case nextstate