httpx 0.11.1 → 0.13.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9b21be17b0c6a6d3e2f60a97582b88f33dc1f62e09e5555cabfc2d6bea1f06a7
-  data.tar.gz: e0ca927150864a80a8050cc9fbc2c23e58b55452b536ce4301eceecebf1c6d37
+  metadata.gz: e3859c516c1cdddc64f8105fafd5425da26074582b1590cc87f65401a909e0e2
+  data.tar.gz: 209b5c1fba921d69f9aa2167ccb0f4b6b45a1a28667364eefef8026104004f53
 SHA512:
-  metadata.gz: 1493d50a5b49dd7c83f48078d582f874cc204d3e5df3442d7ea8308b31e6fd011e30cc3fdc5f1825d0daebd270c22af387700ab8f6d2fb9183d2e1fb63bd7265
-  data.tar.gz: ac48ea08cacc65a4948fea8a2abdd50fe7d7fd30c6260400cd61197ea7e6e8d87de2b4750ca1452f242569bbf6dc304c14865f809e7871a4f55d33e7e5543c03
+  metadata.gz: 3862b4879291aa944b92c50d232abc6413a170f2bcd501d6f33a1ca55a99a1411dd0d4f2b6a50b92a8b1465fa42b654ef658c15a91154205bfd60bcdc983c5a5
+  data.tar.gz: 6b2f721cc2bb8e0fe7267517dcd146fe50f448676bcec64e9744d5f358232d8d9c364acd6717e0df3a72bfcb1d154a7e4b74252324cf9a248f1fe062242eb4fe

data/README.md

@@ -113,7 +113,7 @@ The test suite runs against [httpbin proxied over nghttp2](https://nghttp2.org/h
 
 ## Supported Rubies
 
-All Rubies greater or equal to 2.1, and always latest JRuby.
+All Rubies greater or equal to 2.1, and always latest JRuby and Truffleruby.
 
 **Note**: This gem is tested against all latest patch versions, i.e. if you're using 2.2.0 and you experience some issue, please test it against 2.2.10 (latest patch version of 2.2) before creating an issue.
 
@@ -133,7 +133,7 @@ All Rubies greater or equal to 2.1, and always latest JRuby.
 
 If your requirement is to run requests over HTTP/2 and TLS, make sure you run a version of the gem which compiles OpenSSL 1.0.2 (Ruby 2.3 and higher are guaranteed to).
 
-JRuby's `openssl` is based on Bouncy Castle, which is massively outdated and still doesn't implement ALPN. So HTTP/2 over TLS/ALPN negotiation is off until JRuby figures this out.
+In order to use HTTP/2 under JRuby, [check this link](https://gitlab.com/honeyryderchuck/httpx/-/wikis/JRuby-Truffleruby-Other-Rubies) to know what to do.
 
 ### Known bugs
 

data/doc/release_notes/0_11_1.md

@@ -1 +1,5 @@
-0_11_1.md
+# 0.11.1
+
+## Bugfixes
+
+Fixed a bug related to the `:compression` plugin trying to process a last empty DATA frame from an HTTP/2 response, after it had been closed.

data/doc/release_notes/0_11_2.md

@@ -0,0 +1,5 @@
+# 0.11.2
+
+## Bugfixes
+
+The `:cookies` plugin wasn't able to parse `Expires` values, as it was using `Time.httpdate` to parse timestamps, which is RFC 2616-compliant, whereas cookies datetime values need to be RFC 6265-compliant.

data/doc/release_notes/0_11_3.md

@@ -0,0 +1,5 @@
+# 0.11.3
+
+## Bugfixes
+
+`EOFError` was being thrown when HTTP/1.1 non-chunked responses which don't contain a "content-length" header. The RFC mandates that the socket should be consumed until the server closes it, so a patch was implemented, to return the available response in such cases.

data/doc/release_notes/0_12_0.md

@@ -0,0 +1,55 @@
+# 0.12.0
+
+## Features
+
+### AWS Sigv4 Authentication Plugin
+
+A new plugin, `:aws_sigv4`, is now shipped with `httpx`. It implements the [AWS Signature Version 4 request signing process](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html), a well documented way of authenticating requests to AWS services, which has since been adopted by other cloud providers, such as Google Cloud Storage.
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#sessionaws_sigv4_authentication
+
+For convenience, there's a derivative plugin, `:aws_sdk_authentication`, which builds on top of `:aws_sigv4`, and integrates with the `aws-sdk-core` gem, maintained by AWS, to resolve the authentication credentials (p.ex. if you support ephemeral access keys).
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#sessionaws_sdk_authentication
+
+Other FAQ: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#faqs
+
+### HTTP/2 support for JRuby
+
+`jruby-openssl` doesn't support ALPN protocol negotiation, nor are there plans to implement, which limited the seamless HTTP/2 usage in `httpx`. A new connection adapter was therefore added specifically for JRuby, where ssl/tls connections will be handled using ffi-based openssl bindings, provided you bundle `ffi-compiler` and `concurrent-ruby`, and install a TLS/1.2-compatible `openssl` package.
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/JRuby-Truffleruby-Other-Rubies#http2
+
+## Improvements
+
+
+### truffleruby support
+
+`httpx` supports and tests against `truffleruby` (known to run tests since v20.3, passing all tests since v21).
+
+### Performance
+
+Several optimizations were introduced:
+
+* Reduction in read/write system calls;
+* more usage of `String#byteslice` in parsing (instead of string mutation);
+* Avoid selection on connections with no outstanding requests;
+
+They all contributed to a massive performance improvement, itself reflected in test runs, which need half the time they used to to complete.
+
+### APIs
+
+* `HTTPX::ErrorResponse#to_s` now uses the exception full message, instead of just the backtrace.
+
+## Bugfixes
+
+* HTTP/2 stream protocol errors do not cause the process to hang (instead, error responnses are yielded);
+* Fixed body stream bugs on retries when error causing retry would happen mid-transfer;
+* Fixed `:multipart` plugin body rewind on retries to start the transfer from the beginning;
+* Fixed auto-load of  `:proxy` plugin when `HTTPS_PROXY` or `HTTP_PROXY` is set;
+* Errno::EPIPE errors mid transfer now cause `httpx` to read from the server and get the appropriate HTTP error response;
+* Make sure that all requests have an error responnse if the error happens early;
+* Fixed TCP handshake Errno::INPROGRESS handling inside TLS connnections, which was causing the process to hang in a high handshake contention scenario;
+* Do not call the event loop if there's nothing to listen on (the DoH resolver was being listened on even if there was nothing to be request);
+* Fixed double event registry for DoH resolvers;
+* 

data/doc/release_notes/0_13_0.md

@@ -0,0 +1,58 @@
+# 0.13.0
+
+## Features
+
+### Upgrade plugin
+
+A new plugin, `:upgrade`, is now available. This plugin allows one to "hook" on HTTP/1.1's protocol upgrade mechanism (see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocol_upgrade_mechanism), which is the mechanism that browsers use to initiate websockets (there is an example of how to use `httpx` to start a websocket client connection [in the tests](https://gitlab.com/honeyryderchuck/httpx/-/blob/master/test/support/requests/plugins/upgrade.rb))
+
+You can read more about the `:upgrade` plugin in the [wiki](https://honeyryderchuck.gitlab.io/httpx/wiki/Connection-Upgrade).
+
+It's the basis of two plugins:
+
+#### `:h2c`
+
+This plugin was been rewritten on top of the `:upgrade` plugin, and handles upgrading a plaintext (non-"https") HTTP/1.1 connection, into an HTTP/2 connection.
+
+https://honeyryderchuck.gitlab.io/httpx/wiki/Connection-Upgrade#h2c
+
+#### `:upgrade/h2`
+
+This plugin handles when a server responds to a request with an `Upgrade: h2` header, does the following requests to the same origin via HTTP/2 prior knowledge (bypassing the necessity for ALPN negotiation, which is the whole point of the feature).
+
+https://honeyryderchuck.gitlab.io/httpx/wiki/Connection-Upgrade#h2
+
+### `:addresses` option
+
+The `:addresses` option is now available. You can use it to pass a list of IPs to connect to:
+
+```ruby
+# will not resolve example.com, and instead connect to one of the IPs passed.
+HTTPX.get("http://example.com", addresses: %w[172.5.3.1 172.5.3.2]))
+```
+
+You should also use it to connect to HTTP servers bound to a UNIX socket, in which case you'll have to provide a path:
+
+```ruby
+HTTPX.get("http://example.com", addresses: %w[/path/to/usocket]))
+```
+
+The `:transport_options` are therefore deprecated, and will be moved in a major version.
+
+## Improvements
+
+Some internal improvements that allow certainn plugins not to "leak" globally, such as the `:compression` plugin, which used to enable compression for all the `httpx` sessions from the same process. It doesn't anymore.
+
+Using exceptionless nonblocking connect calls in the supported rubies.
+
+Removed unneeded APIs around the Options object (`with_` methods, or the defined options list).
+
+## Bugfixes
+
+HTTP/1.1 persistent connections were closing after each request after the max requests was reached. It's fixed, and the new connection will also be persistent.
+
+When passing open IO objects for origins (the `:io` option), `httpx` was still trying to resolve the origin's domain. This not only didn't make sense, it broke if the domain is unresolvable. It has been fixed.
+
+Fixed usage of `:io` option when passed an "authority/io" hash.
+
+Fixing some issues around trying to connnect to the next available IPAddress when the previous one was unreachable or ETIMEDOUT.

data/doc/release_notes/0_13_1.md

@@ -0,0 +1,5 @@
+# 0.13.1
+
+## Bugfixes
+
+Rescue `Errno::EALREADY` on calls to `connect_nonblock(exception: false)` (there are exceptionns after all...).

data/lib/httpx.rb

@@ -19,7 +19,6 @@ require "httpx/headers"
 require "httpx/request"
 require "httpx/response"
 require "httpx/chainable"
-require "httpx/session"
 
 # Top-Level Namespace
 #
@@ -59,3 +58,5 @@ module HTTPX
 
   extend Chainable
 end
+
+require "httpx/session"

data/lib/httpx/adapters/faraday.rb

@@ -123,11 +123,9 @@ module Faraday
           end
 
           def method_missing(meth, *args, &blk)
-            if @env && @env.respond_to?(meth)
-              @env.__send__(meth, *args, &blk)
-            else
-              super
-            end
+            return super unless @env && @env.respond_to?(meth)
+
+            @env.__send__(meth, *args, &blk)
           end
         end
 
@@ -197,7 +195,7 @@ module Faraday
           response_headers.merge!(response.headers)
         end
         @app.call(env)
-      rescue OpenSSL::SSL::SSLError => e
+      rescue ::HTTPX::TLSError => e
         raise SSL_ERROR, e
       rescue Errno::ECONNABORTED,
              Errno::ECONNREFUSED,

data/lib/httpx/altsvc.rb

@@ -37,6 +37,7 @@ module HTTPX
     end
 
     def emit(request, response)
+      return unless response.respond_to?(:headers)
       # Alt-Svc
       return unless response.headers.key?("alt-svc")
 

data/lib/httpx/chainable.rb

@@ -17,12 +17,12 @@ module HTTPX
     # :nocov:
     def timeout(**args)
       warn ":#{__method__} is deprecated, use :with_timeout instead"
-      branch(default_options.with(timeout: args))
+      with(timeout: args)
     end
 
     def headers(headers)
       warn ":#{__method__} is deprecated, use :with_headers instead"
-      branch(default_options.with(headers: headers))
+      with(headers: headers)
     end
     # :nocov:
 

data/lib/httpx/connection.rb

@@ -71,6 +71,8 @@ module HTTPX
       @inflight = 0
       @keep_alive_timeout = options.timeout.keep_alive_timeout
       @keep_alive_timer = nil
+
+      self.addresses = options.addresses if options.addresses
     end
 
     # this is a semi-private method, to be used by the resolver
@@ -105,6 +107,8 @@ module HTTPX
 
       return false if exhausted?
 
+      return false unless connection.addresses
+
       !(@io.addresses & connection.addresses).empty? && @options == connection.options
     end
 
@@ -170,7 +174,7 @@ module HTTPX
       end
 
       # if the write buffer is full, we drain it
-      return :w if @write_buffer.full?
+      return :w unless @write_buffer.empty?
 
       return @parser.interests if @parser
 
@@ -251,11 +255,18 @@ module HTTPX
 
     def consume
       catch(:called) do
+        epiped = false
         loop do
           parser.consume
 
-          # we exit if there's no more data to process
-          if @pending.size.zero? && @inflight.zero?
+          # we exit if there's no more requests to process
+          #
+          # this condition takes into account:
+          #
+          # * the number of inflight requests
+          # * the number of pending requests
+          # * whether the write buffer has bytes (i.e. for close handshake)
+          if @pending.size.zero? && @inflight.zero? && @write_buffer.empty?
             log(level: 3) { "NO MORE REQUESTS..." }
             return
           end
@@ -265,9 +276,17 @@ module HTTPX
           read_drained = false
           write_drained = nil
 
-          # dread
+          #
+          # tight read loop.
+          #
+          # read as much of the socket as possible.
+          #
+          # this tight loop reads all the data it can from the socket and pipes it to
+          # its parser.
+          #
           loop do
             siz = @io.read(@window_size, @read_buffer)
+            log(level: 3, color: :cyan) { "IO READ: #{siz} bytes..." }
             unless siz
               ex = EOFError.new("descriptor closed")
               ex.set_backtrace(caller)
@@ -275,27 +294,53 @@ module HTTPX
               return
             end
 
+            # socket has been drained. mark and exit the read loop.
             if siz.zero?
               read_drained = @read_buffer.empty?
+              epiped = false
               break
             end
 
             parser << @read_buffer.to_s
 
+            # continue reading if possible.
+            break if interests == :w && !epiped
+
+            # exit the read loop if connection is preparing to be closed
             break if @state == :closing || @state == :closed
 
-            # for HTTP/2, we just want to write goaway frame
-          end unless @state == :closing
+            # exit #consume altogether if all outstanding requests have been dealt with
+            return if @pending.size.zero? && @inflight.zero?
+          end unless (interests == :w || @state == :closing) && !epiped
 
-          # dwrite
+          #
+          # tight write loop.
+          #
+          # flush as many bytes as the sockets allow.
+          #
           loop do
+            # buffer has been drainned, mark and exit the write loop.
             if @write_buffer.empty?
               # we only mark as drained on the first loop
               write_drained = write_drained.nil? && @inflight.positive?
+
               break
             end
 
-            siz = @io.write(@write_buffer)
+            begin
+              siz = @io.write(@write_buffer)
+            rescue Errno::EPIPE
+              # this can happen if we still have bytes in the buffer to send to the server, but
+              # the server wants to respond immediately with some message, or an error. An example is
+              # when one's uploading a big file to an unintended endpoint, and the server stops the
+              # consumption, and responds immediately with an authorization of even method not allowed error.
+              # at this point, we have to let the connection switch to read-mode.
+              log(level: 2) { "pipe broken, could not flush buffer..." }
+              epiped = true
+              read_drained = false
+              break
+            end
+            log(level: 3, color: :cyan) { "IO WRITE: #{siz} bytes..." }
             unless siz
               ex = EOFError.new("descriptor closed")
               ex.set_backtrace(caller)
@@ -303,21 +348,28 @@ module HTTPX
               return
             end
 
+            # socket closed for writing. mark and exit the write loop.
             if siz.zero?
               write_drained = !@write_buffer.empty?
               break
             end
 
-            break if @state == :closing || @state == :closed
+            # exit write loop if marked to consume from peer, or is closing.
+            break if interests == :r || @state == :closing || @state == :closed
 
             write_drained = false
-          end
+          end unless interests == :r
 
           # return if socket is drained
-          if read_drained && write_drained
-            log(level: 3) { "WAITING FOR EVENTS..." }
-            return
-          end
+          next unless (interests != :r || read_drained) &&
+                      (interests != :w || write_drained)
+
+          # gotta go back to the event loop. It happens when:
+          #
+          # * the socket is drained of bytes or it's not the interest of the conn to read;
+          # * theres nothing more to write, or it's not in the interest of the conn to write;
+          log(level: 3) { "(#{interests}): WAITING FOR EVENTS..." }
+          return
         end
       end
     end
@@ -424,33 +476,33 @@ module HTTPX
           remove_instance_variable(:@total_timeout)
         end
 
-        @io.close if @io
-        @read_buffer.clear
-        if @keep_alive_timer
-          @keep_alive_timer.cancel
-          remove_instance_variable(:@keep_alive_timer)
-        end
-
-        remove_instance_variable(:@timeout) if defined?(@timeout)
+        purge_after_closed
       when :already_open
         nextstate = :open
         send_pending
       end
       @state = nextstate
-    rescue Errno::EHOSTUNREACH
-      # at this point, all addresses from the IO object have failed
-      reset
-      emit(:unreachable)
-      throw(:jump_tick)
     rescue Errno::ECONNREFUSED,
            Errno::EADDRNOTAVAIL,
-           OpenSSL::SSL::SSLError => e
+           Errno::EHOSTUNREACH,
+           TLSError => e
       # connect errors, exit gracefully
       handle_error(e)
       @state = :closed
       emit(:close)
     end
 
+    def purge_after_closed
+      @io.close if @io
+      @read_buffer.clear
+      if @keep_alive_timer
+        @keep_alive_timer.cancel
+        remove_instance_variable(:@keep_alive_timer)
+      end
+
+      remove_instance_variable(:@timeout) if defined?(@timeout)
+    end
+
     def handle_response
       @inflight -= 1
       return unless @inflight.zero?

data/lib/httpx/connection/http1.rb

@@ -10,7 +10,7 @@ module HTTPX
     MAX_REQUESTS = 100
     CRLF = "\r\n"
 
-    attr_reader :pending
+    attr_reader :pending, :requests
 
     def initialize(buffer, options)
       @options = Options.new(options)
@@ -74,9 +74,10 @@ module HTTPX
     end
 
     def consume
-      requests_limit = [@max_concurrent_requests, @max_requests, @requests.size].min
+      requests_limit = [@max_requests, @requests.size].min
+      concurrent_requests_limit = [@max_concurrent_requests, requests_limit].min
       @requests.each_with_index do |request, idx|
-        break if idx >= requests_limit
+        break if idx >= concurrent_requests_limit
         next if request.state == :done
 
         request.headers["connection"] ||= request.options.persistent || idx < requests_limit - 1 ? "keep-alive" : "close"
@@ -115,7 +116,7 @@ module HTTPX
       response = @request.response
       log(level: 2) { "trailer headers received" }
 
-      log(color: :yellow) { h.each.map { |f, v| "-> HEADER: #{f}: #{v}" }.join("\n") }
+      log(color: :yellow) { h.each.map { |f, v| "-> HEADER: #{f}: #{v.join(", ")}" }.join("\n") }
       response.merge_headers(h)
     end
 
@@ -161,6 +162,16 @@ module HTTPX
     end
 
     def handle_error(ex)
+      if (ex.is_a?(EOFError) || ex.is_a?(TimeoutError)) && @request && @request.response &&
+         !@request.response.headers.key?("content-length") &&
+         !@request.response.headers.key?("transfer-encoding")
+        # if the response does not contain a content-length header, the server closing the
+        # connnection is the indicator of response consumed.
+        # https://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.4.4
+        catch(:called) { on_complete }
+        return
+      end
+
       if @pipelining
         disable
       else
@@ -224,6 +235,8 @@ module HTTPX
 
     def disable_pipelining
       return if @requests.empty?
+      # do not disable pipelining if already set to 1 request at a time
+      return if @max_concurrent_requests == 1
 
       @requests.each do |r|
         r.transition(:idle)
@@ -240,7 +253,7 @@ module HTTPX
       @pipelining = false
     end
 
-    def set_request_headers(request)
+    def set_protocol_headers(request)
       request.headers["host"] ||= request.authority
       request.headers["connection"] ||= request.options.persistent ? "keep-alive" : "close"
       if !request.headers.key?("content-length") &&
@@ -254,7 +267,6 @@ module HTTPX
     end
 
     def handle(request)
-      set_request_headers(request)
       catch(:buffer_full) do
         request.transition(:headers)
         join_headers(request) if request.state == :headers
@@ -270,6 +282,7 @@ module HTTPX
       log(color: :yellow) { "<- HEADLINE: #{buffer.chomp.inspect}" }
       @buffer << buffer
       buffer.clear
+      set_protocol_headers(request)
       request.headers.each do |field, value|
         buffer << "#{capitalized(field)}: #{value}" << CRLF
         log(color: :yellow) { "<- HEADER: #{buffer.chomp}" }