httpx 0.11.3 → 0.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e1612c9f4696a7a6ae0508ea66f864654e28470e742b91b57cd14ab5b43adad2
-  data.tar.gz: 791569d7282f7cb3e451d245e332871c170e4571ca5e10ead757f93afffe3d78
+  metadata.gz: 5770fadc8d4604ccb0fb274c7fc157802315f68c749a83973ee5596fee8effb1
+  data.tar.gz: 4f24cca053093be31636405dcb740f5c6b2599197d9ebdfb90cbf127e86a5ffc
 SHA512:
-  metadata.gz: 96e8bc5d07f59b21ccdbc54bb11bce9b1deb53eb128c4d6c2652a0dfca6c925cbe084e82f352857004420cc5753fbacc3a68e2bc4d885937a2afbfa5c6138836
-  data.tar.gz: b74cbac2346d110a97e6ab9a13f2a62db7ed23ce1454df510f9c8cb6ea58314edff2724b9cc269cdf8d5fdf00e9ce6c5c28eff864605f6ce01a9e2f4b9078911
+  metadata.gz: 48fe64207a82e2b8db91b73cc6252ff48ea624e57dcbcbfa30e5f6986e1936e61bbdc83cece34c470dd6e318f41845a306a57e2b5c8710084444b6193786a1eb
+  data.tar.gz: 70b43fc624a8452187a730e39f55d9e92d16438babbd76aa2346c5b3dd0c2cad7f8e5e787f7c4ab1917c52ed750c7656f2f6c5e92a2ec0fce9cc0a76e5994d1e

data/README.md

@@ -113,7 +113,7 @@ The test suite runs against [httpbin proxied over nghttp2](https://nghttp2.org/h
 
 ## Supported Rubies
 
-All Rubies greater or equal to 2.1, and always latest JRuby.
+All Rubies greater or equal to 2.1, and always latest JRuby and Truffleruby.
 
 **Note**: This gem is tested against all latest patch versions, i.e. if you're using 2.2.0 and you experience some issue, please test it against 2.2.10 (latest patch version of 2.2) before creating an issue.
 
@@ -133,7 +133,7 @@ All Rubies greater or equal to 2.1, and always latest JRuby.
 
 If your requirement is to run requests over HTTP/2 and TLS, make sure you run a version of the gem which compiles OpenSSL 1.0.2 (Ruby 2.3 and higher are guaranteed to).
 
-JRuby's `openssl` is based on Bouncy Castle, which is massively outdated and still doesn't implement ALPN. So HTTP/2 over TLS/ALPN negotiation is off until JRuby figures this out.
+In order to use HTTP/2 under JRuby, [check this link](https://gitlab.com/honeyryderchuck/httpx/-/wikis/JRuby-Truffleruby-Other-Rubies) to know what to do.
 
 ### Known bugs
 

data/doc/release_notes/0_11_1.md

@@ -1 +1,5 @@
-0_11_1.md
+# 0.11.1
+
+## Bugfixes
+
+Fixed a bug related to the `:compression` plugin trying to process a last empty DATA frame from an HTTP/2 response, after it had been closed.

data/doc/release_notes/0_12_0.md

@@ -0,0 +1,55 @@
+# 0.12.0
+
+## Features
+
+### AWS Sigv4 Authentication Plugin
+
+A new plugin, `:aws_sigv4`, is now shipped with `httpx`. It implements the [AWS Signature Version 4 request signing process](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html), a well documented way of authenticating requests to AWS services, which has since been adopted by other cloud providers, such as Google Cloud Storage.
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#sessionaws_sigv4_authentication
+
+For convenience, there's a derivative plugin, `:aws_sdk_authentication`, which builds on top of `:aws_sigv4`, and integrates with the `aws-sdk-core` gem, maintained by AWS, to resolve the authentication credentials (p.ex. if you support ephemeral access keys).
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#sessionaws_sdk_authentication
+
+Other FAQ: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#faqs
+
+### HTTP/2 support for JRuby
+
+`jruby-openssl` doesn't support ALPN protocol negotiation, nor are there plans to implement, which limited the seamless HTTP/2 usage in `httpx`. A new connection adapter was therefore added specifically for JRuby, where ssl/tls connections will be handled using ffi-based openssl bindings, provided you bundle `ffi-compiler` and `concurrent-ruby`, and install a TLS/1.2-compatible `openssl` package.
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/JRuby-Truffleruby-Other-Rubies#http2
+
+## Improvements
+
+
+### truffleruby support
+
+`httpx` supports and tests against `truffleruby` (known to run tests since v20.3, passing all tests since v21).
+
+### Performance
+
+Several optimizations were introduced:
+
+* Reduction in read/write system calls;
+* more usage of `String#byteslice` in parsing (instead of string mutation);
+* Avoid selection on connections with no outstanding requests;
+
+They all contributed to a massive performance improvement, itself reflected in test runs, which need half the time they used to to complete.
+
+### APIs
+
+* `HTTPX::ErrorResponse#to_s` now uses the exception full message, instead of just the backtrace.
+
+## Bugfixes
+
+* HTTP/2 stream protocol errors do not cause the process to hang (instead, error responnses are yielded);
+* Fixed body stream bugs on retries when error causing retry would happen mid-transfer;
+* Fixed `:multipart` plugin body rewind on retries to start the transfer from the beginning;
+* Fixed auto-load of  `:proxy` plugin when `HTTPS_PROXY` or `HTTP_PROXY` is set;
+* Errno::EPIPE errors mid transfer now cause `httpx` to read from the server and get the appropriate HTTP error response;
+* Make sure that all requests have an error responnse if the error happens early;
+* Fixed TCP handshake Errno::INPROGRESS handling inside TLS connnections, which was causing the process to hang in a high handshake contention scenario;
+* Do not call the event loop if there's nothing to listen on (the DoH resolver was being listened on even if there was nothing to be request);
+* Fixed double event registry for DoH resolvers;
+* 

data/lib/httpx.rb

@@ -19,7 +19,6 @@ require "httpx/headers"
 require "httpx/request"
 require "httpx/response"
 require "httpx/chainable"
-require "httpx/session"
 
 # Top-Level Namespace
 #
@@ -59,3 +58,5 @@ module HTTPX
 
   extend Chainable
 end
+
+require "httpx/session"

data/lib/httpx/adapters/faraday.rb

@@ -123,11 +123,9 @@ module Faraday
           end
 
           def method_missing(meth, *args, &blk)
-            if @env && @env.respond_to?(meth)
-              @env.__send__(meth, *args, &blk)
-            else
-              super
-            end
+            return super unless @env && @env.respond_to?(meth)
+
+            @env.__send__(meth, *args, &blk)
           end
         end
 
@@ -197,7 +195,7 @@ module Faraday
           response_headers.merge!(response.headers)
         end
         @app.call(env)
-      rescue OpenSSL::SSL::SSLError => e
+      rescue ::HTTPX::TLSError => e
         raise SSL_ERROR, e
       rescue Errno::ECONNABORTED,
              Errno::ECONNREFUSED,

data/lib/httpx/altsvc.rb

@@ -37,6 +37,7 @@ module HTTPX
     end
 
     def emit(request, response)
+      return unless response.respond_to?(:headers)
       # Alt-Svc
       return unless response.headers.key?("alt-svc")
 

data/lib/httpx/connection.rb

@@ -170,7 +170,7 @@ module HTTPX
       end
 
       # if the write buffer is full, we drain it
-      return :w if @write_buffer.full?
+      return :w unless @write_buffer.empty?
 
       return @parser.interests if @parser
 
@@ -251,11 +251,18 @@ module HTTPX
 
     def consume
       catch(:called) do
+        epiped = false
         loop do
           parser.consume
 
-          # we exit if there's no more data to process
-          if @pending.size.zero? && @inflight.zero?
+          # we exit if there's no more requests to process
+          #
+          # this condition takes into account:
+          #
+          # * the number of inflight requests
+          # * the number of pending requests
+          # * whether the write buffer has bytes (i.e. for close handshake)
+          if @pending.size.zero? && @inflight.zero? && @write_buffer.empty?
             log(level: 3) { "NO MORE REQUESTS..." }
             return
           end
@@ -265,9 +272,17 @@ module HTTPX
           read_drained = false
           write_drained = nil
 
-          # dread
+          #
+          # tight read loop.
+          #
+          # read as much of the socket as possible.
+          #
+          # this tight loop reads all the data it can from the socket and pipes it to
+          # its parser.
+          #
           loop do
             siz = @io.read(@window_size, @read_buffer)
+            log(level: 3, color: :cyan) { "IO READ: #{siz} bytes..." }
             unless siz
               ex = EOFError.new("descriptor closed")
               ex.set_backtrace(caller)
@@ -275,27 +290,53 @@ module HTTPX
               return
             end
 
+            # socket has been drained. mark and exit the read loop.
             if siz.zero?
               read_drained = @read_buffer.empty?
+              epiped = false
               break
             end
 
             parser << @read_buffer.to_s
 
+            # continue reading if possible.
+            break if interests == :w && !epiped
+
+            # exit the read loop if connection is preparing to be closed
             break if @state == :closing || @state == :closed
 
-            # for HTTP/2, we just want to write goaway frame
-          end unless @state == :closing
+            # exit #consume altogether if all outstanding requests have been dealt with
+            return if @pending.size.zero? && @inflight.zero?
+          end unless (interests == :w || @state == :closing) && !epiped
 
-          # dwrite
+          #
+          # tight write loop.
+          #
+          # flush as many bytes as the sockets allow.
+          #
           loop do
+            # buffer has been drainned, mark and exit the write loop.
             if @write_buffer.empty?
               # we only mark as drained on the first loop
               write_drained = write_drained.nil? && @inflight.positive?
+
               break
             end
 
-            siz = @io.write(@write_buffer)
+            begin
+              siz = @io.write(@write_buffer)
+            rescue Errno::EPIPE
+              # this can happen if we still have bytes in the buffer to send to the server, but
+              # the server wants to respond immediately with some message, or an error. An example is
+              # when one's uploading a big file to an unintended endpoint, and the server stops the
+              # consumption, and responds immediately with an authorization of even method not allowed error.
+              # at this point, we have to let the connection switch to read-mode.
+              log(level: 2) { "pipe broken, could not flush buffer..." }
+              epiped = true
+              read_drained = false
+              break
+            end
+            log(level: 3, color: :cyan) { "IO WRITE: #{siz} bytes..." }
             unless siz
               ex = EOFError.new("descriptor closed")
               ex.set_backtrace(caller)
@@ -303,21 +344,28 @@ module HTTPX
               return
             end
 
+            # socket closed for writing. mark and exit the write loop.
             if siz.zero?
               write_drained = !@write_buffer.empty?
               break
             end
 
-            break if @state == :closing || @state == :closed
+            # exit write loop if marked to consume from peer, or is closing.
+            break if interests == :r || @state == :closing || @state == :closed
 
             write_drained = false
-          end
+          end unless interests == :r
 
           # return if socket is drained
-          if read_drained && write_drained
-            log(level: 3) { "WAITING FOR EVENTS..." }
-            return
-          end
+          next unless (interests != :r || read_drained) &&
+                      (interests != :w || write_drained)
+
+          # gotta go back to the event loop. It happens when:
+          #
+          # * the socket is drained of bytes or it's not the interest of the conn to read;
+          # * theres nothing more to write, or it's not in the interest of the conn to write;
+          log(level: 3) { "(#{interests}): WAITING FOR EVENTS..." }
+          return
         end
       end
     end
@@ -444,7 +492,7 @@ module HTTPX
       throw(:jump_tick)
     rescue Errno::ECONNREFUSED,
            Errno::EADDRNOTAVAIL,
-           OpenSSL::SSL::SSLError => e
+           TLSError => e
       # connect errors, exit gracefully
       handle_error(e)
       @state = :closed

data/lib/httpx/connection/http1.rb

@@ -69,14 +69,16 @@ module HTTPX
 
       return if @requests.include?(request)
 
+      request.once(:headers, &method(:set_protocol_headers))
       @requests << request
       @pipelining = true if @requests.size > 1
     end
 
     def consume
-      requests_limit = [@max_concurrent_requests, @max_requests, @requests.size].min
+      requests_limit = [@max_requests, @requests.size].min
+      concurrent_requests_limit = [@max_concurrent_requests, requests_limit].min
       @requests.each_with_index do |request, idx|
-        break if idx >= requests_limit
+        break if idx >= concurrent_requests_limit
         next if request.state == :done
 
         request.headers["connection"] ||= request.options.persistent || idx < requests_limit - 1 ? "keep-alive" : "close"
@@ -115,7 +117,7 @@ module HTTPX
       response = @request.response
       log(level: 2) { "trailer headers received" }
 
-      log(color: :yellow) { h.each.map { |f, v| "-> HEADER: #{f}: #{v}" }.join("\n") }
+      log(color: :yellow) { h.each.map { |f, v| "-> HEADER: #{f}: #{v.join(", ")}" }.join("\n") }
       response.merge_headers(h)
     end
 
@@ -161,13 +163,13 @@ module HTTPX
     end
 
     def handle_error(ex)
-      if ex.is_a?(EOFError) && @request && @request.response &&
+      if (ex.is_a?(EOFError) || ex.is_a?(TimeoutError)) && @request && @request.response &&
          !@request.response.headers.key?("content-length") &&
          !@request.response.headers.key?("transfer-encoding")
         # if the response does not contain a content-length header, the server closing the
         # connnection is the indicator of response consumed.
         # https://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.4.4
-        on_complete
+        catch(:called) { on_complete }
         return
       end
 
@@ -250,7 +252,7 @@ module HTTPX
       @pipelining = false
     end
 
-    def set_request_headers(request)
+    def set_protocol_headers(request)
       request.headers["host"] ||= request.authority
       request.headers["connection"] ||= request.options.persistent ? "keep-alive" : "close"
       if !request.headers.key?("content-length") &&
@@ -264,7 +266,6 @@ module HTTPX
     end
 
     def handle(request)
-      set_request_headers(request)
       catch(:buffer_full) do
         request.transition(:headers)
         join_headers(request) if request.state == :headers

data/lib/httpx/connection/http2.rb

@@ -42,11 +42,15 @@ module HTTPX
         return @buffer.empty? ? :r : :rw
       end
 
-      return :w unless @pending.empty?
+      return :w if !@pending.empty? && can_buffer_more_requests?
 
       return :w if @streams.each_key.any? { |r| r.interests == :w }
 
-      return :r if @buffer.empty?
+      if @buffer.empty?
+        return if @streams.empty? && @pings.empty?
+
+        return :r
+      end
 
       :rw
     end
@@ -70,10 +74,14 @@ module HTTPX
       @connection << data
     end
 
+    def can_buffer_more_requests?
+      @handshake_completed &&
+        @streams.size < @max_concurrent_requests &&
+        @streams.size < @max_requests
+    end
+
     def send(request)
-      if !@handshake_completed ||
-         @streams.size >= @max_concurrent_requests ||
-         @streams.size >= @max_requests
+      unless can_buffer_more_requests?
         @pending << request
         return
       end
@@ -83,6 +91,7 @@ module HTTPX
         @streams[request] = stream
         @max_requests -= 1
       end
+      request.once(:headers, &method(:set_protocol_headers))
       handle(request, stream)
       true
     rescue HTTP2Next::Error::StreamLimitExceeded
@@ -126,8 +135,6 @@ module HTTPX
       request.path
     end
 
-    def set_request_headers(request); end
-
     def handle(request, stream)
       catch(:buffer_full) do
         request.transition(:headers)
@@ -172,18 +179,18 @@ module HTTPX
       stream.on(:data, &method(:on_stream_data).curry(3)[stream, request])
     end
 
+    def set_protocol_headers(request)
+      request.headers[":scheme"]    = request.scheme
+      request.headers[":method"]    = request.verb.to_s.upcase
+      request.headers[":path"]      = headline_uri(request)
+      request.headers[":authority"] = request.authority
+    end
+
     def join_headers(stream, request)
-      set_request_headers(request)
-      headers = {}
-      headers[":scheme"]    = request.scheme
-      headers[":method"]    = request.verb.to_s.upcase
-      headers[":path"]      = headline_uri(request)
-      headers[":authority"] = request.authority
-      headers = headers.merge(request.headers)
       log(level: 1, color: :yellow) do
-        headers.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{v}" }.join("\n")
+        request.headers.each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{v}" }.join("\n")
       end
-      stream.headers(headers, end_stream: request.empty?)
+      stream.headers(request.headers.each, end_stream: request.empty?)
     end
 
     def join_body(stream, request)
@@ -227,10 +234,15 @@ module HTTPX
     end
 
     def on_stream_close(stream, request, error)
+      log(level: 2) { "#{stream.id}: closing stream" }
+      @drains.delete(request)
+      @streams.delete(request)
+
       if error && error != :no_error
         ex = Error.new(stream.id, error)
         ex.set_backtrace(caller)
-        emit(:error, request, ex)
+        response = ErrorResponse.new(request, ex, request.options)
+        emit(:response, request, response)
       else
         response = request.response
         if response.status == 421
@@ -241,9 +253,6 @@ module HTTPX
           emit(:response, request, response)
         end
       end
-      log(level: 2) { "#{stream.id}: closing stream" }
-
-      @streams.delete(request)
       send(@pending.shift) unless @pending.empty?
       return unless @streams.empty? && exhausted?
 
@@ -328,11 +337,9 @@ module HTTPX
     end
 
     def method_missing(meth, *args, &blk)
-      if @connection.respond_to?(meth)
-        @connection.__send__(meth, *args, &blk)
-      else
-        super
-      end
+      return super unless @connection.respond_to?(meth)
+
+      @connection.__send__(meth, *args, &blk)
     end
   end
   Connection.register "h2", Connection::HTTP2