httpx 0.13.1 → 0.14.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e3859c516c1cdddc64f8105fafd5425da26074582b1590cc87f65401a909e0e2
-  data.tar.gz: 209b5c1fba921d69f9aa2167ccb0f4b6b45a1a28667364eefef8026104004f53
+  metadata.gz: 716a91d8937fe675798bd8ad275cf5944616069149f2ace7b8c04c42c4cfeb19
+  data.tar.gz: cb76bc4343b18a30e1ffb093885e899b7b0cc9c70f40e5b461f2de4d4cce89f6
 SHA512:
-  metadata.gz: 3862b4879291aa944b92c50d232abc6413a170f2bcd501d6f33a1ca55a99a1411dd0d4f2b6a50b92a8b1465fa42b654ef658c15a91154205bfd60bcdc983c5a5
-  data.tar.gz: 6b2f721cc2bb8e0fe7267517dcd146fe50f448676bcec64e9744d5f358232d8d9c364acd6717e0df3a72bfcb1d154a7e4b74252324cf9a248f1fe062242eb4fe
+  metadata.gz: 789cb7389ada8841232580200950c0066e30071ae120323fbb2b4646ae69cf43422c8e181aa02d2820d315c0a4a839a4635d584f4a57218e30f1c66f07cbcb80
+  data.tar.gz: 3815634b93c035a445557b4ae8718b985bbc8ee4776921cba4ee50fdb58dd31769e3f25aaa051f59df8aa96a0ab69a281e5a3ce7b013d762aa258f807afc74b7

data/doc/release_notes/0_10_1.md

@@ -26,7 +26,7 @@ From now on, both headers and the responnse payload will also appear, so expecte
 ## Bugfixes
 
 * HTTP/2 and HTTP/1.1 exhausted connections now get properly migrated into a new connection;
-* HTTP/2 421 responses will now correctly migrate the connection and pendign requests to HTTP/1.1 (a hanging loop was being caused);
+* HTTP/2 421 responses will now correctly migrate the connection and pending requests to HTTP/1.1 (a hanging loop was being caused);
 * HTTP/2 connection failed with a GOAWAY settings timeout will now return error responses (instead of hanging indefinitely);
 * Non-IP proxy name-resolving errors will now move on to the next available proxy in the list (instead of hanging indefinitely);
 * Non-IP DNS resolve errors for `native` and `https` variants will now return the appropriate error response (instead of hanging indefinitely);

data/doc/release_notes/0_13_0.md

@@ -41,7 +41,7 @@ The `:transport_options` are therefore deprecated, and will be moved in a major
 
 ## Improvements
 
-Some internal improvements that allow certainn plugins not to "leak" globally, such as the `:compression` plugin, which used to enable compression for all the `httpx` sessions from the same process. It doesn't anymore.
+Some internal improvements that allow certain plugins not to "leak" globally, such as the `:compression` plugin, which used to enable compression for all the `httpx` sessions from the same process. It doesn't anymore.
 
 Using exceptionless nonblocking connect calls in the supported rubies.
 
@@ -55,4 +55,4 @@ When passing open IO objects for origins (the `:io` option), `httpx` was still t
 
 Fixed usage of `:io` option when passed an "authority/io" hash.
 
-Fixing some issues around trying to connnect to the next available IPAddress when the previous one was unreachable or ETIMEDOUT.
+Fixing some issues around trying to connnect to the next available IPAddress when the previous one was unreachable or ETIMEDOUT.

data/doc/release_notes/0_13_1.md

@@ -2,4 +2,4 @@
 
 ## Bugfixes
 
-Rescue `Errno::EALREADY` on calls to `connect_nonblock(exception: false)` (there are exceptionns after all...).
+Rescue `Errno::EALREADY` on calls to `connect_nonblock(exception: false)` (there are exceptions after all...).

data/doc/release_notes/0_13_2.md

@@ -0,0 +1,9 @@
+# 0.13.1
+
+## Improvements
+
+`UDPSocket#sendmsg_nonblock` is now used in the native resolver.
+
+## Bugfixes
+
+Usage in Windows was buggy, resulting in `Errno::EINVAL` during DNS resolving, when using the native resolver. This was due to a discrepancy between `recvfrom` behaviour in WS Sockets and Linux Sockets. This was fixed by making we the UDP socket never tries to receive before a DNS query has been actually sent.

data/doc/release_notes/0_14_0.md

@@ -0,0 +1,79 @@
+# 0.14.0
+
+## Features
+
+### GRPC plugin
+
+A new plugin, `:grpc`, is now available. This plugin provides a simple DSL to build GRPC services and performing calls using `httpx` under the hood.
+
+Example:
+
+```ruby
+require "httpx"
+
+grpc = HTTPX.plugin(:grpc)
+helloworld_stub = grpc.build_stub("localhost:4545")
+helloworld_svc = helloworld_stub.rpc(:SayHello, HelloRequest, HelloReply)
+result = helloworld_svc.say_hello(HelloRequest.new(name: "Jack")) #=> HelloReply: "Hello Jack"
+```
+
+You can read more about the `:grpc` plugin in the [wiki](https://honeyryderchuck.gitlab.io/httpx/wiki/GRPC).
+
+### :origin
+
+A new `:origin` option is available. You can use it for setting a base URL for subsequent relative paths on that session:
+
+```ruby
+HTTPX.get("/httpbin/get") #=> HTTPX::Error: invalid URI: /httpbin/get
+
+httpbin = HTTPX.with(origin: "https://nghttp2.org")
+httpbin.get("/httpbin/get") #=> #<Response:5420 HTTP/2.0 @status=200 ....
+```
+
+**Note!** The origin is **not** for setting base paths, i.e. if you pass it a relative path, it'll be filtered out in subsequent requests (`HTTPX.with(origin: "https://nghttp2.org/httpbin")` will still use only `"https://nghttp2.org"`).
+
+## Improvements
+
+* setting an unexpected option will now raise an `HTTPX::Error` with an helpful message, instead of a confusing `NoMethodError`:
+
+```ruby
+HTTPX.with(foo: "bar") 
+# before
+#=> NoMethodError
+# after
+#=> HTTPX::Error: unknown option: foo
+
+```
+
+* `HTTPX::Options#def_option` (which can be used for setting custom plugin options) can now be passed a full body string (where the argument is `value`), although it still support the block form. This is the recommended approach, as the block form is based on `define_method`, which would make clients unusable inside ractors.
+
+* Added support for `:wait_for_handshake` under the `http2_settings` option (`false` by default). HTTP/2 connections complete the protocol handshake before requests are sent. When this option is `true`, requests get send in the initial payload, before the HTTP/2 connection is fully acknowledged.
+
+*  441716a5ac0f7707211ebe0048f568cf0b759c3f: The `:stream` plugin has been improved to start streaming the real response as methods are called (instead of a completely separate synchronous one, which is definitely not good):
+
+```ruby
+session = HTTPX.plugin(:stream)
+response = session.get(build_uri("/stream/3"), stream: true)
+
+# before
+response.status # this could block indefinitely, if the request truly streams infinitely.
+
+# after
+response.status # sends the request, and starts streaming the response until status is available.
+response.each {|chunk|...} # and now you can start yielding the chunks...
+```
+
+
+## Bugfixes
+
+* fixed usage of the `:multipart` if `pathname` isn't loaded.
+* fixed HTTP/2 trailers.
+* fixed connection merges with the same origin, which was causing them to be duplicated and breaking further usage. (#125)
+* fixed repeated session callbacks on a connection, by ensure they're set only once.
+* fixed calculation of `content-length` for streaming or chunked compressed requests.
+
+
+## Chore
+
+* using ruby base container images in CI instead.
+* using truffleruby official container image.

data/doc/release_notes/0_14_1.md

@@ -0,0 +1,7 @@
+# 0.14.1
+
+
+## Bugfixes
+
+* fixed: HTTP/2-specific headers were being reused on insecure redirects, thereby creating an invalid request (#128);
+* fixed: multipart request parts weren't using explicity set `:content_type`, instead using file mime type or "text/plain";

data/doc/release_notes/0_14_2.md

@@ -0,0 +1,6 @@
+# 0.14.2
+
+
+## Bugfixes
+
+* fixed: multipart request parts weren't using explicity set `:filename`.

data/doc/release_notes/0_14_3.md

@@ -0,0 +1,5 @@
+# 0.14.3
+
+## Bugfixes
+
+* fixed: HTTP/1 "connection: close" header was "leaking" into subsequent redirect follow, including HTTP/2 requests which would fail due to the invalid header.

data/lib/httpx.rb

@@ -12,12 +12,11 @@ require "httpx/callbacks"
 require "httpx/loggable"
 require "httpx/registry"
 require "httpx/transcoder"
-require "httpx/options"
-require "httpx/timeout"
 require "httpx/pool"
 require "httpx/headers"
 require "httpx/request"
 require "httpx/response"
+require "httpx/options"
 require "httpx/chainable"
 
 # Top-Level Namespace

data/lib/httpx/callbacks.rb

@@ -6,19 +6,28 @@ module HTTPX
       callbacks(type) << action
     end
 
-    def once(event, &block)
-      on(event) do |*args, &callback|
+    def once(type, &block)
+      on(type) do |*args, &callback|
         block.call(*args, &callback)
         :delete
       end
     end
 
+    def only(type, &block)
+      callbacks(type).clear
+      on(type, &block)
+    end
+
     def emit(type, *args)
-      callbacks(type).delete_if { |pr| pr[*args] == :delete }
+      callbacks(type).delete_if { |pr| :delete == pr[*args] } # rubocop:disable Style/YodaCondition
     end
 
     protected
 
+    def callbacks_for?(type)
+      @callbacks.key?(type) && !@callbacks[type].empty?
+    end
+
     def callbacks(type = nil)
       return @callbacks unless type
 

data/lib/httpx/connection.rb

@@ -69,7 +69,7 @@ module HTTPX
       end
 
       @inflight = 0
-      @keep_alive_timeout = options.timeout.keep_alive_timeout
+      @keep_alive_timeout = options.timeout[:keep_alive_timeout]
       @keep_alive_timer = nil
 
       self.addresses = options.addresses if options.addresses
@@ -129,7 +129,7 @@ module HTTPX
     end
 
     def merge(connection)
-      @origins += connection.instance_variable_get(:@origins)
+      @origins |= connection.instance_variable_get(:@origins)
       connection.purge_pending do |req|
         send(req)
       end
@@ -238,9 +238,9 @@ module HTTPX
     def timeout
       return @timeout if defined?(@timeout)
 
-      return @options.timeout.connect_timeout if @state == :idle
+      return @options.timeout[:connect_timeout] if @state == :idle
 
-      @options.timeout.operation_timeout
+      @options.timeout[:operation_timeout]
     end
 
     private
@@ -413,7 +413,7 @@ module HTTPX
         emit(:exhausted)
       end
       parser.on(:origin) do |origin|
-        @origins << origin
+        @origins |= [origin]
       end
       parser.on(:close) do |force|
         transition(:closing)
@@ -443,6 +443,7 @@ module HTTPX
           emit(:misdirected, request)
         else
           response = ErrorResponse.new(request, ex, @options)
+          request.response = response
           request.emit(:response, response)
         end
       end
@@ -451,7 +452,7 @@ module HTTPX
     def transition(nextstate)
       case nextstate
       when :idle
-        @timeout = @current_timeout = @options.timeout.connect_timeout
+        @timeout = @current_timeout = @options.timeout[:connect_timeout]
 
       when :open
         return if @state == :closed
@@ -463,7 +464,7 @@ module HTTPX
 
         send_pending
 
-        @timeout = @current_timeout = @options.timeout.operation_timeout
+        @timeout = @current_timeout = @options.timeout[:operation_timeout]
         emit(:open)
       when :closing
         return unless @state == :open
@@ -542,12 +543,14 @@ module HTTPX
     def handle_error(error)
       parser.handle_error(error) if @parser && parser.respond_to?(:handle_error)
       while (request = @pending.shift)
-        request.emit(:response, ErrorResponse.new(request, error, @options))
+        response = ErrorResponse.new(request, error, @options)
+        request.response = response
+        request.emit(:response, response)
       end
     end
 
     def total_timeout
-      total = @options.timeout.total_timeout
+      total = @options.timeout[:total_timeout]
 
       return unless total
 

data/lib/httpx/connection/http1.rb

@@ -80,7 +80,6 @@ module HTTPX
         break if idx >= concurrent_requests_limit
         next if request.state == :done
 
-        request.headers["connection"] ||= request.options.persistent || idx < requests_limit - 1 ? "keep-alive" : "close"
         handle(request)
       end
     end
@@ -254,12 +253,21 @@ module HTTPX
     end
 
     def set_protocol_headers(request)
-      request.headers["host"] ||= request.authority
-      request.headers["connection"] ||= request.options.persistent ? "keep-alive" : "close"
       if !request.headers.key?("content-length") &&
          request.body.bytesize == Float::INFINITY
         request.chunk!
       end
+
+      connection = if request.options.persistent || request != @requests[-1]
+        "keep-alive"
+      else
+        "close"
+      end
+
+      {
+        "host" => (request.headers["host"] || request.authority),
+        "connection" => connection,
+      }
     end
 
     def headline_uri(request)
@@ -272,23 +280,18 @@ module HTTPX
         join_headers(request) if request.state == :headers
         request.transition(:body)
         join_body(request) if request.state == :body
+        request.transition(:trailers)
+        # HTTP/1.1 trailers should only work for chunked encoding
+        join_trailers(request) if request.body.chunked? && request.state == :trailers
         request.transition(:done)
       end
     end
 
     def join_headers(request)
-      buffer = +""
-      buffer << "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}" << CRLF
-      log(color: :yellow) { "<- HEADLINE: #{buffer.chomp.inspect}" }
-      @buffer << buffer
-      buffer.clear
-      set_protocol_headers(request)
-      request.headers.each do |field, value|
-        buffer << "#{capitalized(field)}: #{value}" << CRLF
-        log(color: :yellow) { "<- HEADER: #{buffer.chomp}" }
-        @buffer << buffer
-        buffer.clear
-      end
+      @buffer << "#{request.verb.to_s.upcase} #{headline_uri(request)} HTTP/#{@version.join(".")}" << CRLF
+      log(color: :yellow) { "<- HEADLINE: #{@buffer.to_s.chomp.inspect}" }
+      extra_headers = set_protocol_headers(request)
+      join_headers2(request.headers.each(extra_headers))
       log { "<- " }
       @buffer << CRLF
     end
@@ -302,6 +305,26 @@ module HTTPX
         @buffer << chunk
         throw(:buffer_full, request) if @buffer.full?
       end
+
+      raise request.drain_error if request.drain_error
+    end
+
+    def join_trailers(request)
+      return unless request.trailers? && request.callbacks_for?(:trailers)
+
+      join_headers2(request.trailers)
+      log { "<- " }
+      @buffer << CRLF
+    end
+
+    def join_headers2(headers)
+      buffer = "".b
+      headers.each do |field, value|
+        buffer << "#{capitalized(field)}: #{value}" << CRLF
+        log(color: :yellow) { "<- HEADER: #{buffer.chomp}" }
+        @buffer << buffer
+        buffer.clear
+      end
     end
 
     UPCASED = {

data/lib/httpx/connection/http2.rb

@@ -21,14 +21,16 @@ module HTTPX
 
     def initialize(buffer, options)
       @options = Options.new(options)
-      @max_concurrent_requests = @options.max_concurrent_requests || MAX_CONCURRENT_REQUESTS
-      @max_requests = @options.max_requests || 0
+      @settings = @options.http2_settings
       @pending = []
       @streams = {}
       @drains  = {}
       @pings = []
       @buffer = buffer
       @handshake_completed = false
+      @wait_for_handshake = @settings.key?(:wait_for_handshake) ? @settings.delete(:wait_for_handshake) : true
+      @max_concurrent_requests = @options.max_concurrent_requests || MAX_CONCURRENT_REQUESTS
+      @max_requests = @options.max_requests || 0
       init_connection
     end
 
@@ -36,7 +38,11 @@ module HTTPX
       # waiting for WINDOW_UPDATE frames
       return :r if @buffer.full?
 
-      return :w if @connection.state == :closed
+      if @connection.state == :closed
+        return unless @handshake_completed
+
+        return :w
+      end
 
       unless (@connection.state == :connected && @handshake_completed)
         return @buffer.empty? ? :r : :rw
@@ -75,9 +81,13 @@ module HTTPX
     end
 
     def can_buffer_more_requests?
-      @handshake_completed &&
+      if @handshake_completed
         @streams.size < @max_concurrent_requests &&
-        @streams.size < @max_requests
+          @streams.size < @max_requests
+      else
+        !@wait_for_handshake &&
+          @streams.size < @max_concurrent_requests
+      end
     end
 
     def send(request)
@@ -140,12 +150,14 @@ module HTTPX
         join_headers(stream, request) if request.state == :headers
         request.transition(:body)
         join_body(stream, request) if request.state == :body
+        request.transition(:trailers)
+        join_trailers(stream, request) if request.state == :trailers && !request.body.empty?
         request.transition(:done)
       end
     end
 
     def init_connection
-      @connection = HTTP2Next::Client.new(@options.http2_settings)
+      @connection = HTTP2Next::Client.new(@settings)
       @connection.max_streams = @max_requests if @connection.respond_to?(:max_streams=) && @max_requests.positive?
       @connection.on(:frame, &method(:on_frame))
       @connection.on(:frame_sent, &method(:on_frame_sent))
@@ -169,6 +181,7 @@ module HTTPX
     public :reset
 
     def handle_stream(stream, request)
+      request.on(:refuse, &method(:on_stream_refuse).curry(3)[stream, request])
       stream.on(:close, &method(:on_stream_close).curry(3)[stream, request])
       stream.on(:half_close) do
         log(level: 2) { "#{stream.id}: waiting for response..." }
@@ -179,18 +192,32 @@ module HTTPX
     end
 
     def set_protocol_headers(request)
-      request.headers[":scheme"]    = request.scheme
-      request.headers[":method"]    = request.verb.to_s.upcase
-      request.headers[":path"]      = headline_uri(request)
-      request.headers[":authority"] = request.authority
+      {
+        ":scheme" => request.scheme,
+        ":method" => request.verb.to_s.upcase,
+        ":path" => headline_uri(request),
+        ":authority" => request.authority,
+      }
     end
 
     def join_headers(stream, request)
-      set_protocol_headers(request)
+      extra_headers = set_protocol_headers(request)
+      log(level: 1, color: :yellow) do
+        request.headers.merge(extra_headers).each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{v}" }.join("\n")
+      end
+      stream.headers(request.headers.each(extra_headers), end_stream: request.empty?)
+    end
+
+    def join_trailers(stream, request)
+      unless request.trailers?
+        stream.data("", end_stream: true) if request.callbacks_for?(:trailers)
+        return
+      end
+
       log(level: 1, color: :yellow) do
-        request.headers.each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{v}" }.join("\n")
+        request.trailers.each.map { |k, v| "#{stream.id}: -> HEADER: #{k}: #{v}" }.join("\n")
       end
-      stream.headers(request.headers.each, end_stream: request.empty?)
+      stream.headers(request.trailers.each, end_stream: true)
     end
 
     def join_body(stream, request)
@@ -201,13 +228,15 @@ module HTTPX
         next_chunk = request.drain_body
         log(level: 1, color: :green) { "#{stream.id}: -> DATA: #{chunk.bytesize} bytes..." }
         log(level: 2, color: :green) { "#{stream.id}: -> #{chunk.inspect}" }
-        stream.data(chunk, end_stream: !next_chunk)
-        if next_chunk && @buffer.full?
+        stream.data(chunk, end_stream: !(next_chunk || request.trailers? || request.callbacks_for?(:trailers)))
+        if next_chunk && (@buffer.full? || request.body.unbounded_body?)
           @drains[request] = next_chunk
           throw(:buffer_full)
         end
         chunk = next_chunk
       end
+
+      on_stream_refuse(stream, request, request.drain_error) if request.drain_error
     end
 
     ######
@@ -215,6 +244,11 @@ module HTTPX
     ######
 
     def on_stream_headers(stream, request, h)
+      if request.response && request.response.version == "2.0"
+        on_stream_trailers(stream, request, h)
+        return
+      end
+
       log(color: :yellow) do
         h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{v}" }.join("\n")
       end
@@ -227,12 +261,24 @@ module HTTPX
       handle(request, stream) if request.expects?
     end
 
+    def on_stream_trailers(stream, request, h)
+      log(color: :yellow) do
+        h.map { |k, v| "#{stream.id}: <- HEADER: #{k}: #{v}" }.join("\n")
+      end
+      request.response.merge_headers(h)
+    end
+
     def on_stream_data(stream, request, data)
       log(level: 1, color: :green) { "#{stream.id}: <- DATA: #{data.bytesize} bytes..." }
       log(level: 2, color: :green) { "#{stream.id}: <- #{data.inspect}" }
       request.response << data
     end
 
+    def on_stream_refuse(stream, request, error)
+      stream.close
+      on_stream_close(stream, request, error)
+    end
+
     def on_stream_close(stream, request, error)
       log(level: 2) { "#{stream.id}: closing stream" }
       @drains.delete(request)