httpx 0.7.0 → 0.10.0

data/doc/release_notes/0_6_6.md

@@ -0,0 +1,19 @@
+# 0.6.6
+
+## Features
+
+* The `retries` plugin receives two new options:
+  * `retry_on`: a callable that receives the failed response as an argument; the return value will determine whether there'll be a retried request.
+  * `retry_after`: time (in seconds) after which there request will be retried. Can be an integer or a callable that receives the request and returns an integer (one can do exponential back-off like that, for example).
+* Added support for DNS-over-HTTPS GET requests as per the latest spec.
+
+## Improvements
+
+* `HTTPX.plugins` got deprecated; basically, it's great until you have to pass options to a plugin, and then it just works (not). The recommended way to load multiple plugins is `HTTPX.plugin(...).plugin(...)`.
+
+
+## Bugfixes
+
+* fixed a proxy bug where an `Alt-Svc` response header would make the client try to connect. Just like connection coalescing and the ORIGIN frame, it ignores it when going through a proxy.  
+
+

data/doc/release_notes/0_6_7.md

@@ -0,0 +1,5 @@
+# 0.6.7
+
+## Bugfixes
+
+* An error was reported when using the follow plugin allowing insecure redirects: if the insecure redirect would be for the same host, and the original request was performed with HTTP/2, the library would try to coalesce the request, and blocks the reactor. A check was made to ensure that connection only coalesces if both are https connections.

data/doc/release_notes/0_7_0.md

@@ -0,0 +1,46 @@
+# 0.7.0
+
+
+## Features
+
+New option: `:max_requests`. This is a connection-level option signalizing how many requests can be performed on a connection. Although the HTTP/1 parser defined this well, in HTTP/2 this wasn't very clear, so: by definition, the remote MAX_CONCURRENT_STREAMS setting will be used to define it, unless the user explicitly passed the option. You can also pass `:max_requests => Float::INFINITY` if you know that the server allows more requests than that on a connection.
+
+New plugin: `:expect`. 
+
+Although there was support for `expect: 100-continue` header already when passed, this plugin can:
+
+* automatically set the header on requests with body;
+* execute the flow;
+* recover from 417 status errors (i.e. try again without it);
+* send body after X seconds if no 100 response came;
+
+Suport for `with_` methods for the session. As long as the suffix is a valid attribute, it's just like that:
+
+```ruby
+HTTPX.with_timeout(...).with_ssl(...)
+# same as:
+# HTTPX.with(timeout: ..., ssl: ...)
+```
+
+## Improvements
+
+### Connections
+
+The following improvements make the `persistent` plugin way more resilient:
+
+* Better balancing of HTTP/2 connections by distributing requests among X connections depending of how many requests they can process.
+* Exhausted connections can off-load to a new same-origin connection (such as, when the server negotiates less `MAX_CONCURRENT_STREAMS` than what's expected).
+
+### Timeouts
+
+(Timeouts will be one of the main improvements from the 0.7.x series)
+
+`:total_timeout` is now a connection-level directive, which means that this feature will actually make more sense and account for all requests in a block at the same time, instead of one-by-one.
+
+### Options
+
+Option setters were being bypassed, therefore a lot of the type-checks defined there weren't effectively being picked upon, which could have led to weird user errors.
+
+## Bugfixes
+
+* fixed the `push_promise` plugin integration (wasn't working well since `http-2-next` was adopted);

data/doc/release_notes/0_8_0.md

@@ -0,0 +1,27 @@
+# 0.8.0
+
+
+## Features
+
+* `keep_alive_timeout`: for persistent connections, the keep alive timeout will set the connection to be closed if not reused for a request **after** the last received response;
+
+## Improvements
+
+* using `max_requests` for HTTP/1 pipelining as well;
+* `retries` plugin now works with plain HTTP responses (not just error responses);
+* reduced the number of string allocations from log labels;
+* performance: a lot of improvements were made to optimize the "waiting for IO events" phase, which dramatically reduced the CPU usage and make the performance of the library more on-par with other ruby HTTP gems for the 1-shot request scenario.
+
+
+## Bugfixes
+
+* fixed `HTTPX::Response#copy_to`;
+* fixed `compression` plugin not properly compressing request bodies using `gzip`;
+* fixed `compression` plugin not handling `content-encoding: identity` payloads;
+* do not overwrite user-defined `max_requests`on HTTP2 connection handshake;
+* `retries` plugin: connection was blocking when a request with body was retried;
+* `alt-svc: clear` response header was causing the process to hang;
+
+## Tests
+
+* Code coverage improved to 91%;

data/doc/release_notes/0_8_1.md

@@ -0,0 +1,8 @@
+# 0.8.1
+
+
+## Bugfixes
+
+* fixing HTTP/2 handshake IO interests calculation;
+* fixed the double ctrl+f issue when terminating an ongoing HTTP/2 request;
+* fixed connection comparison when passing headers;

data/doc/release_notes/0_8_2.md

@@ -0,0 +1,7 @@
+# 0.8.2
+
+## Features
+
+* `:expect` plugin now supports a new option, `:expect_threshold_size`, meaning: the byte size threshold below which no `expect` header will be sent with requests with payload.
+* `:compression` plugin now supports a new option, `:compression_threshold_size`, meaning: the bytesize threshold below which request payload won't be compressed before being sent.
+* for HTTP/2 connections, when `keep_alive_timeout` expires, a `PING` frame is used to check connection availability; if successful, the connection will be reused.

data/doc/release_notes/0_9_0.md

@@ -0,0 +1,38 @@
+# 0.9.0
+
+## Features
+
+### Multiple requests with specific options
+
+You can now pass a third element to the "request element" of an array to `.request`.
+
+```ruby
+requests = [
+  [:post, "https://url/post", { form: { foo: "bar" } }],
+  [:post, "https://url/post", { form: { foo: "bar2" } }]
+]
+HTTPX.request(requests)
+# or, if you want to pass options common to all requests
+HTTPX.request(requests, max_concurrent_requests: 1)
+```
+
+
+### HTTPX::Session#build_request
+
+`HTTPX::Session::build_request` is now public API from a session. You can now build requests before you send them. These request objects are still considered somewhat "internal", so consider them immutable and **do not rely on its API**. Just pass them forward.
+
+Note: this API is only available for instantiated session, so there is no `HTTPX.build_request`.
+
+
+```ruby
+
+HTTPX.wrap do |http|
+  requests = [
+    http.build_request(:post, "https://url/post", { form: { foo: "bar" } }),
+    http.build_request(:post, "https://url/post", { form: { foo: "bar2" } })
+  ]
+  http.request(requests)
+  # or, if you want to pass options common to all requests
+  http.request(requests, max_concurrent_requests: 1)
+end
+```

data/lib/httpx.rb

@@ -5,6 +5,8 @@ require "httpx/version"
 require "httpx/extensions"
 
 require "httpx/errors"
+require "httpx/utils"
+require "httpx/domain_name"
 require "httpx/altsvc"
 require "httpx/callbacks"
 require "httpx/loggable"

data/lib/httpx/adapters/faraday.rb

@@ -121,7 +121,7 @@ module Faraday
           end
 
           def respond_to_missing?(meth)
-            @env.respond_to?(meth)
+            @env.respond_to?(meth) || super
           end
 
           def method_missing(meth, *args, &blk)

data/lib/httpx/altsvc.rb

@@ -42,7 +42,23 @@ module HTTPX
 
       origin = request.origin
       host = request.uri.host
-      parse(response.headers["alt-svc"]) do |alt_origin, alt_params|
+
+      altsvc = response.headers["alt-svc"]
+
+      # https://tools.ietf.org/html/rfc7838#section-3
+      # A field value containing the special value "clear" indicates that the
+      # origin requests all alternatives for that origin to be invalidated
+      # (including those specified in the same response, in case of an
+      # invalid reply containing both "clear" and alternative services).
+      if altsvc == "clear"
+        @altsvc_mutex.synchronize do
+          @altsvcs[origin].clear
+        end
+
+        return
+      end
+
+      parse(altsvc) do |alt_origin, alt_params|
         alt_origin.host ||= host
         yield(alt_origin, origin, alt_params)
       end
@@ -73,7 +89,7 @@ module HTTPX
         alt_proto, alt_origin = alt_origin.split("=")
         alt_origin = alt_origin[1..-2] if alt_origin.start_with?("\"") && alt_origin.end_with?("\"")
         if alt_origin.start_with?(":")
-          alt_origin = "dummy#{alt_origin}"
+          alt_origin = "#{alt_proto}://dummy#{alt_origin}"
           uri = URI.parse(alt_origin)
           uri.host = nil
           uri

data/lib/httpx/chainable.rb

@@ -3,13 +3,15 @@
 module HTTPX
   module Chainable
     %i[head get post put delete trace options connect patch].each do |meth|
-      define_method meth do |*uri, **options|
-        request(meth, uri, **options)
-      end
+      class_eval(<<-MOD, __FILE__, __LINE__ + 1)
+        def #{meth}(*uri, **options)
+          request(:#{meth}, uri, **options)
+        end
+      MOD
     end
 
-    def request(verb, uri, **options)
-      branch(default_options).request(verb, uri, **options)
+    def request(*args, **options)
+      branch(default_options).request(*args, **options)
     end
 
     # :nocov:
@@ -32,11 +34,11 @@ module HTTPX
       branch(default_options).wrap(&blk)
     end
 
-    def plugin(*args, **opts)
+    def plugin(*args, **opts, &blk)
       klass = is_a?(Session) ? self.class : Session
       klass = Class.new(klass)
       klass.instance_variable_set(:@default_options, klass.default_options.merge(default_options))
-      klass.plugin(*args, **opts).new
+      klass.plugin(*args, **opts, &blk).new
     end
 
     # deprecated
@@ -57,7 +59,6 @@ module HTTPX
       @options || Options.new
     end
 
-    # :nodoc:
     def branch(options, &blk)
       return self.class.new(options, &blk) if is_a?(Session)
 

data/lib/httpx/connection.rb

@@ -51,7 +51,7 @@ module HTTPX
     def initialize(type, uri, options)
       @type = type
       @origins = [uri.origin]
-      @origin = URI(uri.origin)
+      @origin = Utils.uri(uri.origin)
       @options = Options.new(options)
       @window_size = @options.window_size
       @read_buffer = Buffer.new(BUFFER_SIZE)
@@ -67,6 +67,10 @@ module HTTPX
       else
         transition(:idle)
       end
+
+      @inflight = 0
+      @keep_alive_timeout = options.timeout.keep_alive_timeout
+      @keep_alive_timer = nil
     end
 
     # this is a semi-private method, to be used by the resolver
@@ -138,14 +142,15 @@ module HTTPX
       end
     end
 
-    def purge_pending
+    def purge_pending(&block)
       pendings = []
-      pendings << @parser.pending if @parser
+      if @parser
+        @inflight -= @parser.pending.size
+        pendings << @parser.pending
+      end
       pendings << @pending
       pendings.each do |pending|
-        pending.reject! do |request|
-          yield request
-        end
+        pending.reject!(&block)
       end
     end
 
@@ -168,22 +173,45 @@ module HTTPX
     end
 
     def interests
-      return :w if @state == :idle
+      # connecting
+      if connecting?
+        connect
+
+        return @io.interests if connecting?
+      end
 
-      :rw
+      # if the write buffer is full, we drain it
+      return :w if @write_buffer.full?
+
+      return @parser.interests if @parser
+
+      nil
     end
 
     def to_io
+      @io.to_io
+    end
+
+    def call
       case @state
-      when :idle
-        transition(:open)
+      when :closed
+        return
+      when :closing
+        consume
+        transition(:closed)
+        emit(:close)
+      when :open
+        consume
       end
-      @io.to_io
+      nil
     end
 
     def close
       @parser.close if @parser
-      transition(:closing)
+      return unless @keep_alive_timer
+
+      @keep_alive_timer.cancel
+      remove_instance_variable(:@keep_alive_timer)
     end
 
     def reset
@@ -195,26 +223,25 @@ module HTTPX
     def send(request)
       if @parser && !@write_buffer.full?
         request.headers["alt-used"] = @origin.authority if match_altsvcs?(request.uri)
+        if @keep_alive_timer
+          # when pushing a request into an existing connection, we have to check whether there
+          # is the possibility that the connection might have extended the keep alive timeout.
+          # for such cases, we want to ping for availability before deciding to shovel requests.
+          if @keep_alive_timer.fires_in.negative?
+            @pending << request
+            parser.ping
+            return
+          end
+
+          @keep_alive_timer.pause
+        end
+        @inflight += 1
         parser.send(request)
       else
         @pending << request
       end
     end
 
-    def call
-      case @state
-      when :closed
-        return
-      when :closing
-        dwrite
-        transition(:closed)
-        emit(:close)
-      when :open
-        consume
-      end
-      nil
-    end
-
     def timeout
       return @timeout if defined?(@timeout)
 
@@ -225,54 +252,91 @@ module HTTPX
 
     private
 
+    def connect
+      transition(:open)
+    end
+
     def exhausted?
       @parser && parser.exhausted?
     end
 
     def consume
       catch(:called) do
-        dread
-        dwrite
-        parser.consume
-      end
-    end
-
-    def dread(wsize = @window_size)
-      loop do
-        siz = @io.read(wsize, @read_buffer)
-        unless siz
-          ex = EOFError.new("descriptor closed")
-          ex.set_backtrace(caller)
-          on_error(ex)
-          return
-        end
-        return if siz.zero?
-
-        log { "READ: #{siz} bytes..." }
-        parser << @read_buffer.to_s
-        return if @state == :closing || @state == :closed
-      end
-    end
-
-    def dwrite
-      loop do
-        return if @write_buffer.empty?
-
-        siz = @io.write(@write_buffer)
-        unless siz
-          ex = EOFError.new("descriptor closed")
-          ex.set_backtrace(caller)
-          on_error(ex)
-          return
+        loop do
+          parser.consume
+
+          # we exit if there's no more data to process
+          if @pending.size.zero? && @inflight.zero?
+            log(level: 3) { "NO MORE REQUESTS..." }
+            return
+          end
+
+          @timeout = @current_timeout
+
+          read_drained = false
+          write_drained = nil
+
+          # dread
+          loop do
+            siz = @io.read(@window_size, @read_buffer)
+            unless siz
+              ex = EOFError.new("descriptor closed")
+              ex.set_backtrace(caller)
+              on_error(ex)
+              return
+            end
+
+            if siz.zero?
+              read_drained = @read_buffer.empty?
+              break
+            end
+
+            parser << @read_buffer.to_s
+
+            break if @state == :closing || @state == :closed
+
+            # for HTTP/2, we just want to write goaway frame
+          end unless @state == :closing
+
+          # dwrite
+          loop do
+            if @write_buffer.empty?
+              # we only mark as drained on the first loop
+              write_drained = write_drained.nil? && @inflight.positive?
+              break
+            end
+
+            siz = @io.write(@write_buffer)
+            unless siz
+              ex = EOFError.new("descriptor closed")
+              ex.set_backtrace(caller)
+              on_error(ex)
+              return
+            end
+
+            if siz.zero?
+              write_drained = !@write_buffer.empty?
+              break
+            end
+
+            break if @state == :closing || @state == :closed
+
+            write_drained = false
+          end
+
+          # return if socket is drained
+          if read_drained && write_drained
+            log(level: 3) { "WAITING FOR EVENTS..." }
+            return
+          end
         end
-        log { "WRITE: #{siz} bytes..." }
-        return if siz.zero?
-        return if @state == :closing || @state == :closed
       end
     end
 
     def send_pending
       while !@write_buffer.full? && (request = @pending.shift)
+        @inflight += 1
+        @keep_alive_timer.pause if @keep_alive_timer
         parser.send(request)
       end
     end
@@ -292,12 +356,15 @@ module HTTPX
         AltSvc.emit(request, response) do |alt_origin, origin, alt_params|
           emit(:altsvc, alt_origin, origin, alt_params)
         end
+        handle_response
         request.emit(:response, response)
       end
       parser.on(:altsvc) do |alt_origin, origin, alt_params|
         emit(:altsvc, alt_origin, origin, alt_params)
       end
 
+      parser.on(:pong, &method(:send_pending))
+
       parser.on(:promise) do |request, stream|
         request.emit(:promise, parser, stream)
       end
@@ -307,14 +374,21 @@ module HTTPX
       parser.on(:origin) do |origin|
         @origins << origin
       end
-      parser.on(:close) do
+      parser.on(:close) do |force|
         transition(:closing)
+        if force
+          transition(:closed)
+          emit(:close)
+        end
       end
       parser.on(:reset) do
-        transition(:closing)
-        unless parser.empty?
+        if parser.empty?
+          reset
+        else
+          transition(:closing)
           transition(:closed)
           emit(:reset)
+          @parser.reset if @parser
           transition(:idle)
           transition(:open)
         end
@@ -335,6 +409,9 @@ module HTTPX
 
     def transition(nextstate)
       case nextstate
+      when :idle
+        @timeout = @current_timeout = @options.timeout.connect_timeout
+
       when :open
         return if @state == :closed
 
@@ -344,6 +421,8 @@ module HTTPX
         return unless @io.connected?
 
         send_pending
+
+        @timeout = @current_timeout = @options.timeout.operation_timeout
         emit(:open)
       when :closing
         return unless @state == :open
@@ -358,6 +437,11 @@ module HTTPX
 
         @io.close
         @read_buffer.clear
+        if @keep_alive_timer
+          @keep_alive_timer.cancel
+          remove_instance_variable(:@keep_alive_timer)
+        end
+
         remove_instance_variable(:@timeout) if defined?(@timeout)
       when :already_open
         nextstate = :open
@@ -371,7 +455,6 @@ module HTTPX
       throw(:jump_tick)
     rescue Errno::ECONNREFUSED,
            Errno::EADDRNOTAVAIL,
-           Errno::EHOSTUNREACH,
            OpenSSL::SSL::SSLError => e
       # connect errors, exit gracefully
       handle_error(e)
@@ -379,21 +462,43 @@ module HTTPX
       emit(:close)
     end
 
-    def on_error(ex)
-      handle_error(ex)
-      reset
+    def handle_response
+      @inflight -= 1
+      return unless @inflight.zero?
+
+      if @keep_alive_timer
+        @keep_alive_timer.resume
+        @keep_alive_timer.reset
+      else
+        @keep_alive_timer = @timers.after(@keep_alive_timeout) do
+          unless @inflight.zero?
+            log { "(#{@origin}): keep alive timeout expired" }
+            parser.ping
+          end
+        end
+      end
     end
 
-    def handle_error(error)
+    def on_error(error)
       if error.instance_of?(TimeoutError)
         if @timeout
           @timeout -= error.timeout
           return unless @timeout <= 0
         end
 
-        error = error.to_connection_error if connecting?
+        if @total_timeout && @total_timeout.fires_in.negative?
+          ex = TotalTimeoutError.new(@total_timeout.interval, "Timed out after #{@total_timeout.interval} seconds")
+          ex.set_backtrace(error.backtrace)
+          error = ex
+        elsif connecting?
+          error = error.to_connection_error
+        end
       end
+      handle_error(error)
+      reset
+    end
 
+    def handle_error(error)
       parser.handle_error(error) if @parser && parser.respond_to?(:handle_error)
       while (request = @pending.shift)
         request.emit(:response, ErrorResponse.new(request, error, @options))
@@ -408,8 +513,8 @@ module HTTPX
       @total_timeout ||= @timers.after(total) do
         ex = TotalTimeoutError.new(total, "Timed out after #{total} seconds")
         ex.set_backtrace(caller)
-        @parser.close if @parser
         on_error(ex)
+        @parser.close if @parser
       end
     end
   end