httpx 0.8.0 → 0.10.1

data/doc/release_notes/0_6_5.md

@@ -0,0 +1,22 @@
+# 0.6.5
+
+This release fixes important bugs, and automates the PKI for the test suite.
+
+## Features
+
+* `resolver_options` can now receive a `:cache` flag (default: `true`). This bypasses caching and forces the lookup;
+
+## Improvements
+
+* Building the TLS certs necessary for the test suite has been scripted, after the initial certs expired and brought the CI to a halt;
+* All DNS resolvers have a functional test, both for the happy as well as the error case;
+* Added functional tests for HTTP and HTTPS proxy with authentication, making all proxy options now tested with authentication;
+
+
+## Bugfixes
+
+* native and https DNS resolvers weren't usable after a resolving error;
+* system DNS resolver could halt the system after a dns resolving error;
+* fixed system halt on HTTP proxy authentication error;
+
+

data/doc/release_notes/0_6_6.md

@@ -0,0 +1,19 @@
+# 0.6.6
+
+## Features
+
+* The `retries` plugin receives two new options:
+  * `retry_on`: a callable that receives the failed response as an argument; the return value will determine whether there'll be a retried request.
+  * `retry_after`: time (in seconds) after which there request will be retried. Can be an integer or a callable that receives the request and returns an integer (one can do exponential back-off like that, for example).
+* Added support for DNS-over-HTTPS GET requests as per the latest spec.
+
+## Improvements
+
+* `HTTPX.plugins` got deprecated; basically, it's great until you have to pass options to a plugin, and then it just works (not). The recommended way to load multiple plugins is `HTTPX.plugin(...).plugin(...)`.
+
+
+## Bugfixes
+
+* fixed a proxy bug where an `Alt-Svc` response header would make the client try to connect. Just like connection coalescing and the ORIGIN frame, it ignores it when going through a proxy.  
+
+

data/doc/release_notes/0_6_7.md

@@ -0,0 +1,5 @@
+# 0.6.7
+
+## Bugfixes
+
+* An error was reported when using the follow plugin allowing insecure redirects: if the insecure redirect would be for the same host, and the original request was performed with HTTP/2, the library would try to coalesce the request, and blocks the reactor. A check was made to ensure that connection only coalesces if both are https connections.

data/doc/release_notes/0_7_0.md

@@ -0,0 +1,46 @@
+# 0.7.0
+
+
+## Features
+
+New option: `:max_requests`. This is a connection-level option signalizing how many requests can be performed on a connection. Although the HTTP/1 parser defined this well, in HTTP/2 this wasn't very clear, so: by definition, the remote MAX_CONCURRENT_STREAMS setting will be used to define it, unless the user explicitly passed the option. You can also pass `:max_requests => Float::INFINITY` if you know that the server allows more requests than that on a connection.
+
+New plugin: `:expect`. 
+
+Although there was support for `expect: 100-continue` header already when passed, this plugin can:
+
+* automatically set the header on requests with body;
+* execute the flow;
+* recover from 417 status errors (i.e. try again without it);
+* send body after X seconds if no 100 response came;
+
+Suport for `with_` methods for the session. As long as the suffix is a valid attribute, it's just like that:
+
+```ruby
+HTTPX.with_timeout(...).with_ssl(...)
+# same as:
+# HTTPX.with(timeout: ..., ssl: ...)
+```
+
+## Improvements
+
+### Connections
+
+The following improvements make the `persistent` plugin way more resilient:
+
+* Better balancing of HTTP/2 connections by distributing requests among X connections depending of how many requests they can process.
+* Exhausted connections can off-load to a new same-origin connection (such as, when the server negotiates less `MAX_CONCURRENT_STREAMS` than what's expected).
+
+### Timeouts
+
+(Timeouts will be one of the main improvements from the 0.7.x series)
+
+`:total_timeout` is now a connection-level directive, which means that this feature will actually make more sense and account for all requests in a block at the same time, instead of one-by-one.
+
+### Options
+
+Option setters were being bypassed, therefore a lot of the type-checks defined there weren't effectively being picked upon, which could have led to weird user errors.
+
+## Bugfixes
+
+* fixed the `push_promise` plugin integration (wasn't working well since `http-2-next` was adopted);

data/doc/release_notes/0_8_0.md

@@ -0,0 +1,27 @@
+# 0.8.0
+
+
+## Features
+
+* `keep_alive_timeout`: for persistent connections, the keep alive timeout will set the connection to be closed if not reused for a request **after** the last received response;
+
+## Improvements
+
+* using `max_requests` for HTTP/1 pipelining as well;
+* `retries` plugin now works with plain HTTP responses (not just error responses);
+* reduced the number of string allocations from log labels;
+* performance: a lot of improvements were made to optimize the "waiting for IO events" phase, which dramatically reduced the CPU usage and make the performance of the library more on-par with other ruby HTTP gems for the 1-shot request scenario.
+
+
+## Bugfixes
+
+* fixed `HTTPX::Response#copy_to`;
+* fixed `compression` plugin not properly compressing request bodies using `gzip`;
+* fixed `compression` plugin not handling `content-encoding: identity` payloads;
+* do not overwrite user-defined `max_requests`on HTTP2 connection handshake;
+* `retries` plugin: connection was blocking when a request with body was retried;
+* `alt-svc: clear` response header was causing the process to hang;
+
+## Tests
+
+* Code coverage improved to 91%;

data/doc/release_notes/0_8_1.md

@@ -0,0 +1,8 @@
+# 0.8.1
+
+
+## Bugfixes
+
+* fixing HTTP/2 handshake IO interests calculation;
+* fixed the double ctrl+f issue when terminating an ongoing HTTP/2 request;
+* fixed connection comparison when passing headers;

data/doc/release_notes/0_8_2.md

@@ -0,0 +1,7 @@
+# 0.8.2
+
+## Features
+
+* `:expect` plugin now supports a new option, `:expect_threshold_size`, meaning: the byte size threshold below which no `expect` header will be sent with requests with payload.
+* `:compression` plugin now supports a new option, `:compression_threshold_size`, meaning: the bytesize threshold below which request payload won't be compressed before being sent.
+* for HTTP/2 connections, when `keep_alive_timeout` expires, a `PING` frame is used to check connection availability; if successful, the connection will be reused.

data/doc/release_notes/0_9_0.md

@@ -0,0 +1,38 @@
+# 0.9.0
+
+## Features
+
+### Multiple requests with specific options
+
+You can now pass a third element to the "request element" of an array to `.request`.
+
+```ruby
+requests = [
+  [:post, "https://url/post", { form: { foo: "bar" } }],
+  [:post, "https://url/post", { form: { foo: "bar2" } }]
+]
+HTTPX.request(requests)
+# or, if you want to pass options common to all requests
+HTTPX.request(requests, max_concurrent_requests: 1)
+```
+
+
+### HTTPX::Session#build_request
+
+`HTTPX::Session::build_request` is now public API from a session. You can now build requests before you send them. These request objects are still considered somewhat "internal", so consider them immutable and **do not rely on its API**. Just pass them forward.
+
+Note: this API is only available for instantiated session, so there is no `HTTPX.build_request`.
+
+
+```ruby
+
+HTTPX.wrap do |http|
+  requests = [
+    http.build_request(:post, "https://url/post", { form: { foo: "bar" } }),
+    http.build_request(:post, "https://url/post", { form: { foo: "bar2" } })
+  ]
+  http.request(requests)
+  # or, if you want to pass options common to all requests
+  http.request(requests, max_concurrent_requests: 1)
+end
+```

data/lib/httpx.rb

@@ -5,6 +5,8 @@ require "httpx/version"
 require "httpx/extensions"
 
 require "httpx/errors"
+require "httpx/utils"
+require "httpx/domain_name"
 require "httpx/altsvc"
 require "httpx/callbacks"
 require "httpx/loggable"

data/lib/httpx/adapters/faraday.rb

@@ -121,7 +121,7 @@ module Faraday
           end
 
           def respond_to_missing?(meth)
-            @env.respond_to?(meth)
+            @env.respond_to?(meth) || super
           end
 
           def method_missing(meth, *args, &blk)

data/lib/httpx/chainable.rb

@@ -10,8 +10,8 @@ module HTTPX
       MOD
     end
 
-    def request(verb, uri, **options)
-      branch(default_options).request(verb, uri, **options)
+    def request(*args, **options)
+      branch(default_options).request(*args, **options)
     end
 
     # :nocov:
@@ -34,20 +34,23 @@ module HTTPX
       branch(default_options).wrap(&blk)
     end
 
-    def plugin(*args, **opts)
+    def plugin(*args, **opts, &blk)
       klass = is_a?(Session) ? self.class : Session
       klass = Class.new(klass)
       klass.instance_variable_set(:@default_options, klass.default_options.merge(default_options))
-      klass.plugin(*args, **opts).new
+      klass.plugin(*args, **opts, &blk).new
     end
 
     # deprecated
+    # :nocov:
     def plugins(*args, **opts)
+      warn ":#{__method__} is deprecated, use :plugin instead"
       klass = is_a?(Session) ? self.class : Session
       klass = Class.new(klass)
       klass.instance_variable_set(:@default_options, klass.default_options.merge(default_options))
       klass.plugins(*args, **opts).new
     end
+    # :nocov:
 
     def with(options, &blk)
       branch(default_options.merge(options), &blk)
@@ -59,7 +62,6 @@ module HTTPX
       @options || Options.new
     end
 
-    # :nodoc:
     def branch(options, &blk)
       return self.class.new(options, &blk) if is_a?(Session)
 
@@ -67,12 +69,10 @@ module HTTPX
     end
 
     def method_missing(meth, *args, **options)
-      if meth =~ /\Awith_(.+)/
-        option = Regexp.last_match(1).to_sym
-        with(option => (args.first || options))
-      else
-        super
-      end
+      return super unless meth =~ /\Awith_(.+)/
+
+      option = Regexp.last_match(1).to_sym
+      with(option => (args.first || options))
     end
 
     def respond_to_missing?(meth, *args)

data/lib/httpx/connection.rb

@@ -51,7 +51,7 @@ module HTTPX
     def initialize(type, uri, options)
       @type = type
       @origins = [uri.origin]
-      @origin = URI(uri.origin)
+      @origin = Utils.uri(uri.origin)
       @options = Options.new(options)
       @window_size = @options.window_size
       @read_buffer = Buffer.new(BUFFER_SIZE)
@@ -88,8 +88,6 @@ module HTTPX
 
       return false if exhausted?
 
-      return false if @keep_alive_timer && @keep_alive_timer.fires_in.negative?
-
       (
         (
           @origins.include?(uri.origin) &&
@@ -107,8 +105,6 @@ module HTTPX
 
       return false if exhausted?
 
-      return false if @keep_alive_timer && @keep_alive_timer.fires_in.negative?
-
       !(@io.addresses & connection.addresses).empty? && @options == connection.options
     end
 
@@ -124,8 +120,8 @@ module HTTPX
       end
     end
 
-    def create_idle
-      self.class.new(@type, @origin, @options)
+    def create_idle(options = {})
+      self.class.new(@type, @origin, @options.merge(options))
     end
 
     def merge(connection)
@@ -135,18 +131,7 @@ module HTTPX
       end
     end
 
-    def unmerge(connection)
-      @origins -= connection.instance_variable_get(:@origins)
-      purge_pending do |request|
-        request.uri.origin == connection.origin && begin
-          request.transition(:idle)
-          connection.send(request)
-          true
-        end
-      end
-    end
-
-    def purge_pending
+    def purge_pending(&block)
       pendings = []
       if @parser
         @inflight -= @parser.pending.size
@@ -154,9 +139,7 @@ module HTTPX
       end
       pendings << @pending
       pendings.each do |pending|
-        pending.reject! do |request|
-          yield request
-        end
+        pending.reject!(&block)
       end
     end
 
@@ -229,8 +212,19 @@ module HTTPX
     def send(request)
       if @parser && !@write_buffer.full?
         request.headers["alt-used"] = @origin.authority if match_altsvcs?(request.uri)
+        if @keep_alive_timer
+          # when pushing a request into an existing connection, we have to check whether there
+          # is the possibility that the connection might have extended the keep alive timeout.
+          # for such cases, we want to ping for availability before deciding to shovel requests.
+          if @keep_alive_timer.fires_in.negative?
+            @pending << request
+            parser.ping
+            return
+          end
+
+          @keep_alive_timer.pause
+        end
         @inflight += 1
-        @keep_alive_timer.pause if @keep_alive_timer
         parser.send(request)
       else
         @pending << request
@@ -281,8 +275,6 @@ module HTTPX
               return
             end
 
-            log { "READ: #{siz} bytes..." }
-
             if siz.zero?
               read_drained = @read_buffer.empty?
               break
@@ -310,7 +302,6 @@ module HTTPX
               on_error(ex)
               return
             end
-            log { "WRITE: #{siz} bytes..." }
 
             if siz.zero?
               write_drained = !@write_buffer.empty?
@@ -361,6 +352,8 @@ module HTTPX
         emit(:altsvc, alt_origin, origin, alt_params)
       end
 
+      parser.on(:pong, &method(:send_pending))
+
       parser.on(:promise) do |request, stream|
         request.emit(:promise, parser, stream)
       end
@@ -395,7 +388,7 @@ module HTTPX
       parser.on(:error) do |request, ex|
         case ex
         when MisdirectedRequestError
-          emit(:uncoalesce, request.uri)
+          emit(:misdirected, request)
         else
           response = ErrorResponse.new(request, ex, @options)
           request.emit(:response, response)
@@ -431,7 +424,7 @@ module HTTPX
           remove_instance_variable(:@total_timeout)
         end
 
-        @io.close
+        @io.close if @io
         @read_buffer.clear
         if @keep_alive_timer
           @keep_alive_timer.cancel
@@ -451,7 +444,6 @@ module HTTPX
       throw(:jump_tick)
     rescue Errno::ECONNREFUSED,
            Errno::EADDRNOTAVAIL,
-           Errno::EHOSTUNREACH,
            OpenSSL::SSL::SSLError => e
       # connect errors, exit gracefully
       handle_error(e)
@@ -469,8 +461,8 @@ module HTTPX
       else
         @keep_alive_timer = @timers.after(@keep_alive_timeout) do
           unless @inflight.zero?
-            log { "(#{object_id})) keep alive timeout expired, closing..." }
-            reset
+            log { "(#{@origin}): keep alive timeout expired" }
+            parser.ping
           end
         end
       end

data/lib/httpx/connection/http1.rb

@@ -21,6 +21,7 @@ module HTTPX
       @version = [1, 1]
       @pending = []
       @requests = []
+      @handshake_completed = false
     end
 
     def interests
@@ -78,6 +79,7 @@ module HTTPX
         break if idx >= requests_limit
         next if request.state == :done
 
+        request.headers["connection"] ||= request.options.persistent || idx < requests_limit - 1 ? "keep-alive" : "close"
         handle(request)
       end
     end
@@ -154,6 +156,7 @@ module HTTPX
       @parser.reset!
       @max_requests -= 1
       manage_connection(response)
+
       send(@pending.shift) unless @pending.empty?
     end
 
@@ -170,23 +173,39 @@ module HTTPX
       end
     end
 
+    def ping
+      emit(:reset)
+      emit(:exhausted)
+    end
+
     private
 
     def manage_connection(response)
       connection = response.headers["connection"]
       case connection
-      when /keep\-alive/i
+      when /keep-alive/i
+        if @handshake_completed
+          if @max_requests.zero?
+            @pending.concat(@requests)
+            @requests.clear
+            emit(:exhausted)
+          end
+          return
+        end
+
         keep_alive = response.headers["keep-alive"]
         return unless keep_alive
 
         parameters = Hash[keep_alive.split(/ *, */).map do |pair|
           pair.split(/ *= */)
         end]
-        @max_requests = parameters["max"].to_i if parameters.key?("max")
+        @max_requests = parameters["max"].to_i - 1 if parameters.key?("max")
+
         if parameters.key?("timeout")
           keep_alive_timeout = parameters["timeout"].to_i
           emit(:timeout, keep_alive_timeout)
         end
+        @handshake_completed = true
       when /close/i
         disable
       when nil
@@ -206,7 +225,14 @@ module HTTPX
     def disable_pipelining
       return if @requests.empty?
 
-      @requests.each { |r| r.transition(:idle) }
+      @requests.each do |r|
+        r.transition(:idle)
+
+        # when we disable pipelining, we still want to try keep-alive.
+        # only when keep-alive with one request fails, do we fallback to
+        # connection: close.
+        r.headers["connection"] = "close" if @max_concurrent_requests == 1
+      end
       # server doesn't handle pipelining, and probably
       # doesn't support keep-alive. Fallback to send only
       # 1 keep alive request.
@@ -216,7 +242,7 @@ module HTTPX
 
     def set_request_headers(request)
       request.headers["host"] ||= request.authority
-      request.headers["connection"] ||= "keep-alive"
+      request.headers["connection"] ||= request.options.persistent ? "keep-alive" : "close"
       if !request.headers.key?("content-length") &&
          request.body.bytesize == Float::INFINITY
         request.chunk!