httpx 1.6.3 → 1.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f49b29ea3703f6f40abe3cd82d455235b0a0b50a694bd8fa55839ac471d32bbb
-  data.tar.gz: ce933bb3c35d9434f810d4fc83acb244a1ad0db187123d5f01dcbbbd83622a47
+  metadata.gz: a9c30e22a2d406a61ef87a58fddd607bb2b1fad7b50d837d8db062d4a538a2d2
+  data.tar.gz: 9c5e1997b9c03434071c59b2deb2b004f7b1a1077c2ccaed03b9e3a1db7aafe5
 SHA512:
-  metadata.gz: 258cb32129840347a1a37633fb6273133187c98636b63ffcc0b8064f39e3b5642d693ee5da5d7776529625fbe4b68b5193b405c2e0658b2c9b1193b49e316bdc
-  data.tar.gz: 387397ab1954b6abf6a8a54d8764fb25e786fb1bd293401e8afd57f823464f67210527fee0b9e94b34cc3d2931143a5d94598fb6241f5728df77a8f1e045a5bf
+  metadata.gz: 87d99c4971b99f086f7811be81a8453e71bd2535120b5e560a2fc837d50e8a040e70ab1b2b191beee4b481fcea5063576d8895d318a0089183c21cb59fdd0f24
+  data.tar.gz: 85469cf5b5822990367f1e5591798a1512de90a60226853da4b4e6b00dde8816d6218dd2aacc9b5d85f84f6cb1637dd4f7b37836863f44144f05329e4fbdf7f5

data/doc/release_notes/0_11_0.md

@@ -21,7 +21,7 @@ stub_http_request(:get, "https://www.google.com").and_return(status: 200, body:
 
 ```
 
-Read more about it in the [webmock integration documentation](https://os85.gitlab.io/httpx/wiki/Webmock-Adapter).
+Read more about it in the [webmock integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Webmock-Adapter).
 
 ### Datadog Adapter
 
@@ -40,7 +40,7 @@ A trace will be emitted for every request, so this should be an interesting visu
 
 Customization options and traces are similar to what [the net-http adapter provides](https://docs.datadoghq.com/tracing/setup_overview/setup/ruby/#nethttp).
 
-Read more about it in the [datadog integration documentation](https://os85.gitlab.io/httpx/wiki/Datadog-Adapter).
+Read more about it in the [datadog integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Datadog-Adapter).
 
 ## Improvements
 
@@ -52,7 +52,7 @@ Read more about it in the [datadog integration documentation](https://os85.gitla
 HTTPX.plugin(:multipart).post(uri, form: {file: File.new("path/to/file")})
 ```
 
-Read more about it in the [multipart plugin documentation](https://os85.gitlab.io/httpx/wiki/Multipart-Uploads), including also about why this was made.
+Read more about it in the [multipart plugin documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Multipart-Uploads), including also about why this was made.
 
 ### Expect Plugin
 

data/doc/release_notes/1_6_3.md

@@ -7,8 +7,8 @@
 ## Improvements
 
 * `system` resolver now works in a non-blocking manner, initiating the dns query in a separate thread and waiting on the pipe after that (it was blocking the main thread during resolution before).
-* reduce allocation to a sinfle shared option object when headers are passed as a session-level option, like `HTTPX.with(headers: geaders).get(...)`
-* privilege using `String#replace` in buffer operations (instead of "clean-then-append").
+* reduce allocation to a single shared option object when headers are passed as a session-level option, like `HTTPX.with(headers: headers).get(...)`
+* favour using `String#replace` in buffer operations (instead of "clean-then-append").
 * using `Array#unshift` instead of `Array#concat` in order to ensure that request ordering is respected in the face of an in-between error which requires reconnect-and-resend.
 * replaced more internal callback indirection with plain method calls.
 

data/doc/release_notes/1_7_0.md

@@ -0,0 +1,149 @@
+# 1.7.0
+
+## Features
+
+### All AUTH plugin improvements!!
+
+#### `:auth`
+
+The `:auth` plugin can now be used with a dynamic callable object (methods, procs...) to generate the token.
+
+```ruby
+# static token, pre 1.7.0
+HTTPX.plugin(:auth).authorization("API-TOKEN")
+# dynamically generate token!
+HTTPX.plugin(:auth).authorization { generate_new_ephemeral_token }
+```
+
+The `.authorization` method is now syntactic sugar for a new option, `:auth_header_value`, which can be used directly, alongside a `:auth_header_type`:
+
+```ruby
+HTTPX.plugin(:auth).authorization("API-TOKEN")
+HTTPX.plugin(:auth).authorization { generate_new_ephemeral_token }
+HTTPX.plugin(:auth).authorization("Bearer API-TOKEN")
+# same as
+HTTPX.plugin(:auth, auth_header_value: "API-TOKEN")
+HTTPX.plugin(:auth, auth_header_value: -> { generate_new_ephemeral_token })
+HTTPX.plugin(:auth, auth_header_type: "Bearer", auth_header_value: "API-TOKEN")
+```
+
+A new option `:generate_auth_value_on_retry` (which can be passed a callable receiving a response object) is now available; when used alongside the `:retries` plugin, it'll use the callable passed to the `.authorization` method to generate a new token before retrying the request:
+
+```ruby
+authed = HTTPX.plugin(:retries).plugin(:auth, generate_auth_value_on_retry: ->(res) {
+  res.status == 401
+}).authorization { generate_new_ephemeral_token }
+authed.get("https://example.com")
+```
+
+Read more about it in the [auth plugin wiki](https://honeyryderchuck.gitlab.io/httpx/wiki/Auth).
+
+#### `:oauth`
+
+The `:oauth` plugin implementation was revamped to make use of the `:auth` plugin new functionality, in order to make managing an oauth session more seamless.
+
+Take the following example:
+
+```ruby
+session = HTTPX.plugin(:oauth).with_oauth_options(
+  issuer: server.origin,
+  client_id: "CLIENT_ID",
+  client_secret: "SECRET",
+)
+session.get("https://example.com") #=> will load server metadata, request an access token, and perform the request with the access token.
+# 2 hours later...
+session.get("https://example.com")
+# it'll reuse the same acces token, and if the request fails with 401, it'll request a new
+# access token using the refresh token grant (when supported by the token issuer), and
+# reperform the original request with the new access token.
+```
+
+A new option, `:oauth_options`, is now available. The same parameters previously supported by the `:oauth_session` options are supported.
+
+The following components are therefore deprecated and scheduled for removal in a future major version:
+
+* `:oauth_session` option
+* `.oauth_auth` session method
+* `.with_access_token` session method
+
+#### `:bearer_auth`, `:digest_auth`; `:ntlm_auth`
+
+The `:auth` plugin is now the foundation of each of these plugins, which haven't suffered major API changes.
+
+Read more about it in the [auth plugin wiki](https://honeyryderchuck.gitlab.io/httpx/wiki/OAuth).
+
+### `:retries` plugin: `:retry_after` backoff algorithms
+
+The `:retries` plugins supports two new possible values for the `:retry_after` option: `:exponential_backoff` and `:polynomial_backoff`. They'll implement the respective calculation per each retry of a given request.
+
+```ruby
+# will wait 1, 2, 4, 8, 16 seconds... depending of how many retries it can wait for
+session = HTTPX.plugin(:retries, retry_after: :exponential_backoff)
+```
+
+Read more about it in the [retries plugin wiki](https://honeyryderchuck.gitlab.io/httpx/wiki/Retries).
+
+### Ractor compatibility
+
+`httpx` can be used within a ractor:
+
+```ruby
+# ruby 4.0 syntax
+response = Ractor.new(uri) do |uri|
+  HTTPX.get(uri)
+end.value
+```
+
+Bear in mind that, if you're connection via HTTPS, you'll need make sure you're using version 4.0 or higher of the `openssl` gem.
+
+The test suite isn't exhaustive for ractors yet, but most plugins should also be ractor-compatible. If they don't work, that's a bug, and you're recommended to report it.
+
+## Improvements
+
+* When encoding the `:json` param to send it as `application/json` payload, (example: `HTTPX.post("https://example.com", json: { foo: "bar })`), and the method uses the `json` standard library, it'll use `JSON.generate` (instead of `JSON.dump`) to encode the JSON payload. The reason is that, unlike `JSON.dump`, it doesn't rely on access to a global mutable hash, and is therefore ractor-safe.
+* `:stream` plugin: the stream response class (the object that is returned in request calls is a stream response) can be extended now. You can add a `StreamResponseMethods` method to your plugin. Read more about it in the documentation.
+* The resolver name cache (used by the native and https resolvers) was remade into a LRU cache, and will therefore not keep on growing when `httpx` is used to connect to a huge number of hostnames in a process.
+* the native and https DNS resolvers will ignore answers with SERVFAIL code while there are retries left (some resolvers use such error code for rate limiting).
+* `:timeout` option values are now validated, and an error is raised when passing an unrecognized timeout option (which is a good layer of protection for typos).
+* pool: try passing the scheduler to a thread waiting on a connection, to avoid the current case where a connection may be checked-in-then-immediately-out-after when doing multiple requests in a loop, never giving a chance to others and potentially making the pool time out.
+* headers deep-freeze and dup.
+
+## Bugfixes
+
+* recover and close connection when an `IOError` is raised while waiting for IO readiness (could cause busy loops during HTTP/2 termination handshake).
+* `:stream_bidi` plugin: improve thread-safety of buffer operations when the session is used from multiple threads.
+* `:stream_bidi` plugin: added missing methods to signal in order to comply with the Selectable API (it was reported as raising `NoMethodError` under certain conditions).
+* `:stream_bidi` plugin: can support non-bidirectional stream requests using the same session.
+* `:stream` plugin: is now compatible with fiber scheduler engines (via the `:fiber_concurrency` plugin).
+* `:stream` plugin: make sure that stream long-running requests do not share the same connection as regular threads.
+* `:digest_auth` plugin: can now support qop values wrapped inside parentheses in the `www-authenticate` header (i.e. `qop="('auth',)"`).
+* https resolver: handle 3XX redirect responses in HTTP DNS queries.
+* https resolver: do not close HTTP connections whhich are shared across AAAA and A resolution paths when its in use by one of them.
+* fix access to private method from `http-2` which was made public in more recent versions, but not in older still-supported versions.
+* fixed resolver log message using a "connection" label.
+* `HTTPX::Response.copy_to` will explicitly close the response at the end; given that the body file can be moved as a result, there is no guarantee that the response is still usable, so might as well just close it altogether.
+* selector: avoid skipping persistent connections in the selector to deactivate due to iterate-and-modify.
+
+## Breaking Changes
+
+### `:digest_auth` error
+
+The main error class for the `:digest_auth` plugin has been moved to a different location. If you were rescuing the `HTTPX::Plugins::DigestAuth::DigestError` error, you should now point to the `HTTPX::Authentication::Digest::Error`.
+
+### `:stream` plugin: `build_request` should receive `stream: true` for stream requests
+
+In case you're building request objects before passing them to the session, you're now forced to create them with the `:stream` option on:
+
+```ruby
+session = HTTPX.plugin(:stream)
+
+# before
+req = session.build_request("GET", "https://example.com/stream")
+session.request(req, stream: true)
+
+# after
+req = session.build_request("GET", "https://example.com/stream", stream: true)
+session.request(req)
+```
+
+Previous code may still work in a few cases, but it is not guaranteed to work on all cases.

data/doc/release_notes/1_7_1.md

@@ -0,0 +1,21 @@
+# 1.7.1
+
+## Improvements
+
+* fixed timers handling in the selector loop which caused them to be traversed-for-drop twice on each tick.
+* connection: take proxy connecting states when transitioning to `:closing` state.
+* refactored name resolution cache to a cache adapter API with a default memory cache which keeps the current behaviour and will allow to add others.
+  * a new option, `:resolver_cache`, was added, which is `:memory` by default.
+  * this fixes an issue introduced for multi-ractor support where the cache store would not be thread-safe when used in a non main ractor.
+* `:auth` plugin: when loaded with the `:retries` plugin, and the auth value method is dynamic/callable, will recover out-of-the-box from 401 HTTP responses by retrying the request with a newly-generated token.
+  * the `:rate_limiter` plugin will use this work to retry rate-limited responses without having to set setting `:retry_change_requests` to true, thereby eliminating a potential issue with non-idempotent requests.
+
+## Bugfixes
+
+* HTTP1 parser: clear buffer on reset.
+* http1 fix: handle the case in `#handle_error` where the response is an error response
+* `:stream_bidi` plugin: fix internal state preventing bidi requests from being retried.
+* `:stream_bidi` plugin: will only allow initial request body being passed using `:body` param (others, like `:json`, will raise an exception)
+* https resolver: return `:idle` on `#state` calls when no connection is available (sometimes called in internal log messages).
+* selector loop fix: when there are no selectables and an interval is passed, sleep instead of returning (thereby avoiding potential busy loop).
+*

data/lib/httpx/adapters/datadog.rb

@@ -80,7 +80,7 @@ module Datadog::Tracing
             else
               span.set_tag(TAG_STATUS_CODE, response.status.to_s)
 
-              span.set_error(::HTTPX::HTTPError.new(response)) if response.status >= 400 && response.status <= 599
+              span.set_error(::HTTPX::HTTPError.new(response)) if response.status.between?(400, 599)
 
               span.set_tags(
                 Datadog.configuration.tracing.header_tags.response_tags(response.headers.to_h)

data/lib/httpx/adapters/faraday.rb

@@ -9,7 +9,7 @@ module Faraday
     class HTTPX < Faraday::Adapter
       def initialize(app = nil, opts = {}, &block)
         @connection = @bind = nil
-        super(app, opts, &block)
+        super
       end
 
       module RequestMixin

data/lib/httpx/adapters/webmock.rb

@@ -38,12 +38,15 @@ module WebMock
 
           return build_error_response(request, webmock_response.exception) if webmock_response.exception
 
-          request.options.response_class.new(request,
-                                             webmock_response.status[0],
-                                             "2.0",
-                                             webmock_response.headers).tap do |res|
-            res.mocked = true
-          end
+          request
+            .options
+            .response_class
+            .new(
+              request,
+              webmock_response.status[0],
+              "2.0",
+              webmock_response.headers
+            ).tap(&:mock!)
         end
 
         def build_error_response(request, exception)
@@ -72,17 +75,23 @@ module WebMock
       end
 
       module ResponseMethods
-        attr_accessor :mocked
-
         def initialize(*)
           super
           @mocked = false
         end
+
+        def mock!
+          @mocked = true
+        end
+
+        def mocked?
+          @mocked
+        end
       end
 
       module ResponseBodyMethods
         def decode_chunk(chunk)
-          return chunk if @response.mocked
+          return chunk if @response.mocked?
 
           super
         end

data/lib/httpx/altsvc.rb

@@ -8,6 +8,8 @@ module HTTPX
     module ConnectionMixin
       using URIExtensions
 
+      H2_ALTSVC_SCHEMES = %w[https h2].freeze
+
       def send(request)
         request.headers["alt-used"] = @origin.authority if @parser && !@write_buffer.full? && match_altsvcs?(request.uri)
 
@@ -41,12 +43,12 @@ module HTTPX
       end
 
       def altsvc_match?(uri, other_uri)
-        other_uri = URI(other_uri)
+        other_uri = URI(other_uri) #: http_uri
 
         uri.origin == other_uri.origin || begin
           case uri.scheme
           when "h2"
-            (other_uri.scheme == "https" || other_uri.scheme == "h2") &&
+            H2_ALTSVC_SCHEMES.include?(other_uri.scheme) &&
               uri.host == other_uri.host &&
               uri.port == other_uri.port
           else

data/lib/httpx/connection/http1.rb

@@ -10,6 +10,11 @@ module HTTPX
     MAX_REQUESTS = 200
     CRLF = "\r\n"
 
+    UPCASED = {
+      "www-authenticate" => "WWW-Authenticate",
+      "http2-settings" => "HTTP2-Settings",
+      "content-md5" => "Content-MD5",
+    }.freeze
     attr_reader :pending, :requests
 
     attr_accessor :max_concurrent_requests
@@ -192,9 +197,10 @@ module HTTPX
     end
 
     def handle_error(ex, request = nil)
-      if (ex.is_a?(EOFError) || ex.is_a?(TimeoutError)) && @request && @request.response &&
-         !@request.response.headers.key?("content-length") &&
-         !@request.response.headers.key?("transfer-encoding")
+      if (ex.is_a?(EOFError) || ex.is_a?(TimeoutError)) && @request &&
+         (response = @request.response) && response.is_a?(Response) &&
+         !response.headers.key?("content-length") &&
+         !response.headers.key?("transfer-encoding")
         # if the response does not contain a content-length header, the server closing the
         # connnection is the indicator of response consumed.
         # https://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.4.4
@@ -386,12 +392,6 @@ module HTTPX
       end
     end
 
-    UPCASED = {
-      "www-authenticate" => "WWW-Authenticate",
-      "http2-settings" => "HTTP2-Settings",
-      "content-md5" => "Content-MD5",
-    }.freeze
-
     def capitalized(field)
       UPCASED[field] || field.split("-").map(&:capitalize).join("-")
     end

data/lib/httpx/connection/http2.rb

@@ -3,6 +3,8 @@
 require "securerandom"
 require "http/2"
 
+HTTP2::Connection.__send__(:public, :send_buffer) if HTTP2::VERSION < "1.1.1"
+
 module HTTPX
   class Connection::HTTP2
     include Callbacks

data/lib/httpx/connection.rb

@@ -100,14 +100,13 @@ module HTTPX
     def match?(uri, options)
       return false if !used? && (@state == :closing || @state == :closed)
 
-      (
-        @origins.include?(uri.origin) &&
+      @origins.include?(uri.origin) &&
         # if there is more than one origin to match, it means that this connection
         # was the result of coalescing. To prevent blind trust in the case where the
         # origin came from an ORIGIN frame, we're going to verify the hostname with the
         # SSL certificate
-        (@origins.size == 1 || @origin == uri.origin || (@io.is_a?(SSL) && @io.verify_hostname(uri.host)))
-      ) && @options == options
+        (@origins.size == 1 || @origin == uri.origin || (@io.is_a?(SSL) && @io.verify_hostname(uri.host))) &&
+        @options == options
     end
 
     def mergeable?(connection)
@@ -146,10 +145,6 @@ module HTTPX
       end
     end
 
-    def create_idle(options = {})
-      self.class.new(@origin, @options.merge(options))
-    end
-
     def merge(connection)
       @origins |= connection.instance_variable_get(:@origins)
       if @ssl_session.nil? && connection.ssl_session
@@ -729,7 +724,7 @@ module HTTPX
 
         disconnect
       when :closing
-        return unless @state == :idle || @state == :open
+        return unless connecting? || @state == :open
 
         unless @write_buffer.empty?
           # preset state before handshake, as error callbacks
@@ -855,6 +850,9 @@ module HTTPX
       end
     end
 
+    # recover internal state and emit all relevant error responses when +error+ was raised.
+    # this takes an optiona +request+ which may have already been handled and can be opted out
+    # in the state recovery process.
     def handle_error(error, request = nil)
       parser.handle_error(error, request) if @parser && @parser.respond_to?(:handle_error)
       while (req = @pending.shift)

data/lib/httpx/domain_name.rb

@@ -55,7 +55,7 @@ module HTTPX
       def new(domain)
         return domain if domain.is_a?(self)
 
-        super(domain)
+        super
       end
 
       # Normalizes a _domain_ using the Punycode algorithm as necessary.

data/lib/httpx/headers.rb

@@ -42,12 +42,12 @@ module HTTPX
     # dupped initialization
     def initialize_dup(orig)
       super
-      @headers = orig.instance_variable_get(:@headers).dup
+      @headers = orig.instance_variable_get(:@headers).transform_values(&:dup)
     end
 
     # freezes the headers hash
     def freeze
-      @headers.freeze
+      @headers.each_value(&:freeze).freeze
       super
     end
 

data/lib/httpx/io/tcp.rb

@@ -111,7 +111,7 @@ module HTTPX
       raise e if @ip_index.negative?
 
       log { "failed connecting to #{@ip} (#{e.message}), evict from cache and trying next..." }
-      Resolver.cached_lookup_evict(@hostname, @ip)
+      @options.resolver_cache.evict(@hostname, @ip)
 
       @io = build_socket
       retry

data/lib/httpx/loggable.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "fiber" if RUBY_VERSION < "3.0.0"
+
 module HTTPX
   module Loggable
     COLORS = {