httpx 0.10.2 → 0.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2dee09976c993c9a7cb840410ce8758e68faaae6d13d1101a0cf207b59eebdd4
-  data.tar.gz: 444e4c0373f4c3faed53b1b2238e338aaac70af61b24657b2b79ce3f509bffde
+  metadata.gz: 5770fadc8d4604ccb0fb274c7fc157802315f68c749a83973ee5596fee8effb1
+  data.tar.gz: 4f24cca053093be31636405dcb740f5c6b2599197d9ebdfb90cbf127e86a5ffc
 SHA512:
-  metadata.gz: 4d113f172d81c88f155b0bc00c1a42c76fc01c6eed31df693c2ed5c43190e4887c6331492b33acb3104c16de49ec34023fd3c22eb199bfa363392af5fd8ce414
-  data.tar.gz: defb46daafc3fba4f13e765531b61892e5fedd96de9ffdd4ab0e10132ac4df3a5e99e94baf032578cef2919c29069ac3c080c25e56b93907e8451c573a245972
+  metadata.gz: 48fe64207a82e2b8db91b73cc6252ff48ea624e57dcbcbfa30e5f6986e1936e61bbdc83cece34c470dd6e318f41845a306a57e2b5c8710084444b6193786a1eb
+  data.tar.gz: 70b43fc624a8452187a730e39f55d9e92d16438babbd76aa2346c5b3dd0c2cad7f8e5e787f7c4ab1917c52ed750c7656f2f6c5e92a2ec0fce9cc0a76e5994d1e

data/README.md

@@ -1,7 +1,7 @@
 # HTTPX: A Ruby HTTP library for tomorrow... and beyond!
 
 [![Gem Version](https://badge.fury.io/rb/httpx.svg)](http://rubygems.org/gems/httpx)
-[![pipeline status](https://gitlab.com/honeyryderchuck/httpx/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/httpx/commits/master)
+[![pipeline status](https://gitlab.com/honeyryderchuck/httpx/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/httpx/pipelines?page=1&scope=all&ref=master)
 [![coverage report](https://gitlab.com/honeyryderchuck/httpx/badges/master/coverage.svg?job=coverage)](https://honeyryderchuck.gitlab.io/httpx/coverage/#_AllFiles)
 
 HTTPX is an HTTP client library for the Ruby programming language.
@@ -85,7 +85,15 @@ However if the server supports HTTP/1.1, it will try to use HTTP pipelining, fal
 
 ### Clean API
 
-`HTTPX` acknowledges the ease-of-use of the [http gem](https://github.com/httprb/http) API (itself inspired by python [requests](http://docs.python-requests.org/en/latest/) library). It therefore aims at reusing the same facade, extending it for the use cases which the http gem doesn't support.
+`httpx` builds all functions around the `HTTPX` module, so that all calls can compose of each other. Here are a few examples:
+
+```ruby
+response = HTTPX.get("https://www.google.com")
+response = HTTPX.post("https://www.nghttp2.org/httpbin/post", params: {name: "John", age: "22"})
+response = HTTPX.plugin(:basic_authentication)
+                .basic_authentication("user", "pass")
+                .get("https://www.google.com")
+```
 
 ### Lightweight
 
@@ -105,7 +113,7 @@ The test suite runs against [httpbin proxied over nghttp2](https://nghttp2.org/h
 
 ## Supported Rubies
 
-All Rubies greater or equal to 2.1, and always latest JRuby.
+All Rubies greater or equal to 2.1, and always latest JRuby and Truffleruby.
 
 **Note**: This gem is tested against all latest patch versions, i.e. if you're using 2.2.0 and you experience some issue, please test it against 2.2.10 (latest patch version of 2.2) before creating an issue.
 
@@ -125,7 +133,7 @@ All Rubies greater or equal to 2.1, and always latest JRuby.
 
 If your requirement is to run requests over HTTP/2 and TLS, make sure you run a version of the gem which compiles OpenSSL 1.0.2 (Ruby 2.3 and higher are guaranteed to).
 
-JRuby's `openssl` is based on Bouncy Castle, which is massively outdated and still doesn't implement ALPN. So HTTP/2 over TLS/ALPN negotiation is off until JRuby figures this out.
+In order to use HTTP/2 under JRuby, [check this link](https://gitlab.com/honeyryderchuck/httpx/-/wikis/JRuby-Truffleruby-Other-Rubies) to know what to do.
 
 ### Known bugs
 
@@ -135,7 +143,7 @@ Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryde
 
 * Discuss your contribution in an issue
 * Fork it
-* Make your changes, add some test
+* Make your changes, add some tests
 * Ensure all tests pass (`bundle exec rake test`)
 * Open a Merge Request (that's Pull Request in Github-ish)
 * Wait for feedback

data/doc/release_notes/0_11_0.md

@@ -0,0 +1,76 @@
+# 0.11.0
+
+## Features
+
+### Webmock Adapter
+
+`httpx` can now be integrated with `webmock`, a popular HTTP requests stubbing library.
+
+```ruby
+# minitest
+require "webmock/minitest"
+require "httpx/adapters/webmock"
+
+# in rspec
+require "webmock/rspec"
+require "httpx/adapters/webmock"
+
+# and now you're free for mocking
+WebMock.enable!
+stub_http_request(:get, "https://www.google.com").and_return(status: 200, body: "here's google")
+
+```
+
+Read more about it in the [webmock integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Webmock-Adapter).
+
+### Datadog Adapter
+
+`httpx` ships with integration for [ddtrace, datadog's official tracing client](https://github.com/DataDog/dd-trace-rb). You just need to initialize it the following way:
+
+```ruby
+require "ddtrace"
+require "httpx/adapters/datadog"
+
+Datadog.configure do |c|
+  c.use :httpx
+end
+```
+
+A trace will be emitted for every request, so this should be an interesting visualization if concurrent requests are sent.
+
+Customization options and traces are similar to what [the net-http adapter provides](https://docs.datadoghq.com/tracing/setup_overview/setup/ruby/#nethttp).
+
+Read more about it in the [datadog integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Datadog-Adapter).
+
+## Improvements
+
+### Own multipart request encoder
+
+`httpx` now ships with its own multipart formdata encoder, and does not rely on `http-form_data` anymore:
+
+```ruby
+HTTPX.plugin(:multipart).post(uri, form: {file: File.new("path/to/file")})
+```
+
+Read more about it in the [multipart plugin documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Multipart-Uploads), including also about why this was made.
+
+### Expect Plugin
+
+The `:expect` plugin now works reliably when the server does not support the `expect: 100-continue` header, i.e. it'll upload the body after a certain timeout. Building onn that, two behaviours are now implemented:
+
+* A cache of domains which did not respond to the `expect` header is now kept, so that subsequent requests can skip the timeout and immediately upload the payload.
+* If the "100 Continue" response arrives **after** the timeout expired and the body has been uploaded, the domain is removed from the cache, and subsequent requests will send the `expect` header.
+
+### SNI/Host options
+
+Some extension of the API was applied in order to support custom TLS negotiation parameters. You can now pass `:hostname` under the `:ssl` options, and this will be used for the SNI part of the TLS negotiation. This is useful in scenarios where a proxy certificate doesn't apply for the host one wants to send the request to:
+
+```ruby
+response = session.get(proxy_ip, headers: { "host" => upstream_hostname }, ssl: { hostname: sni_hostname }
+```
+
+## Bugfixes
+
+A default 5 second timeout is in-place when using the DNS `:system` resolver, as it was found out that. when using the `resolv` library, the DNS query will not be retried otherwise. You can change this setting py passing `resolver_options: { timeouts: ANOTHER_TIMEOUT}`. In the future, this may become another timeout option, however.
+
+

data/doc/release_notes/0_11_1.md

@@ -0,0 +1,5 @@
+# 0.11.1
+
+## Bugfixes
+
+Fixed a bug related to the `:compression` plugin trying to process a last empty DATA frame from an HTTP/2 response, after it had been closed.

data/doc/release_notes/0_11_2.md

@@ -0,0 +1,5 @@
+# 0.11.2
+
+## Bugfixes
+
+The `:cookies` plugin wasn't able to parse `Expires` values, as it was using `Time.httpdate` to parse timestamps, which is RFC 2616-compliant, whereas cookies datetime values need to be RFC 6265-compliant.

data/doc/release_notes/0_11_3.md

@@ -0,0 +1,5 @@
+# 0.11.3
+
+## Bugfixes
+
+`EOFError` was being thrown when HTTP/1.1 non-chunked responses which don't contain a "content-length" header. The RFC mandates that the socket should be consumed until the server closes it, so a patch was implemented, to return the available response in such cases.

data/doc/release_notes/0_12_0.md

@@ -0,0 +1,55 @@
+# 0.12.0
+
+## Features
+
+### AWS Sigv4 Authentication Plugin
+
+A new plugin, `:aws_sigv4`, is now shipped with `httpx`. It implements the [AWS Signature Version 4 request signing process](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html), a well documented way of authenticating requests to AWS services, which has since been adopted by other cloud providers, such as Google Cloud Storage.
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#sessionaws_sigv4_authentication
+
+For convenience, there's a derivative plugin, `:aws_sdk_authentication`, which builds on top of `:aws_sigv4`, and integrates with the `aws-sdk-core` gem, maintained by AWS, to resolve the authentication credentials (p.ex. if you support ephemeral access keys).
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#sessionaws_sdk_authentication
+
+Other FAQ: https://gitlab.com/honeyryderchuck/httpx/-/wikis/AWS-Sigv4#faqs
+
+### HTTP/2 support for JRuby
+
+`jruby-openssl` doesn't support ALPN protocol negotiation, nor are there plans to implement, which limited the seamless HTTP/2 usage in `httpx`. A new connection adapter was therefore added specifically for JRuby, where ssl/tls connections will be handled using ffi-based openssl bindings, provided you bundle `ffi-compiler` and `concurrent-ruby`, and install a TLS/1.2-compatible `openssl` package.
+
+See how to use it here: https://gitlab.com/honeyryderchuck/httpx/-/wikis/JRuby-Truffleruby-Other-Rubies#http2
+
+## Improvements
+
+
+### truffleruby support
+
+`httpx` supports and tests against `truffleruby` (known to run tests since v20.3, passing all tests since v21).
+
+### Performance
+
+Several optimizations were introduced:
+
+* Reduction in read/write system calls;
+* more usage of `String#byteslice` in parsing (instead of string mutation);
+* Avoid selection on connections with no outstanding requests;
+
+They all contributed to a massive performance improvement, itself reflected in test runs, which need half the time they used to to complete.
+
+### APIs
+
+* `HTTPX::ErrorResponse#to_s` now uses the exception full message, instead of just the backtrace.
+
+## Bugfixes
+
+* HTTP/2 stream protocol errors do not cause the process to hang (instead, error responnses are yielded);
+* Fixed body stream bugs on retries when error causing retry would happen mid-transfer;
+* Fixed `:multipart` plugin body rewind on retries to start the transfer from the beginning;
+* Fixed auto-load of  `:proxy` plugin when `HTTPS_PROXY` or `HTTP_PROXY` is set;
+* Errno::EPIPE errors mid transfer now cause `httpx` to read from the server and get the appropriate HTTP error response;
+* Make sure that all requests have an error responnse if the error happens early;
+* Fixed TCP handshake Errno::INPROGRESS handling inside TLS connnections, which was causing the process to hang in a high handshake contention scenario;
+* Do not call the event loop if there's nothing to listen on (the DoH resolver was being listened on even if there was nothing to be request);
+* Fixed double event registry for DoH resolvers;
+* 

data/lib/httpx.rb

@@ -19,7 +19,6 @@ require "httpx/headers"
 require "httpx/request"
 require "httpx/response"
 require "httpx/chainable"
-require "httpx/session"
 
 # Top-Level Namespace
 #
@@ -59,3 +58,5 @@ module HTTPX
 
   extend Chainable
 end
+
+require "httpx/session"

data/lib/httpx/adapters/datadog.rb

@@ -0,0 +1,205 @@
+# frozen_string_literal: true
+
+require "ddtrace/contrib/integration"
+require "ddtrace/contrib/rest_client/configuration/settings"
+require "ddtrace/contrib/rest_client/patcher"
+
+module Datadog
+  module Contrib
+    module HTTPX
+      # HTTPX Datadog Plugin
+      #
+      # Enables tracing for httpx requests. A span will be created for each individual requests,
+      # and it'll trace since the moment it is fed to the connection, until the moment the response is
+      # fed back to the session.
+      #
+      module Plugin
+        class RequestTracer
+          SPAN_REQUEST = "httpx.request"
+
+          def initialize(request)
+            @request = request
+          end
+
+          def call
+            return if skip_tracing?
+
+            @request.on(:response, &method(:finish))
+
+            verb = @request.verb.to_s.upcase
+            uri = @request.uri
+
+            @span = datadog_pin.tracer.trace(SPAN_REQUEST)
+            service_name = datadog_config[:split_by_domain] ? uri.host : datadog_pin.service_name
+
+            begin
+              @span.service = service_name
+              @span.span_type = Datadog::Ext::HTTP::TYPE_OUTBOUND
+              @span.resource = verb
+
+              Datadog::HTTPPropagator.inject!(@span.context, @request.headers) if datadog_pin.tracer.enabled && !skip_distributed_tracing?
+
+              # Add additional request specific tags to the span.
+
+              @span.set_tag(Datadog::Ext::HTTP::URL, @request.path)
+              @span.set_tag(Datadog::Ext::HTTP::METHOD, verb)
+
+              @span.set_tag(Datadog::Ext::NET::TARGET_HOST, uri.host)
+              @span.set_tag(Datadog::Ext::NET::TARGET_PORT, uri.port.to_s)
+
+              # Tag as an external peer service
+              @span.set_tag(Datadog::Ext::Integration::TAG_PEER_SERVICE, @span.service)
+
+              # Set analytics sample rate
+              if Contrib::Analytics.enabled?(datadog_config[:analytics_enabled])
+                Contrib::Analytics.set_sample_rate(@span, datadog_config[:analytics_sample_rate])
+              end
+            rescue StandardError => e
+              Datadog.logger.error("error preparing span for http request: #{e}")
+            end
+          rescue StandardError => e
+            Datadog.logger.debug("Failed to start span: #{e}")
+          end
+
+          def finish(response)
+            return unless @span
+
+            if response.respond_to?(:error)
+              @span.set_error(response.error)
+            else
+              @span.set_tag(Datadog::Ext::HTTP::STATUS_CODE, response.status.to_s)
+
+              @span.set_error(::HTTPX::HTTPError.new(response)) if response.status >= 400 && response.status <= 599
+            end
+
+            @span.finish
+          end
+
+          private
+
+          def skip_tracing?
+            return true if @request.headers.key?(Datadog::Ext::Transport::HTTP::HEADER_META_TRACER_VERSION)
+
+            return false unless @datadog_pin
+
+            span = @datadog_pin.tracer.active_span
+
+            return true if span && (span.name == SPAN_REQUEST)
+
+            false
+          end
+
+          def skip_distributed_tracing?
+            return !datadog_pin.config[:distributed_tracing] if datadog_pin.config && datadog_pin.config.key?(:distributed_tracing)
+
+            !Datadog.configuration[:httpx][:distributed_tracing]
+          end
+
+          def datadog_pin
+            @datadog_pin ||= begin
+              service = datadog_config[:service_name]
+              tracer = datadog_config[:tracer]
+
+              Datadog::Pin.new(
+                service,
+                app: "httpx",
+                app_type: Datadog::Ext::AppTypes::WEB,
+                tracer: -> { tracer }
+              )
+            end
+          end
+
+          def datadog_config
+            @datadog_config ||= Datadog.configuration[:httpx, @request.uri.host]
+          end
+        end
+
+        module ConnectionMethods
+          def send(request)
+            RequestTracer.new(request).call
+            super
+          end
+        end
+      end
+
+      module Configuration
+        # Default settings for httpx
+        #
+        class Settings < Datadog::Contrib::Configuration::Settings
+          option :service_name, default: "httpx"
+          option :distributed_tracing, default: true
+          option :split_by_domain, default: false
+
+          option :enabled do |o|
+            o.default { env_to_bool("DD_TRACE_HTTPX_ENABLED", true) }
+            o.lazy
+          end
+
+          option :analytics_enabled do |o|
+            o.default { env_to_bool(%w[DD_TRACE_HTTPX_ANALYTICS_ENABLED DD_HTTPX_ANALYTICS_ENABLED], false) }
+            o.lazy
+          end
+
+          option :analytics_sample_rate do |o|
+            o.default { env_to_float(%w[DD_TRACE_HTTPX_ANALYTICS_SAMPLE_RATE DD_HTTPX_ANALYTICS_SAMPLE_RATE], 1.0) }
+            o.lazy
+          end
+
+          option :error_handler, default: Datadog::Tracer::DEFAULT_ON_ERROR
+        end
+      end
+
+      # Patcher enables patching of 'httpx' with datadog components.
+      #
+      module Patcher
+        include Datadog::Contrib::Patcher
+
+        module_function
+
+        def target_version
+          Integration.version
+        end
+
+        # loads a session instannce with the datadog plugin, and replaces the
+        # base HTTPX::Session with the patched session class.
+        def patch
+          datadog_session = ::HTTPX.plugin(Plugin)
+
+          ::HTTPX.send(:remove_const, :Session)
+          ::HTTPX.send(:const_set, :Session, datadog_session.class)
+        end
+      end
+
+      # Datadog Integration for HTTPX.
+      #
+      class Integration
+        include Contrib::Integration
+
+        # MINIMUM_VERSION = Gem::Version.new('0.11.0')
+        MINIMUM_VERSION = Gem::Version.new("0.10.2")
+
+        register_as :httpx
+
+        def self.version
+          Gem.loaded_specs["httpx"] && Gem.loaded_specs["httpx"].version
+        end
+
+        def self.loaded?
+          defined?(::HTTPX::Request)
+        end
+
+        def self.compatible?
+          super && version >= MINIMUM_VERSION
+        end
+
+        def default_configuration
+          Configuration::Settings.new
+        end
+
+        def patcher
+          Patcher
+        end
+      end
+    end
+  end
+end

data/lib/httpx/adapters/faraday.rb

@@ -65,7 +65,6 @@ module Faraday
         plugin(:compression)
         plugin(:persistent)
 
-        # :nocov:
         module ReasonPlugin
           if RUBY_VERSION < "2.5"
             def self.load_dependencies(*)
@@ -88,7 +87,6 @@ module Faraday
             end
           end
         end
-        # :nocov:
         plugin(ReasonPlugin)
       end
 
@@ -125,11 +123,9 @@ module Faraday
           end
 
           def method_missing(meth, *args, &blk)
-            if @env && @env.respond_to?(meth)
-              @env.__send__(meth, *args, &blk)
-            else
-              super
-            end
+            return super unless @env && @env.respond_to?(meth)
+
+            @env.__send__(meth, *args, &blk)
           end
         end
 
@@ -199,7 +195,7 @@ module Faraday
           response_headers.merge!(response.headers)
         end
         @app.call(env)
-      rescue OpenSSL::SSL::SSLError => e
+      rescue ::HTTPX::TLSError => e
         raise SSL_ERROR, e
       rescue Errno::ECONNABORTED,
              Errno::ECONNREFUSED,