httpx 0.10.0 → 0.11.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3e3727a5374ad3c2d6d7c5dbfd61040e3aa3ea6c74aa6ad12d662a5bd0b9a7af
-  data.tar.gz: 7b1a2aa15418b03fc276b81cfaf0565bbce59dd531c2bfc10f19ddc60506213b
+  metadata.gz: 3455bd74f9d6c0cf65936bae67bf74b4d1d5fde21d254446cb4a4615a65b9bb9
+  data.tar.gz: a8f4c6d743667725fcfb3f43c42ef0efbcdc9772f84ecd45f558d2d4c94c3345
 SHA512:
-  metadata.gz: cda3ab9929396bdbad4ff38472dea4d7af34946e34cf995228ad5d52d05e37664b53f66e0a0ac8dd10193e55de52b4f7575f203bea8202b5474f0c3cca450d24
-  data.tar.gz: 025a985fc7c05abbb1bd66d23356d5165f7536d6565464e0729c618da624fc9f428e1046bba7c09dc5d7c23ada0587f73fb23e42128c96847857bcc1ccd6d408
+  metadata.gz: 835072956f05fcb3a42b3e128d64e5753409aa32e875426dda24d8ddbbffdd0fc6585a6d48abd48cd470962f0a57492c4b74afe529c4ab5c750d3039fdbe2b3d
+  data.tar.gz: 8c8be2d19b36b4ad37f00a9f8ea9378e2f60e6e65e4f9513e84b24139480f529650e74c2a3c36a200d0e318ce86c2b15be37a2522032225ba0a9055cbac4534b

data/README.md

@@ -1,7 +1,7 @@
 # HTTPX: A Ruby HTTP library for tomorrow... and beyond!
 
 [![Gem Version](https://badge.fury.io/rb/httpx.svg)](http://rubygems.org/gems/httpx)
-[![pipeline status](https://gitlab.com/honeyryderchuck/httpx/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/httpx/commits/master)
+[![pipeline status](https://gitlab.com/honeyryderchuck/httpx/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/httpx/pipelines?page=1&scope=all&ref=master)
 [![coverage report](https://gitlab.com/honeyryderchuck/httpx/badges/master/coverage.svg?job=coverage)](https://honeyryderchuck.gitlab.io/httpx/coverage/#_AllFiles)
 
 HTTPX is an HTTP client library for the Ruby programming language.
@@ -85,7 +85,15 @@ However if the server supports HTTP/1.1, it will try to use HTTP pipelining, fal
 
 ### Clean API
 
-`HTTPX` acknowledges the ease-of-use of the [http gem](https://github.com/httprb/http) API (itself inspired by python [requests](http://docs.python-requests.org/en/latest/) library). It therefore aims at reusing the same facade, extending it for the use cases which the http gem doesn't support.
+`httpx` builds all functions around the `HTTPX` module, so that all calls can compose of each other. Here are a few examples:
+
+```ruby
+response = HTTPX.get("https://www.google.com")
+response = HTTPX.post("https://www.nghttp2.org/httpbin/post", params: {name: "John", age: "22"})
+response = HTTPX.plugin(:basic_authentication)
+                .basic_authentication("user", "pass")
+                .get("https://www.google.com")
+```
 
 ### Lightweight
 
@@ -135,7 +143,7 @@ Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryde
 
 * Discuss your contribution in an issue
 * Fork it
-* Make your changes, add some test
+* Make your changes, add some tests
 * Ensure all tests pass (`bundle exec rake test`)
 * Open a Merge Request (that's Pull Request in Github-ish)
 * Wait for feedback

data/doc/release_notes/0_10_1.md

@@ -0,0 +1,37 @@
+# 0.10.1
+
+## Improvements
+
+### URL-encoded nested params
+
+url encoder now supports nested params, which is a standard of rack-based frameworks:
+
+```ruby
+HTTPX.post("https://httpbin.org/post", form: { a: { b: 1 }, c: [2, 3] })
+# a[b]=1&c[]=2&c[]=3
+```
+
+This encoding scheme is now the standard for URL-encoded request bodies, query params, and `:multipart` plugin requests.
+
+### Socks4 IPv6 addresses
+
+HTTPX supports IPv6 Socks4 proxies now. This support is restricted to rubies where `IPAddr#hton` is implemented though, so you are encouraged to upgrade.
+
+## More verbose HTTP Errors
+
+`HTTPX::Response#raise_for_status` was raising exceptions for client/server HTTP errors codes (4xx/5xx). However, only the status code was part of the message.
+
+From now on, both headers and the responnse payload will also appear, so expected more verbosity, but also more meaningful information.
+
+## Bugfixes
+
+* HTTP/2 and HTTP/1.1 exhausted connections now get properly migrated into a new connection;
+* HTTP/2 421 responses will now correctly migrate the connection and pendign requests to HTTP/1.1 (a hanging loop was being caused);
+* HTTP/2 connection failed with a GOAWAY settings timeout will now return error responses (instead of hanging indefinitely);
+* Non-IP proxy name-resolving errors will now move on to the next available proxy in the list (instead of hanging indefinitely);
+* Non-IP DNS resolve errors for `native` and `https` variants will now return the appropriate error response (instead of hanging indefinitely);
+
+## Chore
+
+* `HTTPX.plugins` is now officially deprecated (use `HTTPX.plugin` instead);
+

data/doc/release_notes/0_10_2.md

@@ -0,0 +1,5 @@
+# 0.10.2
+
+## Bugfixes
+
+Support for nested params in multipart forms introduced a bug where top-level params weren't being spread out correctly in the request body.

data/doc/release_notes/0_11_0.md

@@ -0,0 +1,76 @@
+# 0.11.0
+
+## Features
+
+### Webmock Adapter
+
+`httpx` can now be integrated with `webmock`, a popular HTTP requests stubbing library.
+
+```ruby
+# minitest
+require "webmock/minitest"
+require "httpx/adapters/webmock"
+
+# in rspec
+require "webmock/rspec"
+require "httpx/adapters/webmock"
+
+# and now you're free for mocking
+WebMock.enable!
+stub_http_request(:get, "https://www.google.com").and_return(status: 200, body: "here's google")
+
+```
+
+Read more about it in the [webmock integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Webmock-Adapter).
+
+### Datadog Adapter
+
+`httpx` ships with integration for [ddtrace, datadog's official tracing client](https://github.com/DataDog/dd-trace-rb). You just need to initialize it the following way:
+
+```ruby
+require "ddtrace"
+require "httpx/adapters/datadog"
+
+Datadog.configure do |c|
+  c.use :httpx
+end
+```
+
+A trace will be emitted for every request, so this should be an interesting visualization if concurrent requests are sent.
+
+Customization options and traces are similar to what [the net-http adapter provides](https://docs.datadoghq.com/tracing/setup_overview/setup/ruby/#nethttp).
+
+Read more about it in the [datadog integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Datadog-Adapter).
+
+## Improvements
+
+### Own multipart request encoder
+
+`httpx` now ships with its own multipart formdata encoder, and does not rely on `http-form_data` anymore:
+
+```ruby
+HTTPX.plugin(:multipart).post(uri, form: {file: File.new("path/to/file")})
+```
+
+Read more about it in the [multipart plugin documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Multipart-Uploads), including also about why this was made.
+
+### Expect Plugin
+
+The `:expect` plugin now works reliably when the server does not support the `expect: 100-continue` header, i.e. it'll upload the body after a certain timeout. Building onn that, two behaviours are now implemented:
+
+* A cache of domains which did not respond to the `expect` header is now kept, so that subsequent requests can skip the timeout and immediately upload the payload.
+* If the "100 Continue" response arrives **after** the timeout expired and the body has been uploaded, the domain is removed from the cache, and subsequent requests will send the `expect` header.
+
+### SNI/Host options
+
+Some extension of the API was applied in order to support custom TLS negotiation parameters. You can now pass `:hostname` under the `:ssl` options, and this will be used for the SNI part of the TLS negotiation. This is useful in scenarios where a proxy certificate doesn't apply for the host one wants to send the request to:
+
+```ruby
+response = session.get(proxy_ip, headers: { "host" => upstream_hostname }, ssl: { hostname: sni_hostname }
+```
+
+## Bugfixes
+
+A default 5 second timeout is in-place when using the DNS `:system` resolver, as it was found out that. when using the `resolv` library, the DNS query will not be retried otherwise. You can change this setting py passing `resolver_options: { timeouts: ANOTHER_TIMEOUT}`. In the future, this may become another timeout option, however.
+
+

data/doc/release_notes/0_11_1.md

@@ -0,0 +1 @@
+0_11_1.md

data/doc/release_notes/0_11_2.md

@@ -0,0 +1,5 @@
+# 0.11.2
+
+## Bugfixes
+
+The `:cookies` plugin wasn't able to parse `Expires` values, as it was using `Time.httpdate` to parse timestamps, which is RFC 2616-compliant, whereas cookies datetime values need to be RFC 6265-compliant.

data/lib/httpx/adapters/datadog.rb

@@ -0,0 +1,205 @@
+# frozen_string_literal: true
+
+require "ddtrace/contrib/integration"
+require "ddtrace/contrib/rest_client/configuration/settings"
+require "ddtrace/contrib/rest_client/patcher"
+
+module Datadog
+  module Contrib
+    module HTTPX
+      # HTTPX Datadog Plugin
+      #
+      # Enables tracing for httpx requests. A span will be created for each individual requests,
+      # and it'll trace since the moment it is fed to the connection, until the moment the response is
+      # fed back to the session.
+      #
+      module Plugin
+        class RequestTracer
+          SPAN_REQUEST = "httpx.request"
+
+          def initialize(request)
+            @request = request
+          end
+
+          def call
+            return if skip_tracing?
+
+            @request.on(:response, &method(:finish))
+
+            verb = @request.verb.to_s.upcase
+            uri = @request.uri
+
+            @span = datadog_pin.tracer.trace(SPAN_REQUEST)
+            service_name = datadog_config[:split_by_domain] ? uri.host : datadog_pin.service_name
+
+            begin
+              @span.service = service_name
+              @span.span_type = Datadog::Ext::HTTP::TYPE_OUTBOUND
+              @span.resource = verb
+
+              Datadog::HTTPPropagator.inject!(@span.context, @request.headers) if datadog_pin.tracer.enabled && !skip_distributed_tracing?
+
+              # Add additional request specific tags to the span.
+
+              @span.set_tag(Datadog::Ext::HTTP::URL, @request.path)
+              @span.set_tag(Datadog::Ext::HTTP::METHOD, verb)
+
+              @span.set_tag(Datadog::Ext::NET::TARGET_HOST, uri.host)
+              @span.set_tag(Datadog::Ext::NET::TARGET_PORT, uri.port.to_s)
+
+              # Tag as an external peer service
+              @span.set_tag(Datadog::Ext::Integration::TAG_PEER_SERVICE, @span.service)
+
+              # Set analytics sample rate
+              if Contrib::Analytics.enabled?(datadog_config[:analytics_enabled])
+                Contrib::Analytics.set_sample_rate(@span, datadog_config[:analytics_sample_rate])
+              end
+            rescue StandardError => e
+              Datadog.logger.error("error preparing span for http request: #{e}")
+            end
+          rescue StandardError => e
+            Datadog.logger.debug("Failed to start span: #{e}")
+          end
+
+          def finish(response)
+            return unless @span
+
+            if response.respond_to?(:error)
+              @span.set_error(response.error)
+            else
+              @span.set_tag(Datadog::Ext::HTTP::STATUS_CODE, response.status.to_s)
+
+              @span.set_error(::HTTPX::HTTPError.new(response)) if response.status >= 400 && response.status <= 599
+            end
+
+            @span.finish
+          end
+
+          private
+
+          def skip_tracing?
+            return true if @request.headers.key?(Datadog::Ext::Transport::HTTP::HEADER_META_TRACER_VERSION)
+
+            return false unless @datadog_pin
+
+            span = @datadog_pin.tracer.active_span
+
+            return true if span && (span.name == SPAN_REQUEST)
+
+            false
+          end
+
+          def skip_distributed_tracing?
+            return !datadog_pin.config[:distributed_tracing] if datadog_pin.config && datadog_pin.config.key?(:distributed_tracing)
+
+            !Datadog.configuration[:httpx][:distributed_tracing]
+          end
+
+          def datadog_pin
+            @datadog_pin ||= begin
+              service = datadog_config[:service_name]
+              tracer = datadog_config[:tracer]
+
+              Datadog::Pin.new(
+                service,
+                app: "httpx",
+                app_type: Datadog::Ext::AppTypes::WEB,
+                tracer: -> { tracer }
+              )
+            end
+          end
+
+          def datadog_config
+            @datadog_config ||= Datadog.configuration[:httpx, @request.uri.host]
+          end
+        end
+
+        module ConnectionMethods
+          def send(request)
+            RequestTracer.new(request).call
+            super
+          end
+        end
+      end
+
+      module Configuration
+        # Default settings for httpx
+        #
+        class Settings < Datadog::Contrib::Configuration::Settings
+          option :service_name, default: "httpx"
+          option :distributed_tracing, default: true
+          option :split_by_domain, default: false
+
+          option :enabled do |o|
+            o.default { env_to_bool("DD_TRACE_HTTPX_ENABLED", true) }
+            o.lazy
+          end
+
+          option :analytics_enabled do |o|
+            o.default { env_to_bool(%w[DD_TRACE_HTTPX_ANALYTICS_ENABLED DD_HTTPX_ANALYTICS_ENABLED], false) }
+            o.lazy
+          end
+
+          option :analytics_sample_rate do |o|
+            o.default { env_to_float(%w[DD_TRACE_HTTPX_ANALYTICS_SAMPLE_RATE DD_HTTPX_ANALYTICS_SAMPLE_RATE], 1.0) }
+            o.lazy
+          end
+
+          option :error_handler, default: Datadog::Tracer::DEFAULT_ON_ERROR
+        end
+      end
+
+      # Patcher enables patching of 'httpx' with datadog components.
+      #
+      module Patcher
+        include Datadog::Contrib::Patcher
+
+        module_function
+
+        def target_version
+          Integration.version
+        end
+
+        # loads a session instannce with the datadog plugin, and replaces the
+        # base HTTPX::Session with the patched session class.
+        def patch
+          datadog_session = ::HTTPX.plugin(Plugin)
+
+          ::HTTPX.send(:remove_const, :Session)
+          ::HTTPX.send(:const_set, :Session, datadog_session.class)
+        end
+      end
+
+      # Datadog Integration for HTTPX.
+      #
+      class Integration
+        include Contrib::Integration
+
+        # MINIMUM_VERSION = Gem::Version.new('0.11.0')
+        MINIMUM_VERSION = Gem::Version.new("0.10.2")
+
+        register_as :httpx
+
+        def self.version
+          Gem.loaded_specs["httpx"] && Gem.loaded_specs["httpx"].version
+        end
+
+        def self.loaded?
+          defined?(::HTTPX::Request)
+        end
+
+        def self.compatible?
+          super && version >= MINIMUM_VERSION
+        end
+
+        def default_configuration
+          Configuration::Settings.new
+        end
+
+        def patcher
+          Patcher
+        end
+      end
+    end
+  end
+end

data/lib/httpx/adapters/faraday.rb

@@ -65,7 +65,6 @@ module Faraday
         plugin(:compression)
         plugin(:persistent)
 
-        # :nocov:
         module ReasonPlugin
           if RUBY_VERSION < "2.5"
             def self.load_dependencies(*)
@@ -88,7 +87,6 @@ module Faraday
             end
           end
         end
-        # :nocov:
         plugin(ReasonPlugin)
       end
 

data/lib/httpx/adapters/webmock.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+module WebMock
+  module HttpLibAdapters
+    if RUBY_VERSION < "2.5"
+      require "webrick/httpstatus"
+      HTTP_REASONS = WEBrick::HTTPStatus::StatusMessage
+    else
+      require "net/http/status"
+      HTTP_REASONS = Net::HTTP::STATUS_CODES
+    end
+
+    #
+    # HTTPX plugin for webmock.
+    #
+    # Requests are "hijacked" at the session, before they're distributed to a connection.
+    #
+    module Plugin
+      module InstanceMethods
+        private
+
+        def send_requests(*requests, options)
+          request_signatures = requests.map do |request|
+            request_signature = _build_webmock_request_signature(request)
+            WebMock::RequestRegistry.instance.requested_signatures.put(request_signature)
+            request_signature
+          end
+
+          responses = request_signatures.map do |request_signature|
+            WebMock::StubRegistry.instance.response_for_request(request_signature)
+          end
+
+          real_requests = {}
+
+          requests.each_with_index.each_with_object([request_signatures, responses]) do |(request, idx), (sig_reqs, mock_responses)|
+            if (webmock_response = mock_responses[idx])
+              mock_responses[idx] = _build_from_webmock_response(request, webmock_response)
+              WebMock::CallbackRegistry.invoke_callbacks({ lib: :httpx }, sig_reqs[idx], webmock_response)
+              log { "mocking #{request.uri} with #{mock_responses[idx].inspect}" }
+            elsif WebMock.net_connect_allowed?(sig_reqs[idx].uri)
+              log { "performing #{request.uri}" }
+              real_requests[request] = idx
+            else
+              raise WebMock::NetConnectNotAllowedError, sig_reqs[idx]
+            end
+          end
+
+          unless real_requests.empty?
+            reqs = real_requests.keys
+            reqs.zip(super(*reqs, options)).each do |req, res|
+              idx = real_requests[req]
+
+              if WebMock::CallbackRegistry.any_callbacks?
+                webmock_response = _build_webmock_response(req, res)
+                WebMock::CallbackRegistry.invoke_callbacks(
+                  { lib: :httpx, real_request: true }, request_signatures[idx],
+                  webmock_response
+                )
+              end
+
+              responses[idx] = res
+            end
+          end
+
+          responses
+        end
+
+        def _build_webmock_request_signature(request)
+          uri = WebMock::Util::URI.heuristic_parse(request.uri)
+          uri.path = uri.normalized_path.gsub("[^:]//", "/")
+
+          WebMock::RequestSignature.new(
+            request.verb,
+            uri.to_s,
+            body: request.body.each.to_a.join,
+            headers: request.headers.to_h
+          )
+        end
+
+        def _build_webmock_response(_request, response)
+          webmock_response = WebMock::Response.new
+          webmock_response.status = [response.status, HTTP_REASONS[response.status]]
+          webmock_response.body = response.body.to_s
+          webmock_response.headers = response.headers.to_h
+          webmock_response
+        end
+
+        def _build_from_webmock_response(request, webmock_response)
+          return ErrorResponse.new(request, webmock_response.exception, request.options) if webmock_response.exception
+
+          response = request.options.response_class.new(request,
+                                                        webmock_response.status[0],
+                                                        "2.0",
+                                                        webmock_response.headers)
+          response << webmock_response.body.dup
+          response
+        end
+      end
+    end
+
+    class HttpxAdapter < HttpLibAdapter
+      adapter_for :httpx
+
+      class << self
+        def enable!
+          @original_session = ::HTTPX::Session
+
+          webmock_session = ::HTTPX.plugin(Plugin)
+
+          ::HTTPX.send(:remove_const, :Session)
+          ::HTTPX.send(:const_set, :Session, webmock_session.class)
+        end
+
+        def disable!
+          return unless @original_session
+
+          HTTPX.send(:remove_const, :Session)
+          HTTPX.send(:const_set, :Session, @original_session)
+        end
+      end
+    end
+  end
+end