httpx 0.9.0 → 0.11.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e15b3e81842118829ecc251093ce5e80ad0c82dada2c7b2588004b2bf7313f63
-  data.tar.gz: 9d4efcb47f0c269f2c5b9a3c950fe8069c3b76ffb2f9624b557d29a702704712
+  metadata.gz: 9b21be17b0c6a6d3e2f60a97582b88f33dc1f62e09e5555cabfc2d6bea1f06a7
+  data.tar.gz: e0ca927150864a80a8050cc9fbc2c23e58b55452b536ce4301eceecebf1c6d37
 SHA512:
-  metadata.gz: 9fbd146e351da8f603c8b00c236d92c7a2a3c383296736f271c88b0ac310086adf0127af7b630e9087ec45c4a22494128b2efafdb2592de43df4aae4a68ef1f3
-  data.tar.gz: eecb817b76bc0f217b728c7e862f73758e4b2db9c6a943311e0c7d10d8e7fe6bfa8e56a7c0d4593158fb605927813a1e720ec85a9d7e70aae8ecce015f5da647
+  metadata.gz: 1493d50a5b49dd7c83f48078d582f874cc204d3e5df3442d7ea8308b31e6fd011e30cc3fdc5f1825d0daebd270c22af387700ab8f6d2fb9183d2e1fb63bd7265
+  data.tar.gz: ac48ea08cacc65a4948fea8a2abdd50fe7d7fd30c6260400cd61197ea7e6e8d87de2b4750ca1452f242569bbf6dc304c14865f809e7871a4f55d33e7e5543c03

data/LICENSE.txt

@@ -189,3 +189,51 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
+
+
+* lib/httpx/domain_name.rb
+
+This file is derived from the implementation of punycode available at
+here:
+
+https://www.verisign.com/en_US/channel-resources/domain-registry-products/idn-sdks/index.xhtml
+
+Copyright (C) 2000-2002 Verisign Inc., All rights reserved.
+
+Redistribution and use in source and binary forms, with or
+without modification, are permitted provided that the following
+conditions are met:
+
+ 1) Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+
+ 2) Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+
+ 3) Neither the name of the VeriSign Inc. nor the names of its
+    contributors may be used to endorse or promote products derived
+    from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+This software is licensed under the BSD open source license. For more
+information visit www.opensource.org.
+
+Authors:
+ John Colosi (VeriSign)
+ Srikanth Veeramachaneni (VeriSign)
+ Nagesh Chigurupati (Verisign)
+ Praveen Srinivasan(Verisign)

data/README.md

@@ -1,7 +1,7 @@
 # HTTPX: A Ruby HTTP library for tomorrow... and beyond!
 
 [![Gem Version](https://badge.fury.io/rb/httpx.svg)](http://rubygems.org/gems/httpx)
-[![pipeline status](https://gitlab.com/honeyryderchuck/httpx/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/httpx/commits/master)
+[![pipeline status](https://gitlab.com/honeyryderchuck/httpx/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/httpx/pipelines?page=1&scope=all&ref=master)
 [![coverage report](https://gitlab.com/honeyryderchuck/httpx/badges/master/coverage.svg?job=coverage)](https://honeyryderchuck.gitlab.io/httpx/coverage/#_AllFiles)
 
 HTTPX is an HTTP client library for the Ruby programming language.
@@ -18,6 +18,7 @@ Among its features, it supports:
 And also:
 
 * Compression (gzip, deflate, brotli)
+* Streaming Requests
 * Authentication (Basic Auth, Digest Auth)
 * Expect 100-continue
 * Multipart Requests
@@ -25,6 +26,7 @@ And also:
 * HTTP/2 Server Push
 * H2C Upgrade
 * Automatic follow redirects
+* International Domain Names
 
 ## How
 
@@ -83,7 +85,15 @@ However if the server supports HTTP/1.1, it will try to use HTTP pipelining, fal
 
 ### Clean API
 
-`HTTPX` acknowledges the ease-of-use of the [http gem](https://github.com/httprb/http) API (itself inspired by python [requests](http://docs.python-requests.org/en/latest/) library). It therefore aims at reusing the same facade, extending it for the use cases which the http gem doesn't support.
+`httpx` builds all functions around the `HTTPX` module, so that all calls can compose of each other. Here are a few examples:
+
+```ruby
+response = HTTPX.get("https://www.google.com")
+response = HTTPX.post("https://www.nghttp2.org/httpbin/post", params: {name: "John", age: "22"})
+response = HTTPX.plugin(:basic_authentication)
+                .basic_authentication("user", "pass")
+                .get("https://www.google.com")
+```
 
 ### Lightweight
 
@@ -133,7 +143,7 @@ Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryde
 
 * Discuss your contribution in an issue
 * Fork it
-* Make your changes, add some test
+* Make your changes, add some tests
 * Ensure all tests pass (`bundle exec rake test`)
 * Open a Merge Request (that's Pull Request in Github-ish)
 * Wait for feedback

data/doc/release_notes/0_10_0.md

@@ -0,0 +1,66 @@
+# 0.10.0
+
+## Features
+
+### Streaming Requests
+
+The `stream` plugin adds functionality to handle long-lived stream responses, such as the Twitter Streaming API:
+
+```ruby
+http = HTTPX.plugin(:stream)
+
+http.get(stream_api_endpoint, stream: true).each_line do |line|
+  payload = JSON.parse(line)
+  # do smth with this
+end
+```
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Stream
+
+### Rate Limiter
+
+The `rate_limiter` plugin adds functionality for automatically hooking into rate-limiting responses coming from the server, and waits-and-retries them according to what the server advertises.
+
+```ruby
+HTTPX.plugin(:rate_limiter).get(rate_limited_api_endpoint)
+# => 429 Too Many Requests .... Retry-After: 3
+# waits 3 seconds before retrying
+```
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Rate-Limiter
+
+### Ruby 3
+
+This release is the first testing against and targeting Ruby 3 and some of the new features.
+
+It ships with RBS signatures for all of the client-facing APIs. There's non 100% typinng coverage yet, but I'm gradually (pun intended) working on it.
+
+
+## Improvements
+
+### IDN support
+
+Requests where the domains are formed by non-ASCII characters, are now supported (if you're using ruby 2.3 or more recent).
+
+```ruby
+HTTPX.get("http://bücher.ch") # it works!
+```
+
+### cookies plugin full implementation
+
+
+The `cookies` plugin is now independent of 3rd-party gems. The motivation for this was that `http-cookie` was dependent of both `domain_name` and `unf` gems, which are currently unusable in ruby 3, and haven't received any update in the last 3 years.
+
+The implementation is still compliant with RFC6265, and all of the features provided in earlier versions were ported, exceptwhen loading the cookie jar stored in a Netscape-format file or Mozilla sqlite database, which were not documented for `httpx` anyway, and I considered too niche to backport. If you feel `httpx` should support those, do let me know.
+
+Some code from these gems, including the ruby punycode implementation, is now part of the source tree, along with its licenses and attribution mentions.
+
+## Bugfixes
+
+
+Several edge-case bugs have been fixed solely by the integration of RBS runtime type checking, including some bugs around closing a connection pool that can cause loops.
+
+
+## Regressions
+
+`HTTPX::ErrorResponse`'s methods `#headers` and `#reason` were removed, as they didn't provide much value. Consider calling `#raise_for_status` or checking the API (`is_a?(HTTPX::ErrorResponse)` or `respond_to?(:error)` are strategies for this).

data/doc/release_notes/0_10_1.md

@@ -0,0 +1,37 @@
+# 0.10.1
+
+## Improvements
+
+### URL-encoded nested params
+
+url encoder now supports nested params, which is a standard of rack-based frameworks:
+
+```ruby
+HTTPX.post("https://httpbin.org/post", form: { a: { b: 1 }, c: [2, 3] })
+# a[b]=1&c[]=2&c[]=3
+```
+
+This encoding scheme is now the standard for URL-encoded request bodies, query params, and `:multipart` plugin requests.
+
+### Socks4 IPv6 addresses
+
+HTTPX supports IPv6 Socks4 proxies now. This support is restricted to rubies where `IPAddr#hton` is implemented though, so you are encouraged to upgrade.
+
+## More verbose HTTP Errors
+
+`HTTPX::Response#raise_for_status` was raising exceptions for client/server HTTP errors codes (4xx/5xx). However, only the status code was part of the message.
+
+From now on, both headers and the responnse payload will also appear, so expected more verbosity, but also more meaningful information.
+
+## Bugfixes
+
+* HTTP/2 and HTTP/1.1 exhausted connections now get properly migrated into a new connection;
+* HTTP/2 421 responses will now correctly migrate the connection and pendign requests to HTTP/1.1 (a hanging loop was being caused);
+* HTTP/2 connection failed with a GOAWAY settings timeout will now return error responses (instead of hanging indefinitely);
+* Non-IP proxy name-resolving errors will now move on to the next available proxy in the list (instead of hanging indefinitely);
+* Non-IP DNS resolve errors for `native` and `https` variants will now return the appropriate error response (instead of hanging indefinitely);
+
+## Chore
+
+* `HTTPX.plugins` is now officially deprecated (use `HTTPX.plugin` instead);
+

data/doc/release_notes/0_10_2.md

@@ -0,0 +1,5 @@
+# 0.10.2
+
+## Bugfixes
+
+Support for nested params in multipart forms introduced a bug where top-level params weren't being spread out correctly in the request body.

data/doc/release_notes/0_11_0.md

@@ -0,0 +1,76 @@
+# 0.11.0
+
+## Features
+
+### Webmock Adapter
+
+`httpx` can now be integrated with `webmock`, a popular HTTP requests stubbing library.
+
+```ruby
+# minitest
+require "webmock/minitest"
+require "httpx/adapters/webmock"
+
+# in rspec
+require "webmock/rspec"
+require "httpx/adapters/webmock"
+
+# and now you're free for mocking
+WebMock.enable!
+stub_http_request(:get, "https://www.google.com").and_return(status: 200, body: "here's google")
+
+```
+
+Read more about it in the [webmock integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Webmock-Adapter).
+
+### Datadog Adapter
+
+`httpx` ships with integration for [ddtrace, datadog's official tracing client](https://github.com/DataDog/dd-trace-rb). You just need to initialize it the following way:
+
+```ruby
+require "ddtrace"
+require "httpx/adapters/datadog"
+
+Datadog.configure do |c|
+  c.use :httpx
+end
+```
+
+A trace will be emitted for every request, so this should be an interesting visualization if concurrent requests are sent.
+
+Customization options and traces are similar to what [the net-http adapter provides](https://docs.datadoghq.com/tracing/setup_overview/setup/ruby/#nethttp).
+
+Read more about it in the [datadog integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Datadog-Adapter).
+
+## Improvements
+
+### Own multipart request encoder
+
+`httpx` now ships with its own multipart formdata encoder, and does not rely on `http-form_data` anymore:
+
+```ruby
+HTTPX.plugin(:multipart).post(uri, form: {file: File.new("path/to/file")})
+```
+
+Read more about it in the [multipart plugin documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Multipart-Uploads), including also about why this was made.
+
+### Expect Plugin
+
+The `:expect` plugin now works reliably when the server does not support the `expect: 100-continue` header, i.e. it'll upload the body after a certain timeout. Building onn that, two behaviours are now implemented:
+
+* A cache of domains which did not respond to the `expect` header is now kept, so that subsequent requests can skip the timeout and immediately upload the payload.
+* If the "100 Continue" response arrives **after** the timeout expired and the body has been uploaded, the domain is removed from the cache, and subsequent requests will send the `expect` header.
+
+### SNI/Host options
+
+Some extension of the API was applied in order to support custom TLS negotiation parameters. You can now pass `:hostname` under the `:ssl` options, and this will be used for the SNI part of the TLS negotiation. This is useful in scenarios where a proxy certificate doesn't apply for the host one wants to send the request to:
+
+```ruby
+response = session.get(proxy_ip, headers: { "host" => upstream_hostname }, ssl: { hostname: sni_hostname }
+```
+
+## Bugfixes
+
+A default 5 second timeout is in-place when using the DNS `:system` resolver, as it was found out that. when using the `resolv` library, the DNS query will not be retried otherwise. You can change this setting py passing `resolver_options: { timeouts: ANOTHER_TIMEOUT}`. In the future, this may become another timeout option, however.
+
+

data/doc/release_notes/0_11_1.md

@@ -0,0 +1 @@
+0_11_1.md

data/lib/httpx.rb

@@ -5,6 +5,8 @@ require "httpx/version"
 require "httpx/extensions"
 
 require "httpx/errors"
+require "httpx/utils"
+require "httpx/domain_name"
 require "httpx/altsvc"
 require "httpx/callbacks"
 require "httpx/loggable"

data/lib/httpx/adapters/datadog.rb

@@ -0,0 +1,205 @@
+# frozen_string_literal: true
+
+require "ddtrace/contrib/integration"
+require "ddtrace/contrib/rest_client/configuration/settings"
+require "ddtrace/contrib/rest_client/patcher"
+
+module Datadog
+  module Contrib
+    module HTTPX
+      # HTTPX Datadog Plugin
+      #
+      # Enables tracing for httpx requests. A span will be created for each individual requests,
+      # and it'll trace since the moment it is fed to the connection, until the moment the response is
+      # fed back to the session.
+      #
+      module Plugin
+        class RequestTracer
+          SPAN_REQUEST = "httpx.request"
+
+          def initialize(request)
+            @request = request
+          end
+
+          def call
+            return if skip_tracing?
+
+            @request.on(:response, &method(:finish))
+
+            verb = @request.verb.to_s.upcase
+            uri = @request.uri
+
+            @span = datadog_pin.tracer.trace(SPAN_REQUEST)
+            service_name = datadog_config[:split_by_domain] ? uri.host : datadog_pin.service_name
+
+            begin
+              @span.service = service_name
+              @span.span_type = Datadog::Ext::HTTP::TYPE_OUTBOUND
+              @span.resource = verb
+
+              Datadog::HTTPPropagator.inject!(@span.context, @request.headers) if datadog_pin.tracer.enabled && !skip_distributed_tracing?
+
+              # Add additional request specific tags to the span.
+
+              @span.set_tag(Datadog::Ext::HTTP::URL, @request.path)
+              @span.set_tag(Datadog::Ext::HTTP::METHOD, verb)
+
+              @span.set_tag(Datadog::Ext::NET::TARGET_HOST, uri.host)
+              @span.set_tag(Datadog::Ext::NET::TARGET_PORT, uri.port.to_s)
+
+              # Tag as an external peer service
+              @span.set_tag(Datadog::Ext::Integration::TAG_PEER_SERVICE, @span.service)
+
+              # Set analytics sample rate
+              if Contrib::Analytics.enabled?(datadog_config[:analytics_enabled])
+                Contrib::Analytics.set_sample_rate(@span, datadog_config[:analytics_sample_rate])
+              end
+            rescue StandardError => e
+              Datadog.logger.error("error preparing span for http request: #{e}")
+            end
+          rescue StandardError => e
+            Datadog.logger.debug("Failed to start span: #{e}")
+          end
+
+          def finish(response)
+            return unless @span
+
+            if response.respond_to?(:error)
+              @span.set_error(response.error)
+            else
+              @span.set_tag(Datadog::Ext::HTTP::STATUS_CODE, response.status.to_s)
+
+              @span.set_error(::HTTPX::HTTPError.new(response)) if response.status >= 400 && response.status <= 599
+            end
+
+            @span.finish
+          end
+
+          private
+
+          def skip_tracing?
+            return true if @request.headers.key?(Datadog::Ext::Transport::HTTP::HEADER_META_TRACER_VERSION)
+
+            return false unless @datadog_pin
+
+            span = @datadog_pin.tracer.active_span
+
+            return true if span && (span.name == SPAN_REQUEST)
+
+            false
+          end
+
+          def skip_distributed_tracing?
+            return !datadog_pin.config[:distributed_tracing] if datadog_pin.config && datadog_pin.config.key?(:distributed_tracing)
+
+            !Datadog.configuration[:httpx][:distributed_tracing]
+          end
+
+          def datadog_pin
+            @datadog_pin ||= begin
+              service = datadog_config[:service_name]
+              tracer = datadog_config[:tracer]
+
+              Datadog::Pin.new(
+                service,
+                app: "httpx",
+                app_type: Datadog::Ext::AppTypes::WEB,
+                tracer: -> { tracer }
+              )
+            end
+          end
+
+          def datadog_config
+            @datadog_config ||= Datadog.configuration[:httpx, @request.uri.host]
+          end
+        end
+
+        module ConnectionMethods
+          def send(request)
+            RequestTracer.new(request).call
+            super
+          end
+        end
+      end
+
+      module Configuration
+        # Default settings for httpx
+        #
+        class Settings < Datadog::Contrib::Configuration::Settings
+          option :service_name, default: "httpx"
+          option :distributed_tracing, default: true
+          option :split_by_domain, default: false
+
+          option :enabled do |o|
+            o.default { env_to_bool("DD_TRACE_HTTPX_ENABLED", true) }
+            o.lazy
+          end
+
+          option :analytics_enabled do |o|
+            o.default { env_to_bool(%w[DD_TRACE_HTTPX_ANALYTICS_ENABLED DD_HTTPX_ANALYTICS_ENABLED], false) }
+            o.lazy
+          end
+
+          option :analytics_sample_rate do |o|
+            o.default { env_to_float(%w[DD_TRACE_HTTPX_ANALYTICS_SAMPLE_RATE DD_HTTPX_ANALYTICS_SAMPLE_RATE], 1.0) }
+            o.lazy
+          end
+
+          option :error_handler, default: Datadog::Tracer::DEFAULT_ON_ERROR
+        end
+      end
+
+      # Patcher enables patching of 'httpx' with datadog components.
+      #
+      module Patcher
+        include Datadog::Contrib::Patcher
+
+        module_function
+
+        def target_version
+          Integration.version
+        end
+
+        # loads a session instannce with the datadog plugin, and replaces the
+        # base HTTPX::Session with the patched session class.
+        def patch
+          datadog_session = ::HTTPX.plugin(Plugin)
+
+          ::HTTPX.send(:remove_const, :Session)
+          ::HTTPX.send(:const_set, :Session, datadog_session.class)
+        end
+      end
+
+      # Datadog Integration for HTTPX.
+      #
+      class Integration
+        include Contrib::Integration
+
+        # MINIMUM_VERSION = Gem::Version.new('0.11.0')
+        MINIMUM_VERSION = Gem::Version.new("0.10.2")
+
+        register_as :httpx
+
+        def self.version
+          Gem.loaded_specs["httpx"] && Gem.loaded_specs["httpx"].version
+        end
+
+        def self.loaded?
+          defined?(::HTTPX::Request)
+        end
+
+        def self.compatible?
+          super && version >= MINIMUM_VERSION
+        end
+
+        def default_configuration
+          Configuration::Settings.new
+        end
+
+        def patcher
+          Patcher
+        end
+      end
+    end
+  end
+end