httpx 0.8.2 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 44d4a0720ab5167dbff45524a71c6950d271a82532e09ce3b3e01f0d348b3e55
-  data.tar.gz: 02cc85986aeb4290c62977b2388567990cf023fa1675d559fc8fbb597db2c434
+  metadata.gz: e5a33fd07fae25308cb791a086313c8dd8aa05cc7bc24504c92c17e20b692688
+  data.tar.gz: 8be24aad50c50bfea6667a94f7afade82dada7d2512177430ea5b04593f2e296
 SHA512:
-  metadata.gz: 661fc0f5198c045779823ef3c256fad4cb19a84ac44c0e99eee2bb842809ad6cc4558b5ec2bb1b827a44b0c960daa07759d1b21466312527efe5bde1420af391
-  data.tar.gz: fe40c074d0ff47ca8dc7781272c836423c0f254b0b70c4a08f35b1d25d00ba58ba7906fb49d0848e5624f5cf91b6a94cdb3abb9a08ffdc459fa9c7cb81be802f
+  metadata.gz: 3b03f47c4ef852af01d9d4eb6895921e6601d33e8c0261b25e34309c663c481a77ce9c5161124efdb02518b1a7c89f3e3440b823348106d492e3d27daf66ab6e
+  data.tar.gz: e4b2961e9d0b6d11348cf81eae7dbb0e744878c47a51ef584429ab89f32eadc24f9f195ceeb68ecfcb553c19b2a333818cba0e867c0cb992bec66b99dcd1def7

data/LICENSE.txt

@@ -189,3 +189,51 @@
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
+
+
+* lib/httpx/domain_name.rb
+
+This file is derived from the implementation of punycode available at
+here:
+
+https://www.verisign.com/en_US/channel-resources/domain-registry-products/idn-sdks/index.xhtml
+
+Copyright (C) 2000-2002 Verisign Inc., All rights reserved.
+
+Redistribution and use in source and binary forms, with or
+without modification, are permitted provided that the following
+conditions are met:
+
+ 1) Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+
+ 2) Redistributions in binary form must reproduce the above copyright
+    notice, this list of conditions and the following disclaimer in
+    the documentation and/or other materials provided with the
+    distribution.
+
+ 3) Neither the name of the VeriSign Inc. nor the names of its
+    contributors may be used to endorse or promote products derived
+    from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
+OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
+AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+This software is licensed under the BSD open source license. For more
+information visit www.opensource.org.
+
+Authors:
+ John Colosi (VeriSign)
+ Srikanth Veeramachaneni (VeriSign)
+ Nagesh Chigurupati (Verisign)
+ Praveen Srinivasan(Verisign)

data/README.md

@@ -1,8 +1,8 @@
 # HTTPX: A Ruby HTTP library for tomorrow... and beyond!
 
 [![Gem Version](https://badge.fury.io/rb/httpx.svg)](http://rubygems.org/gems/httpx)
-[![pipeline status](https://gitlab.com/honeyryderchuck/httpx/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/httpx/commits/master)
-[![coverage report](https://gitlab.com/honeyryderchuck/httpx/badges/master/coverage.svg)](https://honeyryderchuck.gitlab.io/httpx/coverage/#_AllFiles)
+[![pipeline status](https://gitlab.com/honeyryderchuck/httpx/badges/master/pipeline.svg)](https://gitlab.com/honeyryderchuck/httpx/pipelines?page=1&scope=all&ref=master)
+[![coverage report](https://gitlab.com/honeyryderchuck/httpx/badges/master/coverage.svg?job=coverage)](https://honeyryderchuck.gitlab.io/httpx/coverage/#_AllFiles)
 
 HTTPX is an HTTP client library for the Ruby programming language.
 
@@ -18,6 +18,7 @@ Among its features, it supports:
 And also:
 
 * Compression (gzip, deflate, brotli)
+* Streaming Requests
 * Authentication (Basic Auth, Digest Auth)
 * Expect 100-continue
 * Multipart Requests
@@ -25,6 +26,7 @@ And also:
 * HTTP/2 Server Push
 * H2C Upgrade
 * Automatic follow redirects
+* International Domain Names
 
 ## How
 
@@ -83,7 +85,15 @@ However if the server supports HTTP/1.1, it will try to use HTTP pipelining, fal
 
 ### Clean API
 
-`HTTPX` acknowledges the ease-of-use of the [http gem](https://github.com/httprb/http) API (itself inspired by python [requests](http://docs.python-requests.org/en/latest/) library). It therefore aims at reusing the same facade, extending it for the use cases which the http gem doesn't support.
+`httpx` builds all functions around the `HTTPX` module, so that all calls can compose of each other. Here are a few examples:
+
+```ruby
+response = HTTPX.get("https://www.google.com")
+response = HTTPX.post("https://www.nghttp2.org/httpbin/post", params: {name: "John", age: "22"})
+response = HTTPX.plugin(:basic_authentication)
+                .basic_authentication("user", "pass")
+                .get("https://www.google.com")
+```
 
 ### Lightweight
 
@@ -133,7 +143,7 @@ Doesn't work with ruby 2.4.0 for Windows (see [#36](https://gitlab.com/honeyryde
 
 * Discuss your contribution in an issue
 * Fork it
-* Make your changes, add some test
+* Make your changes, add some tests
 * Ensure all tests pass (`bundle exec rake test`)
 * Open a Merge Request (that's Pull Request in Github-ish)
 * Wait for feedback

data/doc/release_notes/0_10_0.md

@@ -0,0 +1,66 @@
+# 0.10.0
+
+## Features
+
+### Streaming Requests
+
+The `stream` plugin adds functionality to handle long-lived stream responses, such as the Twitter Streaming API:
+
+```ruby
+http = HTTPX.plugin(:stream)
+
+http.get(stream_api_endpoint, stream: true).each_line do |line|
+  payload = JSON.parse(line)
+  # do smth with this
+end
+```
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Stream
+
+### Rate Limiter
+
+The `rate_limiter` plugin adds functionality for automatically hooking into rate-limiting responses coming from the server, and waits-and-retries them according to what the server advertises.
+
+```ruby
+HTTPX.plugin(:rate_limiter).get(rate_limited_api_endpoint)
+# => 429 Too Many Requests .... Retry-After: 3
+# waits 3 seconds before retrying
+```
+
+https://gitlab.com/honeyryderchuck/httpx/-/wikis/Rate-Limiter
+
+### Ruby 3
+
+This release is the first testing against and targeting Ruby 3 and some of the new features.
+
+It ships with RBS signatures for all of the client-facing APIs. There's non 100% typinng coverage yet, but I'm gradually (pun intended) working on it.
+
+
+## Improvements
+
+### IDN support
+
+Requests where the domains are formed by non-ASCII characters, are now supported (if you're using ruby 2.3 or more recent).
+
+```ruby
+HTTPX.get("http://bücher.ch") # it works!
+```
+
+### cookies plugin full implementation
+
+
+The `cookies` plugin is now independent of 3rd-party gems. The motivation for this was that `http-cookie` was dependent of both `domain_name` and `unf` gems, which are currently unusable in ruby 3, and haven't received any update in the last 3 years.
+
+The implementation is still compliant with RFC6265, and all of the features provided in earlier versions were ported, exceptwhen loading the cookie jar stored in a Netscape-format file or Mozilla sqlite database, which were not documented for `httpx` anyway, and I considered too niche to backport. If you feel `httpx` should support those, do let me know.
+
+Some code from these gems, including the ruby punycode implementation, is now part of the source tree, along with its licenses and attribution mentions.
+
+## Bugfixes
+
+
+Several edge-case bugs have been fixed solely by the integration of RBS runtime type checking, including some bugs around closing a connection pool that can cause loops.
+
+
+## Regressions
+
+`HTTPX::ErrorResponse`'s methods `#headers` and `#reason` were removed, as they didn't provide much value. Consider calling `#raise_for_status` or checking the API (`is_a?(HTTPX::ErrorResponse)` or `respond_to?(:error)` are strategies for this).

data/doc/release_notes/0_10_1.md

@@ -0,0 +1,37 @@
+# 0.10.1
+
+## Improvements
+
+### URL-encoded nested params
+
+url encoder now supports nested params, which is a standard of rack-based frameworks:
+
+```ruby
+HTTPX.post("https://httpbin.org/post", form: { a: { b: 1 }, c: [2, 3] })
+# a[b]=1&c[]=2&c[]=3
+```
+
+This encoding scheme is now the standard for URL-encoded request bodies, query params, and `:multipart` plugin requests.
+
+### Socks4 IPv6 addresses
+
+HTTPX supports IPv6 Socks4 proxies now. This support is restricted to rubies where `IPAddr#hton` is implemented though, so you are encouraged to upgrade.
+
+## More verbose HTTP Errors
+
+`HTTPX::Response#raise_for_status` was raising exceptions for client/server HTTP errors codes (4xx/5xx). However, only the status code was part of the message.
+
+From now on, both headers and the responnse payload will also appear, so expected more verbosity, but also more meaningful information.
+
+## Bugfixes
+
+* HTTP/2 and HTTP/1.1 exhausted connections now get properly migrated into a new connection;
+* HTTP/2 421 responses will now correctly migrate the connection and pendign requests to HTTP/1.1 (a hanging loop was being caused);
+* HTTP/2 connection failed with a GOAWAY settings timeout will now return error responses (instead of hanging indefinitely);
+* Non-IP proxy name-resolving errors will now move on to the next available proxy in the list (instead of hanging indefinitely);
+* Non-IP DNS resolve errors for `native` and `https` variants will now return the appropriate error response (instead of hanging indefinitely);
+
+## Chore
+
+* `HTTPX.plugins` is now officially deprecated (use `HTTPX.plugin` instead);
+

data/doc/release_notes/0_10_2.md

@@ -0,0 +1,5 @@
+# 0.10.2
+
+## Bugfixes
+
+Support for nested params in multipart forms introduced a bug where top-level params weren't being spread out correctly in the request body.

data/doc/release_notes/0_11_0.md

@@ -0,0 +1,76 @@
+# 0.11.0
+
+## Features
+
+### Webmock Adapter
+
+`httpx` can now be integrated with `webmock`, a popular HTTP requests stubbing library.
+
+```ruby
+# minitest
+require "webmock/minitest"
+require "httpx/adapters/webmock"
+
+# in rspec
+require "webmock/rspec"
+require "httpx/adapters/webmock"
+
+# and now you're free for mocking
+WebMock.enable!
+stub_http_request(:get, "https://www.google.com").and_return(status: 200, body: "here's google")
+
+```
+
+Read more about it in the [webmock integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Webmock-Adapter).
+
+### Datadog Adapter
+
+`httpx` ships with integration for [ddtrace, datadog's official tracing client](https://github.com/DataDog/dd-trace-rb). You just need to initialize it the following way:
+
+```ruby
+require "ddtrace"
+require "httpx/adapters/datadog"
+
+Datadog.configure do |c|
+  c.use :httpx
+end
+```
+
+A trace will be emitted for every request, so this should be an interesting visualization if concurrent requests are sent.
+
+Customization options and traces are similar to what [the net-http adapter provides](https://docs.datadoghq.com/tracing/setup_overview/setup/ruby/#nethttp).
+
+Read more about it in the [datadog integration documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Datadog-Adapter).
+
+## Improvements
+
+### Own multipart request encoder
+
+`httpx` now ships with its own multipart formdata encoder, and does not rely on `http-form_data` anymore:
+
+```ruby
+HTTPX.plugin(:multipart).post(uri, form: {file: File.new("path/to/file")})
+```
+
+Read more about it in the [multipart plugin documentation](https://honeyryderchuck.gitlab.io/httpx/wiki/Multipart-Uploads), including also about why this was made.
+
+### Expect Plugin
+
+The `:expect` plugin now works reliably when the server does not support the `expect: 100-continue` header, i.e. it'll upload the body after a certain timeout. Building onn that, two behaviours are now implemented:
+
+* A cache of domains which did not respond to the `expect` header is now kept, so that subsequent requests can skip the timeout and immediately upload the payload.
+* If the "100 Continue" response arrives **after** the timeout expired and the body has been uploaded, the domain is removed from the cache, and subsequent requests will send the `expect` header.
+
+### SNI/Host options
+
+Some extension of the API was applied in order to support custom TLS negotiation parameters. You can now pass `:hostname` under the `:ssl` options, and this will be used for the SNI part of the TLS negotiation. This is useful in scenarios where a proxy certificate doesn't apply for the host one wants to send the request to:
+
+```ruby
+response = session.get(proxy_ip, headers: { "host" => upstream_hostname }, ssl: { hostname: sni_hostname }
+```
+
+## Bugfixes
+
+A default 5 second timeout is in-place when using the DNS `:system` resolver, as it was found out that. when using the `resolv` library, the DNS query will not be retried otherwise. You can change this setting py passing `resolver_options: { timeouts: ANOTHER_TIMEOUT}`. In the future, this may become another timeout option, however.
+
+

data/doc/release_notes/0_9_0.md

@@ -0,0 +1,38 @@
+# 0.9.0
+
+## Features
+
+### Multiple requests with specific options
+
+You can now pass a third element to the "request element" of an array to `.request`.
+
+```ruby
+requests = [
+  [:post, "https://url/post", { form: { foo: "bar" } }],
+  [:post, "https://url/post", { form: { foo: "bar2" } }]
+]
+HTTPX.request(requests)
+# or, if you want to pass options common to all requests
+HTTPX.request(requests, max_concurrent_requests: 1)
+```
+
+
+### HTTPX::Session#build_request
+
+`HTTPX::Session::build_request` is now public API from a session. You can now build requests before you send them. These request objects are still considered somewhat "internal", so consider them immutable and **do not rely on its API**. Just pass them forward.
+
+Note: this API is only available for instantiated session, so there is no `HTTPX.build_request`.
+
+
+```ruby
+
+HTTPX.wrap do |http|
+  requests = [
+    http.build_request(:post, "https://url/post", { form: { foo: "bar" } }),
+    http.build_request(:post, "https://url/post", { form: { foo: "bar2" } })
+  ]
+  http.request(requests)
+  # or, if you want to pass options common to all requests
+  http.request(requests, max_concurrent_requests: 1)
+end
+```

data/lib/httpx.rb

@@ -5,6 +5,8 @@ require "httpx/version"
 require "httpx/extensions"
 
 require "httpx/errors"
+require "httpx/utils"
+require "httpx/domain_name"
 require "httpx/altsvc"
 require "httpx/callbacks"
 require "httpx/loggable"

data/lib/httpx/adapters/datadog.rb

@@ -0,0 +1,205 @@
+# frozen_string_literal: true
+
+require "ddtrace/contrib/integration"
+require "ddtrace/contrib/rest_client/configuration/settings"
+require "ddtrace/contrib/rest_client/patcher"
+
+module Datadog
+  module Contrib
+    module HTTPX
+      # HTTPX Datadog Plugin
+      #
+      # Enables tracing for httpx requests. A span will be created for each individual requests,
+      # and it'll trace since the moment it is fed to the connection, until the moment the response is
+      # fed back to the session.
+      #
+      module Plugin
+        class RequestTracer
+          SPAN_REQUEST = "httpx.request"
+
+          def initialize(request)
+            @request = request
+          end
+
+          def call
+            return if skip_tracing?
+
+            @request.on(:response, &method(:finish))
+
+            verb = @request.verb.to_s.upcase
+            uri = @request.uri
+
+            @span = datadog_pin.tracer.trace(SPAN_REQUEST)
+            service_name = datadog_config[:split_by_domain] ? uri.host : datadog_pin.service_name
+
+            begin
+              @span.service = service_name
+              @span.span_type = Datadog::Ext::HTTP::TYPE_OUTBOUND
+              @span.resource = verb
+
+              Datadog::HTTPPropagator.inject!(@span.context, @request.headers) if datadog_pin.tracer.enabled && !skip_distributed_tracing?
+
+              # Add additional request specific tags to the span.
+
+              @span.set_tag(Datadog::Ext::HTTP::URL, @request.path)
+              @span.set_tag(Datadog::Ext::HTTP::METHOD, verb)
+
+              @span.set_tag(Datadog::Ext::NET::TARGET_HOST, uri.host)
+              @span.set_tag(Datadog::Ext::NET::TARGET_PORT, uri.port.to_s)
+
+              # Tag as an external peer service
+              @span.set_tag(Datadog::Ext::Integration::TAG_PEER_SERVICE, @span.service)
+
+              # Set analytics sample rate
+              if Contrib::Analytics.enabled?(datadog_config[:analytics_enabled])
+                Contrib::Analytics.set_sample_rate(@span, datadog_config[:analytics_sample_rate])
+              end
+            rescue StandardError => e
+              Datadog.logger.error("error preparing span for http request: #{e}")
+            end
+          rescue StandardError => e
+            Datadog.logger.debug("Failed to start span: #{e}")
+          end
+
+          def finish(response)
+            return unless @span
+
+            if response.respond_to?(:error)
+              @span.set_error(response.error)
+            else
+              @span.set_tag(Datadog::Ext::HTTP::STATUS_CODE, response.status.to_s)
+
+              @span.set_error(::HTTPX::HTTPError.new(response)) if response.status >= 400 && response.status <= 599
+            end
+
+            @span.finish
+          end
+
+          private
+
+          def skip_tracing?
+            return true if @request.headers.key?(Datadog::Ext::Transport::HTTP::HEADER_META_TRACER_VERSION)
+
+            return false unless @datadog_pin
+
+            span = @datadog_pin.tracer.active_span
+
+            return true if span && (span.name == SPAN_REQUEST)
+
+            false
+          end
+
+          def skip_distributed_tracing?
+            return !datadog_pin.config[:distributed_tracing] if datadog_pin.config && datadog_pin.config.key?(:distributed_tracing)
+
+            !Datadog.configuration[:httpx][:distributed_tracing]
+          end
+
+          def datadog_pin
+            @datadog_pin ||= begin
+              service = datadog_config[:service_name]
+              tracer = datadog_config[:tracer]
+
+              Datadog::Pin.new(
+                service,
+                app: "httpx",
+                app_type: Datadog::Ext::AppTypes::WEB,
+                tracer: -> { tracer }
+              )
+            end
+          end
+
+          def datadog_config
+            @datadog_config ||= Datadog.configuration[:httpx, @request.uri.host]
+          end
+        end
+
+        module ConnectionMethods
+          def send(request)
+            RequestTracer.new(request).call
+            super
+          end
+        end
+      end
+
+      module Configuration
+        # Default settings for httpx
+        #
+        class Settings < Datadog::Contrib::Configuration::Settings
+          option :service_name, default: "httpx"
+          option :distributed_tracing, default: true
+          option :split_by_domain, default: false
+
+          option :enabled do |o|
+            o.default { env_to_bool("DD_TRACE_HTTPX_ENABLED", true) }
+            o.lazy
+          end
+
+          option :analytics_enabled do |o|
+            o.default { env_to_bool(%w[DD_TRACE_HTTPX_ANALYTICS_ENABLED DD_HTTPX_ANALYTICS_ENABLED], false) }
+            o.lazy
+          end
+
+          option :analytics_sample_rate do |o|
+            o.default { env_to_float(%w[DD_TRACE_HTTPX_ANALYTICS_SAMPLE_RATE DD_HTTPX_ANALYTICS_SAMPLE_RATE], 1.0) }
+            o.lazy
+          end
+
+          option :error_handler, default: Datadog::Tracer::DEFAULT_ON_ERROR
+        end
+      end
+
+      # Patcher enables patching of 'httpx' with datadog components.
+      #
+      module Patcher
+        include Datadog::Contrib::Patcher
+
+        module_function
+
+        def target_version
+          Integration.version
+        end
+
+        # loads a session instannce with the datadog plugin, and replaces the
+        # base HTTPX::Session with the patched session class.
+        def patch
+          datadog_session = ::HTTPX.plugin(Plugin)
+
+          ::HTTPX.send(:remove_const, :Session)
+          ::HTTPX.send(:const_set, :Session, datadog_session.class)
+        end
+      end
+
+      # Datadog Integration for HTTPX.
+      #
+      class Integration
+        include Contrib::Integration
+
+        # MINIMUM_VERSION = Gem::Version.new('0.11.0')
+        MINIMUM_VERSION = Gem::Version.new("0.10.2")
+
+        register_as :httpx
+
+        def self.version
+          Gem.loaded_specs["httpx"] && Gem.loaded_specs["httpx"].version
+        end
+
+        def self.loaded?
+          defined?(::HTTPX::Request)
+        end
+
+        def self.compatible?
+          super && version >= MINIMUM_VERSION
+        end
+
+        def default_configuration
+          Configuration::Settings.new
+        end
+
+        def patcher
+          Patcher
+        end
+      end
+    end
+  end
+end