restify 2.1.0 → 3.0.0.rc2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6db18b2b418bd450289f6e0c64f429e0c3f3066e6a45c082af2f34fa87b3219f
-  data.tar.gz: ca2c0abe620c7a679e0c0ec0b12e57448ea7d709b5e0ee18d523183e97085715
+  metadata.gz: 99d1fd08757bc62db3bd58386310893f58224c6f0424980c412448c821021b20
+  data.tar.gz: 97ea53432dcee6d6089332eb1783dfd04108f62f88ce3070f1c06b03d7cb1360
 SHA512:
-  metadata.gz: 85f715444240b80d9a269662779b69791a9240b6d37d69586948aaed3c138c8b8bc11d78e0eadd42d7da600f65e628b589053f33991b66a0a9b2c923248b4999
-  data.tar.gz: 9a1509904bcc695749e2cfb2b95976a38a94013db01aabb7374c270de7af788ac1f8e633a846aafe56e14b3cbb244d3952a99c769d845ea3c68c57f8d4f66c8e
+  metadata.gz: 705892e87eeedcbc9bda88352e2444b9217c01fce181f59fd1e228870c5b812cfc28208bf4d93db436ff65fb94ccc87782320ecd4bb0f17c260e6fc8e90446c5
+  data.tar.gz: d9a471e407b564ce115c1e197f0b0e74d74a5553e174529837b36d81d1e936d84121be97406d0ef075138d04a010d32e541eab9f1bf8d7fc1862b975acf7eed4

data/CHANGELOG.md

@@ -17,6 +17,54 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### Breaks
 
+## 3.0.0.rc2 - (2026-09-25)
+
+---
+
+First release candidate for 3.0. `3.0.0.rc1` was tagged, but not published due to a broken release pipeline.
+
+### New
+
+- Add `Restify::Adapter::Ethon` adapter directly using libcurl in a background event loop
+- Perf: Run event loop in waiting threads to avoid slow handover between threads
+- Add `restify/webmock` to intercept requests with WebMock for all adapters
+
+### Changes
+
+- Restrict requests and redirects to HTTP(S), as libcurl supports many more
+  protocols such as `file://` or `gopher://`
+- Drop EOL Ruby versions 3.1 and 3.2
+- Drop `hitimes` dependency, using `Process.clock_gettime` for timeouts
+- Return resources without data for empty response bodies, even with a JSON
+  or MessagePack content type (#31)
+
+### Fixes
+
+- Use the effective URI after redirects for responses and relations
+- Do not set empty OpenTelemetry attributes for URIs without a port
+- Encode response bodies using the charset from the `Content-Type` header
+- Do not share connections of the Ethon adapter with forked child processes
+- Do not deadlock the Typhoeus adapter with `sync: true` when used from multiple threads
+- Raise the `ResponseError` for the status code, e.g. `NotFound`, instead of a
+  `JSON::ParserError` for error responses with an empty or invalid JSON body (#31)
+
+### Breaks
+
+- Raise minimum Ruby version to 3.3
+- `Restify::Request#uri` is always parsed and returned as `Addressable::URI`
+- Use `Restify::Adapter::Ethon` by default. With WebMock, require
+  `restify/webmock`, as WebMock's Typhoeus integration does not apply anymore.
+- `typhoeus` is an optional dependency now. Add it to your Gemfile to use
+  `Restify::Adapter::Typhoeus`.
+- Invalid response bodies do not raise from `#value!` anymore. The resource is
+  returned, e.g. with relations from headers, and accessing its data raises a
+  `Restify::ParseError` with the parser error as its cause (#31).
+- Remove `activesupport` dependency. `Restify.cache` defaults to `nil`, as the
+  previous default cache did not cache anything (yet).
+- Remove `logging` dependency. Set a `Logger`-compatible logger with
+  `Restify.logger = Logger.new(...)`, nothing is logged by default. Restify does
+  not change `Ethon.logger` anymore.
+
 ## 2.1.0 - (2026-06-20)
 
 ---

data/README.md

@@ -2,7 +2,6 @@
 
 [![Gem Version](https://img.shields.io/gem/v/restify?logo=ruby)](https://rubygems.org/gems/restify)
 [![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/jgraichen/restify/test.yml?logo=github)](https://github.com/jgraichen/restify/actions)
-[![Code Quality](https://codebeat.co/badges/368f8033-bd76-48bc-9777-85f1d4befa94)](https://codebeat.co/projects/github-com-jgraichen-restify-main)
 
 Restify is an hypermedia REST client that does parallel, concurrent and keep-alive requests by default.
 
@@ -14,9 +13,15 @@ Restify is build upon the following libraries:
 
 * [concurrent-ruby](https://github.com/ruby-concurrency/concurrent-ruby)
 * [addressable](https://github.com/sporkmonger/addressable)
-* [typhoeus](https://github.com/typhoeus/typhoeus)
+* [ethon](https://github.com/typhoeus/ethon), using libcurl
 
-The HTTP adapters are mostly run in a background thread and may not survive mid-application forks.
+Restify uses libcurl via Ethon by default. The older [Typhoeus](https://github.com/typhoeus/typhoeus) adapter is still available. To use it, add `typhoeus` to your Gemfile and configure it:
+
+```ruby
+Restify.adapter = Restify::Adapter::Typhoeus.new
+```
+
+The default Ethon adapter should work with forking, e.g. in Puma or Unicorn workers. Requests still pending when forking are aborted in the child process.
 
 Restify includes processors to parse responses and to extract links between resources. The following formats are can be parsed:
 
@@ -102,6 +107,27 @@ puts "#{commit['commit']['message']}"
 
 See commented example in main spec [`spec/restify_spec.rb`](https://github.com/jgraichen/restify/blob/master/spec/restify_spec.rb#L100) or in the `examples` directory.
 
+## Testing with WebMock
+
+Restify can intercept requests for [WebMock](https://github.com/bblimke/webmock), independent of the adapter in use. Load it after WebMock, before WebMock is enabled, e.g. in your `spec_helper.rb`:
+
+```ruby
+require 'webmock/rspec'
+require 'restify/webmock'
+```
+
+Note: Stubbed responses are returned as they are, including redirects, like WebMock does for other libcurl-based libraries. Requests not stubbed are passed on to the adapter if WebMock allows connections.
+
+For tests relying on actual HTTP behavior, e.g. following redirects, run a local stub server instead, so that requests are sent over the network by the adapter.
+
+WebMock's own Typhoeus integration is not needed for Restify and can be disabled. Otherwise, requests are reported a second time, when they are passed on to the Typhoeus adapter:
+
+```ruby
+WebMock.enable!(except: %i[typhoeus])
+```
+
+Keep it enabled if your application uses Typhoeus directly, too.
+
 ## Contributing
 
 1. [Fork it](http://github.com/jgraichen/restify/fork)
@@ -113,7 +139,7 @@ See commented example in main spec [`spec/restify_spec.rb`](https://github.com/j
 
 ## License
 
-Copyright (C) 2014-2025 Jan Graichen
+Copyright (C) 2014-2026 Jan Graichen
 
 This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
 

data/lib/restify/adapter/base.rb

@@ -8,7 +8,7 @@ module Restify
       prepend Telemetry
 
       def call(request)
-        Promise.create do |writer|
+        Promise.create(driver:) do |writer|
           call_native request, writer
         end
       end
@@ -16,6 +16,15 @@ module Restify
       def call_native(_request, _writer)
         raise NotImplementedError.new 'Subclass responsibility'
       end
+
+      private
+
+      # Adapters that can process requests in the thread waiting on the
+      # returned promise return an object responding to
+      # `#drive(promise, timeout)` here, see `Promise#wait`.
+      def driver
+        nil
+      end
     end
   end
 end

data/lib/restify/adapter/ethon.rb

@@ -0,0 +1,462 @@
+# frozen_string_literal: true
+
+require 'ethon'
+require 'nio'
+
+module Restify
+  module Adapter
+    #
+    # An adapter using libcurl's multi interface via `Ethon` in
+    # `socket_action` mode, driven by a nio4r-based event loop.
+    #
+    # Requests are handed over to the event loop using a queue and the
+    # loop is woken up via `NIO::Selector#wakeup`. Both are safe to call
+    # from any thread.
+    #
+    # This is important because the event loop is run from either any
+    # waiting thread or a dedicated background thread.
+    #
+    # The background thread that is spawned on demand and restarted if
+    # it ever dies. A thread waiting on a request's promise takes over
+    # the loop and runs it itself until the promise is complete, see
+    # `#drive`. This avoids handing each request and response between
+    # threads, which is expensive (thread wakeup). Only one thread runs
+    # the loop at a time, therefore all libcurl and selector interaction
+    # is still serialized.
+    #
+    class Ethon < Base
+      include Logging
+
+      DEFAULT_HEADERS = {
+        'Expect' => '',
+        'Transfer-Encoding' => '',
+      }.freeze
+
+      DEFAULT_OPTIONS = {
+        followlocation: true,
+        tcp_keepalive: true,
+        tcp_keepidle: 5,
+        tcp_keepintvl: 5,
+      }.freeze
+
+      # Restify follows relations from URLs in server responses, and
+      # libcurl supports far more than HTTP, e.g. `file://` or `scp://`.
+      # Therefore only HTTP-like protocols must ever be used, both for
+      # the request itself and when following redirects.
+      #
+      # This is intentionally not configurable.
+      PROTOCOLS = %i[http https].freeze
+
+      INTERESTS = {
+        none: nil,
+        in: :r,
+        out: :w,
+        inout: :rw,
+      }.freeze
+
+      def initialize(options: {}, **multi)
+        @options = DEFAULT_OPTIONS.merge(options)
+        @multi_options = multi
+
+        # libcurl only stores the function pointers, therefore the procs
+        # must be referenced here too, or they would be garbage
+        # collected.
+        @socketfunction = method(:on_socket).to_proc
+        @timerfunction  = method(:on_timer).to_proc
+
+        @mutex  = Mutex.new
+        @thread = nil
+
+        # Wake up the background thread from `select` when a waiting
+        # thread wants to run the loop.
+        @loop = LoopLock.new { @selector.wakeup }
+
+        setup
+
+        super()
+      end
+
+      def call_native(request, writer)
+        check_fork!
+
+        easy = convert(request, writer)
+
+        debug 'request:add',
+          tag: request.object_id,
+          method: request.method.upcase,
+          url: request.uri,
+          timeout: request.timeout
+
+        @queue << easy
+
+        # Ensure the event loop is running and let it pick up the new
+        # request.
+        thread
+        @selector.wakeup
+      end
+
+      # Run the event loop in the calling thread until the promise is
+      # complete or the timeout expires, see `Promise#wait`.
+      #
+      # Several threads can wait at the same time, but only one runs the
+      # loop. The others sleep until either their promise completes or
+      # the loop is released. The background thread keeps processing
+      # requests nobody waits on, but releases the loop as soon as a
+      # thread starts waiting.
+      #
+      # This only processes transfers, i.e. fulfills or rejects the
+      # adapter's promises. Callbacks chained with `Promise#then` still
+      # only run in the thread waiting on them.
+      #
+      # Returns false when the loop cannot be run in the calling thread,
+      # i.e. when waiting from within the loop itself.
+      #
+      def drive(promise, timeout) # rubocop:disable Naming/PredicateMethod
+        check_fork!
+
+        return false if @loop.owned?
+
+        while @loop.acquire(timeout) { promise.complete? }
+          begin
+            step(timeout.remaining) until promise.complete? || !timeout.remaining.positive?
+          rescue StandardError => e
+            error(e)
+          ensure
+            @loop.release
+          end
+        end
+
+        true
+      end
+
+      private
+
+      def driver
+        self
+      end
+
+      def setup
+        @pid      = Process.pid
+        @selector = NIO::Selector.new
+        @monitors = {}
+        @timer    = nil
+        @queue    = Queue.new
+
+        @multi = ::Ethon::Multi.new(execution_mode: :socket_action, **@multi_options)
+        @multi.socketfunction = @socketfunction
+        @multi.timerfunction  = @timerfunction
+      end
+
+      def check_fork!
+        return if @pid == Process.pid
+
+        @mutex.synchronize { forked! if @pid != Process.pid }
+      end
+
+      # A forked child process inherits the parent's event loop,
+      # connections and sockets. They must neither be used nor cleaned
+      # up here, as the parent still uses them.
+      #
+      # Therefore, abandon the state without releasing it, and set up
+      # everything again.
+      def forked!
+        abandoned = @multi.easy_handles.dup
+        abandoned << @queue.pop(true) until @queue.empty?
+
+        @multi.handle.autorelease = false
+        abandoned.each {|easy| easy.handle.autorelease = false }
+
+        @thread = nil
+        @loop.reset!
+        setup
+
+        abandoned.each do |easy|
+          easy._restify_writer.reject(
+            Restify::NetworkError.new(easy._restify_request, 'Request started before fork'),
+          )
+        end
+      end
+
+      def convert(request, writer)
+        Easy.new.tap do |easy|
+          easy._otel_span = OpenTelemetry::Trace.current_span
+          easy._restify_request = request
+          easy._restify_writer = writer
+
+          easy.http_request(
+            request.uri.to_s,
+            request.method,
+            request_options(request),
+          )
+
+          easy.on_complete do |completed|
+            complete(completed, request, writer)
+
+            # Wake up threads waiting on the loop to check if their
+            # result is available now.
+            @loop.notify
+          end
+        end
+      end
+
+      def request_options(request)
+        # libcurl requires millisecond-based timeouts so fractional
+        # seconds must be converted and rounded.
+        timeout = request.timeout && (request.timeout * 1000).ceil
+
+        @options.merge(
+          headers: DEFAULT_HEADERS.merge(request.headers),
+          body: request.body,
+          timeout_ms: timeout,
+          connecttimeout_ms: timeout,
+
+          # Merged last, so that neither can be overridden via `options`.
+          protocols: PROTOCOLS,
+          redir_protocols: PROTOCOLS,
+
+          # Do not use signal handlers in libcurl, as it would interfere
+          # with the threads.
+          nosignal: true,
+        )
+      end
+
+      def complete(easy, request, writer)
+        writer.set do
+          code   = easy.return_code
+          status = easy.response_code
+
+          debug 'request:complete',
+            tag: request.object_id,
+            status: status,
+            message: code
+
+          if code != :ok
+            raise Restify::NetworkError.new(
+              request,
+              ::Ethon::Curl.easy_strerror(code),
+            )
+          end
+
+          if status.nil? || status.zero?
+            raise Restify::NetworkError.new(
+              request,
+              'Response without HTTP status',
+            )
+          end
+
+          convert_back(easy, request)
+        end
+      rescue StandardError => e
+        # This runs inside a libcurl callback, therefore no exception
+        # must ever escape from here. Anything reaching this point could
+        # not be handed to the promise anymore.
+        error(e)
+      end
+
+      def convert_back(easy, request)
+        ::Restify::Response.new(
+          request,
+          effective_uri(easy, request),
+          easy.response_code,
+          convert_headers(easy.response_headers),
+          easy.response_body,
+        )
+      end
+
+      def effective_uri(easy, request)
+        url = easy.effective_url
+        url ? Addressable::URI.parse(url) : request.uri
+      end
+
+      def convert_headers(raw)
+        headers = {}
+        return headers if raw.nil?
+
+        # The raw headers can contain multiple blocks, e.g. from
+        # informational responses or when following redirects; only use
+        # the latest ones:
+        block = raw.split(/\r?\n\r?\n/).reject {|b| b.strip.empty? }.last
+
+        # Split on newlines that are not followed by whitespace to keep
+        # folded header values together.
+        block.to_s.split(/\r?\n(?!\s)/).each do |line|
+          line = line.strip
+          next if line.empty? || line.start_with?('HTTP/')
+
+          key, value = line.split(':', 2)
+          next if value.nil?
+
+          key = key.strip.upcase.tr('-', '_')
+          value = value.strip.gsub(/\r?\n\s*/, ' ')
+
+          case (current = headers[key])
+            when nil then headers[key] = value
+            when Array then current << value
+            else headers[key] = [current, value]
+          end
+        end
+
+        headers
+      end
+
+      def thread
+        @mutex.synchronize do
+          # Spawn thread if not yet started, or recreate it if it died.
+          if @thread.nil? || !@thread.status
+            debug 'loop:spawn'
+            @thread = Thread.new { run }
+          end
+
+          @thread
+        end
+      end
+
+      def run
+        loop do
+          @loop.acquire_background
+
+          begin
+            step
+          ensure
+            @loop.release
+          end
+        rescue StandardError => e
+          error(e)
+        end
+      ensure
+        debug 'loop:exit'
+      end
+
+      # Run one iteration of the event loop. Must only be called by the
+      # thread owning the loop.
+      def step(limit = nil)
+        dequeue_all
+
+        # libcurl needs to be notified about its own timeouts, e.g. to
+        # start newly added transfers or to time out stalled ones.
+        timeout!
+
+        timeout = select_timeout
+        timeout = limit if limit && (timeout.nil? || timeout > limit)
+        debug 'loop:select', timeout: timeout
+
+        # nil on timeout; empty array when woken up
+        @selector.select(timeout)&.each do |monitor|
+          # libcurl can remove sockets while earlier events of the same
+          # batch are processed, e.g. when a completed transfer tears
+          # down other connections. Skip monitors that are gone by now.
+          next unless @monitors[monitor.value].equal?(monitor)
+
+          socket_action(monitor.value, readiness(monitor))
+        end
+      end
+
+      def dequeue_all
+        loop do
+          easy = begin
+            @queue.pop(true)
+          rescue ThreadError
+            break
+          end
+
+          add(easy)
+        end
+      end
+
+      def add(easy)
+        # Restore the OpenTelemetry span the request originated from, so
+        # that the Ethon instrumentation picks up the correct parent
+        # when the easy handle is added to libcurl.
+        #
+        # Handle exceptions from Ethon or WebMock too, and reject the
+        # promise, so that the errors can be handled in user code.
+        OpenTelemetry::Trace.with_span(easy._otel_span) do
+          @multi.add(easy)
+        rescue Exception => e # rubocop:disable Lint/RescueException
+          easy._restify_writer.reject(e)
+        end
+      end
+
+      # Seconds until libcurl wants to be called again, or nil to block
+      # until a socket becomes ready or the loop is woken up.
+      def select_timeout
+        return nil unless @timer
+
+        [@timer - now, 0].max
+      end
+
+      def timeout!
+        return unless @timer
+        return if now < @timer
+
+        # Reset before invoking libcurl, as it will set a new timeout
+        # from within the socket action.
+        @timer = nil
+
+        socket_action
+      end
+
+      def readiness(monitor)
+        readiness = []
+        readiness << :in if monitor.readable?
+        readiness << :out if monitor.writable?
+        readiness
+      end
+
+      def socket_action(socket = nil, readiness = 0)
+        @multi.socket_action(socket, readiness)
+      end
+
+      # libcurl: called when asking for readiness monitoring
+      def on_socket(_easy, socket, what, _userp, _socketp)
+        debug 'socket:action', tag: socket, what: what
+
+        if what == :remove
+          @monitors.delete(socket)&.close
+        else
+          monitor = (@monitors[socket] ||= register(socket))
+          monitor.interests = INTERESTS.fetch(what)
+        end
+
+        :ok
+      rescue StandardError => e
+        error(e)
+        :ok
+      end
+
+      # libcurl: called on timeout changes
+      def on_timer(_handle, timeout_ms, _userp)
+        debug 'timer:set', timeout: timeout_ms
+        @timer = timeout_ms.negative? ? nil : now + (timeout_ms / 1000.0)
+
+        :ok
+      rescue StandardError => e
+        error(e)
+        :ok
+      end
+
+      def register(socket)
+        # FD owned by libcurl: IO/ruby must never close it
+        io = IO.for_fd(socket, autoclose: false)
+
+        @selector.register(io, :r).tap do |monitor|
+          monitor.value = socket
+        end
+      end
+
+      def now
+        Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      end
+
+      def _log_prefix
+        "[#{object_id}/#{Thread.current.object_id}]"
+      end
+
+      # Keep track of the OTEL span, the request and the promise writer to
+      # reject on errors in the background thread or after fork
+      class Easy < ::Ethon::Easy
+        attr_accessor :_otel_span, :_restify_request, :_restify_writer
+      end
+    end
+  end
+end