ethon 0.18.0 → 0.19.0

data/README.md

@@ -1,36 +1,44 @@
 [![Gem Version](https://badge.fury.io/rb/ethon.svg)](https://badge.fury.io/rb/ethon)
 [![Build Status](https://github.com/typhoeus/ethon/workflows/Ruby/badge.svg)](https://github.com/typhoeus/ethon/actions/workflows/ruby.yml)
 
-#  Ethon
+# Ethon
 
-In Greek mythology, Ethon, the son of Typhoeus and Echidna, is a gigantic eagle. So much for the history.
-In the modern world, Ethon is a very basic libcurl wrapper using ffi.
+Ethon is a lightweight Ruby wrapper around [libcurl](https://curl.se/libcurl/),
+built with [FFI](https://github.com/ffi/ffi). It exposes:
 
-* [Documentation](http://rubydoc.info/github/typhoeus/ethon/frames/Ethon)
-* [Website](http://typhoeus.github.com/)
-* [Mailing list](http://groups.google.com/group/typhoeus)
+- `Ethon::Easy` for a single transfer.
+- `Ethon::Multi` for concurrent transfers.
+- libcurl options, callbacks, response streaming, debug output, and optional
+  lower-level URL, MIME, share, and header APIs.
 
-## Installation
-
-With bundler:
+The name comes from Greek mythology: Ethon was a gigantic eagle and the son of
+Typhoeus and Echidna.
 
-    gem "ethon"
+* [API documentation](https://rubydoc.info/github/typhoeus/ethon)
+* [Changelog](https://github.com/typhoeus/ethon/blob/master/CHANGELOG.md)
 
-With rubygems:
+## Installation
 
-    gem install ethon
+Add the gem to your bundle:
 
-## Usage
+```sh
+bundle add ethon
+```
 
-Making the first request is simple:
+Or install it directly:
 
-```ruby
-easy = Ethon::Easy.new(url: "www.example.com")
-easy.perform
-#=> :ok
+```sh
+gem install ethon
 ```
 
-You have access to various options, such as following redirects:
+Ethon requires Ruby 2.6 or newer and a libcurl shared library. Install libcurl
+with your system package manager, or point `ETHON_CURL_LIBS` at a specific
+build (see [Older libcurl versions](#older-libcurl-versions)).
+
+## Usage
+
+Create an `Easy` handle for a single transfer. The constructor accepts libcurl
+options such as `followlocation`:
 
 ```ruby
 easy = Ethon::Easy.new(url: "www.example.com", followlocation: true)
@@ -38,24 +46,22 @@ easy.perform
 #=> :ok
 ```
 
-Once you're done you can inspect the response code and body:
+After `perform` returns, read the response:
 
 ```ruby
-easy = Ethon::Easy.new(url: "www.example.com", followlocation: true)
-easy.perform
 easy.response_code
 #=> 200
 easy.response_body
 #=> "<!doctype html><html ..."
 ```
 
-## Http
+## HTTP helpers
 
-In order to make life easier, there are some helpers for making HTTP requests:
+`http_request` configures the URL, method, query parameters, and body:
 
 ```ruby
 easy = Ethon::Easy.new
-easy.http_request("www.example.com", :get, { params: {a: 1} })
+easy.http_request("www.example.com", :get, { params: { a: 1 } })
 easy.perform
 #=> :ok
 ```
@@ -67,33 +73,170 @@ easy.perform
 #=> :ok
 ```
 
-This is really handy when making requests since you don't have to care about setting
-everything up correctly.
+File values in a `:body` hash are sent as `multipart/form-data`. Ethon loads
+the optional `mime-types` gem on the first upload to detect each part's content
+type; without it, parts use `application/octet-stream`.
+
+## Parallel requests
 
-## Http2
-Standard http2 servers require the client to connect once and create a session (multi) and then add simple requests to the multi handler.
-The `perform` method then takes all the requests in the multi handler and sends them to the server.
+Add `Easy` handles to a `Multi` and call `perform`:
 
-See the following example
 ```ruby
 multi = Ethon::Multi.new
-easy = Ethon::Easy.new
+2.times do
+  easy = Ethon::Easy.new(url: "www.example.com")
+  multi.add(easy)
+end
+multi.perform
+```
 
-easy.http_request("www.example.com/get", :get, { http_version: :httpv2_0 })
+## HTTP/2
 
-# Sending a request with http version 2 will send an Upgrade header to the server, which many older servers will not support
-# See below for more info: https://everything.curl.dev/http/http2
-# If this is a problem, send the below:
-easy.http_request("www.example.com/get", :get, { http_version: :httpv2_prior_knowledge })
+Use `:http_version` to request HTTP/2. `:httpv2_tls` negotiates HTTP/2 over
+TLS and uses HTTP/1.1 for plain HTTP. `:httpv2_0` also attempts the HTTP/2
+Upgrade handshake on plain HTTP, which many older servers do not support.
+`:httpv2_prior_knowledge` starts HTTP/2 without negotiation. See the
+[libcurl HTTP/2 documentation](https://everything.curl.dev/http/versions/http2.html)
+for details.
 
-# To set the server to use http2 with https and http1 with http, send the following:
-easy.http_request("www.example.com/get", :get, { http_version: :httpv2_tls })
+HTTP/2 works with a single easy handle. A multi handle lets several requests
+share the connection and multiplex them:
+
+```ruby
+multi = Ethon::Multi.new
+easy = Ethon::Easy.new
+
+easy.http_request(
+  "https://www.example.com/get",
+  :get,
+  { http_version: :httpv2_tls }
+)
 
 multi.add(easy)
 multi.perform
 ```
 
-##  LICENSE
+## Lower-level libcurl APIs
+
+Ethon exposes wrappers for some of libcurl's newer APIs. Optional bindings are
+attached only when the loaded libcurl exports them. Wrappers for the URL, MIME,
+share, header, and transfer helpers raise `NotImplementedError` when the
+underlying function is missing; introspection helpers return `nil` or `[]`.
+
+```ruby
+# URL API
+url = Ethon::Curl::Url.new("https://example.com/a")
+url.get(:host) #=> "example.com"
+url.set(:path, "/b")
+url.to_s       #=> "https://example.com/b"
+
+# MIME API
+easy = Ethon::Easy.new(url: "http://example.com/upload")
+mime = Ethon::Curl::Mime.new(easy)
+part = mime.addpart
+part.name = "file"
+part.file = "/tmp/example.txt"
+part.headers = { "X-Checksum" => "abc123" }
+easy.mime = mime
+
+# Share cookies or DNS data between easy handles
+share = Ethon::Curl::Share.new.share(:cookie)
+easy.share = share
+
+# Structured response headers and option metadata
+easy.perform
+easy.header("Content-Type")
+easy.each_header { |header| puts "#{header.name}: #{header.value}" }
+Ethon::Curl.easy_option("URL")
+Ethon::Curl.protocols
+```
+
+When a share handle is used from more than one thread, set
+`Share#lockfunction=` and `Share#unlockfunction=` so libcurl can synchronize
+access to the shared data.
+
+## Older libcurl versions
+
+Ethon attaches some functions only when the loaded libcurl exports them.
+Wrappers that check for a missing function raise `NotImplementedError`; the
+introspection helpers return `nil` or `[]`. This keeps the gem loadable with
+libcurl versions that predate the URL, MIME, share, or structured-header APIs.
+
+libcurl silently ignores unknown options, which can make a request run with a
+different configuration than intended. Ethon therefore raises
+`Ethon::Errors::UnsupportedOption` when `curl_easy_setopt` reports
+`CURLE_UNKNOWN_OPTION`, and `Ethon::Errors::SetOption` for other rejected
+values. Guard optional settings when supporting older runtimes:
+
+```ruby
+easy.tcp_keepcnt = 30 if Ethon::Curl.supports_option?(:tcp_keepcnt)
+easy.httpsig_key = key if Ethon::Curl.version_at_least?("8.23.0")
+```
+
+`Curl.supports_option?` asks libcurl's own option table on 7.73.0 and newer.
+Older versions cannot answer that question, so it returns `true` for every
+option Ethon knows; setting an unsupported option still raises. Multi options
+are only checked against Ethon's table, because libcurl does not expose their
+metadata.
+
+The deprecated form API is attached only when the loaded libcurl exports it,
+because libcurl can be built with `--disable-form-api`. Multipart bodies use the
+MIME API when available and fall back to the form API otherwise.
+
+Point `ETHON_CURL_LIBS` at a specific libcurl build to test against it:
+
+```sh
+ETHON_CURL_LIBS=/opt/curl/lib/libcurl.so bundle exec rake
+```
+
+CI builds libcurl 7.88 in addition to the system package and 8.23 to catch
+accidental use of newer symbols and options. The 8.23 build uses
+`--disable-form-api`, so the missing form API path is exercised too.
+
+## Debugging
+
+Set `verbose: true` on an `Easy` to collect libcurl's debug messages.
+`Easy#debug_info` returns a `DebugInfo` object; use `#to_a` for every message
+or `#to_h` to group them by type (such as `:text`, `:header_in`, and
+`:header_out`). `Easy#debugfunction` registers an observer. The messages are
+also written through `Ethon.logger` at the debug level, so configure it to see
+them:
+
+```ruby
+Ethon.logger.level = Logger::DEBUG
+```
+
+Response body data is not written to the log; it stays available through
+`Easy#debug_info` and the `debugfunction` observer.
+
+## Thread and fork safety
+
+Ethon can be used from multiple threads, but it does not synchronize access to
+individual handles. Follow [libcurl's thread-safety rules](https://curl.se/libcurl/c/threadsafe.html):
+
+* Separate `Ethon::Easy` and `Ethon::Multi` instances may run concurrently in
+  different threads.
+* Do not use the same `Easy` or `Multi` instance from more than one thread at a
+  time. This includes changing its options while a transfer is running.
+* Callbacks run on the thread performing the transfer. Synchronize any shared
+  application state accessed by a callback.
+* When using threads, set `nosignal: true` unless your libcurl uses a threaded
+  or c-ares resolver. See libcurl's signal guidance for the associated DNS
+  timeout and `SIGPIPE` considerations.
+
+With libcurl older than 7.84.0, create an Ethon handle before starting other
+application threads so libcurl's global initialization happens while the
+process is still single-threaded.
+
+Ethon handles and initialized libcurl state must not be inherited across a
+`fork`. If any `Easy` or `Multi` was created before forking, libcurl is already
+initialized and the child cannot create new handles either; use `fork` followed
+by `exec` in that case. If libcurl was not initialized before forking, create
+all `Easy` and `Multi` instances after the fork. Ethon raises
+`Ethon::Errors::ForkSafety` when a child process tries to use inherited libcurl
+state.
+
+## License
 
 (The MIT License)
 

data/lib/ethon/curl.rb

@@ -19,22 +19,31 @@ module Ethon
     extend Ethon::Curls::FormOptions
     extend Ethon::Curls::Messages
 
+    require 'ethon/curls/share_constants'
+
     # :nodoc:
     def self.windows?
       Libc.windows?
     end
 
     require 'ethon/curls/constants'
+    require 'ethon/curls/introspection_constants'
     require 'ethon/curls/settings'
     require 'ethon/curls/classes'
+    require 'ethon/curls/url_constants'
     extend Ethon::Curls::Functions
 
     @blocking = true
 
     @@initialized = false
+    @@loaded_pid = Process.pid
     @@curl_mutex = Mutex.new
 
     class << self
+      def version_info
+        curl_version_info(VERSION_NOW)
+      end
+
       # This function sets up the program environment that libcurl needs.
       # Think of it as an extension of the library loader.
       #
@@ -56,6 +65,7 @@ module Ethon
       #
       # @raise [ Ethon::Errors::GlobalInit ] If Curl.global_init fails.
       def init
+        check_pid!
         @@curl_mutex.synchronize {
           if not @@initialized
             raise Errors::GlobalInit.new if Curl.global_init(GLOBAL_ALL) != 0
@@ -76,6 +86,7 @@ module Ethon
       # See the description in libcurl of global environment requirements
       # for details of how to use this function.
       def cleanup
+        check_pid!
         @@curl_mutex.synchronize {
           if @@initialized
             Curl.global_cleanup()
@@ -85,6 +96,17 @@ module Ethon
         }
       end
 
+      def check_pid!
+        return if @@loaded_pid == Process.pid
+
+        raise Errors::ForkSafety.new if @@initialized
+
+        @@loaded_pid = Process.pid
+        @@curl_mutex = Mutex.new
+      end
+
     end
   end
 end
+
+require 'ethon/curls/url'

data/lib/ethon/curls/classes.rb

@@ -1,6 +1,193 @@
 # frozen_string_literal: true
+require 'ethon/easy/util'
+
 module Ethon
   module Curl
+    class OwnedPointer < ::FFI::AutoPointer
+      OWNERS_MUTEX = Mutex.new
+      OWNERS = {}
+
+      def initialize(pointer)
+        @pid = Process.pid
+        registered = false
+        # Wrapping the same pointer twice would give two finalizers ownership
+        # of one native allocation. The first free marks it freed only for its
+        # own wrapper, so the other can still be used after free.
+        if pointer && !pointer.null?
+          OWNERS_MUTEX.synchronize do
+            if owner_pid = self.class.owners[pointer.address]
+              raise Errors::ForkSafety.new if owner_pid != @pid
+
+              raise ArgumentError, "native pointer is already owned by an Ethon handle"
+            end
+            self.class.owners[pointer.address] = @pid
+            registered = true
+          end
+        end
+
+        super
+      rescue Exception
+        # Do not reserve an address for a wrapper that never got created.
+        if registered
+          OWNERS_MUTEX.synchronize do
+            self.class.owners.delete(pointer.address) if self.class.owners[pointer.address] == @pid
+          end
+        end
+        raise
+      end
+
+      # Explicitly free the native pointer. The wrapper must not be used
+      # afterwards, because FFI keeps the original address instead of
+      # clearing it.
+      def free
+        @freed = true
+        super
+      end
+
+      def check_pid!
+        raise Errors::EthonError, "#{self.class} has already been freed" if @freed
+        raise Errors::ForkSafety.new if @pid != Process.pid
+      end
+
+      # Every owned pointer registers in the same table so callers can inspect
+      # all live native handles, regardless of their wrapper class.
+      def self.owners
+        OWNERS
+      end
+
+      def self.release(pointer)
+        owner_pid = begin
+          OWNERS_MUTEX.synchronize { owners.delete(pointer.address) }
+        rescue ThreadError, SignalException
+          # Finalizers can run in a trap context where locking is forbidden,
+          # or be interrupted by a signal. The process is exiting at this
+          # point, so the unsynchronized delete is best effort.
+          owners.delete(pointer.address)
+        end
+        native_release(pointer) if owner_pid == Process.pid
+      end
+    end
+
+    class EasyHandle < OwnedPointer
+      attr_accessor :error_buffer, :owner
+
+      def self.native_release(pointer)
+        Curl.easy_cleanup(pointer)
+      end
+    end
+
+    class MultiHandle < OwnedPointer
+      CALLBACKS_MUTEX = Mutex.new
+
+      # libcurl may invoke the socket callback from curl_multi_cleanup. Keep
+      # callbacks referenced until the native handle is done with them.
+      def self.callbacks
+        @callbacks ||= {}
+      end
+
+      def self.retain_callback(pointer, option, callback)
+        key = pointer.address
+        CALLBACKS_MUTEX.synchronize do
+          if callback
+            (callbacks[key] ||= {})[option] = callback
+          elsif (registry = callbacks[key])
+            registry.delete(option)
+            callbacks.delete(key) if registry.empty?
+          end
+        end
+      end
+
+      def self.native_release(pointer)
+        Curl.multi_cleanup(pointer)
+      ensure
+        begin
+          CALLBACKS_MUTEX.synchronize { callbacks.delete(pointer.address) }
+        rescue ThreadError, SignalException
+          # Finalizers can run in a trap context where locking is forbidden.
+          callbacks.delete(pointer.address)
+        end
+      end
+    end
+
+    class Slist < OwnedPointer
+      # Build a curl_slist from lines. Returns the raw pointer, which is owned
+      # by the caller; nil when there are no lines.
+      def self.build(lines)
+        list = nil
+        begin
+          lines.each do |line|
+            # libcurl writes slist entries verbatim. For list options such as
+            # FTP quotes and SMTP recipients, CR or LF would inject another
+            # protocol command, so reject them at the Ethon helper boundary.
+            value = Ethon::Easy::Util.validate_header_line(:slist, line.to_s)
+            appended = Curl.slist_append(list, value)
+            # libcurl returns NULL (not nil) and leaves the old list intact.
+            if appended.nil? || appended.null?
+              raise NoMemoryError, "curl_slist_append failed"
+            end
+
+            list = appended
+          end
+          list
+        rescue Exception
+          Curl.slist_free_all(list) if list
+          raise
+        end
+      end
+
+      def to_a
+        check_pid!
+        SlistNode.to_a(self)
+      end
+
+      def self.native_release(pointer)
+        Curl.slist_free_all(pointer)
+      end
+    end
+
+    class SlistNode < ::FFI::Struct
+      layout :data, :pointer,
+        :next, :pointer
+
+      def self.to_a(pointer)
+        values = []
+        until pointer.null?
+          node = new(pointer)
+          values << node[:data].read_string unless node[:data].null?
+          pointer = node[:next]
+        end
+        values
+      end
+    end
+
+    class FormData < OwnedPointer
+      def self.native_release(pointer)
+        Curl.formfree(pointer) if Curl.respond_to?(:formfree)
+      end
+    end
+
+    # struct curl_blob, used by the CURLOPT_*_BLOB options.
+    class Blob < ::FFI::Struct
+      layout :data, :pointer,
+             :len, :size_t,
+             :flags, :uint
+
+      # Copy a Ruby String into native memory and return a blob that keeps the
+      # copied buffer alive for as long as the blob is retained.
+      def self.from_string(value, flags = 0)
+        string = value.to_s
+        data = ::FFI::MemoryPointer.new(:char, string.bytesize)
+        data.put_bytes(0, string)
+
+        blob = new
+        blob[:data] = data
+        blob[:len] = string.bytesize
+        blob[:flags] = flags
+        blob.instance_variable_set(:@data, data)
+        blob
+      end
+    end
+
     # :nodoc:
     class MsgData < ::FFI::Union
       layout :whatever, :pointer, :code, :easy_code
@@ -12,7 +199,7 @@ module Ethon
     end
 
     class VersionInfoData < ::FFI::Struct
-      layout :curl_version, :uint8,
+      layout :curl_version, :int,
         :version, :string,
         :version_num, :int,
         :host, :string,
@@ -23,26 +210,98 @@ module Ethon
         :protocols, :pointer
     end
 
+    # curl_version_info_data through CURLVERSION_ELEVENTH. Only safe to use
+    # when the runtime age is at least CURLVERSION_ELEVENTH.
+    class VersionInfoDataFull < ::FFI::Struct
+      layout :curl_version, :int,
+        :version, :string,
+        :version_num, :int,
+        :host, :string,
+        :features, :int,
+        :ssl_version, :string,
+        :ssl_version_num, :long,
+        :libz_version, :string,
+        :protocols, :pointer,
+        :ares, :string,
+        :ares_num, :int,
+        :libidn, :string,
+        :iconv_ver_num, :int,
+        :libssh_version, :string,
+        :brotli_ver_num, :uint,
+        :brotli_version, :string,
+        :nghttp2_ver_num, :uint,
+        :nghttp2_version, :string,
+        :quic_version, :string,
+        :cainfo, :string,
+        :capath, :string,
+        :zstd_ver_num, :uint,
+        :zstd_version, :string,
+        :hyper_version, :string,
+        :gsasl_version, :string,
+        :feature_names, :pointer
+    end
+
+    # struct curl_header.
+    class HeaderData < ::FFI::Struct
+      layout :name, :string,
+        :value, :string,
+        :amount, :size_t,
+        :index, :size_t,
+        :origin, :uint,
+        :anchor, :pointer
+
+      def name; self[:name]; end
+      def value; self[:value]; end
+      def amount; self[:amount]; end
+      def index; self[:index]; end
+      def origin; self[:origin]; end
+    end
+
+    # Immutable copy of a curl_header. libcurl reuses the backing struct
+    # between curl_easy_nextheader calls, so values must be copied eagerly.
+    class Header
+      # libcurl documents the five low origin bits and reserves the rest. Mask
+      # the whole reserved range so callers see exactly the CURLH_* bits that
+      # HeaderOrigin describes, including bits libcurl might set later.
+      DOCUMENTED_ORIGIN_BITS = (1 << 5) - 1
+
+      attr_reader :name, :value, :amount, :index, :origin
+
+      def initialize(data)
+        @name = data.name
+        @value = data.value
+        @amount = data.amount
+        @index = data.index
+        @origin = data.origin & DOCUMENTED_ORIGIN_BITS
+      end
+    end
+
+    # struct curl_easyoption.
+    class EasyOptionInfo < ::FFI::Struct
+      layout :name, :string,
+        :id, :easy_option,
+        :type, :easy_type,
+        :flags, :uint
+
+      def name; self[:name]; end
+      def id; self[:id]; end
+      def type; self[:type]; end
+      def flags; self[:flags]; end
+      def alias?; (flags & 1) != 0; end
+    end
+
     # :nodoc:
     class FDSet < ::FFI::Struct
       if Curl.windows?
         layout :fd_count, :uint,
+               # SOCKET is UINT_PTR, so use the platform pointer-sized integer.
                # TODO: Make it future proof by dynamically grabbing FD_SETSIZE.
-               :fd_array, [:uint, 2048]
+               :fd_array, [:size_t, 2048]
 
         def clear; self[:fd_count] = 0; end
       else
         # https://github.com/typhoeus/ethon/issues/182
-        FD_SETSIZE = begin
-          # Allow to override the (new) default cap
-          if ENV['ETHON_FD_SIZE']
-            ENV['ETHON_FD_SIZE']
-
-          # auto-detect ulimit, but cap at 2^16
-          else
-            [::Ethon::Libc.getdtablesize, 65_536].min
-          end
-        end
+        FD_SETSIZE = ::Ethon::Libc.fd_size
 
         layout :fds_bits, [:long, FD_SETSIZE / ::FFI::Type::LONG.size]
 
@@ -61,5 +320,18 @@ module Ethon
                :usec, :suseconds_t
       end
     end
+
+    # :nodoc:
+    class Certinfo < ::FFI::Struct
+      layout :num_of_certs, :int,
+             :certinfo, :pointer
+             
+      def to_a
+        return [] if self[:num_of_certs] == 0
+        self[:certinfo].read_array_of_pointer(self[:num_of_certs]).map do |pointer|
+          SlistNode.to_a(pointer)
+        end
+      end
+    end
   end
 end