curb 0.8.5 → 1.3.6

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 11348a158e7da3d3406acd021be1a15f881ce2c607ff04e60b522a3bd92cd828
+  data.tar.gz: 0c524ce914df77b109a2aad85289386b2bf54028733fcf4ec5657a84abf511b8
+SHA512:
+  metadata.gz: b4ebd6b04943bfc099a6593ede1c70298555ed1a6613958e8628cde43c30b8c7f8b86990d91c7a350481589495fe7d5dc4ad1ac3c313823bb7fb21e483e8910c
+  data.tar.gz: 0a78bc41496259eae9989d25e4a47c57a9a9aed6f79b00e77efae691b487a730e71e6cdf8081ae35d33893a84b033219ae24c40cf47effd8e3d1b0ca3591f222

data/README.md

@@ -0,0 +1,579 @@
+# Curb - Libcurl bindings for Ruby
+
+[![CI](https://github.com/taf2/curb/actions/workflows/ci.yml/badge.svg)](https://github.com/taf2/curb/actions/workflows/ci.yml)
+[![codecov](https://codecov.io/gh/taf2/curb/branch/master/graph/badge.svg)](https://codecov.io/gh/taf2/curb)
+[![Gem Version](https://badge.fury.io/rb/curb.svg)](https://badge.fury.io/rb/curb)
+
+* [CI Build Status](https://github.com/taf2/curb/actions/workflows/ci.yml)
+* [rubydoc rdoc](http://www.rubydoc.info/github/taf2/curb/)
+* [github project](http://github.com/taf2/curb/tree/master)
+
+Curb (probably CUrl-RuBy or something) provides Ruby-language bindings for the
+libcurl(3), a fully-featured client-side URL transfer library.
+cURL and libcurl live at [https://curl.se/libcurl/](https://curl.se/libcurl/) .
+
+Curb is a work-in-progress, and currently only supports libcurl's `easy` and `multi` modes.
+
+A big advantage to Curb over all other known ruby http libraries is it's ability to handle timeouts without the use of threads.
+
+## License
+
+Curb is copyright (c) 2006 Ross Bamford, and released under the terms of the
+Ruby license. See the LICENSE file for the gory details.
+
+## Easy mode
+
+GET request
+```
+  res = Curl.get("https://www.google.com/") {|http|
+    http.timeout = 10 # raise exception if request/response not handled within 10 seconds
+  }
+  puts res.code
+  puts res.head
+  puts res.body
+```
+
+POST request
+```
+  res = Curl.post("https://your-server.com/endpoint", {post: "this"}.to_json) {|http|
+    http.headers["Content-Type"] = "application/json"
+  }
+  puts res.code
+  puts res.head
+  puts res.body
+```
+
+## FTP Support
+
+require 'curb'
+
+### Basic FTP Download
+```ruby
+puts "=== FTP Download Example ==="
+ftp = Curl::Easy.new('ftp://ftp.example.com/remote/file.txt')
+ftp.username = 'user'
+ftp.password = 'password'
+ftp.perform
+puts ftp.body
+```
+
+### FTP Upload
+```ruby
+puts "\n=== FTP Upload Example ==="
+upload = Curl::Easy.new('ftp://ftp.example.com/remote/upload.txt')
+upload.username = 'user'
+upload.password = 'password'
+upload.upload = true
+upload.put_data = File.read('local_file.txt')
+upload.perform
+```
+
+### List Directory Contents
+```ruby
+puts "\n=== FTP Directory Listing Example ==="
+list = Curl::Easy.new('ftp://ftp.example.com/remote/directory/')
+list.username = 'user'
+list.password = 'password'
+list.set(:dirlistonly, 1)
+list.perform
+puts list.body
+```
+
+### FTP over HTTP proxy tunnel (NLST/LIST)
+When listing directories through an HTTP proxy with `proxy_tunnel` (CONNECT), let libcurl manage the passive data connection. Do not send `PASV`/`EPSV` or `NLST` via `easy.ftp_commands` — QUOTE commands run on the control connection and libcurl will not open the data connection, resulting in 425 errors.
+
+To get NLST-like output safely:
+
+```ruby
+list = Curl::Easy.new('ftp://ftp.example.com/remote/directory/')
+list.username = 'user'
+list.password = 'password'
+list.proxy_url = 'http://proxy.example.com:80'
+list.proxy_tunnel = true
+
+# Ask libcurl to perform a listing (names only)
+list.set(:dirlistonly, 1)
+
+# If the proxy or server has trouble with EPSV/EPRT, you can adjust:
+# list.set(:ftp_use_epsv, 0)       # disable EPSV
+# list.set(:ftp_use_eprt, 0)       # disable EPRT (stick to IPv4 PASV)
+# list.set(:ftp_skip_pasv_ip, 1)   # ignore PASV host, reuse control host
+
+list.perform
+puts list.body
+```
+
+If you need a full `LIST` output instead of just names, omit `dirlistonly` and parse the server response accordingly. The key is to let libcurl initiate the data connection (PASV/EPSV) instead of trying to drive it via `ftp_commands`.
+
+#### Full LIST directory listing
+To retrieve the full `LIST` output (permissions, owner, size, timestamp, name), simply do not set `dirlistonly`:
+
+```ruby
+list = Curl::Easy.new('ftp://ftp.example.com/remote/directory/')
+list.username = 'user'
+list.password = 'password'
+
+# Explicitly ensure names+metadata (LIST) rather than NLST
+# list.set(:dirlistonly, 0) # optional; default is LIST for directory URLs
+
+list.perform
+puts list.body # multi-line LIST output
+```
+
+Through an HTTP proxy tunnel, the same considerations apply as the NLST example above — just omit `dirlistonly` and keep the optional EPSV/EPRT/PASV tweaks if needed:
+
+```ruby
+list = Curl::Easy.new('ftp://ftp.example.com/remote/directory/')
+list.username = 'user'
+list.password = 'password'
+list.proxy_url = 'http://proxy.example.com:80'
+list.proxy_tunnel = true
+
+# Optional tweaks if the proxy/server combination struggles
+# list.set(:ftp_use_epsv, 0)
+# list.set(:ftp_use_eprt, 0)
+# list.set(:ftp_skip_pasv_ip, 1)
+
+list.perform
+puts list.body
+```
+
+### Advanced FTP Usage with Various Options
+```
+puts "\n=== Advanced FTP Example ==="
+advanced = Curl::Easy.new do |curl|
+  curl.url = 'ftp://ftp.example.com/remote/file.txt'
+  curl.username = 'user'
+  curl.password = 'password'
+
+  # FTP Options
+  curl.ftp_response_timeout = 30
+  curl.ftp_create_missing_dirs = true   # Create directories if they don't exist
+  curl.ftp_filemethod = Curl::CURL_MULTICWD  # Use multicwd method for traversing paths
+
+  # SSL/TLS Options for FTPS
+  curl.use_ssl = Curl::CURLUSESSL_ALL  # Use SSL/TLS for control and data
+  curl.ssl_verify_peer = true
+  curl.ssl_verify_host = true
+  curl.cacert = "/path/to/cacert.pem"
+
+  # Progress callback
+  curl.on_progress do |dl_total, dl_now, ul_total, ul_now|
+    puts "Download: #{dl_now}/#{dl_total} Upload: #{ul_now}/#{ul_total}"
+    true # must return true to continue
+  end
+
+  # Debug output
+  curl.verbose = true
+  curl.on_debug do |type, data|
+    puts "#{type}: #{data}"
+    true
+  end
+end
+
+advanced.perform
+```
+
+### Parallel FTP Downloads
+```
+puts "\n=== Parallel FTP Downloads Example ==="
+urls = [
+  'ftp://ftp.example.com/file1.txt',
+  'ftp://ftp.example.com/file2.txt',
+  'ftp://ftp.example.com/file3.txt'
+]
+```
+
+### Common options for all connections
+```
+options = {
+  :username => 'user',
+  :password => 'password',
+  :timeout => 30,
+  :on_success => proc { |easy| puts "Successfully downloaded: #{easy.url}" },
+  :on_failure => proc { |easy, code| puts "Failed to download: #{easy.url} (#{code})" }
+}
+
+Curl::Multi.download(urls, options) do |curl, file_path|
+  puts "Completed downloading to: #{file_path}"
+end
+```
+
+## Security considerations
+
+`curb` is a libcurl binding and intentionally supports protocols beyond HTTP.
+Do not pass untrusted URLs to `Curl.get`, `Curl::Easy.new`, or related raw
+helpers without application-level validation. For user-supplied URLs, enable the
+process-wide safety policy before making requests:
+
+```ruby
+Curl.safe! do |config|
+  config.network_policy = :public       # block local/private destination IPs
+  config.max_body_bytes = 1_000_000     # cap buffered/callback response bytes
+end
+
+curl = Curl.get(user_url)               # allows only http/https, including redirects
+```
+
+To allow a different protocol set, configure it explicitly. Redirects default to
+the same protocol list:
+
+```ruby
+Curl.safe! do |config|
+  config.protocols = [:http, :ftp]
+  config.max_body_bytes = 1_000_000
+end
+```
+
+For local per-handle policy instead of process-wide policy, use
+`easy.safe_http!` and `easy.max_body_bytes = ...` before `perform`.
+
+With `network_policy = :public`, curb checks peer addresses when libcurl opens
+the socket and blocks local/private destinations. Proxies, `resolve`,
+`connect_to`, DoH URL overrides, and Unix socket paths are disabled by default
+under this policy unless explicitly allowed in the safety config. Custom DNS
+server overrides are rejected. To use a trusted explicit proxy without
+re-enabling environment proxies, set `allowed_proxy_hosts` and configure
+`easy.proxy_url` on the request.
+
+For stricter egress, combine the public network policy with host and CIDR
+allowlists. Host allowlists gate the configured request URL and, when supported
+by libcurl, each followed redirect before the request is sent. CIDR allowlists
+are checked against the resolved peer address at socket-open time:
+
+```ruby
+Curl.safe! do |config|
+  config.network_policy = :public
+  config.allowed_hosts = ["api.example.com"]
+  config.allowed_proxy_hosts = ["proxy.example.com"]
+  config.allowed_cidrs = ["93.184.216.0/24", "2606:2800:220::/48"]
+end
+```
+
+By default, responses are buffered into `body` when no `on_body` callback is
+configured. For untrusted or large responses, use `on_body`, `download`, and/or
+`max_body_bytes` so a remote endpoint cannot force unbounded memory growth.
+`max_body_bytes` is enforced for downloads as well as buffered responses and
+custom body callbacks.
+
+## You will need
+
+* A working Ruby installation (`2.0.0+` will work but `2.1+` preferred) (it's possible it still works with 1.8.7 but you'd have to tell me if not...)
+* A working libcurl development installation
+(Ideally one of the versions listed in the compatibility chart below that maps to your `curb` version)
+* A sane build environment (e.g. gcc, make)
+
+## Version Compatibility chart
+
+A **non-exhaustive** set of compatibility versions of the libcurl library
+with this gem are as follows. (Note that these are only the ones that have been
+tested and reported to work across a variety of platforms / rubies)
+
+| Gem Version | Release Date   | libcurl versions  |
+| ----------- | -------------- | ----------------- |
+| 1.0.8       | Feb 10, 2025   | 7.58 – 8.12.1     |
+| 1.0.7       | Feb 09, 2025   | 7.58 – 8.12.1     |
+| 1.0.6       | Aug 23, 2024   | 7.58 – 8.12.1     |
+| 1.0.5       | Jan 2023       | 7.58 – 8.12.1     |
+| 1.0.4       | Jan 2023       | 7.58 – 8.12.1     |
+| 1.0.3*      | Dec 2022       | 7.58 – 8.12.1     |
+| 1.0.2*      | Dec 2022       | 7.58 – 8.12.1     |
+| 1.0.1       | Apr 2022       | 7.58 – 8.12.1     |
+| 1.0.0       | Jan 2022       | 7.58 – 8.12.1     |
+| 0.9.8       | Jan 2019       | 7.58 – 7.81       |
+| 0.9.7       | Nov 2018       | 7.56 – 7.60       |
+| 0.9.6       | May 2018       | 7.51 – 7.59       |
+| 0.9.5       | May 2018       | 7.51 – 7.59       |
+| 0.9.4       | Aug 2017       | 7.41 – 7.58       |
+| 0.9.3       | Apr 2016       | 7.26 – 7.58       |
+
+  ```*avoid using these version are known to have issues with segmentation faults```
+
+## Installation...
+
+... will usually be as simple as:
+
+    $ gem install curb
+
+On Windows, make sure you're using the [DevKit](http://rubyinstaller.org/downloads/) and
+the [development version of libcurl](http://curl.se/gknw.net/7.39.0/dist-w32/curl-7.39.0-devel-mingw32.zip). Unzip, then run this in your command
+line (alter paths to your curl location, but remember to use forward slashes):
+
+    gem install curb --platform=ruby -- --with-curl-lib=C:/curl-7.39.0-devel-mingw32/lib --with-curl-include=C:/curl-7.39.0-devel-mingw32/include
+    
+Note that with Windows moving from one method of compiling to another as of Ruby `2.4` (DevKit -> MYSYS2),
+the usage of Ruby `2.4+` with this gem on windows is unlikely to work. It is advised to use the
+latest version of Ruby 2.3 available [HERE](https://dl.bintray.com/oneclick/rubyinstaller/rubyinstaller-2.3.3.exe)
+
+Or, if you downloaded the archive:
+
+    $ rake compile && rake install
+
+If you have a weird setup, you might need extconf options. In this case, pass
+them like so:
+
+    $ rake compile EXTCONF_OPTS='--with-curl-dir=/path/to/libcurl --prefix=/what/ever' && rake install
+
+Curb is tested only on GNU/Linux x86 and Mac OSX - YMMV on other platforms.
+If you do use another platform and experience problems, or if you can
+expand on the above instructions, please report the issue at http://github.com/taf2/curb/issues
+
+On Ubuntu, the dependencies can be satisfied by installing the following packages:
+
+18.04 and onwards
+    
+    $ sudo apt-get install libcurl4 libcurl3-gnutls libcurl4-openssl-dev
+
+< 18.04
+
+    $ sudo apt-get install libcurl3 libcurl3-gnutls libcurl4-openssl-dev
+
+On RedHat:
+
+    $ sudo yum install ruby-devel libcurl-devel openssl-devel
+
+Curb has fairly extensive RDoc comments in the source. You can build the
+documentation with:
+
+    $ rake doc
+
+## Usage & examples
+
+Curb provides two classes:
+
+* `Curl::Easy` - simple API, for day-to-day tasks.
+* `Curl::Multi` - more advanced API, for operating on multiple URLs simultaneously.
+
+To use either, you will need to require the curb gem:
+
+```ruby
+require 'curb'
+```
+
+### Super simple API (less typing)
+
+```ruby
+http = Curl.get("http://www.google.com/")
+puts http.body
+
+http = Curl.post("http://www.google.com/", {:foo => "bar"})
+puts http.body
+
+http = Curl.get("http://www.google.com/") do |http|
+  http.headers['Cookie'] = 'foo=1;bar=2'
+end
+puts http.body
+```
+
+### Simple fetch via HTTP:
+
+```ruby
+c = Curl::Easy.perform("http://www.google.co.uk")
+puts c.body
+```
+
+Same thing, more manual:
+
+```ruby
+c = Curl::Easy.new("http://www.google.co.uk")
+c.perform
+puts c.body
+```
+
+### Additional config:
+
+```ruby
+http = Curl::Easy.perform("http://www.google.co.uk") do |curl|
+  curl.headers["User-Agent"] = "myapp-0.0"
+  curl.verbose = true
+end
+```
+
+Same thing, more manual:
+
+```ruby
+c = Curl::Easy.new("http://www.google.co.uk") do |curl|
+  curl.headers["User-Agent"] = "myapp-0.0"
+  curl.verbose = true
+end
+
+c.perform
+```
+
+### HTTP basic authentication:
+
+```ruby
+c = Curl::Easy.new("http://github.com/")
+c.http_auth_types = :basic
+c.username = 'foo'
+c.password = 'bar'
+c.perform
+```
+
+### HTTP "insecure" SSL connections (like curl -k, --insecure) to avoid Curl::Err::SSLCACertificateError:
+
+```ruby
+c = Curl::Easy.new("https://github.com/")
+c.ssl_verify_peer = false
+c.perform
+```
+
+### Supplying custom handlers:
+
+```ruby
+c = Curl::Easy.new("http://www.google.co.uk")
+
+c.on_body { |data| print(data) }
+c.on_header { |data| print(data) }
+
+c.perform
+```
+
+### Reusing Curls:
+
+```ruby
+c = Curl::Easy.new
+
+["http://www.google.co.uk", "http://www.ruby-lang.org/"].map do |url|
+  c.url = url
+  c.perform
+  c.body
+end
+```
+
+### HTTP POST form:
+
+Note: Instance methods like `easy.http_post(...)` do not accept a URL argument. Set the URL first (for example, `Curl::Easy.new(url)` or `easy.url = url`) and then call `easy.http_post(...)`. If you want to pass the URL directly to the call, use the class/module helpers such as `Curl::Easy.http_post(url, ...)` or `Curl.post(url, ...)`.
+
+```ruby
+c = Curl::Easy.http_post("http://my.rails.box/thing/create",
+                         Curl::PostField.content('thing[name]', 'box'),
+                         Curl::PostField.content('thing[type]', 'storage'))
+```
+
+### HTTP POST file upload:
+
+```ruby
+c = Curl::Easy.new("http://my.rails.box/files/upload")
+c.multipart_form_post = true
+c.http_post(Curl::PostField.file('thing[file]', 'myfile.rb'))
+
+### Custom request target
+
+Some advanced scenarios need a request-target that differs from the URL host/path (for example, absolute-form targets or special values like `*`). If your libcurl supports `CURLOPT_REQUEST_TARGET` (libcurl ≥ 7.55), you can override it:
+
+```ruby
+c = Curl::Easy.new("http://127.0.0.1:9129/methods")
+c.request_target = "http://localhost:9129/methods"  # absolute-form target
+c.headers = { 'Host' => 'example.com' }              # override Host header if needed
+c.perform
+```
+
+For HTTPS, prefer `easy.resolve = ["host:443:IP"]` to keep Host/SNI/certificates aligned.
+```
+
+### Using HTTP/2
+
+```ruby
+c = Curl::Easy.new("https://http2.akamai.com")
+c.set(:HTTP_VERSION, Curl::HTTP_2_0)
+
+c.perform
+puts (c.body.include? "You are using HTTP/2 right now!") ? "HTTP/2" : "HTTP/1.x"
+```
+
+### Multi Interface (Basic HTTP GET):
+
+```ruby
+# make multiple GET requests
+easy_options = {:follow_location => true}
+# Use Curl::CURLPIPE_MULTIPLEX for HTTP/2 multiplexing
+multi_options = {:pipeline => Curl::CURLPIPE_HTTP1}
+
+Curl::Multi.get(['url1','url2','url3','url4','url5'], easy_options, multi_options) do|easy|
+  # do something interesting with the easy response
+  puts easy.last_effective_url
+end
+```
+
+### Multi Interface (Basic HTTP POST):
+
+```ruby
+# make multiple POST requests
+easy_options = {:follow_location => true, :multipart_form_post => true}
+multi_options = {:pipeline => Curl::CURLPIPE_HTTP1}
+
+
+url_fields = [
+  { :url => 'url1', :post_fields => {'f1' => 'v1'} },
+  { :url => 'url2', :post_fields => {'f1' => 'v1'} },
+  { :url => 'url3', :post_fields => {'f1' => 'v1'} }
+]
+
+Curl::Multi.post(url_fields, easy_options, multi_options) do|easy|
+  # do something interesting with the easy response
+  puts easy.last_effective_url
+end
+```
+
+### Multi Interface (Advanced):
+
+```ruby
+responses = {}
+requests = ["http://www.google.co.uk/", "http://www.ruby-lang.org/"]
+m = Curl::Multi.new
+# add a few easy handles
+requests.each do |url|
+  responses[url] = ""
+  c = Curl::Easy.new(url) do|curl|
+    curl.follow_location = true
+    curl.on_body{|data| responses[url] << data; data.size }
+    curl.on_success {|easy| puts "success, add more easy handles" }
+  end
+  m.add(c)
+end
+
+m.perform do
+  puts "idling... can do some work here"
+end
+
+requests.each do|url|
+  puts responses[url]
+end
+```
+
+### Easy Callbacks
+
+* `on_success`  is called when the response code is 2xx
+* `on_redirect` is called when the response code is 3xx
+* `on_missing` is called when the response code is 4xx
+* `on_failure` is called when the response code is 5xx
+* `on_complete` is called in all cases.
+
+### Cookies
+
+- Manual cookies: Set the outgoing `Cookie` header via `easy.cookies = "name=value; other=val"`. This only affects the request header and does not modify libcurl's internal cookie engine.
+- Cookie engine: Enable with `easy.enable_cookies = true`. Optionally set `easy.cookiefile` (to load) and/or `easy.cookiejar` (to persist). Cookies received via `Set-Cookie` go into this engine.
+- Inspect engine cookies: `easy.cookielist` returns an array of strings (Netscape or Set-Cookie format).
+- Modify engine cookies: use `easy.cookielist = ...` or `easy.set(:cookielist, ...)` with either a `Set-Cookie` style string, Netscape cookie lines, or special commands: `"ALL"` (clear), `"SESS"` (remove session cookies), `"FLUSH"` (write to jar), `"RELOAD"` (reload from file).
+- Clearing manual cookies: assign an empty string (`easy.cookies = ''`). Assigning `nil` has no effect in current versions.
+
+Examples:
+
+```ruby
+easy = Curl::Easy.new("https://example.com")
+
+# Use the cookie engine and persist cookies
+easy.enable_cookies = true
+easy.cookiejar = "/tmp/cookies.txt"
+easy.perform
+
+# Later: inspect and tweak engine cookies
+p easy.cookielist
+easy.cookielist = 'ALL' # clear stored cookies
+
+# Send custom Cookie header for a single request
+easy.cookies = "flag=1; session_override=abc"
+easy.perform
+easy.cookies = ''  # clear manual Cookie header
+```