ruby-proxy-headers 0.2.2 → 0.3.0

data/docs/index.md

@@ -0,0 +1,109 @@
+# ruby-proxy-headers
+
+Send and receive custom proxy headers during HTTPS CONNECT tunneling in Ruby.
+
+## The Problem
+
+When making HTTPS requests through a proxy server, a tunnel is established using the HTTP CONNECT method:
+
+```
+Client → CONNECT example.com:443 HTTP/1.1 → Proxy → Target
+       ← 200 Connection established ←
+       ←→ [TLS encrypted traffic] ←→
+```
+
+Standard Ruby HTTP libraries support basic proxy authentication but **do not** allow:
+
+- Sending additional custom headers during CONNECT (e.g., `X-ProxyMesh-Country`)
+- Accessing headers returned by the proxy (e.g., `X-ProxyMesh-IP`)
+
+**ruby-proxy-headers** solves both problems.
+
+## Features
+
+- **Send custom headers** to the proxy during HTTPS CONNECT
+- **Receive proxy response headers** from the CONNECT response
+- **Support for 7 popular Ruby HTTP libraries**
+- **Thread-safe** implementation
+- **Pure Ruby** core (no native extensions required)
+
+## Supported Libraries
+
+| Library | Module | Status |
+|---------|--------|--------|
+| Net::HTTP | `RubyProxyHeaders::NetHTTP` | ✅ |
+| Faraday | `RubyProxyHeaders::FaradayIntegration` | ✅ |
+| HTTParty | `RubyProxyHeaders::ProxyHeadersConnectionAdapter` | ✅ |
+| HTTP.rb | `RubyProxyHeaders::HTTPGem` | ✅, http gem >= 6 |
+| Typhoeus | `RubyProxyHeaders::Typhoeus` | ✅, Typhoeus ~> 1.7 |
+| Excon | `RubyProxyHeaders::ExconIntegration` | Send only |
+| RestClient | `RubyProxyHeaders::RestClient` | ✅ |
+
+## Quick Start
+
+```ruby
+require 'uri'
+require 'ruby_proxy_headers/net_http'
+
+RubyProxyHeaders::NetHTTP.patch!
+
+uri = URI('https://api.ipify.org?format=json')
+proxy = URI('http://user:pass@proxy.example.com:31280')
+http = Net::HTTP.new(uri.host, uri.port, proxy.host, proxy.port, proxy.user, proxy.password)
+http.use_ssl = true
+http.proxy_connect_request_headers = { 'X-ProxyMesh-Country' => 'US' }
+response = http.request(Net::HTTP::Get.new(uri))
+
+puts response.code
+puts response.body
+puts http.last_proxy_connect_response_headers['X-ProxyMesh-IP']
+```
+
+## Use Cases
+
+### ProxyMesh Integration
+
+[ProxyMesh](https://docs.proxymesh.com/) uses custom headers for:
+
+- **Country selection**: `X-ProxyMesh-Country: US`
+- **Session persistence**: `X-ProxyMesh-Session: abc123`
+- **IP assignment feedback**: Returns `X-ProxyMesh-IP` in response
+
+### Proxy Service Headers
+
+Many proxy services use custom headers for:
+
+- Geolocation targeting
+- Session management
+- Usage tracking
+- Load balancing hints
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem 'ruby-proxy-headers'
+```
+
+Then run:
+
+```bash
+bundle install
+```
+
+Or install directly:
+
+```bash
+gem install ruby_proxy_headers
+```
+
+## Requirements
+
+- Ruby >= 3.2
+- OpenSSL (included in standard Ruby)
+
+## Related Projects
+
+- [python-proxy-headers](https://github.com/proxymesh/python-proxy-headers) - Python version
+- [javascript-proxy-headers](https://github.com/proxymeshai/javascript-proxy-headers) - JavaScript/Node.js version

data/docs/net-http.md

@@ -0,0 +1,29 @@
+# Net::HTTP
+
+`RubyProxyHeaders::NetHTTP.patch!` prepends a small wrapper around the installed `Net::HTTP#connect`. The gem does not replace that method with its own copy. Call `patch!` once before opening connections.
+
+Proxy header names are documented at <https://docs.proxymesh.com/>.
+
+## Usage
+
+```ruby
+require 'uri'
+require 'ruby_proxy_headers/net_http'
+
+RubyProxyHeaders::NetHTTP.patch!
+
+uri = URI('https://api.ipify.org?format=json')
+proxy = URI('http://user:pass@proxy.example.com:31280')
+
+http = Net::HTTP.new(uri.host, uri.port, proxy.host, proxy.port, proxy.user, proxy.password)
+http.use_ssl = true
+http.proxy_connect_request_headers = { 'X-ProxyMesh-Country' => 'US' }
+
+response = http.request(Net::HTTP::Get.new(uri))
+puts response.body
+puts http.last_proxy_connect_response_headers['X-ProxyMesh-IP']
+```
+
+`last_proxy_connect_response_headers` is case-insensitive. The same headers are on `RubyProxyHeaders.proxy_connect_response_headers` for the current thread.
+
+An `https://` proxy needs net-http 0.5 or newer, which is what `proxy_use_ssl` requires. Set `http.proxy_use_ssl = true` for that proxy.

data/docs/rest-client.md

@@ -0,0 +1,96 @@
+# RestClient
+
+[RestClient](https://github.com/rest-client/rest-client) is a simple REST client. The `RubyProxyHeaders::RestClient` module adds proxy header support.
+
+## Installation
+
+```ruby
+gem 'rest-client'
+gem 'ruby-proxy-headers'
+```
+
+## Basic Usage
+
+```ruby
+require 'ruby_proxy_headers/rest_client'
+
+response = RubyProxyHeaders::RestClient.get(
+  'https://api.ipify.org?format=json',
+  proxy: 'http://user:pass@proxy.example.com:31280',
+  proxy_headers: { 'X-ProxyMesh-Country' => 'US' }
+)
+
+puts "Status: #{response.code}"
+puts "Body: #{response.body}"
+puts "Proxy IP: #{response.proxy_response_headers['x-proxymesh-ip']}"
+```
+
+## API Reference
+
+### `RubyProxyHeaders::RestClient.get(url, options)`
+
+Make a GET request.
+
+**Parameters:**
+
+- `url` (String) - Target URL
+- `options` (Hash):
+  - `:proxy` (String) - Proxy URL (required)
+  - `:proxy_headers` (Hash) - Headers to send to proxy
+  - `:headers` (Hash) - Request headers
+
+**Returns:** `ProxyResponse`
+
+### `RubyProxyHeaders::RestClient.post(url, options)`
+
+Make a POST request.
+
+**Parameters:**
+
+- Same as `get`, plus:
+  - `:payload` (String/Hash) - Request body
+
+### `RubyProxyHeaders::RestClient.put(url, options)`
+
+Make a PUT request.
+
+### `RubyProxyHeaders::RestClient.delete(url, options)`
+
+Make a DELETE request.
+
+## Response Object
+
+```ruby
+response.code                    # HTTP status code
+response.body                    # Response body
+response.headers                 # Response headers
+response.proxy_response_headers  # Proxy CONNECT response headers
+```
+
+## Example: RESTful API
+
+```ruby
+require 'ruby_proxy_headers/rest_client'
+require 'json'
+
+proxy_opts = {
+  proxy: ENV['PROXY_URL'],
+  proxy_headers: { 'X-ProxyMesh-Country' => 'US' }
+}
+
+# Create
+response = RubyProxyHeaders::RestClient.post(
+  'https://jsonplaceholder.typicode.com/posts',
+  payload: { title: 'Test', body: 'Content', userId: 1 }.to_json,
+  headers: { 'Content-Type' => 'application/json' },
+  **proxy_opts
+)
+puts "Created: #{JSON.parse(response.body)['id']}"
+
+# Read
+response = RubyProxyHeaders::RestClient.get(
+  'https://jsonplaceholder.typicode.com/posts/1',
+  **proxy_opts
+)
+puts "Read: #{JSON.parse(response.body)['title']}"
+```

data/docs/testing.md

@@ -0,0 +1,105 @@
+# Testing
+
+This guide covers running tests for ruby-proxy-headers.
+
+## Running Unit Tests
+
+```bash
+# Install dependencies
+bundle install
+
+# Run all tests
+rake spec
+
+# Run with verbose output
+rake test
+
+# Run specific test file
+bundle exec rspec spec/connection_spec.rb
+```
+
+## Running Integration Tests
+
+Integration tests require an actual proxy server.
+
+### Environment Variables
+
+```bash
+# Required
+export PROXY_URL='http://user:pass@proxy.example.com:31280'
+
+# Optional
+export TEST_URL='https://api.ipify.org?format=json'
+export PROXY_HEADER='X-ProxyMesh-IP'
+export SEND_PROXY_HEADER='X-ProxyMesh-Country'
+export SEND_PROXY_VALUE='US'
+```
+
+### Running Tests
+
+```bash
+# Run all integration tests
+ruby test/test_proxy_headers.rb
+
+# Run specific tests
+ruby test/test_proxy_headers.rb net_http faraday
+
+# List available tests
+ruby test/test_proxy_headers.rb -l
+
+# Verbose output
+ruby test/test_proxy_headers.rb -v
+```
+
+The harness prints `Testing <module>... OK` or `FAIL`. Module names are `net_http`, `faraday`, `httparty`, `excon`, `typhoeus`, `http_gem`, and `rest_client`. Excon checks that the proxied request succeeds. It does not read CONNECT response headers.
+
+## Writing Tests
+
+### Unit Test Example
+
+```ruby
+# spec/my_module_spec.rb
+require 'spec_helper'
+
+RSpec.describe MyModule do
+  describe '#my_method' do
+    it 'does something' do
+      result = MyModule.my_method
+      expect(result).to eq(expected_value)
+    end
+  end
+end
+```
+
+### Integration Test Example
+
+```ruby
+# test/my_integration_test.rb
+require 'uri'
+require_relative '../lib/ruby_proxy_headers'
+
+proxy_url = ENV['PROXY_URL']
+raise 'Set PROXY_URL' unless proxy_url
+
+RubyProxyHeaders::NetHTTP.patch!
+uri = URI('https://api.ipify.org?format=json')
+proxy = URI(proxy_url)
+http = Net::HTTP.new(uri.host, uri.port, proxy.host, proxy.port, proxy.user, proxy.password)
+http.use_ssl = true
+http.proxy_connect_request_headers = { 'X-Test' => 'value' }
+response = http.request(Net::HTTP::Get.new(uri))
+puts response.code
+puts http.last_proxy_connect_response_headers['X-ProxyMesh-IP']
+```
+
+## CI
+
+`.github/workflows/unit_tests.yml` runs `bundle exec rspec` on Ruby 3.2 and 3.3. `.github/workflows/proxy_integration_tests.yml` runs `test/test_proxy_headers.rb` when the `PROXY_URL` Actions secret is set.
+
+## Troubleshooting
+
+Set `open_timeout` and `read_timeout` on the client you are using. For a mock origin with a self-signed certificate, disable verification on that client (`verify_mode`, Faraday `ssl.verify`, HTTP.rb `verify_ssl: false`, or RestClient `verify_ssl: false`). Do not do that against a public site.
+
+```bash
+bundle install
+```

data/docs/typhoeus.md

@@ -0,0 +1,47 @@
+# Typhoeus
+
+[Typhoeus](https://typhoeus.github.io/) wraps libcurl. `RubyProxyHeaders::Typhoeus` sends custom proxy headers with `CURLOPT_PROXYHEADER` and reads the CONNECT response.
+
+Requires Typhoeus 1.7 (Ethon >= 0.19.1) and libcurl with `CURLOPT_PROXYHEADER`. The gem does not depend on Typhoeus at runtime.
+
+Proxy header names are documented at <https://docs.proxymesh.com/>.
+
+## Installation
+
+```ruby
+gem 'typhoeus', '~> 1.7'
+gem 'ruby-proxy-headers'
+```
+
+## Basic Usage
+
+```ruby
+require 'ruby_proxy_headers/typhoeus'
+
+response = RubyProxyHeaders::Typhoeus.get(
+  'https://api.ipify.org?format=json',
+  proxy: 'http://user:pass@proxy.example.com:31280',
+  proxy_headers: { 'X-ProxyMesh-Country' => 'US' }
+)
+
+puts response.code
+puts response.body
+puts response.proxy_response_headers['X-ProxyMesh-IP']
+```
+
+`proxy_headers` are sent only to the proxy (`headeropt: :separate`). On HTTPS that is the CONNECT request. `headers:` is the origin request and is not copied onto CONNECT.
+
+`response.headers` stays the origin response. The CONNECT response, including `X-ProxyMesh-IP`, is `response.proxy_response_headers`. Lookup is case-insensitive.
+
+`post` takes the same proxy arguments plus `body:`.
+
+```ruby
+RubyProxyHeaders::Typhoeus.post(
+  'https://api.ipify.org?format=json',
+  proxy: 'http://user:pass@proxy.example.com:31280',
+  proxy_headers: { 'X-ProxyMesh-Country' => 'US' },
+  body: '{"q":"example"}'
+)
+```
+
+Header names and values are rejected when they contain CR, LF, or NUL.

data/lib/ruby_proxy_headers/connection.rb

@@ -135,14 +135,7 @@ module RubyProxyHeaders
     end
 
     def validate_connect_target!(host, port)
-      if RubyProxyHeaders::INVALID_HEADER_VALUE_RE.match?(host.to_s)
-        raise ArgumentError,
-              "CONNECT target host contains invalid characters (CR, LF, or NUL): #{host.inspect}"
-      end
-      if RubyProxyHeaders::INVALID_HEADER_VALUE_RE.match?(port.to_s)
-        raise ArgumentError,
-              "CONNECT target port contains invalid characters (CR, LF, or NUL): #{port.inspect}"
-      end
+      RubyProxyHeaders.validate_connect_target!(host, port)
     end
 
     def raise_connect_error

data/lib/ruby_proxy_headers/faraday.rb

@@ -14,28 +14,31 @@ end
 module RubyProxyHeaders
   # Faraday adapter that merges proxy CONNECT response headers into Faraday response headers.
   # Use with {RubyProxyHeaders::NetHTTP.patch!}.
+  #
+  # faraday-net_http 3.4 and 3.5 perform the request in the private method
+  # +request_with_wrapped_block+. This adapter does not override that method.
+  # It uses the public +build_connection+ and +call+ methods.
   module FaradayAdapter
+    # Copies CONNECT response headers onto the Faraday response.
     class NetHttp < ::Faraday::Adapter::NetHttp
-      def request_with_wrapped_block(http, env, &block)
-        res = super
-        merge_proxy_headers(env, http)
-        res
+      def build_connection(env)
+        super.tap { |http| Thread.current[:ruby_proxy_headers_faraday_http] = http }
       end
 
-      private
+      def call(env)
+        response = super
+        http = Thread.current[:ruby_proxy_headers_faraday_http]
+        headers = http.last_proxy_connect_response_headers if http
+        if env[:response_headers] && headers
+          headers.each do |key, value|
+            next if value.nil?
 
-      def merge_proxy_headers(env, http)
-        return unless env[:response_headers]
-        return unless http.respond_to?(:last_proxy_connect_response_headers)
-
-        ph = http.last_proxy_connect_response_headers
-        return unless ph.is_a?(Hash)
-
-        ph.each do |k, v|
-          next if v.nil?
-
-          env[:response_headers][k] ||= v
+            env[:response_headers][key] ||= value
+          end
         end
+        response
+      ensure
+        Thread.current[:ruby_proxy_headers_faraday_http] = nil
       end
     end
   end

data/lib/ruby_proxy_headers/http_gem.rb

@@ -1,48 +1,53 @@
 # frozen_string_literal: true
 
+require 'delegate'
+require 'openssl'
+require 'uri'
+require_relative '../ruby_proxy_headers'
+
+begin
+  require 'http'
+rescue LoadError => e
+  raise LoadError, "the http gem >= 6 is required for ruby_proxy_headers/http_gem (#{e.message})"
+end
+
 module RubyProxyHeaders
-  # HTTP.rb (http gem) integration for proxy headers support.
+  # HTTP.rb (the +http+ gem) sends +proxy_headers+ on CONNECT and returns them
+  # as +response.proxy_headers+. This wrapper adds +proxy_response_headers+,
+  # matching the other adapters.
   #
   # @example
   #   client = RubyProxyHeaders::HTTPGem.create_client(
-  #     proxy: 'http://user:pass@proxy:8080',
+  #     proxy: 'http://user:pass@proxy.example.com:31280',
   #     proxy_headers: { 'X-ProxyMesh-Country' => 'US' }
   #   )
-  #   
   #   response = client.get('https://example.com')
-  #   puts response.proxy_response_headers
-  #
+  #   puts response.proxy_response_headers['X-ProxyMesh-IP']
   module HTTPGem
-    # Create an HTTP client with proxy header support.
-    # @param proxy [String] Proxy URL
-    # @param proxy_headers [Hash] Custom headers to send to proxy
-    # @param options [Hash] Additional HTTP.rb options
-    # @return [ProxyClient]
     def self.create_client(proxy:, proxy_headers: {}, **options)
       ProxyClient.new(proxy: proxy, proxy_headers: proxy_headers, **options)
     end
 
-    # Make a GET request with proxy headers.
     def self.get(url, proxy:, proxy_headers: {}, **options)
       create_client(proxy: proxy, proxy_headers: proxy_headers).get(url, **options)
     end
 
-    # Make a POST request with proxy headers.
     def self.post(url, proxy:, proxy_headers: {}, body: nil, **options)
       create_client(proxy: proxy, proxy_headers: proxy_headers).post(url, body: body, **options)
     end
 
-    # Proxy client wrapper for HTTP.rb
+    # HTTP.rb client that sends proxy_headers on CONNECT.
     class ProxyClient
+      attr_reader :last_proxy_response_headers
+
+      REQUEST_OPTIONS = %i[open_timeout read_timeout headers body verify_ssl].freeze
+
       def initialize(proxy:, proxy_headers: {}, **options)
         @proxy = proxy
         @proxy_headers = proxy_headers
-        @options = options
-        @last_proxy_response_headers = nil
+        @default_options = options
       end
 
-      attr_reader :last_proxy_response_headers
-
       def get(url, **options)
         request(:get, url, **options)
       end
@@ -62,32 +67,48 @@ module RubyProxyHeaders
       private
 
       def request(method, url, **options)
-        uri = URI.parse(url)
-
-        # Use our core connection for HTTPS
-        if uri.scheme == 'https'
-          response = RubyProxyHeaders::NetHTTP.request(
-            method,
-            url,
-            proxy: @proxy,
-            proxy_headers: @proxy_headers,
-            headers: options[:headers],
-            body: options[:body]
-          )
-          @last_proxy_response_headers = response.proxy_response_headers
-          return response
+        options = @default_options.merge(options)
+        unknown = options.keys - REQUEST_OPTIONS
+        raise ArgumentError, "unknown option: #{unknown.first}" unless unknown.empty?
+
+        @proxy_headers.each { |key, value| RubyProxyHeaders.validate_header!(key, value) }
+        proxy_uri = URI.parse(@proxy)
+        user = proxy_uri.user && URI.decode_www_form_component(proxy_uri.user)
+        password = proxy_uri.password && URI.decode_www_form_component(proxy_uri.password)
+        client = ::HTTP.via(proxy_uri.host, proxy_uri.port, user, password, @proxy_headers)
+        timeouts = {}
+        timeouts[:connect] = options[:open_timeout] if options[:open_timeout]
+        timeouts[:read] = options[:read_timeout] if options[:read_timeout]
+        client = client.timeout(timeouts) unless timeouts.empty?
+
+        request_options = {}
+        request_options[:headers] = options[:headers] if options[:headers]
+        request_options[:body] = options[:body] unless options[:body].nil?
+        if options[:verify_ssl] == false
+          context = OpenSSL::SSL::SSLContext.new
+          context.verify_mode = OpenSSL::SSL::VERIFY_NONE
+          request_options[:ssl_context] = context
+        end
+
+        response = client.request(method, url, **request_options)
+        # HTTP::Headers is not a Hash. Copy into HeaderMap so callers can
+        # read X-ProxyMesh-IP the same way as the other adapters.
+        mapped = HeaderMap.new
+        response.proxy_headers.each do |name, value|
+          mapped[name] = mapped.key?(name) ? "#{mapped[name]}, #{value}" : value
         end
+        @last_proxy_response_headers = mapped
+        ProxyResponse.new(response, mapped)
+      end
+    end
 
-        # For HTTP, use standard http gem
-        require 'http'
+    # HTTP::Response plus +proxy_response_headers+.
+    class ProxyResponse < SimpleDelegator
+      attr_reader :proxy_response_headers
 
-        proxy_uri = URI.parse(@proxy)
-        ::HTTP.via(
-          proxy_uri.host,
-          proxy_uri.port,
-          proxy_uri.user,
-          proxy_uri.password
-        ).send(method, url, **options)
+      def initialize(response, proxy_headers)
+        super(response)
+        @proxy_response_headers = proxy_headers
       end
     end
   end