http_mimic 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 32ae2902090ddf216e6f33843f1fb53d82d13cda864a3a71fd7318f5f316cdf4
+  data.tar.gz: 907aec51da124505902e104ebf3bce70eb2883490ec8b360f3f0571e6efd98c2
+SHA512:
+  metadata.gz: e47bf2078c03dc6a37ecb3478ee5f7a42724375161c63175067fd6df5c8c76fae90b04867eb6f4b8fe03ab2ad2d919ef64dcfd1389ab20befdcb9e2ff245d492
+  data.tar.gz: 31c4ebd49e1180914bd949dc23bbc4bd7580dea84f898c7a97bc4977a8501120b1d3d5e6301b65ef2bf7cfa9e5c9bbecd8a7f47806f0a8cf42e1e1b4e7b2e45c

data/CHANGELOG.md

@@ -0,0 +1,60 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.3.0] - 2026-08-24
+
+### Added
+- **Live Fingerprint Test Suite**: Added integration tests (`rake test:live`) verifying real-world JA3, JA4, and Akamai HTTP/2 fingerprints against live endpoints (`tls.browserleaks.com`, `tls.peet.ws`).
+- **Standard MIT License**: Included official `LICENSE.txt` and repository metadata.
+- **Rakefile Integration**: Added standard Rake test tasks for unit tests, live tests, and full test suites.
+
+### Changed
+- **Default Impersonation Target**: Upgraded default impersonation from `chrome116` to modern `chrome131` with Post-Quantum (ML-KEM) and modern Client Hints support.
+- **Internationalization**: Fully translated all code comments, error messages, and documentation into English.
+
+## [0.2.0] - 2026-08-24
+
+### Added
+- **Webdrivers-like Auto Driver Management (`HttpMimic::Downloader`)**:
+  - Automatically downloads and unpacks official binaries from [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) releases to `~/.http_mimic/bin`.
+  - Automatic platform detection for macOS (ARM64 / x86_64), Linux (GNU / MUSL, x86_64, aarch64, arm, i386, riscv64, loongarch64), Windows (x86_64, arm64, i686), and FreeBSD.
+  - Manual driver management helpers: `HttpMimic.download_driver!`, `HttpMimic.driver_installed?`, `HttpMimic::Downloader.available_binaries`.
+  - Configurable auto-download, target release version, install directory, and custom GitHub repo.
+- **Expanded Browser Target Support**:
+  - Support for Chrome (`chrome116`, `chrome120`, `chrome123`, `chrome124`, `chrome131`, `chrome133a`, `chrome136`, `chrome142`, `chrome99-110`).
+  - Support for Firefox (`firefox133`, `firefox135`, `firefox144`, `firefox117`, `firefox109`, `firefox102`, `firefox98`).
+  - Support for Safari (`safari180`, `safari170`, `safari155`, `safari153`).
+  - Support for Edge (`edge101`, `edge99`) and Tor (`tor145`).
+
+## [0.1.0] - 2026-08-24
+
+### Added
+- **Initial Release of `HttpMimic`**: A HTTParty-like Ruby client wrapping `curl-impersonate` via `Open3.capture3`.
+- **Core HTTP Verbs**: Support for `get`, `post`, `put`, `patch`, `delete`, `head`, and `options`.
+- **HTTParty-Style API**:
+  - Direct module methods: `HttpMimic.get`, `HttpMimic.post`, etc.
+  - Class-level DSL mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`).
+  - Reusable instance client: `HttpMimic::Client.new(...)`.
+- **Smart Response & Parser**:
+  - Auto-parsing JSON responses and method delegation (`response['key']`, `response.parsed_response`).
+  - Case-insensitive header access (`response.headers['content-type']`).
+  - Automatic `Set-Cookie` header parsing and cookie jar representation (`response.cookies`).
+  - Complete 3xx redirect history tracking (`response.history`).
+  - Rich status helpers (`response.success?`, `response.redirect?`, `response.client_error?`, `response.server_error?`, `response.ok?`).
+- **Comprehensive Request Options**:
+  - Query parameters (auto URL encoding and nested parameters).
+  - Headers, JSON payloads (`json:`), Form URL-encoded data (`body:`), and Multipart (`form:` / `multipart:`).
+  - Basic and Digest authentication (`basic_auth:`, `digest_auth:`, `bearer_token:`).
+  - Timeouts (`timeout:`, `connect_timeout:`).
+  - Proxy and Proxy authentication (`proxy:`, `proxy_auth:`).
+  - SSL/TLS settings (`insecure:`, `ssl_ca_file:`, `ssl_cert:`, `ssl_key:`).
+  - Raw curl arguments passthrough (`curl_options:`).
+- **Safe CLI Execution**:
+  - Uses array arguments in `Open3.capture3` to eliminate shell injection vulnerabilities.
+  - Stdin streaming (`stdin_data`) with `-d @-` to bypass OS command line length limits for large payloads.
+- **Unit Test Suite**:
+  - Comprehensive tests for header handling, cookie parsing, command generation, response parsing, redirect chains, and platform slug detection.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 anxgang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,263 @@
+# HttpMimic
+
+`http_mimic` is a Ruby HTTP Client gem built on top of `Open3.capture3` and `curl-impersonate`.
+It provides an elegant, concise, and intuitive **HTTParty-style** API while leveraging [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) to simulate authentic Chrome, Firefox, Safari, Edge, and Tor TLS / HTTP2 fingerprints (JA3, JA4, Akamai) and handshakes.
+
+It also includes **Webdrivers-like automatic driver management**, automatically downloading and managing `curl-impersonate` binaries across macOS, Linux, and Windows without manual setup.
+
+---
+
+## 🌟 Features
+
+- **Webdrivers-Style Driver Management**:
+  - Automatically downloads and unpacks official binaries from [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) to `~/.http_mimic/bin`.
+  - **Automatic platform & architecture detection** (macOS ARM/Intel, Linux x86_64/aarch64/musl, Windows x86_64/arm64, FreeBSD).
+  - Zero configuration required—installs on first request automatically.
+  - Manual driver management helpers: `HttpMimic.download_driver!`, `HttpMimic.driver_installed?`, `HttpMimic::Downloader.available_binaries`.
+- **Authentic Browser Handshakes & Fingerprints**:
+  - Chrome support: `chrome131` (default), `chrome124`, `chrome120`, `chrome133a`, `chrome136`, `chrome142`, `chrome99-110`.
+  - Firefox support: `firefox135`, `firefox133`, `firefox144`, `firefox117`, `firefox109`, `firefox102`, `firefox98`.
+  - Safari support: `safari180`, `safari170`, `safari155`, `safari153`.
+  - Edge & Tor support: `edge101`, `edge99`, `tor145`.
+- **HTTParty-Style API**:
+  - Direct module methods: `HttpMimic.get`, `HttpMimic.post`, etc.
+  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`).
+  - Reusable instance client: `HttpMimic::Client.new(...)`.
+- **Zero Shell Injection Risk**:
+  - Executes commands with array arguments via `Open3.capture3(*cmd_array)`.
+  - Uses stdin streaming (`-d @-`) to safely handle large payloads without hitting OS command-line limits.
+- **Smart Response Parsing**:
+  - HTTP status helpers: `response.code`, `response.success?`, `response.redirect?`, `response.client_error?`, `response.server_error?`.
+  - Case-insensitive header access: `response.headers['Content-Type']`.
+  - Automatic `Set-Cookie` header parsing: `response.cookies['session_id']`.
+  - Auto-parsed JSON with object delegation: `response['key']`, `response.parsed_response`.
+  - Complete 3xx redirect history tracking: `response.history`.
+- **Comprehensive Request Options**:
+  - Supports `query`, `headers`, `json`, `body` (form data), `cookies`, `timeout`, `connect_timeout`, `proxy`, `basic_auth`, `digest_auth`, `bearer_token`, `insecure`, custom `curl_options`, and more.
+- **Graceful Fallback**:
+  - If a specific binary is unavailable and auto-download is disabled, automatically falls back to system standard `curl`.
+
+---
+
+## 📦 Installation
+
+Add this line to your application's `Gemfile`:
+
+```ruby
+gem 'http_mimic'
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+---
+
+## 🤖 Driver Management
+
+`HttpMimic` automatically downloads the corresponding platform binary of `curl-impersonate` on the first request and saves it to `~/.http_mimic/bin`.
+
+You can also manage drivers manually:
+
+```ruby
+require 'http_mimic'
+
+# Check if driver is installed locally
+HttpMimic.driver_installed? # => true / false
+
+# Manually trigger download (defaults to latest stable release v2.1.1)
+HttpMimic.download_driver!
+
+# Specify a version or force re-download
+HttpMimic.download_driver!(version: 'v2.1.1', force: true)
+
+# List all available browser binary names installed locally
+HttpMimic::Downloader.available_binaries
+# => ["curl_chrome131", "curl_chrome120", "curl_firefox135", "curl_safari180", ...]
+```
+
+---
+
+## 🚀 Quick Start
+
+### 1. Direct Module Calls
+
+```ruby
+require 'http_mimic'
+
+# Send a GET request (simulates Chrome 131 fingerprint by default)
+response = HttpMimic.get(
+  'https://tls.browserleaks.com/json',
+  impersonate: 'chrome131'
+)
+
+puts response.code                    # => 200
+puts response.success?                 # => true
+puts response['ja3_hash']             # => Authentic Chrome 131 JA3 fingerprint
+puts response.headers['content-type'] # => "application/json"
+
+# Send a POST JSON request (simulating Safari 18.0)
+response = HttpMimic.post(
+  'https://httpbin.org/post',
+  json: { name: 'Alice', role: 'admin' },
+  impersonate: 'safari180'
+)
+
+puts response.code            # => 200
+puts response['json']['name'] # => "Alice"
+```
+
+---
+
+### 2. Class Mixin Mode (HTTParty Style)
+
+```ruby
+class BrowserLeaksClient
+  include HttpMimic
+
+  base_uri 'https://tls.browserleaks.com'
+  impersonate 'chrome120' # Default to Chrome 120
+  default_timeout 30
+  headers 'Accept-Language' => 'en-US,en;q=0.9'
+
+  def test_fingerprint
+    get('/json')
+  end
+end
+
+client = BrowserLeaksClient.new
+res = client.test_fingerprint
+
+puts "HTTP Status: #{res.code}"
+puts "JA3 Hash: #{res['ja3_hash']}"
+puts "User-Agent: #{res['user_agent']}"
+```
+
+---
+
+### 3. Instance Mode
+
+```ruby
+client = HttpMimic::Client.new(
+  base_uri: 'https://api.example.com',
+  impersonate: 'firefox135',
+  timeout: 15,
+  headers: {
+    'X-API-KEY' => 'my_api_key'
+  }
+)
+
+# Execute GET
+response = client.get('/v1/users', query: { limit: 10 })
+
+# Execute POST
+response = client.post('/v1/users', json: { username: 'bob' })
+```
+
+---
+
+## ⚙️ Global Configuration
+
+Configure global defaults in an initializer (e.g., `config/initializers/http_mimic.rb`):
+
+```ruby
+HttpMimic.configure do |config|
+  # Browser simulation & request defaults
+  config.default_impersonate     = 'chrome131' # Default browser target
+  config.default_timeout         = 30          # Request timeout (seconds)
+  config.default_connect_timeout = 10          # Connection timeout (seconds)
+  config.follow_redirects        = true        # Automatically follow 3xx redirects
+  config.max_redirects           = 10          # Maximum redirect limit
+  config.fallback_to_curl        = true        # Fall back to system curl if binary is missing
+  config.raise_on_error          = false       # Raise exceptions on HTTP errors / non-zero exits
+  config.debug                   = false       # Print debug logs
+
+  # Webdrivers-like auto-download settings (enabled by default)
+  config.auto_download           = true                                  # Auto-download missing binary
+  config.driver_version          = 'v2.1.1'                              # Target release version
+  config.install_dir             = File.expand_path('~/.http_mimic/bin') # Directory for binaries
+  config.github_repo             = 'lexiforest/curl-impersonate'         # GitHub source repository
+end
+```
+
+---
+
+## 🛠️ Supported Request Options
+
+| Option | Type | Description |
+| :--- | :--- | :--- |
+| `:impersonate` | String | Target browser to mimic (e.g., `'chrome131'`, `'chrome120'`, `'firefox135'`, `'safari180'`, `'tor145'`) |
+| `:binary` | String | Path to a custom `curl-impersonate` executable |
+| `:query` / `:params` | Hash | URL query parameters (supports nested parameters and encoding) |
+| `:headers` | Hash | Custom HTTP request headers |
+| `:json` | Hash / Array | Serialized to JSON with `Content-Type: application/json` |
+| `:body` | Hash / String | Form payload (Hash) or raw request body string |
+| `:cookies` | Hash / String | Request cookies |
+| `:cookie_jar` | String | Path to save cookies (`-c`) |
+| `:cookie_file` | String | Path to read cookies (`-b`) |
+| `:timeout` | Integer / Float | Maximum execution timeout in seconds (`--max-time`) |
+| `:connect_timeout` | Integer / Float | Connection timeout in seconds (`--connect-timeout`) |
+| `:proxy` | String | Proxy address (e.g., `'http://127.0.0.1:8888'`) |
+| `:basic_auth` | Hash | `{ username: 'admin', password: 'secret' }` |
+| `:bearer_token` | String | Appends `Authorization: Bearer <token>` header |
+| `:insecure` | Boolean | Disable SSL certificate verification (`-k`) |
+| `:curl_options` | Array / String | Additional raw curl arguments (e.g., `['--http2', '--compressed']`) |
+
+---
+
+## 📄 Response Object
+
+The `Response` object wraps the HTTP response with convenient methods:
+
+```ruby
+response = HttpMimic.get('https://httpbin.org/get')
+
+# Status information
+response.code           # => 200 (Integer)
+response.status         # => 200
+response.status_message # => "OK"
+response.http_version   # => "2"
+response.success?       # => true (2xx)
+response.redirect?      # => false (3xx)
+response.client_error?  # => false (4xx)
+response.server_error?  # => false (5xx)
+
+# Response body
+response.body           # => Raw Body (String)
+response.parsed_response# => Auto-parsed JSON Hash / Array
+response['key']         # => Direct key access to parsed_response
+
+# Headers & Cookies
+response.headers['content-type'] # => Case-insensitive header access
+response.cookies['session_id']   # => Parsed Set-Cookie store
+response.history                 # => Array of redirect history metadata
+
+# Underlying execution details
+response.exit_code      # => Process exit status (0 for success)
+response.stderr         # => Stderr output from curl
+response.command        # => Array of the exact CLI arguments executed
+```
+
+---
+
+## 🧪 Testing
+
+```bash
+# Run offline unit tests
+bundle exec rake test
+
+# Run live integration tests against external TLS / JA3 / JA4 / Akamai endpoints
+bundle exec rake test:live
+
+# Run all tests
+bundle exec rake test:all
+```
+
+---
+
+## 📄 License
+
+This project is available as open source under the terms of the [MIT License](file:///Users/ivan/work/Tranyi/_gem/http_mimic/LICENSE.txt).
+Source code is hosted on [GitHub](https://github.com/anxgang/http_mimic).

data/http_mimic.gemspec

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'http_mimic/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "http_mimic"
+  spec.version       = HttpMimic::VERSION
+  spec.authors       = ["anxgang"]
+  spec.summary       = "A HTTParty-like Ruby client wrapping curl-impersonate via Open3."
+  spec.description   = "Simplifies making HTTP requests using curl-impersonate to mimic real browser TLS/HTTP2 fingerprints."
+  spec.homepage      = "https://github.com/anxgang/http_mimic"
+  spec.license       = "MIT"
+
+  spec.metadata = {
+    "homepage_uri"    => "https://github.com/anxgang/http_mimic",
+    "source_code_uri" => "https://github.com/anxgang/http_mimic",
+    "bug_tracker_uri" => "https://github.com/anxgang/http_mimic/issues",
+    "changelog_uri"   => "https://github.com/anxgang/http_mimic/blob/main/CHANGELOG.md"
+  }
+
+  spec.required_ruby_version = ">= 2.3.0"
+
+  spec.files         = Dir["lib/**/*.rb", "README.md", "CHANGELOG.md", "LICENSE.txt", "http_mimic.gemspec"]
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", ">= 1.17"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "minitest"
+end

data/lib/http_mimic/client.rb

@@ -0,0 +1,72 @@
+# frozen_string_literal: true
+
+module HttpMimic
+  class Client
+    attr_accessor :default_options, :config
+
+    def initialize(options = {}, config = nil)
+      @default_options = options.dup
+      @config = config || HttpMimic.configuration.dup
+    end
+
+    def get(url, options = {})
+      request(:get, url, options)
+    end
+
+    def post(url, options = {})
+      request(:post, url, options)
+    end
+
+    def put(url, options = {})
+      request(:put, url, options)
+    end
+
+    def patch(url, options = {})
+      request(:patch, url, options)
+    end
+
+    def delete(url, options = {})
+      request(:delete, url, options)
+    end
+
+    def head(url, options = {})
+      request(:head, url, options)
+    end
+
+    def options(url, options = {})
+      request(:options, url, options)
+    end
+
+    def request(method, url, options = {})
+      merged_options = merge_options(@default_options, options)
+      Request.new(method, url, merged_options, @config).perform
+    end
+
+    private
+
+    def merge_options(base, override)
+      result = base.merge(override)
+
+      # Deep merge headers
+      if base[:headers] || override[:headers]
+        base_headers = base[:headers] || {}
+        over_headers = override[:headers] || {}
+        result[:headers] = base_headers.merge(over_headers)
+      end
+
+      # Deep merge query / params
+      base_query = base[:query] || base[:params] || {}
+      over_query = override[:query] || override[:params] || {}
+      if !base_query.empty? || !over_query.empty?
+        result[:query] = base_query.merge(over_query)
+      end
+
+      # Deep merge cookies
+      if base[:cookies].is_a?(Hash) && override[:cookies].is_a?(Hash)
+        result[:cookies] = base[:cookies].merge(override[:cookies])
+      end
+
+      result
+    end
+  end
+end