http_mimic 0.4.0 → 0.5.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9b0a9fff1b74c98ea24c294a491b00ba57e270d24e6c0bb7a669e6d2db264d8a
-  data.tar.gz: e11d3ae615b92abcfedda8547120290d82020acd82230353a127cc97dfc844f6
+  metadata.gz: 1b0b4192818428818cca7cde26b4ce90d6fa847db6e80e2b63c163dd8171bdd9
+  data.tar.gz: de6a71df66c710a30e7cb8fcbe6976653ec6242a6d29fd6959016513da4513be
 SHA512:
-  metadata.gz: 5e0a93bb67bb84582e24df07ea4d51d291a53f0b48ab77ad0bd4a63b666bede4b493bbb405ad34d76602e086eafd76a455668c2a491488d83dd2c0be1060fe66
-  data.tar.gz: c616cafefeb23d78af7320beab5ddc3b8f79aafda440fd1fac889f5a5990417b31716105e8c74cab566e6c0ed6eca04a82a397ddc4242eb59caed519e52739de
+  metadata.gz: 3b4ba6f1355da36e9f1064e62b1b988200f483a58b49524e3dd5350b0c99660d0b93ab650fc9ea8b3b7aa2204f41b7e2ca51c6b439128c267d7660be77b2b644
+  data.tar.gz: dd25b3fbf25dca0d935b910cb2b0981ab77d9dbfe45cfa5b522a99bbca7081016e6323af3fc59c389ef925c6df0b4d0bbf7321c7a93df5154d0d41e328962d76

data/CHANGELOG.md

@@ -5,6 +5,70 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.3] - 2026-09-05
+
+### Added
+- **Automatic Free Proxy Pool (`auto_proxy`)**:
+  - Built-in `HttpMimic::ProxyPool` automatically gathers and caches hundreds of public HTTP proxies from maintained sources (`monosans`, `proxyscrape`, `TheSpeedX`).
+  - Disabled by default (`auto_proxy: false`) with per-request and global configuration opt-in.
+  - Transparent proxy failure retry: automatically detects dead proxies (exit codes 5, 7, 28, 35, 56 or connection errors), marks them dead in the pool, and transparently retries with fresh proxies up to `proxy_retries` (default: 3).
+  - Explicit manual `:proxy` option always takes precedence over `auto_proxy`.
+  - Seamless integration with Obscura and WAF solver handshakes.
+  - Added DSL helpers (`HttpMimic.auto_proxy`, `HttpMimic.proxy_pool`, `HttpMimic.refresh_proxies!`).
+- **Streamlined Documentation**:
+  - Refined README with concise, high-level highlights focusing on core anti-detect capabilities and developer experience.
+
+## [0.5.2] - 2026-09-05
+
+### Added
+- **Anti-Poisoning Failure Protection for CookieStore (`persist_on_failure`, `clear_on_failure`)**:
+  - `persist_on_failure` (default: `false`): Ensures failed/blocked requests (403, 401, retry statuses, or WAF challenge pages) do not save invalid or bot-flagged cookies to disk.
+  - `clear_on_failure` (default: `true`): Automatically purges stored host cookies when a request fails verification or is blocked by WAF, preventing poisoned sessions from breaking subsequent requests.
+  - Added class-level DSL and configuration support (`HttpMimic.persist_on_failure`, `HttpMimic.clear_on_failure`).
+- **Two-Phase Telemetry Handshake in `AkamaiSolver`**:
+  - Automatic multi-round telemetry loop sending second-stage interaction sensor posts after initial cookie acquisition to flip `_abck` tokens from `~-1~` to verified `~0~`.
+- **Intelligent Profile Fallback & WAF Protection**:
+  - `:auto` mode now rotates between distinct browser engines (`:impersonate` [Chrome], `:android`, `:ios`, `:firefox`) and skips plain `:curl` fallback on WAF targets to avoid IP flagging.
+  - Added `:firefox` and `:safari` profile support in `CommandBuilder`.
+  - Added Best Response Preservation: prevents successful/200 progress from being wiped out by later failed fallback attempts.
+
+## [0.5.1] - 2026-08-30
+
+### Added
+- **Two-Phase Cookie & State Pipeline (Tier 1 ➔ Tier 2)**:
+  - Automatically forward validated cookies (`NID`, `AEC`, `_abck`, `cf_clearance`) and custom headers from Tier 1 HTTP / WAF resolution directly into Obscura (`--cookie` flag) during `auto_render_spa` transitions.
+- **Obscura Stealth BoringSSL Integration**:
+  - Downloader now defaults to `*-stealth.tar.gz` release assets, equipping Obscura with native BoringSSL Chrome TLS / JA4 fingerprint impersonation.
+- **Upgraded Obscura to `anxgang/obscura` v0.2.3**:
+  - WebGL context, `REAL_FONT_METRICS` font measurements, and persistent Web Worker `importScripts` support.
+
+## [0.5.0] - 2026-08-30
+
+### Added
+- **Tier 2: Obscura Headless SPA Rendering Support**:
+  - Native integration with [`h4ckf0r0day/obscura`](https://github.com/h4ckf0r0day/obscura)—a lightweight (<100MB) Rust + V8 headless browser engine with built-in stealth anti-detection.
+  - On-demand automatic driver management (`HttpMimic.download_obscura!`, `HttpMimic.obscura_installed?`, `HttpMimic.obscura_path`) across macOS (ARM64/x86_64), Linux (x86_64/aarch64), and Windows.
+  - New rendering helpers: `HttpMimic.render(url, options = {})` and `HttpMimic.spa(url, options = {})`.
+  - **Automatic SPA Detection & Transition (`auto_render_spa`)**:
+    - Intelligent `HttpMimic::SpaDetector` detects unhydrated client-side SPA shells (React `#root`, Vue `#app`, Angular `<app-root>`, Google Dynamic SERP, `<noscript>` prompts).
+    - When `auto_render_spa: true` (or `config.auto_render_spa = true`) is enabled, `HttpMimic.get` automatically transitions from Tier 1 (HTTP) to Tier 2 (Obscura) to render the full dynamic DOM.
+  - Configurable options: `:wait_until` (`networkidle0`, `domcontentloaded`, `load`), `:timeout`, `:proxy`, `:eval`, `:dump`, `:selector`, `:user_agent`, and `:stealth`.
+
+## [0.4.1] - 2026-08-30
+
+### Added
+- **Google Search Guard & BotGuard VM Detection**:
+  - Added `HttpMimic::Waf::GoogleSolver` and `HttpMimic::Waf::Detector.google?` for detecting Google Search Guard (`knitsail`, `enablejs`, `SG_SS`) challenges.
+  - Added automatic Google Search referral context (`Referer: https://www.google.com/`, `sec-fetch-site: same-origin`) in `CommandBuilder#apply_navigation_headers`.
+- **Authentic Hardware & WebGL Pipeline Emulation in QuickJS Context**:
+  - Full WebGL shader compilation and rasterization pipeline (`createShader`, `compileShader`, `getShaderPrecisionFormat`, `createProgram`, `linkProgram`, `readPixels` simulated gradient buffer).
+  - Dynamic font measurement (`REAL_FONT_METRICS`) with exact macOS Chrome `offsetWidth` / `offsetHeight` dimensions across 17 font families and 7 font sizes.
+  - Authentic Chromium `OfflineAudioContext` audio buffer rendering curve.
+  - Physics-based Cubic Bézier Spline mouse movement simulation with velocity decay, natural micro-jitter, and complete click event chains.
+  - Polyfills for `trustedTypes`, `sessionStorage`, `localStorage`, and `document.currentScript`.
+- **Automatic Response Decompression**:
+  - Added `--compressed` flag to CommandBuilder to ensure automatic transparent decompression of `gzip`, `deflate`, `br` (Brotli), and `zstd` payloads.
+
 ## [0.4.0] - 2026-08-30
 
 ### Added

data/README.md

@@ -1,41 +1,27 @@
-# HttpMimic
+# HttpMimic 🎭
 
-`http_mimic` is a Ruby HTTP Client gem built on top of `Open3.capture3` and `curl-impersonate`.
-It provides an elegant, concise, and intuitive **HTTParty-style** API while leveraging [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) to simulate authentic Chrome, Firefox, Safari, Edge, and Tor TLS / HTTP2 fingerprints (JA3, JA4, Akamai) and handshakes.
+> **Next-Generation Anti-Detect HTTP Client for Ruby**  
+> An elegant, high-performance HTTP client combining authentic browser TLS/JA4 fingerprint impersonation, built-in anti-bot challenge solving, lightweight headless SPA rendering, and an automatic free proxy pool into an intuitive HTTParty-style API.
 
-It also includes **Webdrivers-like automatic driver management**, automatically downloading and managing `curl-impersonate` binaries across macOS, Linux, and Windows without manual setup.
+`http_mimic` provides a complete, modern toolchain for accessing protected endpoints and web scraping:
+- **🎭 Authentic Browser Fingerprints**: Simulates authentic Chrome, Safari (macOS & iOS), Android, and Firefox TLS 1.3 / HTTP/2 handshakes and JA3/JA4 signatures.
+- **🛡️ Built-in Anti-Bot Defense**: Automatically detects and solves advanced WAF challenges at microsecond speeds without the overhead of heavy browsers.
+- **⚡ Dual-Tier Engine with Headless SPA Support**: Fast HTTP-first pipeline with seamless escalation to an embedded, lightweight (<100MB) Rust+V8 headless engine (`auto_render_spa`) when dynamic DOM rendering is needed.
+- **🌐 Automatic Free Proxy Pool (`auto_proxy`)**: Built-in, zero-dependency proxy pool with health tracking, auto-rotation, and transparent failure retry.
+- **🍪 Resilient Session Management**: Host-based persistent cookie caching with automatic anti-poisoning protection to prevent tainted sessions.
+- **📦 Zero-Setup Driver Management**: Automatically provisions and manages required native binaries across macOS, Linux, and Windows on demand.
+- **🚀 Intuitive Ruby DSL**: Clean, idiomatic syntax supporting class mixins (`include HttpMimic`), direct module calls (`HttpMimic.get`), and reusable client instances.
 
 ---
 
-## 🌟 Features
-
-- **Webdrivers-Style Driver Management**:
-  - Automatically downloads and unpacks official binaries from [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) to `~/.http_mimic/bin`.
-  - **Automatic platform & architecture detection** (macOS ARM/Intel, Linux x86_64/aarch64/musl, Windows x86_64/arm64, FreeBSD).
-  - Zero configuration required—installs on first request automatically.
-  - Manual driver management helpers: `HttpMimic.download_driver!`, `HttpMimic.driver_installed?`, `HttpMimic::Downloader.available_binaries`.
-- **Authentic Browser Handshakes & Fingerprints**:
-  - Chrome support: `chrome131` (default), `chrome124`, `chrome120`, `chrome133a`, `chrome136`, `chrome142`, `chrome99-110`.
-  - Firefox support: `firefox135`, `firefox133`, `firefox144`, `firefox117`, `firefox109`, `firefox102`, `firefox98`.
-  - Safari support: `safari180`, `safari170`, `safari155`, `safari153`.
-  - Edge & Tor support: `edge101`, `edge99`, `tor145`.
-- **HTTParty-Style API**:
-  - Direct module methods: `HttpMimic.get`, `HttpMimic.post`, etc.
-  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`, `persist_cookies`).
-  - Reusable instance client: `HttpMimic::Client.new(...)`.
-- **Zero Shell Injection Risk**:
-  - Executes commands with array arguments via `Open3.capture3(*cmd_array)`.
-  - Uses stdin streaming (`-d @-`) to safely handle large payloads without hitting OS command-line limits.
-- **Smart Response Parsing**:
-  - HTTP status helpers: `response.code`, `response.success?`, `response.redirect?`, `response.client_error?`, `response.server_error?`.
-  - Case-insensitive header access: `response.headers['Content-Type']`.
-  - Automatic `Set-Cookie` header parsing: `response.cookies['session_id']`.
-  - Auto-parsed JSON with object delegation: `response['key']`, `response.parsed_response`.
-  - Complete 3xx redirect history tracking: `response.history`.
-- **Comprehensive Request Options**:
-  - Supports `query`, `headers`, `json`, `body` (form data), `cookies`, `timeout`, `connect_timeout`, `proxy`, `basic_auth`, `digest_auth`, `bearer_token`, `insecure`, custom `curl_options`, and more.
-- **Graceful Fallback**:
-  - If a specific binary is unavailable and auto-download is disabled, automatically falls back to system standard `curl`.
+## 🌟 Highlights
+
+- **Authentic TLS & HTTP/2 Impersonation**: Native support for `chrome131` (default), `chrome120-142`, `safari180` (macOS/iOS), `firefox135`, `edge101`, and `tor145`.
+- **Intelligent Challenge Resolution**: Automatic WAF challenge mitigation and smart profile fallback to keep requests succeeding.
+- **Client-Side SPA Hydration**: High-speed HTTP execution by default, with optional V8 DOM hydration for complex JavaScript apps (`auto_render_spa: true`).
+- **Free Proxy Pool & Auto-Rotation**: Opt-in proxy management (`auto_proxy: true`) with automatic dead-proxy detection and retry.
+- **Safe & Reliable**: Zero shell injection risk via `Open3.capture3(*cmd_array)`, safe stdin streaming for large payloads, and smart response parsing.
+- **Flexible Integration**: Works seamlessly as a one-off client, a reusable instance, or a class-level DSL.
 
 ---
 
@@ -198,6 +184,8 @@ response = HttpMimic.get('https://example.com/items', persist_cookies: true)
 class Scraper
   include HttpMimic
   persist_cookies true
+  # persist_on_failure false # (default: do not save cookies on 403 / verification failure)
+  # clear_on_failure true    # (default: auto-clear cached cookies if verification fails)
 end
 
 # 3. Manual CookieStore helpers
@@ -206,6 +194,9 @@ HttpMimic.save_cookies('example.com', { session: '123' }) # Save cookies with de
 HttpMimic.clear_cookies!('example.com')                # Clear host or all cookies
 ```
 
+> **🛡️ Anti-Poisoning Failure Protection:**
+> By default, `persist_on_failure` is set to `false`, meaning cookies returned with HTTP error codes (e.g. 403, 401) or unverified WAF challenge pages are **never** persisted to disk. Furthermore, `clear_on_failure: true` automatically wipes tainted host cookies upon verification failure, preventing blocked sessions from poisoning subsequent requests.
+
 ---
 
 ## 🧭 Target-Specific Navigation Headers & Client Hints
@@ -247,6 +238,8 @@ HttpMimic.configure do |config|
   # Host CookieStore Persistence (disabled by default)
   config.persist_cookies         = false                                     # Auto-persist cookies per host
   config.cookie_store_dir        = File.expand_path('~/.http_mimic/cookies') # Directory for cookie store
+  config.persist_on_failure      = false                                     # Do not save cookies if request/verification fails
+  config.clear_on_failure        = true                                      # Purge cached host cookies on 403 / verification failure
 
   # WAF challenge solving (enabled by default)
   config.auto_solve_waf          = true        # Automatically solve detected WAF JS challenges
@@ -278,15 +271,106 @@ end
 | `:body` | Hash / String | Form payload (Hash) or raw request body string |
 | `:cookies` | Hash / String | Request cookies |
 | `:persist_cookies` | Boolean | Automatically load and save cookies for the host across requests |
+| `:persist_on_failure` | Boolean | Save cookies even if request or verification fails (default `false`) |
+| `:clear_on_failure` | Boolean | Automatically clear stored cookies on verification failure / 403 / WAF block (default `true`) |
 | `:cookie_jar` | String | Path to save cookies (`-c`) |
 | `:cookie_file` | String | Path to read cookies (`-b`) |
 | `:timeout` | Integer / Float | Maximum execution timeout in seconds (`--max-time`) |
 | `:connect_timeout` | Integer / Float | Connection timeout in seconds (`--connect-timeout`) |
 | `:proxy` | String | Proxy address (e.g., `'http://127.0.0.1:8888'`) |
+| `:auto_proxy` | Boolean | Automatically fetch and rotate through a pool of free HTTP proxies (default: `false`) |
 | `:basic_auth` | Hash | `{ username: 'admin', password: 'secret' }` |
 | `:bearer_token` | String | Appends `Authorization: Bearer <token>` header |
 | `:insecure` | Boolean | Disable SSL certificate verification (`-k`) |
 | `:curl_options` | Array / String | Additional raw curl arguments (e.g., `['--http2', '--compressed']`) |
+| `:auto_render_spa` | Boolean | Automatically detect unhydrated SPA shells and render via Obscura |
+| `:render` | Symbol | `:spa` or `:obscura` to explicitly render page via Obscura |
+
+---
+
+## ⚡ Obscura SPA Rendering (Tier 2 Engine)
+
+`http_mimic` provides native support for [`h4ckf0r0day/obscura`](https://github.com/h4ckf0r0day/obscura)—a lightweight, single-binary (<100MB) headless browser engine written in Rust with an embedded V8 JavaScript runtime and built-in stealth anti-detection capabilities.
+
+Obscura is auto-downloaded on-demand into `~/.http_mimic/bin/obscura` when SPA rendering is requested.
+
+### 1. Automatic SPA Detection & Rendering (`auto_render_spa`)
+
+Similar to `auto_solve_waf`, when `auto_render_spa: true` is enabled, `HttpMimic` first sends a microsecond-fast HTTP request (Tier 1). If the response is detected as an unhydrated SPA shell (empty React `#root`, Vue `#app`, Angular `<app-root>`, Google Dynamic SERP, or `<noscript>` prompt), it automatically transitions to Obscura (Tier 2) to render the full DOM tree:
+
+```ruby
+# Auto-detects SPA shell and seamlessly renders with Obscura
+response = HttpMimic.get('https://example.com/spa', auto_render_spa: true)
+
+puts response.code    # => 200
+puts response.title   # => Fully hydrated DOM title
+puts response.body    # => Fully rendered HTML with client-side injected DOM nodes
+```
+
+### 2. Direct SPA Rendering
+
+```ruby
+# Direct SPA render
+response = HttpMimic.render('https://example.com/spa')
+
+# or via alias:
+response = HttpMimic.spa('https://example.com/spa')
+```
+
+**Supported Options**:
+- `:wait_until` - `'load'`, `'domcontentloaded'`, `'networkidle0'` (default: `'networkidle0'`)
+- `:timeout` - execution deadline in seconds (default: 30)
+- `:proxy` - proxy address (`socks5://...` or `http://...`)
+- `:eval` - JavaScript expression to evaluate in page context
+- `:dump` - output format: `'html'` (default), `'text'`, `'links'`, `'markdown'`, `'cookies'`
+- `:stealth` - whether stealth anti-detection is enabled (default: `true`)
+
+---
+
+## 🌐 Automatic Free Proxy Pool (`auto_proxy`)
+
+`http_mimic` provides a built-in, zero-dependency Free Proxy Pool manager. When enabled, `HttpMimic` automatically fetches and pools hundreds of free HTTP proxies from maintained public sources (monosans, ProxyScrape, TheSpeedX), automatically marks dead proxies, and transparently retries requests on network or proxy failure.
+
+By default, `auto_proxy` is **`false`** so users can choose when to opt-in.
+
+### 1. Per-Request Opt-in
+
+```ruby
+# Automatically obtain a proxy from the pool and execute request
+response = HttpMimic.get('https://httpbin.org/ip', auto_proxy: true)
+
+# Explicit proxy always takes precedence over auto_proxy:
+response = HttpMimic.get('https://httpbin.org/ip', auto_proxy: true, proxy: 'http://my-dedicated-proxy:8080')
+```
+
+### 2. Global Configuration
+
+```ruby
+HttpMimic.configure do |config|
+  config.auto_proxy = true          # Enable free proxy pool by default
+  config.proxy_retries = 3          # Maximum proxy retry attempts on connection failure
+  config.proxy_pool_ttl = 1800      # Cache duration for fetched proxy list (seconds)
+end
+
+# Or via class-level DSL:
+HttpMimic.auto_proxy = true
+```
+
+### 3. Proxy Pool Management
+
+```ruby
+# Check available pool size
+HttpMimic.proxy_pool.size
+
+# Sample a random proxy from the pool
+HttpMimic.proxy_pool.sample # => "http://185.200.188.234:10001"
+
+# Manually refresh the proxy pool
+HttpMimic.refresh_proxies!
+
+# Load custom proxies into the pool
+HttpMimic.proxy_pool.load(['http://1.2.3.4:8080', 'http://5.6.7.8:3128'])
+```
 
 ---
 

data/lib/http_mimic/command_builder.rb

@@ -94,9 +94,10 @@ module HttpMimic
       args = []
       stdin_data = nil
 
-      # Base flags: silent mode, include HTTP headers
+      # Base flags: silent mode, include HTTP headers, auto-decompress responses
       args << '-s'
       args << '-i'
+      args << '--compressed'
 
       # HTTP Method
       if @method == 'HEAD'
@@ -247,6 +248,10 @@ module HttpMimic
         (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
       when :mobile
         (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
+      when :firefox
+        (options[:firefox_impersonate] || 'firefox135').to_s.downcase
+      when :safari
+        (options[:safari_impersonate] || 'safari180').to_s.downcase
       else
         (options[:impersonate] || config.default_impersonate).to_s.downcase
       end
@@ -387,6 +392,17 @@ module HttpMimic
         headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
         headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
       end
+
+      # Smart Search referral context (e.g. searching from Google homepage)
+      begin
+        parsed_uri = URI.parse(url)
+        if parsed_uri && parsed_uri.host =~ /google\./i && parsed_uri.path =~ /\/search/i
+          headers_to_send['referer'] ||= "#{parsed_uri.scheme || 'https'}://#{parsed_uri.host}/"
+          headers_to_send['sec-fetch-site'] ||= 'same-origin'
+        end
+      rescue StandardError
+        # ignore
+      end
     end
 
     def find_in_download_dir(candidates)

data/lib/http_mimic/configuration.rb

@@ -30,16 +30,31 @@ module HttpMimic
     attr_accessor :qjs_version
     attr_accessor :qjs_github_repo
 
+    # Obscura headless SPA engine settings
+    attr_accessor :obscura_version
+    attr_accessor :obscura_github_repo
+    attr_accessor :obscura_path
+    attr_accessor :auto_render_spa
+
     # WAF & JS challenge solving settings
     attr_accessor :auto_solve_waf
 
     # Shared Host-based CookieStore settings
     attr_accessor :persist_cookies
     attr_accessor :cookie_store_dir
+    attr_accessor :persist_on_failure
+    attr_accessor :clear_on_failure
 
     # Modern Navigation Headers simulation
     attr_accessor :navigation_headers
 
+    # Automatic Free Proxy Pool settings
+    attr_accessor :auto_proxy
+    attr_accessor :proxy_sources
+    attr_accessor :proxy_retries
+    attr_accessor :proxy_pool_ttl
+    attr_accessor :proxy_timeout
+
     def initialize
       @default_impersonate         = 'chrome131'
       @default_mobile_impersonate  = 'safari180_ios'
@@ -68,15 +83,33 @@ module HttpMimic
       @qjs_version                 = 'v0.16.2'
       @qjs_github_repo             = 'quickjs-ng/quickjs'
 
+      # Obscura defaults (Lightweight Rust+V8 headless SPA engine)
+      # Temporarily points to 'anxgang/obscura' (v0.2.3) for Web Worker importScripts support.
+      # Will switch back to 'h4ckf0r0day/obscura' once upstream Pull Request is merged.
+      # @obscura_github_repo       = 'h4ckf0r0day/obscura'
+      @obscura_version             = 'v0.2.3'
+      @obscura_github_repo         = 'anxgang/obscura'
+      @obscura_path                = nil
+      @auto_render_spa             = false
+
       # WAF challenge solving defaults
       @auto_solve_waf              = true
 
       # Shared Host CookieStore defaults
       @persist_cookies             = false
       @cookie_store_dir            = File.expand_path('~/.http_mimic/cookies')
+      @persist_on_failure          = false
+      @clear_on_failure            = true
 
       # Navigation Headers defaults
       @navigation_headers          = true
+
+      # Automatic Free Proxy Pool defaults
+      @auto_proxy                  = false
+      @proxy_sources               = nil
+      @proxy_retries               = 3
+      @proxy_pool_ttl              = 1800
+      @proxy_timeout               = 5
     end
   end
 end

data/lib/http_mimic/downloader.rb

@@ -16,6 +16,9 @@ module HttpMimic
     DEFAULT_QJS_REPO    = 'quickjs-ng/quickjs'
     DEFAULT_QJS_VERSION = 'v0.16.2'
 
+    DEFAULT_OBSCURA_REPO    = 'h4ckf0r0day/obscura'
+    DEFAULT_OBSCURA_VERSION = 'v0.2.1'
+
     class DownloadError < HttpMimic::Error; end
     class UnsupportedPlatformError < HttpMimic::Error; end
 
@@ -165,6 +168,100 @@ module HttpMimic
         File.join(dir, '.qjs_version')
       end
 
+      def download_obscura!(version: nil, install_dir: nil, repo: nil, force: false)
+        target_version = normalize_version(version || HttpMimic.configuration.obscura_version || DEFAULT_OBSCURA_VERSION)
+        target_dir     = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        target_repo    = repo || HttpMimic.configuration.obscura_github_repo || DEFAULT_OBSCURA_REPO
+
+        FileUtils.mkdir_p(target_dir)
+
+        dest_binary = File.join(target_dir, binary_name_for_platform('obscura'))
+
+        if !force && obscura_installed?(version: target_version, install_dir: target_dir)
+          log_info("obscura #{target_version} is already installed in #{target_dir}")
+          return dest_binary
+        end
+
+        asset_name = obscura_platform_asset
+        download_url = "https://github.com/#{target_repo}/releases/download/#{target_version}/#{asset_name}"
+
+        log_info("Downloading Obscura (#{target_version}) [#{asset_name}]...")
+        archive_data = fetch_binary(download_url)
+
+        log_info("Extracting Obscura archive to #{target_dir}...")
+        extract_tar_gz(archive_data, target_dir)
+
+        # Ensure executable permissions on extracted binaries
+        %w[obscura obscura-worker].each do |name|
+          p = File.join(target_dir, binary_name_for_platform(name))
+          File.chmod(0755, p) if File.exist?(p)
+        end
+
+        File.write(obscura_version_file_path(target_dir), target_version)
+
+        log_info("Obscura #{target_version} installation complete! (#{dest_binary})")
+        dest_binary
+      end
+
+      def obscura_installed?(version: nil, install_dir: nil)
+        target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        bin = File.join(target_dir, binary_name_for_platform('obscura'))
+        return false unless File.file?(bin) && File.executable?(bin)
+
+        if version
+          target_version = normalize_version(version)
+          v_file = obscura_version_file_path(target_dir)
+          return false unless File.file?(v_file)
+          return File.read(v_file).strip == target_version
+        end
+
+        true
+      end
+
+      def obscura_path(install_dir: nil)
+        return HttpMimic.configuration.obscura_path if HttpMimic.configuration.obscura_path && File.executable?(HttpMimic.configuration.obscura_path)
+
+        target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        candidate = File.join(target_dir, binary_name_for_platform('obscura'))
+        return candidate if File.file?(candidate) && File.executable?(candidate)
+
+        # Fallback to system PATH
+        sys_bin = `which obscura 2>/dev/null`.strip
+        return sys_bin if !sys_bin.empty? && File.executable?(sys_bin)
+
+        nil
+      end
+
+      def obscura_platform_asset
+        os  = host_os
+        cpu = host_cpu
+
+        case os
+        when :macos
+          case cpu
+          when :arm64 then 'obscura-aarch64-macos-stealth.tar.gz'
+          when :x86_64 then 'obscura-x86_64-macos-stealth.tar.gz'
+          else
+            raise UnsupportedPlatformError, "Unsupported macOS CPU architecture for Obscura: #{cpu}"
+          end
+        when :linux
+          case cpu
+          when :x86_64 then 'obscura-x86_64-linux-stealth.tar.gz'
+          when :aarch64, :arm64 then 'obscura-aarch64-linux-stealth.tar.gz'
+          else
+            raise UnsupportedPlatformError, "Unsupported Linux CPU architecture for Obscura: #{cpu}"
+          end
+        when :windows
+          'obscura-x86_64-windows-stealth.zip'
+        else
+          raise UnsupportedPlatformError, "Unsupported operating system for Obscura: #{RbConfig::CONFIG['host_os']}"
+        end
+      end
+
+      def obscura_version_file_path(dir)
+        File.join(dir, '.obscura_version')
+      end
+
       def binary_path(name, install_dir: nil)
         target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
         candidate = File.join(target_dir, binary_name_for_platform(name))

data/lib/http_mimic/module_methods.rb

@@ -35,6 +35,24 @@ module HttpMimic
       default_options[:proxy] = proxy_str
     end
 
+    def auto_proxy(enabled = nil)
+      return default_options[:auto_proxy] if enabled.nil?
+      default_options[:auto_proxy] = enabled
+    end
+
+    def proxy_pool
+      ProxyPool.instance
+    end
+
+    def proxy_sources(sources = nil)
+      return configuration.proxy_sources if sources.nil?
+      configuration.proxy_sources = sources
+    end
+
+    def refresh_proxies!(force: true)
+      ProxyPool.refresh!(force: force)
+    end
+
     def cookies(c = nil)
       return default_options[:cookies] if c.nil?
       default_options[:cookies] ||= {}
@@ -46,6 +64,21 @@ module HttpMimic
       default_options[:persist_cookies] = enabled
     end
 
+    def persist_on_failure(enabled = nil)
+      return default_options[:persist_on_failure] if enabled.nil?
+      default_options[:persist_on_failure] = enabled
+    end
+
+    def clear_on_failure(enabled = nil)
+      return default_options[:clear_on_failure] if enabled.nil?
+      default_options[:clear_on_failure] = enabled
+    end
+
+    def auto_render_spa(enabled = nil)
+      return default_options[:auto_render_spa] if enabled.nil?
+      default_options[:auto_render_spa] = enabled
+    end
+
     def mode(m = nil)
       return default_options[:mode] if m.nil?
       default_options[:mode] = m
@@ -93,6 +126,12 @@ module HttpMimic
       request(:options, url, options)
     end
 
+    def render(url, options = {})
+      merged = default_options.merge(options)
+      Obscura.render(url, merged)
+    end
+    alias spa render
+
     def request(method, url, options = {})
       merged = default_options.merge(options)