http_mimic 0.5.1 → 0.5.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b09e014134575dac987366c71b0cb6644a13a5b17088947621474f6cfccec5cf
-  data.tar.gz: 748d8d3dadcb69d39312a6c7137c32cca1290b7eaee4d8641fa7bb58bfdd0f80
+  metadata.gz: 9f042a9d602736a448c5341e306e8a46c68fcc38e5159cc881a157a52845d994
+  data.tar.gz: 03f41604be51bc6050e1778cd4411a0c3749ad04ef823c7d99839036ec2a1784
 SHA512:
-  metadata.gz: a4d17399d5f85df9b508151f371b0910e074d70e3c05f11cb291f30f01b71c20d22e998ec5c5466728c77dd2f37b3de4aecc18883a844fb69952ae71a3a5c5be
-  data.tar.gz: 7b0675e724e1c138cd57ea4af5e5a6ce2f057bb50981271f9b81cb894bf7179c2ba760137f0672d742a63b602783bc6b455a71b9da4f778c0637db96ee723470
+  metadata.gz: 53fa3674cd113da005dfa6a91b46ec2481dbe0c0f8e5fefb3594bb0e84d4b7258569c519d818bcb14a10034de61308155878cfb06cd8965f3e9c7e82613a338c
+  data.tar.gz: 47696a2a7f80d0ed2a8ca81236cd9a00f1de6fa8e5ae02bfae3aa2ef2b2965169497e49ee3aec585757c7fae12ac6a63da4398b3c57b720e3ccf6917416e433b

data/CHANGELOG.md

@@ -5,6 +5,63 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.5] - 2026-09-05
+
+### Fixed
+- **QuickJS CPT & Bot Manager Telemetry Resolution (Tier 1 WAF Bypass)**:
+  - **Standard `document.cookie` CookieJar**: Replaced the plain property in `browser_context.js` with a compliant `Map`-backed CookieJar. When Akamai CPT writes `document.cookie = "bm_lso=..."`, existing session cookies (`bm_sz`, `_abck`) are preserved rather than overwritten. This fixes the critical bug where main sensor telemetry fell back to default hash `8888888` instead of the session cookie hash.
+  - **`HTMLScriptElement.src` Property Reflection**: Implemented bidirectional synchronization between `el.src` and `el.getAttribute("src")`. CPT scripts can now successfully inspect `document.currentScript.getAttribute("src")` to locate their challenge parameters and relative post endpoints.
+  - **Accurate `document.currentScript` Execution Pipeline**: `AkamaiSolver.build_driver_script` now assigns `document.currentScript` before evaluating each script block, matching real browser multi-script execution order.
+  - **Hybrid `XMLHttpRequest` with `responseURL` Support**: Retained instance-level methods (`open`, `send`) required by obfuscated Akamai scripts while maintaining prototype-level methods for DOM script hooks, and populating `responseURL` to allow `challenge.html` reload checks to succeed.
+  - **Challenge Query String Preservation in Telemetry POST**: When sending CPT PoW solutions, `AkamaiSolver` now preserves query parameters (e.g. `v=...&t=...`) from the original challenge script URL so that edge servers correctly issue validated `bm_sc` session tokens.
+- **End-to-End Akamai Verification**:
+  - `HttpMimic.get('https://www.adidas.com/om/en/samba-og-shoes/KK2371.html')` now successfully solves the CPT challenge within QuickJS and returns the complete 400KB+ product HTML without invoking Tier 2 Obscura.
+
+## [0.5.4] - 2026-09-05
+
+
+### Fixed
+- **Challenge Page Regression Prevention**:
+  - `HttpMimic::Request` no longer preserves un-bypassed WAF challenge pages (such as Akamai 200 CPT skeletons) as `best_response`. If all resolution and fallback attempts fail, true status codes (e.g. 403) are preserved instead of overwriting with skeleton 200 responses.
+- **Explicit Profile Forwarding in Fallbacks & Retries**:
+  - `determine_profiles` now respects `explicit_profile = options[:profile]`, ensuring targeted profiles (e.g., `:android`) remain active when `auto_fallback: false` and are prioritized during fallback loops.
+  - `AkamaiSolver` now explicitly forwards `profile` and concrete `impersonate` targets during final URL retries to prevent TLS/fingerprint mismatches.
+- **Mobile/Android Emulation Fidelity in QuickJS Browser Polyfill**:
+  - Fixed typo in Android platform identifier (`Linux armv8l` instead of `Linux armv81`).
+  - Mobile environments now correctly emulate zero-length plugin and mimeType lists (removing desktop-only PDF plugins).
+  - Aligned mobile screen dimensions (412x915 portrait-primary) and WebGL renderer parameters (Qualcomm Adreno 640).
+- **Default Impersonate Profile Upgraded to `chrome150`**:
+  - Upgraded default client TLS fingerprint to `chrome150` to satisfy modern Akamai edge HTTP/2 and TLS cipher requirements.
+- **Modern CORS & Fetch Headers in Sensor POSTs**:
+  - Automatically injected `Sec-Fetch-Dest: empty`, `Sec-Fetch-Mode: cors`, `Sec-Fetch-Site: same-origin`, and client hints (`sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform`).
+
+## [0.5.3] - 2026-09-05
+
+### Added
+- **Automatic Free Proxy Pool (`auto_proxy`)**:
+  - Built-in `HttpMimic::ProxyPool` automatically gathers and caches hundreds of public HTTP proxies from maintained sources (`monosans`, `proxyscrape`, `TheSpeedX`).
+  - Disabled by default (`auto_proxy: false`) with per-request and global configuration opt-in.
+  - Transparent proxy failure retry: automatically detects dead proxies (exit codes 5, 7, 28, 35, 56 or connection errors), marks them dead in the pool, and transparently retries with fresh proxies up to `proxy_retries` (default: 3).
+  - Explicit manual `:proxy` option always takes precedence over `auto_proxy`.
+  - Seamless integration with Obscura and WAF solver handshakes.
+  - Added DSL helpers (`HttpMimic.auto_proxy`, `HttpMimic.proxy_pool`, `HttpMimic.refresh_proxies!`).
+- **Streamlined Documentation**:
+  - Refined README with concise, high-level highlights focusing on core anti-detect capabilities and developer experience.
+
+## [0.5.2] - 2026-09-05
+
+### Added
+- **Anti-Poisoning Failure Protection for CookieStore (`persist_on_failure`, `clear_on_failure`)**:
+  - `persist_on_failure` (default: `false`): Ensures failed/blocked requests (403, 401, retry statuses, or WAF challenge pages) do not save invalid or bot-flagged cookies to disk.
+  - `clear_on_failure` (default: `true`): Automatically purges stored host cookies when a request fails verification or is blocked by WAF, preventing poisoned sessions from breaking subsequent requests.
+  - Added class-level DSL and configuration support (`HttpMimic.persist_on_failure`, `HttpMimic.clear_on_failure`).
+- **Two-Phase Telemetry Handshake in `AkamaiSolver`**:
+  - Automatic multi-round telemetry loop sending second-stage interaction sensor posts after initial cookie acquisition to flip `_abck` tokens from `~-1~` to verified `~0~`.
+- **Intelligent Profile Fallback & WAF Protection**:
+  - `:auto` mode now rotates between distinct browser engines (`:impersonate` [Chrome], `:android`, `:ios`, `:firefox`) and skips plain `:curl` fallback on WAF targets to avoid IP flagging.
+  - Added `:firefox` and `:safari` profile support in `CommandBuilder`.
+  - Added Best Response Preservation: prevents successful/200 progress from being wiped out by later failed fallback attempts.
+
 ## [0.5.1] - 2026-08-30
 
 ### Added

data/README.md

@@ -1,41 +1,27 @@
-# HttpMimic
+# HttpMimic 🎭
 
-`http_mimic` is a Ruby HTTP Client gem built on top of `Open3.capture3` and `curl-impersonate`.
-It provides an elegant, concise, and intuitive **HTTParty-style** API while leveraging [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) to simulate authentic Chrome, Firefox, Safari, Edge, and Tor TLS / HTTP2 fingerprints (JA3, JA4, Akamai) and handshakes.
+> **Next-Generation Anti-Detect HTTP Client for Ruby**  
+> An elegant, high-performance HTTP client combining authentic browser TLS/JA4 fingerprint impersonation, built-in anti-bot challenge solving, lightweight headless SPA rendering, and an automatic free proxy pool into an intuitive HTTParty-style API.
 
-It also includes **Webdrivers-like automatic driver management**, automatically downloading and managing `curl-impersonate` binaries across macOS, Linux, and Windows without manual setup.
+`http_mimic` provides a complete, modern toolchain for accessing protected endpoints and web scraping:
+- **🎭 Authentic Browser Fingerprints**: Simulates authentic Chrome, Safari (macOS & iOS), Android, and Firefox TLS 1.3 / HTTP/2 handshakes and JA3/JA4 signatures.
+- **🛡️ Built-in Anti-Bot Defense**: Automatically detects and solves advanced WAF challenges at microsecond speeds without the overhead of heavy browsers.
+- **⚡ Dual-Tier Engine with Headless SPA Support**: Fast HTTP-first pipeline with seamless escalation to an embedded, lightweight (<100MB) Rust+V8 headless engine (`auto_render_spa`) when dynamic DOM rendering is needed.
+- **🌐 Automatic Free Proxy Pool (`auto_proxy`)**: Built-in, zero-dependency proxy pool with health tracking, auto-rotation, and transparent failure retry.
+- **🍪 Resilient Session Management**: Host-based persistent cookie caching with automatic anti-poisoning protection to prevent tainted sessions.
+- **📦 Zero-Setup Driver Management**: Automatically provisions and manages required native binaries across macOS, Linux, and Windows on demand.
+- **🚀 Intuitive Ruby DSL**: Clean, idiomatic syntax supporting class mixins (`include HttpMimic`), direct module calls (`HttpMimic.get`), and reusable client instances.
 
 ---
 
-## 🌟 Features
-
-- **Webdrivers-Style Driver Management**:
-  - Automatically downloads and unpacks official binaries from [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) to `~/.http_mimic/bin`.
-  - **Automatic platform & architecture detection** (macOS ARM/Intel, Linux x86_64/aarch64/musl, Windows x86_64/arm64, FreeBSD).
-  - Zero configuration required—installs on first request automatically.
-  - Manual driver management helpers: `HttpMimic.download_driver!`, `HttpMimic.driver_installed?`, `HttpMimic::Downloader.available_binaries`.
-- **Authentic Browser Handshakes & Fingerprints**:
-  - Chrome support: `chrome131` (default), `chrome124`, `chrome120`, `chrome133a`, `chrome136`, `chrome142`, `chrome99-110`.
-  - Firefox support: `firefox135`, `firefox133`, `firefox144`, `firefox117`, `firefox109`, `firefox102`, `firefox98`.
-  - Safari support: `safari180`, `safari170`, `safari155`, `safari153`.
-  - Edge & Tor support: `edge101`, `edge99`, `tor145`.
-- **HTTParty-Style API**:
-  - Direct module methods: `HttpMimic.get`, `HttpMimic.post`, etc.
-  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`, `persist_cookies`).
-  - Reusable instance client: `HttpMimic::Client.new(...)`.
-- **Zero Shell Injection Risk**:
-  - Executes commands with array arguments via `Open3.capture3(*cmd_array)`.
-  - Uses stdin streaming (`-d @-`) to safely handle large payloads without hitting OS command-line limits.
-- **Smart Response Parsing**:
-  - HTTP status helpers: `response.code`, `response.success?`, `response.redirect?`, `response.client_error?`, `response.server_error?`.
-  - Case-insensitive header access: `response.headers['Content-Type']`.
-  - Automatic `Set-Cookie` header parsing: `response.cookies['session_id']`.
-  - Auto-parsed JSON with object delegation: `response['key']`, `response.parsed_response`.
-  - Complete 3xx redirect history tracking: `response.history`.
-- **Comprehensive Request Options**:
-  - Supports `query`, `headers`, `json`, `body` (form data), `cookies`, `timeout`, `connect_timeout`, `proxy`, `basic_auth`, `digest_auth`, `bearer_token`, `insecure`, custom `curl_options`, and more.
-- **Graceful Fallback**:
-  - If a specific binary is unavailable and auto-download is disabled, automatically falls back to system standard `curl`.
+## 🌟 Highlights
+
+- **Authentic TLS & HTTP/2 Impersonation**: Native support for `chrome131` (default), `chrome120-142`, `safari180` (macOS/iOS), `firefox135`, `edge101`, and `tor145`.
+- **Intelligent Challenge Resolution**: Automatic WAF challenge mitigation and smart profile fallback to keep requests succeeding.
+- **Client-Side SPA Hydration**: High-speed HTTP execution by default, with optional V8 DOM hydration for complex JavaScript apps (`auto_render_spa: true`).
+- **Free Proxy Pool & Auto-Rotation**: Opt-in proxy management (`auto_proxy: true`) with automatic dead-proxy detection and retry.
+- **Safe & Reliable**: Zero shell injection risk via `Open3.capture3(*cmd_array)`, safe stdin streaming for large payloads, and smart response parsing.
+- **Flexible Integration**: Works seamlessly as a one-off client, a reusable instance, or a class-level DSL.
 
 ---
 
@@ -198,6 +184,8 @@ response = HttpMimic.get('https://example.com/items', persist_cookies: true)
 class Scraper
   include HttpMimic
   persist_cookies true
+  # persist_on_failure false # (default: do not save cookies on 403 / verification failure)
+  # clear_on_failure true    # (default: auto-clear cached cookies if verification fails)
 end
 
 # 3. Manual CookieStore helpers
@@ -206,6 +194,9 @@ HttpMimic.save_cookies('example.com', { session: '123' }) # Save cookies with de
 HttpMimic.clear_cookies!('example.com')                # Clear host or all cookies
 ```
 
+> **🛡️ Anti-Poisoning Failure Protection:**
+> By default, `persist_on_failure` is set to `false`, meaning cookies returned with HTTP error codes (e.g. 403, 401) or unverified WAF challenge pages are **never** persisted to disk. Furthermore, `clear_on_failure: true` automatically wipes tainted host cookies upon verification failure, preventing blocked sessions from poisoning subsequent requests.
+
 ---
 
 ## 🧭 Target-Specific Navigation Headers & Client Hints
@@ -247,6 +238,8 @@ HttpMimic.configure do |config|
   # Host CookieStore Persistence (disabled by default)
   config.persist_cookies         = false                                     # Auto-persist cookies per host
   config.cookie_store_dir        = File.expand_path('~/.http_mimic/cookies') # Directory for cookie store
+  config.persist_on_failure      = false                                     # Do not save cookies if request/verification fails
+  config.clear_on_failure        = true                                      # Purge cached host cookies on 403 / verification failure
 
   # WAF challenge solving (enabled by default)
   config.auto_solve_waf          = true        # Automatically solve detected WAF JS challenges
@@ -278,11 +271,14 @@ end
 | `:body` | Hash / String | Form payload (Hash) or raw request body string |
 | `:cookies` | Hash / String | Request cookies |
 | `:persist_cookies` | Boolean | Automatically load and save cookies for the host across requests |
+| `:persist_on_failure` | Boolean | Save cookies even if request or verification fails (default `false`) |
+| `:clear_on_failure` | Boolean | Automatically clear stored cookies on verification failure / 403 / WAF block (default `true`) |
 | `:cookie_jar` | String | Path to save cookies (`-c`) |
 | `:cookie_file` | String | Path to read cookies (`-b`) |
 | `:timeout` | Integer / Float | Maximum execution timeout in seconds (`--max-time`) |
 | `:connect_timeout` | Integer / Float | Connection timeout in seconds (`--connect-timeout`) |
 | `:proxy` | String | Proxy address (e.g., `'http://127.0.0.1:8888'`) |
+| `:auto_proxy` | Boolean | Automatically fetch and rotate through a pool of free HTTP proxies (default: `false`) |
 | `:basic_auth` | Hash | `{ username: 'admin', password: 'secret' }` |
 | `:bearer_token` | String | Appends `Authorization: Bearer <token>` header |
 | `:insecure` | Boolean | Disable SSL certificate verification (`-k`) |
@@ -331,6 +327,53 @@ response = HttpMimic.spa('https://example.com/spa')
 
 ---
 
+## 🌐 Automatic Free Proxy Pool (`auto_proxy`)
+
+`http_mimic` provides a built-in, zero-dependency Free Proxy Pool manager. When enabled, `HttpMimic` automatically fetches and pools hundreds of free HTTP proxies from maintained public sources (monosans, ProxyScrape, TheSpeedX), automatically marks dead proxies, and transparently retries requests on network or proxy failure.
+
+By default, `auto_proxy` is **`false`** so users can choose when to opt-in.
+
+### 1. Per-Request Opt-in
+
+```ruby
+# Automatically obtain a proxy from the pool and execute request
+response = HttpMimic.get('https://httpbin.org/ip', auto_proxy: true)
+
+# Explicit proxy always takes precedence over auto_proxy:
+response = HttpMimic.get('https://httpbin.org/ip', auto_proxy: true, proxy: 'http://my-dedicated-proxy:8080')
+```
+
+### 2. Global Configuration
+
+```ruby
+HttpMimic.configure do |config|
+  config.auto_proxy = true          # Enable free proxy pool by default
+  config.proxy_retries = 3          # Maximum proxy retry attempts on connection failure
+  config.proxy_pool_ttl = 1800      # Cache duration for fetched proxy list (seconds)
+end
+
+# Or via class-level DSL:
+HttpMimic.auto_proxy = true
+```
+
+### 3. Proxy Pool Management
+
+```ruby
+# Check available pool size
+HttpMimic.proxy_pool.size
+
+# Sample a random proxy from the pool
+HttpMimic.proxy_pool.sample # => "http://185.200.188.234:10001"
+
+# Manually refresh the proxy pool
+HttpMimic.refresh_proxies!
+
+# Load custom proxies into the pool
+HttpMimic.proxy_pool.load(['http://1.2.3.4:8080', 'http://5.6.7.8:3128'])
+```
+
+---
+
 ## 📄 Response Object
 
 The `Response` object wraps the HTTP response with convenient methods:

data/http_mimic.gemspec

@@ -21,7 +21,7 @@ Gem::Specification.new do |spec|
 
   spec.required_ruby_version = ">= 2.3.0"
 
-  spec.files         = Dir["lib/**/*.rb", "README.md", "CHANGELOG.md", "LICENSE.txt", "http_mimic.gemspec"]
+  spec.files         = Dir["lib/**/*.{rb,js}", "README.md", "CHANGELOG.md", "LICENSE.txt", "http_mimic.gemspec"]
   spec.require_paths = ["lib"]
 
   spec.add_development_dependency "bundler", ">= 1.17"

data/lib/http_mimic/command_builder.rb

@@ -15,7 +15,7 @@ module HttpMimic
 
     TARGET_BINARY_MAP = {
       # Chrome Desktop
-      'chrome'            => %w[curl_chrome131 curl_chrome124 curl_chrome120 curl_chrome116 curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome'            => %w[curl_chrome150 curl_chrome146 curl_chrome145 curl_chrome136 curl_chrome133a curl_chrome131 curl_chrome124 curl_chrome120 curl_chrome116 curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
       'chrome116'         => %w[curl_chrome116 curl-impersonate-chrome curl_chrome curl-impersonate],
       'chrome120'         => %w[curl_chrome120 curl-impersonate-chrome curl_chrome curl-impersonate],
       'chrome123'         => %w[curl_chrome123 curl-impersonate-chrome curl_chrome curl-impersonate],
@@ -118,10 +118,12 @@ module HttpMimic
       end
 
       # Timeouts
-      timeout = options[:timeout] || options[:read_timeout] || config.default_timeout
+      default_max_time = options[:auto_proxy] ? 12 : config.default_timeout
+      timeout = options[:timeout] || options[:read_timeout] || default_max_time
       args << '--max-time' << timeout.to_s if timeout
 
-      connect_timeout = options[:connect_timeout] || config.default_connect_timeout
+      default_conn_timeout = options[:auto_proxy] ? 4 : config.default_connect_timeout
+      connect_timeout = options[:connect_timeout] || default_conn_timeout
       args << '--connect-timeout' << connect_timeout.to_s if connect_timeout
 
       # SSL / Insecure
@@ -248,6 +250,10 @@ module HttpMimic
         (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
       when :mobile
         (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
+      when :firefox
+        (options[:firefox_impersonate] || 'firefox135').to_s.downcase
+      when :safari
+        (options[:safari_impersonate] || 'safari180').to_s.downcase
       else
         (options[:impersonate] || config.default_impersonate).to_s.downcase
       end

data/lib/http_mimic/configuration.rb

@@ -42,12 +42,21 @@ module HttpMimic
     # Shared Host-based CookieStore settings
     attr_accessor :persist_cookies
     attr_accessor :cookie_store_dir
+    attr_accessor :persist_on_failure
+    attr_accessor :clear_on_failure
 
     # Modern Navigation Headers simulation
     attr_accessor :navigation_headers
 
+    # Automatic Free Proxy Pool settings
+    attr_accessor :auto_proxy
+    attr_accessor :proxy_sources
+    attr_accessor :proxy_retries
+    attr_accessor :proxy_pool_ttl
+    attr_accessor :proxy_timeout
+
     def initialize
-      @default_impersonate         = 'chrome131'
+      @default_impersonate         = 'chrome150'
       @default_mobile_impersonate  = 'safari180_ios'
       @binary_path                 = nil
       @fallback_to_curl            = true
@@ -89,9 +98,18 @@ module HttpMimic
       # Shared Host CookieStore defaults
       @persist_cookies             = false
       @cookie_store_dir            = File.expand_path('~/.http_mimic/cookies')
+      @persist_on_failure          = false
+      @clear_on_failure            = true
 
       # Navigation Headers defaults
       @navigation_headers          = true
+
+      # Automatic Free Proxy Pool defaults
+      @auto_proxy                  = false
+      @proxy_sources               = nil
+      @proxy_retries               = 3
+      @proxy_pool_ttl              = 1800
+      @proxy_timeout               = 5
     end
   end
 end

data/lib/http_mimic/module_methods.rb

@@ -35,6 +35,24 @@ module HttpMimic
       default_options[:proxy] = proxy_str
     end
 
+    def auto_proxy(enabled = nil)
+      return default_options[:auto_proxy] if enabled.nil?
+      default_options[:auto_proxy] = enabled
+    end
+
+    def proxy_pool
+      ProxyPool.instance
+    end
+
+    def proxy_sources(sources = nil)
+      return configuration.proxy_sources if sources.nil?
+      configuration.proxy_sources = sources
+    end
+
+    def refresh_proxies!(force: true)
+      ProxyPool.refresh!(force: force)
+    end
+
     def cookies(c = nil)
       return default_options[:cookies] if c.nil?
       default_options[:cookies] ||= {}
@@ -46,6 +64,16 @@ module HttpMimic
       default_options[:persist_cookies] = enabled
     end
 
+    def persist_on_failure(enabled = nil)
+      return default_options[:persist_on_failure] if enabled.nil?
+      default_options[:persist_on_failure] = enabled
+    end
+
+    def clear_on_failure(enabled = nil)
+      return default_options[:clear_on_failure] if enabled.nil?
+      default_options[:clear_on_failure] = enabled
+    end
+
     def auto_render_spa(enabled = nil)
       return default_options[:auto_render_spa] if enabled.nil?
       default_options[:auto_render_spa] = enabled

data/lib/http_mimic/obscura.rb

@@ -82,13 +82,32 @@ module HttpMimic
 
         code = (status && status.success?) ? 200 : (status ? status.exitstatus : 500)
 
-        raw_headers = "HTTP/2 200 OK\r\ncontent-type: text/html; charset=utf-8\r\nx-rendered-by: obscura\r\n\r\n"
+        # Detect if the rendered output contains an explicit HTTP error block page
+        if code == 200 && stdout
+          if stdout.include?('HTTP 403 - Forbidden') ||
+             (stdout.include?('Reference Error:') && stdout.include?('Akamai')) ||
+             stdout.include?('"page_name": "403 ERROR"')
+            code = 403
+          elsif stdout.include?('404 Not Found') && stdout.include?('<title>404')
+            code = 404
+          end
+        end
+
+        status_msg = if code == 200
+          'OK (Obscura SPA Rendered)'
+        elsif code == 403
+          'Forbidden (WAF Blocked)'
+        else
+          'Error'
+        end
+
+        raw_headers = "HTTP/2 #{code} #{status_msg}\r\ncontent-type: text/html; charset=utf-8\r\nx-rendered-by: obscura\r\n\r\n"
         headers = Headers.new({ 'content-type' => 'text/html; charset=utf-8', 'x-rendered-by' => 'obscura' })
 
         Response.new(
           code: code,
           http_version: 'HTTP/2',
-          status_message: status&.success? ? 'OK (Obscura SPA Rendered)' : 'Error',
+          status_message: status_msg,
           headers: headers,
           cookies: Cookies.new,
           body: stdout,

data/lib/http_mimic/proxy_pool.rb

@@ -0,0 +1,224 @@
+# frozen_string_literal: true
+
+require 'net/http'
+require 'uri'
+require 'set'
+
+module HttpMimic
+  class ProxyPool
+    DEFAULT_SOURCES = [
+      'https://raw.githubusercontent.com/monosans/proxy-list/main/proxies/http.txt',
+      'https://api.proxyscrape.com/v2/?request=displayproxies&protocol=http&timeout=5000&country=all&ssl=all&anonymity=all',
+      'https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt'
+    ].freeze
+
+    class << self
+      def instance
+        @instance ||= new
+      end
+
+      def reset_instance!
+        @instance = nil
+      end
+
+      def get
+        instance.get
+      end
+      alias sample get
+      alias next_proxy get
+
+      def mark_dead(proxy)
+        instance.mark_dead(proxy)
+      end
+
+      def mark_alive(proxy)
+        instance.mark_alive(proxy)
+      end
+
+      def refresh!(force: true)
+        instance.refresh!(force: force)
+      end
+
+      def load(proxies)
+        instance.load(proxies)
+      end
+
+      def all
+        instance.all
+      end
+
+      def available
+        instance.available
+      end
+
+      def dead_proxies
+        instance.dead_proxies
+      end
+
+      def size
+        instance.size
+      end
+
+      def clear!
+        instance.clear!
+      end
+    end
+
+    attr_reader :sources, :ttl, :timeout
+    attr_accessor :proxies
+
+    def initialize(options = {})
+      config = HttpMimic.configuration rescue nil
+      @sources = options[:sources] || config&.proxy_sources || DEFAULT_SOURCES.dup
+      @ttl = options[:ttl] || config&.proxy_pool_ttl || 1800
+      @timeout = options[:timeout] || config&.proxy_timeout || 5
+      @proxies = []
+      @dead_proxies = Set.new
+      @last_fetched_at = nil
+      @mutex = Mutex.new
+    end
+
+    # Retrieve an available proxy from the pool
+    #
+    # @return [String, nil] Proxy URL (e.g. 'http://1.2.3.4:8080') or nil if none available
+    def get
+      @mutex.synchronize do
+        refresh_unlocked(force: false) if should_refresh_unlocked?
+        avail = @proxies - @dead_proxies.to_a
+        if avail.empty? && !@proxies.empty?
+          # If all current proxies are exhausted/dead, clear dead set and try one more refresh
+          @dead_proxies.clear
+          refresh_unlocked(force: true)
+          avail = @proxies - @dead_proxies.to_a
+        end
+        avail.sample
+      end
+    end
+    alias sample get
+    alias next_proxy get
+
+    # Mark a proxy as dead/unusable
+    #
+    # @param proxy [String]
+    def mark_dead(proxy)
+      return unless proxy
+      @mutex.synchronize do
+        @dead_proxies.add(normalize_proxy(proxy))
+      end
+    end
+
+    # Mark a proxy as active/usable
+    #
+    # @param proxy [String]
+    def mark_alive(proxy)
+      return unless proxy
+      @mutex.synchronize do
+        @dead_proxies.delete(normalize_proxy(proxy))
+      end
+    end
+
+    # Explicitly refresh proxy pool from configured sources
+    def refresh!(force: true)
+      @mutex.synchronize do
+        refresh_unlocked(force: force)
+      end
+    end
+
+    # Manually load a custom array of proxies
+    #
+    # @param proxy_list [Array<String>]
+    def load(proxy_list)
+      @mutex.synchronize do
+        @proxies = Array(proxy_list).map { |p| normalize_proxy(p) }.compact.uniq
+        @dead_proxies.clear
+        @last_fetched_at = Time.now
+      end
+    end
+
+    # Returns all loaded proxies
+    def all
+      @mutex.synchronize { @proxies.dup }
+    end
+
+    # Returns all non-dead available proxies
+    def available
+      @mutex.synchronize { (@proxies - @dead_proxies.to_a).dup }
+    end
+
+    # Returns dead proxies
+    def dead_proxies
+      @mutex.synchronize { @dead_proxies.to_a }
+    end
+
+    # Number of available proxies
+    def size
+      @mutex.synchronize { (@proxies - @dead_proxies.to_a).size }
+    end
+
+    # Reset proxy pool
+    def clear!
+      @mutex.synchronize do
+        @proxies.clear
+        @dead_proxies.clear
+        @last_fetched_at = nil
+      end
+    end
+
+    private
+
+    def should_refresh_unlocked?
+      @proxies.empty? || @last_fetched_at.nil? || (Time.now - @last_fetched_at > @ttl)
+    end
+
+    def refresh_unlocked(force: false)
+      return if !force && !should_refresh_unlocked?
+
+      new_proxies = fetch_from_sources
+      if !new_proxies.empty?
+        @proxies = new_proxies
+        @last_fetched_at = Time.now
+      end
+    end
+
+    def fetch_from_sources
+      collected = []
+      sources_to_try = Array(@sources).empty? ? DEFAULT_SOURCES : @sources
+
+      sources_to_try.each do |source_url|
+        begin
+          uri = URI.parse(source_url)
+          http = Net::HTTP.new(uri.host, uri.port)
+          http.use_ssl = (uri.scheme == 'https')
+          http.open_timeout = @timeout
+          http.read_timeout = @timeout
+
+          req = Net::HTTP::Get.new(uri.request_uri)
+          req['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
+
+          res = http.request(req)
+          if res.is_a?(Net::HTTPSuccess) && res.body
+            # Extract IP:PORT matches
+            ips = res.body.scan(/\b(?:\d{1,3}\.){3}\d{1,3}:\d{2,5}\b/)
+            if !ips.empty?
+              collected.concat(ips.take(150))
+              # Stop early if we have collected enough proxies
+              break if collected.size >= 50
+            end
+          end
+        rescue StandardError => e
+          if HttpMimic.configuration.debug
+            puts "[HttpMimic::ProxyPool] Failed to fetch proxy list from #{source_url}: #{e.message}"
+          end
+        end
+      end
+
+      collected.uniq.map { |p| normalize_proxy(p) }
+    end
+
+    def normalize_proxy(proxy)
+      p = proxy.to_s.strip
+      return nil if p.empty?
+      p.start_with?('http://', 'https://', 'socks5://', 'socks4://') ? p : "http://#{p}"
+    end
+  end
+end