http_mimic 0.5.1 → 0.5.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b09e014134575dac987366c71b0cb6644a13a5b17088947621474f6cfccec5cf
-  data.tar.gz: 748d8d3dadcb69d39312a6c7137c32cca1290b7eaee4d8641fa7bb58bfdd0f80
+  metadata.gz: 1b0b4192818428818cca7cde26b4ce90d6fa847db6e80e2b63c163dd8171bdd9
+  data.tar.gz: de6a71df66c710a30e7cb8fcbe6976653ec6242a6d29fd6959016513da4513be
 SHA512:
-  metadata.gz: a4d17399d5f85df9b508151f371b0910e074d70e3c05f11cb291f30f01b71c20d22e998ec5c5466728c77dd2f37b3de4aecc18883a844fb69952ae71a3a5c5be
-  data.tar.gz: 7b0675e724e1c138cd57ea4af5e5a6ce2f057bb50981271f9b81cb894bf7179c2ba760137f0672d742a63b602783bc6b455a71b9da4f778c0637db96ee723470
+  metadata.gz: 3b4ba6f1355da36e9f1064e62b1b988200f483a58b49524e3dd5350b0c99660d0b93ab650fc9ea8b3b7aa2204f41b7e2ca51c6b439128c267d7660be77b2b644
+  data.tar.gz: dd25b3fbf25dca0d935b910cb2b0981ab77d9dbfe45cfa5b522a99bbca7081016e6323af3fc59c389ef925c6df0b4d0bbf7321c7a93df5154d0d41e328962d76

data/CHANGELOG.md

@@ -5,6 +5,33 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.3] - 2026-09-05
+
+### Added
+- **Automatic Free Proxy Pool (`auto_proxy`)**:
+  - Built-in `HttpMimic::ProxyPool` automatically gathers and caches hundreds of public HTTP proxies from maintained sources (`monosans`, `proxyscrape`, `TheSpeedX`).
+  - Disabled by default (`auto_proxy: false`) with per-request and global configuration opt-in.
+  - Transparent proxy failure retry: automatically detects dead proxies (exit codes 5, 7, 28, 35, 56 or connection errors), marks them dead in the pool, and transparently retries with fresh proxies up to `proxy_retries` (default: 3).
+  - Explicit manual `:proxy` option always takes precedence over `auto_proxy`.
+  - Seamless integration with Obscura and WAF solver handshakes.
+  - Added DSL helpers (`HttpMimic.auto_proxy`, `HttpMimic.proxy_pool`, `HttpMimic.refresh_proxies!`).
+- **Streamlined Documentation**:
+  - Refined README with concise, high-level highlights focusing on core anti-detect capabilities and developer experience.
+
+## [0.5.2] - 2026-09-05
+
+### Added
+- **Anti-Poisoning Failure Protection for CookieStore (`persist_on_failure`, `clear_on_failure`)**:
+  - `persist_on_failure` (default: `false`): Ensures failed/blocked requests (403, 401, retry statuses, or WAF challenge pages) do not save invalid or bot-flagged cookies to disk.
+  - `clear_on_failure` (default: `true`): Automatically purges stored host cookies when a request fails verification or is blocked by WAF, preventing poisoned sessions from breaking subsequent requests.
+  - Added class-level DSL and configuration support (`HttpMimic.persist_on_failure`, `HttpMimic.clear_on_failure`).
+- **Two-Phase Telemetry Handshake in `AkamaiSolver`**:
+  - Automatic multi-round telemetry loop sending second-stage interaction sensor posts after initial cookie acquisition to flip `_abck` tokens from `~-1~` to verified `~0~`.
+- **Intelligent Profile Fallback & WAF Protection**:
+  - `:auto` mode now rotates between distinct browser engines (`:impersonate` [Chrome], `:android`, `:ios`, `:firefox`) and skips plain `:curl` fallback on WAF targets to avoid IP flagging.
+  - Added `:firefox` and `:safari` profile support in `CommandBuilder`.
+  - Added Best Response Preservation: prevents successful/200 progress from being wiped out by later failed fallback attempts.
+
 ## [0.5.1] - 2026-08-30
 
 ### Added

data/README.md

@@ -1,41 +1,27 @@
-# HttpMimic
+# HttpMimic 🎭
 
-`http_mimic` is a Ruby HTTP Client gem built on top of `Open3.capture3` and `curl-impersonate`.
-It provides an elegant, concise, and intuitive **HTTParty-style** API while leveraging [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) to simulate authentic Chrome, Firefox, Safari, Edge, and Tor TLS / HTTP2 fingerprints (JA3, JA4, Akamai) and handshakes.
+> **Next-Generation Anti-Detect HTTP Client for Ruby**  
+> An elegant, high-performance HTTP client combining authentic browser TLS/JA4 fingerprint impersonation, built-in anti-bot challenge solving, lightweight headless SPA rendering, and an automatic free proxy pool into an intuitive HTTParty-style API.
 
-It also includes **Webdrivers-like automatic driver management**, automatically downloading and managing `curl-impersonate` binaries across macOS, Linux, and Windows without manual setup.
+`http_mimic` provides a complete, modern toolchain for accessing protected endpoints and web scraping:
+- **🎭 Authentic Browser Fingerprints**: Simulates authentic Chrome, Safari (macOS & iOS), Android, and Firefox TLS 1.3 / HTTP/2 handshakes and JA3/JA4 signatures.
+- **🛡️ Built-in Anti-Bot Defense**: Automatically detects and solves advanced WAF challenges at microsecond speeds without the overhead of heavy browsers.
+- **⚡ Dual-Tier Engine with Headless SPA Support**: Fast HTTP-first pipeline with seamless escalation to an embedded, lightweight (<100MB) Rust+V8 headless engine (`auto_render_spa`) when dynamic DOM rendering is needed.
+- **🌐 Automatic Free Proxy Pool (`auto_proxy`)**: Built-in, zero-dependency proxy pool with health tracking, auto-rotation, and transparent failure retry.
+- **🍪 Resilient Session Management**: Host-based persistent cookie caching with automatic anti-poisoning protection to prevent tainted sessions.
+- **📦 Zero-Setup Driver Management**: Automatically provisions and manages required native binaries across macOS, Linux, and Windows on demand.
+- **🚀 Intuitive Ruby DSL**: Clean, idiomatic syntax supporting class mixins (`include HttpMimic`), direct module calls (`HttpMimic.get`), and reusable client instances.
 
 ---
 
-## 🌟 Features
-
-- **Webdrivers-Style Driver Management**:
-  - Automatically downloads and unpacks official binaries from [`lexiforest/curl-impersonate`](https://github.com/lexiforest/curl-impersonate) to `~/.http_mimic/bin`.
-  - **Automatic platform & architecture detection** (macOS ARM/Intel, Linux x86_64/aarch64/musl, Windows x86_64/arm64, FreeBSD).
-  - Zero configuration required—installs on first request automatically.
-  - Manual driver management helpers: `HttpMimic.download_driver!`, `HttpMimic.driver_installed?`, `HttpMimic::Downloader.available_binaries`.
-- **Authentic Browser Handshakes & Fingerprints**:
-  - Chrome support: `chrome131` (default), `chrome124`, `chrome120`, `chrome133a`, `chrome136`, `chrome142`, `chrome99-110`.
-  - Firefox support: `firefox135`, `firefox133`, `firefox144`, `firefox117`, `firefox109`, `firefox102`, `firefox98`.
-  - Safari support: `safari180`, `safari170`, `safari155`, `safari153`.
-  - Edge & Tor support: `edge101`, `edge99`, `tor145`.
-- **HTTParty-Style API**:
-  - Direct module methods: `HttpMimic.get`, `HttpMimic.post`, etc.
-  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`, `persist_cookies`).
-  - Reusable instance client: `HttpMimic::Client.new(...)`.
-- **Zero Shell Injection Risk**:
-  - Executes commands with array arguments via `Open3.capture3(*cmd_array)`.
-  - Uses stdin streaming (`-d @-`) to safely handle large payloads without hitting OS command-line limits.
-- **Smart Response Parsing**:
-  - HTTP status helpers: `response.code`, `response.success?`, `response.redirect?`, `response.client_error?`, `response.server_error?`.
-  - Case-insensitive header access: `response.headers['Content-Type']`.
-  - Automatic `Set-Cookie` header parsing: `response.cookies['session_id']`.
-  - Auto-parsed JSON with object delegation: `response['key']`, `response.parsed_response`.
-  - Complete 3xx redirect history tracking: `response.history`.
-- **Comprehensive Request Options**:
-  - Supports `query`, `headers`, `json`, `body` (form data), `cookies`, `timeout`, `connect_timeout`, `proxy`, `basic_auth`, `digest_auth`, `bearer_token`, `insecure`, custom `curl_options`, and more.
-- **Graceful Fallback**:
-  - If a specific binary is unavailable and auto-download is disabled, automatically falls back to system standard `curl`.
+## 🌟 Highlights
+
+- **Authentic TLS & HTTP/2 Impersonation**: Native support for `chrome131` (default), `chrome120-142`, `safari180` (macOS/iOS), `firefox135`, `edge101`, and `tor145`.
+- **Intelligent Challenge Resolution**: Automatic WAF challenge mitigation and smart profile fallback to keep requests succeeding.
+- **Client-Side SPA Hydration**: High-speed HTTP execution by default, with optional V8 DOM hydration for complex JavaScript apps (`auto_render_spa: true`).
+- **Free Proxy Pool & Auto-Rotation**: Opt-in proxy management (`auto_proxy: true`) with automatic dead-proxy detection and retry.
+- **Safe & Reliable**: Zero shell injection risk via `Open3.capture3(*cmd_array)`, safe stdin streaming for large payloads, and smart response parsing.
+- **Flexible Integration**: Works seamlessly as a one-off client, a reusable instance, or a class-level DSL.
 
 ---
 
@@ -198,6 +184,8 @@ response = HttpMimic.get('https://example.com/items', persist_cookies: true)
 class Scraper
   include HttpMimic
   persist_cookies true
+  # persist_on_failure false # (default: do not save cookies on 403 / verification failure)
+  # clear_on_failure true    # (default: auto-clear cached cookies if verification fails)
 end
 
 # 3. Manual CookieStore helpers
@@ -206,6 +194,9 @@ HttpMimic.save_cookies('example.com', { session: '123' }) # Save cookies with de
 HttpMimic.clear_cookies!('example.com')                # Clear host or all cookies
 ```
 
+> **🛡️ Anti-Poisoning Failure Protection:**
+> By default, `persist_on_failure` is set to `false`, meaning cookies returned with HTTP error codes (e.g. 403, 401) or unverified WAF challenge pages are **never** persisted to disk. Furthermore, `clear_on_failure: true` automatically wipes tainted host cookies upon verification failure, preventing blocked sessions from poisoning subsequent requests.
+
 ---
 
 ## 🧭 Target-Specific Navigation Headers & Client Hints
@@ -247,6 +238,8 @@ HttpMimic.configure do |config|
   # Host CookieStore Persistence (disabled by default)
   config.persist_cookies         = false                                     # Auto-persist cookies per host
   config.cookie_store_dir        = File.expand_path('~/.http_mimic/cookies') # Directory for cookie store
+  config.persist_on_failure      = false                                     # Do not save cookies if request/verification fails
+  config.clear_on_failure        = true                                      # Purge cached host cookies on 403 / verification failure
 
   # WAF challenge solving (enabled by default)
   config.auto_solve_waf          = true        # Automatically solve detected WAF JS challenges
@@ -278,11 +271,14 @@ end
 | `:body` | Hash / String | Form payload (Hash) or raw request body string |
 | `:cookies` | Hash / String | Request cookies |
 | `:persist_cookies` | Boolean | Automatically load and save cookies for the host across requests |
+| `:persist_on_failure` | Boolean | Save cookies even if request or verification fails (default `false`) |
+| `:clear_on_failure` | Boolean | Automatically clear stored cookies on verification failure / 403 / WAF block (default `true`) |
 | `:cookie_jar` | String | Path to save cookies (`-c`) |
 | `:cookie_file` | String | Path to read cookies (`-b`) |
 | `:timeout` | Integer / Float | Maximum execution timeout in seconds (`--max-time`) |
 | `:connect_timeout` | Integer / Float | Connection timeout in seconds (`--connect-timeout`) |
 | `:proxy` | String | Proxy address (e.g., `'http://127.0.0.1:8888'`) |
+| `:auto_proxy` | Boolean | Automatically fetch and rotate through a pool of free HTTP proxies (default: `false`) |
 | `:basic_auth` | Hash | `{ username: 'admin', password: 'secret' }` |
 | `:bearer_token` | String | Appends `Authorization: Bearer <token>` header |
 | `:insecure` | Boolean | Disable SSL certificate verification (`-k`) |
@@ -331,6 +327,53 @@ response = HttpMimic.spa('https://example.com/spa')
 
 ---
 
+## 🌐 Automatic Free Proxy Pool (`auto_proxy`)
+
+`http_mimic` provides a built-in, zero-dependency Free Proxy Pool manager. When enabled, `HttpMimic` automatically fetches and pools hundreds of free HTTP proxies from maintained public sources (monosans, ProxyScrape, TheSpeedX), automatically marks dead proxies, and transparently retries requests on network or proxy failure.
+
+By default, `auto_proxy` is **`false`** so users can choose when to opt-in.
+
+### 1. Per-Request Opt-in
+
+```ruby
+# Automatically obtain a proxy from the pool and execute request
+response = HttpMimic.get('https://httpbin.org/ip', auto_proxy: true)
+
+# Explicit proxy always takes precedence over auto_proxy:
+response = HttpMimic.get('https://httpbin.org/ip', auto_proxy: true, proxy: 'http://my-dedicated-proxy:8080')
+```
+
+### 2. Global Configuration
+
+```ruby
+HttpMimic.configure do |config|
+  config.auto_proxy = true          # Enable free proxy pool by default
+  config.proxy_retries = 3          # Maximum proxy retry attempts on connection failure
+  config.proxy_pool_ttl = 1800      # Cache duration for fetched proxy list (seconds)
+end
+
+# Or via class-level DSL:
+HttpMimic.auto_proxy = true
+```
+
+### 3. Proxy Pool Management
+
+```ruby
+# Check available pool size
+HttpMimic.proxy_pool.size
+
+# Sample a random proxy from the pool
+HttpMimic.proxy_pool.sample # => "http://185.200.188.234:10001"
+
+# Manually refresh the proxy pool
+HttpMimic.refresh_proxies!
+
+# Load custom proxies into the pool
+HttpMimic.proxy_pool.load(['http://1.2.3.4:8080', 'http://5.6.7.8:3128'])
+```
+
+---
+
 ## 📄 Response Object
 
 The `Response` object wraps the HTTP response with convenient methods:

data/lib/http_mimic/command_builder.rb

@@ -248,6 +248,10 @@ module HttpMimic
         (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
       when :mobile
         (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
+      when :firefox
+        (options[:firefox_impersonate] || 'firefox135').to_s.downcase
+      when :safari
+        (options[:safari_impersonate] || 'safari180').to_s.downcase
       else
         (options[:impersonate] || config.default_impersonate).to_s.downcase
       end

data/lib/http_mimic/configuration.rb

@@ -42,10 +42,19 @@ module HttpMimic
     # Shared Host-based CookieStore settings
     attr_accessor :persist_cookies
     attr_accessor :cookie_store_dir
+    attr_accessor :persist_on_failure
+    attr_accessor :clear_on_failure
 
     # Modern Navigation Headers simulation
     attr_accessor :navigation_headers
 
+    # Automatic Free Proxy Pool settings
+    attr_accessor :auto_proxy
+    attr_accessor :proxy_sources
+    attr_accessor :proxy_retries
+    attr_accessor :proxy_pool_ttl
+    attr_accessor :proxy_timeout
+
     def initialize
       @default_impersonate         = 'chrome131'
       @default_mobile_impersonate  = 'safari180_ios'
@@ -89,9 +98,18 @@ module HttpMimic
       # Shared Host CookieStore defaults
       @persist_cookies             = false
       @cookie_store_dir            = File.expand_path('~/.http_mimic/cookies')
+      @persist_on_failure          = false
+      @clear_on_failure            = true
 
       # Navigation Headers defaults
       @navigation_headers          = true
+
+      # Automatic Free Proxy Pool defaults
+      @auto_proxy                  = false
+      @proxy_sources               = nil
+      @proxy_retries               = 3
+      @proxy_pool_ttl              = 1800
+      @proxy_timeout               = 5
     end
   end
 end

data/lib/http_mimic/module_methods.rb

@@ -35,6 +35,24 @@ module HttpMimic
       default_options[:proxy] = proxy_str
     end
 
+    def auto_proxy(enabled = nil)
+      return default_options[:auto_proxy] if enabled.nil?
+      default_options[:auto_proxy] = enabled
+    end
+
+    def proxy_pool
+      ProxyPool.instance
+    end
+
+    def proxy_sources(sources = nil)
+      return configuration.proxy_sources if sources.nil?
+      configuration.proxy_sources = sources
+    end
+
+    def refresh_proxies!(force: true)
+      ProxyPool.refresh!(force: force)
+    end
+
     def cookies(c = nil)
       return default_options[:cookies] if c.nil?
       default_options[:cookies] ||= {}
@@ -46,6 +64,16 @@ module HttpMimic
       default_options[:persist_cookies] = enabled
     end
 
+    def persist_on_failure(enabled = nil)
+      return default_options[:persist_on_failure] if enabled.nil?
+      default_options[:persist_on_failure] = enabled
+    end
+
+    def clear_on_failure(enabled = nil)
+      return default_options[:clear_on_failure] if enabled.nil?
+      default_options[:clear_on_failure] = enabled
+    end
+
     def auto_render_spa(enabled = nil)
       return default_options[:auto_render_spa] if enabled.nil?
       default_options[:auto_render_spa] = enabled

data/lib/http_mimic/obscura.rb

@@ -82,13 +82,32 @@ module HttpMimic
 
         code = (status && status.success?) ? 200 : (status ? status.exitstatus : 500)
 
-        raw_headers = "HTTP/2 200 OK\r\ncontent-type: text/html; charset=utf-8\r\nx-rendered-by: obscura\r\n\r\n"
+        # Detect if the rendered output contains an explicit HTTP error block page
+        if code == 200 && stdout
+          if stdout.include?('HTTP 403 - Forbidden') ||
+             (stdout.include?('Reference Error:') && stdout.include?('Akamai')) ||
+             stdout.include?('"page_name": "403 ERROR"')
+            code = 403
+          elsif stdout.include?('404 Not Found') && stdout.include?('<title>404')
+            code = 404
+          end
+        end
+
+        status_msg = if code == 200
+          'OK (Obscura SPA Rendered)'
+        elsif code == 403
+          'Forbidden (WAF Blocked)'
+        else
+          'Error'
+        end
+
+        raw_headers = "HTTP/2 #{code} #{status_msg}\r\ncontent-type: text/html; charset=utf-8\r\nx-rendered-by: obscura\r\n\r\n"
         headers = Headers.new({ 'content-type' => 'text/html; charset=utf-8', 'x-rendered-by' => 'obscura' })
 
         Response.new(
           code: code,
           http_version: 'HTTP/2',
-          status_message: status&.success? ? 'OK (Obscura SPA Rendered)' : 'Error',
+          status_message: status_msg,
           headers: headers,
           cookies: Cookies.new,
           body: stdout,

data/lib/http_mimic/proxy_pool.rb

@@ -0,0 +1,224 @@
+# frozen_string_literal: true
+
+require 'net/http'
+require 'uri'
+require 'set'
+
+module HttpMimic
+  class ProxyPool
+    DEFAULT_SOURCES = [
+      'https://raw.githubusercontent.com/monosans/proxy-list/main/proxies/http.txt',
+      'https://api.proxyscrape.com/v2/?request=displayproxies&protocol=http&timeout=5000&country=all&ssl=all&anonymity=all',
+      'https://raw.githubusercontent.com/TheSpeedX/SOCKS-List/master/http.txt'
+    ].freeze
+
+    class << self
+      def instance
+        @instance ||= new
+      end
+
+      def reset_instance!
+        @instance = nil
+      end
+
+      def get
+        instance.get
+      end
+      alias sample get
+      alias next_proxy get
+
+      def mark_dead(proxy)
+        instance.mark_dead(proxy)
+      end
+
+      def mark_alive(proxy)
+        instance.mark_alive(proxy)
+      end
+
+      def refresh!(force: true)
+        instance.refresh!(force: force)
+      end
+
+      def load(proxies)
+        instance.load(proxies)
+      end
+
+      def all
+        instance.all
+      end
+
+      def available
+        instance.available
+      end
+
+      def dead_proxies
+        instance.dead_proxies
+      end
+
+      def size
+        instance.size
+      end
+
+      def clear!
+        instance.clear!
+      end
+    end
+
+    attr_reader :sources, :ttl, :timeout
+    attr_accessor :proxies
+
+    def initialize(options = {})
+      config = HttpMimic.configuration rescue nil
+      @sources = options[:sources] || config&.proxy_sources || DEFAULT_SOURCES.dup
+      @ttl = options[:ttl] || config&.proxy_pool_ttl || 1800
+      @timeout = options[:timeout] || config&.proxy_timeout || 5
+      @proxies = []
+      @dead_proxies = Set.new
+      @last_fetched_at = nil
+      @mutex = Mutex.new
+    end
+
+    # Retrieve an available proxy from the pool
+    #
+    # @return [String, nil] Proxy URL (e.g. 'http://1.2.3.4:8080') or nil if none available
+    def get
+      @mutex.synchronize do
+        refresh_unlocked(force: false) if should_refresh_unlocked?
+        avail = @proxies - @dead_proxies.to_a
+        if avail.empty? && !@proxies.empty?
+          # If all current proxies are exhausted/dead, clear dead set and try one more refresh
+          @dead_proxies.clear
+          refresh_unlocked(force: true)
+          avail = @proxies - @dead_proxies.to_a
+        end
+        avail.sample
+      end
+    end
+    alias sample get
+    alias next_proxy get
+
+    # Mark a proxy as dead/unusable
+    #
+    # @param proxy [String]
+    def mark_dead(proxy)
+      return unless proxy
+      @mutex.synchronize do
+        @dead_proxies.add(normalize_proxy(proxy))
+      end
+    end
+
+    # Mark a proxy as active/usable
+    #
+    # @param proxy [String]
+    def mark_alive(proxy)
+      return unless proxy
+      @mutex.synchronize do
+        @dead_proxies.delete(normalize_proxy(proxy))
+      end
+    end
+
+    # Explicitly refresh proxy pool from configured sources
+    def refresh!(force: true)
+      @mutex.synchronize do
+        refresh_unlocked(force: force)
+      end
+    end
+
+    # Manually load a custom array of proxies
+    #
+    # @param proxy_list [Array<String>]
+    def load(proxy_list)
+      @mutex.synchronize do
+        @proxies = Array(proxy_list).map { |p| normalize_proxy(p) }.compact.uniq
+        @dead_proxies.clear
+        @last_fetched_at = Time.now
+      end
+    end
+
+    # Returns all loaded proxies
+    def all
+      @mutex.synchronize { @proxies.dup }
+    end
+
+    # Returns all non-dead available proxies
+    def available
+      @mutex.synchronize { (@proxies - @dead_proxies.to_a).dup }
+    end
+
+    # Returns dead proxies
+    def dead_proxies
+      @mutex.synchronize { @dead_proxies.to_a }
+    end
+
+    # Number of available proxies
+    def size
+      @mutex.synchronize { (@proxies - @dead_proxies.to_a).size }
+    end
+
+    # Reset proxy pool
+    def clear!
+      @mutex.synchronize do
+        @proxies.clear
+        @dead_proxies.clear
+        @last_fetched_at = nil
+      end
+    end
+
+    private
+
+    def should_refresh_unlocked?
+      @proxies.empty? || @last_fetched_at.nil? || (Time.now - @last_fetched_at > @ttl)
+    end
+
+    def refresh_unlocked(force: false)
+      return if !force && !should_refresh_unlocked?
+
+      new_proxies = fetch_from_sources
+      if !new_proxies.empty?
+        @proxies = new_proxies
+        @last_fetched_at = Time.now
+      end
+    end
+
+    def fetch_from_sources
+      collected = []
+      sources_to_try = Array(@sources).empty? ? DEFAULT_SOURCES : @sources
+
+      sources_to_try.each do |source_url|
+        begin
+          uri = URI.parse(source_url)
+          http = Net::HTTP.new(uri.host, uri.port)
+          http.use_ssl = (uri.scheme == 'https')
+          http.open_timeout = @timeout
+          http.read_timeout = @timeout
+
+          req = Net::HTTP::Get.new(uri.request_uri)
+          req['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36'
+
+          res = http.request(req)
+          if res.is_a?(Net::HTTPSuccess) && res.body
+            # Extract IP:PORT matches
+            ips = res.body.scan(/\b(?:\d{1,3}\.){3}\d{1,3}:\d{2,5}\b/)
+            if !ips.empty?
+              collected.concat(ips.take(150))
+              # Stop early if we have collected enough proxies
+              break if collected.size >= 50
+            end
+          end
+        rescue StandardError => e
+          if HttpMimic.configuration.debug
+            puts "[HttpMimic::ProxyPool] Failed to fetch proxy list from #{source_url}: #{e.message}"
+          end
+        end
+      end
+
+      collected.uniq.map { |p| normalize_proxy(p) }
+    end
+
+    def normalize_proxy(proxy)
+      p = proxy.to_s.strip
+      return nil if p.empty?
+      p.start_with?('http://', 'https://', 'socks5://', 'socks4://') ? p : "http://#{p}"
+    end
+  end
+end

data/lib/http_mimic/request.rb

@@ -16,6 +16,16 @@ module HttpMimic
     def perform
       mode = (options[:mode] || config.mode || :auto).to_sym
 
+      # Automatic Free Proxy Pool integration
+      should_auto_proxy = options.fetch(:auto_proxy, config.auto_proxy)
+      if should_auto_proxy && !options[:proxy]
+        selected_proxy = ProxyPool.get
+        if selected_proxy
+          options[:proxy] = selected_proxy
+          log_debug("[HttpMimic::ProxyPool] Assigned proxy from pool: #{selected_proxy}")
+        end
+      end
+
       # Delegate directly to Obscura headless SPA renderer if requested
       if options[:render] == :spa || options[:render] == :obscura || mode == :spa || mode == :obscura
         return Obscura.render(url, options)
@@ -40,6 +50,8 @@ module HttpMimic
       attempts = []
       profiles_to_try = determine_profiles(mode, auto_fallback)
       waf_solve_attempted = false
+      best_response = nil
+      proxy_retries_left = should_auto_proxy ? (options[:proxy_retries] || config.proxy_retries || 3) : 0
 
       response = nil
       final_status = nil
@@ -47,6 +59,12 @@ module HttpMimic
       final_command = nil
 
       profiles_to_try.each_with_index do |profile, index|
+        # Never fall back to plain curl if a WAF challenge has already been detected/attempted
+        if profile == :curl && waf_solve_attempted
+          log_debug("[HttpMimic] Skipping plain :curl fallback for protected WAF target.")
+          next
+        end
+
         current_opts = options.merge(profile: profile)
 
         builder = CommandBuilder.new(method, url, current_opts, config)
@@ -61,6 +79,25 @@ module HttpMimic
         log_debug("Curl exit status: #{status.exitstatus}")
         log_debug("Curl stderr: #{stderr}") unless stderr.empty?
 
+        # Handle proxy failure retry when auto_proxy is enabled
+        if should_auto_proxy && current_opts[:proxy] && (status.exitstatus != 0)
+          is_proxy_err = [5, 7, 28, 35, 56].include?(status.exitstatus) ||
+                         stderr.include?('Failed to connect to') ||
+                         stderr.include?('tunneling failed') ||
+                         stderr.downcase.include?('proxy')
+
+          if is_proxy_err
+            ProxyPool.mark_dead(current_opts[:proxy])
+            if proxy_retries_left > 0
+              proxy_retries_left -= 1
+              new_proxy = ProxyPool.get
+              log_debug("[HttpMimic::ProxyPool] Proxy #{current_opts[:proxy]} failed (exit #{status.exitstatus}). Retrying with new proxy #{new_proxy} (#{proxy_retries_left} retries left)...")
+              options[:proxy] = new_proxy
+              redo
+            end
+          end
+        end
+
         response = ResponseParser.new(
           stdout,
           exit_status: status,
@@ -69,6 +106,10 @@ module HttpMimic
           request_url: final_url
         ).parse
 
+        if should_auto_proxy && current_opts[:proxy] && response.success?
+          ProxyPool.mark_alive(current_opts[:proxy])
+        end
+
         response.mode_used = profile
         response.fallback_triggered = (index > 0)
 
@@ -86,6 +127,17 @@ module HttpMimic
         final_stderr = stderr
         final_command = full_command
 
+        # Track best response so a 200 (or challenge page) isn't wiped out by a failed fallback
+        if response
+          if best_response.nil?
+            best_response = response
+          elsif response.success? && !best_response.success?
+            best_response = response
+          elsif response.code == 200 && best_response.code != 200
+            best_response = response
+          end
+        end
+
         # Forward any received cookies to subsequent attempts
         if response.cookies && !response.cookies.empty?
           existing_cookies = options[:cookies] ? (options[:cookies].is_a?(Hash) ? options[:cookies] : options[:cookies].to_h) : {}
@@ -104,10 +156,13 @@ module HttpMimic
             solved_resp = Waf.solve(url, response, current_opts)
             if solved_resp
               response = solved_resp
-              is_blocked = (response.code != 0 && retry_statuses.include?(response.code))
+              is_blocked = (response.code != 0 && retry_statuses.include?(response.code)) || Waf::Detector.challenge_page?(response)
               if response.cookies && !response.cookies.empty?
                 options[:cookies] = (options[:cookies] || {}).merge(response.cookies.to_h)
               end
+              if response.success? || (response.code == 200 && best_response&.code != 200)
+                best_response = response
+              end
             end
           end
         end
@@ -119,33 +174,61 @@ module HttpMimic
         log_debug("[HttpMimic] Attempt #{index + 1} with #{profile} resulted in status #{response.code}. Triggering smart fallback to next profile...")
       end
 
-      # Automatic SPA Detection & Obscura rendering fallback
+      # Preserve best response if the last attempt resulted in a regression (e.g. 403 / error after getting 200)
+      if best_response && (response.nil? || response.error? || (final_status && final_status.exitstatus != 0))
+        if best_response.success? || (best_response.code == 200 && response&.code != 200)
+          response = best_response
+        end
+      end
+
+      # Automatic SPA Detection & Obscura rendering fallback for SPA shells & Behavioral Challenges
       auto_render_spa = options.fetch(:auto_render_spa, config.auto_render_spa)
-      if auto_render_spa && response && response.success? && method.to_s.upcase == 'GET'
-        if SpaDetector.spa?(response)
-          log_debug("[HttpMimic] Detected unhydrated SPA shell on #{url}. Automatically rendering with Obscura...")
-          begin
-            spa_opts = options.dup
-            # Forward all validated cookies from Tier 1 (Mode 2: Two-Phase Pipeline)
-            if response.cookies && !response.cookies.empty?
-              tier1_cookies = response.cookies.to_h
-              existing_cookies = spa_opts[:cookies].is_a?(Hash) ? spa_opts[:cookies] : {}
-              spa_opts[:cookies] = existing_cookies.merge(tier1_cookies)
-            end
-            rendered_resp = Obscura.render(url, spa_opts)
-            if rendered_resp && rendered_resp.success?
-              response = rendered_resp
-            end
-          rescue StandardError => e
-            log_debug("[HttpMimic] Automatic Obscura SPA render failed (#{e.message}), keeping Tier 1 response.")
+      auto_solve_waf = options.fetch(:solve_waf, config.auto_solve_waf)
+      should_render_spa = auto_render_spa && response && response.success? && SpaDetector.spa?(response)
+      should_render_cpt = auto_solve_waf && response && Waf::Detector.challenge_page?(response)
+
+      if (should_render_spa || should_render_cpt) && method.to_s.upcase == 'GET'
+        target_reason = should_render_cpt ? 'WAF challenge page' : 'unhydrated SPA shell'
+        log_debug("[HttpMimic] Detected #{target_reason} on #{url}. Automatically rendering with Obscura...")
+        begin
+          spa_opts = options.dup
+          # Forward all validated cookies from Tier 1 (Mode 2: Two-Phase Pipeline)
+          if response.cookies && !response.cookies.empty?
+            tier1_cookies = response.cookies.to_h
+            existing_cookies = spa_opts[:cookies].is_a?(Hash) ? spa_opts[:cookies] : {}
+            spa_opts[:cookies] = existing_cookies.merge(tier1_cookies)
+          end
+          spa_opts[:wait_until] ||= 'load' if should_render_cpt
+          rendered_resp = Obscura.render(url, spa_opts)
+          if rendered_resp && rendered_resp.success?
+            response = rendered_resp
           end
+        rescue StandardError => e
+          log_debug("[HttpMimic] Automatic Obscura render failed (#{e.message}), keeping Tier 1 response.")
         end
       end
 
       # Persist cookies back to store if enabled
-      if should_persist_cookie && host && response && response.cookies && !response.cookies.empty?
-        CookieStore.save(host, response.cookies)
-        log_debug("[HttpMimic::CookieStore] Saved #{response.cookies.size} cookies for #{host}")
+      if should_persist_cookie && host
+        is_success = response && (response.success? || response.redirect?) && !Waf::Detector.challenge_page?(response) && (final_status.nil? || final_status.exitstatus == 0)
+        verification_failed = !is_success && response && (
+          [401, 403].include?(response.code) ||
+          retry_statuses.include?(response.code) ||
+          Waf::Detector.challenge_page?(response)
+        )
+
+        persist_on_failure = options.fetch(:persist_on_failure, config.persist_on_failure)
+        clear_on_failure = options.fetch(:clear_on_failure, config.clear_on_failure)
+
+        if is_success || persist_on_failure
+          if response && response.cookies && !response.cookies.empty?
+            CookieStore.save(host, response.cookies)
+            log_debug("[HttpMimic::CookieStore] Saved #{response.cookies.size} cookies for #{host}")
+          end
+        elsif clear_on_failure && verification_failed
+          CookieStore.clear(host)
+          log_debug("[HttpMimic::CookieStore] Verification failed for #{host} (status: #{response&.code}). Cleared stored cookies.")
+        end
       end
 
       handle_errors(final_status, final_stderr, final_command, response)
@@ -157,7 +240,7 @@ module HttpMimic
     def determine_profiles(mode, auto_fallback)
       case mode
       when :curl, :curl_first
-        auto_fallback ? [:curl, :impersonate, :android, :ios] : [:curl]
+        auto_fallback ? [:curl, :impersonate, :android, :ios, :firefox] : [:curl]
       when :curl_only
         [:curl]
       when :impersonate_only
@@ -168,18 +251,22 @@ module HttpMimic
         [:android]
       when :ios_only
         [:ios]
+      when :firefox_only
+        [:firefox]
+      when :safari_only
+        [:safari]
       when :mobile_first
-        auto_fallback ? [:android, :ios, :impersonate, :curl] : [:mobile]
+        auto_fallback ? [:android, :ios, :impersonate, :firefox] : [:mobile]
       when :android_first
-        auto_fallback ? [:android, :ios, :impersonate, :curl] : [:android]
+        auto_fallback ? [:android, :ios, :impersonate, :firefox] : [:android]
       when :ios_first
-        auto_fallback ? [:ios, :android, :impersonate, :curl] : [:ios]
+        auto_fallback ? [:ios, :android, :impersonate, :firefox] : [:ios]
       when :impersonate_first
-        auto_fallback ? [:impersonate, :android, :ios, :curl] : [:impersonate]
+        auto_fallback ? [:impersonate, :android, :ios, :firefox] : [:impersonate]
       when :auto, :smart
-        auto_fallback ? [:impersonate, :android, :ios, :curl] : [:impersonate]
+        auto_fallback ? [:impersonate, :android, :ios, :firefox] : [:impersonate]
       else
-        auto_fallback ? [:impersonate, :android, :ios, :curl] : [:impersonate]
+        auto_fallback ? [:impersonate, :android, :ios, :firefox] : [:impersonate]
       end
     end
 

data/lib/http_mimic/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module HttpMimic
-  VERSION = "0.5.1"
+  VERSION = "0.5.3"
 end

data/lib/http_mimic/waf/akamai_solver.rb

@@ -22,6 +22,7 @@ module HttpMimic
             impersonate: impersonate,
             cookies: cookies,
             headers: { 'Referer' => target_url },
+            proxy: options[:proxy],
             auto_fallback: false,
             solve_waf: false
           )
@@ -29,22 +30,81 @@ module HttpMimic
 
           sensor_js = script_resp.body
           context_js = File.read(CONTEXT_JS_PATH)
-          cookie_str = cookies.to_cookie_string
 
           doc_title = initial_response.title.to_s
           user_agent = options[:user_agent]
           referer = (options[:headers] || {})['Referer'] || (options[:headers] || {})['referer'] || target_url
 
-          # 2. Run virtual browser simulation inside QuickJS
-          driver_script = <<~JS
-            globalThis.__TARGET_URL__ = #{target_url.to_json};
-            globalThis.__DOCUMENT_TITLE__ = #{doc_title.to_json};
-            globalThis.__INITIAL_COOKIES__ = #{cookie_str.to_json};
-            globalThis.__REFERRER__ = #{referer.to_json};
-            #{user_agent ? "globalThis.__USER_AGENT__ = #{user_agent.to_json};" : ""}
+          # 2. Phase 1: Virtual browser simulation inside QuickJS (initial telemetry)
+          driver_script_1 = build_driver_script(
+            target_url: target_url,
+            doc_title: doc_title,
+            cookie_str: cookies.to_cookie_string,
+            referer: referer,
+            user_agent: user_agent,
+            context_js: context_js,
+            sensor_js: sensor_js,
+            phase: 1
+          )
+
+          result1 = JSRuntime.eval_json(driver_script_1)
+          sensor_posts_1 = result1 && result1['sensor_posts']
+          return nil if sensor_posts_1.nil? || sensor_posts_1.empty?
+
+          # 3. Post Phase 1 sensor data to Akamai endpoint
+          updated_cookies = cookies.dup
+          post_sensor_data(sensor_script_url, target_url, impersonate, updated_cookies, sensor_posts_1, proxy: options[:proxy])
+
+          # 4. Phase 2: If _abck is not verified yet (~0~), perform second round after short pause
+          if !verified_abck?(updated_cookies)
+            sleep 0.8
+            driver_script_2 = build_driver_script(
+              target_url: target_url,
+              doc_title: doc_title,
+              cookie_str: updated_cookies.to_cookie_string,
+              referer: referer,
+              user_agent: user_agent,
+              context_js: context_js,
+              sensor_js: sensor_js,
+              phase: 2
+            )
+
+            result2 = JSRuntime.eval_json(driver_script_2)
+            sensor_posts_2 = result2 && result2['sensor_posts']
+            if sensor_posts_2 && !sensor_posts_2.empty?
+              post_sensor_data(sensor_script_url, target_url, impersonate, updated_cookies, sensor_posts_2, proxy: options[:proxy])
+            end
+          end
+
+          # 5. Retry original target URL with the updated cookies
+          retry_headers = (options[:headers] || {}).merge('Referer' => target_url)
+          HttpMimic.get(
+            target_url,
+            options.merge(
+              cookies: updated_cookies,
+              headers: retry_headers,
+              auto_fallback: false,
+              solve_waf: false
+            )
+          )
+        rescue StandardError => e
+          if HttpMimic.configuration.debug
+            puts "[HttpMimic::Waf::AkamaiSolver] Error solving Akamai challenge: #{e.message}"
+          end
+          nil
+        end
+
+        def build_driver_script(params)
+          <<~JS
+            globalThis.__TARGET_URL__ = #{params[:target_url].to_json};
+            globalThis.__DOCUMENT_TITLE__ = #{params[:doc_title].to_json};
+            globalThis.__INITIAL_COOKIES__ = #{params[:cookie_str].to_json};
+            globalThis.__REFERRER__ = #{params[:referer].to_json};
+            #{params[:user_agent] ? "globalThis.__USER_AGENT__ = #{params[:user_agent].to_json};" : ""}
+            globalThis.__TELEMETRY_PHASE__ = #{params[:phase] || 1};
 
-            #{context_js}
-            #{sensor_js}
+            #{params[:context_js]}
+            #{params[:sensor_js]}
 
             if (typeof globalThis.__simulateHumanInteractions === "function") {
               globalThis.__simulateHumanInteractions();
@@ -57,63 +117,61 @@ module HttpMimic
               sensor_posts: globalThis.__sensor_posts
             });
           JS
+        end
 
-          result = JSRuntime.eval_json(driver_script)
-          sensor_posts = result && result['sensor_posts']
-          return nil if sensor_posts.nil? || sensor_posts.empty?
-
-          # 3. Post sensor data to Akamai endpoint
-          updated_cookies = cookies.dup
+        def post_sensor_data(sensor_script_url, target_url, impersonate, updated_cookies, sensor_posts, proxy: nil)
+          origin_url = URI.parse(target_url).tap { |u| u.path = ''; u.query = nil }.to_s rescue target_url
           sensor_posts.each do |post|
             post_body = post['body']
             next if post_body.nil? || post_body.empty?
 
+            post_url = if post['url'].to_s.empty?
+              target_url
+            elsif post['url'].start_with?('http')
+              post['url']
+            else
+              URI.join(target_url, post['url']).to_s
+            end
+
+            content_type = (post['headers'] && (post['headers']['Content-Type'] || post['headers']['content-type'])) || 'application/json'
+
             post_resp = HttpMimic.post(
-              sensor_script_url,
+              post_url,
               impersonate: impersonate,
               cookies: updated_cookies,
               headers: {
-                'Content-Type' => 'text/plain;charset=UTF-8',
+                'Content-Type' => content_type,
                 'Referer' => target_url,
-                'Origin' => URI.parse(target_url).tap { |u| u.path = ''; u.query = nil }.to_s,
+                'Origin' => origin_url,
                 'Accept' => '*/*'
               },
+              proxy: proxy,
               body: post_body,
               auto_fallback: false,
               solve_waf: false
             )
 
-            post_resp.cookies.each { |k, v| updated_cookies[k] = v }
+            post_resp.cookies.each { |k, v| updated_cookies[k] = v } if post_resp&.cookies
           end
+        end
 
-          # 4. Retry original target URL with the updated cookies
-          retry_headers = (options[:headers] || {}).merge('Referer' => target_url)
-          HttpMimic.get(
-            target_url,
-            options.merge(
-              cookies: updated_cookies,
-              headers: retry_headers,
-              auto_fallback: false,
-              solve_waf: false
-            )
-          )
-        rescue StandardError => e
-          if HttpMimic.configuration.debug
-            puts "[HttpMimic::Waf::AkamaiSolver] Error solving Akamai challenge: #{e.message}"
-          end
-          nil
+        def verified_abck?(cookies)
+          return false unless cookies
+          abck = cookies['_abck'] || (cookies.respond_to?(:key?) && cookies['_abck'])
+          abck.to_s.include?('~0~')
         end
 
         def extract_sensor_script_url(target_url, html)
           return nil if html.nil? || html.empty?
 
-          match = html.match(/<script[^>]*src=["\x27]([^"\x27]*\/[a-zA-Z0-9_-]{10,}\?[a-zA-Z0-9_=-]+)["\x27]/i)
+          match = html.match(/<script[^>]*src=["\x27]([^"\x27]*\/[a-zA-Z0-9_\-\/]+\?[^"\x27\s>]+)["\x27]/i)
           match ||= html.match(/<script[^>]*src=["\x27]([^"\x27]*akam[^"\x27]*)["\x27]/i)
-          match ||= html.match(/<script[^>]*src=["\x27]([^"\x27]*\/[a-zA-Z0-9_\-\/]+\?v=[a-zA-Z0-9_-]+)["\x27]/i)
+          match ||= html.match(/<script[^>]*src=["\x27]([^"\x27]*\/[a-zA-Z0-9_-]{10,}\?[a-zA-Z0-9_=-]+)["\x27]/i)
 
           return nil unless match
 
-          URI.join(target_url, match[1]).to_s
+          raw_url = match[1].gsub('&amp;', '&')
+          URI.join(target_url, raw_url).to_s
         rescue StandardError
           nil
         end

data/lib/http_mimic.rb

@@ -21,6 +21,7 @@ require 'http_mimic/waf'
 require 'http_mimic/cookie_store'
 require 'http_mimic/obscura'
 require 'http_mimic/spa_detector'
+require 'http_mimic/proxy_pool'
 
 module HttpMimic
   extend ModuleMethods

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: http_mimic
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.5.3
 platform: ruby
 authors:
 - anxgang
@@ -73,6 +73,7 @@ files:
 - lib/http_mimic/js_runtime.rb
 - lib/http_mimic/module_methods.rb
 - lib/http_mimic/obscura.rb
+- lib/http_mimic/proxy_pool.rb
 - lib/http_mimic/request.rb
 - lib/http_mimic/response.rb
 - lib/http_mimic/response_parser.rb