http_mimic 0.3.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 11e55edf4c06160f27621a651e4c16eae929ec08767b5b6eff6b007d8a3a1d4b
-  data.tar.gz: e90137a02ee867cd744641c725cd3daef2994b5f4a7216304f7ca189e6d92303
+  metadata.gz: 9b0a9fff1b74c98ea24c294a491b00ba57e270d24e6c0bb7a669e6d2db264d8a
+  data.tar.gz: e11d3ae615b92abcfedda8547120290d82020acd82230353a127cc97dfc844f6
 SHA512:
-  metadata.gz: 77dfc30ee87e63105f5cbf95e818cc4b01841f9862566fdceedad277cb22366c05c37ab14b3ac699ddb30d68a9ead35acc43114a1f882c034360d06807450677
-  data.tar.gz: 4ab621140feebc4c3f9e8897d33edf510785ea6c5cbad5cf08b672ecfde52f6146b6f1a3d695f5129acce0d3c7b64cfdae275803480178500e9c683fb7c053a8
+  metadata.gz: 5e0a93bb67bb84582e24df07ea4d51d291a53f0b48ab77ad0bd4a63b666bede4b493bbb405ad34d76602e086eafd76a455668c2a491488d83dd2c0be1060fe66
+  data.tar.gz: c616cafefeb23d78af7320beab5ddc3b8f79aafda440fd1fac889f5a5990417b31716105e8c74cab566e6c0ed6eca04a82a397ddc4242eb59caed519e52739de

data/CHANGELOG.md

@@ -5,6 +5,22 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2026-08-30
+
+### Added
+- **Target-Specific Navigation Headers & Client Hints Simulation**:
+  - Automatically generates browser-accurate Navigation Headers (`sec-fetch-dest`, `sec-fetch-mode`, `sec-fetch-site`, `upgrade-insecure-requests`) and Client Hints (`sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform`) tailored to the selected impersonation target.
+  - Firefox (`firefox*`, `ff*`), Safari / iOS (`safari*`, `ios*`), and Tor targets strictly omit Chromium-only `sec-ch-ua*` headers to avoid anti-bot fingerprint anomalies and detection.
+  - Android Chrome targets properly send `sec-ch-ua-mobile: ?1` and `sec-ch-ua-platform: "Android"`.
+  - Edge targets properly send `"Microsoft Edge"` brand headers with `"Windows"` platform.
+- **Unified Persistent CookieStore (`persist_cookies`)**:
+  - Added `:persist_cookies` request option and `persist_cookies` class-level DSL method to automatically load and persist session cookies across requests per host into `~/.http_mimic/cookies/`.
+  - Added helper methods: `HttpMimic.load_cookies(host, max_age: nil)`, `HttpMimic.save_cookies(host, cookies, ttl: nil)`, and `HttpMimic.clear_cookies!(host = nil)`.
+- **WAF Challenge Solving & Fallback Performance Optimizations**:
+  - Added single-pass WAF solver guard (`waf_solve_attempted`) to prevent redundant JavaScript solver executions across fallback attempts.
+  - Added cross-attempt cookie accumulation to preserve session state between fallback profiles.
+  - Added `HttpMimic::Waf::Detector.challenge_page?` to detect interstitial challenges disguised under HTTP 200 responses.
+
 ## [0.3.2] - 2026-08-29
 
 ### Added

data/README.md

@@ -21,7 +21,7 @@ It also includes **Webdrivers-like automatic driver management**, automatically
   - Edge & Tor support: `edge101`, `edge99`, `tor145`.
 - **HTTParty-Style API**:
   - Direct module methods: `HttpMimic.get`, `HttpMimic.post`, etc.
-  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`).
+  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`, `persist_cookies`).
   - Reusable instance client: `HttpMimic::Client.new(...)`.
 - **Zero Shell Injection Risk**:
   - Executes commands with array arguments via `Open3.capture3(*cmd_array)`.
@@ -186,6 +186,40 @@ HttpMimic.get(images.first).save('samba_og.jpg')
 
 ---
 
+## 🍪 Persistent Host CookieStore
+
+`HttpMimic` supports persistent host-based cookie caching. Once enabled, session and verification cookies are automatically saved to `~/.http_mimic/cookies/<host>.json` and reused in subsequent requests to the same host:
+
+```ruby
+# 1. Enable per-request
+response = HttpMimic.get('https://example.com/items', persist_cookies: true)
+
+# 2. Or enable in class mixin
+class Scraper
+  include HttpMimic
+  persist_cookies true
+end
+
+# 3. Manual CookieStore helpers
+HttpMimic.load_cookies('example.com')                  # => Hash of active cookies
+HttpMimic.save_cookies('example.com', { session: '123' }) # Save cookies with default TTL
+HttpMimic.clear_cookies!('example.com')                # Clear host or all cookies
+```
+
+---
+
+## 🧭 Target-Specific Navigation Headers & Client Hints
+
+`HttpMimic` automatically constructs browser-accurate HTTP navigation headers on `GET` and `HEAD` requests:
+
+- **Chrome Desktop / Mobile**: Generates `sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform`, and standard Chrome `Accept` headers.
+- **Firefox & Safari / iOS**: **Strictly omits** Chromium-only `sec-ch-ua*` headers to prevent anti-bot detection and fingerprint mismatch anomalies.
+- **Edge**: Generates Microsoft Edge brand `sec-ch-ua` headers.
+
+Can be disabled globally with `config.navigation_headers = false` or per request via `navigation_headers: false`.
+
+---
+
 ## ⚙️ Global Configuration
 
 Configure global defaults in an initializer (e.g., `config/initializers/http_mimic.rb`):
@@ -207,6 +241,16 @@ HttpMimic.configure do |config|
   config.raise_on_error          = false       # Raise exceptions on HTTP errors / non-zero exits
   config.debug                   = false       # Print debug logs
 
+  # Navigation headers & Client Hints simulation (enabled by default)
+  config.navigation_headers      = true        # Browser-accurate sec-ch-ua, sec-fetch-*, Accept headers
+
+  # Host CookieStore Persistence (disabled by default)
+  config.persist_cookies         = false                                     # Auto-persist cookies per host
+  config.cookie_store_dir        = File.expand_path('~/.http_mimic/cookies') # Directory for cookie store
+
+  # WAF challenge solving (enabled by default)
+  config.auto_solve_waf          = true        # Automatically solve detected WAF JS challenges
+
   # Webdrivers-like auto-download settings (enabled by default)
   config.auto_download           = true                                  # Auto-download missing binary
   config.driver_version          = 'v2.1.1'                              # Target release version
@@ -225,12 +269,15 @@ end
 | `:auto_fallback` | Boolean | Whether to automatically retry with alternative profile on blocked status (default `true`) |
 | `:retry_statuses`| Array | Status codes that trigger auto-fallback (default `[403, 429, 503]`) |
 | `:impersonate` | String | Target browser to mimic (e.g., `'chrome131'`, `'chrome120'`, `'firefox135'`, `'safari180'`, `'tor145'`) |
+| `:navigation_headers` | Boolean | Automatically generate browser-accurate navigation headers (`sec-ch-ua`, `sec-fetch-*`, `Accept`) (default `true`) |
+| `:solve_waf` | Boolean | Automatically detect and solve WAF challenges (e.g. Akamai) via embedded QuickJS (default `true`) |
 | `:binary` | String | Path to a custom `curl-impersonate` executable |
 | `:query` / `:params` | Hash | URL query parameters (supports nested parameters and encoding) |
 | `:headers` | Hash | Custom HTTP request headers |
 | `:json` | Hash / Array | Serialized to JSON with `Content-Type: application/json` |
 | `:body` | Hash / String | Form payload (Hash) or raw request body string |
 | `:cookies` | Hash / String | Request cookies |
+| `:persist_cookies` | Boolean | Automatically load and save cookies for the host across requests |
 | `:cookie_jar` | String | Path to save cookies (`-c`) |
 | `:cookie_file` | String | Path to read cookies (`-b`) |
 | `:timeout` | Integer / Float | Maximum execution timeout in seconds (`--max-time`) |

data/lib/http_mimic/command_builder.rb

@@ -208,6 +208,13 @@ module HttpMimic
         headers_to_send['Accept'] ||= '*/*'
         headers_to_send['Accept-Encoding'] ||= 'gzip;q=1.0,deflate;q=0.6,identity;q=0.3'
         headers_to_send['Connection'] ||= 'close'
+      else
+        # Automatic Navigation Headers for browser profiles on GET/HEAD
+        nav_headers_enabled = options.fetch(:navigation_headers, config.navigation_headers)
+        if nav_headers_enabled && %i[get head].include?(method.to_s.downcase.to_sym)
+          target = resolve_target
+          apply_navigation_headers(headers_to_send, target)
+        end
       end
 
       # Append headers
@@ -232,7 +239,20 @@ module HttpMimic
       [binary, args, stdin_data, final_url]
     end
 
-    def resolve_binary
+    def resolve_target
+      case options[:profile]
+      when :android
+        (options[:android_impersonate] || 'chrome131_android').to_s.downcase
+      when :ios
+        (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
+      when :mobile
+        (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
+      else
+        (options[:impersonate] || config.default_impersonate).to_s.downcase
+      end
+    end
+
+    def resolve_binary(target = resolve_target)
       # 0. If profile is explicitly set to :curl, use system curl
       if options[:profile] == :curl
         curl_path = find_executable('curl') || 'curl'
@@ -248,16 +268,6 @@ module HttpMimic
         raise BinaryNotFoundError, "Specified executable does not exist or is not executable: #{explicit}"
       end
 
-      target = case options[:profile]
-               when :android
-                 (options[:android_impersonate] || 'chrome131_android').to_s.downcase
-               when :ios
-                 (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
-               when :mobile
-                 (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
-               else
-                 (options[:impersonate] || config.default_impersonate).to_s.downcase
-               end
       candidate_names = TARGET_BINARY_MAP[target] || ["curl_#{target}", target]
 
       # 2. Check local installation directory (~/.http_mimic/bin)
@@ -304,6 +314,81 @@ module HttpMimic
 
     private
 
+    def apply_navigation_headers(headers_to_send, target)
+      target_str = target.to_s.downcase
+
+      if target_str.start_with?('firefox', 'ff')
+        # Firefox (Gecko): Mozilla does not send Client Hints (sec-ch-ua*)
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.5'
+
+      elsif target_str.start_with?('safari') || target_str == 'ios' || target_str == 'safari_ios' || target_str.include?('_ios')
+        # Safari / WebKit (Desktop & iOS): WebKit does not send Client Hints (sec-ch-ua*)
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      elsif target_str.start_with?('tor')
+        # Tor Browser: No sec-ch-ua headers
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.5'
+
+      elsif target_str.start_with?('edge')
+        # Edge (Chromium): Microsoft Edge sec-ch-ua
+        ver = target_str[/\d+/] || '101'
+        headers_to_send['sec-ch-ua'] ||= "\"Microsoft Edge\";v=\"#{ver}\", \"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?0'
+        headers_to_send['sec-ch-ua-platform'] ||= '"Windows"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      elsif target_str.include?('android')
+        # Android Chrome
+        ver = target_str[/\d+/] || '131'
+        headers_to_send['sec-ch-ua'] ||= "\"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\", \"Google Chrome\";v=\"#{ver}\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?1'
+        headers_to_send['sec-ch-ua-platform'] ||= '"Android"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      else
+        # Chrome Desktop (Default)
+        ver = target_str[/\d+/] || '131'
+        headers_to_send['sec-ch-ua'] ||= "\"Google Chrome\";v=\"#{ver}\", \"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?0'
+        headers_to_send['sec-ch-ua-platform'] ||= '"macOS"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+      end
+    end
+
     def find_in_download_dir(candidates)
       target_dir = File.expand_path(config.install_dir)
       return nil unless Dir.exist?(target_dir)

data/lib/http_mimic/configuration.rb

@@ -26,6 +26,20 @@ module HttpMimic
     attr_accessor :install_dir
     attr_accessor :github_repo
 
+    # QuickJS standalone binary management settings
+    attr_accessor :qjs_version
+    attr_accessor :qjs_github_repo
+
+    # WAF & JS challenge solving settings
+    attr_accessor :auto_solve_waf
+
+    # Shared Host-based CookieStore settings
+    attr_accessor :persist_cookies
+    attr_accessor :cookie_store_dir
+
+    # Modern Navigation Headers simulation
+    attr_accessor :navigation_headers
+
     def initialize
       @default_impersonate         = 'chrome131'
       @default_mobile_impersonate  = 'safari180_ios'
@@ -45,10 +59,24 @@ module HttpMimic
       @retry_statuses              = [403, 429, 503]
 
       # Enable automatic downloading of curl-impersonate driver (lexiforest) by default
-      @auto_download            = true
-      @driver_version           = 'v2.1.1'
-      @install_dir              = File.expand_path('~/.http_mimic/bin')
-      @github_repo              = 'lexiforest/curl-impersonate'
+      @auto_download               = true
+      @driver_version              = 'v2.1.1'
+      @install_dir                 = File.expand_path('~/.http_mimic/bin')
+      @github_repo                 = 'lexiforest/curl-impersonate'
+
+      # QuickJS defaults
+      @qjs_version                 = 'v0.16.2'
+      @qjs_github_repo             = 'quickjs-ng/quickjs'
+
+      # WAF challenge solving defaults
+      @auto_solve_waf              = true
+
+      # Shared Host CookieStore defaults
+      @persist_cookies             = false
+      @cookie_store_dir            = File.expand_path('~/.http_mimic/cookies')
+
+      # Navigation Headers defaults
+      @navigation_headers          = true
     end
   end
 end

data/lib/http_mimic/cookie_store.rb

@@ -0,0 +1,118 @@
+# frozen_string_literal: true
+
+require 'fileutils'
+require 'json'
+require 'uri'
+
+module HttpMimic
+  class CookieStore
+    DEFAULT_TTL = 7200 # 2 hours
+
+    class << self
+      def store_dir
+        File.expand_path(HttpMimic.configuration.cookie_store_dir || '~/.http_mimic/cookies')
+      end
+
+      def cookie_file_for(host)
+        clean_host = sanitize_host(host)
+        File.join(store_dir, "#{clean_host}.json")
+      end
+
+      # Load cookies for a given host
+      # Returns a Hash of cookies if valid and not expired, or empty Hash
+      def load(host, max_age: DEFAULT_TTL)
+        file = cookie_file_for(host)
+        return {} unless File.file?(file)
+
+        data = nil
+        File.open(file, File::RDONLY) do |f|
+          f.flock(File::LOCK_SH)
+          data = JSON.parse(f.read) rescue nil
+        end
+
+        return {} unless data.is_a?(Hash)
+
+        updated_at = data['updated_at'].to_i
+        if !max_age.nil?
+          return {} if (Time.now.to_i - updated_at) > max_age
+        end
+
+        data['cookies'] || {}
+      rescue StandardError => e
+        if HttpMimic.configuration.debug
+          puts "[HttpMimic::CookieStore] Error loading cookies for #{host}: #{e.message}"
+        end
+        {}
+      end
+
+      # Save cookies for a given host with atomic locking
+      def save(host, cookies, ttl: DEFAULT_TTL)
+        return if cookies.nil? || cookies.empty?
+
+        clean_host = sanitize_host(host)
+        dir = store_dir
+        FileUtils.mkdir_p(dir)
+
+        file = cookie_file_for(clean_host)
+        existing_cookies = load(clean_host, max_age: ttl) || {}
+
+        new_cookies = cookies.is_a?(Cookies) ? cookies.to_h : cookies.dup
+        merged = existing_cookies.merge(stringify_keys(new_cookies))
+
+        payload = {
+          'host' => clean_host,
+          'updated_at' => Time.now.to_i,
+          'ttl' => ttl,
+          'cookies' => merged
+        }
+
+        # Thread & Process safe atomic write
+        tmp_file = "#{file}.tmp.#{Process.pid}_#{Time.now.to_f}"
+        File.open(tmp_file, File::RDWR | File::CREAT, 0644) do |f|
+          f.flock(File::LOCK_EX)
+          f.write(JSON.pretty_generate(payload))
+          f.flush
+        end
+        File.rename(tmp_file, file)
+      rescue StandardError => e
+        FileUtils.rm_f(tmp_file) if tmp_file
+        if HttpMimic.configuration.debug
+          puts "[HttpMimic::CookieStore] Error saving cookies for #{host}: #{e.message}"
+        end
+      end
+
+      # Clear stored cookies for a host or all hosts
+      def clear(host = nil)
+        if host
+          file = cookie_file_for(host)
+          FileUtils.rm_f(file)
+        else
+          FileUtils.rm_rf(store_dir)
+        end
+      end
+
+      # Extract host from URL or host string
+      def extract_host(url_or_host)
+        str = url_or_host.to_s.strip
+        if str =~ %r{^https?://}i
+          URI.parse(str).host
+        else
+          str.split(':').first
+        end
+      rescue StandardError
+        str
+      end
+
+      private
+
+      def sanitize_host(host)
+        h = extract_host(host) || 'default'
+        h.gsub(/[^a-zA-Z0-9.\-_]/, '_')
+      end
+
+      def stringify_keys(hash)
+        hash.each_with_object({}) { |(k, v), h| h[k.to_s] = v.to_s }
+      end
+    end
+  end
+end

data/lib/http_mimic/cookies.rb

@@ -47,6 +47,44 @@ module HttpMimic
       @store.values
     end
 
+    def empty?
+      @store.empty?
+    end
+
+    def size
+      @store.size
+    end
+    alias length size
+
+    def delete(name)
+      @cookie_items.delete(name.to_s)
+      @store.delete(name.to_s)
+    end
+
+    def clear
+      @cookie_items.clear
+      @store.clear
+    end
+
+    def merge(other)
+      dup_cookies = self.class.new(@store)
+      if other.is_a?(Cookies)
+        other.each { |k, v| dup_cookies[k] = v }
+      elsif other.is_a?(Hash)
+        other.each { |k, v| dup_cookies[k] = v }
+      end
+      dup_cookies
+    end
+
+    def merge!(other)
+      if other.is_a?(Cookies)
+        other.each { |k, v| self[k] = v }
+      elsif other.is_a?(Hash)
+        other.each { |k, v| self[k] = v }
+      end
+      self
+    end
+
     def to_h
       @store.dup
     end

data/lib/http_mimic/downloader.rb

@@ -13,6 +13,9 @@ module HttpMimic
     DEFAULT_GITHUB_REPO = 'lexiforest/curl-impersonate'
     DEFAULT_VERSION     = 'v2.1.1'
 
+    DEFAULT_QJS_REPO    = 'quickjs-ng/quickjs'
+    DEFAULT_QJS_VERSION = 'v0.16.2'
+
     class DownloadError < HttpMimic::Error; end
     class UnsupportedPlatformError < HttpMimic::Error; end
 
@@ -63,6 +66,105 @@ module HttpMimic
         true
       end
 
+      def download_qjs!(version: nil, install_dir: nil, repo: nil, force: false)
+        target_version = normalize_version(version || HttpMimic.configuration.qjs_version || DEFAULT_QJS_VERSION)
+        target_dir     = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        target_repo    = repo || HttpMimic.configuration.qjs_github_repo || DEFAULT_QJS_REPO
+
+        FileUtils.mkdir_p(target_dir)
+
+        dest_binary = File.join(target_dir, binary_name_for_platform('qjs'))
+
+        if !force && qjs_installed?(version: target_version, install_dir: target_dir)
+          log_info("qjs #{target_version} is already installed in #{target_dir}")
+          return dest_binary
+        end
+
+        asset_name = qjs_platform_asset
+        download_url = "https://github.com/#{target_repo}/releases/download/#{target_version}/#{asset_name}"
+
+        log_info("Downloading QuickJS (#{target_version}) [#{asset_name}]...")
+        binary_data = fetch_binary(download_url)
+
+        File.binwrite(dest_binary, binary_data)
+        File.chmod(0755, dest_binary)
+
+        File.write(qjs_version_file_path(target_dir), target_version)
+
+        log_info("QuickJS #{target_version} installation complete! (#{dest_binary})")
+        dest_binary
+      end
+
+      def qjs_installed?(version: nil, install_dir: nil)
+        target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        qjs_bin = File.join(target_dir, binary_name_for_platform('qjs'))
+        return false unless File.file?(qjs_bin) && File.executable?(qjs_bin)
+
+        if version
+          target_version = normalize_version(version)
+          v_file = qjs_version_file_path(target_dir)
+          return false unless File.file?(v_file)
+          return File.read(v_file).strip == target_version
+        end
+
+        true
+      end
+
+      def qjs_path(install_dir: nil)
+        target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        candidate = File.join(target_dir, binary_name_for_platform('qjs'))
+        return candidate if File.file?(candidate) && File.executable?(candidate)
+
+        # Fallback to system PATH
+        sys_qjs = `which qjs 2>/dev/null`.strip
+        return sys_qjs if !sys_qjs.empty? && File.executable?(sys_qjs)
+
+        nil
+      end
+
+      def qjs_platform_asset
+        os  = host_os
+        cpu = host_cpu
+
+        case os
+        when :macos
+          case cpu
+          when :arm64 then 'qjs-darwin-arm64'
+          when :x86_64 then 'qjs-darwin-x86_64'
+          else
+            raise UnsupportedPlatformError, "Unsupported macOS CPU architecture for QuickJS: #{cpu}"
+          end
+        when :linux
+          case cpu
+          when :x86_64
+            'qjs-linux-x86_64'
+          when :aarch64, :arm64
+            'qjs-linux-aarch64'
+          when :arm
+            'qjs-linux-armv7'
+          when :i386, :i686
+            'qjs-linux-x86'
+          when :riscv64
+            'qjs-linux-riscv64'
+          else
+            raise UnsupportedPlatformError, "Unsupported Linux CPU architecture for QuickJS: #{cpu}"
+          end
+        when :windows
+          case cpu
+          when :x86_64 then 'qjs-windows-x86_64.exe'
+          when :i386, :i686 then 'qjs-windows-x86.exe'
+          else
+            raise UnsupportedPlatformError, "Unsupported Windows CPU architecture for QuickJS: #{cpu}"
+          end
+        else
+          raise UnsupportedPlatformError, "Unsupported operating system for QuickJS: #{RbConfig::CONFIG['host_os']}"
+        end
+      end
+
+      def qjs_version_file_path(dir)
+        File.join(dir, '.qjs_version')
+      end
+
       def binary_path(name, install_dir: nil)
         target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
         candidate = File.join(target_dir, binary_name_for_platform(name))

data/lib/http_mimic/exceptions.rb

@@ -31,4 +31,19 @@ module HttpMimic
 
   # Raised when response parsing fails
   class ResponseParseError < Error; end
+
+  # Raised when JavaScript execution fails in JSRuntime
+  class JSError < Error
+    attr_reader :stderr, :exit_code
+
+    def initialize(message, stderr: nil, exit_code: nil)
+      super(message)
+      @stderr = stderr
+      @exit_code = exit_code
+    end
+  end
+
+  # Raised when JavaScript execution times out
+  class JSTimeoutError < JSError; end
 end
+

data/lib/http_mimic/js_runtime.rb

@@ -0,0 +1,78 @@
+# frozen_string_literal: true
+
+require 'open3'
+require 'json'
+require 'timeout'
+
+module HttpMimic
+  class JSRuntime
+    STDIN_RUNNER = <<~JS
+      const input = std.in.readAsString();
+      const result = eval(input);
+      if (result !== undefined) {
+        if (typeof result === "object" && result !== null) {
+          console.log(JSON.stringify(result));
+        } else {
+          console.log(result);
+        }
+      }
+    JS
+
+    class << self
+      def eval(js_code, timeout: nil, install_dir: nil, auto_download: nil)
+        ensure_qjs_installed!(install_dir: install_dir, auto_download: auto_download)
+
+        qjs_bin = Downloader.qjs_path(install_dir: install_dir)
+        raise BinaryNotFoundError, "QuickJS binary 'qjs' not found. Run HttpMimic.download_qjs! to install." unless qjs_bin
+
+        cmd = [qjs_bin, '--std', '-e', STDIN_RUNNER]
+        effective_timeout = timeout || HttpMimic.configuration.default_timeout
+
+        stdout = nil
+        stderr = nil
+        status = nil
+
+        begin
+          if effective_timeout && effective_timeout > 0
+            Timeout.timeout(effective_timeout) do
+              stdout, stderr, status = Open3.capture3(*cmd, stdin_data: js_code.to_s)
+            end
+          else
+            stdout, stderr, status = Open3.capture3(*cmd, stdin_data: js_code.to_s)
+          end
+        rescue Timeout::Error
+          raise JSTimeoutError.new("JavaScript execution timed out after #{effective_timeout}s")
+        end
+
+        unless status && status.success?
+          raise JSError.new(
+            "JavaScript execution failed: #{stderr.to_s.strip}",
+            stderr: stderr,
+            exit_code: status ? status.exitstatus : nil
+          )
+        end
+
+        stdout.to_s.strip
+      end
+
+      def eval_json(js_code, timeout: nil, install_dir: nil, auto_download: nil)
+        output = eval(js_code, timeout: timeout, install_dir: install_dir, auto_download: auto_download)
+        return nil if output.nil? || output.empty?
+
+        JSON.parse(output)
+      rescue JSON::ParserError => e
+        raise ResponseParseError, "Failed to parse JavaScript JSON output: #{e.message} (Output: #{output.inspect})"
+      end
+
+      private
+
+      def ensure_qjs_installed!(install_dir: nil, auto_download: nil)
+        should_download = auto_download.nil? ? HttpMimic.configuration.auto_download : auto_download
+        return unless should_download
+        return if Downloader.qjs_installed?(install_dir: install_dir)
+
+        Downloader.download_qjs!(install_dir: install_dir)
+      end
+    end
+  end
+end

data/lib/http_mimic/module_methods.rb

@@ -41,6 +41,11 @@ module HttpMimic
       default_options[:cookies].merge!(c)
     end
 
+    def persist_cookies(enabled = nil)
+      return default_options[:persist_cookies] if enabled.nil?
+      default_options[:persist_cookies] = enabled
+    end
+
     def mode(m = nil)
       return default_options[:mode] if m.nil?
       default_options[:mode] = m

data/lib/http_mimic/request.rb

@@ -18,8 +18,22 @@ module HttpMimic
       auto_fallback = options.fetch(:auto_fallback, config.auto_fallback)
       retry_statuses = options[:retry_statuses] || config.retry_statuses || [403, 429, 503]
 
+      # Persistent CookieStore integration
+      should_persist_cookie = options.fetch(:persist_cookies, config.persist_cookies)
+      host = CookieStore.extract_host(url)
+
+      if should_persist_cookie && host
+        stored_cookies = CookieStore.load(host)
+        if stored_cookies && !stored_cookies.empty?
+          log_debug("[HttpMimic::CookieStore] Loaded #{stored_cookies.size} persistent cookies for #{host}")
+          user_cookies = options[:cookies] ? (options[:cookies].is_a?(Hash) ? options[:cookies] : options[:cookies].to_h) : {}
+          options[:cookies] = stored_cookies.merge(user_cookies)
+        end
+      end
+
       attempts = []
       profiles_to_try = determine_profiles(mode, auto_fallback)
+      waf_solve_attempted = false
 
       response = nil
       final_status = nil
@@ -66,7 +80,32 @@ module HttpMimic
         final_stderr = stderr
         final_command = full_command
 
-        is_blocked = (status.exitstatus != 0) || retry_statuses.include?(response.code)
+        # Forward any received cookies to subsequent attempts
+        if response.cookies && !response.cookies.empty?
+          existing_cookies = options[:cookies] ? (options[:cookies].is_a?(Hash) ? options[:cookies] : options[:cookies].to_h) : {}
+          options[:cookies] = response.cookies.to_h.merge(existing_cookies)
+        end
+
+        auto_solve_waf = options.fetch(:solve_waf, config.auto_solve_waf)
+        is_blocked = (status.exitstatus != 0) || retry_statuses.include?(response.code) || Waf::Detector.challenge_page?(response)
+
+        # Only attempt WAF resolution once per request to avoid unnecessary latency on subsequent fallbacks
+        if is_blocked && auto_solve_waf && !waf_solve_attempted && method.to_s.upcase == 'GET'
+          waf_type = Waf::Detector.detect(response)
+          if waf_type
+            waf_solve_attempted = true
+            log_debug("[HttpMimic] Detected #{waf_type.to_s.capitalize} WAF challenge. Attempting to solve with QuickJS...")
+            solved_resp = Waf.solve(url, response, current_opts)
+            if solved_resp
+              response = solved_resp
+              is_blocked = (response.code != 0 && retry_statuses.include?(response.code))
+              if response.cookies && !response.cookies.empty?
+                options[:cookies] = (options[:cookies] || {}).merge(response.cookies.to_h)
+              end
+            end
+          end
+        end
+
         if !is_blocked || (index == profiles_to_try.size - 1)
           break
         end
@@ -74,6 +113,12 @@ module HttpMimic
         log_debug("[HttpMimic] Attempt #{index + 1} with #{profile} resulted in status #{response.code}. Triggering smart fallback to next profile...")
       end
 
+      # Persist cookies back to store if enabled
+      if should_persist_cookie && host && response && response.cookies && !response.cookies.empty?
+        CookieStore.save(host, response.cookies)
+        log_debug("[HttpMimic::CookieStore] Saved #{response.cookies.size} cookies for #{host}")
+      end
+
       handle_errors(final_status, final_stderr, final_command, response)
       response
     end

data/lib/http_mimic/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module HttpMimic
-  VERSION = "0.3.2"
+  VERSION = "0.4.0"
 end

data/lib/http_mimic/waf/akamai_solver.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+require 'uri'
+require 'json'
+
+module HttpMimic
+  module Waf
+    class AkamaiSolver
+      CONTEXT_JS_PATH = File.expand_path('browser_context.js', __dir__)
+
+      class << self
+        def solve(target_url, initial_response, options = {})
+          cookies = initial_response.cookies.dup
+          impersonate = options[:impersonate] || HttpMimic.configuration.default_impersonate || 'chrome131'
+
+          sensor_script_url = extract_sensor_script_url(target_url, initial_response.body)
+          return nil unless sensor_script_url
+
+          # 1. Fetch the dynamic sensor script
+          script_resp = HttpMimic.get(
+            sensor_script_url,
+            impersonate: impersonate,
+            cookies: cookies,
+            headers: { 'Referer' => target_url },
+            auto_fallback: false,
+            solve_waf: false
+          )
+          return nil unless script_resp.success?
+
+          sensor_js = script_resp.body
+          context_js = File.read(CONTEXT_JS_PATH)
+          cookie_str = cookies.to_cookie_string
+
+          doc_title = initial_response.title.to_s
+          user_agent = options[:user_agent]
+          referer = (options[:headers] || {})['Referer'] || (options[:headers] || {})['referer'] || target_url
+
+          # 2. Run virtual browser simulation inside QuickJS
+          driver_script = <<~JS
+            globalThis.__TARGET_URL__ = #{target_url.to_json};
+            globalThis.__DOCUMENT_TITLE__ = #{doc_title.to_json};
+            globalThis.__INITIAL_COOKIES__ = #{cookie_str.to_json};
+            globalThis.__REFERRER__ = #{referer.to_json};
+            #{user_agent ? "globalThis.__USER_AGENT__ = #{user_agent.to_json};" : ""}
+
+            #{context_js}
+            #{sensor_js}
+
+            if (typeof globalThis.__simulateHumanInteractions === "function") {
+              globalThis.__simulateHumanInteractions();
+            }
+            if (typeof globalThis.__drainEventLoop === "function") {
+              globalThis.__drainEventLoop(100);
+            }
+
+            JSON.stringify({
+              sensor_posts: globalThis.__sensor_posts
+            });
+          JS
+
+          result = JSRuntime.eval_json(driver_script)
+          sensor_posts = result && result['sensor_posts']
+          return nil if sensor_posts.nil? || sensor_posts.empty?
+
+          # 3. Post sensor data to Akamai endpoint
+          updated_cookies = cookies.dup
+          sensor_posts.each do |post|
+            post_body = post['body']
+            next if post_body.nil? || post_body.empty?
+
+            post_resp = HttpMimic.post(
+              sensor_script_url,
+              impersonate: impersonate,
+              cookies: updated_cookies,
+              headers: {
+                'Content-Type' => 'text/plain;charset=UTF-8',
+                'Referer' => target_url,
+                'Origin' => URI.parse(target_url).tap { |u| u.path = ''; u.query = nil }.to_s,
+                'Accept' => '*/*'
+              },
+              body: post_body,
+              auto_fallback: false,
+              solve_waf: false
+            )
+
+            post_resp.cookies.each { |k, v| updated_cookies[k] = v }
+          end
+
+          # 4. Retry original target URL with the updated cookies
+          retry_headers = (options[:headers] || {}).merge('Referer' => target_url)
+          HttpMimic.get(
+            target_url,
+            options.merge(
+              cookies: updated_cookies,
+              headers: retry_headers,
+              auto_fallback: false,
+              solve_waf: false
+            )
+          )
+        rescue StandardError => e
+          if HttpMimic.configuration.debug
+            puts "[HttpMimic::Waf::AkamaiSolver] Error solving Akamai challenge: #{e.message}"
+          end
+          nil
+        end
+
+        def extract_sensor_script_url(target_url, html)
+          return nil if html.nil? || html.empty?
+
+          match = html.match(/<script[^>]*src=["\x27]([^"\x27]*\/[a-zA-Z0-9_-]{10,}\?[a-zA-Z0-9_=-]+)["\x27]/i)
+          match ||= html.match(/<script[^>]*src=["\x27]([^"\x27]*akam[^"\x27]*)["\x27]/i)
+          match ||= html.match(/<script[^>]*src=["\x27]([^"\x27]*\/[a-zA-Z0-9_\-\/]+\?v=[a-zA-Z0-9_-]+)["\x27]/i)
+
+          return nil unless match
+
+          URI.join(target_url, match[1]).to_s
+        rescue StandardError
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/http_mimic/waf/detector.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+module HttpMimic
+  module Waf
+    class Detector
+      class << self
+        def detect(response)
+          return nil unless response
+
+          if akamai?(response)
+            :akamai
+          elsif cloudflare?(response)
+            :cloudflare
+          elsif datadome?(response)
+            :datadome
+          else
+            nil
+          end
+        end
+
+        def challenge_page?(response)
+          return false unless response
+          body = response.body.to_s
+          return true if body.include?('sec-if-cpt-container') || body.include?('sec-bc-button-parent')
+          return true if body.include?('challenges.cloudflare.com') || body.include?('cf-turnstile')
+          return true if body.include?('datadome.captcha') || body.include?('geo.captcha-delivery.com')
+          false
+        end
+
+        def akamai?(response)
+          return true if response.headers['set-cookie']&.include?('_abck=')
+          return true if response.cookies.key?('_abck') || response.cookies.key?('bm_sz') || response.cookies.key?('ak_bmsc')
+          return true if response.headers['x-akamai-transformed'] || response.headers['x-reference-error']
+
+          body = response.body.to_s
+          return true if body.include?('Reference Error:') && body.include?('Akamai')
+          return true if body.include?('bmak') || body.include?('sensor_data')
+          return true if body.include?('sec-if-cpt-container') || body.include?('sec-bc-button-parent')
+          return true if body =~ /<script[^>]*src=["\x27]([^"\x27]*\/[a-zA-Z0-9_-]{10,}\?[a-zA-Z0-9_=-]+)["\x27]/
+
+          false
+        end
+
+        def cloudflare?(response)
+          return true if response.headers['server']&.downcase&.include?('cloudflare')
+          return true if response.headers['cf-ray'] || response.cookies.key?('cf_clearance')
+          return true if response.body.to_s.include?('challenges.cloudflare.com')
+
+          false
+        end
+
+        def datadome?(response)
+          return true if response.headers['x-datadome'] || response.cookies.key?('datadome')
+          return true if response.headers['server']&.downcase&.include?('datadome')
+
+          false
+        end
+      end
+    end
+  end
+end

data/lib/http_mimic/waf.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require 'http_mimic/waf/detector'
+require 'http_mimic/waf/akamai_solver'
+
+module HttpMimic
+  module Waf
+    class << self
+      def solve(url, response, options = {})
+        waf_type = Detector.detect(response)
+        return nil unless waf_type
+
+        case waf_type
+        when :akamai
+          AkamaiSolver.solve(url, response, options)
+        else
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/http_mimic.rb

@@ -16,6 +16,9 @@ require 'http_mimic/command_builder'
 require 'http_mimic/request'
 require 'http_mimic/client'
 require 'http_mimic/module_methods'
+require 'http_mimic/js_runtime'
+require 'http_mimic/waf'
+require 'http_mimic/cookie_store'
 
 module HttpMimic
   extend ModuleMethods
@@ -43,6 +46,54 @@ module HttpMimic
       Downloader.installed?(version: version)
     end
 
+    # Manually download and install QuickJS binary driver
+    def download_qjs!(version: nil, force: false)
+      Downloader.download_qjs!(version: version, force: force)
+    end
+
+    # Check if QuickJS driver is installed locally
+    def qjs_installed?(version: nil)
+      Downloader.qjs_installed?(version: version)
+    end
+
+    # Return local path to QuickJS binary
+    def qjs_path
+      Downloader.qjs_path
+    end
+
+    # Evaluate JavaScript code using QuickJS
+    def eval_js(code, options = {})
+      JSRuntime.eval(code, timeout: options[:timeout], install_dir: options[:install_dir], auto_download: options[:auto_download])
+    end
+
+    # Evaluate JavaScript and auto-parse JSON result
+    def eval_js_json(code, options = {})
+      JSRuntime.eval_json(code, timeout: options[:timeout], install_dir: options[:install_dir], auto_download: options[:auto_download])
+    end
+
+    # Automatically detect WAF type from response
+    def detect_waf(response)
+      Waf::Detector.detect(response)
+    end
+
+    # Attempt to solve WAF challenge for a blocked response
+    def solve_waf(url, response, options = {})
+      Waf.solve(url, response, options)
+    end
+
+    # Persistent Host CookieStore helpers
+    def load_cookies(host, max_age: nil)
+      CookieStore.load(host, max_age: max_age)
+    end
+
+    def save_cookies(host, cookies, ttl: nil)
+      CookieStore.save(host, cookies, ttl: ttl || CookieStore::DEFAULT_TTL)
+    end
+
+    def clear_cookies!(host = nil)
+      CookieStore.clear(host)
+    end
+
     def included(base)
       base.extend(ModuleMethods)
       base.send(:include, InstanceMethods)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: http_mimic
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.4.0
 platform: ruby
 authors:
 - anxgang
@@ -65,15 +65,20 @@ files:
 - lib/http_mimic/client.rb
 - lib/http_mimic/command_builder.rb
 - lib/http_mimic/configuration.rb
+- lib/http_mimic/cookie_store.rb
 - lib/http_mimic/cookies.rb
 - lib/http_mimic/downloader.rb
 - lib/http_mimic/exceptions.rb
 - lib/http_mimic/headers.rb
+- lib/http_mimic/js_runtime.rb
 - lib/http_mimic/module_methods.rb
 - lib/http_mimic/request.rb
 - lib/http_mimic/response.rb
 - lib/http_mimic/response_parser.rb
 - lib/http_mimic/version.rb
+- lib/http_mimic/waf.rb
+- lib/http_mimic/waf/akamai_solver.rb
+- lib/http_mimic/waf/detector.rb
 homepage: https://github.com/anxgang/http_mimic
 licenses:
 - MIT