http_mimic 0.3.1 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0b09482a3e618dd6390934e85d83de762cffaa062d5610d606d77bdabfb0ff11
-  data.tar.gz: 55c4dfeee202c24898a1ad5a12bc5a58df9c5a1e0481b7d08be0fc0c4a909e32
+  metadata.gz: 9b0a9fff1b74c98ea24c294a491b00ba57e270d24e6c0bb7a669e6d2db264d8a
+  data.tar.gz: e11d3ae615b92abcfedda8547120290d82020acd82230353a127cc97dfc844f6
 SHA512:
-  metadata.gz: 1b4a75f41696e4b6a89c73096640ce5f9b3963dff9262aea51ead8271022868397e7c413edfc724cd7c0cbb439a2663bf55f5ea28cd060e491cff0445e88fea7
-  data.tar.gz: cf2f82e990474a02e00aebe5d5b0b0ca019a994266279cd84f7fd4dd966e9260b488e1915f2b7eccdf7452e50a838489d4709181ec195bebbc773e6aff62d6d8
+  metadata.gz: 5e0a93bb67bb84582e24df07ea4d51d291a53f0b48ab77ad0bd4a63b666bede4b493bbb405ad34d76602e086eafd76a455668c2a491488d83dd2c0be1060fe66
+  data.tar.gz: c616cafefeb23d78af7320beab5ddc3b8f79aafda440fd1fac889f5a5990417b31716105e8c74cab566e6c0ed6eca04a82a397ddc4242eb59caed519e52739de

data/CHANGELOG.md

@@ -5,6 +5,35 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.4.0] - 2026-08-30
+
+### Added
+- **Target-Specific Navigation Headers & Client Hints Simulation**:
+  - Automatically generates browser-accurate Navigation Headers (`sec-fetch-dest`, `sec-fetch-mode`, `sec-fetch-site`, `upgrade-insecure-requests`) and Client Hints (`sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform`) tailored to the selected impersonation target.
+  - Firefox (`firefox*`, `ff*`), Safari / iOS (`safari*`, `ios*`), and Tor targets strictly omit Chromium-only `sec-ch-ua*` headers to avoid anti-bot fingerprint anomalies and detection.
+  - Android Chrome targets properly send `sec-ch-ua-mobile: ?1` and `sec-ch-ua-platform: "Android"`.
+  - Edge targets properly send `"Microsoft Edge"` brand headers with `"Windows"` platform.
+- **Unified Persistent CookieStore (`persist_cookies`)**:
+  - Added `:persist_cookies` request option and `persist_cookies` class-level DSL method to automatically load and persist session cookies across requests per host into `~/.http_mimic/cookies/`.
+  - Added helper methods: `HttpMimic.load_cookies(host, max_age: nil)`, `HttpMimic.save_cookies(host, cookies, ttl: nil)`, and `HttpMimic.clear_cookies!(host = nil)`.
+- **WAF Challenge Solving & Fallback Performance Optimizations**:
+  - Added single-pass WAF solver guard (`waf_solve_attempted`) to prevent redundant JavaScript solver executions across fallback attempts.
+  - Added cross-attempt cookie accumulation to preserve session state between fallback profiles.
+  - Added `HttpMimic::Waf::Detector.challenge_page?` to detect interstitial challenges disguised under HTTP 200 responses.
+
+## [0.3.2] - 2026-08-29
+
+### Added
+- **Mobile Impersonation & Multi-Stage Auto-Fallback**:
+  - Added support for mobile browser targets: Android Chrome (`chrome131_android`, `chrome99_android`, alias `android`) and iOS Safari (`safari260_ios`, `safari180_ios`, `safari184_ios`, `safari172_ios`, alias `ios`, `mobile`).
+  - Enhanced `:auto` adaptive mode with automatic fallback cascade `[:impersonate, :android, :ios, :curl]` to bypass modern anti-bot systems (e.g. Akamai Bot Manager on Adidas HK) without requiring JavaScript telemetry solving.
+  - Added execution modes: `:mobile_first`, `:android_first`, `:ios_first`, `:mobile_only`, `:android_only`, `:ios_only`.
+- **Binary-Safe Response Parsing & File Saving**:
+  - Fixed binary handling in `ResponseParser#split_headers_and_body` to prevent `ArgumentError: invalid byte sequence in UTF-8` on images, PDFs, compressed files, and binary streams.
+  - Added `response.binary?` and `response.save_to_file(path)` / `response.save(path)` helpers.
+- **Built-in HTML / Scraping Helpers**:
+  - Added `response.title`, `response.og_image`, `response.meta_description`, and `response.extract_images(base_url: nil)`.
+
 ## [0.3.1] - 2026-08-25
 
 ### Added

data/README.md

@@ -21,7 +21,7 @@ It also includes **Webdrivers-like automatic driver management**, automatically
   - Edge & Tor support: `edge101`, `edge99`, `tor145`.
 - **HTTParty-Style API**:
   - Direct module methods: `HttpMimic.get`, `HttpMimic.post`, etc.
-  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`).
+  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`, `persist_cookies`).
   - Reusable instance client: `HttpMimic::Client.new(...)`.
 - **Zero Shell Injection Risk**:
   - Executes commands with array arguments via `Open3.capture3(*cmd_array)`.
@@ -161,28 +161,65 @@ response = client.post('/v1/users', json: { username: 'bob' })
 
 ## 🧠 Smart Adaptive Modes (Zero-Configuration Scraping)
 
-`HttpMimic` provides built-in multi-strategy orchestration so you can fetch protected websites without worrying about which specific WAF (Cloudflare, Akamai, DataDome) protects them:
+`HttpMimic` provides built-in multi-strategy orchestration so you can fetch protected websites without worrying about which specific WAF (Akamai Botman, Cloudflare Turnstile, DataDome, Kasada) protects them:
 
-- **`:auto` (Default & Recommended)**: Tries `curl-impersonate` (Chrome 131) first. If blocked by WAFs like Akamai with `403`/`429`/`503` (which require JS telemetry for browsers but allow standard server clients), it **automatically and seamlessly retries with standard curl + server headers**, directly returning `200 OK`.
-- **`:impersonate_first`**: Prefers `curl-impersonate` and automatically falls back to standard `curl` if blocked.
-- **`:curl_first`**: Prefers standard `curl` and automatically upgrades to `curl-impersonate` if blocked.
-- **`:impersonate_only`**: Strictly uses `curl-impersonate` (no retry/fallback).
-- **`:curl_only`**: Strictly uses standard `curl` (no retry/fallback).
+- **`:auto` (Default & Recommended)**: Tries desktop `curl-impersonate` (`chrome131`) first. If blocked by WAFs like Akamai with `403`/`429`/`503` (which require JS telemetry for desktop browsers), it **automatically cascades through mobile profiles (`android`, `ios`) and server `curl` headers**, reliably retrieving `200 OK`.
+- **`:mobile_first` / `:android_first` / `:ios_first`**: Prefers mobile TLS/HTTP2 fingerprints and falls back to desktop or curl if blocked.
+- **`:impersonate_first`**: Prefers desktop `curl-impersonate` and automatically falls back to mobile and curl if blocked.
+- **`:curl_first`**: Prefers standard server `curl` and automatically upgrades to impersonate profiles if blocked.
+- **`:impersonate_only` / `:mobile_only` / `:curl_only`**: Strictly uses the chosen profile (no fallback).
 
 ```ruby
-# 1. No-Brain Auto Mode (Works automatically for both Cloudflare & Akamai):
-response = HttpMimic.get('https://www.asics.com/us/en-us/gt-2000-15/p/ANA_1011C235-750.html')
+# 1. No-Brain Auto Mode (Works automatically for Cloudflare, Akamai, etc.):
+response = HttpMimic.get('https://www.adidas.com.hk/en/KI8139.html')
 puts response.code                 # => 200
-puts response.mode_used            # => :curl
+puts response.mode_used            # => :android
 puts response.fallback_triggered?  # => true
 
-# 2. Per-request mode override:
-response = HttpMimic.get(url, mode: :curl_first)
-response = HttpMimic.get(url, mode: :impersonate_only)
+# 2. Extract images and metadata with built-in helpers:
+puts response.title                # => "SAMBA OG SHOES - Brown | adidas Hong Kong"
+images = response.extract_images   # => ["https://assets.adidas.com/images/.../KI8139_01_00_standard.jpg", ...]
+
+# 3. Fetch and save binary images directly:
+HttpMimic.get(images.first).save('samba_og.jpg')
+```
+
+---
+
+## 🍪 Persistent Host CookieStore
+
+`HttpMimic` supports persistent host-based cookie caching. Once enabled, session and verification cookies are automatically saved to `~/.http_mimic/cookies/<host>.json` and reused in subsequent requests to the same host:
+
+```ruby
+# 1. Enable per-request
+response = HttpMimic.get('https://example.com/items', persist_cookies: true)
+
+# 2. Or enable in class mixin
+class Scraper
+  include HttpMimic
+  persist_cookies true
+end
+
+# 3. Manual CookieStore helpers
+HttpMimic.load_cookies('example.com')                  # => Hash of active cookies
+HttpMimic.save_cookies('example.com', { session: '123' }) # Save cookies with default TTL
+HttpMimic.clear_cookies!('example.com')                # Clear host or all cookies
 ```
 
 ---
 
+## 🧭 Target-Specific Navigation Headers & Client Hints
+
+`HttpMimic` automatically constructs browser-accurate HTTP navigation headers on `GET` and `HEAD` requests:
+
+- **Chrome Desktop / Mobile**: Generates `sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform`, and standard Chrome `Accept` headers.
+- **Firefox & Safari / iOS**: **Strictly omits** Chromium-only `sec-ch-ua*` headers to prevent anti-bot detection and fingerprint mismatch anomalies.
+- **Edge**: Generates Microsoft Edge brand `sec-ch-ua` headers.
+
+Can be disabled globally with `config.navigation_headers = false` or per request via `navigation_headers: false`.
+
+---
+
 ## ⚙️ Global Configuration
 
 Configure global defaults in an initializer (e.g., `config/initializers/http_mimic.rb`):
@@ -204,6 +241,16 @@ HttpMimic.configure do |config|
   config.raise_on_error          = false       # Raise exceptions on HTTP errors / non-zero exits
   config.debug                   = false       # Print debug logs
 
+  # Navigation headers & Client Hints simulation (enabled by default)
+  config.navigation_headers      = true        # Browser-accurate sec-ch-ua, sec-fetch-*, Accept headers
+
+  # Host CookieStore Persistence (disabled by default)
+  config.persist_cookies         = false                                     # Auto-persist cookies per host
+  config.cookie_store_dir        = File.expand_path('~/.http_mimic/cookies') # Directory for cookie store
+
+  # WAF challenge solving (enabled by default)
+  config.auto_solve_waf          = true        # Automatically solve detected WAF JS challenges
+
   # Webdrivers-like auto-download settings (enabled by default)
   config.auto_download           = true                                  # Auto-download missing binary
   config.driver_version          = 'v2.1.1'                              # Target release version
@@ -222,12 +269,15 @@ end
 | `:auto_fallback` | Boolean | Whether to automatically retry with alternative profile on blocked status (default `true`) |
 | `:retry_statuses`| Array | Status codes that trigger auto-fallback (default `[403, 429, 503]`) |
 | `:impersonate` | String | Target browser to mimic (e.g., `'chrome131'`, `'chrome120'`, `'firefox135'`, `'safari180'`, `'tor145'`) |
+| `:navigation_headers` | Boolean | Automatically generate browser-accurate navigation headers (`sec-ch-ua`, `sec-fetch-*`, `Accept`) (default `true`) |
+| `:solve_waf` | Boolean | Automatically detect and solve WAF challenges (e.g. Akamai) via embedded QuickJS (default `true`) |
 | `:binary` | String | Path to a custom `curl-impersonate` executable |
 | `:query` / `:params` | Hash | URL query parameters (supports nested parameters and encoding) |
 | `:headers` | Hash | Custom HTTP request headers |
 | `:json` | Hash / Array | Serialized to JSON with `Content-Type: application/json` |
 | `:body` | Hash / String | Form payload (Hash) or raw request body string |
 | `:cookies` | Hash / String | Request cookies |
+| `:persist_cookies` | Boolean | Automatically load and save cookies for the host across requests |
 | `:cookie_jar` | String | Path to save cookies (`-c`) |
 | `:cookie_file` | String | Path to read cookies (`-b`) |
 | `:timeout` | Integer / Float | Maximum execution timeout in seconds (`--max-time`) |
@@ -257,18 +307,26 @@ response.redirect?      # => false (3xx)
 response.client_error?  # => false (4xx)
 response.server_error?  # => false (5xx)
 
-# Response body
-response.body           # => Raw Body (String)
+# Response body & File operations
+response.body           # => Raw Body (String / binary bytes)
+response.binary?        # => true if content is an image, pdf, or binary stream
+response.save('image.jpg') # => Directly saves response body to file
 response.parsed_response# => Auto-parsed JSON Hash / Array
 response['key']         # => Direct key access to parsed_response
 
+# HTML & Scraping Helpers
+response.title          # => HTML page title
+response.og_image       # => OpenGraph image URL
+response.meta_description # => Meta description
+response.extract_images # => Array of all resolved image URLs in HTML
+
 # Headers & Cookies
 response.headers['content-type'] # => Case-insensitive header access
 response.cookies['session_id']   # => Parsed Set-Cookie store
 response.history                 # => Array of redirect history metadata
 
 # Underlying execution & multi-strategy details
-response.mode_used            # => :impersonate or :curl
+response.mode_used            # => :impersonate, :android, :ios, or :curl
 response.fallback_triggered?  # => true if smart fallback was executed
 response.attempts             # => Array of execution metadata for each attempt
 response.exit_code            # => Process exit status (0 for success)

data/lib/http_mimic/command_builder.rb

@@ -14,40 +14,70 @@ module HttpMimic
     ].freeze
 
     TARGET_BINARY_MAP = {
-      'chrome'       => %w[curl_chrome131 curl_chrome124 curl_chrome120 curl_chrome116 curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome116'    => %w[curl_chrome116 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome120'    => %w[curl_chrome120 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome123'    => %w[curl_chrome123 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome124'    => %w[curl_chrome124 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome131'    => %w[curl_chrome131 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome133a'   => %w[curl_chrome133a curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome136'    => %w[curl_chrome136 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome142'    => %w[curl_chrome142 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome110'    => %w[curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome107'    => %w[curl_chrome107 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome104'    => %w[curl_chrome104 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome101'    => %w[curl_chrome101 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome100'    => %w[curl_chrome100 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome99'     => %w[curl_chrome99 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'firefox'      => %w[curl_firefox135 curl_firefox133 curl_ff117 curl_firefox117 curl_ff109 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox135'   => %w[curl_firefox135 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox133'   => %w[curl_firefox133 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox117'   => %w[curl_ff117 curl_firefox117 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox109'   => %w[curl_ff109 curl_firefox109 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox102'   => %w[curl_ff102 curl_firefox102 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox98'    => %w[curl_ff98 curl_firefox98 curl-impersonate-ff curl_firefox curl-impersonate],
-      'safari'       => %w[curl_safari180 curl_safari170 curl_safari155 curl_safari153 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari180'    => %w[curl_safari180 curl_safari18_0 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari170'    => %w[curl_safari170 curl_safari17_0 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari155'    => %w[curl_safari155 curl_safari15_5 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari15_5'   => %w[curl_safari155 curl_safari15_5 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari153'    => %w[curl_safari153 curl_safari15_3 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari15_3'   => %w[curl_safari153 curl_safari15_3 curl-impersonate-safari curl_safari curl-impersonate],
-      'edge'         => %w[curl_edge101 curl_edge99 curl-impersonate-edge curl_edge curl-impersonate],
-      'edge101'      => %w[curl_edge101 curl-impersonate-edge curl_edge curl-impersonate],
-      'edge99'       => %w[curl_edge99 curl-impersonate-edge curl_edge curl-impersonate],
-      'tor'          => %w[curl_tor145 curl_tor curl-impersonate],
-      'tor145'       => %w[curl_tor145 curl_tor curl-impersonate]
+      # Chrome Desktop
+      'chrome'            => %w[curl_chrome131 curl_chrome124 curl_chrome120 curl_chrome116 curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome116'         => %w[curl_chrome116 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome120'         => %w[curl_chrome120 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome123'         => %w[curl_chrome123 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome124'         => %w[curl_chrome124 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome131'         => %w[curl_chrome131 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome133a'        => %w[curl_chrome133a curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome136'         => %w[curl_chrome136 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome142'         => %w[curl_chrome142 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome145'         => %w[curl_chrome145 curl_chrome146 curl_chrome150 curl_chrome131 curl-impersonate],
+      'chrome146'         => %w[curl_chrome146 curl_chrome150 curl_chrome145 curl_chrome131 curl-impersonate],
+      'chrome150'         => %w[curl_chrome150 curl_chrome146 curl_chrome145 curl_chrome131 curl-impersonate],
+      'chrome110'         => %w[curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome107'         => %w[curl_chrome107 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome104'         => %w[curl_chrome104 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome101'         => %w[curl_chrome101 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome100'         => %w[curl_chrome100 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome99'          => %w[curl_chrome99 curl-impersonate-chrome curl_chrome curl-impersonate],
+
+      # Mobile / Android
+      'chrome131_android' => %w[curl_chrome131_android curl_chrome131 curl-impersonate-chrome curl-impersonate],
+      'chrome99_android'  => %w[curl_chrome99_android curl_chrome99 curl-impersonate-chrome curl-impersonate],
+      'android'           => %w[curl_chrome131_android curl_safari180_ios curl_chrome99_android curl_chrome131 curl-impersonate],
+      'chrome_android'    => %w[curl_chrome131_android curl_chrome99_android curl_chrome131 curl-impersonate],
+
+      # Firefox
+      'firefox'           => %w[curl_firefox135 curl_firefox133 curl_ff117 curl_firefox117 curl_ff109 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox147'        => %w[curl_firefox147 curl_firefox144 curl_firefox135 curl-impersonate-ff curl-impersonate],
+      'firefox144'        => %w[curl_firefox144 curl_firefox147 curl_firefox135 curl-impersonate-ff curl-impersonate],
+      'firefox135'        => %w[curl_firefox135 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox133'        => %w[curl_firefox133 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox117'        => %w[curl_ff117 curl_firefox117 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox109'        => %w[curl_ff109 curl_firefox109 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox102'        => %w[curl_ff102 curl_firefox102 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox98'         => %w[curl_ff98 curl_firefox98 curl-impersonate-ff curl_firefox curl-impersonate],
+
+      # Safari Desktop
+      'safari'            => %w[curl_safari180 curl_safari170 curl_safari184 curl_safari260 curl_safari155 curl_safari153 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari2601'        => %w[curl_safari2601 curl_safari260 curl_safari180 curl-impersonate-safari curl-impersonate],
+      'safari260'         => %w[curl_safari260 curl_safari2601 curl_safari180 curl-impersonate-safari curl-impersonate],
+      'safari184'         => %w[curl_safari184 curl_safari180 curl-impersonate-safari curl-impersonate],
+      'safari180'         => %w[curl_safari180 curl_safari18_0 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari170'         => %w[curl_safari170 curl_safari17_0 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari155'         => %w[curl_safari155 curl_safari15_5 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari15_5'        => %w[curl_safari155 curl_safari15_5 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari153'         => %w[curl_safari153 curl_safari15_3 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari15_3'        => %w[curl_safari153 curl_safari15_3 curl-impersonate-safari curl_safari curl-impersonate],
+
+      # Safari iOS / Mobile
+      'safari180_ios'     => %w[curl_safari180_ios curl_safari184_ios curl_safari260_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'safari184_ios'     => %w[curl_safari184_ios curl_safari180_ios curl_safari260_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'safari260_ios'     => %w[curl_safari260_ios curl_safari184_ios curl_safari180_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'safari172_ios'     => %w[curl_safari172_ios curl_safari180_ios curl_safari184_ios curl_safari180 curl-impersonate],
+      'ios'               => %w[curl_safari180_ios curl_safari184_ios curl_safari260_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'safari_ios'        => %w[curl_safari180_ios curl_safari184_ios curl_safari260_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'mobile'            => %w[curl_safari180_ios curl_chrome131_android curl_safari184_ios curl_safari260_ios curl-impersonate],
+
+      # Edge & Tor
+      'edge'              => %w[curl_edge101 curl_edge99 curl-impersonate-edge curl_edge curl-impersonate],
+      'edge101'           => %w[curl_edge101 curl-impersonate-edge curl_edge curl-impersonate],
+      'edge99'            => %w[curl_edge99 curl-impersonate-edge curl_edge curl-impersonate],
+      'tor'               => %w[curl_tor145 curl_tor curl-impersonate],
+      'tor145'            => %w[curl_tor145 curl_tor curl-impersonate]
     }.freeze
 
     attr_reader :method, :url, :options, :config
@@ -178,6 +208,13 @@ module HttpMimic
         headers_to_send['Accept'] ||= '*/*'
         headers_to_send['Accept-Encoding'] ||= 'gzip;q=1.0,deflate;q=0.6,identity;q=0.3'
         headers_to_send['Connection'] ||= 'close'
+      else
+        # Automatic Navigation Headers for browser profiles on GET/HEAD
+        nav_headers_enabled = options.fetch(:navigation_headers, config.navigation_headers)
+        if nav_headers_enabled && %i[get head].include?(method.to_s.downcase.to_sym)
+          target = resolve_target
+          apply_navigation_headers(headers_to_send, target)
+        end
       end
 
       # Append headers
@@ -202,7 +239,20 @@ module HttpMimic
       [binary, args, stdin_data, final_url]
     end
 
-    def resolve_binary
+    def resolve_target
+      case options[:profile]
+      when :android
+        (options[:android_impersonate] || 'chrome131_android').to_s.downcase
+      when :ios
+        (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
+      when :mobile
+        (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
+      else
+        (options[:impersonate] || config.default_impersonate).to_s.downcase
+      end
+    end
+
+    def resolve_binary(target = resolve_target)
       # 0. If profile is explicitly set to :curl, use system curl
       if options[:profile] == :curl
         curl_path = find_executable('curl') || 'curl'
@@ -218,7 +268,6 @@ module HttpMimic
         raise BinaryNotFoundError, "Specified executable does not exist or is not executable: #{explicit}"
       end
 
-      target = (options[:impersonate] || config.default_impersonate).to_s.downcase
       candidate_names = TARGET_BINARY_MAP[target] || ["curl_#{target}", target]
 
       # 2. Check local installation directory (~/.http_mimic/bin)
@@ -265,6 +314,81 @@ module HttpMimic
 
     private
 
+    def apply_navigation_headers(headers_to_send, target)
+      target_str = target.to_s.downcase
+
+      if target_str.start_with?('firefox', 'ff')
+        # Firefox (Gecko): Mozilla does not send Client Hints (sec-ch-ua*)
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.5'
+
+      elsif target_str.start_with?('safari') || target_str == 'ios' || target_str == 'safari_ios' || target_str.include?('_ios')
+        # Safari / WebKit (Desktop & iOS): WebKit does not send Client Hints (sec-ch-ua*)
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      elsif target_str.start_with?('tor')
+        # Tor Browser: No sec-ch-ua headers
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.5'
+
+      elsif target_str.start_with?('edge')
+        # Edge (Chromium): Microsoft Edge sec-ch-ua
+        ver = target_str[/\d+/] || '101'
+        headers_to_send['sec-ch-ua'] ||= "\"Microsoft Edge\";v=\"#{ver}\", \"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?0'
+        headers_to_send['sec-ch-ua-platform'] ||= '"Windows"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      elsif target_str.include?('android')
+        # Android Chrome
+        ver = target_str[/\d+/] || '131'
+        headers_to_send['sec-ch-ua'] ||= "\"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\", \"Google Chrome\";v=\"#{ver}\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?1'
+        headers_to_send['sec-ch-ua-platform'] ||= '"Android"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      else
+        # Chrome Desktop (Default)
+        ver = target_str[/\d+/] || '131'
+        headers_to_send['sec-ch-ua'] ||= "\"Google Chrome\";v=\"#{ver}\", \"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?0'
+        headers_to_send['sec-ch-ua-platform'] ||= '"macOS"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+      end
+    end
+
     def find_in_download_dir(candidates)
       target_dir = File.expand_path(config.install_dir)
       return nil unless Dir.exist?(target_dir)

data/lib/http_mimic/configuration.rb

@@ -2,7 +2,9 @@
 
 module HttpMimic
   class Configuration
+    # Browser simulation & request defaults
     attr_accessor :default_impersonate
+    attr_accessor :default_mobile_impersonate
     attr_accessor :binary_path
     attr_accessor :fallback_to_curl
     attr_accessor :default_timeout
@@ -24,28 +26,57 @@ module HttpMimic
     attr_accessor :install_dir
     attr_accessor :github_repo
 
+    # QuickJS standalone binary management settings
+    attr_accessor :qjs_version
+    attr_accessor :qjs_github_repo
+
+    # WAF & JS challenge solving settings
+    attr_accessor :auto_solve_waf
+
+    # Shared Host-based CookieStore settings
+    attr_accessor :persist_cookies
+    attr_accessor :cookie_store_dir
+
+    # Modern Navigation Headers simulation
+    attr_accessor :navigation_headers
+
     def initialize
-      @default_impersonate      = 'chrome131'
-      @binary_path              = nil
-      @fallback_to_curl         = true
-      @default_timeout          = 30
-      @default_connect_timeout  = 10
-      @follow_redirects         = true
-      @max_redirects            = 10
-      @raise_on_error           = false
-      @logger                   = nil
-      @debug                    = false
-
-      # Multi-strategy defaults: :auto seamlessly falls back between impersonate and curl
-      @mode                     = :auto
-      @auto_fallback            = true
-      @retry_statuses           = [403, 429, 503]
+      @default_impersonate         = 'chrome131'
+      @default_mobile_impersonate  = 'safari180_ios'
+      @binary_path                 = nil
+      @fallback_to_curl            = true
+      @default_timeout             = 30
+      @default_connect_timeout     = 10
+      @follow_redirects            = true
+      @max_redirects               = 10
+      @raise_on_error              = false
+      @logger                      = nil
+      @debug                       = false
+
+      # Multi-strategy defaults: :auto seamlessly falls back between impersonate, mobile, and curl
+      @mode                        = :auto
+      @auto_fallback               = true
+      @retry_statuses              = [403, 429, 503]
 
       # Enable automatic downloading of curl-impersonate driver (lexiforest) by default
-      @auto_download            = true
-      @driver_version           = 'v2.1.1'
-      @install_dir              = File.expand_path('~/.http_mimic/bin')
-      @github_repo              = 'lexiforest/curl-impersonate'
+      @auto_download               = true
+      @driver_version              = 'v2.1.1'
+      @install_dir                 = File.expand_path('~/.http_mimic/bin')
+      @github_repo                 = 'lexiforest/curl-impersonate'
+
+      # QuickJS defaults
+      @qjs_version                 = 'v0.16.2'
+      @qjs_github_repo             = 'quickjs-ng/quickjs'
+
+      # WAF challenge solving defaults
+      @auto_solve_waf              = true
+
+      # Shared Host CookieStore defaults
+      @persist_cookies             = false
+      @cookie_store_dir            = File.expand_path('~/.http_mimic/cookies')
+
+      # Navigation Headers defaults
+      @navigation_headers          = true
     end
   end
 end

data/lib/http_mimic/cookie_store.rb

@@ -0,0 +1,118 @@
+# frozen_string_literal: true
+
+require 'fileutils'
+require 'json'
+require 'uri'
+
+module HttpMimic
+  class CookieStore
+    DEFAULT_TTL = 7200 # 2 hours
+
+    class << self
+      def store_dir
+        File.expand_path(HttpMimic.configuration.cookie_store_dir || '~/.http_mimic/cookies')
+      end
+
+      def cookie_file_for(host)
+        clean_host = sanitize_host(host)
+        File.join(store_dir, "#{clean_host}.json")
+      end
+
+      # Load cookies for a given host
+      # Returns a Hash of cookies if valid and not expired, or empty Hash
+      def load(host, max_age: DEFAULT_TTL)
+        file = cookie_file_for(host)
+        return {} unless File.file?(file)
+
+        data = nil
+        File.open(file, File::RDONLY) do |f|
+          f.flock(File::LOCK_SH)
+          data = JSON.parse(f.read) rescue nil
+        end
+
+        return {} unless data.is_a?(Hash)
+
+        updated_at = data['updated_at'].to_i
+        if !max_age.nil?
+          return {} if (Time.now.to_i - updated_at) > max_age
+        end
+
+        data['cookies'] || {}
+      rescue StandardError => e
+        if HttpMimic.configuration.debug
+          puts "[HttpMimic::CookieStore] Error loading cookies for #{host}: #{e.message}"
+        end
+        {}
+      end
+
+      # Save cookies for a given host with atomic locking
+      def save(host, cookies, ttl: DEFAULT_TTL)
+        return if cookies.nil? || cookies.empty?
+
+        clean_host = sanitize_host(host)
+        dir = store_dir
+        FileUtils.mkdir_p(dir)
+
+        file = cookie_file_for(clean_host)
+        existing_cookies = load(clean_host, max_age: ttl) || {}
+
+        new_cookies = cookies.is_a?(Cookies) ? cookies.to_h : cookies.dup
+        merged = existing_cookies.merge(stringify_keys(new_cookies))
+
+        payload = {
+          'host' => clean_host,
+          'updated_at' => Time.now.to_i,
+          'ttl' => ttl,
+          'cookies' => merged
+        }
+
+        # Thread & Process safe atomic write
+        tmp_file = "#{file}.tmp.#{Process.pid}_#{Time.now.to_f}"
+        File.open(tmp_file, File::RDWR | File::CREAT, 0644) do |f|
+          f.flock(File::LOCK_EX)
+          f.write(JSON.pretty_generate(payload))
+          f.flush
+        end
+        File.rename(tmp_file, file)
+      rescue StandardError => e
+        FileUtils.rm_f(tmp_file) if tmp_file
+        if HttpMimic.configuration.debug
+          puts "[HttpMimic::CookieStore] Error saving cookies for #{host}: #{e.message}"
+        end
+      end
+
+      # Clear stored cookies for a host or all hosts
+      def clear(host = nil)
+        if host
+          file = cookie_file_for(host)
+          FileUtils.rm_f(file)
+        else
+          FileUtils.rm_rf(store_dir)
+        end
+      end
+
+      # Extract host from URL or host string
+      def extract_host(url_or_host)
+        str = url_or_host.to_s.strip
+        if str =~ %r{^https?://}i
+          URI.parse(str).host
+        else
+          str.split(':').first
+        end
+      rescue StandardError
+        str
+      end
+
+      private
+
+      def sanitize_host(host)
+        h = extract_host(host) || 'default'
+        h.gsub(/[^a-zA-Z0-9.\-_]/, '_')
+      end
+
+      def stringify_keys(hash)
+        hash.each_with_object({}) { |(k, v), h| h[k.to_s] = v.to_s }
+      end
+    end
+  end
+end