http_mimic 0.3.2 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 11e55edf4c06160f27621a651e4c16eae929ec08767b5b6eff6b007d8a3a1d4b
-  data.tar.gz: e90137a02ee867cd744641c725cd3daef2994b5f4a7216304f7ca189e6d92303
+  metadata.gz: b09e014134575dac987366c71b0cb6644a13a5b17088947621474f6cfccec5cf
+  data.tar.gz: 748d8d3dadcb69d39312a6c7137c32cca1290b7eaee4d8641fa7bb58bfdd0f80
 SHA512:
-  metadata.gz: 77dfc30ee87e63105f5cbf95e818cc4b01841f9862566fdceedad277cb22366c05c37ab14b3ac699ddb30d68a9ead35acc43114a1f882c034360d06807450677
-  data.tar.gz: 4ab621140feebc4c3f9e8897d33edf510785ea6c5cbad5cf08b672ecfde52f6146b6f1a3d695f5129acce0d3c7b64cfdae275803480178500e9c683fb7c053a8
+  metadata.gz: a4d17399d5f85df9b508151f371b0910e074d70e3c05f11cb291f30f01b71c20d22e998ec5c5466728c77dd2f37b3de4aecc18883a844fb69952ae71a3a5c5be
+  data.tar.gz: 7b0675e724e1c138cd57ea4af5e5a6ce2f057bb50981271f9b81cb894bf7179c2ba760137f0672d742a63b602783bc6b455a71b9da4f778c0637db96ee723470

data/CHANGELOG.md

@@ -5,6 +5,59 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.1] - 2026-08-30
+
+### Added
+- **Two-Phase Cookie & State Pipeline (Tier 1 ➔ Tier 2)**:
+  - Automatically forward validated cookies (`NID`, `AEC`, `_abck`, `cf_clearance`) and custom headers from Tier 1 HTTP / WAF resolution directly into Obscura (`--cookie` flag) during `auto_render_spa` transitions.
+- **Obscura Stealth BoringSSL Integration**:
+  - Downloader now defaults to `*-stealth.tar.gz` release assets, equipping Obscura with native BoringSSL Chrome TLS / JA4 fingerprint impersonation.
+- **Upgraded Obscura to `anxgang/obscura` v0.2.3**:
+  - WebGL context, `REAL_FONT_METRICS` font measurements, and persistent Web Worker `importScripts` support.
+
+## [0.5.0] - 2026-08-30
+
+### Added
+- **Tier 2: Obscura Headless SPA Rendering Support**:
+  - Native integration with [`h4ckf0r0day/obscura`](https://github.com/h4ckf0r0day/obscura)—a lightweight (<100MB) Rust + V8 headless browser engine with built-in stealth anti-detection.
+  - On-demand automatic driver management (`HttpMimic.download_obscura!`, `HttpMimic.obscura_installed?`, `HttpMimic.obscura_path`) across macOS (ARM64/x86_64), Linux (x86_64/aarch64), and Windows.
+  - New rendering helpers: `HttpMimic.render(url, options = {})` and `HttpMimic.spa(url, options = {})`.
+  - **Automatic SPA Detection & Transition (`auto_render_spa`)**:
+    - Intelligent `HttpMimic::SpaDetector` detects unhydrated client-side SPA shells (React `#root`, Vue `#app`, Angular `<app-root>`, Google Dynamic SERP, `<noscript>` prompts).
+    - When `auto_render_spa: true` (or `config.auto_render_spa = true`) is enabled, `HttpMimic.get` automatically transitions from Tier 1 (HTTP) to Tier 2 (Obscura) to render the full dynamic DOM.
+  - Configurable options: `:wait_until` (`networkidle0`, `domcontentloaded`, `load`), `:timeout`, `:proxy`, `:eval`, `:dump`, `:selector`, `:user_agent`, and `:stealth`.
+
+## [0.4.1] - 2026-08-30
+
+### Added
+- **Google Search Guard & BotGuard VM Detection**:
+  - Added `HttpMimic::Waf::GoogleSolver` and `HttpMimic::Waf::Detector.google?` for detecting Google Search Guard (`knitsail`, `enablejs`, `SG_SS`) challenges.
+  - Added automatic Google Search referral context (`Referer: https://www.google.com/`, `sec-fetch-site: same-origin`) in `CommandBuilder#apply_navigation_headers`.
+- **Authentic Hardware & WebGL Pipeline Emulation in QuickJS Context**:
+  - Full WebGL shader compilation and rasterization pipeline (`createShader`, `compileShader`, `getShaderPrecisionFormat`, `createProgram`, `linkProgram`, `readPixels` simulated gradient buffer).
+  - Dynamic font measurement (`REAL_FONT_METRICS`) with exact macOS Chrome `offsetWidth` / `offsetHeight` dimensions across 17 font families and 7 font sizes.
+  - Authentic Chromium `OfflineAudioContext` audio buffer rendering curve.
+  - Physics-based Cubic Bézier Spline mouse movement simulation with velocity decay, natural micro-jitter, and complete click event chains.
+  - Polyfills for `trustedTypes`, `sessionStorage`, `localStorage`, and `document.currentScript`.
+- **Automatic Response Decompression**:
+  - Added `--compressed` flag to CommandBuilder to ensure automatic transparent decompression of `gzip`, `deflate`, `br` (Brotli), and `zstd` payloads.
+
+## [0.4.0] - 2026-08-30
+
+### Added
+- **Target-Specific Navigation Headers & Client Hints Simulation**:
+  - Automatically generates browser-accurate Navigation Headers (`sec-fetch-dest`, `sec-fetch-mode`, `sec-fetch-site`, `upgrade-insecure-requests`) and Client Hints (`sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform`) tailored to the selected impersonation target.
+  - Firefox (`firefox*`, `ff*`), Safari / iOS (`safari*`, `ios*`), and Tor targets strictly omit Chromium-only `sec-ch-ua*` headers to avoid anti-bot fingerprint anomalies and detection.
+  - Android Chrome targets properly send `sec-ch-ua-mobile: ?1` and `sec-ch-ua-platform: "Android"`.
+  - Edge targets properly send `"Microsoft Edge"` brand headers with `"Windows"` platform.
+- **Unified Persistent CookieStore (`persist_cookies`)**:
+  - Added `:persist_cookies` request option and `persist_cookies` class-level DSL method to automatically load and persist session cookies across requests per host into `~/.http_mimic/cookies/`.
+  - Added helper methods: `HttpMimic.load_cookies(host, max_age: nil)`, `HttpMimic.save_cookies(host, cookies, ttl: nil)`, and `HttpMimic.clear_cookies!(host = nil)`.
+- **WAF Challenge Solving & Fallback Performance Optimizations**:
+  - Added single-pass WAF solver guard (`waf_solve_attempted`) to prevent redundant JavaScript solver executions across fallback attempts.
+  - Added cross-attempt cookie accumulation to preserve session state between fallback profiles.
+  - Added `HttpMimic::Waf::Detector.challenge_page?` to detect interstitial challenges disguised under HTTP 200 responses.
+
 ## [0.3.2] - 2026-08-29
 
 ### Added

data/README.md

@@ -21,7 +21,7 @@ It also includes **Webdrivers-like automatic driver management**, automatically
   - Edge & Tor support: `edge101`, `edge99`, `tor145`.
 - **HTTParty-Style API**:
   - Direct module methods: `HttpMimic.get`, `HttpMimic.post`, etc.
-  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`).
+  - Class mixin via `include HttpMimic` (`base_uri`, `headers`, `default_params`, `default_timeout`, `impersonate`, `proxy`, `cookies`, `persist_cookies`).
   - Reusable instance client: `HttpMimic::Client.new(...)`.
 - **Zero Shell Injection Risk**:
   - Executes commands with array arguments via `Open3.capture3(*cmd_array)`.
@@ -186,6 +186,40 @@ HttpMimic.get(images.first).save('samba_og.jpg')
 
 ---
 
+## 🍪 Persistent Host CookieStore
+
+`HttpMimic` supports persistent host-based cookie caching. Once enabled, session and verification cookies are automatically saved to `~/.http_mimic/cookies/<host>.json` and reused in subsequent requests to the same host:
+
+```ruby
+# 1. Enable per-request
+response = HttpMimic.get('https://example.com/items', persist_cookies: true)
+
+# 2. Or enable in class mixin
+class Scraper
+  include HttpMimic
+  persist_cookies true
+end
+
+# 3. Manual CookieStore helpers
+HttpMimic.load_cookies('example.com')                  # => Hash of active cookies
+HttpMimic.save_cookies('example.com', { session: '123' }) # Save cookies with default TTL
+HttpMimic.clear_cookies!('example.com')                # Clear host or all cookies
+```
+
+---
+
+## 🧭 Target-Specific Navigation Headers & Client Hints
+
+`HttpMimic` automatically constructs browser-accurate HTTP navigation headers on `GET` and `HEAD` requests:
+
+- **Chrome Desktop / Mobile**: Generates `sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform`, and standard Chrome `Accept` headers.
+- **Firefox & Safari / iOS**: **Strictly omits** Chromium-only `sec-ch-ua*` headers to prevent anti-bot detection and fingerprint mismatch anomalies.
+- **Edge**: Generates Microsoft Edge brand `sec-ch-ua` headers.
+
+Can be disabled globally with `config.navigation_headers = false` or per request via `navigation_headers: false`.
+
+---
+
 ## ⚙️ Global Configuration
 
 Configure global defaults in an initializer (e.g., `config/initializers/http_mimic.rb`):
@@ -207,6 +241,16 @@ HttpMimic.configure do |config|
   config.raise_on_error          = false       # Raise exceptions on HTTP errors / non-zero exits
   config.debug                   = false       # Print debug logs
 
+  # Navigation headers & Client Hints simulation (enabled by default)
+  config.navigation_headers      = true        # Browser-accurate sec-ch-ua, sec-fetch-*, Accept headers
+
+  # Host CookieStore Persistence (disabled by default)
+  config.persist_cookies         = false                                     # Auto-persist cookies per host
+  config.cookie_store_dir        = File.expand_path('~/.http_mimic/cookies') # Directory for cookie store
+
+  # WAF challenge solving (enabled by default)
+  config.auto_solve_waf          = true        # Automatically solve detected WAF JS challenges
+
   # Webdrivers-like auto-download settings (enabled by default)
   config.auto_download           = true                                  # Auto-download missing binary
   config.driver_version          = 'v2.1.1'                              # Target release version
@@ -225,12 +269,15 @@ end
 | `:auto_fallback` | Boolean | Whether to automatically retry with alternative profile on blocked status (default `true`) |
 | `:retry_statuses`| Array | Status codes that trigger auto-fallback (default `[403, 429, 503]`) |
 | `:impersonate` | String | Target browser to mimic (e.g., `'chrome131'`, `'chrome120'`, `'firefox135'`, `'safari180'`, `'tor145'`) |
+| `:navigation_headers` | Boolean | Automatically generate browser-accurate navigation headers (`sec-ch-ua`, `sec-fetch-*`, `Accept`) (default `true`) |
+| `:solve_waf` | Boolean | Automatically detect and solve WAF challenges (e.g. Akamai) via embedded QuickJS (default `true`) |
 | `:binary` | String | Path to a custom `curl-impersonate` executable |
 | `:query` / `:params` | Hash | URL query parameters (supports nested parameters and encoding) |
 | `:headers` | Hash | Custom HTTP request headers |
 | `:json` | Hash / Array | Serialized to JSON with `Content-Type: application/json` |
 | `:body` | Hash / String | Form payload (Hash) or raw request body string |
 | `:cookies` | Hash / String | Request cookies |
+| `:persist_cookies` | Boolean | Automatically load and save cookies for the host across requests |
 | `:cookie_jar` | String | Path to save cookies (`-c`) |
 | `:cookie_file` | String | Path to read cookies (`-b`) |
 | `:timeout` | Integer / Float | Maximum execution timeout in seconds (`--max-time`) |
@@ -240,6 +287,47 @@ end
 | `:bearer_token` | String | Appends `Authorization: Bearer <token>` header |
 | `:insecure` | Boolean | Disable SSL certificate verification (`-k`) |
 | `:curl_options` | Array / String | Additional raw curl arguments (e.g., `['--http2', '--compressed']`) |
+| `:auto_render_spa` | Boolean | Automatically detect unhydrated SPA shells and render via Obscura |
+| `:render` | Symbol | `:spa` or `:obscura` to explicitly render page via Obscura |
+
+---
+
+## ⚡ Obscura SPA Rendering (Tier 2 Engine)
+
+`http_mimic` provides native support for [`h4ckf0r0day/obscura`](https://github.com/h4ckf0r0day/obscura)—a lightweight, single-binary (<100MB) headless browser engine written in Rust with an embedded V8 JavaScript runtime and built-in stealth anti-detection capabilities.
+
+Obscura is auto-downloaded on-demand into `~/.http_mimic/bin/obscura` when SPA rendering is requested.
+
+### 1. Automatic SPA Detection & Rendering (`auto_render_spa`)
+
+Similar to `auto_solve_waf`, when `auto_render_spa: true` is enabled, `HttpMimic` first sends a microsecond-fast HTTP request (Tier 1). If the response is detected as an unhydrated SPA shell (empty React `#root`, Vue `#app`, Angular `<app-root>`, Google Dynamic SERP, or `<noscript>` prompt), it automatically transitions to Obscura (Tier 2) to render the full DOM tree:
+
+```ruby
+# Auto-detects SPA shell and seamlessly renders with Obscura
+response = HttpMimic.get('https://example.com/spa', auto_render_spa: true)
+
+puts response.code    # => 200
+puts response.title   # => Fully hydrated DOM title
+puts response.body    # => Fully rendered HTML with client-side injected DOM nodes
+```
+
+### 2. Direct SPA Rendering
+
+```ruby
+# Direct SPA render
+response = HttpMimic.render('https://example.com/spa')
+
+# or via alias:
+response = HttpMimic.spa('https://example.com/spa')
+```
+
+**Supported Options**:
+- `:wait_until` - `'load'`, `'domcontentloaded'`, `'networkidle0'` (default: `'networkidle0'`)
+- `:timeout` - execution deadline in seconds (default: 30)
+- `:proxy` - proxy address (`socks5://...` or `http://...`)
+- `:eval` - JavaScript expression to evaluate in page context
+- `:dump` - output format: `'html'` (default), `'text'`, `'links'`, `'markdown'`, `'cookies'`
+- `:stealth` - whether stealth anti-detection is enabled (default: `true`)
 
 ---
 

data/lib/http_mimic/command_builder.rb

@@ -94,9 +94,10 @@ module HttpMimic
       args = []
       stdin_data = nil
 
-      # Base flags: silent mode, include HTTP headers
+      # Base flags: silent mode, include HTTP headers, auto-decompress responses
       args << '-s'
       args << '-i'
+      args << '--compressed'
 
       # HTTP Method
       if @method == 'HEAD'
@@ -208,6 +209,13 @@ module HttpMimic
         headers_to_send['Accept'] ||= '*/*'
         headers_to_send['Accept-Encoding'] ||= 'gzip;q=1.0,deflate;q=0.6,identity;q=0.3'
         headers_to_send['Connection'] ||= 'close'
+      else
+        # Automatic Navigation Headers for browser profiles on GET/HEAD
+        nav_headers_enabled = options.fetch(:navigation_headers, config.navigation_headers)
+        if nav_headers_enabled && %i[get head].include?(method.to_s.downcase.to_sym)
+          target = resolve_target
+          apply_navigation_headers(headers_to_send, target)
+        end
       end
 
       # Append headers
@@ -232,7 +240,20 @@ module HttpMimic
       [binary, args, stdin_data, final_url]
     end
 
-    def resolve_binary
+    def resolve_target
+      case options[:profile]
+      when :android
+        (options[:android_impersonate] || 'chrome131_android').to_s.downcase
+      when :ios
+        (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
+      when :mobile
+        (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
+      else
+        (options[:impersonate] || config.default_impersonate).to_s.downcase
+      end
+    end
+
+    def resolve_binary(target = resolve_target)
       # 0. If profile is explicitly set to :curl, use system curl
       if options[:profile] == :curl
         curl_path = find_executable('curl') || 'curl'
@@ -248,16 +269,6 @@ module HttpMimic
         raise BinaryNotFoundError, "Specified executable does not exist or is not executable: #{explicit}"
       end
 
-      target = case options[:profile]
-               when :android
-                 (options[:android_impersonate] || 'chrome131_android').to_s.downcase
-               when :ios
-                 (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
-               when :mobile
-                 (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
-               else
-                 (options[:impersonate] || config.default_impersonate).to_s.downcase
-               end
       candidate_names = TARGET_BINARY_MAP[target] || ["curl_#{target}", target]
 
       # 2. Check local installation directory (~/.http_mimic/bin)
@@ -304,6 +315,92 @@ module HttpMimic
 
     private
 
+    def apply_navigation_headers(headers_to_send, target)
+      target_str = target.to_s.downcase
+
+      if target_str.start_with?('firefox', 'ff')
+        # Firefox (Gecko): Mozilla does not send Client Hints (sec-ch-ua*)
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.5'
+
+      elsif target_str.start_with?('safari') || target_str == 'ios' || target_str == 'safari_ios' || target_str.include?('_ios')
+        # Safari / WebKit (Desktop & iOS): WebKit does not send Client Hints (sec-ch-ua*)
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      elsif target_str.start_with?('tor')
+        # Tor Browser: No sec-ch-ua headers
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.5'
+
+      elsif target_str.start_with?('edge')
+        # Edge (Chromium): Microsoft Edge sec-ch-ua
+        ver = target_str[/\d+/] || '101'
+        headers_to_send['sec-ch-ua'] ||= "\"Microsoft Edge\";v=\"#{ver}\", \"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?0'
+        headers_to_send['sec-ch-ua-platform'] ||= '"Windows"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      elsif target_str.include?('android')
+        # Android Chrome
+        ver = target_str[/\d+/] || '131'
+        headers_to_send['sec-ch-ua'] ||= "\"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\", \"Google Chrome\";v=\"#{ver}\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?1'
+        headers_to_send['sec-ch-ua-platform'] ||= '"Android"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+
+      else
+        # Chrome Desktop (Default)
+        ver = target_str[/\d+/] || '131'
+        headers_to_send['sec-ch-ua'] ||= "\"Google Chrome\";v=\"#{ver}\", \"Chromium\";v=\"#{ver}\", \"Not_A Brand\";v=\"24\""
+        headers_to_send['sec-ch-ua-mobile'] ||= '?0'
+        headers_to_send['sec-ch-ua-platform'] ||= '"macOS"'
+        headers_to_send['sec-fetch-dest'] ||= 'document'
+        headers_to_send['sec-fetch-mode'] ||= 'navigate'
+        headers_to_send['sec-fetch-site'] ||= 'none'
+        headers_to_send['sec-fetch-user'] ||= '?1'
+        headers_to_send['upgrade-insecure-requests'] ||= '1'
+        headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
+        headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
+      end
+
+      # Smart Search referral context (e.g. searching from Google homepage)
+      begin
+        parsed_uri = URI.parse(url)
+        if parsed_uri && parsed_uri.host =~ /google\./i && parsed_uri.path =~ /\/search/i
+          headers_to_send['referer'] ||= "#{parsed_uri.scheme || 'https'}://#{parsed_uri.host}/"
+          headers_to_send['sec-fetch-site'] ||= 'same-origin'
+        end
+      rescue StandardError
+        # ignore
+      end
+    end
+
     def find_in_download_dir(candidates)
       target_dir = File.expand_path(config.install_dir)
       return nil unless Dir.exist?(target_dir)

data/lib/http_mimic/configuration.rb

@@ -26,6 +26,26 @@ module HttpMimic
     attr_accessor :install_dir
     attr_accessor :github_repo
 
+    # QuickJS standalone binary management settings
+    attr_accessor :qjs_version
+    attr_accessor :qjs_github_repo
+
+    # Obscura headless SPA engine settings
+    attr_accessor :obscura_version
+    attr_accessor :obscura_github_repo
+    attr_accessor :obscura_path
+    attr_accessor :auto_render_spa
+
+    # WAF & JS challenge solving settings
+    attr_accessor :auto_solve_waf
+
+    # Shared Host-based CookieStore settings
+    attr_accessor :persist_cookies
+    attr_accessor :cookie_store_dir
+
+    # Modern Navigation Headers simulation
+    attr_accessor :navigation_headers
+
     def initialize
       @default_impersonate         = 'chrome131'
       @default_mobile_impersonate  = 'safari180_ios'
@@ -45,10 +65,33 @@ module HttpMimic
       @retry_statuses              = [403, 429, 503]
 
       # Enable automatic downloading of curl-impersonate driver (lexiforest) by default
-      @auto_download            = true
-      @driver_version           = 'v2.1.1'
-      @install_dir              = File.expand_path('~/.http_mimic/bin')
-      @github_repo              = 'lexiforest/curl-impersonate'
+      @auto_download               = true
+      @driver_version              = 'v2.1.1'
+      @install_dir                 = File.expand_path('~/.http_mimic/bin')
+      @github_repo                 = 'lexiforest/curl-impersonate'
+
+      # QuickJS defaults
+      @qjs_version                 = 'v0.16.2'
+      @qjs_github_repo             = 'quickjs-ng/quickjs'
+
+      # Obscura defaults (Lightweight Rust+V8 headless SPA engine)
+      # Temporarily points to 'anxgang/obscura' (v0.2.3) for Web Worker importScripts support.
+      # Will switch back to 'h4ckf0r0day/obscura' once upstream Pull Request is merged.
+      # @obscura_github_repo       = 'h4ckf0r0day/obscura'
+      @obscura_version             = 'v0.2.3'
+      @obscura_github_repo         = 'anxgang/obscura'
+      @obscura_path                = nil
+      @auto_render_spa             = false
+
+      # WAF challenge solving defaults
+      @auto_solve_waf              = true
+
+      # Shared Host CookieStore defaults
+      @persist_cookies             = false
+      @cookie_store_dir            = File.expand_path('~/.http_mimic/cookies')
+
+      # Navigation Headers defaults
+      @navigation_headers          = true
     end
   end
 end

data/lib/http_mimic/cookie_store.rb

@@ -0,0 +1,118 @@
+# frozen_string_literal: true
+
+require 'fileutils'
+require 'json'
+require 'uri'
+
+module HttpMimic
+  class CookieStore
+    DEFAULT_TTL = 7200 # 2 hours
+
+    class << self
+      def store_dir
+        File.expand_path(HttpMimic.configuration.cookie_store_dir || '~/.http_mimic/cookies')
+      end
+
+      def cookie_file_for(host)
+        clean_host = sanitize_host(host)
+        File.join(store_dir, "#{clean_host}.json")
+      end
+
+      # Load cookies for a given host
+      # Returns a Hash of cookies if valid and not expired, or empty Hash
+      def load(host, max_age: DEFAULT_TTL)
+        file = cookie_file_for(host)
+        return {} unless File.file?(file)
+
+        data = nil
+        File.open(file, File::RDONLY) do |f|
+          f.flock(File::LOCK_SH)
+          data = JSON.parse(f.read) rescue nil
+        end
+
+        return {} unless data.is_a?(Hash)
+
+        updated_at = data['updated_at'].to_i
+        if !max_age.nil?
+          return {} if (Time.now.to_i - updated_at) > max_age
+        end
+
+        data['cookies'] || {}
+      rescue StandardError => e
+        if HttpMimic.configuration.debug
+          puts "[HttpMimic::CookieStore] Error loading cookies for #{host}: #{e.message}"
+        end
+        {}
+      end
+
+      # Save cookies for a given host with atomic locking
+      def save(host, cookies, ttl: DEFAULT_TTL)
+        return if cookies.nil? || cookies.empty?
+
+        clean_host = sanitize_host(host)
+        dir = store_dir
+        FileUtils.mkdir_p(dir)
+
+        file = cookie_file_for(clean_host)
+        existing_cookies = load(clean_host, max_age: ttl) || {}
+
+        new_cookies = cookies.is_a?(Cookies) ? cookies.to_h : cookies.dup
+        merged = existing_cookies.merge(stringify_keys(new_cookies))
+
+        payload = {
+          'host' => clean_host,
+          'updated_at' => Time.now.to_i,
+          'ttl' => ttl,
+          'cookies' => merged
+        }
+
+        # Thread & Process safe atomic write
+        tmp_file = "#{file}.tmp.#{Process.pid}_#{Time.now.to_f}"
+        File.open(tmp_file, File::RDWR | File::CREAT, 0644) do |f|
+          f.flock(File::LOCK_EX)
+          f.write(JSON.pretty_generate(payload))
+          f.flush
+        end
+        File.rename(tmp_file, file)
+      rescue StandardError => e
+        FileUtils.rm_f(tmp_file) if tmp_file
+        if HttpMimic.configuration.debug
+          puts "[HttpMimic::CookieStore] Error saving cookies for #{host}: #{e.message}"
+        end
+      end
+
+      # Clear stored cookies for a host or all hosts
+      def clear(host = nil)
+        if host
+          file = cookie_file_for(host)
+          FileUtils.rm_f(file)
+        else
+          FileUtils.rm_rf(store_dir)
+        end
+      end
+
+      # Extract host from URL or host string
+      def extract_host(url_or_host)
+        str = url_or_host.to_s.strip
+        if str =~ %r{^https?://}i
+          URI.parse(str).host
+        else
+          str.split(':').first
+        end
+      rescue StandardError
+        str
+      end
+
+      private
+
+      def sanitize_host(host)
+        h = extract_host(host) || 'default'
+        h.gsub(/[^a-zA-Z0-9.\-_]/, '_')
+      end
+
+      def stringify_keys(hash)
+        hash.each_with_object({}) { |(k, v), h| h[k.to_s] = v.to_s }
+      end
+    end
+  end
+end

data/lib/http_mimic/cookies.rb

@@ -47,6 +47,44 @@ module HttpMimic
       @store.values
     end
 
+    def empty?
+      @store.empty?
+    end
+
+    def size
+      @store.size
+    end
+    alias length size
+
+    def delete(name)
+      @cookie_items.delete(name.to_s)
+      @store.delete(name.to_s)
+    end
+
+    def clear
+      @cookie_items.clear
+      @store.clear
+    end
+
+    def merge(other)
+      dup_cookies = self.class.new(@store)
+      if other.is_a?(Cookies)
+        other.each { |k, v| dup_cookies[k] = v }
+      elsif other.is_a?(Hash)
+        other.each { |k, v| dup_cookies[k] = v }
+      end
+      dup_cookies
+    end
+
+    def merge!(other)
+      if other.is_a?(Cookies)
+        other.each { |k, v| self[k] = v }
+      elsif other.is_a?(Hash)
+        other.each { |k, v| self[k] = v }
+      end
+      self
+    end
+
     def to_h
       @store.dup
     end