http_mimic 0.4.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9b0a9fff1b74c98ea24c294a491b00ba57e270d24e6c0bb7a669e6d2db264d8a
-  data.tar.gz: e11d3ae615b92abcfedda8547120290d82020acd82230353a127cc97dfc844f6
+  metadata.gz: b09e014134575dac987366c71b0cb6644a13a5b17088947621474f6cfccec5cf
+  data.tar.gz: 748d8d3dadcb69d39312a6c7137c32cca1290b7eaee4d8641fa7bb58bfdd0f80
 SHA512:
-  metadata.gz: 5e0a93bb67bb84582e24df07ea4d51d291a53f0b48ab77ad0bd4a63b666bede4b493bbb405ad34d76602e086eafd76a455668c2a491488d83dd2c0be1060fe66
-  data.tar.gz: c616cafefeb23d78af7320beab5ddc3b8f79aafda440fd1fac889f5a5990417b31716105e8c74cab566e6c0ed6eca04a82a397ddc4242eb59caed519e52739de
+  metadata.gz: a4d17399d5f85df9b508151f371b0910e074d70e3c05f11cb291f30f01b71c20d22e998ec5c5466728c77dd2f37b3de4aecc18883a844fb69952ae71a3a5c5be
+  data.tar.gz: 7b0675e724e1c138cd57ea4af5e5a6ce2f057bb50981271f9b81cb894bf7179c2ba760137f0672d742a63b602783bc6b455a71b9da4f778c0637db96ee723470

data/CHANGELOG.md

@@ -5,6 +5,43 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.1] - 2026-08-30
+
+### Added
+- **Two-Phase Cookie & State Pipeline (Tier 1 ➔ Tier 2)**:
+  - Automatically forward validated cookies (`NID`, `AEC`, `_abck`, `cf_clearance`) and custom headers from Tier 1 HTTP / WAF resolution directly into Obscura (`--cookie` flag) during `auto_render_spa` transitions.
+- **Obscura Stealth BoringSSL Integration**:
+  - Downloader now defaults to `*-stealth.tar.gz` release assets, equipping Obscura with native BoringSSL Chrome TLS / JA4 fingerprint impersonation.
+- **Upgraded Obscura to `anxgang/obscura` v0.2.3**:
+  - WebGL context, `REAL_FONT_METRICS` font measurements, and persistent Web Worker `importScripts` support.
+
+## [0.5.0] - 2026-08-30
+
+### Added
+- **Tier 2: Obscura Headless SPA Rendering Support**:
+  - Native integration with [`h4ckf0r0day/obscura`](https://github.com/h4ckf0r0day/obscura)—a lightweight (<100MB) Rust + V8 headless browser engine with built-in stealth anti-detection.
+  - On-demand automatic driver management (`HttpMimic.download_obscura!`, `HttpMimic.obscura_installed?`, `HttpMimic.obscura_path`) across macOS (ARM64/x86_64), Linux (x86_64/aarch64), and Windows.
+  - New rendering helpers: `HttpMimic.render(url, options = {})` and `HttpMimic.spa(url, options = {})`.
+  - **Automatic SPA Detection & Transition (`auto_render_spa`)**:
+    - Intelligent `HttpMimic::SpaDetector` detects unhydrated client-side SPA shells (React `#root`, Vue `#app`, Angular `<app-root>`, Google Dynamic SERP, `<noscript>` prompts).
+    - When `auto_render_spa: true` (or `config.auto_render_spa = true`) is enabled, `HttpMimic.get` automatically transitions from Tier 1 (HTTP) to Tier 2 (Obscura) to render the full dynamic DOM.
+  - Configurable options: `:wait_until` (`networkidle0`, `domcontentloaded`, `load`), `:timeout`, `:proxy`, `:eval`, `:dump`, `:selector`, `:user_agent`, and `:stealth`.
+
+## [0.4.1] - 2026-08-30
+
+### Added
+- **Google Search Guard & BotGuard VM Detection**:
+  - Added `HttpMimic::Waf::GoogleSolver` and `HttpMimic::Waf::Detector.google?` for detecting Google Search Guard (`knitsail`, `enablejs`, `SG_SS`) challenges.
+  - Added automatic Google Search referral context (`Referer: https://www.google.com/`, `sec-fetch-site: same-origin`) in `CommandBuilder#apply_navigation_headers`.
+- **Authentic Hardware & WebGL Pipeline Emulation in QuickJS Context**:
+  - Full WebGL shader compilation and rasterization pipeline (`createShader`, `compileShader`, `getShaderPrecisionFormat`, `createProgram`, `linkProgram`, `readPixels` simulated gradient buffer).
+  - Dynamic font measurement (`REAL_FONT_METRICS`) with exact macOS Chrome `offsetWidth` / `offsetHeight` dimensions across 17 font families and 7 font sizes.
+  - Authentic Chromium `OfflineAudioContext` audio buffer rendering curve.
+  - Physics-based Cubic Bézier Spline mouse movement simulation with velocity decay, natural micro-jitter, and complete click event chains.
+  - Polyfills for `trustedTypes`, `sessionStorage`, `localStorage`, and `document.currentScript`.
+- **Automatic Response Decompression**:
+  - Added `--compressed` flag to CommandBuilder to ensure automatic transparent decompression of `gzip`, `deflate`, `br` (Brotli), and `zstd` payloads.
+
 ## [0.4.0] - 2026-08-30
 
 ### Added

data/README.md

@@ -287,6 +287,47 @@ end
 | `:bearer_token` | String | Appends `Authorization: Bearer <token>` header |
 | `:insecure` | Boolean | Disable SSL certificate verification (`-k`) |
 | `:curl_options` | Array / String | Additional raw curl arguments (e.g., `['--http2', '--compressed']`) |
+| `:auto_render_spa` | Boolean | Automatically detect unhydrated SPA shells and render via Obscura |
+| `:render` | Symbol | `:spa` or `:obscura` to explicitly render page via Obscura |
+
+---
+
+## ⚡ Obscura SPA Rendering (Tier 2 Engine)
+
+`http_mimic` provides native support for [`h4ckf0r0day/obscura`](https://github.com/h4ckf0r0day/obscura)—a lightweight, single-binary (<100MB) headless browser engine written in Rust with an embedded V8 JavaScript runtime and built-in stealth anti-detection capabilities.
+
+Obscura is auto-downloaded on-demand into `~/.http_mimic/bin/obscura` when SPA rendering is requested.
+
+### 1. Automatic SPA Detection & Rendering (`auto_render_spa`)
+
+Similar to `auto_solve_waf`, when `auto_render_spa: true` is enabled, `HttpMimic` first sends a microsecond-fast HTTP request (Tier 1). If the response is detected as an unhydrated SPA shell (empty React `#root`, Vue `#app`, Angular `<app-root>`, Google Dynamic SERP, or `<noscript>` prompt), it automatically transitions to Obscura (Tier 2) to render the full DOM tree:
+
+```ruby
+# Auto-detects SPA shell and seamlessly renders with Obscura
+response = HttpMimic.get('https://example.com/spa', auto_render_spa: true)
+
+puts response.code    # => 200
+puts response.title   # => Fully hydrated DOM title
+puts response.body    # => Fully rendered HTML with client-side injected DOM nodes
+```
+
+### 2. Direct SPA Rendering
+
+```ruby
+# Direct SPA render
+response = HttpMimic.render('https://example.com/spa')
+
+# or via alias:
+response = HttpMimic.spa('https://example.com/spa')
+```
+
+**Supported Options**:
+- `:wait_until` - `'load'`, `'domcontentloaded'`, `'networkidle0'` (default: `'networkidle0'`)
+- `:timeout` - execution deadline in seconds (default: 30)
+- `:proxy` - proxy address (`socks5://...` or `http://...`)
+- `:eval` - JavaScript expression to evaluate in page context
+- `:dump` - output format: `'html'` (default), `'text'`, `'links'`, `'markdown'`, `'cookies'`
+- `:stealth` - whether stealth anti-detection is enabled (default: `true`)
 
 ---
 

data/lib/http_mimic/command_builder.rb

@@ -94,9 +94,10 @@ module HttpMimic
       args = []
       stdin_data = nil
 
-      # Base flags: silent mode, include HTTP headers
+      # Base flags: silent mode, include HTTP headers, auto-decompress responses
       args << '-s'
       args << '-i'
+      args << '--compressed'
 
       # HTTP Method
       if @method == 'HEAD'
@@ -387,6 +388,17 @@ module HttpMimic
         headers_to_send['accept'] ||= 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
         headers_to_send['accept-language'] ||= 'en-US,en;q=0.9'
       end
+
+      # Smart Search referral context (e.g. searching from Google homepage)
+      begin
+        parsed_uri = URI.parse(url)
+        if parsed_uri && parsed_uri.host =~ /google\./i && parsed_uri.path =~ /\/search/i
+          headers_to_send['referer'] ||= "#{parsed_uri.scheme || 'https'}://#{parsed_uri.host}/"
+          headers_to_send['sec-fetch-site'] ||= 'same-origin'
+        end
+      rescue StandardError
+        # ignore
+      end
     end
 
     def find_in_download_dir(candidates)

data/lib/http_mimic/configuration.rb

@@ -30,6 +30,12 @@ module HttpMimic
     attr_accessor :qjs_version
     attr_accessor :qjs_github_repo
 
+    # Obscura headless SPA engine settings
+    attr_accessor :obscura_version
+    attr_accessor :obscura_github_repo
+    attr_accessor :obscura_path
+    attr_accessor :auto_render_spa
+
     # WAF & JS challenge solving settings
     attr_accessor :auto_solve_waf
 
@@ -68,6 +74,15 @@ module HttpMimic
       @qjs_version                 = 'v0.16.2'
       @qjs_github_repo             = 'quickjs-ng/quickjs'
 
+      # Obscura defaults (Lightweight Rust+V8 headless SPA engine)
+      # Temporarily points to 'anxgang/obscura' (v0.2.3) for Web Worker importScripts support.
+      # Will switch back to 'h4ckf0r0day/obscura' once upstream Pull Request is merged.
+      # @obscura_github_repo       = 'h4ckf0r0day/obscura'
+      @obscura_version             = 'v0.2.3'
+      @obscura_github_repo         = 'anxgang/obscura'
+      @obscura_path                = nil
+      @auto_render_spa             = false
+
       # WAF challenge solving defaults
       @auto_solve_waf              = true
 

data/lib/http_mimic/downloader.rb

@@ -16,6 +16,9 @@ module HttpMimic
     DEFAULT_QJS_REPO    = 'quickjs-ng/quickjs'
     DEFAULT_QJS_VERSION = 'v0.16.2'
 
+    DEFAULT_OBSCURA_REPO    = 'h4ckf0r0day/obscura'
+    DEFAULT_OBSCURA_VERSION = 'v0.2.1'
+
     class DownloadError < HttpMimic::Error; end
     class UnsupportedPlatformError < HttpMimic::Error; end
 
@@ -165,6 +168,100 @@ module HttpMimic
         File.join(dir, '.qjs_version')
       end
 
+      def download_obscura!(version: nil, install_dir: nil, repo: nil, force: false)
+        target_version = normalize_version(version || HttpMimic.configuration.obscura_version || DEFAULT_OBSCURA_VERSION)
+        target_dir     = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        target_repo    = repo || HttpMimic.configuration.obscura_github_repo || DEFAULT_OBSCURA_REPO
+
+        FileUtils.mkdir_p(target_dir)
+
+        dest_binary = File.join(target_dir, binary_name_for_platform('obscura'))
+
+        if !force && obscura_installed?(version: target_version, install_dir: target_dir)
+          log_info("obscura #{target_version} is already installed in #{target_dir}")
+          return dest_binary
+        end
+
+        asset_name = obscura_platform_asset
+        download_url = "https://github.com/#{target_repo}/releases/download/#{target_version}/#{asset_name}"
+
+        log_info("Downloading Obscura (#{target_version}) [#{asset_name}]...")
+        archive_data = fetch_binary(download_url)
+
+        log_info("Extracting Obscura archive to #{target_dir}...")
+        extract_tar_gz(archive_data, target_dir)
+
+        # Ensure executable permissions on extracted binaries
+        %w[obscura obscura-worker].each do |name|
+          p = File.join(target_dir, binary_name_for_platform(name))
+          File.chmod(0755, p) if File.exist?(p)
+        end
+
+        File.write(obscura_version_file_path(target_dir), target_version)
+
+        log_info("Obscura #{target_version} installation complete! (#{dest_binary})")
+        dest_binary
+      end
+
+      def obscura_installed?(version: nil, install_dir: nil)
+        target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        bin = File.join(target_dir, binary_name_for_platform('obscura'))
+        return false unless File.file?(bin) && File.executable?(bin)
+
+        if version
+          target_version = normalize_version(version)
+          v_file = obscura_version_file_path(target_dir)
+          return false unless File.file?(v_file)
+          return File.read(v_file).strip == target_version
+        end
+
+        true
+      end
+
+      def obscura_path(install_dir: nil)
+        return HttpMimic.configuration.obscura_path if HttpMimic.configuration.obscura_path && File.executable?(HttpMimic.configuration.obscura_path)
+
+        target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
+        candidate = File.join(target_dir, binary_name_for_platform('obscura'))
+        return candidate if File.file?(candidate) && File.executable?(candidate)
+
+        # Fallback to system PATH
+        sys_bin = `which obscura 2>/dev/null`.strip
+        return sys_bin if !sys_bin.empty? && File.executable?(sys_bin)
+
+        nil
+      end
+
+      def obscura_platform_asset
+        os  = host_os
+        cpu = host_cpu
+
+        case os
+        when :macos
+          case cpu
+          when :arm64 then 'obscura-aarch64-macos-stealth.tar.gz'
+          when :x86_64 then 'obscura-x86_64-macos-stealth.tar.gz'
+          else
+            raise UnsupportedPlatformError, "Unsupported macOS CPU architecture for Obscura: #{cpu}"
+          end
+        when :linux
+          case cpu
+          when :x86_64 then 'obscura-x86_64-linux-stealth.tar.gz'
+          when :aarch64, :arm64 then 'obscura-aarch64-linux-stealth.tar.gz'
+          else
+            raise UnsupportedPlatformError, "Unsupported Linux CPU architecture for Obscura: #{cpu}"
+          end
+        when :windows
+          'obscura-x86_64-windows-stealth.zip'
+        else
+          raise UnsupportedPlatformError, "Unsupported operating system for Obscura: #{RbConfig::CONFIG['host_os']}"
+        end
+      end
+
+      def obscura_version_file_path(dir)
+        File.join(dir, '.obscura_version')
+      end
+
       def binary_path(name, install_dir: nil)
         target_dir = File.expand_path(install_dir || HttpMimic.configuration.install_dir)
         candidate = File.join(target_dir, binary_name_for_platform(name))

data/lib/http_mimic/module_methods.rb

@@ -46,6 +46,11 @@ module HttpMimic
       default_options[:persist_cookies] = enabled
     end
 
+    def auto_render_spa(enabled = nil)
+      return default_options[:auto_render_spa] if enabled.nil?
+      default_options[:auto_render_spa] = enabled
+    end
+
     def mode(m = nil)
       return default_options[:mode] if m.nil?
       default_options[:mode] = m
@@ -93,6 +98,12 @@ module HttpMimic
       request(:options, url, options)
     end
 
+    def render(url, options = {})
+      merged = default_options.merge(options)
+      Obscura.render(url, merged)
+    end
+    alias spa render
+
     def request(method, url, options = {})
       merged = default_options.merge(options)
 

data/lib/http_mimic/obscura.rb

@@ -0,0 +1,139 @@
+# frozen_string_literal: true
+
+require 'open3'
+require 'uri'
+require 'json'
+
+module HttpMimic
+  module Obscura
+    class ExecutionError < HttpMimic::Error; end
+    class BinaryNotFoundError < HttpMimic::Error; end
+
+    class << self
+      # Renders a web page using the Obscura headless SPA engine and returns an HttpMimic::Response
+      #
+      # @param url [String, URI] The target URL to render
+      # @param options [Hash] Rendering options
+      # @option options [String] :wait_until ('networkidle0') Wait condition (e.g. 'load', 'domcontentloaded', 'networkidle0')
+      # @option options [Integer] :timeout Per-command timeout in seconds
+      # @option options [String] :proxy Proxy URL (e.g. 'socks5://127.0.0.1:1080' or 'http://...')
+      # @option options [String] :eval JavaScript expression to evaluate on the page
+      # @option options [String] :dump Output format ('html', 'text', 'links')
+      # @return [HttpMimic::Response]
+      def render(url, options = {})
+        bin = resolve_binary
+        raise BinaryNotFoundError, "Obscura binary not found. Run HttpMimic.download_obscura! or configure obscura_path." unless bin
+
+        args = [bin, 'fetch', url.to_s]
+
+        # Dump format: default to html (can be 'html', 'text', 'links', 'markdown', 'original', 'cookies')
+        dump_format = options[:dump] || 'html'
+        args << '--dump' << dump_format.to_s
+
+        # Stealth mode (enabled by default)
+        args << '--stealth' if options.fetch(:stealth, true)
+
+        # Wait condition
+        if options[:wait_until]
+          args << '--wait-until' << options[:wait_until].to_s
+        end
+
+        # Wait delay
+        if options[:wait]
+          args << '--wait' << options[:wait].to_i.to_s
+        end
+
+        # Timeout in seconds (integer)
+        timeout = options[:timeout] || HttpMimic.configuration.default_timeout
+        args << '--timeout' << timeout.to_i.to_s if timeout
+
+        # User Agent
+        if options[:user_agent]
+          args << '--user-agent' << options[:user_agent].to_s
+        end
+
+        # Proxy
+        if options[:proxy]
+          args << '--proxy' << options[:proxy].to_s
+        end
+
+        # Cookies: pass cookies string or hash (Two-Phase Pipeline support)
+        if options[:cookies] || options[:cookie]
+          cookie_val = options[:cookies] || options[:cookie]
+          cookie_str = if cookie_val.is_a?(Hash)
+            cookie_val.map { |k, v| "#{k}=#{v}" }.join('; ')
+          elsif cookie_val.is_a?(Array)
+            cookie_val.join('; ')
+          else
+            cookie_val.to_s
+          end
+          args << '--cookie' << cookie_str unless cookie_str.empty?
+        end
+
+        # Optional JS evaluation
+        args << '--eval' << options[:eval].to_s if options[:eval]
+
+        # Selector
+        args << '--selector' << options[:selector].to_s if options[:selector]
+
+        log_debug("Executing Obscura SPA render: #{args.join(' ')}")
+
+        stdout, stderr, status = Open3.capture3(*args)
+
+        code = (status && status.success?) ? 200 : (status ? status.exitstatus : 500)
+
+        raw_headers = "HTTP/2 200 OK\r\ncontent-type: text/html; charset=utf-8\r\nx-rendered-by: obscura\r\n\r\n"
+        headers = Headers.new({ 'content-type' => 'text/html; charset=utf-8', 'x-rendered-by' => 'obscura' })
+
+        Response.new(
+          code: code,
+          http_version: 'HTTP/2',
+          status_message: status&.success? ? 'OK (Obscura SPA Rendered)' : 'Error',
+          headers: headers,
+          cookies: Cookies.new,
+          body: stdout,
+          parsed_response: nil,
+          raw_headers: raw_headers,
+          history: [],
+          request_url: url.to_s,
+          stderr: stderr,
+          exit_code: status ? status.exitstatus : 0,
+          command: args.join(' ')
+        )
+      end
+
+      def resolve_binary
+        # 1. Configured custom path
+        custom_path = HttpMimic.configuration.obscura_path
+        if custom_path && (File.file?(custom_path) || File.executable?(custom_path))
+          return custom_path
+        end
+
+        # 2. Check install directory or system PATH
+        installed_path = Downloader.obscura_path
+        return installed_path if installed_path
+
+        # 3. Auto-download if enabled
+        if HttpMimic.configuration.auto_download
+          begin
+            return Downloader.download_obscura!
+          rescue StandardError => e
+            HttpMimic.configuration.logger&.warn("[HttpMimic::Obscura] Auto-download failed: #{e.message}")
+          end
+        end
+
+        nil
+      end
+
+      private
+
+      def log_debug(msg)
+        if HttpMimic.configuration.logger
+          HttpMimic.configuration.logger.debug("[HttpMimic::Obscura] #{msg}")
+        elsif HttpMimic.configuration.debug
+          puts "[HttpMimic::Obscura] #{msg}"
+        end
+      end
+    end
+  end
+end

data/lib/http_mimic/request.rb

@@ -15,6 +15,12 @@ module HttpMimic
 
     def perform
       mode = (options[:mode] || config.mode || :auto).to_sym
+
+      # Delegate directly to Obscura headless SPA renderer if requested
+      if options[:render] == :spa || options[:render] == :obscura || mode == :spa || mode == :obscura
+        return Obscura.render(url, options)
+      end
+
       auto_fallback = options.fetch(:auto_fallback, config.auto_fallback)
       retry_statuses = options[:retry_statuses] || config.retry_statuses || [403, 429, 503]
 
@@ -113,6 +119,29 @@ module HttpMimic
         log_debug("[HttpMimic] Attempt #{index + 1} with #{profile} resulted in status #{response.code}. Triggering smart fallback to next profile...")
       end
 
+      # Automatic SPA Detection & Obscura rendering fallback
+      auto_render_spa = options.fetch(:auto_render_spa, config.auto_render_spa)
+      if auto_render_spa && response && response.success? && method.to_s.upcase == 'GET'
+        if SpaDetector.spa?(response)
+          log_debug("[HttpMimic] Detected unhydrated SPA shell on #{url}. Automatically rendering with Obscura...")
+          begin
+            spa_opts = options.dup
+            # Forward all validated cookies from Tier 1 (Mode 2: Two-Phase Pipeline)
+            if response.cookies && !response.cookies.empty?
+              tier1_cookies = response.cookies.to_h
+              existing_cookies = spa_opts[:cookies].is_a?(Hash) ? spa_opts[:cookies] : {}
+              spa_opts[:cookies] = existing_cookies.merge(tier1_cookies)
+            end
+            rendered_resp = Obscura.render(url, spa_opts)
+            if rendered_resp && rendered_resp.success?
+              response = rendered_resp
+            end
+          rescue StandardError => e
+            log_debug("[HttpMimic] Automatic Obscura SPA render failed (#{e.message}), keeping Tier 1 response.")
+          end
+        end
+      end
+
       # Persist cookies back to store if enabled
       if should_persist_cookie && host && response && response.cookies && !response.cookies.empty?
         CookieStore.save(host, response.cookies)

data/lib/http_mimic/spa_detector.rb

@@ -0,0 +1,95 @@
+# frozen_string_literal: true
+
+module HttpMimic
+  module SpaDetector
+    # Framework mount points that indicate an unhydrated SPA client shell
+    EMPTY_MOUNT_REGEX = %r{
+      <div[^>]+id=["'](?:root|app|__next|__nuxt|svelte)["'][^>]*>\s*</div>|
+      <app-root[^>]*>\s*</app-root>|
+      <div[^>]+id=["']app["'][^>]*>\s*<!--\s*(?:app-content)?\s*-->\s*</div>
+    }ix
+
+    # Noscript patterns requiring JavaScript execution
+    NOSCRIPT_JS_REQUIRED_REGEX = %r{
+      <noscript[^>]*>[\s\S]*?(?:enable\s+javascript|javascript\s+is\s+required|need\s+to\s+enable\s+javascript|requires\s+javascript)[\s\S]*?</noscript>
+    }ix
+
+    # Google Search Guard dynamic SPA shell indicators
+    GOOGLE_SHELL_REGEX = %r{
+      (?:/httpservice/retry/enablejs|SG_REL|emsg=SG_REL|knitsail)
+    }ix
+
+    class << self
+      # Returns true if the HTTP response represents an unhydrated Single Page Application (SPA) shell
+      #
+      # @param response [HttpMimic::Response]
+      # @return [Boolean]
+      def spa?(response)
+        return false unless response && response.success?
+        return false if response.respond_to?(:binary?) && response.binary?
+
+        content_type = response.headers['content-type'].to_s.downcase
+        return false unless content_type.empty? || content_type.include?('text/html') || content_type.include?('application/xhtml')
+
+        body = response.body.to_s
+        return false if body.strip.empty?
+
+        # 1. Google Dynamic SERP / Search Guard JS Shell
+        if google_serp_shell?(body)
+          return true
+        end
+
+        # 2. Modern SPA framework empty mount points (React, Vue, Angular, Next, Nuxt, Svelte)
+        if empty_mount_point?(body)
+          return true
+        end
+
+        # 3. Noscript requirement message
+        if noscript_js_required?(body)
+          return true
+        end
+
+        # 4. Small body with multiple scripts and virtually no visible text
+        if low_text_js_heavy_shell?(body)
+          return true
+        end
+
+        false
+      end
+
+      def empty_mount_point?(body)
+        body.match?(EMPTY_MOUNT_REGEX)
+      end
+
+      def noscript_js_required?(body)
+        body.match?(NOSCRIPT_JS_REQUIRED_REGEX)
+      end
+
+      def google_serp_shell?(body)
+        return false unless body.match?(GOOGLE_SHELL_REGEX)
+
+        # If it has the Google shell indicators AND lacks rendered organic result elements
+        !body.include?('<h3 class=') && !body.include?('id="search"') && !body.include?('id="rso"')
+      end
+
+      def low_text_js_heavy_shell?(body)
+        return false if body.length > 25_000 # Larger HTML files typically have SSR content
+
+        # Check if there are script tags
+        script_count = body.scan(/<script/i).size
+        return false if script_count.zero?
+
+        # Strip all HTML tags, scripts, and styles to get raw visible text length
+        visible_text = body
+                       .gsub(/<script[\s\S]*?<\/script>/i, '')
+                       .gsub(/<style[\s\S]*?<\/style>/i, '')
+                       .gsub(/<[^>]+>/, ' ')
+                       .gsub(/\s+/, ' ')
+                       .strip
+
+        # If visible text is virtually nonexistent but page has multiple scripts
+        visible_text.length < 80 && script_count >= 2
+      end
+    end
+  end
+end

data/lib/http_mimic/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module HttpMimic
-  VERSION = "0.4.0"
+  VERSION = "0.5.1"
 end

data/lib/http_mimic/waf/detector.rb

@@ -9,6 +9,8 @@ module HttpMimic
 
           if akamai?(response)
             :akamai
+          elsif google?(response)
+            :google
           elsif cloudflare?(response)
             :cloudflare
           elsif datadome?(response)
@@ -24,6 +26,16 @@ module HttpMimic
           return true if body.include?('sec-if-cpt-container') || body.include?('sec-bc-button-parent')
           return true if body.include?('challenges.cloudflare.com') || body.include?('cf-turnstile')
           return true if body.include?('datadome.captcha') || body.include?('geo.captcha-delivery.com')
+          return true if body.include?('/httpservice/retry/enablejs') || (body.include?('knitsail') && body.include?('SG_SS'))
+          false
+        end
+
+        def google?(response)
+          return false unless response
+          body = response.body.to_s
+          return true if body.include?('/httpservice/retry/enablejs')
+          return true if body.include?('knitsail') && body.include?('SG_SS')
+          return true if response.headers['server']&.downcase&.include?('gws') && body.include?('enablejs')
           false
         end
 

data/lib/http_mimic/waf/google_solver.rb

@@ -0,0 +1,124 @@
+# frozen_string_literal: true
+
+require 'json'
+require 'uri'
+
+module HttpMimic
+  module Waf
+    class GoogleSolver
+      class << self
+        def solve(target_url, response, options = {})
+          return nil unless response
+
+          html = response.body.to_s
+          return nil if html.empty?
+
+          uri = URI.parse(target_url) rescue nil
+          base_url = uri ? "#{uri.scheme}://#{uri.host}" : "https://www.google.com"
+
+          impersonate = options[:impersonate] || HttpMimic.configuration.default_impersonate || 'chrome131'
+          current_cookies = (response.cookies || Cookies.new).dup
+
+          # Strategy 1: Session warmup from Google root if cookies are sparse
+          if current_cookies.empty? || !current_cookies.key?('NID')
+            warmup_resp = HttpMimic.get(
+              "#{base_url}/",
+              impersonate: impersonate,
+              cookies: current_cookies,
+              auto_fallback: false,
+              solve_waf: false,
+              persist_cookies: false
+            )
+            warmup_resp.cookies.each { |k, v| current_cookies[k] = v } if warmup_resp&.cookies
+          end
+
+          # Strategy 2: Attempt QuickJS solve if script payload exists
+          scripts = html.scan(/<script[^>]*>([\s\S]*?)<\/script>/i).map(&:first)
+          if scripts.any? { |s| s.include?('knitsail') || s.include?('SG_SS') || s.include?('closureDynamicButton') }
+            sg_ss_cookie = solve_with_quickjs(target_url, scripts, current_cookies)
+            current_cookies['SG_SS'] = sg_ss_cookie if sg_ss_cookie && !sg_ss_cookie.start_with?('E:')
+          end
+
+          # Strategy 3: Retry search query with same-origin referral headers and updated cookies
+          retry_headers = (options[:headers] || {}).dup
+          retry_headers['Referer'] ||= "#{base_url}/"
+          retry_headers['sec-fetch-site'] ||= 'same-origin'
+
+          retry_opts = options.merge(
+            cookies: current_cookies,
+            headers: retry_headers,
+            auto_fallback: false,
+            solve_waf: false
+          )
+
+          HttpMimic.get(target_url, **retry_opts)
+        rescue StandardError => e
+          HttpMimic.logger.debug("[HttpMimic::Waf::GoogleSolver] Failed to solve Google challenge: #{e.message}") if HttpMimic.logger
+          nil
+        end
+
+        private
+
+        def solve_with_quickjs(target_url, scripts, cookies)
+          context_js_path = File.expand_path('browser_context.js', __dir__)
+          context_js = File.read(context_js_path)
+          cookie_str = cookies.respond_to?(:to_cookie_string) ? cookies.to_cookie_string : cookies.to_s
+
+          driver_script = <<~JS
+            globalThis.__TARGET_URL__ = #{target_url.to_json};
+            globalThis.__DOCUMENT_TITLE__ = "Google Search";
+            globalThis.__INITIAL_COOKIES__ = #{cookie_str.to_json};
+            globalThis.__REFERRER__ = "https://www.google.com/";
+
+            #{context_js}
+
+            // Google specific environment enhancements
+            globalThis.sessionStorage = {
+              _data: {},
+              getItem(k) { return this._data[k] || null; },
+              setItem(k, v) { this._data[k] = String(v); },
+              removeItem(k) { delete this._data[k]; },
+              clear() { this._data = {}; }
+            };
+            globalThis.localStorage = globalThis.sessionStorage;
+            globalThis._F_css = function() {};
+            globalThis.google = { c: { c: { a: true } } };
+
+            #{scripts.join("\n;\n")}
+
+            // Extract p token and invoke knitsail if present
+            var allScripts = #{scripts.join("\n").to_json};
+            var pMatch = allScripts.match(/var p=\\x27([^\\x27]+)\\x27/);
+            var pVal = pMatch ? pMatch[1] : null;
+
+            var solved_token = null;
+            if (globalThis.knitsail && typeof globalThis.knitsail.a === "function" && pVal) {
+              try {
+                globalThis.knitsail.a(pVal, function(resultCallback) {
+                  if (typeof resultCallback === "function") {
+                    resultCallback(function(token) {
+                      solved_token = token;
+                    }, [{}]);
+                  }
+                }, false, undefined, undefined, undefined, undefined, true);
+              } catch(e) {}
+            }
+
+            if (typeof globalThis.__drainEventLoop === "function") {
+              globalThis.__drainEventLoop(50);
+            }
+
+            JSON.stringify({
+              token: solved_token
+            });
+          JS
+
+          result = JSRuntime.eval_json(driver_script)
+          result ? result['token'] : nil
+        rescue StandardError
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/http_mimic/waf.rb

@@ -2,6 +2,7 @@
 
 require 'http_mimic/waf/detector'
 require 'http_mimic/waf/akamai_solver'
+require 'http_mimic/waf/google_solver'
 
 module HttpMimic
   module Waf
@@ -13,6 +14,8 @@ module HttpMimic
         case waf_type
         when :akamai
           AkamaiSolver.solve(url, response, options)
+        when :google
+          GoogleSolver.solve(url, response, options)
         else
           nil
         end

data/lib/http_mimic.rb

@@ -19,6 +19,8 @@ require 'http_mimic/module_methods'
 require 'http_mimic/js_runtime'
 require 'http_mimic/waf'
 require 'http_mimic/cookie_store'
+require 'http_mimic/obscura'
+require 'http_mimic/spa_detector'
 
 module HttpMimic
   extend ModuleMethods
@@ -94,6 +96,30 @@ module HttpMimic
       CookieStore.clear(host)
     end
 
+    # Obscura headless SPA engine helpers
+    def download_obscura!(version: nil, force: false)
+      Downloader.download_obscura!(version: version, force: force)
+    end
+
+    def obscura_installed?(version: nil)
+      Downloader.obscura_installed?(version: version)
+    end
+
+    def obscura_path
+      Downloader.obscura_path
+    end
+
+    # Render SPA page using Obscura headless engine
+    def render(url, options = {})
+      Obscura.render(url, options)
+    end
+    alias spa render
+
+    # Check if a response represents an unhydrated SPA shell
+    def spa?(response)
+      SpaDetector.spa?(response)
+    end
+
     def included(base)
       base.extend(ModuleMethods)
       base.send(:include, InstanceMethods)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: http_mimic
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.1
 platform: ruby
 authors:
 - anxgang
@@ -72,13 +72,16 @@ files:
 - lib/http_mimic/headers.rb
 - lib/http_mimic/js_runtime.rb
 - lib/http_mimic/module_methods.rb
+- lib/http_mimic/obscura.rb
 - lib/http_mimic/request.rb
 - lib/http_mimic/response.rb
 - lib/http_mimic/response_parser.rb
+- lib/http_mimic/spa_detector.rb
 - lib/http_mimic/version.rb
 - lib/http_mimic/waf.rb
 - lib/http_mimic/waf/akamai_solver.rb
 - lib/http_mimic/waf/detector.rb
+- lib/http_mimic/waf/google_solver.rb
 homepage: https://github.com/anxgang/http_mimic
 licenses:
 - MIT