http_mimic 0.3.0 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 32ae2902090ddf216e6f33843f1fb53d82d13cda864a3a71fd7318f5f316cdf4
-  data.tar.gz: 907aec51da124505902e104ebf3bce70eb2883490ec8b360f3f0571e6efd98c2
+  metadata.gz: 11e55edf4c06160f27621a651e4c16eae929ec08767b5b6eff6b007d8a3a1d4b
+  data.tar.gz: e90137a02ee867cd744641c725cd3daef2994b5f4a7216304f7ca189e6d92303
 SHA512:
-  metadata.gz: e47bf2078c03dc6a37ecb3478ee5f7a42724375161c63175067fd6df5c8c76fae90b04867eb6f4b8fe03ab2ad2d919ef64dcfd1389ab20befdcb9e2ff245d492
-  data.tar.gz: 31c4ebd49e1180914bd949dc23bbc4bd7580dea84f898c7a97bc4977a8501120b1d3d5e6301b65ef2bf7cfa9e5c9bbecd8a7f47806f0a8cf42e1e1b4e7b2e45c
+  metadata.gz: 77dfc30ee87e63105f5cbf95e818cc4b01841f9862566fdceedad277cb22366c05c37ab14b3ac699ddb30d68a9ead35acc43114a1f882c034360d06807450677
+  data.tar.gz: 4ab621140feebc4c3f9e8897d33edf510785ea6c5cbad5cf08b672ecfde52f6146b6f1a3d695f5129acce0d3c7b64cfdae275803480178500e9c683fb7c053a8

data/CHANGELOG.md

@@ -5,6 +5,28 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.2] - 2026-08-29
+
+### Added
+- **Mobile Impersonation & Multi-Stage Auto-Fallback**:
+  - Added support for mobile browser targets: Android Chrome (`chrome131_android`, `chrome99_android`, alias `android`) and iOS Safari (`safari260_ios`, `safari180_ios`, `safari184_ios`, `safari172_ios`, alias `ios`, `mobile`).
+  - Enhanced `:auto` adaptive mode with automatic fallback cascade `[:impersonate, :android, :ios, :curl]` to bypass modern anti-bot systems (e.g. Akamai Bot Manager on Adidas HK) without requiring JavaScript telemetry solving.
+  - Added execution modes: `:mobile_first`, `:android_first`, `:ios_first`, `:mobile_only`, `:android_only`, `:ios_only`.
+- **Binary-Safe Response Parsing & File Saving**:
+  - Fixed binary handling in `ResponseParser#split_headers_and_body` to prevent `ArgumentError: invalid byte sequence in UTF-8` on images, PDFs, compressed files, and binary streams.
+  - Added `response.binary?` and `response.save_to_file(path)` / `response.save(path)` helpers.
+- **Built-in HTML / Scraping Helpers**:
+  - Added `response.title`, `response.og_image`, `response.meta_description`, and `response.extract_images(base_url: nil)`.
+
+## [0.3.1] - 2026-08-25
+
+### Added
+- **Smart Adaptive Multi-Strategy Modes (`:auto`, `:impersonate_first`, `:curl_first`, `:impersonate_only`, `:curl_only`)**:
+  - **Zero-Configuration Protection Bypass (`:auto` mode, enabled by default)**: Seamlessly tries `curl-impersonate` first, and if blocked by WAFs with `403`/`429`/`503` (e.g. Akamai bot challenges requiring JS), automatically falls back to standard `curl` with server client headers to reliably retrieve 200 OK.
+  - Per-request and class-level configurable execution mode (`mode:`, `auto_fallback:`, `retry_statuses:`).
+  - Enhanced `Response` object with `response.mode_used`, `response.fallback_triggered?`, and `response.attempts` metadata.
+  - Added live integration test for Akamai-protected site bypass (`test_smart_auto_fallback_on_akamai_protected_site`).
+
 ## [0.3.0] - 2026-08-24
 
 ### Added

data/README.md

@@ -142,6 +142,7 @@ puts "User-Agent: #{res['user_agent']}"
 ```ruby
 client = HttpMimic::Client.new(
   base_uri: 'https://api.example.com',
+  mode: :auto,
   impersonate: 'firefox135',
   timeout: 15,
   headers: {
@@ -158,12 +159,44 @@ response = client.post('/v1/users', json: { username: 'bob' })
 
 ---
 
+## 🧠 Smart Adaptive Modes (Zero-Configuration Scraping)
+
+`HttpMimic` provides built-in multi-strategy orchestration so you can fetch protected websites without worrying about which specific WAF (Akamai Botman, Cloudflare Turnstile, DataDome, Kasada) protects them:
+
+- **`:auto` (Default & Recommended)**: Tries desktop `curl-impersonate` (`chrome131`) first. If blocked by WAFs like Akamai with `403`/`429`/`503` (which require JS telemetry for desktop browsers), it **automatically cascades through mobile profiles (`android`, `ios`) and server `curl` headers**, reliably retrieving `200 OK`.
+- **`:mobile_first` / `:android_first` / `:ios_first`**: Prefers mobile TLS/HTTP2 fingerprints and falls back to desktop or curl if blocked.
+- **`:impersonate_first`**: Prefers desktop `curl-impersonate` and automatically falls back to mobile and curl if blocked.
+- **`:curl_first`**: Prefers standard server `curl` and automatically upgrades to impersonate profiles if blocked.
+- **`:impersonate_only` / `:mobile_only` / `:curl_only`**: Strictly uses the chosen profile (no fallback).
+
+```ruby
+# 1. No-Brain Auto Mode (Works automatically for Cloudflare, Akamai, etc.):
+response = HttpMimic.get('https://www.adidas.com.hk/en/KI8139.html')
+puts response.code                 # => 200
+puts response.mode_used            # => :android
+puts response.fallback_triggered?  # => true
+
+# 2. Extract images and metadata with built-in helpers:
+puts response.title                # => "SAMBA OG SHOES - Brown | adidas Hong Kong"
+images = response.extract_images   # => ["https://assets.adidas.com/images/.../KI8139_01_00_standard.jpg", ...]
+
+# 3. Fetch and save binary images directly:
+HttpMimic.get(images.first).save('samba_og.jpg')
+```
+
+---
+
 ## ⚙️ Global Configuration
 
 Configure global defaults in an initializer (e.g., `config/initializers/http_mimic.rb`):
 
 ```ruby
 HttpMimic.configure do |config|
+  # Multi-strategy & Smart Fallback
+  config.mode                    = :auto       # :auto (default), :impersonate_first, :curl_first, :impersonate_only, :curl_only
+  config.auto_fallback           = true        # Automatically retry with alternative profile if blocked
+  config.retry_statuses          = [403, 429, 503] # Status codes that trigger auto-fallback
+
   # Browser simulation & request defaults
   config.default_impersonate     = 'chrome131' # Default browser target
   config.default_timeout         = 30          # Request timeout (seconds)
@@ -188,6 +221,9 @@ end
 
 | Option | Type | Description |
 | :--- | :--- | :--- |
+| `:mode` | Symbol | Execution strategy: `:auto` (default), `:impersonate_first`, `:curl_first`, `:impersonate_only`, `:curl_only` |
+| `:auto_fallback` | Boolean | Whether to automatically retry with alternative profile on blocked status (default `true`) |
+| `:retry_statuses`| Array | Status codes that trigger auto-fallback (default `[403, 429, 503]`) |
 | `:impersonate` | String | Target browser to mimic (e.g., `'chrome131'`, `'chrome120'`, `'firefox135'`, `'safari180'`, `'tor145'`) |
 | `:binary` | String | Path to a custom `curl-impersonate` executable |
 | `:query` / `:params` | Hash | URL query parameters (supports nested parameters and encoding) |
@@ -224,20 +260,31 @@ response.redirect?      # => false (3xx)
 response.client_error?  # => false (4xx)
 response.server_error?  # => false (5xx)
 
-# Response body
-response.body           # => Raw Body (String)
+# Response body & File operations
+response.body           # => Raw Body (String / binary bytes)
+response.binary?        # => true if content is an image, pdf, or binary stream
+response.save('image.jpg') # => Directly saves response body to file
 response.parsed_response# => Auto-parsed JSON Hash / Array
 response['key']         # => Direct key access to parsed_response
 
+# HTML & Scraping Helpers
+response.title          # => HTML page title
+response.og_image       # => OpenGraph image URL
+response.meta_description # => Meta description
+response.extract_images # => Array of all resolved image URLs in HTML
+
 # Headers & Cookies
 response.headers['content-type'] # => Case-insensitive header access
 response.cookies['session_id']   # => Parsed Set-Cookie store
 response.history                 # => Array of redirect history metadata
 
-# Underlying execution details
-response.exit_code      # => Process exit status (0 for success)
-response.stderr         # => Stderr output from curl
-response.command        # => Array of the exact CLI arguments executed
+# Underlying execution & multi-strategy details
+response.mode_used            # => :impersonate, :android, :ios, or :curl
+response.fallback_triggered?  # => true if smart fallback was executed
+response.attempts             # => Array of execution metadata for each attempt
+response.exit_code            # => Process exit status (0 for success)
+response.stderr               # => Stderr output from curl
+response.command              # => Array of the exact CLI arguments executed
 ```
 
 ---

data/http_mimic.gemspec

@@ -14,7 +14,6 @@ Gem::Specification.new do |spec|
   spec.license       = "MIT"
 
   spec.metadata = {
-    "homepage_uri"    => "https://github.com/anxgang/http_mimic",
     "source_code_uri" => "https://github.com/anxgang/http_mimic",
     "bug_tracker_uri" => "https://github.com/anxgang/http_mimic/issues",
     "changelog_uri"   => "https://github.com/anxgang/http_mimic/blob/main/CHANGELOG.md"

data/lib/http_mimic/command_builder.rb

@@ -14,40 +14,70 @@ module HttpMimic
     ].freeze
 
     TARGET_BINARY_MAP = {
-      'chrome'       => %w[curl_chrome131 curl_chrome124 curl_chrome120 curl_chrome116 curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome116'    => %w[curl_chrome116 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome120'    => %w[curl_chrome120 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome123'    => %w[curl_chrome123 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome124'    => %w[curl_chrome124 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome131'    => %w[curl_chrome131 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome133a'   => %w[curl_chrome133a curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome136'    => %w[curl_chrome136 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome142'    => %w[curl_chrome142 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome110'    => %w[curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome107'    => %w[curl_chrome107 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome104'    => %w[curl_chrome104 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome101'    => %w[curl_chrome101 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome100'    => %w[curl_chrome100 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'chrome99'     => %w[curl_chrome99 curl-impersonate-chrome curl_chrome curl-impersonate],
-      'firefox'      => %w[curl_firefox135 curl_firefox133 curl_ff117 curl_firefox117 curl_ff109 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox135'   => %w[curl_firefox135 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox133'   => %w[curl_firefox133 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox117'   => %w[curl_ff117 curl_firefox117 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox109'   => %w[curl_ff109 curl_firefox109 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox102'   => %w[curl_ff102 curl_firefox102 curl-impersonate-ff curl_firefox curl-impersonate],
-      'firefox98'    => %w[curl_ff98 curl_firefox98 curl-impersonate-ff curl_firefox curl-impersonate],
-      'safari'       => %w[curl_safari180 curl_safari170 curl_safari155 curl_safari153 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari180'    => %w[curl_safari180 curl_safari18_0 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari170'    => %w[curl_safari170 curl_safari17_0 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari155'    => %w[curl_safari155 curl_safari15_5 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari15_5'   => %w[curl_safari155 curl_safari15_5 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari153'    => %w[curl_safari153 curl_safari15_3 curl-impersonate-safari curl_safari curl-impersonate],
-      'safari15_3'   => %w[curl_safari153 curl_safari15_3 curl-impersonate-safari curl_safari curl-impersonate],
-      'edge'         => %w[curl_edge101 curl_edge99 curl-impersonate-edge curl_edge curl-impersonate],
-      'edge101'      => %w[curl_edge101 curl-impersonate-edge curl_edge curl-impersonate],
-      'edge99'       => %w[curl_edge99 curl-impersonate-edge curl_edge curl-impersonate],
-      'tor'          => %w[curl_tor145 curl_tor curl-impersonate],
-      'tor145'       => %w[curl_tor145 curl_tor curl-impersonate]
+      # Chrome Desktop
+      'chrome'            => %w[curl_chrome131 curl_chrome124 curl_chrome120 curl_chrome116 curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome116'         => %w[curl_chrome116 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome120'         => %w[curl_chrome120 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome123'         => %w[curl_chrome123 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome124'         => %w[curl_chrome124 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome131'         => %w[curl_chrome131 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome133a'        => %w[curl_chrome133a curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome136'         => %w[curl_chrome136 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome142'         => %w[curl_chrome142 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome145'         => %w[curl_chrome145 curl_chrome146 curl_chrome150 curl_chrome131 curl-impersonate],
+      'chrome146'         => %w[curl_chrome146 curl_chrome150 curl_chrome145 curl_chrome131 curl-impersonate],
+      'chrome150'         => %w[curl_chrome150 curl_chrome146 curl_chrome145 curl_chrome131 curl-impersonate],
+      'chrome110'         => %w[curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome107'         => %w[curl_chrome107 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome104'         => %w[curl_chrome104 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome101'         => %w[curl_chrome101 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome100'         => %w[curl_chrome100 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome99'          => %w[curl_chrome99 curl-impersonate-chrome curl_chrome curl-impersonate],
+
+      # Mobile / Android
+      'chrome131_android' => %w[curl_chrome131_android curl_chrome131 curl-impersonate-chrome curl-impersonate],
+      'chrome99_android'  => %w[curl_chrome99_android curl_chrome99 curl-impersonate-chrome curl-impersonate],
+      'android'           => %w[curl_chrome131_android curl_safari180_ios curl_chrome99_android curl_chrome131 curl-impersonate],
+      'chrome_android'    => %w[curl_chrome131_android curl_chrome99_android curl_chrome131 curl-impersonate],
+
+      # Firefox
+      'firefox'           => %w[curl_firefox135 curl_firefox133 curl_ff117 curl_firefox117 curl_ff109 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox147'        => %w[curl_firefox147 curl_firefox144 curl_firefox135 curl-impersonate-ff curl-impersonate],
+      'firefox144'        => %w[curl_firefox144 curl_firefox147 curl_firefox135 curl-impersonate-ff curl-impersonate],
+      'firefox135'        => %w[curl_firefox135 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox133'        => %w[curl_firefox133 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox117'        => %w[curl_ff117 curl_firefox117 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox109'        => %w[curl_ff109 curl_firefox109 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox102'        => %w[curl_ff102 curl_firefox102 curl-impersonate-ff curl_firefox curl-impersonate],
+      'firefox98'         => %w[curl_ff98 curl_firefox98 curl-impersonate-ff curl_firefox curl-impersonate],
+
+      # Safari Desktop
+      'safari'            => %w[curl_safari180 curl_safari170 curl_safari184 curl_safari260 curl_safari155 curl_safari153 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari2601'        => %w[curl_safari2601 curl_safari260 curl_safari180 curl-impersonate-safari curl-impersonate],
+      'safari260'         => %w[curl_safari260 curl_safari2601 curl_safari180 curl-impersonate-safari curl-impersonate],
+      'safari184'         => %w[curl_safari184 curl_safari180 curl-impersonate-safari curl-impersonate],
+      'safari180'         => %w[curl_safari180 curl_safari18_0 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari170'         => %w[curl_safari170 curl_safari17_0 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari155'         => %w[curl_safari155 curl_safari15_5 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari15_5'        => %w[curl_safari155 curl_safari15_5 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari153'         => %w[curl_safari153 curl_safari15_3 curl-impersonate-safari curl_safari curl-impersonate],
+      'safari15_3'        => %w[curl_safari153 curl_safari15_3 curl-impersonate-safari curl_safari curl-impersonate],
+
+      # Safari iOS / Mobile
+      'safari180_ios'     => %w[curl_safari180_ios curl_safari184_ios curl_safari260_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'safari184_ios'     => %w[curl_safari184_ios curl_safari180_ios curl_safari260_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'safari260_ios'     => %w[curl_safari260_ios curl_safari184_ios curl_safari180_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'safari172_ios'     => %w[curl_safari172_ios curl_safari180_ios curl_safari184_ios curl_safari180 curl-impersonate],
+      'ios'               => %w[curl_safari180_ios curl_safari184_ios curl_safari260_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'safari_ios'        => %w[curl_safari180_ios curl_safari184_ios curl_safari260_ios curl_safari172_ios curl_safari180 curl-impersonate],
+      'mobile'            => %w[curl_safari180_ios curl_chrome131_android curl_safari184_ios curl_safari260_ios curl-impersonate],
+
+      # Edge & Tor
+      'edge'              => %w[curl_edge101 curl_edge99 curl-impersonate-edge curl_edge curl-impersonate],
+      'edge101'           => %w[curl_edge101 curl-impersonate-edge curl_edge curl-impersonate],
+      'edge99'            => %w[curl_edge99 curl-impersonate-edge curl_edge curl-impersonate],
+      'tor'               => %w[curl_tor145 curl_tor curl-impersonate],
+      'tor145'            => %w[curl_tor145 curl_tor curl-impersonate]
     }.freeze
 
     attr_reader :method, :url, :options, :config
@@ -171,6 +201,15 @@ module HttpMimic
         args << '-A' << options[:user_agent].to_s
       end
 
+      # Profile-specific headers & HTTP version defaults (for server/curl mode)
+      if options[:profile] == :curl
+        args << '--http1.1' unless (options[:curl_options] || []).to_s.include?('--http')
+        headers_to_send['User-Agent'] ||= (options[:user_agent] || 'Ruby')
+        headers_to_send['Accept'] ||= '*/*'
+        headers_to_send['Accept-Encoding'] ||= 'gzip;q=1.0,deflate;q=0.6,identity;q=0.3'
+        headers_to_send['Connection'] ||= 'close'
+      end
+
       # Append headers
       headers_to_send.each do |k, v|
         if v.is_a?(Array)
@@ -194,6 +233,13 @@ module HttpMimic
     end
 
     def resolve_binary
+      # 0. If profile is explicitly set to :curl, use system curl
+      if options[:profile] == :curl
+        curl_path = find_executable('curl') || 'curl'
+        return curl_path if executable?(curl_path)
+        raise BinaryNotFoundError, "System curl executable was not found."
+      end
+
       # 1. Use explicit binary_path from options or config if provided
       explicit = options[:binary] || config.binary_path
       if explicit
@@ -202,7 +248,16 @@ module HttpMimic
         raise BinaryNotFoundError, "Specified executable does not exist or is not executable: #{explicit}"
       end
 
-      target = (options[:impersonate] || config.default_impersonate).to_s.downcase
+      target = case options[:profile]
+               when :android
+                 (options[:android_impersonate] || 'chrome131_android').to_s.downcase
+               when :ios
+                 (options[:ios_impersonate] || 'safari260_ios').to_s.downcase
+               when :mobile
+                 (options[:mobile_impersonate] || 'chrome131_android').to_s.downcase
+               else
+                 (options[:impersonate] || config.default_impersonate).to_s.downcase
+               end
       candidate_names = TARGET_BINARY_MAP[target] || ["curl_#{target}", target]
 
       # 2. Check local installation directory (~/.http_mimic/bin)

data/lib/http_mimic/configuration.rb

@@ -2,7 +2,9 @@
 
 module HttpMimic
   class Configuration
+    # Browser simulation & request defaults
     attr_accessor :default_impersonate
+    attr_accessor :default_mobile_impersonate
     attr_accessor :binary_path
     attr_accessor :fallback_to_curl
     attr_accessor :default_timeout
@@ -13,6 +15,11 @@ module HttpMimic
     attr_accessor :logger
     attr_accessor :debug
 
+    # Multi-strategy and smart auto-fallback settings
+    attr_accessor :mode
+    attr_accessor :auto_fallback
+    attr_accessor :retry_statuses
+
     # Webdrivers-like auto download and driver management settings
     attr_accessor :auto_download
     attr_accessor :driver_version
@@ -20,16 +27,22 @@ module HttpMimic
     attr_accessor :github_repo
 
     def initialize
-      @default_impersonate      = 'chrome131'
-      @binary_path              = nil
-      @fallback_to_curl         = true
-      @default_timeout          = 30
-      @default_connect_timeout  = 10
-      @follow_redirects         = true
-      @max_redirects            = 10
-      @raise_on_error           = false
-      @logger                   = nil
-      @debug                    = false
+      @default_impersonate         = 'chrome131'
+      @default_mobile_impersonate  = 'safari180_ios'
+      @binary_path                 = nil
+      @fallback_to_curl            = true
+      @default_timeout             = 30
+      @default_connect_timeout     = 10
+      @follow_redirects            = true
+      @max_redirects               = 10
+      @raise_on_error              = false
+      @logger                      = nil
+      @debug                       = false
+
+      # Multi-strategy defaults: :auto seamlessly falls back between impersonate, mobile, and curl
+      @mode                        = :auto
+      @auto_fallback               = true
+      @retry_statuses              = [403, 429, 503]
 
       # Enable automatic downloading of curl-impersonate driver (lexiforest) by default
       @auto_download            = true

data/lib/http_mimic/module_methods.rb

@@ -41,6 +41,21 @@ module HttpMimic
       default_options[:cookies].merge!(c)
     end
 
+    def mode(m = nil)
+      return default_options[:mode] if m.nil?
+      default_options[:mode] = m
+    end
+
+    def auto_fallback(enabled = nil)
+      return default_options[:auto_fallback] if enabled.nil?
+      default_options[:auto_fallback] = enabled
+    end
+
+    def retry_statuses(statuses = nil)
+      return default_options[:retry_statuses] if statuses.nil?
+      default_options[:retry_statuses] = statuses
+    end
+
     def default_options
       @default_options ||= {}
     end

data/lib/http_mimic/request.rb

@@ -14,34 +14,101 @@ module HttpMimic
     end
 
     def perform
-      builder = CommandBuilder.new(method, url, options, config)
-      binary, args, stdin_data, final_url = builder.build
-
-      full_command = [binary] + args
-
-      log_debug("Executing HttpMimic command: #{full_command.join(' ')}")
-      log_debug("Stdin data: #{stdin_data}") if stdin_data
-
-      stdout, stderr, status = execute_open3(full_command, stdin_data)
-
-      log_debug("Curl exit status: #{status.exitstatus}")
-      log_debug("Curl stderr: #{stderr}") unless stderr.empty?
-
-      response = ResponseParser.new(
-        stdout,
-        exit_status: status,
-        stderr: stderr,
-        command: full_command,
-        request_url: final_url
-      ).parse
+      mode = (options[:mode] || config.mode || :auto).to_sym
+      auto_fallback = options.fetch(:auto_fallback, config.auto_fallback)
+      retry_statuses = options[:retry_statuses] || config.retry_statuses || [403, 429, 503]
+
+      attempts = []
+      profiles_to_try = determine_profiles(mode, auto_fallback)
+
+      response = nil
+      final_status = nil
+      final_stderr = nil
+      final_command = nil
+
+      profiles_to_try.each_with_index do |profile, index|
+        current_opts = options.merge(profile: profile)
+
+        builder = CommandBuilder.new(method, url, current_opts, config)
+        binary, args, stdin_data, final_url = builder.build
+        full_command = [binary] + args
+
+        log_debug("Executing HttpMimic command (attempt #{index + 1}, profile: #{profile}): #{full_command.join(' ')}")
+        log_debug("Stdin data: #{stdin_data}") if stdin_data
+
+        stdout, stderr, status = execute_open3(full_command, stdin_data)
+
+        log_debug("Curl exit status: #{status.exitstatus}")
+        log_debug("Curl stderr: #{stderr}") unless stderr.empty?
+
+        response = ResponseParser.new(
+          stdout,
+          exit_status: status,
+          stderr: stderr,
+          command: full_command,
+          request_url: final_url
+        ).parse
+
+        response.mode_used = profile
+        response.fallback_triggered = (index > 0)
+
+        attempts << {
+          attempt: index + 1,
+          profile: profile,
+          command: full_command,
+          code: response.code,
+          exit_code: status.exitstatus,
+          success: response.success?
+        }
+        response.attempts = attempts
+
+        final_status = status
+        final_stderr = stderr
+        final_command = full_command
+
+        is_blocked = (status.exitstatus != 0) || retry_statuses.include?(response.code)
+        if !is_blocked || (index == profiles_to_try.size - 1)
+          break
+        end
 
-      handle_errors(status, stderr, full_command, response)
+        log_debug("[HttpMimic] Attempt #{index + 1} with #{profile} resulted in status #{response.code}. Triggering smart fallback to next profile...")
+      end
 
+      handle_errors(final_status, final_stderr, final_command, response)
       response
     end
 
     private
 
+    def determine_profiles(mode, auto_fallback)
+      case mode
+      when :curl, :curl_first
+        auto_fallback ? [:curl, :impersonate, :android, :ios] : [:curl]
+      when :curl_only
+        [:curl]
+      when :impersonate_only
+        [:impersonate]
+      when :mobile_only
+        [:mobile]
+      when :android_only
+        [:android]
+      when :ios_only
+        [:ios]
+      when :mobile_first
+        auto_fallback ? [:android, :ios, :impersonate, :curl] : [:mobile]
+      when :android_first
+        auto_fallback ? [:android, :ios, :impersonate, :curl] : [:android]
+      when :ios_first
+        auto_fallback ? [:ios, :android, :impersonate, :curl] : [:ios]
+      when :impersonate_first
+        auto_fallback ? [:impersonate, :android, :ios, :curl] : [:impersonate]
+      when :auto, :smart
+        auto_fallback ? [:impersonate, :android, :ios, :curl] : [:impersonate]
+      else
+        auto_fallback ? [:impersonate, :android, :ios, :curl] : [:impersonate]
+      end
+    end
+
     def execute_open3(command_array, stdin_data)
       if stdin_data
         Open3.capture3(*command_array, stdin_data: stdin_data)
@@ -54,6 +121,7 @@ module HttpMimic
 
     def handle_errors(status, stderr, command, response)
       raise_error = options.fetch(:raise_on_error, config.raise_on_error)
+      return unless status
       exit_code = status.exitstatus
 
       if exit_code != 0
@@ -72,7 +140,7 @@ module HttpMimic
         if raise_error
           raise error_class.new(message, exit_code: exit_code, stderr: stderr, command: command, response: response)
         end
-      elsif raise_error && response.error?
+      elsif raise_error && response&.error?
         raise CommandError.new("HTTP request failed (status #{response.code})", exit_code: exit_code, stderr: stderr, command: command, response: response)
       end
     end

data/lib/http_mimic/response.rb

@@ -16,22 +16,34 @@ module HttpMimic
                 :exit_code,
                 :command
 
+    attr_accessor :mode_used,
+                  :fallback_triggered,
+                  :attempts
+
     alias status code
+    alias strategy_used mode_used
 
     def initialize(attributes = {})
-      @code            = attributes[:code] || 0
-      @http_version    = attributes[:http_version]
-      @status_message  = attributes[:status_message]
-      @headers         = attributes[:headers] || Headers.new
-      @cookies         = attributes[:cookies] || Cookies.new
-      @body            = attributes[:body] || ''
-      @parsed_response = attributes[:parsed_response]
-      @raw_headers     = attributes[:raw_headers] || ''
-      @history         = attributes[:history] || []
-      @request_url     = attributes[:request_url]
-      @stderr          = attributes[:stderr] || ''
-      @exit_code       = attributes[:exit_code] || 0
-      @command         = attributes[:command]
+      @code               = attributes[:code] || 0
+      @http_version       = attributes[:http_version]
+      @status_message     = attributes[:status_message]
+      @headers            = attributes[:headers] || Headers.new
+      @cookies            = attributes[:cookies] || Cookies.new
+      @body               = attributes[:body] || ''
+      @parsed_response    = attributes[:parsed_response]
+      @raw_headers        = attributes[:raw_headers] || ''
+      @history            = attributes[:history] || []
+      @request_url        = attributes[:request_url]
+      @stderr             = attributes[:stderr] || ''
+      @exit_code          = attributes[:exit_code] || 0
+      @command            = attributes[:command]
+      @mode_used          = attributes[:mode_used] || :impersonate
+      @fallback_triggered = attributes[:fallback_triggered] || false
+      @attempts           = attributes[:attempts] || []
+    end
+
+    def fallback_triggered?
+      !!@fallback_triggered
     end
 
     def success?
@@ -55,6 +67,76 @@ module HttpMimic
       client_error? || server_error?
     end
 
+    def binary?
+      content_type = headers['content-type'].to_s.downcase
+      ResponseParser::BINARY_MIME_KEYWORDS.any? { |kw| content_type.include?(kw) } ||
+        body.to_s.b[0, 1024]&.include?("\x00".b)
+    end
+
+    def save_to_file(filepath)
+      require 'fileutils'
+      FileUtils.mkdir_p(File.dirname(filepath))
+      File.binwrite(filepath, body)
+      filepath
+    end
+    alias save save_to_file
+
+    def title
+      return nil if binary?
+      body[/<title[^>]*>(.*?)<\/title>/im, 1]&.strip
+    end
+
+    def og_image
+      return nil if binary?
+      body[/<meta\s+[^>]*property=["']og:image["'][^>]*content=["']([^"']+)["']/im, 1] ||
+        body[/<meta\s+[^>]*content=["']([^"']+)["'][^>]*property=["']og:image["']/im, 1]
+    end
+
+    def meta_description
+      return nil if binary?
+      body[/<meta\s+[^>]*name=["']description["'][^>]*content=["']([^"']+)["']/im, 1] ||
+        body[/<meta\s+[^>]*content=["']([^"']+)["'][^>]*name=["']description["']/im, 1]
+    end
+
+    def extract_images(base_url: nil)
+      return [] if binary?
+
+      base = base_url || request_url || ''
+      images = []
+
+      # 1. Match img tags (src, data-src, data-zoom-image, data-original, srcset)
+      body.scan(/<img\s+[^>]*>/i).each do |img_tag|
+        %w[src data-src data-zoom-image data-original data-high-res-src data-full-size-image-url].each do |attr|
+          if match = img_tag.match(/#{attr}=["']([^"']+)["']/i)
+            images << match[1].strip
+          end
+        end
+
+        if match = img_tag.match(/srcset=["']([^"']+)["']/i)
+          match[1].split(',').each do |item|
+            url = item.strip.split(/\s+/).first
+            images << url if url && !url.empty?
+          end
+        end
+      end
+
+      # 2. Match og:image meta tag
+      if og = og_image
+        images << og
+      end
+
+      # 3. Match raw image URLs found in document / script payloads
+      body.scan(/https?:[^\s"'<>]+\.(?:jpg|jpeg|png|webp|avif|gif)/i).each do |raw_url|
+        images << raw_url
+      end
+
+      # Clean, resolve relative URLs to absolute, and deduplicate
+      images.map do |img|
+        clean_url = img.gsub(/&amp;/, '&').strip
+        resolve_url(clean_url, base)
+      end.compact.reject(&:empty?).uniq
+    end
+
     def [](key)
       if parsed_response.is_a?(Hash) || parsed_response.is_a?(Array)
         parsed_response[key]
@@ -80,6 +162,20 @@ module HttpMimic
       "#<#{self.class.name}:0x#{object_id.to_s(16)} @code=#{code} @status_message=#{status_message.inspect} @headers=#{headers.to_h.inspect} @parsed_response=#{parsed_response.inspect}>"
     end
 
+    private
+
+    def resolve_url(url, base)
+      return nil if url.nil? || url.empty? || url.start_with?('data:', 'javascript:', 'blob:', '#')
+      return url if url =~ /\Ahttps?:\/\//i
+      return "https:#{url}" if url.start_with?('//')
+
+      return url if base.nil? || base.empty?
+      require 'uri'
+      URI.join(base, url).to_s
+    rescue URI::InvalidURIError
+      url
+    end
+
     def method_missing(method_name, *args, &block)
       if parsed_response.respond_to?(method_name)
         parsed_response.public_send(method_name, *args, &block)

data/lib/http_mimic/response_parser.rb

@@ -5,6 +5,8 @@ require 'json'
 module HttpMimic
   class ResponseParser
     HTTP_STATUS_LINE_REGEX = /\AHTTP\/(?<version>[\d\.]+)\s+(?<code>\d{3})(?:\s+(?<message>.*))?/i
+    HTTP_STATUS_LINE_B     = /\AHTTP\/(?:[\d\.]+)\s+\d{3}/i
+    BINARY_MIME_KEYWORDS   = %w[image/ audio/ video/ pdf octet-stream zip gzip tar compressed stream font wasm].freeze
 
     attr_reader :raw_output, :exit_status, :stderr, :command, :request_url
 
@@ -17,8 +19,8 @@ module HttpMimic
     end
 
     def parse
-      # Split header blocks and body
-      header_blocks, body = split_headers_and_body(@raw_output)
+      # Split header blocks and body in binary-safe manner
+      header_blocks, raw_body = split_headers_and_body(@raw_output)
 
       final_header_block = header_blocks.last || ''
       history_header_blocks = header_blocks.size > 1 ? header_blocks[0...-1] : []
@@ -39,6 +41,14 @@ module HttpMimic
         }
       end
 
+      # Process body: retain raw bytes for binary, or UTF-8 encode for text
+      content_type = headers['content-type'].to_s.downcase
+      body = if binary_content?(content_type, raw_body)
+               raw_body
+             else
+               raw_body.dup.force_encoding('UTF-8').scrub
+             end
+
       # Parse body (auto-detect JSON)
       parsed_body = parse_body(body, headers)
 
@@ -61,32 +71,36 @@ module HttpMimic
 
     private
 
-    # Split raw_output into header blocks and body by \r?\n\r?\n
+    # Split raw_output into header blocks and body in a binary-safe manner
     def split_headers_and_body(text)
-      return [[], ''] if text.nil? || text.empty?
-
-      # Normalize line endings
-      normalized = text.gsub(/\r\n/, "\n")
-      parts = normalized.split("\n\n")
+      return [[], ''.b] if text.nil? || text.empty?
 
+      remaining = text.to_s.b
       header_blocks = []
-      body_index = 0
 
-      parts.each_with_index do |part, idx|
-        trimmed = part.strip
-        if trimmed =~ HTTP_STATUS_LINE_REGEX
-          header_blocks << part
-          body_index = idx + 1
+      while remaining =~ HTTP_STATUS_LINE_B
+        crlf_idx = remaining.index("\r\n\r\n".b)
+        lf_idx   = remaining.index("\n\n".b)
+
+        break unless crlf_idx || lf_idx
+
+        if crlf_idx && lf_idx
+          delim_pos = [crlf_idx, lf_idx].min
+          delim_len = (delim_pos == crlf_idx) ? 4 : 2
+        elsif crlf_idx
+          delim_pos = crlf_idx
+          delim_len = 4
         else
-          # Once a non-HTTP status block is encountered, the rest is body
-          break
+          delim_pos = lf_idx
+          delim_len = 2
         end
-      end
 
-      # Combine remaining parts as body
-      body = parts[body_index..-1] ? parts[body_index..-1].join("\n\n") : ''
+        header_block = remaining[0...delim_pos].force_encoding('UTF-8').scrub
+        header_blocks << header_block
+        remaining = remaining[(delim_pos + delim_len)..-1] || ''.b
+      end
 
-      [header_blocks, body]
+      [header_blocks, remaining]
     end
 
     def parse_header_block(block)
@@ -129,12 +143,22 @@ module HttpMimic
       [code, http_version, status_message, headers, cookies]
     end
 
+    def binary_content?(content_type, body)
+      return true if BINARY_MIME_KEYWORDS.any? { |kw| content_type.include?(kw) }
+
+      # Check for null bytes in the initial portion of body
+      sample = body[0, 1024]
+      sample&.include?("\x00".b)
+    end
+
     def parse_body(body, headers)
       return nil if body.nil? || body.empty?
 
       content_type = headers['content-type'].to_s.downcase
+      return body if binary_content?(content_type, body)
 
-      if content_type.include?('json') || body.strip.start_with?('{', '[')
+      trimmed = body.strip
+      if content_type.include?('json') || trimmed.start_with?('{', '[')
         begin
           JSON.parse(body)
         rescue JSON::ParserError

data/lib/http_mimic/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module HttpMimic
-  VERSION = "0.3.0"
+  VERSION = "0.3.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: http_mimic
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.2
 platform: ruby
 authors:
 - anxgang
@@ -78,7 +78,6 @@ homepage: https://github.com/anxgang/http_mimic
 licenses:
 - MIT
 metadata:
-  homepage_uri: https://github.com/anxgang/http_mimic
   source_code_uri: https://github.com/anxgang/http_mimic
   bug_tracker_uri: https://github.com/anxgang/http_mimic/issues
   changelog_uri: https://github.com/anxgang/http_mimic/blob/main/CHANGELOG.md