http_mimic 0.5.3 → 0.5.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1b0b4192818428818cca7cde26b4ce90d6fa847db6e80e2b63c163dd8171bdd9
-  data.tar.gz: de6a71df66c710a30e7cb8fcbe6976653ec6242a6d29fd6959016513da4513be
+  metadata.gz: 9f042a9d602736a448c5341e306e8a46c68fcc38e5159cc881a157a52845d994
+  data.tar.gz: 03f41604be51bc6050e1778cd4411a0c3749ad04ef823c7d99839036ec2a1784
 SHA512:
-  metadata.gz: 3b4ba6f1355da36e9f1064e62b1b988200f483a58b49524e3dd5350b0c99660d0b93ab650fc9ea8b3b7aa2204f41b7e2ca51c6b439128c267d7660be77b2b644
-  data.tar.gz: dd25b3fbf25dca0d935b910cb2b0981ab77d9dbfe45cfa5b522a99bbca7081016e6323af3fc59c389ef925c6df0b4d0bbf7321c7a93df5154d0d41e328962d76
+  metadata.gz: 53fa3674cd113da005dfa6a91b46ec2481dbe0c0f8e5fefb3594bb0e84d4b7258569c519d818bcb14a10034de61308155878cfb06cd8965f3e9c7e82613a338c
+  data.tar.gz: 47696a2a7f80d0ed2a8ca81236cd9a00f1de6fa8e5ae02bfae3aa2ef2b2965169497e49ee3aec585757c7fae12ac6a63da4398b3c57b720e3ccf6917416e433b

data/CHANGELOG.md

@@ -5,6 +5,36 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.5] - 2026-09-05
+
+### Fixed
+- **QuickJS CPT & Bot Manager Telemetry Resolution (Tier 1 WAF Bypass)**:
+  - **Standard `document.cookie` CookieJar**: Replaced the plain property in `browser_context.js` with a compliant `Map`-backed CookieJar. When Akamai CPT writes `document.cookie = "bm_lso=..."`, existing session cookies (`bm_sz`, `_abck`) are preserved rather than overwritten. This fixes the critical bug where main sensor telemetry fell back to default hash `8888888` instead of the session cookie hash.
+  - **`HTMLScriptElement.src` Property Reflection**: Implemented bidirectional synchronization between `el.src` and `el.getAttribute("src")`. CPT scripts can now successfully inspect `document.currentScript.getAttribute("src")` to locate their challenge parameters and relative post endpoints.
+  - **Accurate `document.currentScript` Execution Pipeline**: `AkamaiSolver.build_driver_script` now assigns `document.currentScript` before evaluating each script block, matching real browser multi-script execution order.
+  - **Hybrid `XMLHttpRequest` with `responseURL` Support**: Retained instance-level methods (`open`, `send`) required by obfuscated Akamai scripts while maintaining prototype-level methods for DOM script hooks, and populating `responseURL` to allow `challenge.html` reload checks to succeed.
+  - **Challenge Query String Preservation in Telemetry POST**: When sending CPT PoW solutions, `AkamaiSolver` now preserves query parameters (e.g. `v=...&t=...`) from the original challenge script URL so that edge servers correctly issue validated `bm_sc` session tokens.
+- **End-to-End Akamai Verification**:
+  - `HttpMimic.get('https://www.adidas.com/om/en/samba-og-shoes/KK2371.html')` now successfully solves the CPT challenge within QuickJS and returns the complete 400KB+ product HTML without invoking Tier 2 Obscura.
+
+## [0.5.4] - 2026-09-05
+
+
+### Fixed
+- **Challenge Page Regression Prevention**:
+  - `HttpMimic::Request` no longer preserves un-bypassed WAF challenge pages (such as Akamai 200 CPT skeletons) as `best_response`. If all resolution and fallback attempts fail, true status codes (e.g. 403) are preserved instead of overwriting with skeleton 200 responses.
+- **Explicit Profile Forwarding in Fallbacks & Retries**:
+  - `determine_profiles` now respects `explicit_profile = options[:profile]`, ensuring targeted profiles (e.g., `:android`) remain active when `auto_fallback: false` and are prioritized during fallback loops.
+  - `AkamaiSolver` now explicitly forwards `profile` and concrete `impersonate` targets during final URL retries to prevent TLS/fingerprint mismatches.
+- **Mobile/Android Emulation Fidelity in QuickJS Browser Polyfill**:
+  - Fixed typo in Android platform identifier (`Linux armv8l` instead of `Linux armv81`).
+  - Mobile environments now correctly emulate zero-length plugin and mimeType lists (removing desktop-only PDF plugins).
+  - Aligned mobile screen dimensions (412x915 portrait-primary) and WebGL renderer parameters (Qualcomm Adreno 640).
+- **Default Impersonate Profile Upgraded to `chrome150`**:
+  - Upgraded default client TLS fingerprint to `chrome150` to satisfy modern Akamai edge HTTP/2 and TLS cipher requirements.
+- **Modern CORS & Fetch Headers in Sensor POSTs**:
+  - Automatically injected `Sec-Fetch-Dest: empty`, `Sec-Fetch-Mode: cors`, `Sec-Fetch-Site: same-origin`, and client hints (`sec-ch-ua`, `sec-ch-ua-mobile`, `sec-ch-ua-platform`).
+
 ## [0.5.3] - 2026-09-05
 
 ### Added

data/http_mimic.gemspec

@@ -21,7 +21,7 @@ Gem::Specification.new do |spec|
 
   spec.required_ruby_version = ">= 2.3.0"
 
-  spec.files         = Dir["lib/**/*.rb", "README.md", "CHANGELOG.md", "LICENSE.txt", "http_mimic.gemspec"]
+  spec.files         = Dir["lib/**/*.{rb,js}", "README.md", "CHANGELOG.md", "LICENSE.txt", "http_mimic.gemspec"]
   spec.require_paths = ["lib"]
 
   spec.add_development_dependency "bundler", ">= 1.17"

data/lib/http_mimic/command_builder.rb

@@ -15,7 +15,7 @@ module HttpMimic
 
     TARGET_BINARY_MAP = {
       # Chrome Desktop
-      'chrome'            => %w[curl_chrome131 curl_chrome124 curl_chrome120 curl_chrome116 curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
+      'chrome'            => %w[curl_chrome150 curl_chrome146 curl_chrome145 curl_chrome136 curl_chrome133a curl_chrome131 curl_chrome124 curl_chrome120 curl_chrome116 curl_chrome110 curl-impersonate-chrome curl_chrome curl-impersonate],
       'chrome116'         => %w[curl_chrome116 curl-impersonate-chrome curl_chrome curl-impersonate],
       'chrome120'         => %w[curl_chrome120 curl-impersonate-chrome curl_chrome curl-impersonate],
       'chrome123'         => %w[curl_chrome123 curl-impersonate-chrome curl_chrome curl-impersonate],
@@ -118,10 +118,12 @@ module HttpMimic
       end
 
       # Timeouts
-      timeout = options[:timeout] || options[:read_timeout] || config.default_timeout
+      default_max_time = options[:auto_proxy] ? 12 : config.default_timeout
+      timeout = options[:timeout] || options[:read_timeout] || default_max_time
       args << '--max-time' << timeout.to_s if timeout
 
-      connect_timeout = options[:connect_timeout] || config.default_connect_timeout
+      default_conn_timeout = options[:auto_proxy] ? 4 : config.default_connect_timeout
+      connect_timeout = options[:connect_timeout] || default_conn_timeout
       args << '--connect-timeout' << connect_timeout.to_s if connect_timeout
 
       # SSL / Insecure

data/lib/http_mimic/configuration.rb

@@ -56,7 +56,7 @@ module HttpMimic
     attr_accessor :proxy_timeout
 
     def initialize
-      @default_impersonate         = 'chrome131'
+      @default_impersonate         = 'chrome150'
       @default_mobile_impersonate  = 'safari180_ios'
       @binary_path                 = nil
       @fallback_to_curl            = true

data/lib/http_mimic/request.rb

@@ -48,8 +48,8 @@ module HttpMimic
       end
 
       attempts = []
-      profiles_to_try = determine_profiles(mode, auto_fallback)
-      waf_solve_attempted = false
+      profiles_to_try = determine_profiles(mode, auto_fallback, options[:profile])
+      waf_solve_attempts = 0
       best_response = nil
       proxy_retries_left = should_auto_proxy ? (options[:proxy_retries] || config.proxy_retries || 3) : 0
 
@@ -60,7 +60,7 @@ module HttpMimic
 
       profiles_to_try.each_with_index do |profile, index|
         # Never fall back to plain curl if a WAF challenge has already been detected/attempted
-        if profile == :curl && waf_solve_attempted
+        if profile == :curl && waf_solve_attempts > 0
           log_debug("[HttpMimic] Skipping plain :curl fallback for protected WAF target.")
           next
         end
@@ -79,25 +79,6 @@ module HttpMimic
         log_debug("Curl exit status: #{status.exitstatus}")
         log_debug("Curl stderr: #{stderr}") unless stderr.empty?
 
-        # Handle proxy failure retry when auto_proxy is enabled
-        if should_auto_proxy && current_opts[:proxy] && (status.exitstatus != 0)
-          is_proxy_err = [5, 7, 28, 35, 56].include?(status.exitstatus) ||
-                         stderr.include?('Failed to connect to') ||
-                         stderr.include?('tunneling failed') ||
-                         stderr.downcase.include?('proxy')
-
-          if is_proxy_err
-            ProxyPool.mark_dead(current_opts[:proxy])
-            if proxy_retries_left > 0
-              proxy_retries_left -= 1
-              new_proxy = ProxyPool.get
-              log_debug("[HttpMimic::ProxyPool] Proxy #{current_opts[:proxy]} failed (exit #{status.exitstatus}). Retrying with new proxy #{new_proxy} (#{proxy_retries_left} retries left)...")
-              options[:proxy] = new_proxy
-              redo
-            end
-          end
-        end
-
         response = ResponseParser.new(
           stdout,
           exit_status: status,
@@ -106,7 +87,23 @@ module HttpMimic
           request_url: final_url
         ).parse
 
-        if should_auto_proxy && current_opts[:proxy] && response.success?
+        # Handle proxy failure retry when auto_proxy is enabled
+        is_proxy_failure = should_auto_proxy && current_opts[:proxy] && proxy_failure?(status, stderr, response)
+        if is_proxy_failure
+          ProxyPool.mark_dead(current_opts[:proxy])
+          if proxy_retries_left > 0
+            proxy_retries_left -= 1
+            new_proxy = ProxyPool.get
+            log_debug("[HttpMimic::ProxyPool] Proxy #{current_opts[:proxy]} failed (exit #{status&.exitstatus}, code #{response&.code}, empty body: #{response&.body.to_s.empty?}). Retrying with new proxy #{new_proxy} (#{proxy_retries_left} retries left)...")
+            options[:proxy] = new_proxy
+            redo
+          else
+            log_debug("[HttpMimic::ProxyPool] Proxy retries exhausted (#{current_opts[:proxy]} failed). Aborting profile fallback.")
+            break
+          end
+        end
+
+        if should_auto_proxy && current_opts[:proxy] && response.success? && !is_proxy_failure
           ProxyPool.mark_alive(current_opts[:proxy])
         end
 
@@ -127,14 +124,18 @@ module HttpMimic
         final_stderr = stderr
         final_command = full_command
 
-        # Track best response so a 200 (or challenge page) isn't wiped out by a failed fallback
-        if response
-          if best_response.nil?
-            best_response = response
-          elsif response.success? && !best_response.success?
-            best_response = response
-          elsif response.code == 200 && best_response.code != 200
-            best_response = response
+        # Track best response so a 200 isn't wiped out by a failed fallback.
+        # Only accept responses from successful curl executions that are not proxy failures or WAF challenges.
+        if response && (status.nil? || status.exitstatus == 0) && !is_proxy_failure
+          is_challenge = Waf::Detector.challenge_page?(response)
+          if !is_challenge
+            if best_response.nil?
+              best_response = response
+            elsif response.success? && !best_response.success?
+              best_response = response
+            elsif response.code == 200 && best_response.code != 200
+              best_response = response
+            end
           end
         end
 
@@ -145,13 +146,13 @@ module HttpMimic
         end
 
         auto_solve_waf = options.fetch(:solve_waf, config.auto_solve_waf)
-        is_blocked = (status.exitstatus != 0) || retry_statuses.include?(response.code) || Waf::Detector.challenge_page?(response)
+        is_blocked = (status.exitstatus != 0) || is_proxy_failure || retry_statuses.include?(response.code) || Waf::Detector.challenge_page?(response)
 
-        # Only attempt WAF resolution once per request to avoid unnecessary latency on subsequent fallbacks
-        if is_blocked && auto_solve_waf && !waf_solve_attempted && method.to_s.upcase == 'GET'
+        # Only attempt WAF resolution if origin server returned challenge (not a proxy error)
+        if is_blocked && !is_proxy_failure && auto_solve_waf && waf_solve_attempts < 2 && method.to_s.upcase == 'GET'
           waf_type = Waf::Detector.detect(response)
           if waf_type
-            waf_solve_attempted = true
+            waf_solve_attempts += 1
             log_debug("[HttpMimic] Detected #{waf_type.to_s.capitalize} WAF challenge. Attempting to solve with QuickJS...")
             solved_resp = Waf.solve(url, response, current_opts)
             if solved_resp
@@ -160,7 +161,8 @@ module HttpMimic
               if response.cookies && !response.cookies.empty?
                 options[:cookies] = (options[:cookies] || {}).merge(response.cookies.to_h)
               end
-              if response.success? || (response.code == 200 && best_response&.code != 200)
+              is_challenge_res = Waf::Detector.challenge_page?(response)
+              if !is_challenge_res && (response.success? || (response.code == 200 && best_response&.code != 200))
                 best_response = response
               end
             end
@@ -175,17 +177,16 @@ module HttpMimic
       end
 
       # Preserve best response if the last attempt resulted in a regression (e.g. 403 / error after getting 200)
-      if best_response && (response.nil? || response.error? || (final_status && final_status.exitstatus != 0))
+      if best_response && !Waf::Detector.challenge_page?(best_response) && (response.nil? || response.error? || (final_status && final_status.exitstatus != 0) || Waf::Detector.challenge_page?(response))
         if best_response.success? || (best_response.code == 200 && response&.code != 200)
           response = best_response
         end
       end
 
-      # Automatic SPA Detection & Obscura rendering fallback for SPA shells & Behavioral Challenges
+      # Automatic SPA Detection & Obscura rendering fallback (strictly requires auto_render_spa: true)
       auto_render_spa = options.fetch(:auto_render_spa, config.auto_render_spa)
-      auto_solve_waf = options.fetch(:solve_waf, config.auto_solve_waf)
       should_render_spa = auto_render_spa && response && response.success? && SpaDetector.spa?(response)
-      should_render_cpt = auto_solve_waf && response && Waf::Detector.challenge_page?(response)
+      should_render_cpt = auto_render_spa && response && Waf::Detector.challenge_page?(response)
 
       if (should_render_spa || should_render_cpt) && method.to_s.upcase == 'GET'
         target_reason = should_render_cpt ? 'WAF challenge page' : 'unhydrated SPA shell'
@@ -231,13 +232,34 @@ module HttpMimic
         end
       end
 
+      # If the final response was an intercepted dummy 200 from a broken/intercepting proxy, override code
+      if response && should_auto_proxy && proxy_failure?(final_status, final_stderr, response)
+        if response.code == 200 && response.body.to_s.empty?
+          response.instance_variable_set(:@code, 0)
+          response.instance_variable_set(:@status_message, 'Proxy Interception Failure (Empty Body)')
+        end
+      end
+
+      # If the final response is an un-bypassed WAF challenge page, annotate status message so it's clear
+      if response && response.challenge_page? && (response.status_message.to_s.empty? || response.status_message == 'OK')
+        type_str = response.challenge_type ? response.challenge_type.to_s.capitalize : 'WAF'
+        response.instance_variable_set(:@status_message, "Challenge Required (#{type_str})")
+      end
+
       handle_errors(final_status, final_stderr, final_command, response)
       response
     end
 
     private
 
-    def determine_profiles(mode, auto_fallback)
+    def determine_profiles(mode, auto_fallback, explicit_profile = nil)
+      if explicit_profile
+        p_sym = explicit_profile.to_sym
+        return [p_sym] unless auto_fallback
+        defaults = [:impersonate, :android, :ios, :firefox, :safari, :curl]
+        return ([p_sym] + defaults).uniq
+      end
+
       case mode
       when :curl, :curl_first
         auto_fallback ? [:curl, :impersonate, :android, :ios, :firefox] : [:curl]
@@ -270,6 +292,37 @@ module HttpMimic
       end
     end
 
+    def proxy_failure?(status, stderr, response)
+      # 1. Non-zero exit status from curl when using proxy (connection failed, SSL error, empty reply, etc.)
+      return true if status.nil? || status.exitstatus != 0
+
+      # 2. Origin server did not return a valid response (e.g. only CONNECT tunnel block or code 0)
+      return true if response.nil? || response.code == 0
+
+      # 3. Known proxy-level error codes
+      return true if [407, 502, 503, 504].include?(response.code)
+
+      # 4. Proxy fake 200 OK interception detection:
+      # A GET request that returns 200 with empty body is almost always a proxy interception / dummy response
+      if method.to_s.upcase == 'GET' && response.code == 200 && response.body.to_s.empty?
+        server_hdr  = response.headers['server'].to_s.downcase
+        via_hdr     = response.headers['via'].to_s.downcase
+        content_len = response.headers['content-length']
+
+        is_proxy_marker = server_hdr.include?('proxy') ||
+                          server_hdr.include?('squid') ||
+                          server_hdr.include?('console') ||
+                          server_hdr.include?('privoxy') ||
+                          server_hdr.include?('tinyproxy') ||
+                          via_hdr.include?('proxy') ||
+                          content_len == '0'
+
+        return true if is_proxy_marker
+      end
+
+      false
+    end
+
     def execute_open3(command_array, stdin_data)
       if stdin_data
         Open3.capture3(*command_array, stdin_data: stdin_data)

data/lib/http_mimic/response.rb

@@ -46,8 +46,26 @@ module HttpMimic
       !!@fallback_triggered
     end
 
+    def challenge_page?
+      if defined?(HttpMimic::Waf::Detector)
+        @challenge_page ||= HttpMimic::Waf::Detector.challenge_page?(self)
+      else
+        false
+      end
+    end
+    alias challenge? challenge_page?
+    alias blocked? challenge_page?
+
+    def challenge_type
+      if defined?(HttpMimic::Waf::Detector)
+        @challenge_type ||= HttpMimic::Waf::Detector.detect(self)
+      else
+        nil
+      end
+    end
+
     def success?
-      code >= 200 && code < 300
+      exit_code == 0 && (code >= 200 && code < 300) && !challenge_page?
     end
     alias ok? success?
 
@@ -64,7 +82,7 @@ module HttpMimic
     end
 
     def error?
-      client_error? || server_error?
+      exit_code != 0 || client_error? || server_error? || challenge_page?
     end
 
     def binary?
@@ -159,7 +177,8 @@ module HttpMimic
     end
 
     def inspect
-      "#<#{self.class.name}:0x#{object_id.to_s(16)} @code=#{code} @status_message=#{status_message.inspect} @headers=#{headers.to_h.inspect} @parsed_response=#{parsed_response.inspect}>"
+      challenge_info = challenge_page? ? " @challenge=true (#{challenge_type || 'WAF'})" : ""
+      "#<#{self.class.name}:0x#{object_id.to_s(16)} @code=#{code}#{challenge_info} @status_message=#{status_message.inspect} @headers=#{headers.to_h.inspect} @parsed_response=#{parsed_response.inspect}>"
     end
 
     private

data/lib/http_mimic/response_parser.rb

@@ -6,6 +6,7 @@ module HttpMimic
   class ResponseParser
     HTTP_STATUS_LINE_REGEX = /\AHTTP\/(?<version>[\d\.]+)\s+(?<code>\d{3})(?:\s+(?<message>.*))?/i
     HTTP_STATUS_LINE_B     = /\AHTTP\/(?:[\d\.]+)\s+\d{3}/i
+    PROXY_CONNECT_REGEX    = /\AHTTP\/[\d\.]+\s+200\s+Connection\s+established/i
     BINARY_MIME_KEYWORDS   = %w[image/ audio/ video/ pdf octet-stream zip gzip tar compressed stream font wasm].freeze
 
     attr_reader :raw_output, :exit_status, :stderr, :command, :request_url
@@ -22,6 +23,11 @@ module HttpMimic
       # Split header blocks and body in binary-safe manner
       header_blocks, raw_body = split_headers_and_body(@raw_output)
 
+      # Strip proxy CONNECT tunnel handshake blocks (e.g. "HTTP/1.1 200 Connection established")
+      while header_blocks.first && header_blocks.first =~ PROXY_CONNECT_REGEX
+        header_blocks.shift
+      end
+
       final_header_block = header_blocks.last || ''
       history_header_blocks = header_blocks.size > 1 ? header_blocks[0...-1] : []
 

data/lib/http_mimic/spa_detector.rb

@@ -25,7 +25,7 @@ module HttpMimic
       # @param response [HttpMimic::Response]
       # @return [Boolean]
       def spa?(response)
-        return false unless response && response.success?
+        return false unless response && (response.code >= 200 && response.code < 300)
         return false if response.respond_to?(:binary?) && response.binary?
 
         content_type = response.headers['content-type'].to_s.downcase

data/lib/http_mimic/version.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
 module HttpMimic
-  VERSION = "0.5.3"
+  VERSION = "0.5.5"
 end
+