fetch_util 0.5.0 → 0.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1ede7613063d932c011dcc3923412b3b41bfbdeadab49df501af635b0b7ffbd0
-  data.tar.gz: 5bd369cd22d7ca45c63046f707b12f09ad817d3b71267b3f5ccfd83455fb2c81
+  metadata.gz: 6d9dd7f8e6ba30f2d7ad5c9818fc214add21a18baa69f0e1081fe124d7781c35
+  data.tar.gz: 22e9bbc8ad10f85745e2791f196ce2633f4b0a795c95ac94f867ef3696d2053f
 SHA512:
-  metadata.gz: 5103c8780482705beb59a40e16f31e3635f0f6512dffe3522ef7996b79652a45e81e00c716f085b2bacf85685dcd61f717477310864e5459a895cc7b5e9312ca
-  data.tar.gz: 9118e9c1d177fbaf92bb19fac644b37419dee8712cd839586a350fe1f09b4f3434f9334c20a694693c0a2f6af9e57cef1f9aa1f61a5017c91c6a13e3344d8932
+  metadata.gz: f0115bb99e4c882fd46a06ac57b8350aaa464e651b5f7fb3f38e869a25b48b6ddde29a96a1f0fa955805c3de5459f436cacea8180dbe8246c662b896c2a5c991
+  data.tar.gz: 1813278203b73f64ce2bb26e2936ee42374d7b72fc1365622cba926d736672ac078efc5f03e6972703d47616f531483d74ad4e75c444c6665bef246186798af1

data/CHANGELOG.md

@@ -2,6 +2,25 @@
 
 ## Unreleased
 
+## v0.5.1 - 2026-07-15
+
+### Added
+
+- Allow browser fetches to wait within the configured timeout for site-delivered Anubis JavaScript/WebAssembly proof-of-work to resolve, while retaining explicit interstitial results when it does not.
+- Add Yahoo as a direct search source with source-specific organic-card parsing and safe result-wrapper decoding.
+
+### Fixed
+
+- Retry eligible Yahoo transport, HTTP 429, and HTTP 5xx failures up to two times after the initial request under the existing search deadline.
+- Preserve candidates when source relevance is uncertain, including valid later-ranked evidence, and normalize scoped, negated, snake_case, and camelCase query evidence without turning uncertainty into an empty query.
+- Stabilize default quoted, delimiter-bearing, and scoped searches with a healthy Yahoo authority set while preserving explicit source unions and normal fallback aggregation.
+
+### Changed
+
+- Define agent retrieval as the primary product contract while keeping custom solvers, CAPTCHA handling, access circumvention, and high-throughput scraping behavior out of scope.
+- Clarify that direct HTTP search challenges are diagnosed rather than executed.
+- Change the default direct search sources to Brave, Bing, and Yahoo.
+
 ## v0.5.0 - 2026-07-13
 
 ### Added

data/README.md

@@ -1,9 +1,11 @@
 # fetch_util
 
-Reliable browser-backed fetching for Ruby.
+Reliable browser-backed fetching for agents and Ruby applications.
 
 `fetch_util` renders modern pages, inspects the live DOM, classifies page shape, and returns compact markdown plus structured metadata.
 
+Its primary product contract is agent retrieval: use it to discover and fetch public web information, then inspect structured warnings when access is incomplete or ambiguous.
+
 It also provides a plain-Ruby regulatory inspector for machine-readable crawl, index, and text-and-data-mining signals such as `robots.txt`, `X-Robots-Tag`, robots meta tags, and TDM reservation metadata.
 
 It helps applications distinguish between content pages and access/interstitial states such as consent prompts, login-required pages, and challenge screens. When original content is not available, it returns a compact summary with warnings rather than pretending the page was extracted successfully.
@@ -12,11 +14,11 @@ It helps applications distinguish between content pages and access/interstitial
 
 The easiest way to explain `fetch_util` is in three steps:
 
-- `Render` - load the page in Chromium, inspect the rendered DOM, and read page metadata.
+- `Render` - load the page in Chromium, inspect the rendered DOM, and read page metadata. If a site delivers a JavaScript or WebAssembly proof-of-work challenge, the normal browser session may execute it and preserve its resulting cookies while waiting within the configured browser timeout.
 - `Classify` - identify whether the page is an article, list/index, docs page, search result, or an interstitial/access-limited state.
 - `Shape` - return compact markdown, normalized URLs, and warning metadata so the result is usable by agents, LLM workflows, and ordinary Ruby applications.
 
-In short: `fetch_util` makes the web easier to build on.
+This is bounded native browser execution, not a custom challenge solver. If the challenge does not resolve within the bound, the result remains an explicit interstitial with warnings.
 
 ## Installation
 
@@ -75,11 +77,15 @@ fetch_util regulatory https://example.com/article --sources=machine,human
 
 ### Search
 
-Search uses direct HTTP requests to the supported sources, in parallel, rather than the browser fetcher. The default sources are `brave` and `bing`; explicit `--source` values may be any of `brave`, `bing`, `duckduckgo`, `google`, or `ecosia`.
+Search uses direct HTTP requests to the supported sources in parallel. The default sources are `brave`, `bing`, and `yahoo`; explicit `--source` values may be any of `brave`, `bing`, `duckduckgo`, `google`, `ecosia`, or `yahoo`.
+
+Search always emits one JSON object. The normal payload is exactly `{ "query": ..., "results": [...] }`. Results are interleaved by source rank, deduplicated by normalized URL, and retain every eligible result unless an explicit `--limit N` is supplied. `--limit` is applied after aggregation; there is no default result cap. Known Bing, Google, DuckDuckGo, and Yahoo result wrappers are decoded before destination validation.
 
-Search always emits one JSON object. The normal payload is exactly `{ "query": ..., "results": [...] }`. Results are interleaved by source rank, deduplicated by normalized URL, and retain every eligible result unless an explicit `--limit N` is supplied. `--limit` is applied after aggregation; there is no default result cap. Known Bing, Google, and DuckDuckGo result wrappers are decoded before destination validation.
+Each search has one finite deadline shared by its source requests and parsing. After the initial Yahoo request, Yahoo may retry generic transport failures reported as `failed`, HTTP 429, or HTTP 5xx responses up to two times within that same deadline. Direct HTTP search challenges are diagnosed, not executed or bypassed. A source can be `ok`, `empty`, or `failed`; finite reasons include `challenge`, `failed`, `host`, `http_status`, `parse`, `query_mismatch`, `redirect`, `size`, and `timeout`.
 
-Each search has one finite deadline shared by its source requests and parsing. Challenges are reported, not bypassed. A source can be `ok`, `empty`, or `failed`; failure reasons include `challenge`, `failed`, `host`, `http_status`, `parse`, `query_mismatch`, `redirect`, `size`, and `timeout`. Normal source failures do not discard healthy peer results. With `--verbose-search`, the payload additionally contains ordered finite source `diagnostics`, and each result contains ordered `sources` and per-source `ranks`.
+`query_mismatch` is a candidate-bearing relevance warning, not a terminal query failure. The transport retains parsed candidates even when another source succeeds, so an uncertain lexical check cannot erase valid later-ranked evidence or force an empty query. Explicit source unions expose those retained candidates; the separate default structured-query source policy below may select Yahoo's URL set. Scoped operators and negated terms are excluded from lexical evidence, and snake_case/camelCase identifiers share token boundaries.
+
+For default structured queries containing quoted phrases, delimiters, or scoped operators, a healthy Yahoo response defines the returned URL set; matching peer URLs retain their source provenance and ranks. Explicit `--source` selections preserve the requested source union, and normal aggregation remains in effect when Yahoo is unavailable or uncertain. This source policy applies no result-count cap: every eligible authority result is returned unless the caller supplies `--limit`. With `--verbose-search`, the payload additionally contains ordered finite source `diagnostics`, and each result contains ordered `sources` and per-source `ranks`.
 
 For agent discovery, use an explicit first-pass budget, choose only the best 1-3 direct result URLs, then fetch those destinations and inspect JSON `warnings`, `suspect`, and `content_type` when needed. Add `--verbose-search` when results are empty or suspicious, or when source health matters:
 
@@ -109,8 +115,8 @@ Useful result fields:
 
 ## Common Options
 
-- `timeout:` browser timeout in seconds
-- `wait:` additional settle delay after page load
+- `timeout:` browser timeout in seconds; it is also the bounded observation budget for a delivered Anubis challenge
+- `wait:` settle interval used by applicable post-load stabilization paths; it does not control the challenge-completion budget
 - `wait_for_idle:` wait for Ferrum network idle before extraction
 - `limit:` search-only explicit maximum result count; omitted by default, search returns every result in the fetched responses
 - `idle_duration:` idle duration passed to Ferrum when `wait_for_idle` is enabled
@@ -180,14 +186,16 @@ pp FetchUtil.regulatory(
 
 - Extracts articles, list/index pages, and search pages into compact markdown.
 - Uses page classification to select extraction logic appropriate to the rendered page type.
-- Detects consent prompts, login-required pages, and challenge/interstitial screens and reports them with concise summaries and warning tags.
+- Detects consent prompts, login-required pages, and challenge/interstitial screens and reports them with concise summaries and warning tags. A delivered JavaScript/WebAssembly proof-of-work may complete in the normal browser session within `timeout`; unresolved challenges remain explicit interstitials.
 - Cleans up docs/reference pages aggressively enough for agent consumption.
 - Preserves `final_url`, `canonical_url`, and warning metadata so callers can reason about redirects, mismatches, and interstitials.
 - Extracts regulatory allow/disallow signals from `robots.txt`, page headers/meta tags, and TDM reservation metadata without caching raw page bodies.
 
 ## Compliance Boundaries
 
-`fetch_util` is for rendering and summarizing publicly delivered page output. It may identify consent prompts, login-required pages, and challenge/interstitial states and return warning metadata for them. It is not intended to bypass account requirements, paywalls, verification systems, or other access controls.
+`fetch_util` is primarily an agent retrieval tool for rendering and summarizing publicly delivered page output. Browser fetch may allow a site's delivered JavaScript/WebAssembly proof-of-work, such as Anubis, to complete within the configured browser timeout and normal cookie session. This is bounded native browser execution, not a custom solver.
+
+Unresolved challenges remain explicit interstitials. CAPTCHA solving, account or paywall circumvention, identity deception, and high-throughput scraping optimization are out of scope. Direct HTTP search challenges are diagnosed rather than executed.
 
 Browser-profile normalization is intentionally limited to reducing obvious runtime inconsistencies that would otherwise change page behavior during extraction.
 

data/SKILL.md

@@ -10,6 +10,7 @@ Use this skill for agent web retrieval.
 
 Start by loading this skill, then use `fetch_util` as the default first tool for ordinary web-page retrieval and search.
 Use it first because it usually returns cleaner, more agent-friendly markdown than generic web fetch, and it is cheap enough for repeated multi-pass retrieval.
+`fetch_util` is primarily an agent retrieval tool: prefer bounded access to public information and inspect warnings when content is incomplete.
 
 ## Default Rule
 
@@ -27,6 +28,13 @@ Use it first because it usually returns cleaner, more agent-friendly markdown th
 - searching the web and returning compact agent-friendly results
 - getting cleaner markdown from noisy pages, docs, search pages, or light interstitials
 
+## Browser Challenges
+
+- Browser fetch may let the site's delivered JavaScript or WebAssembly proof-of-work, such as Anubis, complete within the configured browser `timeout` and the normal cookie session.
+- This is bounded native browser execution, not a custom solver. `timeout` supplies the challenge-observation budget; `wait` only affects applicable post-load stabilization paths.
+- If the challenge remains unresolved, treat the result as an explicit interstitial and inspect `warnings`, `suspect`, and `content_type`.
+- CAPTCHA solving, account or paywall circumvention, identity deception, and high-throughput scraping optimization are out of scope.
+
 ## Core Commands
 
 Fetch known URLs (returns pure markdown by default):
@@ -62,9 +70,11 @@ bundle exec exe/fetch_util search ruby language --limit 8
 - if the user gives you URLs, use `fetch_util fetch` first
 - if the user needs discovery, use `fetch_util search` first; for a context-efficient first pass, consider an explicit budget such as `--limit 8`
 - add `--verbose-search` and inspect ordered source `diagnostics` when results are empty or suspicious, or when source health matters
-- search defaults to direct HTTP Brave and Bing; explicit sources are `brave`, `bing`, `duckduckgo`, `google`, and `ecosia`
+- search defaults to direct HTTP Brave, Bing, and Yahoo; explicit sources are `brave`, `bing`, `duckduckgo`, `google`, `ecosia`, and `yahoo`
 - search always emits one JSON object and normally returns exactly `{query, results}`; `--verbose-search` adds source diagnostics plus per-result source provenance and ranks
-- search has one finite shared source deadline, does not bypass challenges, decodes known engine wrappers, and preserves healthy peer results when a source fails; `empty` and `query_mismatch` are finite source outcomes
+- search has one finite shared source deadline, diagnoses direct HTTP challenges rather than executing or bypassing them, decodes known engine wrappers, and preserves healthy peer results when a source fails; after the initial request, Yahoo retries generic `failed`, HTTP 429, and HTTP 5xx outcomes up to two times within the same deadline
+- `query_mismatch` retains candidate evidence at the transport boundary instead of forcing an empty query; explicit source unions expose it, while the separate default structured-query policy may select Yahoo's URL set
+- for default quoted, delimiter-bearing, or scoped queries, a healthy Yahoo result set defines returned URLs while matching peers retain provenance; explicit `--source` selections preserve the requested union
 - after search, select only 1-3 direct result URLs, then run `fetch_util fetch` on those destinations; use `--format json` or `--format jsonl` and inspect `warnings`, `suspect`, and `content_type` as needed
 - if the task is a normal web roundup (for example, checking several news homepages), still use `fetch_util` first; do not skip straight to built-in web fetch just because the URLs are already known
 - if you are in a subagent without the `skill` tool, treat `fetch_util` as a normal installed CLI and call it directly

data/lib/fetch_util/browser/stabilization/anubis.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+module FetchUtil
+  class Browser
+    module Stabilization
+      module Anubis
+        ANUBIS_POLL = 0.25
+
+        private
+
+        def wait_for_anubis_challenge(page)
+          deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @timeout
+          state = anubis_page_state(page)
+          if !valid_anubis_state?(state) && Process.clock_gettime(Process::CLOCK_MONOTONIC) < deadline
+            state = anubis_page_state(page)
+          end
+          return false unless valid_anubis_state?(state) && state["challenge"] == true
+
+          remaining = deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          return false unless remaining.positive?
+
+          retry_until_timeout(remaining, interval: ANUBIS_POLL) do
+            state = anubis_page_state(page)
+            next false unless valid_anubis_state?(state)
+            next false unless state["challenge"] == false
+
+            state["document_ready"] == true && state["body_present"] == true && state["body_text_present"] == true
+          end
+        end
+
+        def valid_anubis_state?(state)
+          state.is_a?(Hash) &&
+            [true, false].include?(state["challenge"]) &&
+            [true, false].include?(state["document_ready"]) &&
+            [true, false].include?(state["body_present"]) &&
+            [true, false].include?(state["body_text_present"]) &&
+            state["url"].is_a?(String) && !state["url"].empty?
+        end
+
+        def anubis_page_state(page)
+          safe_evaluate(page, <<~JS, default: {})
+            (() => {
+              const body = document.body;
+              const bodyText = body ? (body.innerText || '').replace(/\s+/g, ' ').trim() : '';
+              const anubisRoot = document.querySelector('#anubis_challenge');
+              const anubisScript = document.querySelector('script[src*="/.within.website/x/cmd/anubis/"]');
+              const visibleRoot = anubisRoot && (() => {
+                const style = window.getComputedStyle(anubisRoot);
+                const rect = anubisRoot.getBoundingClientRect();
+                return style.display !== 'none' && style.visibility !== 'hidden' && rect.width > 0 && rect.height > 0;
+              })();
+              const makingSure = /making sure you're not a bot/i.test((document.title || '') + ' ' + bodyText);
+              const protectedChallenge = /protected by anubis/i.test(bodyText) &&
+                /(enable javascript to get past this challenge|please wait a moment while we ensure the security of your connection|loading|calculating|challenge:\s*anubis)/i.test(bodyText);
+              const challenge = Boolean((visibleRoot || anubisScript) && (makingSure || protectedChallenge));
+              return {
+                challenge,
+                document_ready: document.readyState === 'complete',
+                body_present: Boolean(body),
+                body_text_present: bodyText.length > 0,
+                url: location.href
+              };
+            })()
+          JS
+        end
+      end
+    end
+  end
+end

data/lib/fetch_util/browser/stabilization/page_flow.rb

@@ -38,6 +38,8 @@ module FetchUtil
         private
 
         def stabilize_page(page, url)
+          wait_for_anubis_challenge(page)
+
           if (profile = matching_stabilization_profile(url, PAGE_FLOW_STABILIZATION_PROFILES))
             return send(profile.fetch(:strategy), page)
           end

data/lib/fetch_util/browser/stabilization.rb

@@ -5,9 +5,11 @@ module FetchUtil
     module Stabilization
       autoload :PageFlow, "fetch_util/browser/stabilization/page_flow"
       autoload :SpaHydration, "fetch_util/browser/stabilization/spa_hydration"
+      autoload :Anubis, "fetch_util/browser/stabilization/anubis"
 
       include PageFlow
       include SpaHydration
+      include Anubis
     end
   end
 end

data/lib/fetch_util/cli.rb

@@ -71,7 +71,7 @@ module FetchUtil
     end
 
     desc "search QUERY", "Search across configured engines and aggregate results"
-    option :source, type: :array, default: FetchUtil::Searcher::DEFAULT_SOURCES, desc: "Search sources"
+    option :source, type: :array, default: nil, desc: "Search sources"
     option :limit, type: :numeric, default: nil, desc: "Maximum results; omit to return every result in the fetched response"
     option :verbose_search, type: :boolean, default: false, desc: "Include per-result search provenance"
     def search(*terms)

data/lib/fetch_util/search_transport.rb

@@ -34,19 +34,24 @@ module FetchUtil
       "bing" => { url: "https://www.bing.com/search?q=%{query}&setlang=en-US&cc=US", hosts: %w[www.bing.com cn.bing.com] },
       "duckduckgo" => { url: "https://html.duckduckgo.com/html/?q=%{query}", hosts: %w[html.duckduckgo.com] },
       "google" => { url: "https://www.google.com/search?q=%{query}", hosts: %w[www.google.com] },
-      "ecosia" => { url: "https://www.ecosia.org/search?q=%{query}", hosts: %w[www.ecosia.org] }
+      "ecosia" => { url: "https://www.ecosia.org/search?q=%{query}", hosts: %w[www.ecosia.org] },
+      "yahoo" => { url: "https://search.yahoo.com/search?p=%{query}", hosts: %w[search.yahoo.com] }
     }.freeze
     DEFAULT_TIMEOUT = 10.0
+    YAHOO_RECOVERY_ATTEMPTS = 2
     FAILURE_REASONS = %w[challenge failed host http_status parse query_mismatch redirect size timeout].freeze
     RELEVANCE_HEALTH_CANDIDATE_COUNT = 3
     QUERY_FUNCTION_WORDS = %w[
       a an and are at be been being but by can could did do does for from had has have how i if in is it me my no not of on or our shall
       should that the their them then there these they this to was we were what when where which who why will with would you your
     ].freeze
+    SCOPED_QUERY_TERM = /(?:\A|\s)[+-]?[[:alpha:]][[:alnum:]_-]*:(?:"[^"]*"|'[^']*'|\S+)/
+    NEGATED_QUERY_TERM = /(?:\A|\s)-\S+/
     WRAPPER_HOSTS = {
       "bing" => %w[bing.com www.bing.com cn.bing.com],
       "duckduckgo" => %w[duckduckgo.com www.duckduckgo.com html.duckduckgo.com],
-      "google" => %w[google.com www.google.com]
+      "google" => %w[google.com www.google.com],
+      "yahoo" => %w[r.search.yahoo.com]
     }.freeze
 
     def initialize(sources: SOURCES.keys, timeout: DEFAULT_TIMEOUT, clock: nil, http_client: nil, html_parser: nil)
@@ -62,11 +67,28 @@ module FetchUtil
       @html_parser = html_parser || ->(body) { Nokogiri::HTML(body) }
     end
 
-    def search(query)
+    def self.candidates_match_query?(query, candidates)
+      candidates.first(RELEVANCE_HEALTH_CANDIDATE_COUNT).any? do |candidate|
+        candidate_matches_query?(query, candidate)
+      end
+    end
+
+    def self.candidate_matches_query?(query, candidate)
+      query_terms = meaningful_query_terms(normalized_query_text(query))
+      return true if query_terms.length < 3
+
+      candidate_terms = normalized_terms("#{candidate.title} #{candidate.snippet}")
+      (query_terms & candidate_terms).length >= 2
+    end
+
+    def search(query, timeout: @timeout)
       query = query.to_s.strip
       raise ArgumentError, "query must not be empty" if query.empty?
 
-      deadline = clock.call + timeout
+      request_timeout = Float(timeout)
+      raise ArgumentError, "timeout must be positive" unless request_timeout.positive?
+
+      deadline = clock.call + request_timeout
       responses = Array.new(sources.length)
       threads = sources.each_with_index.map do |source, index|
         Thread.new { responses[index] = search_source(source, query, deadline) }
@@ -82,8 +104,10 @@ module FetchUtil
     def search_source(source, query, deadline)
       started_at = clock.call
       url = build_url(source, query)
-      result = http_client.get(url, deadline: deadline, allowed_hosts: SOURCES.fetch(source).fetch(:hosts))
-      return failure(source, result.reason, elapsed_since(started_at), result.final_url) if result.is_a?(HttpFailure)
+      result = yahoo_response(source, url, deadline)
+      if result.is_a?(HttpFailure)
+        return failure(source, result.reason, elapsed_since(started_at), result.final_url)
+      end
       outcome = within_deadline(deadline) do
         document = html_parser.call(result.body)
         next [:failed, "challenge"] if challenge?(source, result, document)
@@ -92,13 +116,23 @@ module FetchUtil
 
         candidates = parse_candidates(source, document)
         next [:ok, candidates] if candidates.empty?
-        next [:failed, "query_mismatch"] unless query_matches_candidates?(query, candidates)
+        next [:suspect, candidates] unless self.class.candidates_match_query?(query, candidates)
 
         [:ok, candidates]
       end
       elapsed_ms = elapsed_since(started_at)
-      return failure(source, outcome.last, elapsed_ms, result.final_url) if outcome.first == :failed
+      return failure(source, outcome[1], elapsed_ms, result.final_url, candidates: outcome[2] || []) if outcome.first == :failed
       return empty(source, elapsed_ms, result.final_url) if outcome.first == :empty
+      if outcome.first == :suspect
+        return SourceResponse.new(
+          source: source,
+          status: "ok",
+          candidates: outcome.last,
+          elapsed_ms: elapsed_ms,
+          final_url: result.final_url,
+          reason: "query_mismatch"
+        )
+      end
 
       candidates = outcome.last
       return failure(source, "parse", elapsed_ms, result.final_url) if candidates.empty?
@@ -113,13 +147,31 @@ module FetchUtil
       failure(source, "failed", elapsed_since(started_at), url)
     end
 
+    def yahoo_response(source, url, deadline)
+      return http_client.get(url, deadline: deadline, allowed_hosts: SOURCES.fetch(source).fetch(:hosts)) unless source == "yahoo"
+
+      result = http_client.get(url, deadline: deadline, allowed_hosts: SOURCES.fetch(source).fetch(:hosts))
+      YAHOO_RECOVERY_ATTEMPTS.times do
+        break unless yahoo_retryable?(result) && deadline > clock.call
+
+        result = http_client.get(url, deadline: deadline, allowed_hosts: SOURCES.fetch(source).fetch(:hosts))
+      end
+      result
+    end
+
+    def yahoo_retryable?(result)
+      return result.reason == "failed" if result.is_a?(HttpFailure)
+
+      result.is_a?(HttpResponse) && (result.status == 429 || result.status.between?(500, 599))
+    end
+
     def build_url(source, query)
       SOURCES.fetch(source).fetch(:url) % { query: URI.encode_www_form_component(query) }
     end
 
     def parse_candidates(source, document)
       result_nodes(source, document).filter_map do |node|
-        next if excluded_node?(node)
+        next if excluded_node?(source, node)
 
         anchor = result_anchor(source, node)
         next unless anchor
@@ -135,37 +187,27 @@ module FetchUtil
       end
     end
 
-    def query_matches_candidates?(query, candidates)
-      normalized_query = normalized_query_text(query)
-      query_terms = meaningful_query_terms(normalized_query)
-      return true unless relevance_gate_applies?(normalized_query, query_terms)
-
-      candidates.first(RELEVANCE_HEALTH_CANDIDATE_COUNT).any? do |candidate|
-        candidate_terms = normalized_terms("#{candidate.title} #{candidate.snippet}")
-        (query_terms & candidate_terms).length >= 2
-      end
+    def self.meaningful_query_terms(query)
+      semantic_query = query.gsub(SCOPED_QUERY_TERM, " ").gsub(NEGATED_QUERY_TERM, " ")
+      normalized_terms(semantic_query).reject { |term| QUERY_FUNCTION_WORDS.include?(term) }
     end
 
-    def relevance_gate_applies?(query, query_terms)
-      return false if query_terms.length < 3
-
-      !query.match?(%r{["']|(?:\A|\s)[+-]?[[:alpha:]][[:alnum:]_-]*:|[+#/]|(?:\b[[:alpha:]]\.){2,}|\d|\b[[:upper:]]{2,}\b})
-    end
-
-    def meaningful_query_terms(query)
-      normalized_terms(query).reject { |term| QUERY_FUNCTION_WORDS.include?(term) }
+    def self.normalized_terms(text)
+      normalized = normalized_query_text(text)
+      normalized = normalized.gsub(/([[:upper:]]+)([[:upper:]][[:lower:]])/, '\1 \2')
+      normalized = normalized.gsub(/([[:lower:]\d])([[:upper:]])/, '\1 \2')
+      normalized.downcase.scan(/[[:alnum:]]+/).uniq
     end
 
-    def normalized_terms(text)
-      normalized_query_text(text).downcase.scan(/[[:alnum:]]+/).uniq
-    end
-
-    def normalized_query_text(text)
+    def self.normalized_query_text(text)
       text.to_s.encode("UTF-8", invalid: :replace, undef: :replace, replace: " ").unicode_normalize(:nfc)
     end
 
+    private_class_method :meaningful_query_terms, :normalized_terms, :normalized_query_text
+
     def result_nodes(source, document)
       return brave_result_nodes(document) if source == "brave"
+      return yahoo_result_nodes(document) if source == "yahoo"
 
       selector = {
         "bing" => "#b_results li.b_algo",
@@ -183,6 +225,11 @@ module FetchUtil
       (current.to_a + legacy).uniq
     end
 
+    def yahoo_result_nodes(document)
+      nodes = document.css("#web .algo, #web .algo-sr, #web .dd.algo").uniq
+      nodes.reject { |node| nodes.any? { |other| other != node && other.ancestors.include?(node) } }
+    end
+
     def legacy_brave_result?(node)
       node.at_css("h2, h3, .title.search-snippet-title") && result_anchor("brave", node)
     end
@@ -197,6 +244,7 @@ module FetchUtil
       when "bing" then node.at_css("h2 a[href]")
       when "duckduckgo" then node.at_css("a.result__a[href], h2 a[href]")
       when "google" then node.at_css("a[href]:has(h3)") || node.at_css("h3")&.ancestors("a[href]")&.first
+      when "yahoo" then node.at_css("h3 a[href], .compTitle a[href]")
       else node.at_css("h2 a[href], h3 a[href], a[href]:has(h3)")
       end
     end
@@ -213,21 +261,31 @@ module FetchUtil
         "bing" => ".b_caption p, .b_paractl",
         "duckduckgo" => ".result__snippet",
         "google" => ".VwiC3b, .aCOpRe, [data-sncf]",
-        "ecosia" => ".result-snippet, .result__description"
+        "ecosia" => ".result-snippet, .result__description",
+        "yahoo" => ".compText, .compText p"
       }.fetch(source)
     end
 
-    def excluded_node?(node)
+    def excluded_node?(source, node)
+      return yahoo_excluded_node?(node) if source == "yahoo"
+
       node.xpath("ancestor-or-self::*").any? do |ancestor|
         value = [ancestor["class"], ancestor["id"], ancestor["data-testid"]].compact.join(" ").downcase
         value.match?(/\b(ad|ads|advert|enrichment|knowledge|llm|nav|pagination|related|answer)\b/)
       end
     end
 
+    def yahoo_excluded_node?(node)
+      node.xpath("ancestor-or-self::*").any? do |ancestor|
+        value = [ancestor["class"], ancestor["id"], ancestor["data-testid"]].compact.join(" ").downcase
+        value.match?(/\b(ad|ads|advert|control|controls|enrichment|knowledge|llm|lookalike|nav|navigation|pagination|related|right[-_ ]?rail|assist)\b/)
+      end
+    end
+
     def destination(source, href)
       value = href.to_s.strip
       value = decode_bing(value) if source == "bing"
-      value = decode_wrapper(source, value) if %w[google duckduckgo].include?(source)
+      value = decode_wrapper(source, value) if %w[google duckduckgo yahoo].include?(source)
       uri = URI.parse(value)
       return unless uri.is_a?(URI::HTTP) && uri.host
 
@@ -252,6 +310,8 @@ module FetchUtil
 
     def decode_wrapper(source, value)
       uri = URI.parse(value)
+      return decode_yahoo_wrapper(uri, value) if source == "yahoo"
+
       key = source == "google" ? %w[q url] : %w[uddg]
       path = wrapper_path(source)
       return value unless uri.path == path && (uri.host.nil? || wrapper_host?(source, uri.host))
@@ -261,6 +321,21 @@ module FetchUtil
       value
     end
 
+    def decode_yahoo_wrapper(uri, value)
+      return value unless wrapper_host?("yahoo", uri.host)
+
+      segment = uri.path.split("/").find { |part| part.start_with?("RU=") }
+      return value unless segment
+
+      decoded = URI::DEFAULT_PARSER.unescape(segment.delete_prefix("RU="))
+      destination = URI.parse(decoded)
+      return nil unless destination.is_a?(URI::HTTP) && destination.host
+
+      destination.to_s
+    rescue URI::InvalidURIError
+      nil
+    end
+
     def engine_url?(source, value)
       host = URI.parse(value).host
       same_engine_host?(source, host) || wrapper_host?(source, host)
@@ -302,7 +377,8 @@ module FetchUtil
         "bing" => /there are no results|no results found/,
         "duckduckgo" => /no results|no more results/,
         "google" => /did not match any documents|no results found/,
-        "ecosia" => /no results found|we couldn't find/
+        "ecosia" => /no results found|we couldn't find/,
+        "yahoo" => /no results found|we couldn't find|did not match any results/
       }
       text.match?(patterns.fetch(source))
     end
@@ -313,10 +389,17 @@ module FetchUtil
       normalized_text(visible).downcase
     end
 
-    def failure(source, reason, elapsed_ms, final_url)
+    def failure(source, reason, elapsed_ms, final_url, candidates: [])
       reason = reason.to_s
       reason = "failed" unless FAILURE_REASONS.include?(reason)
-      SourceResponse.new(source: source, status: "failed", elapsed_ms: elapsed_ms, final_url: final_url, reason: reason)
+      SourceResponse.new(
+        source: source,
+        status: "failed",
+        candidates: candidates,
+        elapsed_ms: elapsed_ms,
+        final_url: final_url,
+        reason: reason
+      )
     end
 
     def empty(source, elapsed_ms, final_url)

data/lib/fetch_util/searcher.rb

@@ -5,8 +5,9 @@ require "uri"
 
 module FetchUtil
   class Searcher
-    DEFAULT_SOURCES = %w[brave bing].freeze
-
+    DEFAULT_SOURCES = %w[brave bing yahoo].freeze
+    STRUCTURED_QUERY_AUTHORITY = "yahoo"
+    STRUCTURED_QUERY = /"[^"]+"|'[^']+'|\(|\)|\[|\]|\{|\}|(?:\A|\s)[+-]?[[:alpha:]][[:alnum:]_-]*:/
     autoload :ResultFiltering, "fetch_util/searcher/result_filtering"
     include ResultFiltering
     private_constant :ResultFiltering
@@ -14,6 +15,7 @@ module FetchUtil
     def initialize(transport: nil, request_log: RequestLog.new, sources: nil, limit: nil, verbose: false,
                    timeout: SearchTransport::DEFAULT_TIMEOUT)
       @request_log = request_log
+      @sources_explicit = !sources.nil?
       @sources = Array(sources || DEFAULT_SOURCES).map(&:to_s).uniq
       unknown = @sources - SearchTransport::SOURCES.keys
       raise ArgumentError, "unsupported search source: #{unknown.first}" if unknown.any?
@@ -33,7 +35,7 @@ module FetchUtil
 
       payload = {
         query: encoded_query,
-        results: formatted_results(apply_limit(aggregate(responses)))
+        results: formatted_results(apply_limit(aggregate(responses, encoded_query)))
       }
       payload[:diagnostics] = diagnostics(responses) if @verbose
       payload
@@ -58,17 +60,18 @@ module FetchUtil
       "search://#{@sources.join(",")}?q=#{CGI.escape(query)}"
     end
 
-    def aggregate(responses)
+    def aggregate(responses, query)
       parsed = {}
-      max_size = 0
+      structured_query = query.match?(STRUCTURED_QUERY)
 
       @sources.each do |source|
         response = responses.find { |item| item.source == source }
-        items = response ? response.candidates.filter_map { |candidate| normalized_candidate(candidate) } : []
-        parsed[source] = items
-        max_size = [max_size, items.length].max
+        parsed[source] = response ? response.candidates.filter_map { |candidate| normalized_candidate(candidate) } : []
       end
 
+      apply_structured_query_authority!(parsed, responses) if structured_query
+      max_size = parsed.values.map(&:length).max || 0
+
       items = []
       seen = {}
 
@@ -92,6 +95,23 @@ module FetchUtil
       items
     end
 
+    def apply_structured_query_authority!(parsed, responses)
+      return if @sources_explicit
+      return unless @sources.include?(STRUCTURED_QUERY_AUTHORITY)
+
+      authority = responses.find { |response| response.source == STRUCTURED_QUERY_AUTHORITY }
+      return unless authority&.status == "ok" && authority.reason.nil?
+
+      authority_urls = parsed.fetch(STRUCTURED_QUERY_AUTHORITY).map { |item| item[:url] }
+      return if authority_urls.empty?
+
+      @sources.each do |source|
+        next if source == STRUCTURED_QUERY_AUTHORITY
+
+        parsed[source].select! { |item| authority_urls.include?(item[:url]) }
+      end
+    end
+
     def build_result(source, item)
       result = {
         title: item[:title],

data/lib/fetch_util/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module FetchUtil
-  VERSION = "0.5.0"
+  VERSION = "0.5.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fetch_util
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.5.1
 platform: ruby
 authors:
 - hmdne
@@ -86,6 +86,7 @@ files:
 - lib/fetch_util/browser/site_stabilization/social_platforms.rb
 - lib/fetch_util/browser/site_stabilization/travel_and_lodging.rb
 - lib/fetch_util/browser/stabilization.rb
+- lib/fetch_util/browser/stabilization/anubis.rb
 - lib/fetch_util/browser/stabilization/page_flow.rb
 - lib/fetch_util/browser/stabilization/spa_hydration.rb
 - lib/fetch_util/cli.rb