ask-web-search 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8843934817bb67da278cc649beebc5beb3ed6af85653209511108f72f06b1da5
-  data.tar.gz: c2f2ab3b68106eb47806824b7e8d86153cf934159024535be0c7eb3b3bbc4f5b
+  metadata.gz: 0e72b7a5d69234d5b2a98512629e00c7887e690abf6cf312f02f24c092f9e464
+  data.tar.gz: 37348a846dbf38c41f41df76d52b475d6fb94e29a0c70033b70bc6ca43b743fa
 SHA512:
-  metadata.gz: df4ec625002df7c953171cea42654a4b4b5ab10fa365cea9ab149991100913d5b8982d1d14d1204a1a596f00c847dc1cb95714309383cb3496913aa7d8137c78
-  data.tar.gz: d1212989bbf9cc31e15acd4f43153fff0c57c23038ccb46a4c557955d62d129ed9ecd0cfaf21c92508d1b985cbb5742dd1849cc2dbc235fa6f49b9109ddfd8a8
+  metadata.gz: 489ae3316e2c88daf33cd31a478ec6a5e774fcd802a4b0c746577934af5db49fe94de9ffa12f1aa9ccde625e81129bc3d86263c9bc5ba259bf602c047153f046
+  data.tar.gz: c99af3305cc4b3417dcfc15bf5f3752d55823fd8e73baac2923431b479ac23cfdee6f3f4d81789a88d2faa9aa3216c95d2c2254dde57dea2f3b5aaaa830728a8

data/README.md

@@ -3,23 +3,45 @@
 [![Gem Version](https://badge.fury.io/rb/ask-web-search.svg)](https://badge.fury.io/rb/ask-web-search)
 
 A web search tool for the ask-rb ecosystem. It provides
-`Ask::Tools::WebSearch`, which searches the web via a local
-[SearXNG](https://docs.searxng.org/) instance and returns numbered markdown
-results for LLM consumption. It has no Rails dependencies; it depends only on
-ask-tools.
+`Ask::Tools::WebSearch`, which searches the web and returns numbered
+markdown results for LLM consumption. It has no Rails dependencies; it
+depends only on ask-tools.
 
-## Prerequisites
+## Backends
 
-A running SearXNG instance. The default is `http://localhost:8888`.
+Two interchangeable backends, chosen automatically at call time:
 
-Start one with Docker:
+1. **TinyFish (recommended — no self-hosting).** Get a free API key at
+   [agent.tinyfish.ai](https://agent.tinyfish.ai/api-keys) and export it:
 
-```sh
-docker run -d --name searxng -p 8888:8080 searxng/searxng
-```
+   ```sh
+   export TINYFISH_API_KEY=...
+   ```
+
+   That's the whole setup — no Docker, no SearXNG instance. Live
+   browser-rendered results, freshness windows, and the news /
+   research-paper verticals all work out of the box.
+
+2. **SearXNG (self-hosted).** For users who want queries to stay local,
+   and as the automatic fallback when TinyFish fails: a running
+   [SearXNG](https://docs.searxng.org/) instance, default
+   `http://localhost:8888`. Start one with Docker:
+
+   ```sh
+   docker run -d --name searxng -p 8888:8080 searxng/searxng
+   ```
+
+   Or use the provided `docker-compose.yml` in the `searxng` directory of
+   this repository.
+
+Routing rules:
 
-Or use the provided `docker-compose.yml` in the `searxng` directory of this
-repository:
+- `TINYFISH_API_KEY` set → TinyFish primary; if TinyFish fails **and** a
+  SearXNG endpoint is explicitly configured, the call falls back to
+  SearXNG automatically (combined errors surface both failures).
+- No key → SearXNG only, exactly as in 0.6.x.
+- `TINYFISH_SEARCH=0` disables TinyFish outright (SearXNG only, even
+  with a key).
 
 ## Installation
 
@@ -29,13 +51,14 @@ gem "ask-web-search"
 
 ## Configuration
 
-Set the `SEARXNG_URL` environment variable to point to your SearXNG instance:
-
 ```sh
-export SEARXNG_URL=http://localhost:8888
+export TINYFISH_API_KEY=...   # TinyFish (recommended): free key, instant search
+export SEARXNG_URL=http://localhost:8888   # SearXNG endpoint (default shown)
 ```
 
-Defaults to `http://localhost:8888`.
+The SearXNG endpoint can also be set in code: `Ask::WebSearch.searxng_url=`.
+TinyFish reads `TINYFISH_API_KEY` from the environment only.
+`Ask::WebSearch.max_retries = 0` disables the retry-with-backoff loop.
 
 ## Quick Start
 
@@ -72,27 +95,65 @@ Ask::WebSearch.search("fed policy", categories: "news")
 Ask::WebSearch.search("retrieval augmented generation", categories: "science")
 ```
 
-- **`time_range:`** restricts results to a SearXNG freshness window —
-  `day`, `week`, `month`, or `year`. Anything else raises `ArgumentError`
+- **`time_range:`** restricts results to a freshness window — `day`,
+  `week`, `month`, or `year`. Anything else raises `ArgumentError`
   naming the valid values. When a windowed search comes back cleanly
   empty, the result says so — "No results found within the day freshness
   window. Retry with a broader time_range or without one." — instead of a
   bare "No results found.", and engine-failure diagnostics suggest a
-  broader window. Caveat: SearXNG delegates date filtering to the engines,
-  and as of SearXNG 2026.6.x every date-capable engine returns zero
-  results under a date filter, so windows currently fail soft with that
-  message. No gem change is needed when the engines are fixed upstream.
-- **`categories:`** scopes the search to a SearXNG category — `news`,
-  `science` (research papers), or any category the instance configures.
-  Accepts a string or anything Array-able (`[:news, :science]` →
-  `news,science`). Values pass through unvalidated: instances configure
-  their own category set, and an unknown category degrades to a clean
-  empty result rather than a failure. The instance must have vertical
-  engines enabled — the `searxng/` compose config in this repository
-  enables bing news + google news (news) and arxiv + pubmed (science);
-  without them SearXNG silently resolves the request against the general
+  broader window. TinyFish maps this to its `recency_minutes` and
+  honors it properly. Caveat on the **SearXNG backend**: SearXNG
+  delegates date filtering to the engines, and as of SearXNG 2026.6.x
+  every date-capable engine returns zero results under a date filter, so
+  windows there fail soft with that message. No gem change is needed
+  when the engines are fixed upstream.
+- **`categories:`** scopes the search to a vertical — `news`, `science`
+  (research papers), or general web (default). Accepts a string or
+  anything Array-able (`[:news, :science]` → `news,science`). TinyFish
+  maps the first recognized value to its `domain_type`
+  (general→web, news→news, science→research_paper; unknown categories
+  are omitted so TinyFish defaults to web). On the **SearXNG backend**
+  values pass through unvalidated: instances configure their own
+  category set, and an unknown category degrades to a clean empty
+  result rather than a failure. The instance must have vertical engines
+  enabled — the `searxng/` compose config in this repository enables
+  bing news + google news (news) and arxiv + pubmed (science); without
+  them SearXNG silently resolves the request against the general
   engines.
 
+## SafeSearch and adult content
+
+The gem returns results exactly as SearXNG produces them — it does no
+content filtering of its own. Whether adult sites appear in ordinary
+searches is decided entirely by the SearXNG instance's SafeSearch
+setting, which SearXNG defaults to **off**. To keep adult sites out of
+ordinary results, configure the instance:
+
+```yaml
+# /etc/searxng/settings.yml
+preferences:
+  lock:
+    - safesearch
+
+search:
+  safe_search: 2
+```
+
+- `safe_search: 2` is strict filtering.
+- Locking the `safesearch` preference forces that level onto every
+  request, so neither this gem, the JSON API, nor the web UI can relax
+  it back to 0.
+- Filtering is enforced **per engine**: SearXNG forwards the level
+  upstream and engines that don't implement SafeSearch (e.g.
+  `duckduckgo_web`, which carries an upstream `TODO: support safesearch`)
+  pass adult results through regardless of the setting. Prefer engines
+  that support it (`bing`, `duckduckgo`, `google`). The `searxng/` compose
+  config in this repository uses `duckduckgo` for this reason.
+
+SafeSearch is best-effort upstream filtering — strict is reliable in
+practice but not a guarantee. Consumers needing a hard guarantee should
+post-filter results by domain.
+
 ## Full documentation
 
 The full ask-rb documentation lives at https://ask-rb.github.io/ask-docs.

data/lib/ask/web_search/version.rb

@@ -1,5 +1,5 @@
 module Ask
   module WebSearch
-    VERSION = "0.6.0"
+    VERSION = "0.7.0"
   end
 end

data/lib/ask/web_search.rb

@@ -6,11 +6,14 @@ require "json"
 require_relative "web_search/version"
 
 module Ask
-  # Searches the web via a local SearXNG instance and returns the results
-  # as clean, numbered markdown for LLM consumption. The capability layer:
-  # one entry point (WebSearch.search) and a configurable endpoint. Tool
-  # framing — name, parameter schema, result wrapping — lives with the
-  # consumers (the MCP server, the agents) that call this library.
+  # Searches the web and returns the results as clean, numbered markdown
+  # for LLM consumption. Two interchangeable backends: TinyFish's hosted
+  # search API — primary when TINYFISH_API_KEY is set, so newcomers need
+  # no self-hosted infrastructure — and a local SearXNG instance for
+  # self-hosters, which is also the automatic fallback when TinyFish
+  # fails. The capability layer: one entry point (WebSearch.search).
+  # Tool framing — name, parameter schema, result wrapping — lives with
+  # the consumers (the MCP server, the agents) that call this library.
   module WebSearch
     # Search errors that carry engine diagnostics — the agent needs to
     # know *which* engines failed and *why* (CAPTCHA, timeout, suspended)
@@ -30,6 +33,7 @@ module Ask
 
     def self.searxng_url=(url)
       @searxng_url = url
+      @searxng_url_set = !url.nil?
     end
 
     # Retry configuration. Set max_retries to 0 to disable retries.
@@ -42,6 +46,49 @@ module Ask
     # unfiltered.
     TIME_RANGES = %w[day week month year].freeze
 
+    # TinyFish's hosted search endpoint — the primary backend for anyone
+    # holding a (free) TINYFISH_API_KEY, so no SearXNG instance is needed.
+    TINYFISH_SEARCH_URL = "https://api.search.tinyfish.ai/client.search.query"
+
+    # Our time_range → TinyFish's recency_minutes (SearXNG's month/year
+    # are approximate engine filters anyway; these are the equivalents).
+    TIME_RANGE_TO_MINUTES = {
+      "day" => 1440,
+      "week" => 10_080,
+      "month" => 43_200,
+      "year" => 525_600
+    }.freeze
+
+    # Our categories → TinyFish's single-value domain_type. The FIRST
+    # recognized value wins (TinyFish is one-domain-per-query); unknown
+    # categories are omitted so TinyFish falls back to its web default.
+    CATEGORIES_TO_DOMAIN_TYPE = {
+      "general" => "web",
+      "web" => "web",
+      "news" => "news",
+      "science" => "research_paper",
+      "research_paper" => "research_paper"
+    }.freeze
+
+    # True when TinyFish should be the primary backend: a key is present
+    # and TINYFISH_SEARCH=0 has not disabled it. Read fresh on every
+    # call so tests and config reloads don't need memo resets.
+    def self.use_tinyfish?
+      return false if ENV["TINYFISH_SEARCH"] == "0"
+
+      !ENV["TINYFISH_API_KEY"].to_s.empty?
+    end
+
+    # True when a SearXNG endpoint was configured explicitly (SEARXNG_URL
+    # or searxng_url=). Gates the TinyFish → SearXNG fallback so a failed
+    # TinyFish search never probes a default localhost instance that may
+    # not exist.
+    def self.searxng_configured?
+      return true if @searxng_url_set
+
+      !ENV["SEARXNG_URL"].to_s.empty?
+    end
+
     def self.max_retries
       return @max_retries if defined?(@max_retries)
 
@@ -89,22 +136,43 @@ module Ask
       search_raw(query, time_range: time_range, categories: categories)[:results]
     end
 
-    # The full SearXNG response: { results: [...], unresponsive: [[name,
-    # reason], ...] }. Same retry logic as #search_results, but preserves
-    # the engine diagnostics the caller needs to explain an empty result.
-    # time_range / categories are normalized here — the single point that
-    # builds the request — and are idempotent, so callers may pass raw or
-    # already-normalized values.
+    # The full response: { results: [...], unresponsive: [[name,
+    # reason], ...] }. Routes to TinyFish when #use_tinyfish?, falling
+    # back to SearXNG (only when explicitly configured) if TinyFish
+    # raises; otherwise straight to SearXNG — byte-for-byte the 0.6.x
+    # behavior for keyless users. Same retry logic either way, but
+    # preserves the engine diagnostics the caller needs to explain an
+    # empty result. time_range / categories are normalized by the
+    # backends that build requests — idempotent, so callers may pass
+    # raw or already-normalized values.
     def self.search_raw(query, time_range: nil, categories: nil)
+      return searxng_search_raw(query, time_range: time_range, categories: categories) unless use_tinyfish?
+
+      begin
+        tinyfish_search_raw(query, time_range: time_range, categories: categories)
+      rescue StandardError => primary_error
+        raise primary_error unless searxng_configured?
+
+        begin
+          searxng_search_raw(query, time_range: time_range, categories: categories)
+        rescue StandardError => fallback_error
+          raise Error, "search failed: TinyFish → #{primary_error.message}; " \
+                       "SearXNG fallback → #{fallback_error.message}"
+        end
+      end
+    end
+
+    # The SearXNG backend: GET {searxng_url}/search with the metasearch
+    # params, parsed into { results:, unresponsive: }. Call directly to
+    # bypass the router (and any TinyFish routing/fallback).
+    def self.searxng_search_raw(query, time_range: nil, categories: nil)
       params = { q: query, format: "json" }
       params[:time_range] = normalize_time_range(time_range) if time_range
       params[:categories] = normalize_categories(categories) if categories
       uri = URI("#{searxng_url}/search")
       uri.query = URI.encode_www_form(params)
-      retries = max_retries || 0
-      attempt = 0
-      begin
-        attempt += 1
+
+      with_retries do
         http = Net::HTTP.new(uri.host, uri.port)
         http.open_timeout = 5
         http.read_timeout = 10
@@ -114,11 +182,44 @@ module Ask
         raise "SearXNG returned #{res.code}: #{res.body}" unless res.code.start_with?("2")
 
         parse_response(JSON.parse(res.body))
-      rescue StandardError
-        raise if attempt > retries
+      end
+    end
 
-        sleep RETRY_BACKOFF[[attempt - 1, RETRY_BACKOFF.size - 1].min]
-        retry
+    # The TinyFish backend: GET the hosted search API (needs
+    # TINYFISH_API_KEY), normalized into the same { results:,
+    # unresponsive: } contract — snippet → content; a single hosted
+    # API has no per-engine status, so unresponsive is always empty
+    # (transport failures raise instead, which the router's fallback
+    # catches). time_range maps to recency_minutes; the FIRST
+    # recognized categories value maps to domain_type.
+    def self.tinyfish_search_raw(query, time_range: nil, categories: nil)
+      params = { query: query }
+      normalized_range = normalize_time_range(time_range)
+      params[:recency_minutes] = TIME_RANGE_TO_MINUTES[normalized_range] if normalized_range
+      normalized_categories = normalize_categories(categories)
+      if normalized_categories
+        domain = CATEGORIES_TO_DOMAIN_TYPE[normalized_categories.split(",").first]
+        params[:domain_type] = domain if domain
+      end
+      uri = URI(TINYFISH_SEARCH_URL)
+      uri.query = URI.encode_www_form(params)
+
+      with_retries do
+        http = Net::HTTP.new(uri.host, uri.port)
+        http.use_ssl = true
+        http.open_timeout = 5
+        http.read_timeout = 10
+        req = Net::HTTP::Get.new(uri)
+        req["X-API-Key"] = ENV["TINYFISH_API_KEY"]
+        req["User-Agent"] = "ask-web-search/#{Ask::WebSearch::VERSION}"
+        res = http.request(req)
+        raise Error, "TinyFish returned #{res.code}: #{res.body[0, 200]}" unless res.code.start_with?("2")
+
+        data = JSON.parse(res.body)
+        results = data.fetch("results", []).map do |r|
+          { url: r["url"], title: r["title"], content: r["snippet"] }
+        end
+        { results: results, unresponsive: [] }
       end
     end
 
@@ -160,6 +261,24 @@ module Ask
       value.empty? ? nil : value
     end
 
+    # Retries the block up to max_retries times on any StandardError
+    # with exponential backoff (RETRY_BACKOFF). Param building and
+    # validation happen OUTSIDE the block in the backends, so an
+    # ArgumentError never sleeps and retries.
+    def self.with_retries
+      attempt = 0
+      begin
+        attempt += 1
+        yield
+      rescue StandardError
+        raise if attempt > (max_retries || 0)
+
+        sleep RETRY_BACKOFF[[attempt - 1, RETRY_BACKOFF.size - 1].min]
+        retry
+      end
+    end
+    private_class_method :with_retries
+
     # Parses the full SearXNG JSON response into { results:, unresponsive: }.
     # results are { url:, title:, content: } entries from results and
     # infoboxes, deduplicated by url. unresponsive is the raw

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ask-web-search
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - Kaka Ruto
@@ -79,10 +79,11 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '13.0'
-description: Searches the web via a local SearXNG instance and returns the results
-  as clean numbered markdown. The capability layer (WebSearch.search); the native
-  Ask::Tools::WebSearch agent tool is an optional integration that registers when
-  ask-tools is present. Configure endpoint via SEARXNG_URL env var.
+description: 'Web search returning results as clean numbered markdown. Two backends:
+  TinyFish''s hosted API (primary with TINYFISH_API_KEY — no self-hosting) and a local
+  SearXNG instance (SEARXNG_URL, also the automatic fallback). The capability layer
+  (WebSearch.search); the native Ask::Tools::WebSearch agent tool is an optional integration
+  that registers when ask-tools is present.'
 email:
 - kaka@myrrlabs.com
 executables: []