searxng 0.1.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2b8849ed3269f41fc39d8b6274f8b0ae92ade2649dd1f6e5111b1799fec375e5
-  data.tar.gz: cd78466743cbad58a93d2453f2128e7bfb2f139a05c84eba0c984166da662e2f
+  metadata.gz: 0752ce85481fa3a6a5846b4326effa0251ff69e77a32912dc046f06e35542832
+  data.tar.gz: 1700a1ad55569d5cbb7137c6b3c6cde6583c8b6198508975e6b66030dfd2bc39
 SHA512:
-  metadata.gz: 38818edbabaf94b55bca1e43cbee38fb8bf7d54319d1b2fe3b56a10653f33152011697dd01e9135d1e05fc23f07629a716578dce791b55addc97ca3f471afd92
-  data.tar.gz: 4a77e1ba98532c75d7da8c5e3da80c5cb53127488c094fa8dfc331de5285afb4a78e5278f332aa92e27d8e489a1273e8e39eea1027d4643988869c9bdd75b610
+  metadata.gz: e85f85f3363f0468cef0d8dd6b5318f2d2eba8ef7268e6c607fbbe318a5140347e395400b747c33716f4ee73a2fa99e6af54398e7e20e42dbfb2e1a76f60593a
+  data.tar.gz: ff609005b3e6993106c5c29855801675e48b25221456d7f4561f43390a31af1002d98a6abc718d407cd3c87bcede1ff8bed6b587cf166c6fdd5dddfb699aa3a8

data/CHANGELOG.md

@@ -1,4 +1,39 @@
-# Changelog
+# CHANGELOG
+
+## Unreleased
+
+## 0.2.1 (2026-10-29)
+
+- Fix cleartext HTTP proxy fetches to dial the `NetworkPolicy`-pinned IP and set `Host` to the original hostname.
+- Fix unbounded `web_url_read` response buffering with a shared byte ceiling across HTTP, FTP, SFTP, SMB, and Gemini.
+- Fix process-wide `@@url_cache` growth with TTL purge, entry cap, byte budget, and a mutex.
+- Fix Gemini TLS verification by loading the default CA store via `SSLContext#set_params`.
+- Fix SMB credentials leaking through `smbclient` argv; pass the password via `PASSWD`.
+- Fix website HTTP errors mislabeled as SearXNG search failures; include redirect `Location` when present.
+- Fix invalid `time_range` / `safesearch` values being silently dropped; raise `ConfigurationError`.
+- Fix IPFS gateway path traversal via unsafe CID or `..` path segments.
+- Fix `timeoutMs` of `0` disabling deadlines; clamp to `1000..30000`.
+- Fix release gate ignoring lint/test failures (`pipefail`) and dirty staged/untracked trees.
+- Fix release confirmation treating EOF as publish; require an explicit Enter for the default.
+- Require a `CHANGELOG.md` version heading matching `VERSION` before gem push.
+- Fix `searxng serve` swallowing startup errors into an unbounded `StringIO`; reopen stderr to `File::NULL` and print configuration failures.
+- Add `NetworkPolicy` coverage for local NAT64 (`64:ff9b:1::/48`), IPv4-compatible embeds, documentation ranges, and bracketed IPv6 hostnames via `uri.hostname`.
+- Document Ruby `>= 3.4` and that `.onion` / `.i2p` are not supported by `web_url_read`.
+- Redact URL userinfo from fetch error URIs; default `maxLength` to `20000`; honor `SEARXNG_USER_AGENT` for URL reads.
+
+## 0.2.0
+
+- MCP: Added `web_url_read` tool (URL fetch + Markdown conversion with pagination/section options).
+- MCP: Added `gemini://` support in `web_url_read` (Gemini fetch + Gemtext to Markdown).
+- MCP: Added `ipfs://` support in `web_url_read` via configurable `IPFS_GATEWAY`.
+- MCP: Added `ftp://`, `sftp://`, and `smb://` support in `web_url_read`.
+- MCP: Added resources `config://server-config` and `help://usage-guide`.
+- MCP: Added startup environment validation for `SEARXNG_URL` and auth variable pairs.
+- Errors: Added richer, user-facing configuration/network/HTTP messages.
+- Client: Added auth env aliases (`AUTH_USERNAME`/`AUTH_PASSWORD`) and proxy auto-detection from `HTTP_PROXY`/`HTTPS_PROXY`/`ALL_PROXY` with `NO_PROXY`.
+- Docs: Added protocol support notes for Gemini, IPFS, Tor, I2P, and SOCKS5-related setups.
+- Docs: Added protocol support and environment notes for FTP/SFTP/SMB.
+- Search tool: Improved no-results response to include actionable guidance.
 
 ## 0.1.0
 

data/README.md

@@ -1,18 +1,12 @@
-# searxng
+# searxng.rb
 
-[![Gem Version](https://badge.fury.io/rb/searxng.svg)](https://badge.fury.io/rb/searxng) [![Test Status](https://github.com/amkisko/searxng.rb/actions/workflows/test.yml/badge.svg)](https://github.com/amkisko/searxng.rb/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/amkisko/searxng.rb/graph/badge.svg)](https://codecov.io/gh/amkisko/searxng.rb)
+[![Gem Version](https://badge.fury.io/rb/searxng.svg?v=0.2.1)](https://badge.fury.io/rb/searxng) [![Test Status](https://github.com/amkisko/searxng.rb/actions/workflows/test.yml/badge.svg)](https://github.com/amkisko/searxng.rb/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/amkisko/searxng.rb/graph/badge.svg)](https://app.codecov.io/github/amkisko/searxng.rb)
 
 Ruby gem providing a SearXNG HTTP client, CLI (search), and MCP (Model Context Protocol) server for web search. Integrates with MCP-compatible clients like Codex, Cursor, Claude, and other MCP-enabled tools.
 
-Sponsored by [Kisko Labs](https://www.kiskolabs.com).
-
-<a href="https://www.kiskolabs.com">
-  <img src="kisko.svg" width="200" alt="Sponsored by Kisko Labs" />
-</a>
-
 ## Requirements
 
-- **Ruby 3.1 or higher** (Ruby 3.0 and earlier are not supported). For managing Ruby versions, [rbenv](https://github.com/rbenv/rbenv) or [mise](https://mise.jdx.dev/) are recommended; system Ruby may be sufficient if it meets the version requirement.
+- **Ruby 3.4 or higher**. For managing Ruby versions, [rbenv](https://github.com/rbenv/rbenv) or [mise](https://mise.jdx.dev/) are recommended; system Ruby may be sufficient if it meets the version requirement.
 
 ## Quick Start
 
@@ -26,15 +20,63 @@ Or add to your Gemfile:
 gem "searxng"
 ```
 
+**Quick start with a local SearXNG instance:** The repo includes a ready-made [docker-compose](examples/docker-compose.yml) and [settings](examples/searxng/settings.yml) so you can run SearXNG locally (JSON format and limiter preconfigured for the gem). From the repo root: `docker compose -f examples/docker-compose.yml up -d`, then `export SEARXNG_URL="http://localhost:8080"`. See [examples/SETUP.md](examples/SETUP.md) for details.
+
+### CLI
+
+```bash
+searxng search "your query"
+searxng search "ruby" --page 2 --language en --time-range day --json
+```
+
+Options: `--url`, `--page`, `--language`, `--time-range` (day|month|year), `--safesearch` (0|1|2), `--json`.
+
+### Ruby API
+
+```ruby
+require "searxng"
+
+client = Searxng::Client.new
+data = client.search("ruby programming", pageno: 1, language: "en")
+
+data[:results].each do |r|
+  puts r[:title], r[:url], r[:content]
+end
+```
+
 ### Configuration
 
-- **SEARXNG_URL** (required for remote): Base URL of your SearXNG instance (e.g. `http://localhost:8080` or `https://search.example.com`). Defaults to `http://localhost:8080` when not set.
-- **SEARXNG_USER** / **SEARXNG_PASSWORD** (optional): Basic auth credentials if your instance is protected.
+- **SEARXNG_URL**: Base URL of your SearXNG instance (e.g. `http://localhost:8080` or `https://search.example.com`). The client defaults to `http://localhost:8080`; MCP server startup validates that this variable is explicitly set and well-formed.
+- **SEARXNG_USER** / **SEARXNG_PASSWORD** (optional): Basic auth credentials if your instance is protected. `AUTH_USERNAME` / `AUTH_PASSWORD` are also accepted for compatibility. Auth vars must be provided as a pair.
 - **SEARXNG_CA_FILE** / **SEARXNG_CA_PATH** (optional): Custom CA certificate file or directory for HTTPS. You can also pass `ca_file:`, `ca_path:`, or `verify_mode:` to `Searxng::Client.new`.
 - **SEARXNG_USER_AGENT** (optional): Custom User-Agent string. The client sends a default identifying the gem; if your instance returns 403 Forbidden (e.g. bot detection), set a custom User-Agent or pass `user_agent:` to `Searxng::Client.new`.
+- **HTTP_PROXY / HTTPS_PROXY / ALL_PROXY / NO_PROXY** (optional): Proxy configuration for HTTP(S) requests.
+- **SEARXNG_WEB_URL_SCHEMES** (optional): Comma-separated protocols enabled for `web_url_read`. The secure default is `http,https,ipfs`; opt in to any required subset of `ftp,sftp,smb,gemini`.
+- **SEARXNG_WEB_URL_CREDENTIAL_HOSTS** (optional): Comma-separated host allowlist permitted to receive credentials from the FTP, SFTP, or SMB environment variables. Credentials embedded directly in a URL remain scoped to that URL.
+- **FTP_USER / FTP_PASSWORD** (optional): FTP credentials used only for hosts in `SEARXNG_WEB_URL_CREDENTIAL_HOSTS`.
+- **SFTP_USER / SFTP_PASSWORD** (optional): SFTP credentials used only for hosts in `SEARXNG_WEB_URL_CREDENTIAL_HOSTS`.
+- **SMB_USER / SMB_PASSWORD / SMB_DOMAIN** (optional): SMB credentials/domain used only for hosts in `SEARXNG_WEB_URL_CREDENTIAL_HOSTS`.
+- **IPFS_GATEWAY** (optional): Gateway used for `ipfs://` URLs (default: `https://ipfs.io`).
+- **GEMINI_INSECURE=1** (optional): disables TLS certificate verification for Gemini fetches in `web_url_read`.
 
 To fully customize the HTTP client (e.g. custom certs, proxy, timeouts), override `#build_http(uri)` in a subclass, or pass a `configure_http:` callable to the client; it is invoked with the `Net::HTTP` instance and the request URI before each request.
 
+## MCP Tools
+
+The MCP server provides the following tools:
+
+1. **searxng_web_search** - Web search via SearXNG
+   - Parameters: `query` (required), `pageno` (optional), `max_results` (optional, default 10), `time_range` (optional), `language` (optional), `safesearch` (optional). Use `max_results` to limit how many results are returned in one response (reduces token usage); use `pageno` for the next page.
+2. **web_url_read** - Fetch a URL and return Markdown content
+   - Parameters: `url` (required), `startChar`, `maxLength`, `section`, `paragraphRange`, `readHeadings`, `timeoutMs` (all optional).
+   - Uses `teplo_core` for HTML/XML to Markdown conversion (with workspace fallback to `textplorer/core-ruby` in development).
+   - Supports `http`, `https`, and `ipfs` by default. `ftp`, `sftp`, `smb`, and `gemini` require explicit opt-in through `SEARXNG_WEB_URL_SCHEMES`.
+
+The MCP server also exposes resources:
+
+- `config://server-config` - current server/environment capabilities
+- `help://usage-guide` - concise usage and configuration guide
+
 ### Cursor IDE Configuration
 
 For Cursor IDE, create or update `.cursor/mcp.json` in your project:
@@ -120,37 +162,18 @@ Useful for testing the `searxng_web_search` tool before integrating with Cursor
 - **MCP Server Integration**: Ready-to-use MCP server with web search tool, compatible with Cursor IDE, Claude Desktop, and other MCP-enabled tools
 - **Configurable HTTP**: Optional custom CA, verify mode, and `configure_http` hook or `build_http` override for proxy/timeouts/certs
 - **Basic Auth**: Optional Basic authentication via options or ENV
+- **Fail-fast MCP validation**: MCP startup validates `SEARXNG_URL` and auth variable pairs
+- **User-facing errors**: Clear network/HTTP/configuration error messages
 
-## Basic Usage
+## Protocol Support Notes
 
-### Ruby API
-
-```ruby
-require "searxng"
-
-client = Searxng::Client.new
-data = client.search("ruby programming", pageno: 1, language: "en")
-
-data[:results].each do |r|
-  puts r[:title], r[:url], r[:content]
-end
-```
-
-### CLI
-
-```bash
-searxng search "your query"
-searxng search "ruby" --page 2 --language en --time-range day --json
-```
-
-Options: `--url`, `--page`, `--language`, `--time-range` (day|month|year), `--safesearch` (0|1|2), `--json`.
-
-## MCP Tools
-
-The MCP server provides the following tools:
-
-1. **searxng_web_search** - Web search via SearXNG
-   - Parameters: `query` (required), `pageno` (optional), `max_results` (optional, default 10), `time_range` (optional), `language` (optional), `safesearch` (optional). Use `max_results` to limit how many results are returned in one response (reduces token usage); use `pageno` for the next page.
+- `http` / `https`: fully supported.
+- `ftp`: opt-in through `SEARXNG_WEB_URL_SCHEMES` (requires the `net-ftp` runtime gem).
+- `sftp`: opt-in through `SEARXNG_WEB_URL_SCHEMES` (requires the `net-sftp` runtime gem).
+- `smb`: opt-in through `SEARXNG_WEB_URL_SCHEMES` and uses `smbclient` available in `PATH`.
+- `gemini`: opt-in through `SEARXNG_WEB_URL_SCHEMES` (Gemini fetch + Gemtext to Markdown conversion).
+- `ipfs`: supported by `web_url_read` through gateway resolution (`ipfs://` -> `IPFS_GATEWAY`).
+- `.onion` / `.i2p`: not supported by `web_url_read` (destination policy resolves hosts locally before any proxy).
 
 ## Examples
 
@@ -196,6 +219,27 @@ For more information, see [CONTRIBUTING.md](CONTRIBUTING.md).
 
 If you discover a security vulnerability, please report it responsibly. See [SECURITY.md](SECURITY.md) for details.
 
+## Links
+
+- [GitHub](https://github.com/amkisko/searxng.rb)
+- [GitLab](https://gitlab.com/amkisko/searxng.rb)
+- [RubyGems](https://rubygems.org/gems/searxng)
+- [Versions Atom](https://rubygems.org/gems/searxng/versions.atom) (feed id `10648244160064`)
+- [libraries.io](https://libraries.io/rubygems/searxng)
+- [Deps.dev](https://deps.dev/rubygems/searxng)
+- [SonarCloud](https://sonarcloud.io/project/overview?id=amkisko_searxng.rb)
+- [Snyk](https://snyk.io/test/github/amkisko/searxng.rb)
+- [Codecov](https://app.codecov.io/github/amkisko/searxng.rb)
+- [OpenSSF Scorecard](https://scorecard.dev/viewer/?uri=github.com/amkisko/searxng.rb)
+
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Sponsors
+
+Sponsored by [Kisko Labs](https://www.kiskolabs.com).
+
+<a href="https://www.kiskolabs.com">
+  <img src="kisko.svg" width="200" alt="Sponsored by Kisko Labs" />
+</a>

data/bin/searxng

@@ -5,13 +5,16 @@ cmd = ARGV.shift
 case cmd
 when "serve"
   unless ENV["DEBUG"]
-    require "stringio"
-    $stderr = StringIO.new
-    $stderr.set_encoding("UTF-8")
+    $stderr.reopen(File::NULL, "w")
   end
   $LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
   require "searxng/server"
-  Searxng::Server.start
+  begin
+    Searxng::Server.start
+  rescue Searxng::ConfigurationError => e
+    warn e.message
+    exit 1
+  end
 when "search"
   $LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
   require "searxng"

data/lib/searxng/client.rb

@@ -2,6 +2,8 @@ require "uri"
 require "net/http"
 require "openssl"
 require "json"
+require "ipaddr"
+require "timeout"
 
 module Searxng
   class Client
@@ -22,9 +24,13 @@ module Searxng
       user_agent: nil,
       configure_http: nil
     )
-      @base_url = (base_url || ENV["SEARXNG_URL"] || DEFAULT_BASE_URL).to_s.chomp("/")
-      @user = user || ENV["SEARXNG_USER"]
-      @password = password || ENV["SEARXNG_PASSWORD"]
+      @base_url = if base_url.nil?
+        present_string(ENV["SEARXNG_URL"]) || DEFAULT_BASE_URL
+      else
+        base_url.to_s.strip
+      end.chomp("/")
+      @user = present_string(user) || present_string(ENV["SEARXNG_USER"]) || present_string(ENV["AUTH_USERNAME"])
+      @password = present_string(password) || present_string(ENV["SEARXNG_PASSWORD"]) || present_string(ENV["AUTH_PASSWORD"])
       @open_timeout = open_timeout
       @read_timeout = read_timeout
       @ca_file = ca_file || ENV["SEARXNG_CA_FILE"]
@@ -35,7 +41,7 @@ module Searxng
     end
 
     def search(query, pageno: 1, time_range: nil, language: "all", safesearch: nil)
-      raise ConfigurationError, "SEARXNG_URL not set" if @base_url.nil? || @base_url.empty?
+      raise ConfigurationError, ErrorMessages.configuration_missing_url if @base_url.nil? || @base_url.empty?
 
       uri = build_search_uri(query, pageno: pageno, time_range: time_range, language: language, safesearch: safesearch)
       response = perform_request(uri)
@@ -47,13 +53,30 @@ module Searxng
     def build_search_uri(query, pageno:, time_range:, language:, safesearch:)
       uri = URI.join("#{@base_url}/", "search")
       params = {"q" => query, "format" => "json", "pageno" => pageno.to_s}
-      params["time_range"] = time_range if time_range && VALID_TIME_RANGES.include?(time_range.to_s)
+      if time_range
+        range = time_range.to_s
+        unless VALID_TIME_RANGES.include?(range)
+          raise ConfigurationError, "Invalid time_range: #{time_range.inspect} (allowed: #{VALID_TIME_RANGES.join(", ")})"
+        end
+        params["time_range"] = range
+      end
       params["language"] = language if language && language != "all"
-      params["safesearch"] = safesearch.to_s if safesearch && VALID_SAFESEARCH.include?(safesearch.to_i)
+      unless safesearch.nil?
+        value = Integer(safesearch, exception: false)
+        unless value && VALID_SAFESEARCH.include?(value)
+          raise ConfigurationError, "Invalid safesearch: #{safesearch.inspect} (allowed: #{VALID_SAFESEARCH.join(", ")})"
+        end
+        params["safesearch"] = value.to_s
+      end
       uri.query = URI.encode_www_form(params)
       uri
     end
 
+    def present_string(value)
+      text = value&.to_s&.strip
+      text unless text.nil? || text.empty?
+    end
+
     def perform_request(uri)
       http = build_http(uri)
       @configure_http&.call(http, uri)
@@ -65,13 +88,19 @@ module Searxng
 
       http.request(request)
     rescue SocketError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET, Errno::EPIPE, Timeout::Error, OpenSSL::SSL::SSLError, EOFError => e
-      raise NetworkError, "Connection failed: #{e.message}"
+      raise NetworkError, ErrorMessages.network_error(e, uri: uri)
     end
 
     # Builds and returns a configured Net::HTTP instance. Override in a subclass to customize
     # timeouts, SSL, proxy, or other options. Called once per request.
     def build_http(uri)
-      http = Net::HTTP.new(uri.host, uri.port)
+      proxy_uri = proxy_uri_for(uri)
+      http_klass = if proxy_uri
+        Net::HTTP::Proxy(proxy_uri.host, proxy_uri.port, proxy_uri.user, proxy_uri.password)
+      else
+        Net::HTTP
+      end
+      http = http_klass.new(uri.host, uri.port)
       http.use_ssl = (uri.scheme == "https")
       http.open_timeout = @open_timeout
       http.read_timeout = @read_timeout
@@ -103,7 +132,7 @@ module Searxng
         }
       else
         raise APIError.new(
-          "SearXNG returned #{response.code} #{response.message}",
+          ErrorMessages.api_error(response.code.to_i, response.message),
           status_code: response.code.to_i,
           response_data: response.body,
           uri: uri.to_s
@@ -117,6 +146,51 @@ module Searxng
       )
     end
 
+    def proxy_uri_for(uri)
+      return nil if no_proxy_match?(uri.host)
+
+      raw = if uri.scheme == "https"
+        ENV["HTTPS_PROXY"] || ENV["https_proxy"] || ENV["HTTP_PROXY"] || ENV["http_proxy"] || ENV["ALL_PROXY"] || ENV["all_proxy"]
+      else
+        ENV["HTTP_PROXY"] || ENV["http_proxy"] || ENV["ALL_PROXY"] || ENV["all_proxy"]
+      end
+      return nil if raw.nil? || raw.strip.empty?
+
+      URI(raw.include?("://") ? raw : "http://#{raw}")
+    rescue URI::InvalidURIError, ArgumentError
+      raise ConfigurationError, "Proxy URL is invalid: #{raw.inspect}"
+    end
+
+    def no_proxy_match?(host)
+      return false if host.nil? || host.empty?
+
+      raw = ENV["NO_PROXY"] || ENV["no_proxy"]
+      return false if raw.nil? || raw.strip.empty?
+
+      host_ip = begin
+        IPAddr.new(host)
+      rescue
+        nil
+      end
+      raw.split(",").any? do |entry|
+        token = entry.to_s.strip
+        next false if token.empty?
+        return true if token == "*"
+
+        cidr = begin
+          IPAddr.new(token)
+        rescue
+          nil
+        end
+        if cidr && host_ip
+          next cidr.include?(host_ip)
+        end
+
+        normalized = token.sub(/\A\./, "")
+        host == normalized || host.end_with?(".#{normalized}")
+      end
+    end
+
     def normalize_results(results)
       Array(results).map do |r|
         {

data/lib/searxng/errors.rb

@@ -15,4 +15,68 @@ module Searxng
       @uri = uri
     end
   end
+
+  module ErrorMessages
+    module_function
+
+    def configuration_missing_url
+      "SEARXNG_URL not set. Set it to your SearXNG instance (e.g., http://localhost:8080 or https://search.example.com)"
+    end
+
+    def configuration_invalid_url(url)
+      "SEARXNG_URL has invalid format: #{url.inspect}. Use format: http://localhost:8080 or https://search.example.com"
+    end
+
+    def configuration_invalid_protocol(protocol)
+      "SEARXNG_URL must use http or https protocol, got: #{protocol.inspect}"
+    end
+
+    def configuration_auth_pair
+      "Authentication variables must be set together: SEARXNG_USER with SEARXNG_PASSWORD (or AUTH_USERNAME with AUTH_PASSWORD)"
+    end
+
+    def no_results(query)
+      %(No results found for "#{query}". Try different search terms or check if SearXNG search engines are working.)
+    end
+
+    def network_error(exception, uri: nil, target: "SearXNG server")
+      case exception
+      when SocketError
+        host = uri&.host || "unknown host"
+        %(DNS Error: Cannot resolve hostname "#{host}".)
+      when Errno::ECONNREFUSED
+        "Connection Error: #{target} is not responding."
+      when Errno::ETIMEDOUT, Timeout::Error
+        "Timeout Error: #{target} is too slow to respond."
+      when OpenSSL::SSL::SSLError
+        "SSL Error: Certificate or TLS problem while connecting to #{target}."
+      else
+        message = exception&.message.to_s.strip
+        "Network Error: #{message.empty? ? "Connection failed" : message}"
+      end
+    end
+
+    def api_error(status_code, status_message)
+      case status_code.to_i
+      when 403
+        "SearXNG Error (403): Authentication required or IP blocked."
+      when 404
+        "SearXNG Error (404): Search endpoint not found."
+      when 429
+        "SearXNG Error (429): Rate limit exceeded."
+      when 500..599
+        "SearXNG Error (#{status_code}): Internal server error."
+      else
+        "SearXNG returned #{status_code} #{status_message}".strip
+      end
+    end
+
+    def fetch_http_error(status_code, status_message, location: nil)
+      code = status_code.to_i
+      base = "Website Error (#{code}): #{status_message}".strip
+      return "#{base} Location: #{location}" if location && !location.to_s.strip.empty?
+
+      base
+    end
+  end
 end

data/lib/searxng/network_policy.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+require "ipaddr"
+require "socket"
+require "uri"
+
+module Searxng
+  class NetworkPolicy
+    FORBIDDEN_RANGES = %w[
+      0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8
+      169.254.0.0/16 172.16.0.0/12 192.0.0.0/24 192.0.2.0/24
+      192.88.99.0/24 192.168.0.0/16 198.18.0.0/15 198.51.100.0/24
+      203.0.113.0/24 224.0.0.0/4 240.0.0.0/4
+      ::/128 ::1/128 ::/96 64:ff9b::/96 64:ff9b:1::/48 100::/64
+      2001::/23 2001:db8::/32 2002::/16 3fff::/20
+      fc00::/7 fe80::/10 fec0::/10 ff00::/8
+    ].map { |range| IPAddr.new(range) }.freeze
+
+    def self.validate!(url, allowed_schemes:, allow_userinfo: false)
+      uri = URI.parse(url.to_s)
+      reject! unless allowed_schemes.include?(uri.scheme&.downcase)
+      reject! if hostname_of(uri).to_s.empty? || (uri.userinfo && !allow_userinfo)
+
+      [uri, allowed_ip_address(uri)]
+    rescue URI::InvalidURIError
+      reject!
+    end
+
+    def self.allowed_ip_address(uri)
+      host = hostname_of(uri)
+      addresses = Addrinfo.getaddrinfo(host, uri.port, nil, :STREAM)
+        .map(&:ip_address)
+        .uniq
+      reject! if addresses.empty? || addresses.any? { |address| forbidden?(address) }
+
+      addresses.first
+    rescue SocketError, SystemCallError
+      reject!
+    end
+    private_class_method :allowed_ip_address
+
+    def self.hostname_of(uri)
+      uri.respond_to?(:hostname) ? uri.hostname : uri.host.to_s.delete_prefix("[").delete_suffix("]")
+    end
+    private_class_method :hostname_of
+
+    def self.forbidden?(address)
+      parsed = normalize_address(address)
+      return true if FORBIDDEN_RANGES.any? { |range| range.include?(parsed) }
+      return forbidden?(parsed.native.to_s) if ipv4_compatible?(parsed)
+
+      false
+    rescue IPAddr::InvalidAddressError
+      true
+    end
+    private_class_method :forbidden?
+
+    def self.normalize_address(address)
+      parsed = IPAddr.new(address.split("%", 2).first)
+      parsed.ipv4_mapped? ? parsed.native : parsed
+    end
+    private_class_method :normalize_address
+
+    def self.ipv4_compatible?(parsed)
+      parsed.ipv6? && parsed.respond_to?(:ipv4_compat?) && parsed.ipv4_compat?
+    end
+    private_class_method :ipv4_compatible?
+
+    def self.reject!
+      raise ConfigurationError, "URL destination is not allowed"
+    end
+    private_class_method :reject!
+  end
+end