searxng 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2b8849ed3269f41fc39d8b6274f8b0ae92ade2649dd1f6e5111b1799fec375e5
-  data.tar.gz: cd78466743cbad58a93d2453f2128e7bfb2f139a05c84eba0c984166da662e2f
+  metadata.gz: 312181892379c3ed0181f46a10f535e96803969d7f1b224b87e07fe7fe2d68d4
+  data.tar.gz: e206771ab2279dac1329a08dc1195d254e5992c4532c7e2e5afd95a67b267991
 SHA512:
-  metadata.gz: 38818edbabaf94b55bca1e43cbee38fb8bf7d54319d1b2fe3b56a10653f33152011697dd01e9135d1e05fc23f07629a716578dce791b55addc97ca3f471afd92
-  data.tar.gz: 4a77e1ba98532c75d7da8c5e3da80c5cb53127488c094fa8dfc331de5285afb4a78e5278f332aa92e27d8e489a1273e8e39eea1027d4643988869c9bdd75b610
+  metadata.gz: ce7b9acd2231550ddd32fd27956b1ef366f044a769f7b0034a6f5c5ade98efed0ac96a34856d1015456cc46132b024afb723726117dd3a9df7b82b26282d5ad7
+  data.tar.gz: 5636c48dd7221654b182793931ffb097fe017939fe8fc41781819deda7ed627a921edf508243f16943fa70ade77ee04b59b3ea6ed6be76de63eb34becbe2e13c

data/CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## Unreleased
+
+## 0.2.0
+
+- MCP: Added `web_url_read` tool (URL fetch + Markdown conversion with pagination/section options).
+- MCP: Added `gemini://` support in `web_url_read` (Gemini fetch + Gemtext to Markdown).
+- MCP: Added `ipfs://` support in `web_url_read` via configurable `IPFS_GATEWAY`.
+- MCP: Added `ftp://`, `sftp://`, and `smb://` support in `web_url_read`.
+- MCP: Added resources `config://server-config` and `help://usage-guide`.
+- MCP: Added startup environment validation for `SEARXNG_URL` and auth variable pairs.
+- Errors: Added richer, user-facing configuration/network/HTTP messages.
+- Client: Added auth env aliases (`AUTH_USERNAME`/`AUTH_PASSWORD`) and proxy auto-detection from `HTTP_PROXY`/`HTTPS_PROXY`/`ALL_PROXY` with `NO_PROXY`.
+- Docs: Added protocol support notes for Gemini, IPFS, Tor, I2P, and SOCKS5-related setups.
+- Docs: Added protocol support and environment notes for FTP/SFTP/SMB.
+- Search tool: Improved no-results response to include actionable guidance.
+
 ## 0.1.0
 
 - Initial release.

data/README.md

@@ -1,15 +1,9 @@
-# searxng
+# searxng.rb
 
-[![Gem Version](https://badge.fury.io/rb/searxng.svg)](https://badge.fury.io/rb/searxng) [![Test Status](https://github.com/amkisko/searxng.rb/actions/workflows/test.yml/badge.svg)](https://github.com/amkisko/searxng.rb/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/amkisko/searxng.rb/graph/badge.svg)](https://codecov.io/gh/amkisko/searxng.rb)
+[![Gem Version](https://badge.fury.io/rb/searxng.svg?v=0.2.0)](https://badge.fury.io/rb/searxng) [![Test Status](https://github.com/amkisko/searxng.rb/actions/workflows/test.yml/badge.svg)](https://github.com/amkisko/searxng.rb/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/amkisko/searxng.rb/graph/badge.svg)](https://codecov.io/gh/amkisko/searxng.rb)
 
 Ruby gem providing a SearXNG HTTP client, CLI (search), and MCP (Model Context Protocol) server for web search. Integrates with MCP-compatible clients like Codex, Cursor, Claude, and other MCP-enabled tools.
 
-Sponsored by [Kisko Labs](https://www.kiskolabs.com).
-
-<a href="https://www.kiskolabs.com">
-  <img src="kisko.svg" width="200" alt="Sponsored by Kisko Labs" />
-</a>
-
 ## Requirements
 
 - **Ruby 3.1 or higher** (Ruby 3.0 and earlier are not supported). For managing Ruby versions, [rbenv](https://github.com/rbenv/rbenv) or [mise](https://mise.jdx.dev/) are recommended; system Ruby may be sufficient if it meets the version requirement.
@@ -26,15 +20,61 @@ Or add to your Gemfile:
 gem "searxng"
 ```
 
+**Quick start with a local SearXNG instance:** The repo includes a ready-made [docker-compose](examples/docker-compose.yml) and [settings](examples/searxng/settings.yml) so you can run SearXNG locally (JSON format and limiter preconfigured for the gem). From the repo root: `docker compose -f examples/docker-compose.yml up -d`, then `export SEARXNG_URL="http://localhost:8080"`. See [examples/SETUP.md](examples/SETUP.md) for details.
+
+### CLI
+
+```bash
+searxng search "your query"
+searxng search "ruby" --page 2 --language en --time-range day --json
+```
+
+Options: `--url`, `--page`, `--language`, `--time-range` (day|month|year), `--safesearch` (0|1|2), `--json`.
+
+### Ruby API
+
+```ruby
+require "searxng"
+
+client = Searxng::Client.new
+data = client.search("ruby programming", pageno: 1, language: "en")
+
+data[:results].each do |r|
+  puts r[:title], r[:url], r[:content]
+end
+```
+
 ### Configuration
 
-- **SEARXNG_URL** (required for remote): Base URL of your SearXNG instance (e.g. `http://localhost:8080` or `https://search.example.com`). Defaults to `http://localhost:8080` when not set.
-- **SEARXNG_USER** / **SEARXNG_PASSWORD** (optional): Basic auth credentials if your instance is protected.
+- **SEARXNG_URL**: Base URL of your SearXNG instance (e.g. `http://localhost:8080` or `https://search.example.com`). The client defaults to `http://localhost:8080`; MCP server startup validates that this variable is explicitly set and well-formed.
+- **SEARXNG_USER** / **SEARXNG_PASSWORD** (optional): Basic auth credentials if your instance is protected. `AUTH_USERNAME` / `AUTH_PASSWORD` are also accepted for compatibility. Auth vars must be provided as a pair.
 - **SEARXNG_CA_FILE** / **SEARXNG_CA_PATH** (optional): Custom CA certificate file or directory for HTTPS. You can also pass `ca_file:`, `ca_path:`, or `verify_mode:` to `Searxng::Client.new`.
 - **SEARXNG_USER_AGENT** (optional): Custom User-Agent string. The client sends a default identifying the gem; if your instance returns 403 Forbidden (e.g. bot detection), set a custom User-Agent or pass `user_agent:` to `Searxng::Client.new`.
+- **HTTP_PROXY / HTTPS_PROXY / ALL_PROXY / NO_PROXY** (optional): Proxy configuration for HTTP(S) requests.
+- **FTP_USER / FTP_PASSWORD** (optional): FTP credentials used when URL doesn’t include auth.
+- **SFTP_USER / SFTP_PASSWORD** (optional): SFTP credentials used when URL doesn’t include auth.
+- **SMB_USER / SMB_PASSWORD / SMB_DOMAIN** (optional): SMB credentials/domain used when URL doesn’t include auth.
+- **IPFS_GATEWAY** (optional): Gateway used for `ipfs://` URLs (default: `https://ipfs.io`).
+- **GEMINI_INSECURE=1** (optional): disables TLS certificate verification for Gemini fetches in `web_url_read`.
 
 To fully customize the HTTP client (e.g. custom certs, proxy, timeouts), override `#build_http(uri)` in a subclass, or pass a `configure_http:` callable to the client; it is invoked with the `Net::HTTP` instance and the request URI before each request.
 
+## MCP Tools
+
+The MCP server provides the following tools:
+
+1. **searxng_web_search** - Web search via SearXNG
+   - Parameters: `query` (required), `pageno` (optional), `max_results` (optional, default 10), `time_range` (optional), `language` (optional), `safesearch` (optional). Use `max_results` to limit how many results are returned in one response (reduces token usage); use `pageno` for the next page.
+2. **web_url_read** - Fetch a URL and return Markdown content
+   - Parameters: `url` (required), `startChar`, `maxLength`, `section`, `paragraphRange`, `readHeadings`, `timeoutMs` (all optional).
+   - Uses `teplo_core` for HTML/XML to Markdown conversion (with workspace fallback to `textplorer/core-ruby` in development).
+   - Supports `http`, `https`, `ftp`, `sftp`, `smb`, `gemini`, and `ipfs` URLs (`ipfs://` is resolved via `IPFS_GATEWAY`).
+
+The MCP server also exposes resources:
+
+- `config://server-config` - current server/environment capabilities
+- `help://usage-guide` - concise usage and configuration guide
+
 ### Cursor IDE Configuration
 
 For Cursor IDE, create or update `.cursor/mcp.json` in your project:
@@ -120,37 +160,18 @@ Useful for testing the `searxng_web_search` tool before integrating with Cursor
 - **MCP Server Integration**: Ready-to-use MCP server with web search tool, compatible with Cursor IDE, Claude Desktop, and other MCP-enabled tools
 - **Configurable HTTP**: Optional custom CA, verify mode, and `configure_http` hook or `build_http` override for proxy/timeouts/certs
 - **Basic Auth**: Optional Basic authentication via options or ENV
+- **Fail-fast MCP validation**: MCP startup validates `SEARXNG_URL` and auth variable pairs
+- **User-facing errors**: Clear network/HTTP/configuration error messages
 
-## Basic Usage
-
-### Ruby API
-
-```ruby
-require "searxng"
-
-client = Searxng::Client.new
-data = client.search("ruby programming", pageno: 1, language: "en")
-
-data[:results].each do |r|
-  puts r[:title], r[:url], r[:content]
-end
-```
-
-### CLI
-
-```bash
-searxng search "your query"
-searxng search "ruby" --page 2 --language en --time-range day --json
-```
-
-Options: `--url`, `--page`, `--language`, `--time-range` (day|month|year), `--safesearch` (0|1|2), `--json`.
-
-## MCP Tools
-
-The MCP server provides the following tools:
+## Protocol Support Notes
 
-1. **searxng_web_search** - Web search via SearXNG
-   - Parameters: `query` (required), `pageno` (optional), `max_results` (optional, default 10), `time_range` (optional), `language` (optional), `safesearch` (optional). Use `max_results` to limit how many results are returned in one response (reduces token usage); use `pageno` for the next page.
+- `http` / `https`: fully supported.
+- `ftp`: supported by `web_url_read` (requires `net-ftp` runtime gem).
+- `sftp`: supported by `web_url_read` (requires `net-sftp` runtime gem).
+- `smb`: supported by `web_url_read` via `smbclient` available in `PATH`.
+- `gemini`: supported by `web_url_read` (Gemini fetch + Gemtext to Markdown conversion).
+- `ipfs`: supported by `web_url_read` through gateway resolution (`ipfs://` -> `IPFS_GATEWAY`).
+- `tor` / `i2p`: supported through `.onion` / `.i2p` hosts over HTTP(S), typically with proxy configuration.
 
 ## Examples
 
@@ -199,3 +220,11 @@ If you discover a security vulnerability, please report it responsibly. See [SEC
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Sponsors
+
+Sponsored by [Kisko Labs](https://www.kiskolabs.com).
+
+<a href="https://www.kiskolabs.com">
+  <img src="kisko.svg" width="200" alt="Sponsored by Kisko Labs" />
+</a>

data/lib/searxng/client.rb

@@ -2,6 +2,8 @@ require "uri"
 require "net/http"
 require "openssl"
 require "json"
+require "ipaddr"
+require "timeout"
 
 module Searxng
   class Client
@@ -23,8 +25,8 @@ module Searxng
       configure_http: nil
     )
       @base_url = (base_url || ENV["SEARXNG_URL"] || DEFAULT_BASE_URL).to_s.chomp("/")
-      @user = user || ENV["SEARXNG_USER"]
-      @password = password || ENV["SEARXNG_PASSWORD"]
+      @user = user || ENV["SEARXNG_USER"] || ENV["AUTH_USERNAME"]
+      @password = password || ENV["SEARXNG_PASSWORD"] || ENV["AUTH_PASSWORD"]
       @open_timeout = open_timeout
       @read_timeout = read_timeout
       @ca_file = ca_file || ENV["SEARXNG_CA_FILE"]
@@ -35,7 +37,7 @@ module Searxng
     end
 
     def search(query, pageno: 1, time_range: nil, language: "all", safesearch: nil)
-      raise ConfigurationError, "SEARXNG_URL not set" if @base_url.nil? || @base_url.empty?
+      raise ConfigurationError, ErrorMessages.configuration_missing_url if @base_url.nil? || @base_url.empty?
 
       uri = build_search_uri(query, pageno: pageno, time_range: time_range, language: language, safesearch: safesearch)
       response = perform_request(uri)
@@ -65,13 +67,19 @@ module Searxng
 
       http.request(request)
     rescue SocketError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET, Errno::EPIPE, Timeout::Error, OpenSSL::SSL::SSLError, EOFError => e
-      raise NetworkError, "Connection failed: #{e.message}"
+      raise NetworkError, ErrorMessages.network_error(e, uri: uri)
     end
 
     # Builds and returns a configured Net::HTTP instance. Override in a subclass to customize
     # timeouts, SSL, proxy, or other options. Called once per request.
     def build_http(uri)
-      http = Net::HTTP.new(uri.host, uri.port)
+      proxy_uri = proxy_uri_for(uri)
+      http_klass = if proxy_uri
+        Net::HTTP::Proxy(proxy_uri.host, proxy_uri.port, proxy_uri.user, proxy_uri.password)
+      else
+        Net::HTTP
+      end
+      http = http_klass.new(uri.host, uri.port)
       http.use_ssl = (uri.scheme == "https")
       http.open_timeout = @open_timeout
       http.read_timeout = @read_timeout
@@ -103,7 +111,7 @@ module Searxng
         }
       else
         raise APIError.new(
-          "SearXNG returned #{response.code} #{response.message}",
+          ErrorMessages.api_error(response.code.to_i, response.message),
           status_code: response.code.to_i,
           response_data: response.body,
           uri: uri.to_s
@@ -117,6 +125,51 @@ module Searxng
       )
     end
 
+    def proxy_uri_for(uri)
+      return nil if no_proxy_match?(uri.host)
+
+      raw = if uri.scheme == "https"
+        ENV["HTTPS_PROXY"] || ENV["https_proxy"] || ENV["HTTP_PROXY"] || ENV["http_proxy"] || ENV["ALL_PROXY"] || ENV["all_proxy"]
+      else
+        ENV["HTTP_PROXY"] || ENV["http_proxy"] || ENV["ALL_PROXY"] || ENV["all_proxy"]
+      end
+      return nil if raw.nil? || raw.strip.empty?
+
+      URI(raw.include?("://") ? raw : "http://#{raw}")
+    rescue
+      nil
+    end
+
+    def no_proxy_match?(host)
+      return false if host.nil? || host.empty?
+
+      raw = ENV["NO_PROXY"] || ENV["no_proxy"]
+      return false if raw.nil? || raw.strip.empty?
+
+      host_ip = begin
+        IPAddr.new(host)
+      rescue
+        nil
+      end
+      raw.split(",").any? do |entry|
+        token = entry.to_s.strip
+        next false if token.empty?
+        return true if token == "*"
+
+        cidr = begin
+          IPAddr.new(token)
+        rescue
+          nil
+        end
+        if cidr && host_ip
+          next cidr.include?(host_ip)
+        end
+
+        normalized = token.sub(/\A\./, "")
+        host == normalized || host.end_with?(".#{normalized}")
+      end
+    end
+
     def normalize_results(results)
       Array(results).map do |r|
         {

data/lib/searxng/errors.rb

@@ -15,4 +15,60 @@ module Searxng
       @uri = uri
     end
   end
+
+  module ErrorMessages
+    module_function
+
+    def configuration_missing_url
+      "SEARXNG_URL not set. Set it to your SearXNG instance (e.g., http://localhost:8080 or https://search.example.com)"
+    end
+
+    def configuration_invalid_url(url)
+      "SEARXNG_URL has invalid format: #{url.inspect}. Use format: http://localhost:8080 or https://search.example.com"
+    end
+
+    def configuration_invalid_protocol(protocol)
+      "SEARXNG_URL must use http or https protocol, got: #{protocol.inspect}"
+    end
+
+    def configuration_auth_pair
+      "Authentication variables must be set together: SEARXNG_USER with SEARXNG_PASSWORD (or AUTH_USERNAME with AUTH_PASSWORD)"
+    end
+
+    def no_results(query)
+      %(No results found for "#{query}". Try different search terms or check if SearXNG search engines are working.)
+    end
+
+    def network_error(exception, uri: nil, target: "SearXNG server")
+      case exception
+      when SocketError
+        host = uri&.host || "unknown host"
+        %(DNS Error: Cannot resolve hostname "#{host}".)
+      when Errno::ECONNREFUSED
+        "Connection Error: #{target} is not responding."
+      when Errno::ETIMEDOUT, Timeout::Error
+        "Timeout Error: #{target} is too slow to respond."
+      when OpenSSL::SSL::SSLError
+        "SSL Error: Certificate or TLS problem while connecting to #{target}."
+      else
+        message = exception&.message.to_s.strip
+        "Network Error: #{message.empty? ? "Connection failed" : message}"
+      end
+    end
+
+    def api_error(status_code, status_message)
+      case status_code.to_i
+      when 403
+        "SearXNG Error (403): Authentication required or IP blocked."
+      when 404
+        "SearXNG Error (404): Search endpoint not found."
+      when 429
+        "SearXNG Error (429): Rate limit exceeded."
+      when 500..599
+        "SearXNG Error (#{status_code}): Internal server error."
+      else
+        "SearXNG returned #{status_code} #{status_message}".strip
+      end
+    end
+  end
 end

data/lib/searxng/server.rb

@@ -1,5 +1,13 @@
 require "fast_mcp"
 require "searxng"
+require "json"
+require "net/http"
+require "openssl"
+require "open3"
+require "socket"
+require "time"
+require "timeout"
+require "uri"
 
 FastMcp = MCP unless defined?(FastMcp)
 
@@ -53,15 +61,47 @@ module Searxng
     end
 
     def self.start
+      validate_environment!
       server = FastMcp::Server.new(
         name: "searxng",
         version: Searxng::VERSION,
         logger: NullLogger.new
       )
-      server.register_tool(SearxngWebSearchTool)
+      register_tools(server)
+      register_resources(server)
       server.start
     end
 
+    def self.register_tools(server)
+      server.register_tool(SearxngWebSearchTool)
+      server.register_tool(WebUrlReadTool)
+    end
+
+    def self.register_resources(server)
+      server.register_resource(ServerConfigResource)
+      server.register_resource(UsageGuideResource)
+    end
+
+    def self.validate_environment!
+      searxng_url = ENV["SEARXNG_URL"]
+      if searxng_url.nil? || searxng_url.strip.empty?
+        raise ConfigurationError, ErrorMessages.configuration_missing_url
+      end
+
+      uri = URI.parse(searxng_url)
+      unless %w[http https].include?(uri.scheme)
+        raise ConfigurationError, ErrorMessages.configuration_invalid_protocol(uri.scheme)
+      end
+
+      user = ENV["SEARXNG_USER"] || ENV["AUTH_USERNAME"]
+      password = ENV["SEARXNG_PASSWORD"] || ENV["AUTH_PASSWORD"]
+      if (user && !password) || (!user && password)
+        raise ConfigurationError, ErrorMessages.configuration_auth_pair
+      end
+    rescue URI::InvalidURIError
+      raise ConfigurationError, ErrorMessages.configuration_invalid_url(searxng_url)
+    end
+
     class BaseTool < FastMcp::Tool
       protected
 
@@ -73,6 +113,7 @@ module Searxng
     class SearxngWebSearchTool < BaseTool
       tool_name "searxng_web_search"
       description "Performs a web search using the SearXNG API. Use this to find information on the web. Aggregates results from multiple search engines."
+      annotations(readOnlyHint: true, openWorldHint: true) if respond_to?(:annotations)
 
       arguments do
         required(:query).filled(:string).description("The search query")
@@ -98,6 +139,7 @@ module Searxng
 
       def format_search_result(data, max_results: 10, pageno: 1)
         out = []
+        query = data[:query].to_s
         data[:infoboxes]&.each do |ib|
           out << "Infobox: #{ib[:infobox]}"
           out << "ID: #{ib[:id]}"
@@ -107,7 +149,7 @@ module Searxng
         results = data[:results] || []
         total = data[:number_of_results]
         if results.empty?
-          out << "No results found"
+          out << ErrorMessages.no_results(query.empty? ? "your query" : query)
         else
           limit = [max_results.to_i, 1].max
           shown = results.first(limit)
@@ -127,5 +169,535 @@ module Searxng
         out.join("\n").strip
       end
     end
+
+    class WebUrlReadTool < BaseTool
+      tool_name "web_url_read"
+      description "Reads a URL and converts HTML/XML content into Markdown. Supports character pagination, section extraction, paragraph ranges, and heading-only mode."
+      annotations(readOnlyHint: true, openWorldHint: true) if respond_to?(:annotations)
+
+      CACHE_TTL_SECONDS = 300
+      @@url_cache = {} # rubocop:disable Style/ClassVars
+
+      arguments do
+        required(:url).filled(:string).description("URL to fetch and read")
+        optional(:startChar).filled(:integer).description("Starting character position (default: 0)")
+        optional(:maxLength).filled(:integer).description("Maximum characters to return")
+        optional(:section).filled(:string).description("Extract by heading text")
+        optional(:paragraphRange).filled(:string).description("Paragraph range, e.g. '1-5', '3', '10-'")
+        optional(:readHeadings).filled(:bool).description("Return only headings when true")
+        optional(:timeoutMs).filled(:integer).description("Request timeout in ms (default: 10000)")
+      end
+
+      def call(url:, **kwargs)
+        start_char = kwargs.fetch(:startChar, 0)
+        max_length = kwargs[:maxLength]
+        section = kwargs[:section]
+        paragraph_range = kwargs[:paragraphRange]
+        read_headings = kwargs.fetch(:readHeadings, false)
+        timeout_ms = kwargs.fetch(:timeoutMs, 10_000)
+
+        normalized = normalize_url(url)
+        uri = URI.parse(normalized)
+        markdown = case uri.scheme
+        when "http", "https"
+          html = fetch_html(normalized, timeout_ms: timeout_ms.to_i)
+          convert_to_markdown(html, normalized)
+        when "ftp"
+          content = fetch_ftp(uri, timeout_ms: timeout_ms.to_i)
+          convert_fetched_content(content, normalized)
+        when "sftp"
+          content = fetch_sftp(uri, timeout_ms: timeout_ms.to_i)
+          convert_fetched_content(content, normalized)
+        when "smb"
+          content = fetch_smb(uri, timeout_ms: timeout_ms.to_i)
+          convert_fetched_content(content, normalized)
+        when "gemini"
+          gemtext = fetch_gemini(uri, timeout_ms: timeout_ms.to_i)
+          gemtext_to_markdown(gemtext, uri)
+        when "ipfs"
+          gateway_url = resolve_ipfs_url(uri)
+          html = fetch_html(gateway_url, timeout_ms: timeout_ms.to_i)
+          convert_to_markdown(html, gateway_url)
+        else
+          raise ConfigurationError, %(Unsupported URL scheme "#{uri.scheme}". Supported schemes: http, https, ftp, sftp, smb, gemini, ipfs.)
+        end
+        apply_options(markdown, start_char: start_char, max_length: max_length, section: section, paragraph_range: paragraph_range, read_headings: read_headings)
+      end
+
+      private
+
+      def normalize_url(url)
+        raw = url.to_s.strip
+        uri = URI.parse(raw)
+        if uri.scheme
+          scheme = uri.scheme.downcase
+          return raw if uri.host && %w[http https ftp sftp smb gemini ipfs].include?(scheme)
+
+          raise ConfigurationError, %(Unsupported URL scheme "#{uri.scheme}". Supported schemes: http, https, ftp, sftp, smb, gemini, ipfs.)
+        end
+
+        candidate = "https://#{raw}"
+        parsed = URI.parse(candidate)
+        return candidate if parsed.host
+
+        raise ArgumentError
+      rescue URI::InvalidURIError, ArgumentError
+        raise ConfigurationError, %(URL Format Error: Invalid URL "#{url}")
+      end
+
+      def fetch_html(url, timeout_ms:)
+        now = Time.now.to_i
+        cached = @@url_cache[url]
+        if cached && (now - cached[:at] <= CACHE_TTL_SECONDS)
+          return cached[:html]
+        end
+
+        uri = URI.parse(url)
+        client = get_client
+        http = client.send(:build_http, uri)
+        request = Net::HTTP::Get.new(uri)
+        request["User-Agent"] = "searxng-ruby/#{Searxng::VERSION} web_url_read"
+        response = nil
+        Timeout.timeout(timeout_ms / 1000.0) { response = http.request(request) }
+        unless response.is_a?(Net::HTTPSuccess)
+          raise APIError.new(
+            ErrorMessages.api_error(response.code.to_i, response.message),
+            status_code: response.code.to_i,
+            response_data: response.body,
+            uri: url
+          )
+        end
+
+        body = response.body.to_s
+        if body.strip.empty?
+          raise APIError.new("Content Error: Website returned empty content.", status_code: response.code.to_i, uri: url)
+        end
+
+        @@url_cache[url] = {at: now, html: body}
+        body
+      rescue Timeout::Error
+        raise NetworkError, "Timeout Error: #{URI.parse(url).host} took longer than #{timeout_ms}ms to respond"
+      rescue SocketError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET, Errno::EPIPE, OpenSSL::SSL::SSLError, EOFError => e
+        raise NetworkError, ErrorMessages.network_error(e, uri: URI.parse(url), target: "website")
+      end
+
+      def convert_to_markdown(html, base_url)
+        load_teplo_core!
+        ast = TeploCore.parse_html(html)
+        markdown = TeploCore.ast_to_markdown(ast, base_url)
+        if markdown.to_s.strip.empty?
+          "Content Warning: Page fetched but appears empty after conversion (#{base_url}). May contain only media or require JavaScript."
+        else
+          markdown
+        end
+      rescue => e
+        raise APIError.new("Conversion Error: Cannot convert HTML to Markdown (#{base_url}) - #{e.message}", uri: base_url)
+      end
+
+      def convert_fetched_content(content, base_url)
+        text = content.to_s.encode("UTF-8", invalid: :replace, undef: :replace, replace: "")
+        return text if text.strip.empty?
+
+        looks_like_markup = text.match?(/\A\s*<(?:!doctype|html|body|xml|rss|feed|svg|\?xml)/i)
+        looks_like_markup ? convert_to_markdown(text, base_url) : text
+      end
+
+      def fetch_ftp(uri, timeout_ms:)
+        begin
+          require "net/ftp"
+        rescue LoadError
+          raise ConfigurationError, "FTP support requires the 'net-ftp' gem."
+        end
+
+        body = +""
+        Timeout.timeout(timeout_ms / 1000.0) do
+          ftp = Net::FTP.new
+          begin
+            ftp.connect(uri.host, uri.port || 21)
+            ftp.read_timeout = [timeout_ms / 1000.0, 1.0].max
+            ftp.open_timeout = [timeout_ms / 1000.0, 1.0].max
+            user = uri.user || ENV["FTP_USER"] || "anonymous"
+            pass = uri.password || ENV["FTP_PASSWORD"] || "anonymous@"
+            ftp.login(user, pass)
+            path = uri.path.to_s
+            raise APIError.new("FTP path is required", uri: uri.to_s) if path.empty? || path == "/"
+
+            ftp.retrbinary("RETR #{path}", 16_384) { |chunk| body << chunk }
+          ensure
+            ftp.close unless ftp.closed?
+          end
+        end
+        body
+      rescue Timeout::Error
+        raise NetworkError, "Timeout Error: #{uri.host} took longer than #{timeout_ms}ms to respond"
+      rescue SocketError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET, Errno::EPIPE, EOFError => e
+        raise NetworkError, ErrorMessages.network_error(e, uri: uri, target: "FTP server")
+      rescue => e
+        if e.class.name.start_with?("Net::FTP")
+          raise APIError.new("FTP Error: #{e.message}", uri: uri.to_s)
+        end
+        raise e
+      end
+
+      def fetch_sftp(uri, timeout_ms:)
+        begin
+          require "net/sftp"
+        rescue LoadError
+          raise ConfigurationError, "SFTP support requires the 'net-sftp' gem."
+        end
+
+        user = uri.user || ENV["SFTP_USER"]
+        password = uri.password || ENV["SFTP_PASSWORD"]
+        raise ConfigurationError, "SFTP requires username (sftp://user@host/path or SFTP_USER)." unless user
+
+        path = uri.path.to_s
+        raise APIError.new("SFTP path is required", uri: uri.to_s) if path.empty? || path == "/"
+
+        result = nil
+        Timeout.timeout(timeout_ms / 1000.0) do
+          Net::SFTP.start(uri.host, user, password: password, port: uri.port || 22, non_interactive: true, verify_host_key: :never) do |sftp|
+            result = sftp.download!(path)
+          end
+        end
+        result.to_s
+      rescue Timeout::Error
+        raise NetworkError, "Timeout Error: #{uri.host} took longer than #{timeout_ms}ms to respond"
+      rescue SocketError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET, Errno::EPIPE, EOFError => e
+        raise NetworkError, ErrorMessages.network_error(e, uri: uri, target: "SFTP server")
+      rescue => e
+        raise e if e.is_a?(ConfigurationError)
+        if e.class.name.include?("StatusException")
+          description = e.respond_to?(:description) ? e.description : e.message
+          raise APIError.new("SFTP Error: #{description}", uri: uri.to_s)
+        end
+        if e.class.name.include?("AuthenticationFailed")
+          raise APIError.new("SFTP authentication failed", uri: uri.to_s)
+        end
+        raise APIError.new("SFTP Error: #{e.message}", uri: uri.to_s)
+      end
+
+      def fetch_smb(uri, timeout_ms:)
+        share, remote_path = parse_smb_path(uri)
+        user = uri.user || ENV["SMB_USER"]
+        pass = uri.password || ENV["SMB_PASSWORD"]
+        domain = ENV["SMB_DOMAIN"]
+
+        auth = if user
+          full_user = (domain && !domain.empty?) ? "#{domain}\\#{user}" : user
+          "#{full_user}%#{pass}"
+        end
+
+        escaped_remote = remote_path.gsub('"', '\"')
+        cmd = ["smbclient", "//#{uri.host}/#{share}", "-c", %(get "#{escaped_remote}" -)]
+        if auth
+          cmd << "-U" << auth
+        else
+          cmd << "-N"
+        end
+
+        stdout = +""
+        stderr = +""
+        status = nil
+        Timeout.timeout(timeout_ms / 1000.0) do
+          stdout, stderr, status = Open3.capture3(*cmd)
+        end
+
+        raise APIError.new("SMB Error: #{stderr.strip.empty? ? "request failed" : stderr.strip}", uri: uri.to_s) unless status.success?
+
+        stdout
+      rescue Errno::ENOENT
+        raise ConfigurationError, "SMB support requires 'smbclient' to be installed and available in PATH."
+      rescue Timeout::Error
+        raise NetworkError, "Timeout Error: #{uri.host} took longer than #{timeout_ms}ms to respond"
+      rescue SocketError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET, Errno::EPIPE, EOFError => e
+        raise NetworkError, ErrorMessages.network_error(e, uri: uri, target: "SMB server")
+      end
+
+      def parse_smb_path(uri)
+        segments = uri.path.to_s.split("/").reject(&:empty?)
+        share = segments.shift
+        remote = segments.join("/")
+        raise APIError.new("SMB path must include share and file (smb://host/share/path/file)", uri: uri.to_s) if share.to_s.empty? || remote.empty?
+
+        [share, remote]
+      end
+
+      def fetch_gemini(uri, timeout_ms:)
+        response_header = nil
+        body = +""
+        Timeout.timeout(timeout_ms / 1000.0) do
+          tcp = TCPSocket.new(uri.host, uri.port || 1965)
+          begin
+            ctx = OpenSSL::SSL::SSLContext.new
+            ctx.verify_mode = (ENV["GEMINI_INSECURE"] == "1") ? OpenSSL::SSL::VERIFY_NONE : OpenSSL::SSL::VERIFY_PEER
+            ssl = OpenSSL::SSL::SSLSocket.new(tcp, ctx)
+            ssl.hostname = uri.host if ssl.respond_to?(:hostname=)
+            ssl.connect
+            ssl.write("#{uri}\r\n")
+            response_header = ssl.gets("\r\n")
+            body = ssl.read.to_s
+            ssl.close
+          ensure
+            tcp.close unless tcp.closed?
+          end
+        end
+
+        unless response_header
+          raise APIError.new("Gemini response error: empty response header", uri: uri.to_s)
+        end
+
+        status, meta = response_header.strip.split(/\s+/, 2)
+        status_code = status.to_i
+        case status_code
+        when 20..29
+          body
+        when 30..39
+          raise APIError.new("Gemini redirect (#{status_code}): #{meta}", status_code: status_code, uri: uri.to_s)
+        when 40..49
+          raise APIError.new("Gemini temporary failure (#{status_code}): #{meta}", status_code: status_code, uri: uri.to_s)
+        when 50..59
+          raise APIError.new("Gemini permanent failure (#{status_code}): #{meta}", status_code: status_code, uri: uri.to_s)
+        when 60..69
+          raise APIError.new("Gemini certificate required (#{status_code}): #{meta}", status_code: status_code, uri: uri.to_s)
+        else
+          raise APIError.new("Gemini response error (#{status_code}): #{meta}", status_code: status_code, uri: uri.to_s)
+        end
+      rescue Timeout::Error
+        raise NetworkError, "Timeout Error: #{uri.host} took longer than #{timeout_ms}ms to respond"
+      rescue SocketError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET, Errno::EPIPE, OpenSSL::SSL::SSLError, EOFError => e
+        raise NetworkError, ErrorMessages.network_error(e, uri: uri, target: "Gemini server")
+      end
+
+      def gemtext_to_markdown(gemtext, base_uri)
+        lines = gemtext.to_s.split("\n")
+        out = lines.map do |line|
+          if line.start_with?("=>")
+            target, *label_parts = line.sub(/\A=>\s*/, "").split(/\s+/)
+            label = label_parts.join(" ").strip
+            next "" if target.to_s.empty?
+            resolved = resolve_relative_uri(base_uri, target)
+            display = label.empty? ? resolved : label
+            "- [#{display}](#{resolved})"
+          elsif line.start_with?("### ")
+            "### #{line[4..].to_s.strip}"
+          elsif line.start_with?("## ")
+            "## #{line[3..].to_s.strip}"
+          elsif line.start_with?("# ")
+            "# #{line[2..].to_s.strip}"
+          elsif line.start_with?("> ")
+            "> #{line[2..]}"
+          else
+            line
+          end
+        end
+        out.join("\n")
+      end
+
+      def resolve_relative_uri(base_uri, target)
+        target_uri = URI.parse(target)
+        return target if target_uri.scheme
+
+        URI.join(base_uri.to_s, target).to_s
+      rescue URI::InvalidURIError
+        target
+      end
+
+      def resolve_ipfs_url(uri)
+        gateway = ENV["IPFS_GATEWAY"] || "https://ipfs.io"
+        gateway = gateway.sub(%r{/+\z}, "")
+        parsed_gateway = URI.parse(gateway)
+        unless %w[http https].include?(parsed_gateway.scheme)
+          raise ConfigurationError, "IPFS_GATEWAY must use http or https, got: #{parsed_gateway.scheme.inspect}"
+        end
+
+        cid = uri.host.to_s
+        path = uri.path.to_s
+        query = uri.query ? "?#{uri.query}" : ""
+        "#{gateway}/ipfs/#{cid}#{path}#{query}"
+      rescue URI::InvalidURIError
+        raise ConfigurationError, "IPFS_GATEWAY has invalid format: #{gateway.inspect}"
+      end
+
+      def load_teplo_core!
+        return if defined?(TeploCore)
+
+        require "teplo_core"
+      rescue LoadError
+        local = File.expand_path("../../../textplorer/core-ruby/lib/teplo_core", __dir__)
+        if File.exist?("#{local}.rb")
+          require local
+        else
+          raise ConfigurationError, "web_url_read requires teplo_core. Install the gem or provide textplorer/core-ruby in the expected workspace location."
+        end
+      end
+
+      def apply_options(markdown, start_char:, max_length:, section:, paragraph_range:, read_headings:)
+        result = markdown.to_s
+        return extract_headings(result) if read_headings
+
+        if section && !section.strip.empty?
+          section_text = extract_section(result, section)
+          result = section_text.empty? ? %(Section "#{section}" not found in the content.) : section_text
+        end
+
+        if paragraph_range && !paragraph_range.strip.empty?
+          paragraph_text = extract_paragraph_range(result, paragraph_range)
+          result = paragraph_text.empty? ? %(Paragraph range "#{paragraph_range}" is invalid or out of bounds.) : paragraph_text
+        end
+
+        start = [start_char.to_i, 0].max
+        result = (start >= result.length) ? "" : result[start..]
+
+        if max_length
+          max = max_length.to_i
+          result = result[0, max] if max.positive?
+        end
+        result
+      end
+
+      def extract_section(markdown, heading)
+        lines = markdown.split("\n")
+        section_regex = /^\#{1,6}\s*.*#{Regexp.escape(heading)}.*$/i
+
+        start_index = -1
+        current_level = 0
+        lines.each_with_index do |line, idx|
+          next unless line.match?(section_regex)
+
+          start_index = idx
+          current_level = line[/^#+/].to_s.length
+          break
+        end
+        return "" if start_index < 0
+
+        end_index = lines.length
+        lines[(start_index + 1)..]&.each_with_index do |line, offset|
+          level = line[/^#+/].to_s.length
+          next if level.zero? || level > current_level
+
+          end_index = start_index + 1 + offset
+          break
+        end
+
+        lines[start_index...end_index].join("\n")
+      end
+
+      def extract_paragraph_range(markdown, range)
+        paragraphs = markdown.split(/\n{2,}/).map(&:strip).reject(&:empty?)
+        match = range.to_s.strip.match(/\A(\d+)(?:-(\d*))?\z/)
+        return "" unless match
+
+        start = match[1].to_i - 1
+        return "" if start.negative? || start >= paragraphs.length
+
+        if match[2].nil?
+          paragraphs[start].to_s
+        elsif match[2].empty?
+          paragraphs[start..].join("\n\n")
+        else
+          ending = match[2].to_i
+          paragraphs[start...ending].join("\n\n")
+        end
+      end
+
+      def extract_headings(markdown)
+        headings = markdown.split("\n").select { |line| line.match?(/^\#{1,6}\s+/) }
+        headings.empty? ? "No headings found in the content." : headings.join("\n")
+      end
+    end
+
+    class ServerConfigResource < FastMcp::Resource
+      uri "config://server-config"
+      resource_name "Server Configuration"
+      description "Current SearXNG MCP server configuration and capabilities"
+      mime_type "application/json"
+
+      def content
+        searxng_url = ENV["SEARXNG_URL"].to_s
+        user = ENV["SEARXNG_USER"] || ENV["AUTH_USERNAME"]
+        password = ENV["SEARXNG_PASSWORD"] || ENV["AUTH_PASSWORD"]
+        config = {
+          server_info: {
+            name: "searxng",
+            version: Searxng::VERSION
+          },
+          environment: {
+            searxng_url: safe_url(searxng_url),
+            has_auth: !!(user && password),
+            has_proxy: !!(ENV["HTTP_PROXY"] || ENV["HTTPS_PROXY"] || ENV["http_proxy"] || ENV["https_proxy"]),
+            has_no_proxy: !!(ENV["NO_PROXY"] || ENV["no_proxy"])
+          },
+          capabilities: {
+            tools: %w[searxng_web_search web_url_read],
+            resources: %w[config://server-config help://usage-guide],
+            transport: ["stdio"]
+          }
+        }
+        JSON.pretty_generate(config)
+      end
+
+      private
+
+      def safe_url(raw)
+        return "(not configured)" if raw.nil? || raw.strip.empty?
+
+        uri = URI.parse(raw)
+        uri.user = nil
+        uri.password = nil
+        uri.to_s
+      rescue URI::InvalidURIError
+        "(invalid URL)"
+      end
+    end
+
+    class UsageGuideResource < FastMcp::Resource
+      uri "help://usage-guide"
+      resource_name "Usage Guide"
+      description "Short guide for using SearXNG MCP tools and environment variables"
+      mime_type "text/markdown"
+
+      def content
+        <<~MD
+          # SearXNG MCP Server Help
+
+          ## Tools
+          1. `searxng_web_search` - Search the web via SearXNG.
+          2. `web_url_read` - Fetch a URL and return Markdown-converted content.
+
+          ## Required Environment
+          - `SEARXNG_URL` (must be `http://` or `https://`)
+
+          ## Optional Environment
+          - `SEARXNG_USER` and `SEARXNG_PASSWORD` (or `AUTH_USERNAME` and `AUTH_PASSWORD`)
+          - `HTTP_PROXY` / `HTTPS_PROXY`
+          - `ALL_PROXY` (for shared proxy config)
+          - `NO_PROXY`
+          - `FTP_USER` / `FTP_PASSWORD` (optional FTP credentials)
+          - `SFTP_USER` / `SFTP_PASSWORD` (optional SFTP credentials)
+          - `SMB_USER` / `SMB_PASSWORD` / `SMB_DOMAIN` (optional SMB credentials)
+          - `IPFS_GATEWAY` (default: `https://ipfs.io`)
+          - `GEMINI_INSECURE=1` (optional, disables TLS verification for Gemini)
+
+          ## Protocol Notes
+          - `http` / `https`: fully supported (`searxng_web_search` + `web_url_read`)
+          - `ftp`: supported in `web_url_read` (file retrieval via FTP)
+          - `sftp`: supported in `web_url_read` (requires `net-sftp` gem)
+          - `smb`: supported in `web_url_read` via `smbclient` command
+          - `gemini`: supported in `web_url_read` (Gemini fetch + Gemtext conversion)
+          - `ipfs`: supported via HTTP gateway mapping (`ipfs://...` -> `IPFS_GATEWAY`)
+          - `tor` / `i2p`: supported via `.onion` / `.i2p` hosts over HTTP(S) with proxy configuration
+          - `socks5`: configure via proxy environment and run through a local bridge/proxy compatible with your client setup
+
+          ## Examples
+          - Search: `{"query":"latest ruby news","time_range":"day"}`
+          - Read URL: `{"url":"https://example.com","section":"Introduction","maxLength":2000}`
+          - Read FTP: `{"url":"ftp://ftp.example.com/path/file.txt"}`
+          - Read SFTP: `{"url":"sftp://user@example.com/path/file.txt"}`
+          - Read SMB: `{"url":"smb://fileserver/share/path/file.txt"}`
+          - Read Gemini: `{"url":"gemini://geminiprotocol.net"}`
+          - Read IPFS: `{"url":"ipfs://bafybeigdyrzt.../index.html"}`
+        MD
+      end
+    end
   end
 end

data/lib/searxng/version.rb

@@ -1,3 +1,3 @@
 module Searxng
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

data/sig/searxng.rbs

@@ -12,6 +12,16 @@ module Searxng
     attr_reader uri: String?
   end
 
+  module ErrorMessages
+    def self.configuration_missing_url: () -> String
+    def self.configuration_invalid_url: (untyped url) -> String
+    def self.configuration_invalid_protocol: (untyped protocol) -> String
+    def self.configuration_auth_pair: () -> String
+    def self.no_results: (String query) -> String
+    def self.network_error: (Exception exception, ?uri: URI::Generic?, ?target: String) -> String
+    def self.api_error: (Integer status_code, String status_message) -> String
+  end
+
   class Client
     def initialize: (
       ?base_url: String?,
@@ -23,7 +33,7 @@ module Searxng
       ?ca_path: String?,
       ?verify_mode: Integer?,
       ?user_agent: String?,
-      ?configure_http: (Net::HTTP, URI::Generic) -> void
+      ?configure_http: ((Net::HTTP, URI::Generic) -> void)?
     ) -> void
     def search: (
       String query,

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: searxng
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Andrei Makarov