prescient 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 167f50351b78b35d1ae63d3487285db48c07799ad015b5d49f46dd3c451afa49
-  data.tar.gz: 91e1895635015120c75b0b461fea017d8ec7c4b5ebd81f1d2b0c86ba37690e1d
+  metadata.gz: 176552789656cbda6e78951ab04eb0ecdfa9ffa53dcf88159bec5cdbf403cdef
+  data.tar.gz: 5173a6bdd607dc6aca121079ee6e81243a550575f4702c57133f637a31aac798
 SHA512:
-  metadata.gz: 6cdf4ed9b05525692898cdd88153f092c23f5caac1d422962bdc768055bb225e960e45a9fb877adb17a7c686e8bf608db09dea1b3e24ebc15767071255910285
-  data.tar.gz: ecec27b8c215dfe5fb92e00f68fd6a5ae66c8284342525c9ae635214318032fc4970a72d2944679e455915f2ba8de8425a07706530ca6c309bedc5d5d4ff114a
+  metadata.gz: fb309c19aa67f200e806b227f392ff5181e5ef7bc1ea00a6f9a0d4cf5ebbce989c0d9b472885a1c2795efaa7789a8ed61f912453fd20594fba029f8c7bc126d4
+  data.tar.gz: 70187123148e3c0aad70f88fa469c91e3c55200c1c36b9452697109e2746cfe0fe70ef8358b766b3f99b967884aba4b846f3a8cb634c76fe9803ad45925d0e50

data/CHANGELOG.md

@@ -2,6 +2,28 @@
 
 ## Unreleased
 
+## [0.7.0] - 2025-08-17
+
+### Added
+
+- Added an explicit external-tool contract with a configurable SearXNG web-search adapter.
+- Added YAML/schema configuration for tool registration, environment references, bounded requests, and normalized search results.
+- Added `prescient search` and an annotated `web_search` configuration example.
+- Added an optional development SearXNG service to `docker-compose.yml` for runnable web-search examples.
+- Added opt-in search-to-provider context assembly through `Prescient.search_and_generate` and `prescient search --generate`.
+- Added `POST /v1/search/generate` for REST API consumers to opt in to search-result context.
+- Added `POST /v1/search` for normalized raw external-tool results without AI generation.
+- Added SearchApi as a hosted web-search adapter with engine, location, language,
+  country, API-key, timeout, and result-limit configuration.
+- Added capability groups for ordered tool-adapter fallback on transient
+  connection and rate-limit failures.
+- Documented SearXNG and SearchApi setup and CLI usage side by side.
+- Added explicit `--generate` opt-in behavior to the web-search example.
+- Added `SEARXNG_URL` environment defaults for automatic `web_search` registration.
+- Expanded `prescient config example` with documented SearXNG settings,
+  environment references, custom tool names, and direct-versus-generated usage.
+- Organized CLI help into global and search-specific option sections.
+
 ## [0.6.0] - 2025-08-15
 
 ### Added

data/INTEGRATION_GUIDE.md

@@ -11,7 +11,7 @@ and [examples guide](examples/README.md).
 
 ```ruby
 # Add to your Gemfile
-gem 'prescient', '~> 0.6.0'
+gem 'prescient', '~> 0.7.0'
 ```
 
 ### 2. Replace Existing AI Service
@@ -190,6 +190,8 @@ This makes the following routes available under `/prescient`:
 | `GET` | `/v1/capabilities` | Provider capabilities |
 | `GET` | `/v1/health` | Provider health |
 | `POST` | `/v1/generate` | Text generation |
+| `POST` | `/v1/search` | Normalized external-tool search |
+| `POST` | `/v1/search/generate` | Search with opt-in AI generation |
 | `POST` | `/v1/embeddings` | Single embedding |
 | `POST` | `/v1/embeddings/batch` | Bounded batch embeddings |
 
@@ -202,6 +204,16 @@ curl -X POST http://localhost:3000/prescient/v1/generate \
   -H "Authorization: Bearer ${PRESCIENT_API_TOKEN}" \
   -H 'Content-Type: application/json' \
   -d '{"prompt":"Explain Ruby fibers"}'
+
+curl -X POST http://localhost:3000/prescient/v1/search/generate \
+  -H "Authorization: Bearer ${PRESCIENT_API_TOKEN}" \
+  -H 'Content-Type: application/json' \
+  -d '{"query":"Ruby HTTP clients","provider":"openai","limit":5}'
+
+curl -X POST http://localhost:3000/prescient/v1/search \
+  -H "Authorization: Bearer ${PRESCIENT_API_TOKEN}" \
+  -H 'Content-Type: application/json' \
+  -d '{"query":"Ruby HTTP clients","limit":5}'
 ```
 
 Responses include a request ID. Request bodies are size-limited, batch inputs

data/README.md

@@ -2,11 +2,12 @@
 
 [![Gem Version](https://img.shields.io/gem/v/prescient?logo=rubygems&logoColor=white)](https://rubygems.org/gems/prescient)
 [![Requires Ruby 3.1+](https://img.shields.io/badge/Requires-Ruby%203.1%2B-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org/)
+[![Docker](https://img.shields.io/badge/Docker-GHCR-2496ED?logo=docker&logoColor=white)](https://github.com/kanutocd/prescient/pkgs/container/prescient)
 [![CI](https://github.com/kanutocd/prescient/actions/workflows/ci.yml/badge.svg)](https://github.com/kanutocd/prescient/actions/workflows/ci.yml)
 [![Security](https://img.shields.io/github/actions/workflow/status/kanutocd/prescient/security.yml?branch=main&event=push&label=Security)](https://github.com/kanutocd/prescient/actions/workflows/security.yml)
 [![License](https://img.shields.io/badge/License-MIT-22C55E)](LICENSE.txt)
 
-Prescient is a boring AI provider abstraction for Ruby. Configure your AI providers once, then use the same interface regardless of whether the request is handled by OpenAI, Anthropic, Ollama, Hugging Face, Google Gemini, Mistral, DeepSeek, or xAI. Prescient handles provider selection, retries, health checks, and fallback.
+Prescient is a boring AI provider gateway implemented in Ruby. Configure your AI providers once, then use them through a consistent Ruby API, CLI, or REST API. Prescient handles provider selection, retries, health checks, and fallback across configured providers, including OpenAI, Anthropic, Ollama, Hugging Face, Google Gemini, Mistral, DeepSeek, and xAI.
 
 For focused guidance, see the **[examples guide](https://github.com/kanutocd/prescient/tree/main/examples)**,
 **[Rails integration guide](https://github.com/kanutocd/prescient/blob/main/INTEGRATION_GUIDE.md)**, and
@@ -14,13 +15,15 @@ For focused guidance, see the **[examples guide](https://github.com/kanutocd/pre
 
 ## Features
 
-- **Unified Interface**: Single API for multiple AI providers
-- **Local and Cloud Support**: Ollama for local/private deployments, cloud APIs for scale
-- **Embedding Generation**: Vector embeddings for semantic search and AI applications
-- **Text Completion**: Chat completions with context support
-- **Error Handling**: Robust error handling with automatic retries
-- **Health Monitoring**: Built-in health checks for all providers
-- **Flexible Configuration**: YAML, environment variable, and programmatic configuration
+- **Provider abstraction** — One consistent interface across supported AI providers
+- **Multiple interfaces** — Ruby API, CLI, and Rack-compatible REST API
+- **Text and embeddings** — Generate responses and embeddings with provider/model selection
+- **Reliability controls** — Retries, health checks, and fallback across configured providers
+- **Declarative configuration** — Versioned YAML, environment references, and JSON Schema validation
+- **Prompt and context customization** — Configurable prompt templates and context formatting
+- **External tools** — Explicit, normalized web-search integration with SearXNG and SearchApi
+- **Local and cloud support** — Ollama alongside hosted providers
+- **Optional integrations** — Docker deployment and pgvector support without making either mandatory
 
 ## Supported Providers
 
@@ -242,6 +245,8 @@ run Prescient::API.new(
 Available endpoints include:
 
 - **`POST /v1/generate`**
+- **`POST /v1/search`**
+- **`POST /v1/search/generate`**
 - **`POST /v1/embeddings`**
 - **`POST /v1/embeddings/batch`**
 - **`GET /v1/providers`**
@@ -347,25 +352,139 @@ YAML values, then built-in defaults. Use `prescient config validate` to check a
 configuration before running an operation, or `prescient config example` to
 generate an annotated starter file.
 
-Prompt templates can also be configured per provider. Use the YAML mapping for
-multiline templates, or pass a template file to a single CLI operation:
+### External Tools
+
+External tools are opt-in capability adapters, separate from AI providers.
+Supported web-search adapters are [SearXNG](https://searxng.org/) and
+[SearchApi](https://www.searchapi.io/):
+
+#### SearXNG
+
+Setting `SEARXNG_URL` registers the default `web_search` tool for CLI and Ruby
+environment-based configuration. YAML or programmatic configuration can be
+used when more control is needed.
 
 ```yaml
-providers:
-  openai:
-    type: openai
-    api_key_env: OPENAI_API_KEY
-    chat_model: gpt-4.1-mini
-    prompt_templates:
-      system_prompt: You are a concise assistant.
-      no_context_template: "%{system_prompt}\n\nUser: %{query}"
-      with_context_template: "%{system_prompt}\n\nContext:\n%{context}\n\nUser: %{query}"
+tools:
+  web_search:
+    type: searxng
+    url_env: SEARXNG_URL
+    language: en
+    categories: [general, news]
+    timeout: 5
+    max_results: 5
 ```
 
+Run the SearXNG-backed tool:
+
 ```bash
-prescient generate --prompt-templates-file prompts.yml "Summarize this"
+docker compose up -d searxng
+SEARXNG_URL=http://localhost:8080 bundle exec prescient search \
+  --format json "Ruby HTTP clients"
+```
+
+#### SearchApi
+
+[SearchApi](https://www.searchapi.io/) uses a hosted Google search engine and
+requires an API key:
+
+```yaml
+tools:
+  web_search:
+    type: searchapi
+    api_key_env: SEARCHAPI_API_KEY
+    engine: google
+    location: New York
+    hl: en
+    gl: us
+    timeout: 10
+    max_results: 5
+```
+
+The `engine` value can select another SearchApi web or product engine when its
+response uses `organic_results`, such as `bing`, `yahoo`, `yandex`,
+`amazon_search`, or `walmart_search`.
+
+Adapters can be grouped under one logical capability for ordered fallback:
+
+```yaml
+tools:
+  web_search:
+    adapters:
+      - type: searxng
+        url_env: SEARXNG_URL
+      - type: searchapi
+        api_key_env: SEARCHAPI_API_KEY
+        engine: google
+```
+
+Adapters are tried in order. Fallback is limited to transient connection and
+rate-limit failures; invalid configuration, authentication failures, and
+malformed responses are not retried with another adapter.
+
+Run the SearchApi-backed tool:
+
+```bash
+SEARCHAPI_API_KEY=your-key bundle exec prescient search \
+  --config prescient.yml --format json "Ruby HTTP clients"
+```
+
+Invoke a configured tool explicitly from Ruby or the CLI:
+
+```ruby
+result = Prescient.tool(:web_search).search('Ruby HTTP clients', limit: 3)
+```
+
+```bash
+bundle exec prescient search --config prescient.yml \
+  --tool web_search --format json "Ruby HTTP clients"
 ```
 
+Results use a normalized envelope containing `tool`, `query`, `source`, and
+`results` entries with `title`, `url`, `snippet`, and `source`. Requests have
+bounded query length, timeout, result count, and response size. Tool execution
+is explicit; Prescient does not autonomously invoke tools, and tool endpoints
+are not exposed as raw tool endpoints through `Prescient::API`; the API exposes
+the explicit combined search-and-generation operation below. Other adapters can
+implement the same contract without changing provider integrations.
+
+Search results are not sent to an AI provider by default. Opt in when you want
+the normalized results assembled as generation context:
+
+```ruby
+response = Prescient.search_and_generate(
+  'Ruby HTTP clients',
+  tool: :web_search,
+  provider: :openai,
+)
+```
+
+The CLI exposes the same opt-in behavior with `--generate`:
+
+```bash
+prescient search --generate --provider openai "Ruby HTTP clients"
+```
+
+The REST API exposes the same opt-in behavior:
+
+Raw normalized search results are available without generation:
+
+```bash
+curl -X POST http://localhost:9292/v1/search \
+  -H 'Content-Type: application/json' \
+  -d '{"query":"Ruby HTTP clients","limit":5}'
+```
+
+```bash
+curl -X POST http://localhost:9292/v1/search/generate \
+  -H 'Content-Type: application/json' \
+  -d '{"query":"Ruby HTTP clients","provider":"openai","limit":5}'
+```
+
+Use `fallback: false` to disable provider fallback for the request. The
+response is the normalized AI provider response; omit this endpoint and use
+`POST /v1/generate` when search context is not wanted.
+
 ### Programmatic Configuration
 
 ```ruby
@@ -602,6 +721,24 @@ client = Prescient.client(:customer_service)
 response = client.generate_response("What's your return policy?")
 ```
 
+Templates can also be configured in YAML or overridden for one CLI operation:
+
+```yaml
+providers:
+  openai:
+    type: openai
+    api_key_env: OPENAI_API_KEY
+    chat_model: gpt-4.1-mini
+    prompt_templates:
+      system_prompt: You are a concise assistant.
+      no_context_template: "%{system_prompt}\n\nUser: %{query}"
+      with_context_template: "%{system_prompt}\n\nContext:\n%{context}\n\nUser: %{query}"
+```
+
+```bash
+prescient generate --prompt-templates-file prompts.yml "Summarize this"
+```
+
 ### Template Placeholders
 
 - `%{system_prompt}` - The system/role instruction
@@ -1132,6 +1269,8 @@ The included `docker-compose.yml` provides:
 
 - **ollama**: Ollama AI service with persistent model storage
 - **ollama-init**: Automatically pulls required models on startup
+- **searxng**: Optional SearXNG web-search service with JSON output enabled
+- **postgres**: Optional PostgreSQL database with pgvector support
 - **redis**: Optional caching layer for embeddings
 - **prescient-app**: Example Ruby application container
 

data/docker-compose.yml

@@ -91,6 +91,26 @@ services:
       retries: 3
       start_period: 30s
 
+  # Optional SearXNG instance for the external web-search example.
+  searxng:
+    image: searxng/searxng:latest
+    container_name: prescient-searxng
+    ports:
+      - "8080:8080"
+    environment:
+      - SEARXNG_BASE_URL=http://localhost:8080/
+      - SEARXNG_SECRET=${SEARXNG_SECRET:-prescient-development-secret}
+    volumes:
+      - ./searxng/settings.yml:/etc/searxng/settings.yml:ro
+      - searxng_cache:/var/cache/searxng
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/')"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
+
   # Optional: Redis for caching embeddings (useful for development)
   redis:
     image: redis:7-alpine
@@ -145,6 +165,8 @@ volumes:
     driver: local
   redis_data:
     driver: local
+  searxng_cache:
+    driver: local
 
 networks:
   default:

data/examples/README.md

@@ -19,6 +19,10 @@ bundle install
   and similarity search.
 - `rest_api.ru` — a tiny Rack-compatible application that mounts
   `Prescient::API` and lists its endpoints at `/`.
+- `web_search.rb` — explicit SearXNG tool invocation with normalized JSON output.
+
+The same `web_search` capability can use SearchApi instead of SearXNG when the
+tool is configured with `type: searchapi` and `SEARCHAPI_API_KEY`.
 
 Run the REST API example with a Rack server such as `rackup`:
 
@@ -29,6 +33,17 @@ PRESCIENT_API_TOKEN=change-me BUNDLE_WITH=rack_example \
 curl http://localhost:9292/
 ```
 
+The endpoint catalog includes `POST /v1/search/generate`. With a configured
+SearXNG tool and AI provider, call it explicitly to feed search results into
+generation:
+
+```bash
+curl -X POST http://localhost:9292/v1/search/generate \
+  -H 'Authorization: Bearer change-me' \
+  -H 'Content-Type: application/json' \
+  -d '{"query":"Ruby HTTP clients","provider":"openai","limit":5}'
+```
+
 Running `bundle exec ruby examples/rest_api.ru` directly prints the same
 endpoint catalog without starting a server.
 
@@ -57,6 +72,25 @@ configuration. The scripts are demonstrations rather than isolated test
 fixtures; they may make real provider requests when the configured service is
 available.
 
+The web-search example requires a reachable SearXNG instance:
+
+```bash
+docker compose up -d searxng
+SEARXNG_URL=http://localhost:8080 bundle exec ruby examples/web_search.rb "Ruby HTTP clients"
+```
+
+The example returns normalized search results directly by default. Opt in to
+feeding those results to the configured AI provider with `--generate`:
+
+```bash
+SEARXNG_URL=http://localhost:8080 PRESCIENT_PROVIDER=openai \
+  bundle exec ruby examples/web_search.rb --generate "Ruby HTTP clients"
+```
+
+Omit `--generate` to keep the search results direct. `PRESCIENT_PROVIDER` is
+only used with `--generate` and may be omitted when the default provider is
+configured.
+
 See the [main README](../README.md) for configuration, fallback behavior,
 prompt templates, context exclusions, embeddings, and the public API. Rails
 applications can also use the [integration guide](../INTEGRATION_GUIDE.md),

data/examples/web_search.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require 'json'
+require_relative '../lib/prescient'
+
+generate = ARGV.delete('--generate')
+if ARGV.include?('--help')
+  puts 'Usage: ruby examples/web_search.rb [--generate] [QUERY]'
+  puts '  --generate  Feed normalized search results to the configured AI provider'
+  puts '  PRESCIENT_PROVIDER  Provider used with --generate (default: configured provider)'
+  exit
+end
+
+query = ARGV.empty? ? 'Ruby HTTP clients' : ARGV.join(' ')
+
+Prescient.configure do |config|
+  config.add_tool(
+    :web_search,
+    Prescient::Tool::SearXNG,
+    url: ENV.fetch('SEARXNG_URL', 'http://localhost:8080'),
+  )
+end
+
+result = if generate
+           Prescient.search_and_generate(
+             query,
+             provider: ENV['PRESCIENT_PROVIDER']&.to_sym,
+             limit:    20,
+           )
+         else
+           Prescient.tool(:web_search).search(query, limit: 20)
+         end
+
+puts JSON.pretty_generate(result)

data/lib/prescient/api.rb

@@ -27,6 +27,8 @@ class Prescient::API
     ['GET',  '/v1/capabilities']     => :capabilities_response,
     ['GET',  '/v1/health']           => :health_response,
     ['POST', '/v1/generate']         => :generate_response,
+    ['POST', '/v1/search']           => :search_response,
+    ['POST', '/v1/search/generate']  => :search_generate_response,
     ['POST', '/v1/embeddings']       => :embeddings_response,
     ['POST', '/v1/embeddings/batch'] => :batch_embeddings_response,
   }.freeze
@@ -88,6 +90,56 @@ class Prescient::API
     json_response(200, result, request_id)
   end
 
+  def search_generate_response(env, _query, request_id)
+    payload = request_payload(env)
+    query = required_string(payload, 'query')
+    tool = search_tool_name(payload)
+    fallback = search_fallback(payload)
+    limit = search_limit(payload)
+
+    result = Prescient.search_and_generate(
+      query,
+      tool:            tool,
+      provider:        payload['provider']&.to_sym,
+      limit:           limit,
+      enable_fallback: fallback,
+      **generation_options(payload),
+    )
+    json_response(200, result, request_id)
+  end
+
+  def search_response(env, _query, request_id)
+    payload = request_payload(env)
+    query = required_string(payload, 'query')
+    tool_name = search_tool_name(payload)
+    tool = Prescient.tool(tool_name)
+    raise Prescient::ToolConfigurationError, "tool not configured: #{tool_name}" unless tool
+
+    result = tool.search(query, limit: search_limit(payload))
+    json_response(200, result, request_id)
+  end
+
+  def search_tool_name(payload)
+    value = payload.fetch('tool', 'web_search')
+    raise ArgumentError, 'tool must be a non-empty string' unless value.is_a?(String) && !value.empty?
+
+    value.to_sym
+  end
+
+  def search_fallback(payload)
+    fallback = payload.key?('fallback') ? payload['fallback'] : true
+    raise ArgumentError, 'fallback must be boolean' unless [true, false].include?(fallback)
+
+    fallback
+  end
+
+  def search_limit(payload)
+    limit = payload['limit']
+    raise ArgumentError, 'limit must be a positive integer' if limit && (!limit.is_a?(Integer) || !limit.positive?)
+
+    limit
+  end
+
   def embeddings_response(env, _query, request_id)
     payload = request_payload(env)
     input = required_string(payload, 'input')