unique_web_search 2026.30.0.dev2__tar.gz → 2026.30.0.dev4__tar.gz

unique_web_search-2026.30.0.dev4/PKG-INFO

@@ -0,0 +1,146 @@
+Metadata-Version: 2.3
+Name: unique-web-search
+Version: 2026.30.0.dev4
+Summary: 
+Author: Andreas Hauri, Gustav Hartz, Rami Azouz
+Author-email: Andreas Hauri <andreas@unique.ch>, Gustav Hartz <gustav.hartz.ext@unique.ch>, Rami Azouz <rami.ext@unique.ch>
+License: Proprietary
+Requires-Dist: typing-extensions>=4.14.1,<5
+Requires-Dist: pydantic>=2.12.3,<3
+Requires-Dist: pydantic-settings>=2.10.1,<3
+Requires-Dist: timeout-decorator>=0.5.0,<1
+Requires-Dist: markdownify>=0.14.1,<1
+Requires-Dist: fake-useragent>=2.2.0,<3
+Requires-Dist: firecrawl>=3.3.2,<4
+Requires-Dist: tavily-python>=0.7.11,<1
+Requires-Dist: unidecode>=1.4.0,<2
+Requires-Dist: azure-ai-projects>=1.0.0,<2
+Requires-Dist: azure-identity>=1.25.0,<2
+Requires-Dist: unique-toolkit[monitoring]>=2026.30.0.dev20,<2026.30.0rc0
+Requires-Dist: azure-core>=1.36.0,<2
+Requires-Dist: google-cloud-aiplatform>=1.128.0,<2
+Requires-Dist: google-auth>=2.43.0,<3
+Requires-Dist: google-generativeai>=0.8.5,<0.9
+Requires-Dist: rapidfuzz>=3.0.0,<4
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: certifi>=2025.11.12
+Requires-Dist: jinja2>=3.1.6
+Requires-Dist: unique-search-proxy-core>=2026.30.0.dev5,<2026.30.0rc0
+Requires-Dist: unique-search-proxy-sdk>=2026.30.0.dev3,<2026.30.0rc0
+Requires-Python: >=3.12, <4.0
+Description-Content-Type: text/markdown
+
+# Unique Web Search
+
+Configurable web search tool for Unique assistants: search the web, read pages, and return citation-ready content chunks.
+
+## Search Proxy (recommended)
+
+Outbound web access should go through **Unique Search Proxy** (`connectors/unique_search_proxy/`). When `SEARCH_PROXY_BASE_URL` is set, search and crawl calls use `unique-search-proxy-sdk`; credentials and provider egress stay on the proxy pod.
+
+| Mode | When | Behaviour |
+|------|------|-----------|
+| **Proxy (recommended)** | `SEARCH_PROXY_BASE_URL` set | `_proxy_search` / `_proxy_crawl` via the SDK |
+| **Legacy** | URL unset | Direct provider calls from this process (`_legacy_search` / `_legacy_crawl`) |
+
+- **New standard engines (Brave, Perplexity) are proxy-only** — legacy raises.
+- **Custom API** always runs locally (never routed through the proxy).
+- Future provider work targets Search Proxy only; legacy is preserved for migrated engines.
+
+Packages involved: [`unique-search-proxy-core`](../../connectors/unique_search_proxy/unique_search_proxy_core/README.md) (schemas), [`unique-search-proxy-sdk`](../../connectors/unique_search_proxy/unique_search_proxy_sdk/README.md) (HTTP), [`unique-search-proxy`](../../connectors/unique_search_proxy/unique_search_proxy_client/README.md) (server). System overview: [`connectors/unique_search_proxy/README.md`](../../connectors/unique_search_proxy/README.md).
+
+## Architecture
+
+![Web Search Tool Architecture](docs/images/architecture-diagram.svg)
+
+## Search engine kinds
+
+### Standard search
+
+Query → normalised results (URL / title / snippet / optional content). Optional page crawl via the configured crawler.
+
+| Engine | Proxy | Notes |
+|--------|-------|-------|
+| Google | legacy + proxy | Requires scraping by default |
+| Brave | **proxy-only** | Rich snippets; `ExposableParam` knobs |
+| Perplexity | **proxy-only** | Content extraction knobs; `ExposableParam` |
+| Custom API | always local | Your REST endpoint |
+
+Standard engines that live in proxy-core use **`ExposableParam`** (`expose` + `value`) for optional provider knobs. See [Search Engines README](./src/unique_web_search/services/search_engine/README.md).
+
+### Agent search (grounding)
+
+An upstream AI agent / grounded model searches and returns opaque text that this tool parses into `WebSearchResult`s.
+
+| Engine | Proxy | Notes |
+|--------|-------|-------|
+| Bing (Azure AI Foundry grounding) | legacy + proxy | Configurable `requires_scraping` |
+| VertexAI (Gemini grounding) | legacy + proxy | Configurable `requires_scraping` |
+
+Agent engines do **not** use `ExposableParam`.
+
+## Web crawlers (page readers)
+
+| Crawler | Auth | Notes |
+|---------|------|-------|
+| Basic | none | HTTP + content-type toggles; tool-local `url_blocked_patterns` (legacy) |
+| Tavily | `TAVILY_API_KEY` | Extract API |
+| Jina | `JINA_API_KEY` | Reader API |
+| Firecrawl | `FIRECRAWL_API_KEY` | Batch scrape |
+
+No Crawl4AI. Crawler configs have **no exposable LLM knobs**.
+
+## Execution modes
+
+| Mode | Status | Behaviour |
+|------|--------|-----------|
+| **V1** | Stable | Refine query → search → crawl → process |
+| **V2** | Stable | Parallel plan steps (search / read URL) |
+| **V3** | Experimental | Agent loop: snippet `search` vs on-demand `read_urls` |
+
+Proxy vs legacy is transparent to executors.
+
+## Detailed subsystem docs
+
+- [Search Engines](./src/unique_web_search/services/search_engine/README.md)
+- [Crawlers](./src/unique_web_search/services/crawlers/README.md)
+- [Executors](./src/unique_web_search/services/executors/README.md)
+
+## Configuration
+
+- `SEARCH_PROXY_BASE_URL` — enables the Search Proxy client
+- Provider API keys (on the proxy pod when proxy is enabled; otherwise on assistants-core for legacy)
+- Search / crawler / content-processing settings via `WebSearchConfig`
+
+## Dependency management (uv.lock + min/latest testing)
+
+This package is a **library** and uses `uv` for dependency management.
+
+We run tests additionally with minimal dependencies to ensure that the listed ranges are valid. NOTE: We use lowest-direct, not lowest.
+Lowest attempts to use the lowest possible dependency versions _transitively_ causing issues if a dependency has incorrect metadata. Example:
+- google-cloud-aiplatform says it works with shapely<3.0.0.
+- The lowest resolver assumes 1.0 which needs python 2 -> breaks
+Therefore we use lowest-direct which only sets our direct dependencies to lowest. However, this only correctly verifies our min dependencies
+if our code correctly lists all the required dependencies and never imports a transitive dependency. We therefore use deptry to ensure we
+don't use transitive dependencies and that we have no unused dependencies.
+
+### Test locally
+
+- **Latest deps and deptry**:
+
+```bash
+cd tool_packages/unique_web_search
+uv sync
+uv run pytest
+uv run deptry
+```
+
+- **Min deps**:
+
+```bash
+cd tool_packages/unique_web_search
+uv venv
+uv pip install -e . --resolution=lowest-direct
+uv export --only-group dev --no-hashes | uv pip install -r -
+uv run --no-sync pytest
+```

unique_web_search-2026.30.0.dev4/README.md

@@ -0,0 +1,114 @@
+# Unique Web Search
+
+Configurable web search tool for Unique assistants: search the web, read pages, and return citation-ready content chunks.
+
+## Search Proxy (recommended)
+
+Outbound web access should go through **Unique Search Proxy** (`connectors/unique_search_proxy/`). When `SEARCH_PROXY_BASE_URL` is set, search and crawl calls use `unique-search-proxy-sdk`; credentials and provider egress stay on the proxy pod.
+
+| Mode | When | Behaviour |
+|------|------|-----------|
+| **Proxy (recommended)** | `SEARCH_PROXY_BASE_URL` set | `_proxy_search` / `_proxy_crawl` via the SDK |
+| **Legacy** | URL unset | Direct provider calls from this process (`_legacy_search` / `_legacy_crawl`) |
+
+- **New standard engines (Brave, Perplexity) are proxy-only** — legacy raises.
+- **Custom API** always runs locally (never routed through the proxy).
+- Future provider work targets Search Proxy only; legacy is preserved for migrated engines.
+
+Packages involved: [`unique-search-proxy-core`](../../connectors/unique_search_proxy/unique_search_proxy_core/README.md) (schemas), [`unique-search-proxy-sdk`](../../connectors/unique_search_proxy/unique_search_proxy_sdk/README.md) (HTTP), [`unique-search-proxy`](../../connectors/unique_search_proxy/unique_search_proxy_client/README.md) (server). System overview: [`connectors/unique_search_proxy/README.md`](../../connectors/unique_search_proxy/README.md).
+
+## Architecture
+
+![Web Search Tool Architecture](docs/images/architecture-diagram.svg)
+
+## Search engine kinds
+
+### Standard search
+
+Query → normalised results (URL / title / snippet / optional content). Optional page crawl via the configured crawler.
+
+| Engine | Proxy | Notes |
+|--------|-------|-------|
+| Google | legacy + proxy | Requires scraping by default |
+| Brave | **proxy-only** | Rich snippets; `ExposableParam` knobs |
+| Perplexity | **proxy-only** | Content extraction knobs; `ExposableParam` |
+| Custom API | always local | Your REST endpoint |
+
+Standard engines that live in proxy-core use **`ExposableParam`** (`expose` + `value`) for optional provider knobs. See [Search Engines README](./src/unique_web_search/services/search_engine/README.md).
+
+### Agent search (grounding)
+
+An upstream AI agent / grounded model searches and returns opaque text that this tool parses into `WebSearchResult`s.
+
+| Engine | Proxy | Notes |
+|--------|-------|-------|
+| Bing (Azure AI Foundry grounding) | legacy + proxy | Configurable `requires_scraping` |
+| VertexAI (Gemini grounding) | legacy + proxy | Configurable `requires_scraping` |
+
+Agent engines do **not** use `ExposableParam`.
+
+## Web crawlers (page readers)
+
+| Crawler | Auth | Notes |
+|---------|------|-------|
+| Basic | none | HTTP + content-type toggles; tool-local `url_blocked_patterns` (legacy) |
+| Tavily | `TAVILY_API_KEY` | Extract API |
+| Jina | `JINA_API_KEY` | Reader API |
+| Firecrawl | `FIRECRAWL_API_KEY` | Batch scrape |
+
+No Crawl4AI. Crawler configs have **no exposable LLM knobs**.
+
+## Execution modes
+
+| Mode | Status | Behaviour |
+|------|--------|-----------|
+| **V1** | Stable | Refine query → search → crawl → process |
+| **V2** | Stable | Parallel plan steps (search / read URL) |
+| **V3** | Experimental | Agent loop: snippet `search` vs on-demand `read_urls` |
+
+Proxy vs legacy is transparent to executors.
+
+## Detailed subsystem docs
+
+- [Search Engines](./src/unique_web_search/services/search_engine/README.md)
+- [Crawlers](./src/unique_web_search/services/crawlers/README.md)
+- [Executors](./src/unique_web_search/services/executors/README.md)
+
+## Configuration
+
+- `SEARCH_PROXY_BASE_URL` — enables the Search Proxy client
+- Provider API keys (on the proxy pod when proxy is enabled; otherwise on assistants-core for legacy)
+- Search / crawler / content-processing settings via `WebSearchConfig`
+
+## Dependency management (uv.lock + min/latest testing)
+
+This package is a **library** and uses `uv` for dependency management.
+
+We run tests additionally with minimal dependencies to ensure that the listed ranges are valid. NOTE: We use lowest-direct, not lowest.
+Lowest attempts to use the lowest possible dependency versions _transitively_ causing issues if a dependency has incorrect metadata. Example:
+- google-cloud-aiplatform says it works with shapely<3.0.0.
+- The lowest resolver assumes 1.0 which needs python 2 -> breaks
+Therefore we use lowest-direct which only sets our direct dependencies to lowest. However, this only correctly verifies our min dependencies
+if our code correctly lists all the required dependencies and never imports a transitive dependency. We therefore use deptry to ensure we
+don't use transitive dependencies and that we have no unused dependencies.
+
+### Test locally
+
+- **Latest deps and deptry**:
+
+```bash
+cd tool_packages/unique_web_search
+uv sync
+uv run pytest
+uv run deptry
+```
+
+- **Min deps**:
+
+```bash
+cd tool_packages/unique_web_search
+uv venv
+uv pip install -e . --resolution=lowest-direct
+uv export --only-group dev --no-hashes | uv pip install -r -
+uv run --no-sync pytest
+```

{unique_web_search-2026.30.0.dev2 → unique_web_search-2026.30.0.dev4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "unique_web_search"
-version = "2026.30.0.dev2"
+version = "2026.30.0.dev4"
 description = ""
 readme = "README.md"
 requires-python = ">=3.12,<4.0"
@@ -17,13 +17,12 @@ dependencies = [
     "timeout-decorator>=0.5.0,<1",
     "markdownify>=0.14.1,<1",
     "fake-useragent>=2.2.0,<3",
-    "crawl4ai>=0.6.3,<1",
     "firecrawl>=3.3.2,<4",
     "tavily-python>=0.7.11,<1",
     "unidecode>=1.4.0,<2",
     "azure-ai-projects>=1.0.0,<2",
     "azure-identity>=1.25.0,<2",
-    "unique-toolkit[monitoring]>=2026.30.0.dev12,<2026.30.0rc0",
+    "unique-toolkit[monitoring]>=2026.30.0.dev20,<2026.30.0rc0",
     "azure-core>=1.36.0,<2",
     "google-cloud-aiplatform>=1.128.0,<2",
     "google-auth>=2.43.0,<3",
@@ -32,9 +31,8 @@ dependencies = [
     "httpx>=0.28.1",
     "certifi>=2025.11.12",
     "jinja2>=3.1.6",
-    "playwright>=1.58.0",
-    "unique-search-proxy-core>=2026.30.0.dev2,<2026.30.0rc0",
-    "unique-search-proxy-sdk>=2026.30.0.dev2,<2026.30.0rc0",
+    "unique-search-proxy-core>=2026.30.0.dev5,<2026.30.0rc0",
+    "unique-search-proxy-sdk>=2026.30.0.dev3,<2026.30.0rc0",
 ]
 
 [build-system]

{unique_web_search-2026.30.0.dev2 → unique_web_search-2026.30.0.dev4}/src/unique_web_search/config.py

@@ -180,7 +180,7 @@ class WebSearchConfig(BaseToolConfig):
         default_factory=DefaultCrawler,  # type: ignore (This type is computed at runtime so pyright is not able to infer it)
         title="Web Page Reader",
         description="Choose and configure how web pages are fetched and converted to readable text.",
-        discriminator="crawler_type",
+        discriminator="crawler",
     )
 
     content_processor_config: ContentProcessorConfig = Field(

{unique_web_search-2026.30.0.dev2 → unique_web_search-2026.30.0.dev4}/src/unique_web_search/service.py

@@ -3,6 +3,7 @@ from datetime import datetime
 from time import time
 
 from typing_extensions import override
+from unique_search_proxy_core.context import RequestContext
 from unique_search_proxy_core.param_policy.exposed_params import ExposedParams
 from unique_search_proxy_core.search_engines.base import BaseSearchEngineConfig
 from unique_toolkit._common.chunk_relevancy_sorter.service import ChunkRelevancySorter
@@ -72,13 +73,22 @@ class WebSearchTool(Tool[WebSearchConfig]):
             language_model_orchestrator or configuration.token_counting_language_model
         )
 
+        self.chunk_relevancy_sorter = ChunkRelevancySorter(self.event)
+        self.company_id = self.event.company_id
+        self.request_context = RequestContext(
+            company_id=self.event.company_id,
+            user_id=self.event.user_id,
+            chat_id=self.event.payload.chat_id,
+        )
         self.search_engine_service = get_search_engine_service(
             self.config.search_engine_config,
             self.language_model_service,
+            request_context=self.request_context,
+        )
+        self.crawler_service = get_crawler_service(
+            self.config.crawler_config,
+            request_context=self.request_context,
         )
-        self.crawler_service = get_crawler_service(self.config.crawler_config)
-        self.chunk_relevancy_sorter = ChunkRelevancySorter(self.event)
-        self.company_id = self.event.company_id
         self.chat_history_token_length = 0
         self.chat_history_chat_messages = self._chat_service.get_full_history()
 

unique_web_search-2026.30.0.dev4/src/unique_web_search/services/crawlers/README.md

@@ -0,0 +1,110 @@
+# Crawlers
+
+Web-page readers used by Unique Web Search to fetch URLs and convert them to text.
+
+## Architecture
+
+Crawler configuration lives in **`unique_search_proxy_core.crawlers`**. The tool registers implementations via `CRAWLER_REGISTRY` and dispatches through `BaseCrawler`.
+
+```mermaid
+flowchart LR
+  Config["proxy-core *Config"]
+  Registry["CRAWLER_REGISTRY"]
+  Base["BaseCrawler"]
+  SDK["SDK crawl"]
+  Legacy["_legacy_crawl"]
+
+  Config --> Registry
+  Registry --> Base
+  Base -->|"SEARCH_PROXY_BASE_URL set"| SDK
+  Base -->|"unset"| Legacy
+```
+
+### Key invariants
+
+- **No exposable params.** Crawlers have no LLM override surface — only deployment defaults + `urls`.
+- **Proxy body = config fields + urls.** `BaseCrawler._proxy_crawl` dumps `config.model_dump(exclude={"crawler", ...})` into the SDK call.
+- **Legacy retained** when the search proxy is disabled.
+
+## Available crawlers
+
+| Name | Config source | Auth | Notes |
+|------|---------------|------|-------|
+| Basic | tool-local `BasicConfig` extends proxy-core | none | HTTP fetch + processors; `content_types` toggles; `url_blocked_patterns` (tool-local, stripped before proxy) |
+| Tavily | `TavilyConfig` | `TAVILY_API_KEY` | Extract API |
+| Jina | `JinaConfig` | `JINA_API_KEY` | Reader API |
+| Firecrawl | `FirecrawlConfig` | `FIRECRAWL_API_KEY` | Batch scrape |
+
+There is **no Crawl4AI** crawler.
+
+`WebSearchConfig.crawler_config` is a discriminated union on `crawler` (`Basic` / `Tavily` / `Jina` / `Firecrawl`).
+
+---
+
+## Basic
+
+| Field | Default | Notes |
+|-------|---------|-------|
+| `timeout` | from base | Per-request timeout (seconds) |
+| `content_types` | html/xhtml/plain/markdown on; pdf off | Which MIME types to process |
+| `max_concurrent_requests` | `10` | Parallelism |
+| `url_blocked_patterns` | tool-local | Regex URL blocklist; **not** sent to the proxy |
+
+```python
+from unique_search_proxy_core.crawlers import CrawlerType
+from unique_web_search.services.crawlers import BasicConfig, get_crawler_service
+
+config = BasicConfig(crawler=CrawlerType.BASIC, timeout=30)
+markdowns = await get_crawler_service(config).crawl(["https://example.com"])
+```
+
+---
+
+## Tavily
+
+| Field | Default | Notes |
+|-------|---------|-------|
+| `extract_depth` | `advanced` | `basic` / `advanced` |
+| `format` | `markdown` | `markdown` / `text` |
+| `query` | unset | Optional rerank intent |
+| `chunks_per_source` | unset | Requires `query` |
+| `include_images` / `include_favicon` / `include_usage` | `false` | Response extras |
+
+---
+
+## Jina
+
+| Field | Default | Notes |
+|-------|---------|-------|
+| `return_format` | `markdown` | `markdown` / `html` / `text` / … |
+| `engine` | `browser` | `auto` / `browser` / `direct` / `cf-browser-rendering` |
+| `page_timeout` | unset | Seconds; falls back to crawl `timeout` |
+| `max_concurrent_requests` | `10` | |
+| `no_cache` | `false` | |
+| `do_not_track` | `true` | |
+| selectors / summaries | unset / false | `target_selector`, `wait_for_selector`, `remove_selector`, image/link options, `locale`, `referer`, `proxy_url` |
+
+---
+
+## Firecrawl
+
+| Field | Default | Notes |
+|-------|---------|-------|
+| `only_main_content` | `true` | Strip chrome before markdown |
+| `only_clean_content` | `false` | Extra LLM clean pass |
+| `max_concurrency` | unset | Batch parallelism |
+| `ignore_invalid_urls` | `true` | |
+| `wait_for` | `0` | ms delay before fetch |
+| `mobile` | `false` | |
+| `block_ads` | `true` | |
+| `remove_base64_images` | `true` | |
+| `proxy` | `auto` | Firecrawl proxy tier |
+| `include_tags` / `exclude_tags` / `scrape_headers` / `max_age` | unset | Optional |
+
+---
+
+## Adding a crawler
+
+1. Define config (+ request model) in `unique_search_proxy_core.crawlers.<name>`.
+2. Add a tool module with `@register_crawler(...)` implementing `_legacy_crawl`.
+3. Modules are auto-discovered; keep static unions in `__init__.py` and cover with `tests/test_crawler_registry.py`.

unique_web_search-2026.30.0.dev4/src/unique_web_search/services/crawlers/__init__.py

@@ -0,0 +1,60 @@
+from pydantic import BaseModel
+from unique_search_proxy_core.crawlers import CrawlerType
+from unique_search_proxy_core.crawlers.firecrawl.schema import FirecrawlConfig
+from unique_search_proxy_core.crawlers.jina.schema import JinaConfig
+from unique_search_proxy_core.crawlers.tavily.schema import TavilyConfig
+
+from unique_web_search.services.crawlers.base import BaseCrawler
+from unique_web_search.services.crawlers.registry import (
+    CRAWLER_REGISTRY,
+    get_crawler_service,
+)
+
+CRAWLER_REGISTRY.autodiscover(
+    __path__,
+    __name__,
+    exclude=frozenset({"url_safety", "utils", "registry"}),
+)
+
+from unique_web_search.services.crawlers.basic import (  # noqa: E402
+    BasicConfig,
+    BasicCrawler,
+)
+from unique_web_search.services.crawlers.firecrawl import (  # noqa: E402
+    FirecrawlCrawler,
+)
+from unique_web_search.services.crawlers.jina import JinaCrawler  # noqa: E402
+from unique_web_search.services.crawlers.tavily import TavilyCrawler  # noqa: E402
+
+CrawlerTypes = BasicCrawler | FirecrawlCrawler | JinaCrawler | TavilyCrawler
+
+CrawlerConfigTypes = BasicConfig | FirecrawlConfig | JinaConfig | TavilyConfig
+
+CRAWLER_NAME_TO_CONFIG = CRAWLER_REGISTRY.name_to_config()
+
+
+def get_crawler_config_types_from_names(crawler_names: list[str]) -> type[BaseModel]:
+    return CRAWLER_REGISTRY.config_types_from_names(crawler_names)
+
+
+def get_default_crawler_config(crawler_names: list[str]) -> type[BaseModel]:
+    return CRAWLER_REGISTRY.default_config(crawler_names)
+
+
+__all__ = [
+    "BasicCrawler",
+    "BasicConfig",
+    "FirecrawlCrawler",
+    "FirecrawlConfig",
+    "JinaCrawler",
+    "JinaConfig",
+    "TavilyCrawler",
+    "TavilyConfig",
+    "CrawlerType",
+    "BaseCrawler",
+    "get_crawler_service",
+    "CrawlerConfigTypes",
+    "CrawlerTypes",
+    "get_crawler_config_types_from_names",
+    "get_default_crawler_config",
+]

unique_web_search-2026.30.0.dev4/src/unique_web_search/services/crawlers/base.py

@@ -0,0 +1,70 @@
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from typing import Generic, TypeVar
+
+from unique_search_proxy_core.context import LOCAL_REQUEST_CONTEXT, RequestContext
+from unique_search_proxy_core.crawlers.base import BaseCrawlerConfig
+from unique_search_proxy_core.url_safety import (
+    CrawlTargetValidationError,
+    ResolvedCrawlTarget,
+    UrlSafetyService,
+)
+
+from unique_web_search.metrics import crawl_blocked
+from unique_web_search.services.proxy.bridge import (
+    open_search_proxy_client,
+    search_proxy_client_enabled,
+)
+from unique_web_search.services.proxy.mappers import map_crawl_response
+
+CrawlerConfig = TypeVar(
+    "CrawlerConfig",
+    bound=BaseCrawlerConfig,
+)
+
+
+class BaseCrawler(ABC, Generic[CrawlerConfig]):
+    """Base class for web-page crawlers.
+
+    The search-proxy path is implemented entirely here. Subclasses only provide
+    the direct ``_legacy_crawl`` implementation used when the proxy is disabled.
+    """
+
+    def __init__(
+        self,
+        config: CrawlerConfig,
+        *,
+        request_context: RequestContext = LOCAL_REQUEST_CONTEXT,
+    ):
+        self.config = config
+        self._request_context = request_context
+
+    async def crawl(self, urls: list[str]) -> list[str]:
+        if search_proxy_client_enabled:
+            return await self._proxy_crawl(urls)
+
+        try:
+            targets = await UrlSafetyService.validate_batch_urls(urls)
+        except CrawlTargetValidationError as exc:
+            for target in exc.blocked_targets:
+                crawl_blocked.labels(reason_category=target.category).inc()
+            raise
+        return await self._legacy_crawl(targets)
+
+    async def _proxy_crawl(self, urls: list[str]) -> list[str]:
+        """Dump deployment config fields into the generic proxy crawl call."""
+        params = self.config.model_dump(exclude={"crawler"}, exclude_none=True)
+        async with open_search_proxy_client(
+            timeout=float(self.config.timeout),
+            context=self._request_context,
+        ) as client:
+            response = await client.crawl.crawl(
+                urls=urls,
+                crawler=self.config.crawler,
+                **params,
+            )
+        return map_crawl_response(response, urls)
+
+    @abstractmethod
+    async def _legacy_crawl(self, targets: list[ResolvedCrawlTarget]) -> list[str]: ...