unique-search-proxy-core 2026.30.0.dev2__tar.gz → 2026.30.0.dev4__tar.gz

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: unique-search-proxy-core
-Version: 2026.30.0.dev2
+Version: 2026.30.0.dev4
 Summary: Shared Pydantic types for the Unique Search Proxy API
 Author: ThePhilAz
 Author-email: ThePhilAz <rami.azouz@philico.com>
@@ -8,7 +8,7 @@ Requires-Dist: httpx>=0.28.0,<0.29
 Requires-Dist: pydantic>=2.12.5,<3.0.0
 Requires-Dist: pydantic-settings>=2.10.1,<3
 Requires-Dist: pyhumps>=3.8.0,<4
-Requires-Dist: unique-toolkit[monitoring]>=2026.30.0.dev10,<2026.30.0rc0
+Requires-Dist: unique-toolkit[monitoring]>=2026.30.0.dev15,<2026.30.0rc0
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 
@@ -82,7 +82,7 @@ Admin JSON → GoogleConfig {expose, value}
 | `merge(overrides, *, query)` (instance) | search only | deployment defaults + LLM overrides + query → validated `request_model()` instance |
 | `provider_query_params(request)` (classmethod) | search only | request serialized for the upstream provider, minus `_provider_param_exclude_fields` (Google adds `search_engine_id`) |
 
-Agent engines (`BingAgentConfig`, …) and crawlers (`BasicConfig`, …) only implement `request_model()`; crawlers additionally keep `merge_crawler_config_and_invocation`.
+Agent engines (`BingAgentConfig`, …) and crawlers (`BasicConfig`, …) only implement `request_model()`. Crawlers have no exposable params and no `merge` — a crawl request is simply the deployment config fields plus `urls`.
 
 ### 3.2 ExposableParam — search-engine only
 
@@ -121,15 +121,6 @@ request = google_config.merge({"gl": "de"}, query="EU AI Act")
 
 Deactivated knobs (`value=None`) are dropped, overrides win over admin defaults, `engine` always comes from the config. The proxy receives a flat body; it does not resolve deployment config over HTTP.
 
-### 3.5 merge_crawler_config_and_invocation — crawler only
-
-```python
-from unique_search_proxy_core.crawlers import merge_crawler_config_and_invocation
-
-request = merge_crawler_config_and_invocation(basic_config, {"urls": ["https://example.com"]})
-# → validated BasicCrawlRequest ready for POST /v1/crawl
-```
-
 ---
 
 ## 4. Architecture (modules)
@@ -169,7 +160,7 @@ flowchart TB
 | `providers/schema.py` | JSON Schema + defaults for deployment UIs (`provider_config_json_schema`, …) |
 | `search_engines/` | Config models with the config-owned API (`request_model` / `exposed_params_model` / `merge` / `provider_query_params`), request union |
 | `agent_engines/` | Agent config/request models (`request_model`), output schema |
-| `crawlers/` | `*Config` deployment models + derived `*CrawlRequest` bodies, `merge_crawler_config_and_invocation` |
+| `crawlers/` | `*Config` deployment models + derived `*CrawlRequest` bodies (`request_model()`) |
 
 ---
 
@@ -234,7 +225,7 @@ from unique_search_proxy_core import (
 - Discriminated provider configs (`engine`, `crawler` Literal discriminators)
 - **Search-only:** `ExposableParam` policy; config-owned `request_model` / `exposed_params_model` / `merge` / `provider_query_params`
 - **Agent:** `BingAgentConfig.request_model()` (injects `query`; excludes `output_schema`)
-- **Crawl:** `BasicConfig.request_model()` (injects `urls`); `merge_crawler_config_and_invocation`
+- **Crawl:** `BasicConfig.request_model()` (injects `urls`); no exposable params / no merge
 - CamelCase JSON aliases on all models
 - Zero server dependencies (import-linter enforced in the client package)
 

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/README.md

@@ -68,7 +68,7 @@ Admin JSON → GoogleConfig {expose, value}
 | `merge(overrides, *, query)` (instance) | search only | deployment defaults + LLM overrides + query → validated `request_model()` instance |
 | `provider_query_params(request)` (classmethod) | search only | request serialized for the upstream provider, minus `_provider_param_exclude_fields` (Google adds `search_engine_id`) |
 
-Agent engines (`BingAgentConfig`, …) and crawlers (`BasicConfig`, …) only implement `request_model()`; crawlers additionally keep `merge_crawler_config_and_invocation`.
+Agent engines (`BingAgentConfig`, …) and crawlers (`BasicConfig`, …) only implement `request_model()`. Crawlers have no exposable params and no `merge` — a crawl request is simply the deployment config fields plus `urls`.
 
 ### 3.2 ExposableParam — search-engine only
 
@@ -107,15 +107,6 @@ request = google_config.merge({"gl": "de"}, query="EU AI Act")
 
 Deactivated knobs (`value=None`) are dropped, overrides win over admin defaults, `engine` always comes from the config. The proxy receives a flat body; it does not resolve deployment config over HTTP.
 
-### 3.5 merge_crawler_config_and_invocation — crawler only
-
-```python
-from unique_search_proxy_core.crawlers import merge_crawler_config_and_invocation
-
-request = merge_crawler_config_and_invocation(basic_config, {"urls": ["https://example.com"]})
-# → validated BasicCrawlRequest ready for POST /v1/crawl
-```
-
 ---
 
 ## 4. Architecture (modules)
@@ -155,7 +146,7 @@ flowchart TB
 | `providers/schema.py` | JSON Schema + defaults for deployment UIs (`provider_config_json_schema`, …) |
 | `search_engines/` | Config models with the config-owned API (`request_model` / `exposed_params_model` / `merge` / `provider_query_params`), request union |
 | `agent_engines/` | Agent config/request models (`request_model`), output schema |
-| `crawlers/` | `*Config` deployment models + derived `*CrawlRequest` bodies, `merge_crawler_config_and_invocation` |
+| `crawlers/` | `*Config` deployment models + derived `*CrawlRequest` bodies (`request_model()`) |
 
 ---
 
@@ -220,7 +211,7 @@ from unique_search_proxy_core import (
 - Discriminated provider configs (`engine`, `crawler` Literal discriminators)
 - **Search-only:** `ExposableParam` policy; config-owned `request_model` / `exposed_params_model` / `merge` / `provider_query_params`
 - **Agent:** `BingAgentConfig.request_model()` (injects `query`; excludes `output_schema`)
-- **Crawl:** `BasicConfig.request_model()` (injects `urls`); `merge_crawler_config_and_invocation`
+- **Crawl:** `BasicConfig.request_model()` (injects `urls`); no exposable params / no merge
 - CamelCase JSON aliases on all models
 - Zero server dependencies (import-linter enforced in the client package)
 

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "unique-search-proxy-core"
-version = "2026.30.0.dev2"
+version = "2026.30.0.dev4"
 description = "Shared Pydantic types for the Unique Search Proxy API"
 authors = [{ name = "ThePhilAz", email = "rami.azouz@philico.com" }]
 readme = "README.md"
@@ -10,7 +10,7 @@ dependencies = [
     "pydantic>=2.12.5,<3.0.0",
     "pydantic-settings>=2.10.1,<3",
     "pyhumps>=3.8.0,<4",
-    "unique-toolkit[monitoring]>=2026.30.0.dev10,<2026.30.0rc0",
+    "unique-toolkit[monitoring]>=2026.30.0.dev15,<2026.30.0rc0",
 
 ]
 

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/crawlers/__init__.py

@@ -17,7 +17,6 @@ from unique_search_proxy_core.crawlers.config_types import (
     parse_crawl_request,
     parse_crawler_config,
 )
-from unique_search_proxy_core.crawlers.params import merge_crawler_config_and_invocation
 
 __all__ = [
     "BaseCrawler",
@@ -31,7 +30,6 @@ __all__ = [
     "CrawlerType",
     "build_crawl_request_union",
     "crawler_config_from_request",
-    "merge_crawler_config_and_invocation",
     "parse_crawl_request",
     "parse_crawler_config",
 ]

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/crawlers/base.py

@@ -8,7 +8,7 @@ from pydantic import BaseModel, Field
 
 from unique_search_proxy_core.param_policy.derive import derive_request_model
 from unique_search_proxy_core.param_policy.request_base import CrawlRequestBase
-from unique_search_proxy_core.schema import CrawlUrlResult, deployment_model_config
+from unique_search_proxy_core.schema import CrawlUrlResult, camelized_model_config
 
 if TYPE_CHECKING:
     from httpx import AsyncClient
@@ -29,7 +29,12 @@ class CrawlerType(StrEnum):
 class BaseCrawlerConfig(BaseModel, Generic[CrawlerTypeT]):
     """Deployment config for a crawler; each crawler narrows ``crawler`` with a Literal."""
 
-    model_config = deployment_model_config
+    # Mirrors ``BaseSearchEngineConfig``: no ``extra='forbid'``. Emitting
+    # ``additionalProperties: false`` on every branch of the ``crawlerConfig``
+    # discriminated ``oneOf`` breaks the RJSF admin form, which carries sibling
+    # default fields across branches and then matches zero subschemas. Runtime
+    # ``/v1/crawl`` request bodies stay strict via ``CrawlRequestBase``.
+    model_config = camelized_model_config
 
     #: Name of the derived request model; set by every concrete crawler config.
     _request_model_name: ClassVar[str]

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/crawlers/basic/schema.py

@@ -1,8 +1,9 @@
 from __future__ import annotations
 
-from typing import ClassVar, Literal
+from typing import Annotated, ClassVar, Literal
 
 from pydantic import Field
+from unique_toolkit._common.pydantic.rjsf_tags import RJSFMetaTag
 
 from unique_search_proxy_core.crawlers.base import BaseCrawlerConfig, CrawlerType
 from unique_search_proxy_core.crawlers.basic.content_types import ContentTypeToggles
@@ -13,7 +14,13 @@ class BasicConfig(BaseCrawlerConfig[CrawlerType.BASIC]):
 
     _request_model_name: ClassVar[str] = "BasicCrawlRequest"
 
-    crawler: Literal[CrawlerType.BASIC] = CrawlerType.BASIC
+    crawler: Annotated[
+        Literal[CrawlerType.BASIC], RJSFMetaTag.SpecialWidget.hidden()
+    ] = Field(
+        default=CrawlerType.BASIC,
+        title="Crawler",
+        description="Provider discriminator; must be `Basic` for this config.",
+    )
 
     content_types: ContentTypeToggles = Field(
         default_factory=ContentTypeToggles,

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/crawlers/firecrawl/schema.py

@@ -1,20 +1,44 @@
 from __future__ import annotations
 
-from typing import ClassVar, Literal
+from typing import Annotated, ClassVar, Literal, TypeAlias
 
 from pydantic import Field
+from unique_toolkit._common.pydantic.rjsf_tags import RJSFMetaTag
 
 from unique_search_proxy_core.crawlers.base import BaseCrawlerConfig, CrawlerType
+from unique_search_proxy_core.schema import DeactivatedNone
 
 FirecrawlProxyMode = Literal["basic", "enhanced", "auto"]
 
+MaxConcurrencyOrNone: TypeAlias = (
+    Annotated[int, Field(title="Integer", ge=1)] | DeactivatedNone
+)
+TagListOrNone: TypeAlias = (
+    Annotated[
+        list[str],
+        Field(title="Tags"),
+        RJSFMetaTag({"ui:options": {"orderable": False}}),
+    ]
+    | DeactivatedNone
+)
+ScrapeHeadersOrNone: TypeAlias = (
+    Annotated[dict[str, str], Field(title="Headers")] | DeactivatedNone
+)
+MaxAgeOrNone: TypeAlias = Annotated[int, Field(title="Integer", ge=0)] | DeactivatedNone
+
 
 class FirecrawlConfig(BaseCrawlerConfig[CrawlerType.FIRECRAWL]):
     """Deployment config for Firecrawl (scrape or batch scrape)."""
 
     _request_model_name: ClassVar[str] = "FirecrawlCrawlRequest"
 
-    crawler: Literal[CrawlerType.FIRECRAWL] = CrawlerType.FIRECRAWL
+    crawler: Annotated[
+        Literal[CrawlerType.FIRECRAWL], RJSFMetaTag.SpecialWidget.hidden()
+    ] = Field(
+        default=CrawlerType.FIRECRAWL,
+        title="Crawler",
+        description="Provider discriminator; must be `Firecrawl` for this config.",
+    )
 
     only_main_content: bool = Field(
         default=True,
@@ -26,9 +50,8 @@ class FirecrawlConfig(BaseCrawlerConfig[CrawlerType.FIRECRAWL]):
         title="Only clean content",
         description="LLM pass to remove residual boilerplate from markdown.",
     )
-    max_concurrency: int | None = Field(
+    max_concurrency: MaxConcurrencyOrNone = Field(
         default=None,
-        ge=1,
         title="Max concurrency",
         description="Maximum concurrent scrapes for this batch job.",
     )
@@ -63,24 +86,23 @@ class FirecrawlConfig(BaseCrawlerConfig[CrawlerType.FIRECRAWL]):
         title="Proxy mode",
         description="Firecrawl proxy tier: `basic`, `enhanced`, or `auto`.",
     )
-    include_tags: list[str] | None = Field(
+    include_tags: TagListOrNone = Field(
         default=None,
         title="Include tags",
         description="HTML tags to include in scrape output.",
     )
-    exclude_tags: list[str] | None = Field(
+    exclude_tags: TagListOrNone = Field(
         default=None,
         title="Exclude tags",
         description="HTML tags to exclude from scrape output.",
     )
-    scrape_headers: dict[str, str] | None = Field(
+    scrape_headers: ScrapeHeadersOrNone = Field(
         default=None,
         title="Scrape headers",
         description="Headers sent to the target page (cookies, user-agent, etc.).",
     )
-    max_age: int | None = Field(
+    max_age: MaxAgeOrNone = Field(
         default=None,
-        ge=0,
         title="Max age",
         description=(
             "Return cached page content younger than this age in milliseconds."

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/crawlers/jina/schema.py

@@ -1,22 +1,46 @@
 from __future__ import annotations
 
-from typing import ClassVar, Literal
+from typing import Annotated, ClassVar, Literal, TypeAlias
 
 from pydantic import Field
+from unique_toolkit._common.pydantic.rjsf_tags import RJSFMetaTag
 
 from unique_search_proxy_core.crawlers.base import BaseCrawlerConfig, CrawlerType
+from unique_search_proxy_core.schema import DeactivatedNone
 
 JinaReturnFormat = Literal["markdown", "html", "text", "screenshot", "pageshot"]
 JinaEngine = Literal["auto", "browser", "direct", "cf-browser-rendering"]
 JinaRetainImages = Literal["none", "all", "alt", "all_p", "alt_p"]
 
+StrOrNone: TypeAlias = Annotated[str, Field(title="String")] | DeactivatedNone
+PageTimeoutOrNone: TypeAlias = (
+    Annotated[int, Field(title="Integer", ge=1, le=180)] | DeactivatedNone
+)
+CssSelectorListOrNone: TypeAlias = (
+    Annotated[
+        list[str],
+        Field(title="CSS Selectors"),
+        RJSFMetaTag({"ui:options": {"orderable": False}}),
+    ]
+    | DeactivatedNone
+)
+RetainImagesOrNone: TypeAlias = (
+    Annotated[JinaRetainImages, Field(title="Retain Images")] | DeactivatedNone
+)
+
 
 class JinaConfig(BaseCrawlerConfig[CrawlerType.JINA]):
     """Deployment config for the Jina Reader crawler."""
 
     _request_model_name: ClassVar[str] = "JinaCrawlRequest"
 
-    crawler: Literal[CrawlerType.JINA] = CrawlerType.JINA
+    crawler: Annotated[
+        Literal[CrawlerType.JINA], RJSFMetaTag.SpecialWidget.hidden()
+    ] = Field(
+        default=CrawlerType.JINA,
+        title="Crawler",
+        description="Provider discriminator; must be `Jina` for this config.",
+    )
 
     return_format: JinaReturnFormat = Field(
         default="markdown",
@@ -34,10 +58,8 @@ class JinaConfig(BaseCrawlerConfig[CrawlerType.JINA]):
             "`cf-browser-rendering` for JS-heavy sites."
         ),
     )
-    page_timeout: int | None = Field(
+    page_timeout: PageTimeoutOrNone = Field(
         default=None,
-        ge=1,
-        le=180,
         title="Page load timeout",
         description=(
             "Max seconds to wait for the page to load (Jina Reader). "
@@ -56,17 +78,17 @@ class JinaConfig(BaseCrawlerConfig[CrawlerType.JINA]):
         title="Bypass cache",
         description="Bypass Jina cache and fetch fresh content.",
     )
-    target_selector: list[str] | None = Field(
+    target_selector: CssSelectorListOrNone = Field(
         default=None,
         title="Target selector",
         description="CSS selectors to focus extraction on specific page elements.",
     )
-    wait_for_selector: list[str] | None = Field(
+    wait_for_selector: CssSelectorListOrNone = Field(
         default=None,
         title="Wait for selector",
         description="CSS selectors to wait for before returning content.",
     )
-    remove_selector: list[str] | None = Field(
+    remove_selector: CssSelectorListOrNone = Field(
         default=None,
         title="Remove selector",
         description="CSS selectors for page regions to strip (headers, footers, etc.).",
@@ -91,22 +113,22 @@ class JinaConfig(BaseCrawlerConfig[CrawlerType.JINA]):
         title="Include iframes",
         description="Include iframe content in the reader response.",
     )
-    retain_images: JinaRetainImages | None = Field(
+    retain_images: RetainImagesOrNone = Field(
         default=None,
         title="Retain images",
         description="Control which images are retained in markdown output.",
     )
-    locale: str | None = Field(
+    locale: StrOrNone = Field(
         default=None,
         title="Locale",
         description="Browser locale used to render the page (e.g. `en-US`).",
     )
-    referer: str | None = Field(
+    referer: StrOrNone = Field(
         default=None,
         title="Referer",
         description="Referer header sent when fetching the target URL.",
     )
-    proxy_url: str | None = Field(
+    proxy_url: StrOrNone = Field(
         default=None,
         title="Proxy URL",
         description="Proxy URL used by Jina Reader to access the target page.",

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/crawlers/tavily/schema.py

@@ -1,21 +1,34 @@
 from __future__ import annotations
 
-from typing import ClassVar, Literal
+from typing import Annotated, ClassVar, Literal, TypeAlias
 
 from pydantic import Field, model_validator
+from unique_toolkit._common.pydantic.rjsf_tags import RJSFMetaTag
 
 from unique_search_proxy_core.crawlers.base import BaseCrawlerConfig, CrawlerType
+from unique_search_proxy_core.schema import DeactivatedNone
 
 TavilyExtractDepth = Literal["basic", "advanced"]
 TavilyExtractFormat = Literal["markdown", "text"]
 
+StrOrNone: TypeAlias = Annotated[str, Field(title="String")] | DeactivatedNone
+ChunksPerSourceOrNone: TypeAlias = (
+    Annotated[int, Field(title="Integer", ge=1, le=5)] | DeactivatedNone
+)
+
 
 class TavilyConfig(BaseCrawlerConfig[CrawlerType.TAVILY]):
     """Deployment config for the Tavily Extract crawler."""
 
     _request_model_name: ClassVar[str] = "TavilyCrawlRequest"
 
-    crawler: Literal[CrawlerType.TAVILY] = CrawlerType.TAVILY
+    crawler: Annotated[
+        Literal[CrawlerType.TAVILY], RJSFMetaTag.SpecialWidget.hidden()
+    ] = Field(
+        default=CrawlerType.TAVILY,
+        title="Crawler",
+        description="Provider discriminator; must be `Tavily` for this config.",
+    )
 
     extract_depth: TavilyExtractDepth = Field(
         default="advanced",
@@ -30,7 +43,7 @@ class TavilyConfig(BaseCrawlerConfig[CrawlerType.TAVILY]):
         title="Output format",
         description="Extracted content format: `markdown` or `text`.",
     )
-    query: str | None = Field(
+    query: StrOrNone = Field(
         default=None,
         title="Rerank query",
         description=(
@@ -38,10 +51,8 @@ class TavilyConfig(BaseCrawlerConfig[CrawlerType.TAVILY]):
             "`chunks_per_source` may be used."
         ),
     )
-    chunks_per_source: int | None = Field(
+    chunks_per_source: ChunksPerSourceOrNone = Field(
         default=None,
-        ge=1,
-        le=5,
         title="Chunks per source",
         description=(
             "Max relevant chunks per URL when `query` is set (1–5). "

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/param_policy/request_base.py

@@ -9,10 +9,7 @@ from __future__ import annotations
 
 from pydantic import BaseModel, Field
 
-from unique_search_proxy_core.schema import (
-    camelized_model_config,
-    deployment_model_config,
-)
+from unique_search_proxy_core.schema import camelized_model_config
 
 
 class SearchRequestBase(BaseModel):
@@ -27,11 +24,7 @@ class AgentRequestBase(SearchRequestBase):
 
 
 class CrawlRequestBase(BaseModel):
-    """Base for derived ``POST /v1/crawl`` bodies: carries the required ``urls``.
+    """Base for derived ``POST /v1/crawl`` bodies: carries the required ``urls``."""
 
-    Uses ``deployment_model_config`` (``extra='forbid'``) to match the crawler
-    configs' schema (``additionalProperties: false``).
-    """
-
-    model_config = deployment_model_config
+    model_config = camelized_model_config
     urls: list[str] = Field(..., min_length=1, description="URLs to crawl")

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/schema.py

@@ -35,15 +35,6 @@ def get_model_config(title: str | None = None) -> ConfigDict:
 camelized_model_config = get_model_config()
 
 
-deployment_model_config = ConfigDict(
-    alias_generator=to_camel,
-    field_title_generator=field_title_generator,
-    model_title_generator=model_title_generator,
-    populate_by_name=True,
-    extra="forbid",
-)
-
-
 # Marks the deactivated branch of an optional config value. Locally declared so
 # core has no runtime dependency on unique-toolkit for this one-line alias.
 DeactivatedNone = Annotated[

{unique_search_proxy_core-2026.30.0.dev2 → unique_search_proxy_core-2026.30.0.dev4}/unique_search_proxy_core/url_safety/redirect.py

@@ -17,6 +17,13 @@ _LOGGER = logging.getLogger(__name__)
 
 _REDIRECT_STATUS_CODES = frozenset({301, 302, 303, 307, 308})
 
+# Browser-like UA avoids CDN bot stalls (e.g. Akamai) that make fail-closed
+# redirect probing falsely block legitimate public pages.
+_REDIRECT_PROBE_USER_AGENT = (
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
+    "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
+)
+
 ValidateUrlFn = Callable[[str], Awaitable[tuple[str, str] | None]]
 
 
@@ -32,7 +39,11 @@ async def resolve_redirect_chain(
     """
     current = url
     timeout = url_safety_settings.redirect_timeout_seconds
-    async with httpx.AsyncClient(follow_redirects=False, timeout=timeout) as client:
+    async with httpx.AsyncClient(
+        follow_redirects=False,
+        timeout=timeout,
+        headers={"User-Agent": _REDIRECT_PROBE_USER_AGENT},
+    ) as client:
         for _ in range(url_safety_settings.max_redirect_hops):
             error = await validate_url(current)
             if error is not None:

unique_search_proxy_core-2026.30.0.dev2/unique_search_proxy_core/crawlers/params.py

@@ -1,49 +0,0 @@
-"""Crawler request merge helpers (no HTTP)."""
-
-from __future__ import annotations
-
-from typing import Any
-
-from pydantic import BaseModel
-
-from unique_search_proxy_core.crawlers.base import BaseCrawlerConfig
-from unique_search_proxy_core.crawlers.config_types import crawler_config_from_request
-from unique_search_proxy_core.param_policy import URLS_FIELD
-
-CRAWLER_FIELD = "crawler"
-TIMEOUT_FIELD = "timeout"
-
-_DEPLOYMENT_DEFAULT_EXCLUDED_FIELDS = frozenset({CRAWLER_FIELD, URLS_FIELD})
-
-
-def crawler_config_defaults(config: BaseModel) -> dict[str, Any]:
-    """Deployment defaults merged into each flat crawl request."""
-    defaults: dict[str, Any] = {}
-    for field_name in type(config).model_fields:
-        if field_name in _DEPLOYMENT_DEFAULT_EXCLUDED_FIELDS:
-            continue
-        defaults[field_name] = getattr(config, field_name)
-    return defaults
-
-
-def merge_crawler_config_and_invocation(
-    config: BaseCrawlerConfig,
-    invocation: dict[str, Any],
-) -> BaseModel:
-    """Merge deployment config defaults with caller/LLM args into a flat crawl request."""
-    request_model = config.request_model()
-    defaults = crawler_config_defaults(config)
-    merged: dict[str, Any] = {**defaults, **invocation}
-    merged[CRAWLER_FIELD] = getattr(config, CRAWLER_FIELD)
-    request = request_model.model_validate(merged)
-    crawler_config_from_request(request)
-    return request
-
-
-__all__ = [
-    "CRAWLER_FIELD",
-    "TIMEOUT_FIELD",
-    "URLS_FIELD",
-    "crawler_config_defaults",
-    "merge_crawler_config_and_invocation",
-]