undetecta 0.1.0__tar.gz

undetecta-0.1.0/.gitignore

@@ -0,0 +1,55 @@
+# Dependencies
+node_modules/
+.pnpm-store/
+
+# Build outputs
+dist/
+build/
+
+# Python
+__pycache__/
+*.py[cod]
+.venv/
+*.egg-info/
+.pytest_cache/
+.coverage
+htmlcov/
+.ruff_cache/
+.mypy_cache/
+
+# Environment
+.env
+.env.local
+.env.prod
+.env.*.local
+
+# IDE
+.idea/
+.vscode/
+*.code-workspace
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Turbo
+.turbo/
+
+# Logs
+*.log
+logs/
+
+# Temp
+tmp/
+.tmp/
+*.tmp
+
+# Project specific
+browser_profiles/
+screenshots/
+.browser/
+user_data_dir/
+
+# Git worktrees
+.worktrees/
+worktrees/

undetecta-0.1.0/DESIGN.md

@@ -0,0 +1,581 @@
+# Undetecta Python SDK Architecture Design
+
+## Overview
+
+The `undetecta` Python package provides a type-safe, ergonomic interface for the Undetecta API. It mirrors the JavaScript SDK's API surface while following Python best practices (PEP 517, PEP 621, type hints).
+
+## Design Principles
+
+1. **Type Safety**: Full type hints using Python's `typing` module with Pydantic for runtime validation
+2. **Ergonomics**: Simple, intuitive API that follows common Python client patterns (requests, httpx)
+3. **Async Support**: First-class async support with httpx for modern Python applications
+4. **Error Handling**: Structured exception hierarchy for predictable error handling
+5. **Modern Packaging**: `pyproject.toml` with PEP 517/621 compliance
+6. **Minimal Dependencies**: Only essential dependencies (httpx, pydantic)
+
+## API Surface Analysis
+
+The Undetecta API has the following public endpoints:
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| POST | `/v1/scrape` | Scrape a URL (sync response) |
+| POST | `/v1/search` | Web search (sync response) |
+| GET | `/v1/health` | Health check |
+
+**Note**: Admin endpoints (users, API keys, proxies) are internal and not exposed in the public client.
+
+### Request/Response Pattern
+
+All responses follow the `result()` wrapper pattern:
+
+```python
+# Success response
+{"success": True, "data": T}
+
+# Error response
+{"success": False, "error": {"code": str, "message": str}}
+```
+
+### Authentication
+
+Authentication is via `x-api-key` header.
+
+## Package Structure
+
+```
+undetecta/
+├── pyproject.toml         # PEP 621 package configuration
+├── README.md              # Package documentation
+├── LICENSE                # MIT License
+├── src/
+│   └── undetecta/
+│       ├── __init__.py            # Main entry point, exports
+│       ├── _client.py             # UndetectaClient class
+│       ├── _types.py              # Type definitions and Pydantic models
+│       ├── _errors.py             # Exception classes
+│       ├── _scrape.py             # Scrape API methods
+│       ├── _search.py             # Search API methods
+│       └── _transport/
+│           ├── __init__.py
+│           ├── _http.py           # HTTP transport using httpx
+│           └── _retry.py          # Retry logic with exponential backoff
+├── tests/
+│   ├── __init__.py
+│   ├── test_client.py
+│   ├── test_scrape.py
+│   ├── test_search.py
+│   └── test_errors.py
+├── .gitignore
+├── .python-version          # Python version (3.10+)
+└── CHANGELOG.md
+```
+
+**Note**: Leading underscores on modules (`_client.py`) indicate private/internal modules. The public API is exported through `__init__.py`.
+
+## Client Architecture
+
+### Main Client Class
+
+```python
+class UndetectaClient:
+    """Undetecta API client."""
+
+    def __init__(
+        self,
+        *,
+        api_key: str,
+        base_url: str = "https://api.undetecta.com",
+        timeout: float = 60.0,
+        max_retries: int = 3,
+    ) -> None: ...
+
+    # Core methods
+    async def scrape(self, request: ScrapeRequest) -> ScrapeResponse: ...
+    async def search(self, request: SearchRequest) -> SearchResponse: ...
+    async def health(self) -> HealthResponse: ...
+
+    # Sync variants (optional, convenience)
+    def scrape_sync(self, request: ScrapeRequest) -> ScrapeResponse: ...
+    def search_sync(self, request: SearchRequest) -> SearchResponse: ...
+
+    # Configuration (read-only)
+    @property
+    def api_key(self) -> str: ...
+    @property
+    def base_url(self) -> str: ...
+    @property
+    def timeout(self) -> float: ...
+    @property
+    def max_retries(self) -> int: ...
+```
+
+### Synchronous Support
+
+For users who prefer synchronous code, the SDK provides blocking methods via `httpx.Client`:
+
+```python
+# Using async (default)
+client = UndetectaClient(api_key="sk_...")
+result = await client.scrape(ScrapeRequest(url="https://example.com"))
+
+# Using sync convenience methods
+result = client.scrape_sync(ScrapeRequest(url="https://example.com"))
+```
+
+### Type System
+
+All types use Python's `typing` module with Pydantic for validation:
+
+```python
+# Enums (using StrEnum for Python 3.11+)
+class ScrapeFormat(StrEnum):
+    HTML = "html"
+    RAW_HTML = "rawHtml"
+    MARKDOWN = "markdown"
+    LINKS = "links"
+    SCREENSHOT = "screenshot"
+    BRANDING = "branding"
+
+class WaitUntil(StrEnum):
+    LOAD = "load"
+    DOM_CONTENT_LOADED = "domcontentloaded"
+    NETWORK_IDLE = "networkidle"
+
+class JobStatus(StrEnum):
+    PENDING = "pending"
+    RUNNING = "running"
+    COMPLETED = "completed"
+    FAILED = "failed"
+    STOPPED = "stopped"
+
+# Request models (Pydantic)
+class ScrapeRequest(BaseModel):
+    url: str
+    formats: list[ScrapeFormat] = [ScrapeFormat.MARKDOWN]
+    timeout: int = 30000
+    wait_until: WaitUntil = WaitUntil.DOM_CONTENT_LOADED
+    # ... 30+ other optional fields
+
+# Response models (Pydantic)
+class ScrapeMetadata(BaseModel):
+    status_code: int | None = None
+    url: str | None = None
+    title: str | None = None
+    description: str | None = None
+    # ...
+
+class ScrapeResponse(BaseModel):
+    id: str
+    status: JobStatus
+    created_at: str  # ISO 8601 datetime
+    completed_at: str | None = None
+    metadata: ScrapeMetadata | None = None
+    markdown: str | None = None
+    html: str | None = None
+    # ...
+```
+
+### Export Structure
+
+```python
+# src/undetecta/__init__.py
+"""Undetecta API Client."""
+
+from undetecta._client import UndetectaClient
+from undetecta._types import (
+    ScrapeFormat,
+    ScrapeRequest,
+    ScrapeResponse,
+    SearchRequest,
+    SearchResponse,
+    WaitUntil,
+    # ... other types
+)
+from undetecta._errors import (
+    UndetectaError,
+    ApiKeyError,
+    RateLimitError,
+    ValidationError,
+    NetworkError,
+    TimeoutError,
+    NotFoundError,
+    ServerError,
+)
+
+__all__ = [
+    # Client
+    "UndetectaClient",
+    # Types
+    "ScrapeFormat",
+    "ScrapeRequest",
+    "ScrapeResponse",
+    "SearchRequest",
+    "SearchResponse",
+    "WaitUntil",
+    # ...
+    # Errors
+    "UndetectaError",
+    "ApiKeyError",
+    "RateLimitError",
+    "ValidationError",
+    "NetworkError",
+    "TimeoutError",
+    "NotFoundError",
+    "ServerError",
+]
+```
+
+## Error Handling Strategy
+
+### Exception Hierarchy
+
+```python
+class UndetectaError(Exception):
+    """Base class for all Undetecta errors."""
+
+    def __init__(self, code: str, message: str, status_code: int | None = None) -> None: ...
+
+class ApiKeyError(UndetectaError):
+    """Raised when API key is invalid or missing (401)."""
+
+class RateLimitError(UndetectaError):
+    """Raised when rate limit is exceeded (429)."""
+
+class ValidationError(UndetectaError):
+    """Raised when request validation fails (400)."""
+
+class NotFoundError(UndetectaError):
+    """Raised when a resource is not found (404)."""
+
+class ServerError(UndetectaError):
+    """Raised when the server returns a 5xx error."""
+
+class NetworkError(UndetectaError):
+    """Raised when network request fails."""
+
+class TimeoutError(UndetectaError):
+    """Raised when request times out."""
+```
+
+### Error Handling Pattern
+
+```python
+from undetecta import UndetectaClient, ApiKeyError, RateLimitError
+
+client = UndetectaClient(api_key="sk_...")
+
+try:
+    result = await client.scrape(ScrapeRequest(url="https://example.com"))
+except ApiKeyError as e:
+    print(f"Invalid API key: {e.message}")
+except RateLimitError as e:
+    print(f"Rate limited, retry later: {e.message}")
+except ValidationError as e:
+    print(f"Invalid request: {e.message}")
+except UndetectaError as e:
+    print(f"API error: {e.code} - {e.message}")
+```
+
+## HTTP Transport
+
+### Using httpx
+
+The SDK uses `httpx` for HTTP requests:
+
+1. **Async-first**: Primary API uses `httpx.AsyncClient`
+2. **Sync fallback**: Sync variants use `httpx.Client`
+3. **Automatic retries**: Exponential backoff for 5xx and network errors
+4. **Timeout handling**: Per-request timeout with `httpx.Timeout`
+5. **Headers injection**: `x-api-key`, `Content-Type`, `User-Agent`
+
+```python
+import httpx
+
+class _HttpTransport:
+    """Internal HTTP transport with retry logic."""
+
+    async def _request(
+        self,
+        method: str,
+        url: str,
+        *,
+        json: dict[str, Any] | None = None,
+        max_retries: int = 3,
+    ) -> dict[str, Any]: ...
+```
+
+### Retry Strategy
+
+- **Retryable status codes**: 408, 429, 500-599
+- **Backoff**: Exponential with jitter (base: 1s, max: 30s)
+- **Max retries**: Configurable (default: 3)
+- **Non-retryable**: 4xx errors (except 429)
+
+## Configuration
+
+### pyproject.toml
+
+```toml
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "undetecta"
+version = "0.1.0"
+description = "Python SDK for the Undetecta API"
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [
+    { name = "Undetecta", email = "sdk@undetecta.com" }
+]
+keywords = ["scraping", "automation", "anti-detection"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+
+dependencies = [
+    "httpx>=0.27.0",
+    "pydantic>=2.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0.0",
+    "pytest-asyncio>=0.24.0",
+    "pytest-cov>=5.0.0",
+    "ruff>=0.8.0",
+    "mypy>=1.0.0",
+]
+
+[project.urls]
+Homepage = "https://undetecta.com"
+Documentation = "https://github.com/undetecta/sdk-python"
+Repository = "https://github.com/undetecta/sdk-python"
+Issues = "https://github.com/undetecta/sdk-python/issues"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/undetecta"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "W", "UP"]
+ignore = ["E501"]
+
+[tool.mypy]
+python_version = "3.10"
+strict = true
+warn_return_any = true
+warn_unused_ignores = true
+```
+
+## Usage Examples
+
+### Basic Scraping
+
+```python
+import asyncio
+from undetecta import UndetectaClient, ScrapeRequest, ScrapeFormat
+
+async def main():
+    client = UndetectaClient(api_key="sk_...")
+
+    result = await client.scrape(ScrapeRequest(
+        url="https://example.com",
+        formats=[ScrapeFormat.MARKDOWN]
+    ))
+
+    print(result.markdown)
+
+asyncio.run(main())
+```
+
+### Synchronous Usage
+
+```python
+from undetecta import UndetectaClient, ScrapeRequest
+
+client = UndetectaClient(api_key="sk_...")
+result = client.scrape_sync(ScrapeRequest(url="https://example.com"))
+print(result.markdown)
+```
+
+### Advanced Scraping
+
+```python
+result = await client.scrape(ScrapeRequest(
+    url="https://example.com",
+    formats=[ScrapeFormat.MARKDOWN, ScrapeFormat.SCREENSHOT, ScrapeFormat.LINKS],
+    screenshot_options=ScreenshotOptions(
+        full_page=True,
+        format="png"
+    ),
+    wait_for_selector=".main-content",
+    actions=[
+        Action(type="click", selector=".cookie-accept")
+    ]
+))
+```
+
+### Search
+
+```python
+result = await client.search(SearchRequest(
+    query="python web scraping",
+    limit=10,
+    sources=[SearchSource.WEB],
+    scrape_options=SearchScrapeOptions(
+        formats=[ScrapeFormat.MARKDOWN]
+    )
+))
+
+for item in result.web or []:
+    print(f"{item.title}: {item.url}")
+    if item.markdown:
+        print(item.markdown[:200])
+```
+
+### Custom Configuration
+
+```python
+client = UndetectaClient(
+    api_key="sk_...",
+    base_url="https://api.undetecta.loc:6363",  # Local development
+    timeout=120.0,
+    max_retries=5
+)
+```
+
+## Dependencies
+
+### Runtime
+- `httpx>=0.27.0`: Async HTTP client with sync support
+- `pydantic>=2.0.0`: Runtime validation and type definitions
+
+### Development
+- `pytest>=8.0.0`: Testing framework
+- `pytest-asyncio>=0.24.0`: Async test support
+- `pytest-cov>=5.0.0`: Coverage reporting
+- `ruff>=0.8.0`: Fast Python linter
+- `mypy>=1.0.0`: Static type checking
+
+## Type Definitions Mapping
+
+The Python SDK types mirror the JavaScript SDK and Zod schemas:
+
+| JavaScript/TS | Python | Pydantic |
+|---------------|--------|----------|
+| `type ScrapeFormat = ...` | `class ScrapeFormat(StrEnum)` | N/A |
+| `interface ScrapeOptions` | `class ScrapeRequest(BaseModel)` | `BaseModel` |
+| `interface ScrapeJobResponse` | `class ScrapeResponse(BaseModel)` | `BaseModel` |
+| `enum WaitUntil` | `class WaitUntil(StrEnum)` | N/A |
+| `type JobStatus = ...` | `class JobStatus(StrEnum)` | N/A |
+
+### Naming Conventions
+
+- **Request types**: `{Operation}Request` (e.g., `ScrapeRequest`)
+- **Response types**: `{Operation}Response` (e.g., `ScrapeResponse`)
+- **Enums**: `PascalCase` with `StrEnum` for value serialization
+- **Fields**: `snake_case` (Python convention)
+  - JS `waitForSelector` → Python `wait_for_selector`
+  - JS `onlyMainContent` → Python `only_main_content`
+
+## Publishing to PyPI
+
+### Release Process
+
+1. **Version bump**: Update `version` in `pyproject.toml`
+2. **Build**: `python -m build`
+3. **Check**: `twine check dist/*`
+4. **Upload**: `twine upload dist/*`
+
+### Trusted Publishers (Recommended)
+
+Configure PyPI to use trusted publishers (no tokens needed):
+
+```toml
+[tool.hatch.publish.index]
+disable = true
+
+# Or via PyPI dashboard: GitHub Actions -> OIDC
+```
+
+### CI/CD
+
+```yaml
+# .github/workflows/release.yml
+name: Release to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write  # Required for trusted publishing
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install build
+      - run: python -m build
+      - uses: pypa/gh-action-pypi-publish@release/v1
+```
+
+## Testing Strategy
+
+### Test Structure
+
+```python
+# tests/test_client.py
+import pytest
+from undetecta import UndetectaClient, ScrapeRequest
+from unittest.mock import AsyncMock, patch
+
+@pytest.mark.asyncio
+async def test_scrape_success():
+    client = UndetectaClient(api_key="test-key")
+    # Mock HTTP response
+    # Assert response structure
+```
+
+### Test Coverage
+
+- Unit tests for each module
+- HTTP client mocking
+- Error handling tests
+- Type validation tests
+
+## Implementation Order
+
+1. **Package setup**: `pyproject.toml`, directory structure
+2. **Type definitions**: `_types.py` with Pydantic models
+3. **Error classes**: `_errors.py`
+4. **HTTP transport**: `_transport/_http.py`, `_transport/_retry.py`
+5. **Client**: `_client.py`
+6. **API methods**: `_scrape.py`, `_search.py`
+7. **Entry point**: `__init__.py` with exports
+8. **Tests**: Unit tests for each module
+9. **Documentation**: README with examples
+
+## Future Enhancements (Out of Scope for MVP)
+
+1. Webhook support for async jobs
+2. Streaming responses
+3. Batch operations
+4. Context manager support
+5. OpenTelemetry integration
+6. Sphinx documentation