crawlora 1.5.0.dev1__tar.gz

crawlora-1.5.0.dev1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Crawlora
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

crawlora-1.5.0.dev1/MANIFEST.in

@@ -0,0 +1,2 @@
+recursive-include examples *.py
+recursive-include docs *.md

crawlora-1.5.0.dev1/PKG-INFO

@@ -0,0 +1,213 @@
+Metadata-Version: 2.4
+Name: crawlora
+Version: 1.5.0.dev1
+Summary: Python SDK for the public Crawlora API.
+Author: Crawlora
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Crawlora-org/crawlora-python-sdk
+Project-URL: Repository, https://github.com/Crawlora-org/crawlora-python-sdk
+Project-URL: Issues, https://github.com/Crawlora-org/crawlora-python-sdk/issues
+Keywords: crawlora,sdk,web-scraping,api-client
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typing-extensions>=4.8; python_version < "3.11"
+Provides-Extra: async
+Requires-Dist: httpx>=0.27; extra == "async"
+Dynamic: license-file
+
+# Crawlora Python SDK
+
+Python client for the public Crawlora API. Use it to call Crawlora scraping,
+search, marketplace, media, maps, finance, and usage endpoints with generated
+type stubs for editor and type-checker support.
+
+- Runtime: Python 3.10+
+- Auth: `x-api-key`
+- Default API base URL: `https://api.crawlora.net/api/v1`
+- Reference: [operations](docs/operations.md) and [recipes](docs/recipes.md)
+
+## Install
+
+The Python SDK is currently distributed from Git beta tags:
+
+```sh
+pip install "git+https://github.com/Crawlora-org/crawlora-python-sdk.git@latest"
+```
+
+For reproducible builds, pin a released tag:
+
+```sh
+pip install "git+https://github.com/Crawlora-org/crawlora-python-sdk.git@TAG"
+```
+
+## API Key
+
+Create or sign in to your Crawlora account at [crawlora.net](https://crawlora.net),
+then create an API key in the dashboard.
+
+```sh
+read -r CRAWLORA_API_KEY
+export CRAWLORA_API_KEY
+```
+
+## First Request
+
+```python
+import os
+from crawlora import CrawloraClient
+
+crawlora = CrawloraClient(api_key=os.environ["CRAWLORA_API_KEY"])
+
+response = crawlora.bing.search(
+    q="coffee shops",
+    count=10,
+)
+
+print(response["data"]["results"][0])
+```
+
+Endpoint groups are generated from the public API contract, so common calls are
+available as methods such as `crawlora.bing.search(...)`,
+`crawlora.youtube.transcript(...)`, and `crawlora.google.map_search(...)`.
+
+## Typed Dynamic Calls
+
+You can also call by operation id. Literal operation ids are covered by the
+generated `.pyi` stubs, so type checkers can infer the matching parameter and
+response aliases:
+
+```python
+response = crawlora.request("bing-search", {
+    "q": "coffee shops",
+    "count": 10,
+})
+```
+
+Generated stubs include operation ids, endpoint groups, keyword parameters,
+enum values, response aliases, and reserved request options.
+
+## Configuration
+
+```python
+crawlora = CrawloraClient(
+    api_key=os.environ["CRAWLORA_API_KEY"],
+    base_url="https://api.crawlora.net/api/v1",
+    timeout=30,
+    retries=2,
+    retry_delay=0.25,
+    headers={"x-client": "my-app"},
+)
+```
+
+Per-request options are available through reserved keyword arguments. Header
+names are matched case-insensitively, so request headers can override default
+auth, user-agent, and content headers without duplicating variants such as
+`x-api-key` and `X-API-KEY`:
+
+```python
+response = crawlora.bing.search(
+    q="coffee shops",
+    _timeout=10,
+    _headers={"x-request-id": "search-001"},
+)
+```
+
+## Text Responses
+
+Most endpoints return JSON. `_response_type` must be `auto`, `json`, or
+`text`. Endpoints that support alternate text output, such as YouTube
+transcripts, can opt into text mode:
+
+```python
+transcript = crawlora.youtube.transcript(
+    id="VIDEO_ID",
+    format="text",
+    _response_type="text",
+)
+
+print(transcript)
+```
+
+## Errors
+
+Failed API calls raise `CrawloraError`:
+
+```python
+from crawlora import CrawloraError
+
+try:
+    crawlora.bing.search(q="coffee shops")
+except CrawloraError as error:
+    print(error.status, error.code, error.body)
+    raise
+```
+
+The error includes `status`, optional API `code`, parsed `body`, `raw_body`,
+response `headers`, and the underlying parser or transport exception as
+`__cause__` when available. Retryable responses honor positive `Retry-After`
+headers, capped at 30 seconds. Timeout-like transport failures use the
+`Crawlora request timed out` SDK message.
+
+`CrawloraError` has three subclasses for branching on the failure kind:
+`CrawloraClientError` (4xx, request rejected), `CrawloraServerError` (5xx), and
+`CrawloraNetworkError` (transport failure or timeout before a response).
+
+## Async
+
+`AsyncCrawloraClient` mirrors the synchronous client for asyncio applications:
+
+```python
+from crawlora import AsyncCrawloraClient
+
+crawlora = AsyncCrawloraClient(api_key="YOUR_API_KEY")
+result = await crawlora.bing.search(q="coffee shops")
+```
+
+It reuses the same validation, retries, and `Retry-After` handling, running each
+request in a worker thread so the package stays dependency-free.
+
+## Pagination
+
+`client.paginate` yields successive pages, advancing the page/offset query
+parameter and stopping when a page returns no data:
+
+```python
+for page in crawlora.paginate("ebay-seller-feedback", {"seller": "acme"}):
+    for review in page["data"]:
+        print(review)
+```
+
+`AsyncCrawloraClient.paginate` is the `async for` equivalent. Override detection
+with `page_param`, `start`, `step`, and `max_pages`.
+
+## Examples
+
+Runnable examples live under `examples/` and skip cleanly when required
+environment variables are missing:
+
+```sh
+python3 examples/bing_search.py
+python3 examples/youtube_transcript.py
+```
+
+Set `CRAWLORA_BASE_URL` to point examples at a staging or local API.
+
+## Package Notes
+
+The import name is `crawlora`:
+
+```python
+from crawlora import CrawloraClient
+```
+
+The future PyPI package target is also `crawlora`, but registry publication is
+not enabled yet. Until then, install from an explicit Git beta tag or the
+moving `latest` tag as shown above.

crawlora-1.5.0.dev1/README.md

@@ -0,0 +1,188 @@
+# Crawlora Python SDK
+
+Python client for the public Crawlora API. Use it to call Crawlora scraping,
+search, marketplace, media, maps, finance, and usage endpoints with generated
+type stubs for editor and type-checker support.
+
+- Runtime: Python 3.10+
+- Auth: `x-api-key`
+- Default API base URL: `https://api.crawlora.net/api/v1`
+- Reference: [operations](docs/operations.md) and [recipes](docs/recipes.md)
+
+## Install
+
+The Python SDK is currently distributed from Git beta tags:
+
+```sh
+pip install "git+https://github.com/Crawlora-org/crawlora-python-sdk.git@latest"
+```
+
+For reproducible builds, pin a released tag:
+
+```sh
+pip install "git+https://github.com/Crawlora-org/crawlora-python-sdk.git@TAG"
+```
+
+## API Key
+
+Create or sign in to your Crawlora account at [crawlora.net](https://crawlora.net),
+then create an API key in the dashboard.
+
+```sh
+read -r CRAWLORA_API_KEY
+export CRAWLORA_API_KEY
+```
+
+## First Request
+
+```python
+import os
+from crawlora import CrawloraClient
+
+crawlora = CrawloraClient(api_key=os.environ["CRAWLORA_API_KEY"])
+
+response = crawlora.bing.search(
+    q="coffee shops",
+    count=10,
+)
+
+print(response["data"]["results"][0])
+```
+
+Endpoint groups are generated from the public API contract, so common calls are
+available as methods such as `crawlora.bing.search(...)`,
+`crawlora.youtube.transcript(...)`, and `crawlora.google.map_search(...)`.
+
+## Typed Dynamic Calls
+
+You can also call by operation id. Literal operation ids are covered by the
+generated `.pyi` stubs, so type checkers can infer the matching parameter and
+response aliases:
+
+```python
+response = crawlora.request("bing-search", {
+    "q": "coffee shops",
+    "count": 10,
+})
+```
+
+Generated stubs include operation ids, endpoint groups, keyword parameters,
+enum values, response aliases, and reserved request options.
+
+## Configuration
+
+```python
+crawlora = CrawloraClient(
+    api_key=os.environ["CRAWLORA_API_KEY"],
+    base_url="https://api.crawlora.net/api/v1",
+    timeout=30,
+    retries=2,
+    retry_delay=0.25,
+    headers={"x-client": "my-app"},
+)
+```
+
+Per-request options are available through reserved keyword arguments. Header
+names are matched case-insensitively, so request headers can override default
+auth, user-agent, and content headers without duplicating variants such as
+`x-api-key` and `X-API-KEY`:
+
+```python
+response = crawlora.bing.search(
+    q="coffee shops",
+    _timeout=10,
+    _headers={"x-request-id": "search-001"},
+)
+```
+
+## Text Responses
+
+Most endpoints return JSON. `_response_type` must be `auto`, `json`, or
+`text`. Endpoints that support alternate text output, such as YouTube
+transcripts, can opt into text mode:
+
+```python
+transcript = crawlora.youtube.transcript(
+    id="VIDEO_ID",
+    format="text",
+    _response_type="text",
+)
+
+print(transcript)
+```
+
+## Errors
+
+Failed API calls raise `CrawloraError`:
+
+```python
+from crawlora import CrawloraError
+
+try:
+    crawlora.bing.search(q="coffee shops")
+except CrawloraError as error:
+    print(error.status, error.code, error.body)
+    raise
+```
+
+The error includes `status`, optional API `code`, parsed `body`, `raw_body`,
+response `headers`, and the underlying parser or transport exception as
+`__cause__` when available. Retryable responses honor positive `Retry-After`
+headers, capped at 30 seconds. Timeout-like transport failures use the
+`Crawlora request timed out` SDK message.
+
+`CrawloraError` has three subclasses for branching on the failure kind:
+`CrawloraClientError` (4xx, request rejected), `CrawloraServerError` (5xx), and
+`CrawloraNetworkError` (transport failure or timeout before a response).
+
+## Async
+
+`AsyncCrawloraClient` mirrors the synchronous client for asyncio applications:
+
+```python
+from crawlora import AsyncCrawloraClient
+
+crawlora = AsyncCrawloraClient(api_key="YOUR_API_KEY")
+result = await crawlora.bing.search(q="coffee shops")
+```
+
+It reuses the same validation, retries, and `Retry-After` handling, running each
+request in a worker thread so the package stays dependency-free.
+
+## Pagination
+
+`client.paginate` yields successive pages, advancing the page/offset query
+parameter and stopping when a page returns no data:
+
+```python
+for page in crawlora.paginate("ebay-seller-feedback", {"seller": "acme"}):
+    for review in page["data"]:
+        print(review)
+```
+
+`AsyncCrawloraClient.paginate` is the `async for` equivalent. Override detection
+with `page_param`, `start`, `step`, and `max_pages`.
+
+## Examples
+
+Runnable examples live under `examples/` and skip cleanly when required
+environment variables are missing:
+
+```sh
+python3 examples/bing_search.py
+python3 examples/youtube_transcript.py
+```
+
+Set `CRAWLORA_BASE_URL` to point examples at a staging or local API.
+
+## Package Notes
+
+The import name is `crawlora`:
+
+```python
+from crawlora import CrawloraClient
+```
+
+The future PyPI package target is also `crawlora`, but registry publication is
+not enabled yet. Until then, install from an explicit Git beta tag or the
+moving `latest` tag as shown above.

crawlora-1.5.0.dev1/crawlora/__init__.py

@@ -0,0 +1,24 @@
+from .async_client import AsyncCrawloraClient
+from .client import (
+    VERSION,
+    CrawloraClient,
+    CrawloraClientError,
+    CrawloraError,
+    CrawloraNetworkError,
+    CrawloraServerError,
+)
+from .operations import GROUPS, OPERATION_COUNT, OPERATIONS, OperationId
+
+__all__ = [
+    "AsyncCrawloraClient",
+    "CrawloraClient",
+    "CrawloraError",
+    "CrawloraClientError",
+    "CrawloraServerError",
+    "CrawloraNetworkError",
+    "GROUPS",
+    "OPERATIONS",
+    "OPERATION_COUNT",
+    "OperationId",
+    "VERSION",
+]

crawlora-1.5.0.dev1/crawlora/_pagination.py

@@ -0,0 +1,44 @@
+"""Shared pagination helpers used by the sync and async clients.
+
+This module deliberately has no `.pyi` stub so type checkers read its inline
+annotations directly (the `client.pyi` stub shadows `client.py`).
+"""
+
+from __future__ import annotations
+
+from typing import Any, Mapping
+
+PAGE_PARAM_NAMES = ("page", "offset")
+
+
+def detect_page_param(operation: Mapping[str, Any]) -> str | None:
+    names = {parameter["name"] for parameter in operation.get("queryParams", [])}
+    for candidate in PAGE_PARAM_NAMES:
+        if candidate in names:
+            return candidate
+    return None
+
+
+def page_is_empty(response: Any) -> bool:
+    data = response
+    if isinstance(response, Mapping) and "data" in response:
+        data = response["data"]
+    if data is None:
+        return True
+    if isinstance(data, (list, tuple, dict, str)):
+        return len(data) == 0
+    return not data
+
+
+def default_start(page_param: str) -> int:
+    return 0 if page_param == "offset" else 1
+
+
+def default_items(response: Any) -> list[Any]:
+    """Default item extractor: the response's ``data`` list (Crawlora envelope),
+    or the response itself when it is already a list."""
+    if isinstance(response, Mapping) and isinstance(response.get("data"), list):
+        return list(response["data"])
+    if isinstance(response, list):
+        return list(response)
+    return []

crawlora-1.5.0.dev1/crawlora/_transport_sync.py

@@ -0,0 +1,114 @@
+"""Keep-alive HTTP transport for the synchronous client (standard library only).
+
+Maintains a small pool of reusable connections per ``(scheme, host, port)`` so
+the sync client avoids a fresh TCP + TLS handshake on every request. Each
+request checks out its own connection, so the transport is safe to use from
+multiple threads (e.g. under ``max_concurrency``). This module is stub-free so
+type checkers read its inline annotations directly.
+
+The transport returns a lightweight response object exposing ``status``,
+``headers`` (a dict), and ``body`` (bytes) — the only fields the client reads.
+"""
+
+from __future__ import annotations
+
+import http.client
+import threading
+from dataclasses import dataclass
+from typing import Any, Mapping
+from urllib.parse import urlsplit
+from urllib.request import Request
+
+
+def _title_case(name: str) -> str:
+    return "-".join(part.capitalize() for part in name.split("-"))
+
+
+@dataclass(frozen=True)
+class _PooledResponse:
+    status: int
+    headers: Mapping[str, str]
+    body: bytes
+
+
+class KeepAliveTransport:
+    """Connection-pooling transport. Drop-in for the urlopen transport: callable
+    as ``transport(request, timeout) -> response``."""
+
+    def __init__(self, max_per_host: int = 8) -> None:
+        self._lock = threading.Lock()
+        self._pools: dict[tuple, list[http.client.HTTPConnection]] = {}
+        self._max_per_host = max_per_host
+
+    def __call__(self, request: Request, timeout: float) -> _PooledResponse:
+        parts = urlsplit(request.full_url)
+        key = (parts.scheme, parts.hostname, parts.port)
+        path = parts.path or "/"
+        if parts.query:
+            path = f"{path}?{parts.query}"
+        method = request.get_method()
+        # Send canonical HTTP title-case header names (matching the urlopen
+        # transport's behavior), so receivers see e.g. "X-Api-Key".
+        headers = {_title_case(name): value for name, value in request.header_items()}
+        body = request.data
+
+        last_exc: Exception | None = None
+        for attempt in range(2):
+            conn = self._checkout(key, parts, timeout)
+            try:
+                conn.request(method, path, body=body, headers=headers)
+                response = conn.getresponse()
+                data = response.read()
+                result = _PooledResponse(response.status, dict(response.getheaders()), data)
+            except (http.client.HTTPException, ConnectionError, OSError) as exc:
+                # Likely a stale pooled connection the server already closed;
+                # discard it and retry once on a fresh connection.
+                last_exc = exc
+                self._close(conn)
+                if attempt == 1:
+                    raise
+                continue
+            if response.will_close:
+                self._close(conn)
+            else:
+                self._checkin(key, conn)
+            return result
+        raise last_exc if last_exc else RuntimeError("keep-alive transport failed")
+
+    def close(self) -> None:
+        with self._lock:
+            pools = list(self._pools.values())
+            self._pools.clear()
+        for pool in pools:
+            for conn in pool:
+                self._close(conn)
+
+    def _checkout(self, key: tuple, parts: Any, timeout: float) -> http.client.HTTPConnection:
+        with self._lock:
+            pool = self._pools.get(key)
+            if pool:
+                conn = pool.pop()
+                conn.timeout = timeout
+                return conn
+        return self._new(parts, timeout)
+
+    def _checkin(self, key: tuple, conn: http.client.HTTPConnection) -> None:
+        with self._lock:
+            pool = self._pools.setdefault(key, [])
+            if len(pool) < self._max_per_host:
+                pool.append(conn)
+                return
+        self._close(conn)
+
+    @staticmethod
+    def _new(parts: Any, timeout: float) -> http.client.HTTPConnection:
+        if parts.scheme == "https":
+            return http.client.HTTPSConnection(parts.hostname, parts.port or 443, timeout=timeout)
+        return http.client.HTTPConnection(parts.hostname, parts.port or 80, timeout=timeout)
+
+    @staticmethod
+    def _close(conn: http.client.HTTPConnection) -> None:
+        try:
+            conn.close()
+        except Exception:
+            pass